diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-12-20 16:37:47 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-12-20 16:37:47 +0300 |
| commit | aee0a117a889461ce8ced6fcf73207fe017f1d99 (patch) | |
| tree | 891d9ef189227a8445d83f35c1b0fc99573f4380 /doc | |
| parent | 8d46af3258650d305f53b819eabf7ab18d22f59e (diff) | |
Add latest changes from gitlab-org/gitlab@14-6-stable-eev14.6.0-rc42
Diffstat (limited to 'doc')
648 files changed, 14019 insertions, 9936 deletions
diff --git a/doc/.markdownlint/markdownlint-no-trailing-spaces.yml b/doc/.markdownlint/markdownlint-no-trailing-spaces.yml index 71903ae423d..8720fbafcb3 100644 --- a/doc/.markdownlint/markdownlint-no-trailing-spaces.yml +++ b/doc/.markdownlint/markdownlint-no-trailing-spaces.yml @@ -1,4 +1,4 @@ --- # Extended Markdown configuration to enforce no-trailing-spaces rule -"extends": "../../.markdownlint.yml" -"no-trailing-spaces": true +extends: "../../.markdownlint.yml" +no-trailing-spaces: true diff --git a/doc/.vale/gitlab/Dropdown.yml b/doc/.vale/gitlab/Dropdown.yml new file mode 100644 index 00000000000..691d44d1a48 --- /dev/null +++ b/doc/.vale/gitlab/Dropdown.yml @@ -0,0 +1,14 @@ +--- +# Suggestion: gitlab.Dropdown +# +# Catches many ways the phrase 'dropdown list' can be fumbled. +# +# For a list of all options, see https://errata-ai.github.io/vale/styles/ +extends: existence +message: 'Use "dropdown list".' +link: https://docs.gitlab.com/ee/development/documentation/styleguide/word_list.html#dropdown-list +level: suggestion +ignorecase: true +tokens: + - drop-down( [\w]*)? + - dropdown(?! list) diff --git a/doc/.vale/gitlab/ElementDescriptors.yml b/doc/.vale/gitlab/ElementDescriptors.yml index 254da16d00c..36f1202aef1 100644 --- a/doc/.vale/gitlab/ElementDescriptors.yml +++ b/doc/.vale/gitlab/ElementDescriptors.yml @@ -10,5 +10,4 @@ link: https://docs.gitlab.com/ee/development/documentation/styleguide/index.html level: suggestion ignorecase: true swap: - button: 'if possible, rewrite to not use' - area: 'use "section" instead of' + button: 'if possible, rewrite to remove' diff --git a/doc/.vale/gitlab/SubstitutionWarning.yml b/doc/.vale/gitlab/SubstitutionWarning.yml index 885b58cc937..fefc0f85cf4 100644 --- a/doc/.vale/gitlab/SubstitutionWarning.yml +++ b/doc/.vale/gitlab/SubstitutionWarning.yml @@ -21,6 +21,6 @@ swap: repo: repository timezone: time zone utilize: use - administrator access: the Administrator role - administrator permission: the Administrator role - administrator permissions: the Administrator role + administrator permission: the administrator access level + administrator permissions: the administrator access level + administrator role: the administrator access level diff --git a/doc/.vale/gitlab/Acronyms.yml b/doc/.vale/gitlab/Uppercase.yml index 23285fd0038..ae011748748 100644 --- a/doc/.vale/gitlab/Acronyms.yml +++ b/doc/.vale/gitlab/Uppercase.yml @@ -1,11 +1,11 @@ --- -# Warning: gitlab.Acronyms +# Warning: gitlab.Uppercase # -# Checks for unexpanded acronyms. +# Checks for use of all uppercase letters with unknown reason. # -# For a list of all options, see https://errata-ai.gitbook.io/vale/getting-started/styles +# For a list of all options, see https://docs.errata.ai/vale/styles. extends: conditional -message: '"%s" has no definition.' +message: "'%s' is uppercase. Use lowercase or `backticks` if possible. Otherwise add this word to the rule's exception list." link: https://about.gitlab.com/handbook/marketing/growth-marketing/content/editorial-team/#acronyms level: warning ignorecase: false diff --git a/doc/.vale/gitlab/spelling-exceptions.txt b/doc/.vale/gitlab/spelling-exceptions.txt index df228e61278..5ed8dc92249 100644 --- a/doc/.vale/gitlab/spelling-exceptions.txt +++ b/doc/.vale/gitlab/spelling-exceptions.txt @@ -128,7 +128,7 @@ crosslinked crosslinking crosslinks Crossplane -CrowdIn +Crowdin CSV cybersecurity Dangerfile diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 2062016ef03..06ad16bbcba 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -9,8 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan](https://about.gitlab.com/pricing/). -GitLab system administrators can also take advantage of the logs located on the -file system. See [the logs system documentation](logs.md#audit_jsonlog) for more details. +GitLab system administrators can also view all audit events by accessing the [`audit_json.log` file](logs.md#audit_jsonlog). You can: @@ -31,6 +30,11 @@ permission level, who added a new user, or who removed a user. - Track which users have access to a certain group of projects in GitLab, and who gave them that permission level. +## Retention policy + +There is no retention policy in place for audit events. +See the [Specify a retention period for audit events](https://gitlab.com/gitlab-org/gitlab/-/issues/8137) for more information. + ## List of events There are two kinds of events logged: @@ -97,7 +101,8 @@ From there, you can see the following actions: - 2FA enforcement or grace period changed. - Roles allowed to create project changed. - Group CI/CD variable added, removed, or protected status changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.3. -- Compliance framework created, updated, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340649) in GitLab 14.6. +- Compliance framework created, updated, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340649) in GitLab 14.5. +- Event streaming destination created, updated, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344664) in GitLab 14.6. Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events) @@ -128,6 +133,10 @@ From there, you can see the following actions: - Release was updated - Release milestone associations changed - Permission to approve merge requests by committers was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9) +- Permission to approve merge requests by committers was updated. + - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9. + - Message for event [changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72623/diffs) in GitLab 14.6. + - Permission to approve merge requests by authors was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9) - Number of required approvals was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9) - Added or removed users and groups from project approval groups ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213603) in GitLab 13.2) @@ -203,7 +212,7 @@ Events visible in Audit Events views until more events are logged. ### "Deleted User" events -Audit events can be created for a user after the user is deleted. The user name associated with the event is set to +Audit events can be created for a user after the user is deleted. The user name associated with the event is set to "Deleted User" because the actual user name is unknowable. For example, if a deleted user's access to a project is removed automatically due to expiration, the audit event is created for "Deleted User". We are [investigating](https://gitlab.com/gitlab-org/gitlab/-/issues/343933) whether this is avoidable. diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md index 14c48231a3d..5fa10c4c119 100644 --- a/doc/administration/auth/atlassian.md +++ b/doc/administration/auth/atlassian.md @@ -50,9 +50,10 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu gitlab_rails['omniauth_providers'] = [ { name: "atlassian_oauth2", + # label: "Provider name", # optional label for login button, defaults to "Atlassian" app_id: "YOUR_CLIENT_ID", app_secret: "YOUR_CLIENT_SECRET", - args: { scope: 'offline_access read:jira-user read:jira-work', prompt: 'consent' } + args: { scope: "offline_access read:jira-user read:jira-work", prompt: "consent" } } ] ``` @@ -60,10 +61,12 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu For installations from source: ```yaml - - name: "atlassian_oauth2", - app_id: "YOUR_CLIENT_ID", - app_secret: "YOUR_CLIENT_SECRET", - args: { scope: 'offline_access read:jira-user read:jira-work', prompt: 'consent' } + - { name: "atlassian_oauth2", + # label: "Provider name", # optional label for login button, defaults to "Atlassian" + app_id: "YOUR_CLIENT_ID", + app_secret: "YOUR_CLIENT_SECRET", + args: { scope: "offline_access read:jira-user read:jira-work", prompt: "consent" } + } ``` 1. Change `YOUR_CLIENT_ID` and `YOUR_CLIENT_SECRET` to the Client credentials you received in [application registration](#atlassian-application-registration) steps. diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md index 19ee143a72a..4220e552196 100644 --- a/doc/administration/auth/authentiq.md +++ b/doc/administration/auth/authentiq.md @@ -36,12 +36,13 @@ Authentiq generates a Client ID and the accompanying Client Secret for you to us ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "authentiq", - "app_id" => "YOUR_CLIENT_ID", - "app_secret" => "YOUR_CLIENT_SECRET", - "args" => { - "scope": 'aq:name email~rs address aq:push' - } + name: "authentiq", + # label: "Provider name", # optional label for login button, defaults to "Authentiq" + app_id: "YOUR_CLIENT_ID", + app_secret: "YOUR_CLIENT_SECRET", + args: { + "scope": 'aq:name email~rs address aq:push' + } } ] ``` @@ -50,6 +51,7 @@ Authentiq generates a Client ID and the accompanying Client Secret for you to us ```yaml - { name: 'authentiq', + # label: 'Provider name', # optional label for login button, defaults to "Authentiq" app_id: 'YOUR_CLIENT_ID', app_secret: 'YOUR_CLIENT_SECRET', args: { diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md index d137489a838..718a2919ed0 100644 --- a/doc/administration/auth/cognito.md +++ b/doc/administration/auth/cognito.md @@ -56,25 +56,25 @@ Include the code block in the `/etc/gitlab/gitlab.rb` file: gitlab_rails['omniauth_allow_single_sign_on'] = ['cognito'] gitlab_rails['omniauth_providers'] = [ { - "name" => "cognito", - # "label" => "Cognito", - # "icon" => nil, # Optional icon URL - "app_id" => "CLIENT ID", - "app_secret" => "CLIENT SECRET", - "args" => { - "scope" => "openid profile email", + name: "cognito", + label: "Provider name", # optional label for login button, defaults to "Cognito" + icon: nil, # Optional icon URL + app_id: "CLIENT ID", + app_secret: "CLIENT SECRET", + args: { + scope: "openid profile email", client_options: { - 'site' => 'https://your_domain.auth.your_region.amazoncognito.com', - 'authorize_url' => '/oauth2/authorize', - 'token_url' => '/oauth2/token', - 'user_info_url' => '/oauth2/userInfo' + site: "https://your_domain.auth.your_region.amazoncognito.com", + authorize_url: "/oauth2/authorize", + token_url: "/oauth2/token", + user_info_url: "/oauth2/userInfo" }, user_response_structure: { root_path: [], - id_path: ['sub'], - attributes: { nickname: 'email', name: 'email', email: 'email' } + id_path: ["sub"], + attributes: { nickname: "email", name: "email", email: "email" } }, - name: 'cognito', + name: "cognito", strategy_class: "OmniAuth::Strategies::OAuth2Generic" } } diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index 466e208a52e..265bba8a9b1 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -46,11 +46,12 @@ this provider also allows Crowd authentication for Git-over-https requests. ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "crowd", - "args" => { - "crowd_server_url" => "CROWD_SERVER_URL", - "application_name" => "YOUR_APP_NAME", - "application_password" => "YOUR_APP_PASSWORD" + name: "crowd", + # label: "Provider name", # optional label for login button, defaults to "Crowd" + args: { + crowd_server_url: "CROWD_SERVER_URL", + application_name: "YOUR_APP_NAME", + application_password: "YOUR_APP_PASSWORD" } } ] @@ -60,6 +61,7 @@ this provider also allows Crowd authentication for Git-over-https requests. ```yaml - { name: 'crowd', + # label: 'Provider name', # optional label for login button, defaults to "Crowd" args: { crowd_server_url: 'CROWD_SERVER_URL', application_name: 'YOUR_APP_NAME', diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index 26e523cb802..9298b04cbc1 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # JWT OmniAuth provider **(FREE SELF)** To enable the JWT OmniAuth provider, you must register your application with JWT. -JWT will provide you with a secret key for you to use. +JWT provides you with a secret key for you to use. 1. On your GitLab server, open the configuration file. @@ -32,14 +32,15 @@ JWT will provide you with a secret key for you to use. ```ruby gitlab_rails['omniauth_providers'] = [ - { name: 'jwt', + { name: "jwt", + label: "Provider name", # optional label for login button, defaults to "Jwt" args: { - secret: 'YOUR_APP_SECRET', - algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' - uid_claim: 'email', - required_claims: ['name', 'email'], - info_map: { name: 'name', email: 'email' }, - auth_url: 'https://example.com/', + secret: "YOUR_APP_SECRET", + algorithm: "HS256", # Supported algorithms: "RS256", "RS384", "RS512", "ES256", "ES384", "ES512", "HS256", "HS384", "HS512" + uid_claim: "email", + required_claims: ["name", "email"], + info_map: { name: "name", email: "email" }, + auth_url: "https://example.com/", valid_within: 3600 # 1 hour } } @@ -50,6 +51,7 @@ JWT will provide you with a secret key for you to use. ```yaml - { name: 'jwt', + label: 'Provider name', # optional label for login button, defaults to "Jwt" args: { secret: 'YOUR_APP_SECRET', algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' @@ -72,9 +74,9 @@ JWT will provide you with a secret key for you to use. installed GitLab via Omnibus or from source respectively. On the sign in page there should now be a JWT icon below the regular sign in form. -Click the icon to begin the authentication process. JWT will ask the user to +Click the icon to begin the authentication process. JWT asks the user to sign in and authorize the GitLab application. If everything goes well, the user -will be redirected to GitLab and will be signed in. +is redirected to GitLab and signed in. <!-- ## Troubleshooting diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 9047cfae1e9..f551c362784 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -23,7 +23,7 @@ Users added through LDAP: - Take a [licensed seat](../../../subscriptions/self_managed/index.md#billable-users). - Can authenticate with Git using either their GitLab username or their email and LDAP password, - even if password authentication for Git + even if password authentication for Git [is disabled](../../../user/admin_area/settings/sign_in_restrictions.md#password-authentication-enabled). The LDAP DN is associated with existing GitLab users when: @@ -41,7 +41,7 @@ If an existing GitLab user wants to enable LDAP sign-in for themselves, they sho GitLab has multiple mechanisms to verify a user is still active in LDAP. If the user is no longer active in LDAP, they are placed in an `ldap_blocked` status and are signed out. They are unable to sign in using any authentication provider until they are -reactivated in LDAP. +reactivated in LDAP. Users are considered inactive in LDAP when they: @@ -52,7 +52,8 @@ Users are considered inactive in LDAP when they: Status is checked for all LDAP users: -- When signing in using any authentication provider. +- When signing in using any authentication provider. [In GitLab 14.4 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/343298), status was + checked only when signing in using LDAP directly. - Once per hour for active web sessions or Git requests using tokens or SSH keys. - When performing Git over HTTP requests using LDAP username and password. - Once per day during [User Sync](ldap_synchronization.md#user-sync). @@ -221,6 +222,51 @@ These LDAP sync configuration settings are available: | `external_groups` | An array of CNs of groups containing users that should be considered external. Not `cn=interns` or the full DN. | **{dotted-circle}** No | `['interns', 'contractors']` | | `sync_ssh_keys` | The LDAP attribute containing a user's public SSH key. | **{dotted-circle}** No | `'sshPublicKey'` or false if not set | +### Use multiple LDAP servers **(PREMIUM SELF)** + +If you have users on multiple LDAP servers, you can configure GitLab to use them. To add additional LDAP servers: + +1. Duplicate the [`main` LDAP configuration](#configure-ldap). +1. Edit each duplicate configuration with the details of the additional servers. + - For each additional server, choose a different provider ID, like `main`, `secondary`, or `tertiary`. Use lowercase + alphanumeric characters. GitLab uses the provider ID to associate each user with a specific LDAP server. + - For each entry, use a unique `label` value. These values are used for the tab names on the sign-in page. + +#### Example of multiple LDAP servers + +The following example shows how to configure three LDAP servers in `gitlab.rb`: + +```ruby +gitlab_rails['ldap_enabled'] = true +gitlab_rails['ldap_servers'] = { +'main' => { + 'label' => 'GitLab AD', + 'host' => 'ad.example.org', + 'port' => 636, + ... + }, + +'secondary' => { + 'label' => 'GitLab Secondary AD', + 'host' => 'ad-secondary.example.net', + 'port' => 636, + ... + }, + +'tertiary' => { + 'label' => 'GitLab Tertiary AD', + 'host' => 'ad-tertiary.example.net', + 'port' => 636, + ... + } + +} +``` + +This example results in the following sign-in page: + + + ### Set up LDAP user filter To limit all GitLab access to a subset of the LDAP users on your LDAP server, first narrow the @@ -451,56 +497,6 @@ If initially your LDAP configuration looked like: 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. -## Multiple LDAP servers **(PREMIUM SELF)** - -With GitLab, you can configure multiple LDAP servers that your GitLab instance -connects to. - -To add another LDAP server: - -1. Duplicate the settings under [the main configuration](#configure-ldap). -1. Edit them to match the additional LDAP server. - -Be sure to choose a different provider ID made of letters a-z and numbers 0-9. -This ID is stored in the database so that GitLab can remember which LDAP -server a user belongs to. - - - -Based on the example illustrated on the image above, -our `gitlab.rb` configuration would look like: - -```ruby -gitlab_rails['ldap_enabled'] = true -gitlab_rails['ldap_servers'] = { -'main' => { - 'label' => 'GitLab AD', - 'host' => 'ad.example.org', - 'port' => 636, - ... - }, - -'secondary' => { - 'label' => 'GitLab Secondary AD', - 'host' => 'ad-secondary.example.net', - 'port' => 636, - ... - }, - -'tertiary' => { - 'label' => 'GitLab Tertiary AD', - 'host' => 'ad-tertiary.example.net', - 'port' => 636, - ... - } - -} -``` - -If you configure multiple LDAP servers, use a unique naming convention for the -`label` section of each entry. That label is used as the display name of the tab -shown on the sign-in page. - ## Disable anonymous LDAP authentication GitLab doesn't support TLS client authentication. Complete these steps on your LDAP server. diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index aa40060c4c1..63e4490e332 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -106,7 +106,7 @@ here are some questions to ask yourself: - Does the user pass through the [configured `user_filter`](index.md#set-up-ldap-user-filter)? If one is not configured, this question can be ignored. If it is, then the user must also pass through this filter to be allowed to sign in. - - Refer to our docs on [debugging the `user_filter`](#debug-ldap-user-filter). + - Refer to our documentation on [debugging the `user_filter`](#debug-ldap-user-filter). If the above are both okay, the next place to look for the problem is the logs themselves while reproducing the issue. @@ -316,7 +316,7 @@ LDAP search error: No Such Object User Update (0.4ms) UPDATE "users" SET "state" = $1, "updated_at" = $2 WHERE "users"."id" = $3 [["state", "ldap_blocked"], ["updated_at", "2019-10-18 15:46:22.902177"], ["id", 20]] ``` -Once the user is found in LDAP, the rest of the output updates the GitLab +After the user is found in LDAP, the rest of the output updates the GitLab database with any changes. #### Query a user in LDAP @@ -337,8 +337,8 @@ Gitlab::Auth::Ldap::Person.find_by_uid('<uid>', adapter) #### Membership(s) not granted Sometimes you may think a particular user should be added to a GitLab group via -LDAP group sync, but for some reason it's not happening. There are several -things to check to debug the situation. +LDAP group sync, but for some reason it's not happening. You can check several +things to debug the situation. - Ensure LDAP configuration has a `group_base` specified. [This configuration](ldap_synchronization.md#group-sync) is required for group sync to work properly. @@ -421,7 +421,7 @@ Started syncing 'ldapmain' provider for 'my_group' group ``` The following entry shows an array of all user DNs GitLab sees in the LDAP server. -These are the users for a single LDAP group, not a GitLab group. If +These DNs are the users for a single LDAP group, not a GitLab group. If you have multiple LDAP groups linked to this GitLab group, you see multiple log entries like this - one for each LDAP group. If you don't see an LDAP user DN in this log entry, LDAP is not returning the user when we do the lookup. @@ -545,7 +545,7 @@ updates the stored DN to the new value so both values now match what's in LDAP. If the email has changed and the DN has not, GitLab finds the user with -the DN and update its own record of the user's email to match the one in LDAP. +the DN and updates its own record of the user's email to match the one in LDAP. However, if the primary email _and_ the DN change in LDAP, then GitLab has no way of identifying the correct LDAP record of the user and, as a @@ -563,7 +563,7 @@ email address are removed first. This is because emails have to be unique in Git Go to the [rails console](#rails-console) and then run: ```ruby -# Each entry will have to include the old username and the new email +# Each entry must include the old username and the new email emails = { 'ORIGINAL_USERNAME' => 'NEW_EMAIL_ADDRESS', ... @@ -582,8 +582,8 @@ for each of these users. ## Expired license causes errors with multiple LDAP servers -Using [multiple LDAP servers](index.md#multiple-ldap-servers) requires a valid license. An expired -license can cause: +Using [multiple LDAP servers](index.md#use-multiple-ldap-servers) requires a valid license. An expired license can +cause: - `502` errors in the web interface. - The following error in logs (the actual strategy name depends on the name configured in `/etc/gitlab/gitlab.rb`): @@ -686,7 +686,7 @@ For more information, see the [official `ldapsearch` documentation](https://linu ### Using **AdFind** (Windows) -You can use the [`AdFind`](https://social.technet.microsoft.com/wiki/contents/articles/7535.adfind-command-examples.aspx) utility (on Windows based systems) to test that your LDAP server is accessible and authentication is working correctly. This is a freeware utility built by [Joe Richards](http://www.joeware.net/freetools/tools/adfind/index.htm). +You can use the [`AdFind`](https://social.technet.microsoft.com/wiki/contents/articles/7535.adfind-command-examples.aspx) utility (on Windows based systems) to test that your LDAP server is accessible and authentication is working correctly. AdFind is a freeware utility built by [Joe Richards](http://www.joeware.net/freetools/tools/adfind/index.htm). **Return all objects** @@ -719,9 +719,8 @@ For instructions about how to use the rails console, refer to this #### Enable debug output -This provides debug output that is useful to see -what GitLab is doing and with what. This value is not persisted, and is only -enabled for this session in the rails console. +This provides debug output that shows what GitLab is doing and with what. +This value is not persisted, and is only enabled for this session in the Rails console. To enable debug output in the rails console, [enter the rails console](#rails-console) and run: diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index 2673a8374ec..8ccd8fecbcf 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -127,8 +127,8 @@ following. 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. -To take advantage of group sync, group owners or maintainers must [create one -or more LDAP group links](#add-group-links). +To take advantage of group sync, group Owners or users with the [Maintainer role](../../../user/permissions.md) must +[create one or more LDAP group links](#add-group-links). ### Add group links diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index b8c443ae4d4..7ab1f2f5feb 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -35,22 +35,23 @@ The OpenID Connect provides you with a client's details and secret for you to us ```ruby gitlab_rails['omniauth_providers'] = [ - { 'name' => 'openid_connect', - 'label' => '<your_oidc_label>', - 'icon' => '<custom_provider_icon>', - 'args' => { - 'name' => 'openid_connect', - 'scope' => ['openid','profile','email'], - 'response_type' => 'code', - 'issuer' => '<your_oidc_url>', - 'discovery' => true, - 'client_auth_method' => 'query', - 'uid_field' => '<uid_field>', - 'send_scope_to_token_endpoint' => 'false', - 'client_options' => { - 'identifier' => '<your_oidc_client_id>', - 'secret' => '<your_oidc_client_secret>', - 'redirect_uri' => '<your_gitlab_url>/users/auth/openid_connect/callback' + { + name: "openid_connect", + label: "Provider name", # optional label for login button, defaults to "Openid Connect" + icon: "<custom_provider_icon>", + args: { + name: "openid_connect", + scope: ["openid","profile","email"], + response_type: "code", + issuer: "<your_oidc_url>", + discovery: true, + client_auth_method: "query", + uid_field: "<uid_field>", + send_scope_to_token_endpoint: "false", + client_options: { + identifier: "<your_oidc_client_id>", + secret: "<your_oidc_client_secret>", + redirect_uri: "<your_gitlab_url>/users/auth/openid_connect/callback" } } } @@ -61,7 +62,7 @@ The OpenID Connect provides you with a client's details and secret for you to us ```yaml - { name: 'openid_connect', - label: '<your_oidc_label>', + label: 'Provider name', # optional label for login button, defaults to "Openid Connect" icon: '<custom_provider_icon>', args: { name: 'openid_connect', @@ -136,20 +137,20 @@ for more details: ```ruby gitlab_rails['omniauth_providers'] = [ { - 'name' => 'openid_connect', - 'label' => 'Google OpenID', - 'args' => { - 'name' => 'openid_connect', - 'scope' => ['openid', 'profile', 'email'], - 'response_type' => 'code', - 'issuer' => 'https://accounts.google.com', - 'client_auth_method' => 'query', - 'discovery' => true, - 'uid_field' => 'preferred_username', - 'client_options' => { - 'identifier' => '<YOUR PROJECT CLIENT ID>', - 'secret' => '<YOUR PROJECT CLIENT SECRET>', - 'redirect_uri' => 'https://example.com/users/auth/openid_connect/callback', + name: "openid_connect", + label: "Google OpenID", # optional label for login button, defaults to "Openid Connect" + args: { + name: "openid_connect", + scope: ["openid", "profile", "email"], + response_type: "code", + issuer: "https://accounts.google.com", + client_auth_method: "query", + discovery: true, + uid_field: "preferred_username", + client_options: { + identifier: "<YOUR PROJECT CLIENT ID>", + secret: "<YOUR PROJECT CLIENT SECRET>", + redirect_uri: "https://example.com/users/auth/openid_connect/callback", } } } @@ -173,20 +174,20 @@ Example Omnibus configuration block: ```ruby gitlab_rails['omniauth_providers'] = [ { - 'name' => 'openid_connect', - 'label' => 'Azure OIDC', - 'args' => { - 'name' => 'openid_connect', - 'scope' => ['openid', 'profile', 'email'], - 'response_type' => 'code', - 'issuer' => 'https://login.microsoftonline.com/<YOUR-TENANT-ID>/v2.0', - 'client_auth_method' => 'query', - 'discovery' => true, - 'uid_field' => 'preferred_username', - 'client_options' => { - 'identifier' => '<YOUR APP CLIENT ID>', - 'secret' => '<YOUR APP CLIENT SECRET>', - 'redirect_uri' => 'https://gitlab.example.com/users/auth/openid_connect/callback' + name: "openid_connect", + label: "Azure OIDC", # optional label for login button, defaults to "Openid Connect" + args: { + name: "openid_connect", + scope: ["openid", "profile", "email"], + response_type: "code", + issuer: "https://login.microsoftonline.com/<YOUR-TENANT-ID>/v2.0", + client_auth_method: "query", + discovery: true, + uid_field: "preferred_username", + client_options: { + identifier: "<YOUR APP CLIENT ID>", + secret: "<YOUR APP CLIENT SECRET>", + redirect_uri: "https://gitlab.example.com/users/auth/openid_connect/callback" } } } @@ -302,21 +303,21 @@ The trailing forward slash is required. ```ruby gitlab_rails['omniauth_providers'] = [ { - 'name' => 'openid_connect', - 'label' => 'Azure B2C OIDC', - 'args' => { - 'name' => 'openid_connect', - 'scope' => ['openid'], - 'response_mode' => 'query', - 'response_type' => 'id_token', - 'issuer' => 'https://<YOUR-DOMAIN>/tfp/<YOUR-TENANT-ID>/b2c_1a_signup_signin/v2.0/', - 'client_auth_method' => 'query', - 'discovery' => true, - 'send_scope_to_token_endpoint' => true, - 'client_options' => { - 'identifier' => '<YOUR APP CLIENT ID>', - 'secret' => '<YOUR APP CLIENT SECRET>', - 'redirect_uri' => 'https://gitlab.example.com/users/auth/openid_connect/callback' + name: "openid_connect", + label: "Azure B2C OIDC", # optional label for login button, defaults to "Openid Connect" + args: { + name: "openid_connect", + scope: ["openid"], + response_mode: "query", + response_type: "id_token", + issuer: "https://<YOUR-DOMAIN>/tfp/<YOUR-TENANT-ID>/b2c_1a_signup_signin/v2.0/", + client_auth_method: "query", + discovery: true, + send_scope_to_token_endpoint: true, + client_options: { + identifier: "<YOUR APP CLIENT ID>", + secret: "<YOUR APP CLIENT SECRET>", + redirect_uri: "https://gitlab.example.com/users/auth/openid_connect/callback" } } }] @@ -359,20 +360,20 @@ Example Omnibus configuration block: ```ruby gitlab_rails['omniauth_providers'] = [ { - 'name' => 'openid_connect', - 'label' => 'Keycloak', - 'args' => { - 'name' => 'openid_connect', - 'scope' => ['openid', 'profile', 'email'], - 'response_type' => 'code', - 'issuer' => 'https://keycloak.example.com/auth/realms/myrealm', - 'client_auth_method' => 'query', - 'discovery' => true, - 'uid_field' => 'preferred_username', - 'client_options' => { - 'identifier' => '<YOUR CLIENT ID>', - 'secret' => '<YOUR CLIENT SECRET>', - 'redirect_uri' => 'https://gitlab.example.com/users/auth/openid_connect/callback' + name: "openid_connect", + label: "Keycloak", # optional label for login button, defaults to "Openid Connect" + args: { + name: "openid_connect", + scope: ["openid", "profile", "email"], + response_type: "code", + issuer: "https://keycloak.example.com/auth/realms/myrealm", + client_auth_method: "query", + discovery: true, + uid_field: "preferred_username", + client_options: { + identifier: "<YOUR CLIENT ID>", + secret: "<YOUR CLIENT SECRET>", + redirect_uri: "https://gitlab.example.com/users/auth/openid_connect/callback" } } } @@ -436,21 +437,21 @@ To use symmetric key encryption: ```ruby gitlab_rails['omniauth_providers'] = [ { - 'name' => 'openid_connect', - 'label' => 'Keycloak', - 'args' => { - 'name' => 'openid_connect', - 'scope' => ['openid', 'profile', 'email'], - 'response_type' => 'code', - 'issuer' => 'https://keycloak.example.com/auth/realms/myrealm', - 'client_auth_method' => 'query', - 'discovery' => true, - 'uid_field' => 'preferred_username', - 'jwt_secret_base64' => '<YOUR BASE64-ENCODED SECRET>', - 'client_options' => { - 'identifier' => '<YOUR CLIENT ID>', - 'secret' => '<YOUR CLIENT SECRET>', - 'redirect_uri' => 'https://gitlab.example.com/users/auth/openid_connect/callback' + name: "openid_connect", + label: "Keycloak", # optional label for login button, defaults to "Openid Connect" + args: { + name: "openid_connect", + scope: ["openid", "profile", "email"], + response_type: "code", + issuer: "https://keycloak.example.com/auth/realms/myrealm", + client_auth_method: "query", + discovery: true, + uid_field: "preferred_username", + jwt_secret_base64: "<YOUR BASE64-ENCODED SECRET>", + client_options: { + identifier: "<YOUR CLIENT ID>", + secret: "<YOUR CLIENT SECRET>", + redirect_uri: "https://gitlab.example.com/users/auth/openid_connect/callback" } } } diff --git a/doc/administration/cicd.md b/doc/administration/cicd.md index d53290f1d5d..a7bd07d5d38 100644 --- a/doc/administration/cicd.md +++ b/doc/administration/cicd.md @@ -58,9 +58,9 @@ For Omnibus GitLab installations: sudo gitlab-ctl reconfigure ``` -## Set the `needs:` job limit **(FREE SELF)** +## Set the `needs` job limit **(FREE SELF)** -The maximum number of jobs that can be defined in `needs:` defaults to 50. +The maximum number of jobs that can be defined in `needs` defaults to 50. A GitLab administrator with [access to the GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session) can choose a custom limit. For example, to set the limit to `100`: diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index 93b24007de8..b5c0a6ee76a 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -4,13 +4,15 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install the Kubernetes Agent Server (KAS) **(PREMIUM SELF)** +# Install the GitLab Agent Server (KAS) **(FREE SELF)** -The Kubernetes Agent Server (KAS) is a GitLab backend service dedicated to -managing [Kubernetes Agents](../../user/clusters/agent/index.md). +> [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. + +The GitLab Agent Server (KAS) is a GitLab backend service dedicated to +managing the [GitLab Agent](../../user/clusters/agent/index.md). The KAS is already installed and available in GitLab.com under `wss://kas.gitlab.com`. -See [how to use GitLab.com's KAS](../../user/clusters/agent/install/index.md#set-up-the-kubernetes-agent-server). +See [how to use GitLab.com's KAS](../../user/clusters/agent/install/index.md#set-up-the-agent-server). This document describes how to install a KAS for GitLab self-managed instances. ## Installation options @@ -27,7 +29,7 @@ You can also opt to use an [external KAS](#use-an-external-kas-installation). For [Omnibus](https://docs.gitlab.com/omnibus/) package installations: -1. Edit `/etc/gitlab/gitlab.rb` to enable the Kubernetes Agent Server: +1. Edit `/etc/gitlab/gitlab.rb` to enable the Agent Server: ```ruby gitlab_kas['enable'] = true diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index a05495c024e..7cecc0c30fd 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -73,7 +73,7 @@ These features can also help with compliance requirements: - [**Generate reports on permission levels of users**](../user/admin_area/index.md#user-permission-export) (for instances): Administrators can generate a report listing all users' access permissions for groups and projects in the instance. -- [**Lock project membership to group**](../user/group/index.md#prevent-members-from-being-added-to-a-group) (for +- [**Lock project membership to group**](../user/group/index.md#prevent-members-from-being-added-to-projects-in-a-group) (for groups): Group owners can prevent new members from being added to projects within a group. - [**LDAP group sync**](auth/ldap/ldap_synchronization.md#group-sync) (for instances): Gives administrators the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index 45f27a8a8f2..92b8342f251 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -1,272 +1,9 @@ --- -stage: Enablement -group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'postgresql/database_load_balancing.md' +remove_date: '2022-02-19' --- -# Database Load Balancing **(FREE SELF)** +This file was moved to [another location](postgresql/database_load_balancing.md). -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60894) from GitLab Premium to GitLab Free in 14.0. - -Distribute read-only queries among multiple database servers. - -## Overview - -Database load balancing improves the distribution of database workloads across -multiple computing resources. Load balancing aims to optimize resource use, -maximize throughput, minimize response time, and avoid overload of any single -resource. Using multiple components with load balancing instead of a single -component may increase reliability and availability through redundancy. -[_Wikipedia article_](https://en.wikipedia.org/wiki/Load_balancing_(computing)) - -When database load balancing is enabled in GitLab, the load is balanced using -a simple round-robin algorithm, without any external dependencies such as Redis. - -In the following image, you can see the load is balanced rather evenly among -all the secondaries (`db4`, `db5`, `db6`). Because `SELECT` queries are not -sent to the primary (unless necessary), the primary (`db3`) hardly has any load. - - - -## Requirements - -For load balancing to work, you need at least PostgreSQL 11 or newer, -[**MySQL is not supported**](../install/requirements.md#database). You also need to make sure that you have -at least 1 secondary in [hot standby](https://www.postgresql.org/docs/11/hot-standby.html) mode. - -Load balancing also requires that the configured hosts **always** point to the -primary, even after a database failover. Furthermore, the additional hosts to -balance load among must **always** point to secondary databases. This means that -you should put a load balancer in front of every database, and have GitLab connect -to those load balancers. - -For example, say you have a primary (`db1.gitlab.com`) and two secondaries, -`db2.gitlab.com` and `db3.gitlab.com`. For this setup, you need to have 3 -load balancers, one for every host. For example: - -- `primary.gitlab.com` forwards to `db1.gitlab.com` -- `secondary1.gitlab.com` forwards to `db2.gitlab.com` -- `secondary2.gitlab.com` forwards to `db3.gitlab.com` - -Now let's say that a failover happens and db2 becomes the new primary. This -means forwarding should now happen as follows: - -- `primary.gitlab.com` forwards to `db2.gitlab.com` -- `secondary1.gitlab.com` forwards to `db1.gitlab.com` -- `secondary2.gitlab.com` forwards to `db3.gitlab.com` - -GitLab does not take care of this for you, so you need to do so yourself. - -Finally, load balancing requires that GitLab can connect to all hosts using the -same credentials and port as configured in the -[Enabling load balancing](#enabling-load-balancing) section. Using -different ports or credentials for different hosts is not supported. - -## Use cases - -- For GitLab instances with thousands of users and high traffic, you can use - database load balancing to reduce the load on the primary database and - increase responsiveness, thus resulting in faster page load inside GitLab. - -## Enabling load balancing - -For the environment in which you want to use load balancing, you'll need to add -the following. This balances the load between `host1.example.com` and -`host2.example.com`. - -**In Omnibus installations:** - -1. Edit `/etc/gitlab/gitlab.rb` and add the following line: - - ```ruby - gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com'] } - ``` - -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - ---- - -**In installations from source:** - -1. Edit `/home/git/gitlab/config/database.yml` and add or amend the following lines: - - ```yaml - production: - username: gitlab - database: gitlab - encoding: unicode - load_balancing: - hosts: - - host1.example.com - - host2.example.com - ``` - -1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. - -### Load balancing for Sidekiq - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334494) in GitLab 14.1, load balancing for Sidekick is enabled by default. - -Sidekiq jobs mostly write to the primary database, but there are read-only jobs that can benefit -from the use of Sidekiq load balancing. -These jobs can use load balancing and database replicas to read the application state. -This allows to offload the primary database. - -For Sidekiq, we can define -[data consistency](../development/sidekiq_style_guide.md#job-data-consistency-strategies) -requirements for a specific job. - -## Service Discovery **(PREMIUM SELF)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883) in GitLab 11.0. - -Service discovery allows GitLab to automatically retrieve a list of secondary -databases to use, instead of having to manually specify these in the -`database.yml` configuration file. Service discovery works by periodically -checking a DNS A record, using the IPs returned by this record as the addresses -for the secondaries. For service discovery to work, all you need is a DNS server -and an A record containing the IP addresses of your secondaries. - -To use service discovery you need to change your `database.yml` configuration -file so it looks like the following: - -```yaml -production: - username: gitlab - database: gitlab - encoding: unicode - load_balancing: - discover: - nameserver: localhost - record: secondary.postgresql.service.consul - record_type: A - port: 8600 - interval: 60 - disconnect_timeout: 120 -``` - -Here, the `discover:` section specifies the configuration details to use for -service discovery. - -### Configuration - -The following options can be set: - -| Option | Description | Default | -|----------------------|---------------------------------------------------------------------------------------------------|-----------| -| `nameserver` | The nameserver to use for looking up the DNS record. | localhost | -| `record` | The record to look up. This option is required for service discovery to work. | | -| `record_type` | Optional record type to look up, this can be either A or SRV (GitLab 12.3 and later) | A | -| `port` | The port of the nameserver. | 8600 | -| `interval` | The minimum time in seconds between checking the DNS record. | 60 | -| `disconnect_timeout` | The time in seconds after which an old connection is closed, after the list of hosts was updated. | 120 | -| `use_tcp` | Lookup DNS resources using TCP instead of UDP | false | - -If `record_type` is set to `SRV`, then GitLab continues to use round-robin algorithm -and ignores the `weight` and `priority` in the record. Since SRV records usually -return hostnames instead of IPs, GitLab needs to look for the IPs of returned hostnames -in the additional section of the SRV response. If no IP is found for a hostname, GitLab -needs to query the configured `nameserver` for ANY record for each such hostname looking for A or AAAA -records, eventually dropping this hostname from rotation if it can't resolve its IP. - -The `interval` value specifies the _minimum_ time between checks. If the A -record has a TTL greater than this value, then service discovery honors said -TTL. For example, if the TTL of the A record is 90 seconds, then service -discovery waits at least 90 seconds before checking the A record again. - -When the list of hosts is updated, it might take a while for the old connections -to be terminated. The `disconnect_timeout` setting can be used to enforce an -upper limit on the time it takes to terminate all old database connections. - -Some nameservers (like [Consul](https://www.consul.io/docs/discovery/dns#udp-based-dns-queries)) can return a truncated list of hosts when -queried over UDP. To overcome this issue, you can use TCP for querying by setting -`use_tcp` to `true`. - -## Balancing queries - -Read-only `SELECT` queries balance among all the secondary hosts. -Everything else (including transactions) executes on the primary. -Queries such as `SELECT ... FOR UPDATE` are also executed on the primary. - -## Prepared statements - -Prepared statements don't work well with load balancing and are disabled -automatically when load balancing is enabled. This should have no impact on -response timings. - -## Primary sticking - -After a write has been performed, GitLab sticks to using the primary for a -certain period of time, scoped to the user that performed the write. GitLab -reverts back to using secondaries when they have either caught up, or after 30 -seconds. - -## Failover handling - -In the event of a failover or an unresponsive database, the load balancer -tries to use the next available host. If no secondaries are available the -operation is performed on the primary instead. - -If a connection error occurs while writing data, the -operation is retried up to 3 times using an exponential back-off. - -When using load balancing, you should be able to safely restart a database server -without it immediately leading to errors being presented to the users. - -## Logging - -The load balancer logs various events in -[`database_load_balancing.log`](logs.md#database_load_balancinglog), such as - -- When a host is marked as offline -- When a host comes back online -- When all secondaries are offline -- When a read is retried on a different host due to a query conflict - -The log is structured with each entry a JSON object containing at least: - -- An `event` field useful for filtering. -- A human-readable `message` field. -- Some event-specific metadata. For example, `db_host` -- Contextual information that is always logged. For example, `severity` and `time`. - -For example: - -```json -{"severity":"INFO","time":"2019-09-02T12:12:01.728Z","correlation_id":"abcdefg","event":"host_online","message":"Host came back online","db_host":"111.222.333.444","db_port":null,"tag":"rails.database_load_balancing","environment":"production","hostname":"web-example-1","fqdn":"gitlab.example.com","path":null,"params":null} -``` - -## Handling Stale Reads **(PREMIUM SELF)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526) in GitLab 10.3. - -To prevent reading from an outdated secondary the load balancer checks if it -is in sync with the primary. If the data is determined to be recent enough the -secondary is used, otherwise it is ignored. To reduce the overhead of -these checks we only perform these checks at certain intervals. - -There are three configuration options that influence this behavior: - -| Option | Description | Default | -|------------------------------|----------------------------------------------------------------------------------------------------------------|------------| -| `max_replication_difference` | The amount of data (in bytes) a secondary is allowed to lag behind when it hasn't replicated data for a while. | 8 MB | -| `max_replication_lag_time` | The maximum number of seconds a secondary is allowed to lag behind before we stop using it. | 60 seconds | -| `replica_check_interval` | The minimum number of seconds we have to wait before checking the status of a secondary. | 60 seconds | - -The defaults should be sufficient for most users. Should you want to change them -you can specify them in `config/database.yml` like so: - -```yaml -production: - username: gitlab - database: gitlab - encoding: unicode - load_balancing: - hosts: - - host1.example.com - - host2.example.com - max_replication_difference: 16777216 # 16 MB - max_replication_lag_time: 30 - replica_check_interval: 30 -``` +<!-- This redirect file can be deleted after <2022-02-19>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 21e32d145bd..22159b6e9db 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -31,6 +31,7 @@ You can use the following environment variables to override certain values: | `GITLAB_EMAIL_REPLY_TO` | string | The email address used in the **Reply-To** field in emails sent by GitLab. | | `GITLAB_EMAIL_SUBJECT_SUFFIX` | string | The email subject suffix used in emails sent by GitLab. | | `GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`). | +| `GITLAB_MARKUP_TIMEOUT` | string | Timeout, in seconds, for `rest2html` and `pod2html` commands executed by the [`gitlab-markup` gem](https://gitlab.com/gitlab-org/gitlab-markup/). Default is `10`. | | `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation. | | `GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for runners. | | `RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging`, or `test`. | diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 3c7af309f78..b207be47aa1 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -60,11 +60,11 @@ What is not covered: NOTE: Before following any of those steps, make sure you have `root` access to the -**secondary** to promote it, since there isn't provided an automated way to +**secondary** to promote it, because there isn't provided an automated way to promote a Geo replica and perform a failover. NOTE: -GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). On the **secondary** site: @@ -88,7 +88,7 @@ A common cause of replication failures is the data being missing on the **primary** site - you can resolve these failures by restoring the data from backup, or removing references to the missing data. -The maintenance window won't end until Geo replication and verification is +The maintenance window doesn't end until Geo replication and verification is completely finished. To keep the window as short as possible, you should ensure these processes are close to 100% as possible during active use. @@ -122,7 +122,7 @@ follow these steps to avoid unnecessary data loss: From this point, users are unable to view their data or make changes on the **primary** site. They are also unable to log in to the **secondary** site. - However, existing sessions need to work for the remainder of the maintenance period, and + However, existing sessions must work for the remainder of the maintenance period, and so public data is accessible throughout. 1. Verify the **primary** site is blocked to HTTP traffic by visiting it in browser via @@ -135,10 +135,10 @@ follow these steps to avoid unnecessary data loss: 1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. - 1. On the Sidekiq dhasboard, select **Cron**. + 1. On the Sidekiq dashboard, select **Cron**. 1. Select `Disable All` to disable any non-Geo periodic background jobs. 1. Select `Enable` for the `geo_sidekiq_cron_config_worker` cron job. - This job will re-enable several other cron jobs that are essential for planned + This job re-enables several other cron jobs that are essential for planned failover to complete successfully. 1. Finish replicating and verifying all data: @@ -176,7 +176,7 @@ follow these steps to avoid unnecessary data loss: At this point, your **secondary** site contains an up-to-date copy of everything the **primary** site has, meaning nothing is lost when you fail over. -1. In this final step, you need to permanently disable the **primary** site. +1. In this final step, you must permanently disable the **primary** site. WARNING: When the **primary** site goes offline, there may be data saved on the **primary** site @@ -204,7 +204,7 @@ follow these steps to avoid unnecessary data loss: ``` NOTE: - (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being + (**CentOS only**) In CentOS 6 or older, it is challenging to prevent GitLab from being started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. @@ -216,7 +216,7 @@ follow these steps to avoid unnecessary data loss: - If you do not have SSH access to the **primary** site, take the machine offline and prevent it from rebooting. Since there are many ways you may prefer to accomplish - this, we avoid a single recommendation. You may need to: + this, we avoid a single recommendation. You may have to: - Reconfigure the load balancers. - Change DNS records (for example, point the **primary** DNS record to the diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 8a4f2ed4306..5a6f9eb8be7 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -52,7 +52,7 @@ Before following any of those steps, make sure you have `root` access to the promote a Geo replica and perform a failover. NOTE: -GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses appears to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). On the **secondary** site, navigate to the **Admin Area > Geo** dashboard to review its status. Replicated objects (shown in green) should be close to 100%, @@ -73,7 +73,7 @@ A common cause of replication failures is the data being missing on the **primary** site - you can resolve these failures by restoring the data from backup, or removing references to the missing data. -The maintenance window won't end until Geo replication and verification is +The maintenance window does not end until Geo replication and verification is completely finished. To keep the window as short as possible, you should ensure these processes are close to 100% as possible during active use. @@ -123,7 +123,7 @@ follow these steps to avoid unnecessary data loss: 1. On the Sidekiq dhasboard, select **Cron**. 1. Select `Disable All` to disable any non-Geo periodic background jobs. 1. Select `Enable` for the `geo_sidekiq_cron_config_worker` cron job. - This job will re-enable several other cron jobs that are essential for planned + This job re-enables several other cron jobs that are essential for planned failover to complete successfully. 1. Finish replicating and verifying all data: diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 30d8d765dc5..2cb1a424ce8 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -2,17 +2,19 @@ stage: Enablement group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- # Geo **(PREMIUM SELF)** -Geo is the solution for widely distributed development teams and for providing a warm-standby as part of a disaster recovery strategy. +Geo is the solution for widely distributed development teams and for providing +a warm-standby as part of a disaster recovery strategy. ## Overview WARNING: -Geo undergoes significant changes from release to release. Upgrades **are** supported and [documented](#updating-geo), but you should ensure that you're using the right version of the documentation for your installation. +Geo undergoes significant changes from release to release. Upgrades are +supported and [documented](#updating-geo), but you should ensure that you're +using the right version of the documentation for your installation. Fetching large repositories can take a long time for teams located far from a single GitLab instance. @@ -69,8 +71,9 @@ Keep in mind that: - **Secondary** sites talk to the **primary** site to: - Get user data for logins (API). - Replicate repositories, LFS Objects, and Attachments (HTTPS + JWT). -- In GitLab Premium 10.0 and later, the **primary** site no longer talks to **secondary** sites to notify for changes (API). -- Pushing directly to a **secondary** site (for both HTTP and SSH, including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. +- The **primary** site doesn't talk to **secondary** sites to notify for changes (API). +- You can push directly to a **secondary** site (for both HTTP and SSH, + including Git LFS). - There are [limitations](#limitations) when using Geo. ### Architecture @@ -111,21 +114,18 @@ In **secondary** sites, there is an additional daemon: [Geo Log Cursor](#geo-log The following are required to run Geo: -- An operating system that supports OpenSSH 6.9+ (needed for +- An operating system that supports OpenSSH 6.9 or later (needed for [fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md)) The following operating systems are known to ship with a current version of OpenSSH: - - [CentOS](https://www.centos.org) 7.4+ - - [Ubuntu](https://ubuntu.com) 16.04+ -- PostgreSQL 12+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) -- Git 2.9+ -- Git-lfs 2.4.2+ on the user side when using LFS + - [CentOS](https://www.centos.org) 7.4 or later + - [Ubuntu](https://ubuntu.com) 16.04 or later +- PostgreSQL 12 or later with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) +- Git 2.9 or later +- Git-lfs 2.4.2 or later on the user side when using LFS - All sites must run the same GitLab version. Additionally, check the GitLab [minimum requirements](../../install/requirements.md), -and we recommend you use: - -- At least GitLab Enterprise Edition 10.0 for basic Geo features. -- The latest version for a better experience. +and we recommend you use the latest version of GitLab for a better experience. ### Firewall rules @@ -311,7 +311,8 @@ For answers to common questions, see the [Geo FAQ](replication/faq.md). ## Log files -In GitLab 9.5 and later, Geo stores structured log messages in a `geo.log` file. For Omnibus installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. +Geo stores structured log messages in a `geo.log` file. For Omnibus GitLab +installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. This file contains information about when Geo attempts to sync repositories and files. Each line in the file contains a separate JSON entry that can be ingested into. For example, Elasticsearch or Splunk. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 88f1ad5b490..3cbde77903d 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -247,7 +247,7 @@ You can safely skip this step if your **primary** site uses a CA-issued HTTPS ce If your **primary** site is using a self-signed certificate for *HTTPS* support, you need to add that certificate to the **secondary** site's trust store. Retrieve the certificate from the **primary** site and follow -[these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html) +[these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) on the **secondary** site. ### Step 5. Enable Git access over HTTP/HTTPS diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index c98436157fc..31bc473d74b 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo data types support **(PREMIUM SELF)** +# Supported Geo data types **(PREMIUM SELF)** A Geo data type is a specific class of data that is required by one or more GitLab features to store relevant information. @@ -14,7 +14,7 @@ To replicate data produced by these features with Geo, we use several strategies ## Data types -We currently distinguish between three different data types: +We distinguish between three different data types: - [Git repositories](#git-repositories) - [Blobs](#blobs) @@ -35,9 +35,9 @@ verification methods: | Git | Project Snippets | Geo with Gitaly | Gitaly Checksum | | Git | Personal Snippets | Geo with Gitaly | Gitaly Checksum | | Git | Group wiki repository | Geo with Gitaly | _Not implemented_ | -| Blobs | User uploads _(file system)_ | Geo with API | _Not implemented_ | +| Blobs | User uploads _(file system)_ | Geo with API | SHA256 checksum | | Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | LFS objects _(file system)_ | Geo with API | _Not implemented_ | +| Blobs | LFS objects _(file system)_ | Geo with API | SHA256 checksum | | Blobs | LFS objects _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | CI job artifacts _(file system)_ | Geo with API | _Not implemented_ | | Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | @@ -51,11 +51,11 @@ verification methods: | Blobs | Infrastructure registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | +| Blobs | External Merge Request Diffs _(file system)_ | Geo with API | SHA256 checksum | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Pipeline artifacts _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Pipeline artifacts _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum | -| Blobs | Pages _(file system)_ | Geo with API | _Not implemented_ | +| Blobs | Pipeline artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Pages _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Pages _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo sites. @@ -66,15 +66,20 @@ verification methods: A GitLab instance can have one or more repository shards. Each shard has a Gitaly instance that is responsible for allowing access and operations on the locally stored Git repositories. It can run -on a machine with a single disk, multiple disks mounted as a single mount-point (like with a RAID array), -or using LVM. +on a machine: -It requires no special file system and can work with NFS or a mounted Storage Appliance (there may be -performance limitations when using a remote file system). +- With a single disk. +- With multiple disks mounted as a single mount-point (like with a RAID array). +- Using LVM. -Geo will trigger garbage collection in Gitaly to [deduplicate forked repositories](../../../development/git_object_deduplication.md#git-object-deduplication-and-gitlab-geo) on Geo secondary sites. +GitLab does not require a special file system and can work with: -Communication is done via Gitaly's own gRPC API. There are three possible ways of synchronization: +- NFS. +- A mounted Storage Appliance (there may be performance limitations when using a remote file system). + +Geo triggers garbage collection in Gitaly to [deduplicate forked repositories](../../../development/git_object_deduplication.md#git-object-deduplication-and-gitlab-geo) on Geo secondary sites. + +Communication is done via Gitaly's own gRPC API, with three possible ways of synchronization: - Using regular Git clone/fetch from one Geo site to another (with special authentication). - Using repository snapshots (for when the first method fails or repository is corrupt). @@ -90,7 +95,7 @@ They all live in the same shard and share the same base name with a `-wiki` and for Wiki and Design Repository cases. Besides that, there are snippet repositories. They can be connected to a project or to some specific user. -Both types will be synced to a secondary site. +Both types are synced to a secondary site. ### Blobs @@ -102,7 +107,7 @@ GitLab stores files and blobs such as Issue attachments or LFS objects into eith - Hosted by you (like MinIO). - A Storage Appliance that exposes an Object Storage-compatible API. -When using the file system store instead of Object Storage, you need to use network mounted file systems +When using the file system store instead of Object Storage, use network mounted file systems to run GitLab when using more than one node. With respect to replication and verification: @@ -118,17 +123,17 @@ GitLab relies on data stored in multiple databases, for different use-cases. PostgreSQL is the single point of truth for user-generated content in the Web interface, like issues content, comments as well as permissions and credentials. -PostgreSQL can also hold some level of cached data like HTML rendered Markdown, cached merge-requests diff (this can -also be configured to be offloaded to object storage). +PostgreSQL can also hold some level of cached data like HTML-rendered Markdown and cached merge-requests diff. +This can also be configured to be offloaded to object storage. We use PostgreSQL's own replication functionality to replicate data from the **primary** to **secondary** sites. We use Redis both as a cache store and to hold persistent data for our background jobs system. Because both use-cases have data that are exclusive to the same Geo site, we don't replicate it between sites. -Elasticsearch is an optional database, that can enable advanced searching capabilities, like improved Advanced Search -in both source-code level and user generated content in Issues / Merge-Requests and discussions. Currently it's not -supported in Geo. +Elasticsearch is an optional database that for advanced searching capabilities. It can improve search +in both source-code level, and user generated content in issues, merge requests, and discussions. +Elasticsearch is not supported in Geo. ## Limitations on replication/verification @@ -142,7 +147,6 @@ these epics/issues: - [Geo: Improve the self-service Geo replication framework](https://gitlab.com/groups/gitlab-org/-/epics/3761) - [Geo: Move existing blobs to framework](https://gitlab.com/groups/gitlab-org/-/epics/3588) - [Geo: Add unreplicated data types](https://gitlab.com/groups/gitlab-org/-/epics/893) -- [Geo: Support GitLab Pages](https://gitlab.com/groups/gitlab-org/-/epics/589) ### Replicated data types behind a feature flag @@ -174,35 +178,35 @@ Feature.enable(:geo_package_file_replication) WARNING: Features not on this list, or with **No** in the **Replicated** column, are not replicated to a **secondary** site. Failing over without manually -replicating data from those features will cause the data to be **lost**. -If you wish to use those features on a **secondary** site, or to execute a failover +replicating data from those features causes the data to be **lost**. +To use those features on a **secondary** site, or to execute a failover successfully, you must replicate their data using some other means. -|Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | -|:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:------------------------------------------------------------------------|:------------------------------------------------------------------------------|:------| -|[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -|[Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | -|[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | -|[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | -|[Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | -|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Behind feature flag `geo_lfs_object_replication`, enabled by default. | -|[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -|[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -|[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. Job logs also verified on transfer. | -|[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | -|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | -|[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | -|[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (14.0) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | No | Designs also require replication of LFS objects and Uploads. | -|[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.12) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0| -|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification is under development, behind the feature flag `geo_merge_request_diff_verification`, introduced in 14.0.| -|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | -|[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_pages_deployment_replication`, enabled by default. | -|[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | -|[Elasticsearch integration](../../../integration/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries currently use the same ES cluster as the primary. | -|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked by [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for disaster recovery purposes because it can be recreated from external sources. | -|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | +|Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | +|:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------|:------------------------------------------------------------------------------|:------| +|[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +|[Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | +|[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | +|[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | +|[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification is behind the feature flag `geo_upload_verification` introduced and enabled by default in 14.6. | +|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | **Yes** (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Replication is behind the feature flag `geo_lfs_object_replication`, enabled by default. Verification is behind the feature flag `geo_lfs_object_verification` introduced and enabled by default in 14.6. | +|[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +|[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +|[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. Job logs also verified on transfer. | +|[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes. | +|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | +|[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | +|[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (14.0) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | No | Designs also require replication of LFS objects and Uploads. | +|[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.12) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0. | +|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification is behind the feature flag `geo_merge_request_diff_verification`, enabled by default in 14.6.| +|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | +|[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_pages_deployment_replication`, enabled by default. Verification is behind the feature flag `geo_pages_deployment_verification`, enabled by default in 14.6. | +|[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | +|[Elasticsearch integration](../../../integration/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries use the same ES cluster as the primary. | +|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked by [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for disaster recovery purposes because it can be recreated from external sources. | +|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | #### Limitation of verification for files in Object Storage diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index 02a65f0b8e1..0fa469e57cd 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -20,7 +20,7 @@ To disable Geo, follow these steps: 1. [Remove the primary site from the UI](#remove-the-primary-site-from-the-ui). 1. [Remove secondary replication slots](#remove-secondary-replication-slots). 1. [Remove Geo-related configuration](#remove-geo-related-configuration). -1. [(Optional) Revert PostgreSQL settings to use a password and listen on an IP](#optional-revert-postgresql-settings-to-use-a-password-and-listen-on-an-ip). +1. [Optional. Revert PostgreSQL settings to use a password and listen on an IP](#optional-revert-postgresql-settings-to-use-a-password-and-listen-on-an-ip). ## Remove all secondary Geo sites diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index 70a6e506c28..e613a9b5670 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -50,6 +50,8 @@ attachments and avatars, and the whole database. This means user accounts, issues, merge requests, groups, project data, and so on, will be available for query. +For more details, see the [supported Geo data types](datatypes.md). + ## Can I `git push` to a **secondary** site? Yes! Pushing directly to a **secondary** site (for both HTTP and SSH, including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in GitLab 11.3. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 432d042608c..673d8388af1 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -559,7 +559,7 @@ to start again from scratch, there are a few steps that can help you: You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future as soon as you confirmed that you don't need it anymore, to save disk space. -1. _(Optional)_ Rename other data folders and create new ones +1. Optional. Rename other data folders and create new ones WARNING: You may still have files on the **secondary** node that have been removed from the **primary** node, but this diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md deleted file mode 100644 index f07c8d547a4..00000000000 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'updating_the_geo_sites.md' -remove_date: '2021-11-23' ---- - -This file was moved to [another location](updating_the_geo_sites.md). - -<!-- This redirect file can be deleted after <2021-11-23>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 1b22a5f0991..883e335ff94 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -11,6 +11,10 @@ Review this page for update instructions for your version. These steps accompany the [general steps](updating_the_geo_sites.md#general-update-steps) for updating Geo nodes. +## Updating to 14.4 + +There is [an issue in GitLab 14.4.0 through 14.4.2](../../../update/index.md#1440) that can affect Geo and other features that rely on cronjobs. We recommend upgrading to GitLab 14.4.3 or later. + ## Updating to 14.1, 14.2, 14.3 ### Multi-arch images @@ -50,7 +54,7 @@ If you are running a version prior to 14.1 and are using Geo and multi-arch cont ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). ## Updating to GitLab 14.0/14.1 @@ -64,7 +68,7 @@ If you are running an affected version and need to remove your Primary site, you ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). ## Updating to GitLab 13.12 @@ -90,7 +94,7 @@ Geo::LfsObjectRegistry.where(state: 0, success: true).update_all(state: 2) ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). ## Updating to GitLab 13.11 @@ -98,20 +102,20 @@ We found an [issue with Git clone/pull through HTTP(s)](https://gitlab.com/gitla ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). ## Updating to GitLab 13.10 ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). ## Updating to GitLab 13.9 ### Error during zero-downtime update: "cannot drop column asset_proxy_whitelist" We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160) -that will prevent upgrades to GitLab 13.9.0, 13.9.1, 13.9.2 and 13.9.3 when following the zero-downtime steps. It is necessary +that prevents upgrades to GitLab 13.9.0, 13.9.1, 13.9.2 and 13.9.3 when following the zero-downtime steps. It is necessary to perform the following additional steps for the zero-downtime update: 1. Before running the final `sudo gitlab-rake db:migrate` command on the deploy node, @@ -132,7 +136,7 @@ to perform the following additional steps for the zero-downtime update: ``` If you have already run the final `sudo gitlab-rake db:migrate` command on the deploy node and have -encountered the [column rename issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160), you will +encountered the [column rename issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160), you might see the following error: ```shell @@ -148,7 +152,7 @@ More details are available [in this issue](https://gitlab.com/gitlab-org/gitlab/ ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). ## Updating to GitLab 13.7 @@ -168,7 +172,7 @@ on Geo secondaries. This issue is fixed in GitLab 13.6.1 and later. In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) dependency for the tracking database. -The FDW server, user, and the extension will be removed during the upgrade +The FDW server, user, and the extension is removed during the upgrade process on each secondary node. The GitLab settings related to the FDW in the `/etc/gitlab/gitlab.rb` have been deprecated and can be safely removed. diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index 2b8c0d1e6fa..ebd71757e91 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -7,11 +7,14 @@ type: howto # Geo proxying for secondary sites **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5914) in GitLab 14.4 [with a flag](../../feature_flags.md) named `geo_secondary_proxy`. Disabled by default. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5914) in GitLab 14.4 [with a flag](../../feature_flags.md) named `geo_secondary_proxy`. Disabled by default. +> - [Enabled by default for unified URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/325732) in GitLab 14.6. +> - [Disabled by default for different URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/325732) in GitLab 14.6 [with a flag](../../feature_flags.md) named `geo_secondary_proxy_separate_urls`. FLAG: -On self-managed GitLab, by default this feature is not available. See below to [Set up a unified URL for Geo sites](#set-up-a-unified-url-for-geo-sites). -The feature is not ready for production use. +On self-managed GitLab, this feature is only available by default for Geo sites using a unified URL. See below to +[set up a unified URL for Geo sites](#set-up-a-unified-url-for-geo-sites). +The feature is not ready for production use with separate URLs. Use Geo proxying to: @@ -66,7 +69,11 @@ a single URL used by all Geo sites, including the primary. is using the secondary proxying and set the `URL` field to the single URL. Make sure the primary site is also using this URL. -### Enable secondary proxying +In Kubernetes, you can use the same domain under `global.hosts.domain` as for the primary site. + +## Disable Geo proxying + +You can disable the secondary proxying on each Geo site, separately, by following these steps with Omnibus-based packages: 1. SSH into each application node (serving user traffic directly) on your secondary Geo site and add the following environment variable: @@ -77,7 +84,7 @@ a single URL used by all Geo sites, including the primary. ```ruby gitlab_workhorse['env'] = { - "GEO_SECONDARY_PROXY" => "1" + "GEO_SECONDARY_PROXY" => "0" } ``` @@ -87,11 +94,15 @@ a single URL used by all Geo sites, including the primary. gitlab-ctl reconfigure ``` -1. SSH into one node running Rails on your primary Geo site and enable the Geo secondary proxy feature flag: +In Kubernetes, you can use `--set gitlab.webservice.extraEnv.GEO_SECONDARY_PROXY="0"`, +or specify the following in your values file: - ```shell - sudo gitlab-rails runner "Feature.enable(:geo_secondary_proxy)" - ``` +```yaml +gitlab: + webservice: + extraEnv: + GEO_SECONDARY_PROXY: "0" +``` ## Enable Geo proxying with Separate URLs @@ -99,6 +110,36 @@ The ability to use proxying with separate URLs is still in development. You can ["Geo secondary proxying with separate URLs" epic](https://gitlab.com/groups/gitlab-org/-/epics/6865) for progress. +To try out this feature, enable the `geo_secondary_proxy_separate_urls` feature flag. +SSH into one node running Rails on your primary Geo site and run: + +```shell +sudo gitlab-rails runner "Feature.enable(:geo_secondary_proxy_separate_urls)" +``` + +In Kubernetes, you can run the same command in the toolbox pod. Refer to the +[Kubernetes cheat sheet](../../troubleshooting/kubernetes_cheat_sheet.md#gitlab-specific-kubernetes-information) +for details. + +## Limitations + +- When secondary proxying is used, the asynchronous Geo replication can cause unexpected issues for accelerated + data types that may be replicated to the Geo secondaries with a delay. + + For example, we found a potential issue where + [replication lag introduces read-after-write inconsistencies](https://gitlab.com/gitlab-org/gitlab/-/issues/345267). + If the replication lag is high enough, this can result in Git reads receiving stale data when hitting a secondary. + +- Non-Rails requests are not proxied, so other services may need to use a separate, non-unified URL to ensure requests + are always sent to the primary. These services include: + + - GitLab Container Registry - [can be configured to use a separate domain](../../packages/container_registry.md#configure-container-registry-under-its-own-domain). + - GitLab Pages - should always use a separate domain, as part of [the prerequisites for running GitLab Pages](../../pages/index.md#prerequisites). + +- With a unified URL, Let's Encrypt can't generate certificates unless it can reach both IPs through the same domain. + To use TLS certificates with Let's Encrypt, you can manually point the domain to one of the Geo sites, generate + the certificate, then copy it to all other sites. + ## Features accelerated by secondary Geo sites Most HTTP traffic sent to a secondary Geo site can be proxied to the primary Geo site. With this architecture, diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index 84dff69ebe7..7d365f73101 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -26,6 +26,7 @@ If you installed GitLab using the Omnibus packages (highly recommended): 1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** site(s). 1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** site(s). See [notes on LDAP](../index.md#ldap). 1. Follow the [Using a Geo Site](../replication/usage.md) guide. +1. [Configure Geo secondary proxying](../secondary_proxy/index.md) to use a single, unified URL for all Geo sites. This step is recommended to accelerate most read requests while transparently proxying writes to the primary Geo site. ## Post-installation documentation diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index d0841f4e607..b31a02aae0a 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -350,6 +350,10 @@ leading to `Error creating pipeline` and `Commit not found` errors, or stale dat As the final step, you must update Gitaly clients to switch from using local Gitaly service to use the Gitaly servers you just configured. +NOTE: +GitLab requires a `default` repository storage to be configured. +[Read more about this limitation](#gitlab-requires-a-default-repository-storage). + This can be risky because anything that prevents your Gitaly clients from reaching the Gitaly servers causes all Gitaly requests to fail. For example, any sort of network, firewall, or name resolution problems. @@ -489,6 +493,18 @@ gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" `path` can be included only for storage shards on the local Gitaly server. If it's excluded, default Git storage directory is used for that storage shard. +### GitLab requires a default repository storage + +When adding Gitaly servers to an environment, you might want to replace the original `default` Gitaly service. However, you can't +reconfigure the GitLab application servers to remove the `default` entry from `git_data_dirs` because GitLab requires a +`git_data_dirs` entry called `default`. [Read more](https://gitlab.com/gitlab-org/gitlab/-/issues/36175) about this limitation. + +To work around the limitation: + +1. Define an additional storage location on the new Gitaly service and configure the additional storage to be `default`. +1. In the [Admin Area](../repository_storage_paths.md#configure-where-new-repositories-are-stored), set `default` to a weight of zero + to prevent repositories being stored there. + ### Disable Gitaly where not required (optional) If you run Gitaly [as a remote service](#run-gitaly-on-its-own-server), consider @@ -605,7 +621,7 @@ To configure Gitaly with TLS: 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Verify Gitaly traffic is being served over TLS by [observing the types of Gitaly connections](#observe-type-of-gitaly-connections). -1. (Optional) Improve security by: +1. Optional. Improve security by: 1. Disabling non-TLS connections by commenting out or deleting `gitaly['listen_addr']` in `/etc/gitlab/gitlab.rb`. 1. Saving the file. @@ -681,7 +697,7 @@ To configure Gitaly with TLS: 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. Verify Gitaly traffic is being served over TLS by [observing the types of Gitaly connections](#observe-type-of-gitaly-connections). -1. (Optional) Improve security by: +1. Optional. Improve security by: 1. Disabling non-TLS connections by commenting out or deleting `listen_addr` in `/home/git/gitaly/config.toml`. 1. Saving the file. diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index c689530e12c..f99bbf21840 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -189,8 +189,7 @@ The availability objectives for Gitaly clusters assuming a single node failure a Writes are replicated asynchronously. Any writes that have not been replicated to the newly promoted primary are lost. - [Strong consistency](#strong-consistency) can be used to avoid loss in some - circumstances. + [Strong consistency](#strong-consistency) prevents loss in some circumstances. - **Recovery Time Objective (RTO):** Less than 10 seconds. Outages are detected by a health check run by each Praefect node every @@ -284,8 +283,7 @@ Gitaly Cluster provides the following features: - [Replication factor](#replication-factor) of repositories for increased redundancy. - [Automatic failover](praefect.md#automatic-failover-and-primary-election-strategies) from the primary Gitaly node to secondary Gitaly nodes. -- Reporting of possible [data loss](praefect.md#check-for-data-loss) if replication queue is - non-empty. +- Reporting of possible [data loss](recovery.md#check-for-data-loss) if replication queue isn't empty. Follow the [Gitaly Cluster epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) for improvements including [horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013). @@ -323,18 +321,26 @@ You can [monitor distribution of reads](#monitor-gitaly-cluster) using Prometheu > - In GitLab 13.3, disabled unless primary-wins voting strategy is disabled. > - From GitLab 13.4, enabled by default. > - From GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency. -> - From GitLab 13.6, primary-wins voting strategy and `gitaly_reference_transactions_primary_wins` feature flag were removed from the source code. +> - From GitLab 13.6, primary-wins voting strategy and the `gitaly_reference_transactions_primary_wins` feature flag was removed. +> - From GitLab 14.0, [Gitaly Cluster only supports strong consistency](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3575), and the `gitaly_reference_transactions` feature flag was removed. -By default, Gitaly Cluster guarantees eventual consistency by replicating all writes to secondary -Gitaly nodes after the write to the primary Gitaly node has happened. +Gitaly Cluster provides strong consistency by writing changes synchronously to all healthy, up-to-date replicas. If a +replica is outdated or unhealthy at the time of the transaction, the write is asynchronously replicated to it. -Praefect can instead provide strong consistency by creating a transaction and writing changes to all -Gitaly nodes at once. +If strong consistency is unavailable, Gitaly Cluster guarantees eventual consistency. In this case. Gitaly Cluster +replicates all writes to secondary Gitaly nodes after the write to the primary Gitaly node has occurred. -If enabled, transactions are only available for a subset of RPCs. For more information, see the -[strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189). +Strong consistency: -For configuration information, see [Configure strong consistency](praefect.md#configure-strong-consistency). +- Is the primary replication method in GitLab 14.0 and later. A subset of operations still use replication jobs + (eventual consistency) instead of strong consistency. Refer to the + [strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189) for more information. +- Must be configured in GitLab versions 13.1 to 13.12. For configuration information, refer to either: + - Documentation on your GitLab instance at `/help`. + - The [13.12 documentation](https://docs.gitlab.com/13.12/ee/administration/gitaly/praefect.html#strong-consistency). +- Is unavailable in GitLab 13.0 and earlier. + +For more information on monitoring strong consistency, see the Gitaly Cluster [Prometheus metrics documentation](#monitor-gitaly-cluster). #### Replication factor @@ -368,6 +374,10 @@ WARNING: Some [known database inconsistency issues](#known-issues) exist in Gitaly Cluster. We recommend you remain on your current service for now. +NOTE: +GitLab requires a `default` repository storage to be configured. +[Read more about this limitation](configure_gitaly.md#gitlab-requires-a-default-repository-storage). + ### Migrate off Gitaly Cluster If you have repositories stored on a Gitaly Cluster, but you'd like to migrate @@ -513,6 +523,10 @@ To monitor [strong consistency](#strong-consistency), you can use the following You can also monitor the [Praefect logs](../logs.md#praefect-logs). +## Recover from failure + +Gitaly Cluster can [recover from certain types of failure](recovery.md). + ## Do not bypass Gitaly GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client, diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index da456131a52..d3a8662080f 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -215,6 +215,38 @@ The database used by Praefect is now configured. If you see Praefect database errors after configuring PostgreSQL, see [troubleshooting steps](troubleshooting.md#relation-does-not-exist-errors). +#### Reads distribution caching + +Praefect performance can be improved by additionally configuring the `database_direct` +settings: + +```ruby +praefect['database_direct_host'] = POSTGRESQL_HOST +praefect['database_direct_port'] = 5432 + +# Use the following to override parameters of direct database connection. +# Comment out where the parameters are the same for both connections. + +praefect['database_direct_user'] = 'praefect' +praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD +praefect['database_direct_dbname'] = 'praefect_production' +#praefect['database_direct_sslmode'] = '...' +#praefect['database_direct_sslcert'] = '...' +#praefect['database_direct_sslkey'] = '...' +#praefect['database_direct_sslrootcert'] = '...' +``` + +Once configured, this connection is automatically used for the +[SQL LISTEN](https://www.postgresql.org/docs/11/sql-listen.html) feature and +allows Praefect to receive notifications from PostgreSQL for cache invalidation. + +Verify this feature is working by looking for the following log entry in the Praefect +log: + +```plaintext +reads distribution caching is enabled by configuration +``` + #### Use PgBouncer To reduce PostgreSQL resource consumption, we recommend setting up and configuring @@ -223,7 +255,7 @@ this, you must point Praefect to PgBouncer by setting Praefect database paramete ```ruby praefect['database_host'] = PGBOUNCER_HOST -praefect['database_port'] = 6432 +praefect['database_port'] = 5432 praefect['database_user'] = 'praefect' praefect['database_password'] = PRAEFECT_SQL_PASSWORD praefect['database_dbname'] = 'praefect_production' @@ -1073,31 +1105,6 @@ To get started quickly: Congratulations! You've configured an observable fault-tolerant Praefect cluster. -## Configure strong consistency - -To enable [strong consistency](index.md#strong-consistency): - -- In GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency. -- In GitLab 13.4 and later, the strong consistency voting strategy has been improved and enabled by default. - Instead of requiring all nodes to agree, only the primary and half of the secondaries need to agree. -- In GitLab 13.3, reference transactions are enabled by default with a primary-wins strategy. - This strategy causes all transactions to succeed for the primary and thus does not ensure strong consistency. - To enable strong consistency, disable the `:gitaly_reference_transactions_primary_wins` feature flag. -- In GitLab 13.2, enable the `:gitaly_reference_transactions` feature flag. -- In GitLab 13.1, enable the `:gitaly_reference_transactions` and `:gitaly_hooks_rpc` - feature flags. - -Changing feature flags requires [access to the Rails console](../feature_flags.md#start-the-gitlab-rails-console). -In the Rails console, enable or disable the flags as required. For example: - -```ruby -Feature.enable(:gitaly_reference_transactions) -Feature.disable(:gitaly_reference_transactions_primary_wins) -``` - -For information on monitoring strong consistency, see the -[relevant documentation](index.md#monitor-gitaly-cluster). - ## Configure replication factor WARNING: @@ -1153,8 +1160,7 @@ Praefect regularly checks the health of each Gitaly node. This is used to automa to a newly-elected primary Gitaly node if the current primary node is found to be unhealthy. We recommend using [repository-specific primary nodes](#repository-specific-primary-nodes). This is -[planned to be the only available election strategy](https://gitlab.com/gitlab-org/gitaly/-/issues/3574) -from GitLab 14.0. +[the only available election strategy](https://gitlab.com/gitlab-org/gitaly/-/issues/3574) from GitLab 14.0. ### Repository-specific primary nodes @@ -1268,7 +1274,7 @@ To migrate existing clusters: ### Deprecated election strategies WARNING: -The below election strategies are deprecated and are scheduled for removal in GitLab 14.0. +The below election strategies are deprecated and were removed in GitLab 14.0. Migrate to [repository-specific primary nodes](#repository-specific-primary-nodes). - **PostgreSQL:** Enabled by default until GitLab 14.0, and equivalent to: @@ -1287,397 +1293,3 @@ Migrate to [repository-specific primary nodes](#repository-specific-primary-node If a sufficient number of health checks fail for the current primary Gitaly node, a new primary is elected. **Do not use with multiple Praefect nodes!** Using with multiple Praefect nodes is likely to result in a split brain. - -## Primary Node Failure - -Gitaly Cluster recovers from a failing primary Gitaly node by promoting a healthy secondary as the -new primary. - -In GitLab 14.1 and later, Gitaly Cluster: - -- Elects a healthy secondary with a fully up to date copy of the repository as the new primary. -- Repository becomes unavailable if there are no fully up to date copies of it on healthy secondaries. - -To minimize data loss in GitLab 13.0 to 14.0, Gitaly Cluster: - -- Switches repositories that are outdated on the new primary to [read-only mode](#read-only-mode). -- Elects the secondary with the least unreplicated writes from the primary to be the new - primary. Because there can still be some unreplicated writes, - [data loss can occur](#check-for-data-loss). - -### Read-only mode - -> - Introduced in GitLab 13.0 as [generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). -> - Between GitLab 13.0 and GitLab 13.2, read-only mode applied to the whole virtual storage and occurred whenever failover occurred. -> - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitaly/-/issues/2862), read-only mode applies on a per-repository basis and only occurs if a new primary is out of date. -new primary. If the failed primary contained unreplicated writes, [data loss can occur](#check-for-data-loss). -> - Removed in GitLab 14.1. Instead, repositories [become unavailable](#unavailable-repositories). - -When Gitaly Cluster switches to a new primary in GitLab 13.0 to 14.0, repositories enter -read-only mode if they are out of date. This can happen after failing over to an outdated -secondary. Read-only mode eases data recovery efforts by preventing writes that may conflict -with the unreplicated writes on other nodes. - -To enable writes again in GitLab 13.0 to 14.0, an administrator can: - -1. [Check](#check-for-data-loss) for data loss. -1. Attempt to [recover](#data-recovery) missing data. -1. Either [enable writes](#enable-writes-or-accept-data-loss) in the virtual storage or - [accept data loss](#enable-writes-or-accept-data-loss) if necessary, depending on the version of - GitLab. - -## Unavailable repositories - -> - From GitLab 13.0 through 14.0, repositories became read-only if they were outdated on the primary but fully up to date on a healthy secondary. `dataloss` sub-command displays read-only repositories by default through these versions. -> - Since GitLab 14.1, Praefect contains more responsive failover logic which immediately fails over to one of the fully up to date secondaries rather than placing the repository in read-only mode. Since GitLab 14.1, the `dataloss` sub-command displays repositories which are unavailable due to having no fully up to date copies on healthy Gitaly nodes. - -A repository is unavailable if all of its up to date replicas are unavailable. Unavailable repositories are -not accessible through Praefect to prevent serving stale data that may break automated tooling. - -### Check for data loss - -The Praefect `dataloss` subcommand identifies: - -- Copies of repositories in GitLab 13.0 to GitLab 14.0 that at are likely to be outdated. - This can help identify potential data loss after a failover. -- Repositories in GitLab 14.1 and later that are unavailable. This helps identify potential - data loss and repositories which are no longer accessible because all of their up-to-date - replicas copies are unavailable. - -The following parameters are available: - -- `-virtual-storage` that specifies which virtual storage to check. Because they might require - an administrator to intervene, the default behavior is to display: - - In GitLab 13.0 to 14.0, copies of read-only repositories. - - In GitLab 14.1 and later, unavailable repositories. -- In GitLab 14.1 and later, [`-partially-unavailable`](#unavailable-replicas-of-available-repositories) - that specifies whether to include in the output repositories that are available but have - some assigned copies that are not available. - -NOTE: -`dataloss` is still in beta and the output format is subject to change. - -To check for repositories with outdated primaries or for unavailable repositories, run: - -```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss [-virtual-storage <virtual-storage>] -``` - -Every configured virtual storage is checked if none is specified: - -```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss -``` - -Repositories are listed in the output that have either: - -- An outdated copy of the repository on the primary, in GitLab 13.0 to GitLab 14.0. -- No healthy and fully up-to-date copies available, in GitLab 14.1 and later. - -The following information is printed for each repository: - -- A repository's relative path to the storage directory identifies each repository and groups the related - information. -- The repository's current status is printed in parentheses next to the disk path: - - In GitLab 13.0 to 14.0, either `(read-only)` if the repository's primary node is outdated - and can't accept writes. Otherwise, `(writable)`. - - In GitLab 14.1 and later, `(unavailable)` is printed next to the disk path if the - repository is unavailable. -- The primary field lists the repository's current primary. If the repository has no primary, the field shows - `No Primary`. -- The In-Sync Storages lists replicas which have replicated the latest successful write and all writes - preceding it. -- The Outdated Storages lists replicas which contain an outdated copy of the repository. Replicas which have no copy - of the repository but should contain it are also listed here. The maximum number of changes the replica is missing - is listed next to replica. It's important to notice that the outdated replicas may be fully up to date or contain - later changes but Praefect can't guarantee it. - -Additional information includes: - -- Whether a node is assigned to host the repository is listed with each node's status. - `assigned host` is printed next to nodes that are assigned to store the repository. The - text is omitted if the node contains a copy of the repository but is not assigned to store - the repository. Such copies aren't kept in sync by Praefect, but may act as replication - sources to bring assigned copies up to date. -- In GitLab 14.1 and later, `unhealthy` is printed next to the copies that are located - on unhealthy Gitaly nodes. - -Example output: - -```shell -Virtual storage: default - Outdated repositories: - @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git (unavailable): - Primary: gitaly-1 - In-Sync Storages: - gitaly-2, assigned host, unhealthy - Outdated Storages: - gitaly-1 is behind by 3 changes or less, assigned host - gitaly-3 is behind by 3 changes or less -``` - -A confirmation is printed out when every repository is available. For example: - -```shell -Virtual storage: default - All repositories are available! -``` - -#### Unavailable replicas of available repositories - -NOTE: -In GitLab 14.0 and earlier, the flag is `-partially-replicated` and the output shows any repositories with assigned nodes with outdated -copies. - -To also list information of repositories which are available but are unavailable from some of the assigned nodes, -use the `-partially-unavailable` flag. - -A repository is available if there is a healthy, up to date replica available. Some of the assigned secondary -replicas may be temporarily unavailable for access while they are waiting to replicate the latest changes. - -```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss [-virtual-storage <virtual-storage>] [-partially-unavailable] -``` - -Example output: - -```shell -Virtual storage: default - Outdated repositories: - @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git: - Primary: gitaly-1 - In-Sync Storages: - gitaly-1, assigned host - Outdated Storages: - gitaly-2 is behind by 3 changes or less, assigned host - gitaly-3 is behind by 3 changes or less -``` - -With the `-partially-unavailable` flag set, a confirmation is printed out if every assigned replica is fully up to -date and healthy. - -For example: - -```shell -Virtual storage: default - All repositories are fully available on all assigned storages! -``` - -### Check repository checksums - -To check a project's repository checksums across on all Gitaly nodes, run the -[replicas Rake task](../raketasks/praefect.md#replica-checksums) on the main GitLab node. - -### Accept data loss - -WARNING: -`accept-dataloss` causes permanent data loss by overwriting other versions of the repository. Data -[recovery efforts](#data-recovery) must be performed before using it. - -If it is not possible to bring one of the up to date replicas back online, you may have to accept data -loss. When accepting data loss, Praefect marks the chosen replica of the repository as the latest version -and replicates it to the other assigned Gitaly nodes. This process overwrites any other version of the -repository so care must be taken. - -```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss --virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name> -``` - -### Enable writes or accept data loss - -WARNING: -`accept-dataloss` causes permanent data loss by overwriting other versions of the repository. -Data [recovery efforts](#data-recovery) must be performed before using it. - -Praefect provides the following subcommands to re-enable writes or accept data loss: - -- In GitLab 13.2 and earlier, `enable-writes` to re-enable virtual storage for writes after - data recovery attempts: - - ```shell - sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml enable-writes -virtual-storage <virtual-storage> - ``` - -- In GitLab 13.3 and later, if it is not possible to bring one of the up to date nodes back - online, you may have to accept data loss: - - ```shell - sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name> - ``` - - When accepting data loss, Praefect: - - 1. Marks the chosen copy of the repository as the latest version. - 1. Replicates the copy to the other assigned Gitaly nodes. - - This process overwrites any other copy of the repository so care must be taken. - -## Data recovery - -If a Gitaly node fails replication jobs for any reason, it ends up hosting outdated versions of the -affected repositories. Praefect provides tools for: - -- [Automatic](#automatic-reconciliation) reconciliation, for GitLab 13.4 and later. -- [Manual](#manual-reconciliation) reconciliation, for: - - GitLab 13.3 and earlier. - - Repositories upgraded to GitLab 13.4 and later without entries in the `repositories` table. In - GitLab 13.6 and later, [a migration is run](https://gitlab.com/gitlab-org/gitaly/-/issues/3033) - when Praefect starts for these repositories. - -These tools reconcile the outdated repositories to bring them fully up to date again. - -### Automatic reconciliation - -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2717) in GitLab 13.4. - -Praefect automatically reconciles repositories that are not up to date. By default, this is done every -five minutes. For each outdated repository on a healthy Gitaly node, the Praefect picks a -random, fully up-to-date replica of the repository on another healthy Gitaly node to replicate from. A -replication job is scheduled only if there are no other replication jobs pending for the target -repository. - -The reconciliation frequency can be changed via the configuration. The value can be any valid -[Go duration value](https://golang.org/pkg/time/#ParseDuration). Values below 0 disable the feature. - -Examples: - -```ruby -praefect['reconciliation_scheduling_interval'] = '5m' # the default value -``` - -```ruby -praefect['reconciliation_scheduling_interval'] = '30s' # reconcile every 30 seconds -``` - -```ruby -praefect['reconciliation_scheduling_interval'] = '0' # disable the feature -``` - -### Manual reconciliation - -WARNING: -The `reconcile` sub-command was removed in GitLab 14.1. Use [automatic reconciliation](#automatic-reconciliation) instead. Manual reconciliation may produce excess replication jobs and is limited in functionality. Manual reconciliation does not work when [repository-specific primary nodes](#repository-specific-primary-nodes) are -enabled. - -The Praefect `reconcile` sub-command allows for the manual reconciliation between two Gitaly nodes. The -command replicates every repository on a later version on the reference storage to the target storage. - -```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml reconcile -virtual <virtual-storage> -reference <up-to-date-storage> -target <outdated-storage> -f -``` - -- Replace the placeholder `<virtual-storage>` with the virtual storage containing the Gitaly node storage to be checked. -- Replace the placeholder `<up-to-date-storage>` with the Gitaly storage name containing up to date repositories. -- Replace the placeholder `<outdated-storage>` with the Gitaly storage name containing outdated repositories. - -### Manually remove repositories - -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3767) in GitLab 14.3. - -The `remove-repository` Praefect sub-command removes repositories from a Gitaly Cluster. It removes -all state associated with a given repository including: - -- On-disk repositories on all relevant Gitaly nodes. -- Any database state tracked by Praefect. - -```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository> -``` - -- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following: - - ```ruby - praefect['virtual_storages'] = { - 'default' => { - ... - }, - 'storage-1' => { - ... - } - } - ``` - - In this example, the virtual storage to specify is `default` or `storage-1`. - -- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_types.md#hashed-storage). - For example: - - ```plaintext - @hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git - ``` - -Parts of the repository can continue to exist after running `remove-repository`. This can be because of: - -- A deletion error. -- An in-flight RPC call targeting the repository. - -If this occurs, run `remove-repository` again. - -### Manually list untracked repositories - -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3926) in GitLab 14.4. - -The `list-untracked-repositories` Praefect sub-command lists repositories of the Gitaly Cluster that both: - -- Exist for at least one Gitaly storage. -- Aren't tracked in the Praefect database. - -The command outputs: - -- Result to `STDOUT` and the command's logs. -- Errors to `STDERR`. - -Each entry is a complete JSON string with a newline at the end (configurable using the -`-delimiter` flag). For example: - -```plaintext -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml list-untracked-repositories -{"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567890.git"} -{"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567891.git"} -``` - -### Manually track repositories - -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5658) in GitLab 14.4. - -The `track-repository` Praefect sub-command adds repositories on disk to the Praefect database to be tracked. - -```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -repository <repository> -``` - -- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following: - - ```ruby - praefect['virtual_storages'] = { - 'default' => { - ... - }, - 'storage-1' => { - ... - } - } - ``` - - In this example, the virtual storage to specify is `default` or `storage-1`. - -- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_types.md#hashed-storage). - For example: - - ```plaintext - @hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git - ``` - -- `-authoritative-storage` is the storage we want Praefect to treat as the primary. Required if - [per-repository replication](#configure-replication-factor) is set as the replication strategy. - -The command outputs: - -- Results to `STDOUT` and the command's logs. -- Errors to `STDERR`. - -This command fails if: - -- The repository is already being tracked by the Praefect database. -- The repository does not exist on disk. diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md new file mode 100644 index 00000000000..e1b9a73908d --- /dev/null +++ b/doc/administration/gitaly/recovery.md @@ -0,0 +1,418 @@ +--- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Recovery options + +Gitaly Cluster can [recover from certain types of failure](recovery.md). + +## Primary Node Failure + +Gitaly Cluster recovers from a failing primary Gitaly node by promoting a healthy secondary as the +new primary. + +In GitLab 14.1 and later, Gitaly Cluster: + +- Elects a healthy secondary with a fully up to date copy of the repository as the new primary. +- Repository becomes unavailable if there are no fully up to date copies of it on healthy secondaries. + +To minimize data loss in GitLab 13.0 to 14.0, Gitaly Cluster: + +- Switches repositories that are outdated on the new primary to [read-only mode](#read-only-mode). +- Elects the secondary with the least unreplicated writes from the primary to be the new + primary. Because there can still be some unreplicated writes, + [data loss can occur](#check-for-data-loss). + +### Read-only mode + +> - Introduced in GitLab 13.0 as [generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). +> - Between GitLab 13.0 and GitLab 13.2, read-only mode applied to the whole virtual storage and occurred whenever failover occurred. +> - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitaly/-/issues/2862), read-only mode applies on a per-repository basis and only occurs if a new primary is out of date. +new primary. If the failed primary contained unreplicated writes, [data loss can occur](#check-for-data-loss). +> - Removed in GitLab 14.1. Instead, repositories [become unavailable](#unavailable-repositories). + +When Gitaly Cluster switches to a new primary in GitLab 13.0 to 14.0, repositories enter +read-only mode if they are out of date. This can happen after failing over to an outdated +secondary. Read-only mode eases data recovery efforts by preventing writes that may conflict +with the unreplicated writes on other nodes. + +To enable writes again in GitLab 13.0 to 14.0, an administrator can: + +1. [Check](#check-for-data-loss) for data loss. +1. Attempt to [recover](#data-recovery) missing data. +1. Either [enable writes](#enable-writes-or-accept-data-loss) in the virtual storage or + [accept data loss](#enable-writes-or-accept-data-loss) if necessary, depending on the version of + GitLab. + +## Unavailable repositories + +> - From GitLab 13.0 through 14.0, repositories became read-only if they were outdated on the primary but fully up to date on a healthy secondary. `dataloss` sub-command displays read-only repositories by default through these versions. +> - Since GitLab 14.1, Praefect contains more responsive failover logic which immediately fails over to one of the fully up to date secondaries rather than placing the repository in read-only mode. Since GitLab 14.1, the `dataloss` sub-command displays repositories which are unavailable due to having no fully up to date copies on healthy Gitaly nodes. + +A repository is unavailable if all of its up to date replicas are unavailable. Unavailable repositories are +not accessible through Praefect to prevent serving stale data that may break automated tooling. + +### Check for data loss + +The Praefect `dataloss` subcommand identifies: + +- Copies of repositories in GitLab 13.0 to GitLab 14.0 that at are likely to be outdated. + This can help identify potential data loss after a failover. +- Repositories in GitLab 14.1 and later that are unavailable. This helps identify potential + data loss and repositories which are no longer accessible because all of their up-to-date + replicas copies are unavailable. + +The following parameters are available: + +- `-virtual-storage` that specifies which virtual storage to check. Because they might require + an administrator to intervene, the default behavior is to display: + - In GitLab 13.0 to 14.0, copies of read-only repositories. + - In GitLab 14.1 and later, unavailable repositories. +- In GitLab 14.1 and later, [`-partially-unavailable`](#unavailable-replicas-of-available-repositories) + that specifies whether to include in the output repositories that are available but have + some assigned copies that are not available. + +NOTE: +`dataloss` is still in beta and the output format is subject to change. + +To check for repositories with outdated primaries or for unavailable repositories, run: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss [-virtual-storage <virtual-storage>] +``` + +Every configured virtual storage is checked if none is specified: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss +``` + +Repositories are listed in the output that have either: + +- An outdated copy of the repository on the primary, in GitLab 13.0 to GitLab 14.0. +- No healthy and fully up-to-date copies available, in GitLab 14.1 and later. + +The following information is printed for each repository: + +- A repository's relative path to the storage directory identifies each repository and groups the related + information. +- The repository's current status is printed in parentheses next to the disk path: + - In GitLab 13.0 to 14.0, either `(read-only)` if the repository's primary node is outdated + and can't accept writes. Otherwise, `(writable)`. + - In GitLab 14.1 and later, `(unavailable)` is printed next to the disk path if the + repository is unavailable. +- The primary field lists the repository's current primary. If the repository has no primary, the field shows + `No Primary`. +- The In-Sync Storages lists replicas which have replicated the latest successful write and all writes + preceding it. +- The Outdated Storages lists replicas which contain an outdated copy of the repository. Replicas which have no copy + of the repository but should contain it are also listed here. The maximum number of changes the replica is missing + is listed next to replica. It's important to notice that the outdated replicas may be fully up to date or contain + later changes but Praefect can't guarantee it. + +Additional information includes: + +- Whether a node is assigned to host the repository is listed with each node's status. + `assigned host` is printed next to nodes that are assigned to store the repository. The + text is omitted if the node contains a copy of the repository but is not assigned to store + the repository. Such copies aren't kept in sync by Praefect, but may act as replication + sources to bring assigned copies up to date. +- In GitLab 14.1 and later, `unhealthy` is printed next to the copies that are located + on unhealthy Gitaly nodes. + +Example output: + +```shell +Virtual storage: default + Outdated repositories: + @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git (unavailable): + Primary: gitaly-1 + In-Sync Storages: + gitaly-2, assigned host, unhealthy + Outdated Storages: + gitaly-1 is behind by 3 changes or less, assigned host + gitaly-3 is behind by 3 changes or less +``` + +A confirmation is printed out when every repository is available. For example: + +```shell +Virtual storage: default + All repositories are available! +``` + +#### Unavailable replicas of available repositories + +NOTE: +In GitLab 14.0 and earlier, the flag is `-partially-replicated` and the output shows any repositories with assigned nodes with outdated +copies. + +To also list information of repositories which are available but are unavailable from some of the assigned nodes, +use the `-partially-unavailable` flag. + +A repository is available if there is a healthy, up to date replica available. Some of the assigned secondary +replicas may be temporarily unavailable for access while they are waiting to replicate the latest changes. + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss [-virtual-storage <virtual-storage>] [-partially-unavailable] +``` + +Example output: + +```shell +Virtual storage: default + Outdated repositories: + @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git: + Primary: gitaly-1 + In-Sync Storages: + gitaly-1, assigned host + Outdated Storages: + gitaly-2 is behind by 3 changes or less, assigned host + gitaly-3 is behind by 3 changes or less +``` + +With the `-partially-unavailable` flag set, a confirmation is printed out if every assigned replica is fully up to +date and healthy. + +For example: + +```shell +Virtual storage: default + All repositories are fully available on all assigned storages! +``` + +### Check repository checksums + +To check a project's repository checksums across on all Gitaly nodes, run the +[replicas Rake task](../raketasks/praefect.md#replica-checksums) on the main GitLab node. + +### Accept data loss + +WARNING: +`accept-dataloss` causes permanent data loss by overwriting other versions of the repository. Data +[recovery efforts](#data-recovery) must be performed before using it. + +If it is not possible to bring one of the up to date replicas back online, you may have to accept data +loss. When accepting data loss, Praefect marks the chosen replica of the repository as the latest version +and replicates it to the other assigned Gitaly nodes. This process overwrites any other version of the +repository so care must be taken. + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss +-virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name> +``` + +### Enable writes or accept data loss + +WARNING: +`accept-dataloss` causes permanent data loss by overwriting other versions of the repository. +Data [recovery efforts](#data-recovery) must be performed before using it. + +Praefect provides the following subcommands to re-enable writes or accept data loss: + +- In GitLab 13.2 and earlier, `enable-writes` to re-enable virtual storage for writes after + data recovery attempts: + + ```shell + sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml enable-writes -virtual-storage <virtual-storage> + ``` + +- In GitLab 13.3 and later, if it is not possible to bring one of the up to date nodes back + online, you may have to accept data loss: + + ```shell + sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name> + ``` + + When accepting data loss, Praefect: + + 1. Marks the chosen copy of the repository as the latest version. + 1. Replicates the copy to the other assigned Gitaly nodes. + + This process overwrites any other copy of the repository so care must be taken. + +## Data recovery + +If a Gitaly node fails replication jobs for any reason, it ends up hosting outdated versions of the +affected repositories. Praefect provides tools for: + +- [Automatic](#automatic-reconciliation) reconciliation, for GitLab 13.4 and later. +- [Manual](#manual-reconciliation) reconciliation, for: + - GitLab 13.3 and earlier. + - Repositories upgraded to GitLab 13.4 and later without entries in the `repositories` table. In + GitLab 13.6 and later, [a migration is run](https://gitlab.com/gitlab-org/gitaly/-/issues/3033) + when Praefect starts for these repositories. + +These tools reconcile the outdated repositories to bring them fully up to date again. + +### Automatic reconciliation + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2717) in GitLab 13.4. + +Praefect automatically reconciles repositories that are not up to date. By default, this is done every +five minutes. For each outdated repository on a healthy Gitaly node, the Praefect picks a +random, fully up-to-date replica of the repository on another healthy Gitaly node to replicate from. A +replication job is scheduled only if there are no other replication jobs pending for the target +repository. + +The reconciliation frequency can be changed via the configuration. The value can be any valid +[Go duration value](https://pkg.go.dev/time#ParseDuration). Values below 0 disable the feature. + +Examples: + +```ruby +praefect['reconciliation_scheduling_interval'] = '5m' # the default value +``` + +```ruby +praefect['reconciliation_scheduling_interval'] = '30s' # reconcile every 30 seconds +``` + +```ruby +praefect['reconciliation_scheduling_interval'] = '0' # disable the feature +``` + +### Manual reconciliation + +WARNING: +The `reconcile` sub-command was removed in GitLab 14.1. Use [automatic reconciliation](#automatic-reconciliation) instead. +Manual reconciliation may produce excess replication jobs and is limited in functionality. Manual reconciliation does not +work when [repository-specific primary nodes](praefect.md#repository-specific-primary-nodes) are enabled. + +The Praefect `reconcile` sub-command allows for the manual reconciliation between two Gitaly nodes. The +command replicates every repository on a later version on the reference storage to the target storage. + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml reconcile -virtual <virtual-storage> -reference <up-to-date-storage> -target <outdated-storage> -f +``` + +- Replace the placeholder `<virtual-storage>` with the virtual storage containing the Gitaly node storage to be checked. +- Replace the placeholder `<up-to-date-storage>` with the Gitaly storage name containing up to date repositories. +- Replace the placeholder `<outdated-storage>` with the Gitaly storage name containing outdated repositories. + +### Manually remove repositories + +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3767) in GitLab 14.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4054) in GitLab 14.6, support for dry-run mode. + +The `remove-repository` Praefect sub-command removes a repository from a Gitaly Cluster, and all state associated with a given repository including: + +- On-disk repositories on all relevant Gitaly nodes. +- Any database state tracked by Praefect. + +In GitLab 14.6 and later, by default, the command operates in dry-run mode. In earlier versions, the command didn't support dry-run mode. For example: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository> +``` + +- Replace `<virtual-storage>` with the name of the virtual storage containing the repository. +- Replace `<repository>` with the relative path of the repository to remove. +- In GitLab 14.6 and later, add `-apply` to run the command outside of dry-run mode and remove the repository. For example: + + ```shell + sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository> -apply + ``` + +- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following: + + ```ruby + praefect['virtual_storages'] = { + 'default' => { + ... + }, + 'storage-1' => { + ... + } + } + ``` + + In this example, the virtual storage to specify is `default` or `storage-1`. + +- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_types.md#hashed-storage). + For example: + + ```plaintext + @hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git + ``` + +Parts of the repository can continue to exist after running `remove-repository`. This can be because of: + +- A deletion error. +- An in-flight RPC call targeting the repository. + +If this occurs, run `remove-repository` again. + +### Manually list untracked repositories + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3926) in GitLab 14.4. + +The `list-untracked-repositories` Praefect sub-command lists repositories of the Gitaly Cluster that both: + +- Exist for at least one Gitaly storage. +- Aren't tracked in the Praefect database. + +The command outputs: + +- Result to `STDOUT` and the command's logs. +- Errors to `STDERR`. + +Each entry is a complete JSON string with a newline at the end (configurable using the +`-delimiter` flag). For example: + +```plaintext +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml list-untracked-repositories +{"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567890.git"} +{"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567891.git"} +``` + +### Manually track repositories + +> - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5658) in GitLab 14.4. +> - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5789) in GitLab 14.6, support for immediate replication. + +The `track-repository` Praefect sub-command adds repositories on disk to the Praefect database to be tracked. + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -repository <repository> -replicate-immediately +``` + +- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following: + + ```ruby + praefect['virtual_storages'] = { + 'default' => { + ... + }, + 'storage-1' => { + ... + } + } + ``` + + In this example, the virtual storage to specify is `default` or `storage-1`. + +- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_types.md#hashed-storage). + For example: + + ```plaintext + @hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git + ``` + +- `-authoritative-storage` is the storage we want Praefect to treat as the primary. Required if + [per-repository replication](praefect.md#configure-replication-factor) is set as the replication strategy. +- `-replicate-immediately`, available in GitLab 14.6 and later, causes the command to replicate the repository to its secondaries immediately. + Otherwise, replication jobs are scheduled for execution in the database and are picked up by a Praefect background process. + +The command outputs: + +- Results to `STDOUT` and the command's logs. +- Errors to `STDERR`. + +This command fails if: + +- The repository is already being tracked by the Praefect database. +- The repository does not exist on disk. diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index d6d93b8af94..fdd281c1a90 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -153,7 +153,7 @@ Confirm the following are all true: - When any user adds or modifies a file from the repository using the GitLab UI, it immediately fails with a red `401 Unauthorized` banner. -- Creating a new project and [initializing it with a README](../../user/project/working_with_projects.md#blank-projects) +- Creating a new project and [initializing it with a README](../../user/project/working_with_projects.md#create-a-blank-project) successfully creates the project but doesn't create the README. - When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on a Gitaly client and reproducing the error, you get `401` errors @@ -328,10 +328,94 @@ experiencing [clock drift](https://en.wikipedia.org/wiki/Clock_drift). Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time server to keep them synchronized if possible. +### Health check warnings + +The following warning in `/var/log/gitlab/praefect/current` can be ignored. + +```plaintext +"error":"full method name not found: /grpc.health.v1.Health/Check", +"msg":"error when looking up method info" +``` + +### File not found errors + +The following errors in `/var/log/gitlab/gitaly/current` can be ignored. +They are caused by the GitLab Rails application checking for specific files +that do not exist in a repository. + +```plaintext +"error":"not found: .gitlab/route-map.yml" +"error":"not found: Dockerfile" +"error":"not found: .gitlab-ci.yml" +``` + ## Troubleshoot Praefect (Gitaly Cluster) The following sections provide possible solutions to Gitaly Cluster errors. +### Check cluster health + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/) in GitLab 14.6. + +The `check` Praefect sub-command runs a series of checks to determine the health of the Gitaly Cluster. + +```shell +gitlab-ctl praefect check +``` + +The following sections describe the checks that are run. + +#### Praefect migrations + +Because Database migrations must be up to date for Praefect to work correctly, checks if Praefect migrations are up to date. + +If this check fails: + +1. See the `schema_migrations` table in the database to see which migrations have run. +1. Run `praefect sql-migrate` to bring the migrations up to date. + +#### Node connectivity and disk access + +Checks if Praefect can reach all of its Gitaly nodes, and if each Gitaly node has read and write access to all of its storages. + +If this check fails: + +1. Confirm the network addresses and tokens are set up correctly: + - In the Praefect configuration. + - In each Gitaly node's configuration. +1. On the Gitaly nodes, check that the `gitaly` process being run as `git`. There might be a permissions issue that is preventing Gitaly from + accessing its storage directories. +1. Confirm that there are no issues with the network that connects Praefect to Gitaly nodes. + +#### Database read and write access + +Checks if Praefect can read from and write to the database. + +If this check fails: + +1. See if the Praefect database is in recovery mode. In recovery mode, tables may be read only. To check, run: + + ```sql + select pg_is_in_recovery() + ``` + +1. Confirm that the user that Praefect uses to connect to PostgreSQL has read and write access to the database. +1. See if the database has been placed into read-only mode. To check, run: + + ```sql + show default_transaction_read_only + ``` + +#### Inaccessible repositories + +Checks how many repositories are inaccessible because they are missing a primary assignment, or their primary is unavailable. + +If this check fails: + +1. See if any Gitaly nodes are down. Run `praefect ping-nodes` to check. +1. Check if there is a high load on the Praefect database. If the Praefect database is slow to respond, it can lead health checks failing to persist + to the database, leading Praefect to think nodes are unhealthy. + ### Praefect errors in logs If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`. @@ -353,17 +437,107 @@ Here are common errors and potential causes: ### Determine primary Gitaly node -To determine the current primary Gitaly node for a specific Praefect node: +To determine the primary node of a repository: -- Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the - [`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json). - This is recommended. -- If you do not have Grafana set up, use the following command on each host of each - Praefect node: +- In GitLab 14.6 and later, use the [`praefect metadata`](#view-repository-metadata) subcommand. +- In GitLab 13.12 to GitLab 14.5 with [repository-specific primaries](praefect.md#repository-specific-primary-nodes), + use the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums). +- With legacy election strategies in GitLab 13.12 and earlier, the primary was the same for all repositories in a virtual storage. + To determine the current primary Gitaly node for a specific virtual storage: - ```shell - curl localhost:9652/metrics | grep gitaly_praefect_primaries` - ``` + - Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the + [`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json). + This is recommended. + - If you do not have Grafana set up, use the following command on each host of each + Praefect node: + + ```shell + curl localhost:9652/metrics | grep gitaly_praefect_primaries` + ``` + +### View repository metadata + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3481) in GitLab 14.6. + +Gitaly Cluster maintains a [metadata database](index.md#components) about the repositories stored on the cluster. Use the `praefect metadata` subcommand +to inspect the metadata for troubleshooting. + +You can retrieve a repository's metadata by its Praefect-assigned repository ID: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id <repository-id> +``` + +You can also retrieve a repository's metadata by its virtual storage and relative path: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -virtual-storage <virtual-storage> -relative-path <relative-path> +``` + +#### Examples + +To retrieve the metadata for a repository with a Praefect-assigned repository ID of 1: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id 1 +``` + +To retrieve the metadata for a repository with virtual storage `default` and relative path `@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git`: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -virtual-storage default -relative-path @hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git +``` + +Either of these examples retrieve the following metadata for an example repository: + +```plaintext +Repository ID: 54771 +Virtual Storage: "default" +Relative Path: "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git" +Replica Path: "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git" +Primary: "gitaly-1" +Generation: 1 +Replicas: +- Storage: "gitaly-1" + Assigned: true + Generation: 1, fully up to date + Healthy: true + Valid Primary: true +- Storage: "gitaly-2" + Assigned: true + Generation: 0, behind by 1 changes + Healthy: true + Valid Primary: false +- Storage: "gitaly-3" + Assigned: true + Generation: replica not yet created + Healthy: false + Valid Primary: false +``` + +#### Available metadata + +The metadata retrieved by `praefect metadata` includes the fields in the following tables. + +| Field | Description | +|:------------------|:-------------------------------------------------------------------------------------------------------------------| +| `Repository ID` | Permanent unique ID assigned to the repository by Praefect. Different to the ID GitLab uses for repositories. | +| `Virtual Storage` | Name of the virtual storage the repository is stored in. | +| `Relative Path` | Repository's path in the virtual storage. | +| `Replica Path` | Where on the Gitaly node's disk the repository's replicas are stored. | +| `Primary` | Current primary of the repository. | +| `Generation` | Used by Praefect to track repository changes. Each write in the repository increments the repository's generation. | +| `Replicas` | A list of replicas that exist or are expected to exist. | + +For each replica, the following metadata is available: + +| `Replicas` Field | Description | +|:-----------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `Storage` | Name of the Gitaly storage that contains the replica. | +| `Assigned` | Indicates whether the replica is expected to exist in the storage. Can be `false` if a Gitaly node is removed from the cluster or if the storage contains an extra copy after the repository's replication factor was decreased. | +| `Generation` | Latest confirmed generation of the replica. It indicates:<br><br>- The replica is fully up to date if the generation matches the repository's generation.<br>- The replica is outdated if the replica's generation is less than the repository's generation.<br>- `replica not yet created` if the replica does not yet exist at all on the storage. | +| `Healthy` | Indicates whether the Gitaly node that is hosting this replica is considered healthy by the consensus of Praefect nodes. | +| `Valid Primary` | Indicates whether the replica is fit to serve as the primary node. If the repository's primary is not a valid primary, a failover occurs on the next write to the repository if there is another replica that is a valid primary. A replica is a valid primary if:<br><br>- It is stored on a healthy Gitaly node.<br>- It is fully up to date.<br>- It is not targeted by a pending deletion job from decreasing replication factor.<br>- It is assigned. | ### Check that repositories are in sync @@ -371,7 +545,7 @@ Is [some cases](index.md#known-issues) the Praefect database can get out of sync a given repository is fully synced on all nodes, run the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums) that checksums the repository on all Gitaly nodes. -The [Praefect dataloss](praefect.md#check-for-data-loss) command only checks the state of the repo in the Praefect database, and cannot +The [Praefect dataloss](recovery.md#check-for-data-loss) command only checks the state of the repo in the Praefect database, and cannot be relied to detect sync problems in this scenario. ### Relation does not exist errors @@ -409,3 +583,21 @@ This indicates that the virtual storage name used in the [`git_data_dirs` setting](praefect.md#gitaly) for GitLab. Resolve this by matching the virtual storage names used in Praefect and GitLab configuration. + +### Gitaly Cluster performance issues on cloud platforms + +Praefect does not require a lot of CPU or memory, and can run on small virtual machines. +Cloud services may place other limits on the resources that small VMs can use, such as +disk IO and network traffic. + +Praefect nodes generate a lot of network traffic. The following symptoms can be observed if their network bandwidth has +been throttled by the cloud service: + +- Poor performance of Git operations. +- High network latency. +- High memory use by Praefect. + +Possible solutions: + +- Provision larger VMs to gain access to larger network traffic allowances. +- Use your cloud service's monitoring and logging to check that the Praefect nodes are not exhausting their traffic allowances. diff --git a/doc/administration/img/db_load_balancing_postgres_stats.png b/doc/administration/img/db_load_balancing_postgres_stats.png Binary files differdeleted file mode 100644 index 8b311616e7b..00000000000 --- a/doc/administration/img/db_load_balancing_postgres_stats.png +++ /dev/null diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 6b390cfc77a..3f54f5dd576 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -10,7 +10,7 @@ GitLab has several features based on receiving incoming email messages: - [Reply by Email](reply_by_email.md): allow GitLab users to comment on issues and merge requests by replying to notification email. -- [New issue by email](../user/project/issues/managing_issues.md#new-issue-via-email): +- [New issue by email](../user/project/issues/managing_issues.md#by-sending-an-email): allow GitLab users to create a new issue by sending an email to a user-specific email address. - [New merge request by email](../user/project/merge_requests/creating_merge_requests.md#by-sending-an-email): @@ -66,6 +66,24 @@ This solution is relatively simple to set up: you just need to create an email address dedicated to receive your users' replies to GitLab notifications. However, this method only supports replies, and not the other features of [incoming email](#incoming-email). +## Accepted headers + +Email is processed correctly when a configured email address is present in one of the following headers: + +- `To` +- `Delivered-To` +- `Envelope-To` or `X-Envelope-To` + +In GitLab 14.6 and later, [Service Desk](../user/project/service_desk.md) +also checks these additional headers. + +Usually, the "To" field contains the email address of the primary receiver. +However, it might not include the configured GitLab email address if: + +- The address is in the "CC" field. +- The address was included when using "Reply all". +- The email was forwarded. + ## Set it up If you want to use Gmail / Google Apps for incoming email, make sure you have diff --git a/doc/administration/index.md b/doc/administration/index.md index 53a3c305aab..d78c9d80b5f 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -31,8 +31,6 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Installing GitLab - [Install](../install/index.md): Requirements, directory structures, and installation methods. - - [Database load balancing](database_load_balancing.md): Distribute database queries among multiple database servers. - - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only). - [Reference architectures](reference_architectures/index.md): Add additional resources to support more users. - [Installing GitLab on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab on Amazon AWS. - [Geo](geo/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. @@ -79,6 +77,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Enabling and disabling features flags](feature_flags.md): how to enable and disable GitLab features deployed behind feature flags. - [Application settings cache expiry interval](application_settings_cache.md) +- [Database Load Balancing](postgresql/database_load_balancing.md): Distribute database queries among multiple database servers. +- [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only). #### Customizing GitLab appearance @@ -133,7 +133,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - Instances. - [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. - [Incoming email](incoming_email.md): Configure incoming emails to allow - users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/managing_issues.md#new-issue-via-email) and + users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/managing_issues.md#by-sending-an-email) and [merge requests by email](../user/project/merge_requests/creating_merge_requests.md#by-sending-an-email), and to enable [Service Desk](../user/project/service_desk.md). - [Postfix for incoming email](reply_by_email_postfix_setup.md): Set up a basic Postfix mail server with IMAP authentication on Ubuntu for incoming diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 0b470146b14..bfe59d5277b 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -88,12 +88,8 @@ requests per user. For more information, read ### Files API -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68561) in GitLab 14.3. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the `files_api_throttling` flag](../administration/feature_flags.md). On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. -The feature is not ready for production use. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68561) in GitLab 14.3 [with a flag](../administration/feature_flags.md) named `files_api_throttling`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75918) in GitLab 14.6. [Feature flag `files_api_throttling`](https://gitlab.com/gitlab-org/gitlab/-/issues/338903) removed. This setting limits the request rate on the Packages API per user or IP address. For more information, read [Files API rate limits](../user/admin_area/settings/files_api_rate_limits.md). @@ -257,7 +253,7 @@ Set the limit to `0` to disable it. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/237891) in GitLab 13.7. The [minimum wait time between pull refreshes](../user/project/repository/mirror/index.md) -defaults to 300 seconds (5 minutes). For example, by default a pull refresh will only run once in a given 300 second period regardless of how many times you try to trigger it. +defaults to 300 seconds (5 minutes). For example, a pull refresh only runs once in a given 300 second period, regardless of how many times you trigger it. This setting applies in the context of pull refreshes invoked via the [projects API](../api/projects.md#start-the-pull-mirroring-process-for-a-project), or when forcing an update by selecting the **Update now** (**{retry}**) button within **Settings > Repository > Mirroring repositories**. This setting has no effect on the automatic 30 minute interval schedule used by Sidekiq for [pull mirroring](../user/project/repository/mirror/pull.md). @@ -400,7 +396,7 @@ limit is checked every time a new trigger is created. If a new trigger would cause the total number of pipeline triggers to exceed the limit, the trigger is considered invalid. -Set the limit to `0` to disable it. Defaults to `0` on self-managed instances. +Set the limit to `0` to disable it. Defaults to `150` on self-managed instances. To set this limit to `100` on a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -551,8 +547,8 @@ Plan.default.actual_limits.update!(pages_file_entries: 100) > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321368) in GitLab 13.12. Disabled by default. > - Enabled on GitLab.com in GitLab 14.3. > - Enabled on self-managed in GitLab 14.4. -> - Feature flag `ci_runner_limits` removed in GitLab 14.4. You can still use `ci_runner_limits_override` - to remove limits for a given scope. +> - Feature flag `ci_runner_limits` removed in GitLab 14.4. +> - Feature flag `ci_runner_limits_override` removed in GitLab 14.6. The total number of registered runners is limited at the group and project levels. Each time a new runner is registered, GitLab checks these limits against runners that have been active in the last 3 months. @@ -739,7 +735,7 @@ See [Environment Dashboard](../ci/environments/environments_dashboard.md#adding- [Deploy boards](../user/project/deploy_boards.md) load information from Kubernetes about Pods and Deployments. However, data over 10 MB for a certain environment read from -Kubernetes won't be shown. +Kubernetes aren't shown. ## Merge requests @@ -762,7 +758,7 @@ prevent any more changes from rendering. For more information about these limits ### Merge request reports size limit -Reports that go over the 20 MB limit won't be loaded. Affected reports: +Reports that go over the 20 MB limit aren't loaded. Affected reports: - [Merge request security reports](../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports) - [CI/CD parameter `artifacts:expose_as`](../ci/yaml/index.md#artifactsexpose_as) @@ -826,7 +822,7 @@ See the [Design Management Limitations](../user/project/issues/design_management > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31009) in GitLab 12.4. Total number of changes (branches or tags) in a single push. If changes are more -than the specified limit, hooks won't be executed. +than the specified limit, hooks are not executed. More information can be found in these docs: @@ -848,16 +844,21 @@ More information can be found in the [Push event activities limit and bulk push > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218017) in GitLab 13.4. -On GitLab.com, the maximum file size for a package that's uploaded to the [GitLab Package Registry](../user/packages/package_registry/index.md) varies by format: +The default maximum file size for a package that's uploaded to the [GitLab Package Registry](../user/packages/package_registry/index.md) varies by format: -- Conan: 5 GB +- Conan: 3 GB - Generic: 5 GB -- Maven: 5 GB -- npm: 5 GB -- NuGet: 5 GB -- PyPI: 5 GB +- Helm: 5 MB +- Maven: 3 GB +- npm: 500 MB +- NuGet: 500 MB +- PyPI: 3 GB +- Terraform: 1 GB -To set this limit for a self-managed installation, run the following in the +The [maximum file sizes on GitLab.com](../user/gitlab_com/index.md#package-registry-limits) +might be different. + +To set these limits for a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby @@ -881,6 +882,9 @@ Plan.default.actual_limits.update!(pypi_max_file_size: 100.megabytes) # For Debian Packages Plan.default.actual_limits.update!(debian_max_file_size: 100.megabytes) +# For Helm Charts +Plan.default.actual_limits.update!(helm_max_file_size: 100.megabytes) + # For Generic Packages Plan.default.actual_limits.update!(generic_packages_max_file_size: 100.megabytes) ``` diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index 62897651166..872cdb239bd 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -12,7 +12,7 @@ If you run a medium-sized self-managed instance (50+ users) of a free version of [either Community Edition or unlicensed Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/), you qualify for a free Instance Review. -1. Sign in as a user with Administrator [role](../user/permissions.md). +1. Sign in as an administrator. 1. In the top menu, click your user icon, and select **Get a free instance review**: diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 07b9ba87d8e..f570c9b559f 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -17,7 +17,7 @@ GitLab uses these credentials to provide access to [web terminals](../../ci/environments/index.md#web-terminals-deprecated) for environments. NOTE: -Only project maintainers and owners can access web terminals. +Only users with at least the [Maintainer role](../../user/permissions.md) for the project access web terminals. ## How it works diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 46a9ee11679..64b5ddbd165 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -2,20 +2,18 @@ stage: Verify group: Testing info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Jobs artifacts administration **(FREE SELF)** This is the administration documentation. For the user guide see [pipelines/job_artifacts](../ci/pipelines/job_artifacts.md). -Artifacts is a list of files and directories which are attached to a job after it -finishes. This feature is enabled by default in all GitLab installations. Keep reading -if you want to know how to disable it. +An artifact is a list of files and directories attached to a job after it +finishes. This feature is enabled by default in all GitLab installations. ## Disabling job artifacts -To disable artifacts site-wide, follow the steps below. +To disable artifacts site-wide: **In Omnibus installations:** @@ -41,7 +39,7 @@ To disable artifacts site-wide, follow the steps below. ## Storing job artifacts GitLab Runner can upload an archive containing the job artifacts to GitLab. By default, -this is done when the job succeeds, but can also be done on failure, or always, via the +this is done when the job succeeds, but can also be done on failure, or always, with the [`artifacts:when`](../ci/yaml/index.md#artifactswhen) parameter. Most artifacts are compressed by GitLab Runner before being sent to the coordinator. The exception to this is @@ -84,8 +82,6 @@ _The artifacts are stored by default in ### Using object storage -> Introduced in GitLab 11.0: Support for `direct_upload` to S3. - If you don't want to use the local disk where GitLab is installed to store the artifacts, you can use an object storage like AWS S3 instead. This configuration relies on valid AWS credentials to be configured already. @@ -108,7 +104,9 @@ In GitLab 13.2 and later, we recommend using the [consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). This section describes the earlier configuration format. -For source installations the following settings are nested under `artifacts:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `artifacts_object_store_`. +For source installations the following settings are nested under `artifacts:` +and then `object_store:`. On Omnibus GitLab installs they are prefixed by +`artifacts_object_store_`. | Setting | Default | Description | |---------------------|---------|-------------| @@ -143,8 +141,9 @@ _The artifacts are stored by default in } ``` - NOTE: For GitLab 9.4+, if you're using AWS IAM profiles, be sure to omit the - AWS access key and secret access key/value pairs. For example: + NOTE: + If you're using AWS IAM profiles, omit the AWS access key and secret access + key/value pairs. For example: ```ruby gitlab_rails['artifacts_object_store_connection'] = { @@ -155,37 +154,7 @@ _The artifacts are stored by default in ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local artifacts to the object storage: - - ```shell - gitlab-rake gitlab:artifacts:migrate - ``` - -1. Optional: Verify all files migrated properly. - From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) - (`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: - - ```shell - gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts; - - total | filesystem | objectstg - ------+------------+----------- - 2409 | 0 | 2409 - ``` - - Verify no files on disk in `artifacts` folder: - - ```shell - sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp/cache | wc -l - ``` - - In some cases, you may need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files) - to clean up orphaned artifacts. - -WARNING: -JUnit test report artifact (`junit.xml.gz`) migration -[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991) -by the `gitlab:artifacts:migrate` script. +1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). **In installations from source:** @@ -209,36 +178,7 @@ _The artifacts are stored by default in ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -1. Migrate any existing local artifacts to the object storage: - - ```shell - sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production - ``` - -1. Optional: Verify all files migrated properly. - From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: - - ```shell - gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts; - - total | filesystem | objectstg - ------+------------+----------- - 2409 | 0 | 2409 - ``` - - Verify no files on disk in `artifacts` folder: - - ```shell - sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp/cache | wc -l - ``` - - In some cases, you may need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files) - to clean up orphaned artifacts. - -WARNING: -JUnit test report artifact (`junit.xml.gz`) migration -[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991) -by the `gitlab:artifacts:migrate` script. +1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). ### OpenStack example @@ -268,11 +208,7 @@ _The uploads are stored by default in ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local artifacts to the object storage: - - ```shell - gitlab-rake gitlab:artifacts:migrate - ``` +1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). --- @@ -303,11 +239,55 @@ _The uploads are stored by default in ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -1. Migrate any existing local artifacts to the object storage: +1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). - ```shell - sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production - ``` +### Migrating to object storage + +After [configuring the object storage](#using-object-storage), use the following task to +migrate existing job artifacts from the local storage to the remote storage. +The processing is done in a background worker and requires **no downtime**. + +**In Omnibus installations:** + +```shell +gitlab-rake gitlab:artifacts:migrate +``` + +**In installations from source:** + +```shell +sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production +``` + +You can optionally track progress and verify that all job artifacts migrated successfully using the +[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): + +- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. + +Verify `objectstg` below (where `store=2`) has count of all job artifacts: + +```shell +gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts; + +total | filesystem | objectstg +------+------------+----------- + 19 | 0 | 19 +``` + +Verify that there are no files on disk in the `artifacts` folder: + +```shell +sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp | wc -l +``` + +In some cases, you need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files) +to clean up orphaned artifacts. + +WARNING: +JUnit test report artifact (`junit.xml.gz`) migration +[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991) +by the `gitlab:artifacts:migrate` Rake task. ### Migrating from object storage to local storage @@ -503,13 +483,13 @@ If you need to manually remove job artifacts associated with multiple jobs while - `3.months.ago` - `1.year.ago` - `erase_erasable_artifacts!` is a synchronous method, and upon execution, the artifacts are removed immediately. - They are not scheduled via some background queue. + `erase_erasable_artifacts!` is a synchronous method, and upon execution the artifacts are immediately removed; + they are not scheduled by a background queue. #### Delete job artifacts and logs from jobs completed before a specific date WARNING: -These commands remove data permanently from the database and from disk. We +These commands remove data permanently from both the database and from disk. We highly recommend running them only under the guidance of a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. @@ -517,7 +497,7 @@ restored, just in case. If you need to manually remove **all** job artifacts associated with multiple jobs, **including job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): -1. Select jobs to be deleted: +1. Select the jobs to be deleted: To select jobs with artifacts for a single project: @@ -538,7 +518,7 @@ If you need to manually remove **all** job artifacts associated with multiple jo admin_user = User.find_by(username: 'username') ``` -1. Erase job artifacts and logs older than a specific date: +1. Erase the job artifacts and logs older than a specific date: ```ruby builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) @@ -563,34 +543,34 @@ If you need to manually remove **all** job artifacts associated with multiple jo ### Error `Downloading artifacts from coordinator... not found` -When a job tries to download artifacts from an earlier job, you might receive an error similar to: +When a job attempts to download artifacts from an earlier job, you might receive an error message similar to: ```plaintext Downloading artifacts from coordinator... not found id=12345678 responseStatus=404 Not Found ``` -This might be caused by a `gitlab.rb` file with the following configuration: +This can be caused by a `gitlab.rb` file with the following configuration: ```ruby gitlab_rails['artifacts_object_store_background_upload'] = false gitlab_rails['artifacts_object_store_direct_upload'] = true ``` -To prevent this, comment out or remove those lines, or switch to their [default values](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template), +To prevent this, comment out or remove those lines, or switch to their [default values](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template), and then run `sudo gitlab-ctl reconfigure`. ### Job artifact upload fails with error 500 If you are using object storage for artifacts and a job artifact fails to upload, -you can check: +review: -- The job log for an error similar to: +- The job log for an error message similar to: ```plaintext WARNING: Uploading artifacts as "archive" to coordinator... failed id=12345 responseStatus=500 Internal Server Error status=500 token=abcd1234 ``` -- The [workhorse log](logs.md#workhorse-logs) for an error similar to: +- The [workhorse log](logs.md#workhorse-logs) for an error message similar to: ```json {"error":"MissingRegion: could not find region configuration","level":"error","msg":"error uploading S3 session","time":"2021-03-16T22:10:55-04:00"} diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index d2f220e3795..3fe6a94ef13 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -56,7 +56,7 @@ In `config/gitlab.yml`: ## Storing LFS objects in remote object storage You can store LFS objects in remote object storage. This allows you -to offload reads and writes to the local disk, and free up disk space significantly. +to reduce reads and writes to the local disk, and free up disk space significantly. GitLab is tightly integrated with `Fog`, so you can refer to its [documentation](http://fog.io/about/provider_documentation.html) to check which storage services can be integrated with GitLab. You can also use external object storage in a private local network. For example, @@ -98,32 +98,6 @@ See [the available connection settings for different providers](../object_storag Here is a configuration example with S3. -### Manual uploading to an object storage - -There are two ways to manually do the same thing as automatic uploading (described above). - -**Option 1: Rake task** - -```shell -gitlab-rake gitlab:lfs:migrate -``` - -**Option 2: Rails console** - -Log into the Rails console: - -```shell -sudo gitlab-rails console -``` - -Upload LFS files manually - -```ruby -LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object| - lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists? -end -``` - ### S3 for Omnibus installations On Omnibus GitLab installations, the settings are prefixed by `lfs_object_store_`: @@ -146,32 +120,10 @@ On Omnibus GitLab installations, the settings are prefixed by `lfs_object_store_ ``` 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local LFS objects to the object storage: - - ```shell - gitlab-rake gitlab:lfs:migrate - ``` - - This migrates existing LFS objects to object storage. New LFS objects +1. [Migrate any existing local LFS objects to the object storage](#migrating-to-object-storage). + New LFS objects are forwarded to object storage unless `gitlab_rails['lfs_object_store_background_upload']` and `gitlab_rails['lfs_object_store_direct_upload']` is set to `false`. -1. (Optional) Verify all files migrated properly. - From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) - (`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: - - ```shell - gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects; - - total | filesystem | objectstg - ------+------------+----------- - 2409 | 0 | 2409 - ``` - - Verify no files on disk in `artifacts` folder: - - ```shell - sudo find /var/opt/gitlab/gitlab-rails/shared/lfs-objects -type f | grep -v tmp/cache | wc -l - ``` ### S3 for installations from source @@ -199,31 +151,68 @@ For source installations the settings are nested under `lfs:` and then ``` 1. Save the file, and then [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. -1. Migrate any existing local LFS objects to the object storage: +1. [Migrate any existing local LFS objects to the object storage](#migrating-to-object-storage). + New LFS objects + are forwarded to object storage unless + `background_upload` and `direct_upload` is set to `false`. - ```shell - sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production - ``` +### Migrating to object storage - This migrates existing LFS objects to object storage. New LFS objects - are forwarded to object storage unless `background_upload` and `direct_upload` is set to - `false`. -1. (Optional) Verify all files migrated properly. - From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: +**Option 1: Rake task** - ```shell - gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects; +After [configuring the object storage](#storing-lfs-objects-in-remote-object-storage), use the following task to +migrate existing LFS objects from the local storage to the remote storage. +The processing is done in a background worker and requires **no downtime**. - total | filesystem | objectstg - ------+------------+----------- - 2409 | 0 | 2409 - ``` +For Omnibus GitLab: + +```shell +sudo gitlab-rake "gitlab:lfs:migrate" +``` - Verify no files on disk in `artifacts` folder: +For installations from source: - ```shell - sudo find /var/opt/gitlab/gitlab-rails/shared/lfs-objects -type f | grep -v tmp/cache | wc -l - ``` +```shell +RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:lfs:migrate +``` + +You can optionally track progress and verify that all packages migrated successfully using the +[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): + +- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. + +Verify `objectstg` below (where `store=2`) has count of all LFS objects: + +```shell +gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects; + +total | filesystem | objectstg +------+------------+----------- + 2409 | 0 | 2409 +``` + +Verify that there are no files on disk in the `objects` folder: + +```shell +sudo find /var/opt/gitlab/gitlab-rails/shared/lfs-objects -type f | grep -v tmp | wc -l +``` + +**Option 2: Rails console** + +Log into the Rails console: + +```shell +sudo gitlab-rails console +``` + +Upload LFS files manually + +```ruby +LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object| + lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists? +end +``` ### Migrating back to local storage diff --git a/doc/administration/logs.md b/doc/administration/logs.md index bf74a96a627..263fe699529 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -245,8 +245,6 @@ The request was processed by `Projects::TreeController`. ## `api_json.log` -> Introduced in GitLab 10.0. - Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/api_json.log` @@ -296,7 +294,7 @@ Depending on your installation method, this file is located at: - Installations from source: `/home/git/gitlab/log/application.log` It helps you discover events happening in your instance such as user creation -and project removal. For example: +and project deletion. For example: ```plaintext October 06, 2014 11:56: User "Administrator" (admin@example.com) was created @@ -367,8 +365,6 @@ like this example: ## `kubernetes.log` -> Introduced in GitLab 11.6. - Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/kubernetes.log` @@ -696,8 +692,6 @@ on a project. ## `importer.log` -> Introduced in GitLab 11.3. - Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/importer.log` @@ -830,7 +824,7 @@ are generated in a location based on your installation method: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15442) in GitLab 12.3. -Contains details of GitLab [Database Load Balancing](database_load_balancing.md). +Contains details of GitLab [Database Load Balancing](postgresql/database_load_balancing.md). Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/database_load_balancing.log` @@ -915,8 +909,6 @@ For example: ## `geo.log` **(PREMIUM SELF)** -> Introduced in 9.5. - Geo stores structured log messages in a `geo.log` file. For Omnibus GitLab installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. @@ -934,8 +926,6 @@ This message shows that Geo detected that a repository update was needed for pro ## `update_mirror_service_json.log` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/commit/7f637e2af7006dc2b1b2649d9affc0b86cfb33c4) in GitLab 11.12. - Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/update_mirror_service_json.log` @@ -1057,9 +1047,9 @@ For Omnibus GitLab installations, GitLab Monitor logs are in `/var/log/gitlab/gi For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/gitlab-exporter/`. -## GitLab Kubernetes Agent Server +## GitLab Agent Server -For Omnibus GitLab installations, GitLab Kubernetes Agent Server logs are +For Omnibus GitLab installations, GitLab Agent Server logs are in `/var/log/gitlab/gitlab-kas/`. ## Praefect Logs diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 90d0b65dbe5..1cf4e5a25ba 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -20,9 +20,8 @@ project called **Monitoring** is created: The project is created specifically for visualizing and configuring the monitoring of your GitLab instance. -When the project and group are created, all administrators are added as maintainers. As an -administrator, you can add new members to the group to give them the -[Maintainer role](../../../user/permissions.md) for the project. +When the project and group are created, all administrators are given the [Maintainer role](../../../user/permissions.md). +As an administrator, you can add new members to the group to give them the Maintainer role for the project. This project can be used to: diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index d798feb71a9..14a560223f9 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -26,11 +26,10 @@ From left to right, the performance bar displays: details for each query: - **In a transaction**: shows up below the query if it was executed in the context of a transaction - - **Role**: shows up when [database load - balancing](../../database_load_balancing.md) is enabled. It shows - which server role was used for the query. "Primary" means that the query - was sent to the read/write primary server. "Replica" means it was sent - to a read-only replica. + - **Role**: shows up when [Database Load Balancing](../../postgresql/database_load_balancing.md) + is enabled. It shows which server role was used for the query. + "Primary" means that the query was sent to the read/write primary server. + "Replica" means it was sent to a read-only replica. - **Config name**: shows up only when the `GITLAB_MULTIPLE_DATABASE_METRICS` environment variable is set. This is used to distinguish between different databases configured for different diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 2f9c1e3bc9c..4a504b07a1b 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -187,16 +187,25 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_repositories` | Gauge | 10.2 | Total number of repositories available on primary | `url` | | `geo_repositories_synced` | Gauge | 10.2 | Number of repositories synced on secondary | `url` | | `geo_repositories_failed` | Gauge | 10.2 | Number of repositories failed to sync on secondary | `url` | -| `geo_lfs_objects` | Gauge | 10.2 | Total number of LFS objects available on primary | `url` | -| `geo_lfs_objects_synced` | Gauge | 10.2 | Number of LFS objects synced on secondary | `url` | -| `geo_lfs_objects_failed` | Gauge | 10.2 | Number of LFS objects failed to sync on secondary | `url` | +| `geo_lfs_objects` | Gauge | 10.2 | Number of LFS objects on primary | `url` | +| `geo_lfs_objects_checksummed` | Gauge | 14.6 | Number of LFS objects checksummed successfully on primary | `url` | +| `geo_lfs_objects_checksum_failed` | Gauge | 14.6 | Number of LFS objects failed to calculate the checksum on primary | `url` | +| `geo_lfs_objects_checksum_total` | Gauge | 14.6 | Number of LFS objects tried to checksum on primary | `url` | +| `geo_lfs_objects_synced` | Gauge | 10.2 | Number of syncable LFS objects synced on secondary | `url` | +| `geo_lfs_objects_failed` | Gauge | 10.2 | Number of syncable LFS objects failed to sync on secondary | `url` | +| `geo_lfs_objects_registry` | Gauge | 14.6 | Number of LFS objects in the registry | `url` | +| `geo_lfs_objects_verified` | Gauge | 14.6 | Number of LFS objects verified on secondary | `url` | +| `geo_lfs_objects_verification_failed` | Gauge | 14.6 | Number of LFS objects' verifications failed on secondary | `url` | +| `geo_lfs_objects_verification_total` | Gauge | 14.6 | Number of LFS objects' verifications tried on secondary | `url` |LFS objects failed to sync on secondary | `url` | +| `geo_attachments` | Gauge | 10.2 | Total number of file attachments available on primary | `url` | +| `geo_attachments_synced` | Gauge | 10.2 | Number of attachments synced on secondary | `url` | +| `geo_attachments_failed` | Gauge | 10.2 | Number of attachments failed to sync on secondary | `url` | | `geo_last_event_id` | Gauge | 10.2 | Database ID of the latest event log entry on the primary | `url` | | `geo_last_event_timestamp` | Gauge | 10.2 | UNIX timestamp of the latest event log entry on the primary | `url` | | `geo_cursor_last_event_id` | Gauge | 10.2 | Last database ID of the event log processed by the secondary | `url` | | `geo_cursor_last_event_timestamp` | Gauge | 10.2 | Last UNIX timestamp of the event log processed by the secondary | `url` | | `geo_status_failed_total` | Counter | 10.2 | Number of times retrieving the status from the Geo Node failed | `url` | | `geo_last_successful_status_check_timestamp` | Gauge | 10.2 | Last timestamp when the status was successfully updated | `url` | -| `geo_lfs_objects_synced_missing_on_primary` | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | `url` | | `geo_job_artifacts_synced_missing_on_primary` | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | `url` | | `geo_repositories_checksummed` | Gauge | 10.7 | Number of repositories checksummed on primary | `url` | | `geo_repositories_checksum_failed` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | `url` | @@ -253,15 +262,15 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_group_wiki_repositories_failed` | Gauge | 13.10 | Number of syncable group wikis failed on secondary | `url` | | `geo_group_wiki_repositories_registry` | Gauge | 13.10 | Number of syncable group wikis in the registry | `url` | | `geo_pages_deployments` | Gauge | 14.3 | Number of pages deployments on primary | `url` | -| `geo_pages_deployments_checksum_total` | Gauge | 14.3 | Number of pages deployments tried to checksum on primary | `url` | -| `geo_pages_deployments_checksummed` | Gauge | 14.3 | Number of pages deployments successfully checksummed on primary | `url` | -| `geo_pages_deployments_checksum_failed` | Gauge | 14.3 | Number of pages deployments failed to calculate the checksum on primary | `url` | +| `geo_pages_deployments_checksum_total` | Gauge | 14.6 | Number of pages deployments tried to checksum on primary | `url` | +| `geo_pages_deployments_checksummed` | Gauge | 14.6 | Number of pages deployments successfully checksummed on primary | `url` | +| `geo_pages_deployments_checksum_failed` | Gauge | 14.6 | Number of pages deployments failed to calculate the checksum on primary | `url` | | `geo_pages_deployments_synced` | Gauge | 14.3 | Number of syncable pages deployments synced on secondary | `url` | | `geo_pages_deployments_failed` | Gauge | 14.3 | Number of syncable pages deployments failed to sync on secondary | `url` | | `geo_pages_deployments_registry` | Gauge | 14.3 | Number of pages deployments in the registry | `url` | -| `geo_pages_deployments_verification_total` | Gauge | 14.3 | Number of pages deployments verifications tried on secondary | `url` | -| `geo_pages_deployments_verified` | Gauge | 14.3 | Number of pages deployments verified on secondary | `url` | -| `geo_pages_deployments_verification_failed` | Gauge | 14.3 | Number of pages deployments verifications failed on secondary | `url` | +| `geo_pages_deployments_verification_total` | Gauge | 14.6 | Number of pages deployments verifications tried on secondary | `url` | +| `geo_pages_deployments_verified` | Gauge | 14.6 | Number of pages deployments verified on secondary | `url` | +| `geo_pages_deployments_verification_failed` | Gauge | 14.6 | Number of pages deployments verifications failed on secondary | `url` | | `limited_capacity_worker_running_jobs` | Gauge | 13.5 | Number of running jobs | `worker` | | `limited_capacity_worker_max_running_jobs` | Gauge | 13.5 | Maximum number of running jobs | `worker` | | `limited_capacity_worker_remaining_work_count` | Gauge | 13.5 | Number of jobs waiting to be enqueued | `worker` | @@ -272,6 +281,12 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_uploads_synced` | Gauge | 14.1 | Number of uploads synced on secondary | `url` | | `geo_uploads_failed` | Gauge | 14.1 | Number of syncable uploads failed to sync on secondary | `url` | | `geo_uploads_registry` | Gauge | 14.1 | Number of uploads in the registry | `url` | +| `geo_uploads_checksum_total` | Gauge | 14.6 | Number of uploads tried to checksum on primary | `url` | +| `geo_uploads_checksummed` | Gauge | 14.6 | Number of uploads successfully checksummed on primary | `url` | +| `geo_uploads_checksum_failed` | Gauge | 14.6 | Number of uploads failed to calculate the checksum on primary | `url` | +| `geo_uploads_verification_total` | Gauge | 14.6 | Number of uploads verifications tried on secondary | `url` | +| `geo_uploads_verified` | Gauge | 14.6 | Number of uploads verified on secondary | `url` | +| `geo_uploads_verification_failed` | Gauge | 14.6 | Number of uploads verifications failed on secondary | `url` | | `gitlab_sli:rails_request_apdex:total` | Counter | 14.4 | The number of request-apdex measurements, [more information the development documentation](../../../development/application_slis/rails_request_apdex.md) | `endpoint_id`, `feature_category`, `request_urgency` | | `gitlab_sli:rails_request_apdex:success_total` | Counter | 14.4 | The number of succesful requests that met the target duration for their urgency. Devide by `gitlab_sli:rails_requests_apdex:total` to get a success ratio | `endpoint_id`, `feature_category`, `request_urgency` | diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index e86ca596955..3268c0fc14c 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -259,9 +259,10 @@ To use an external Prometheus server: - 1.1.1.1:9229 - job_name: gitlab-rails metrics_path: "/-/metrics" + scheme: https static_configs: - targets: - - 1.1.1.1:8080 + - 1.1.1.1 - job_name: gitlab-sidekiq static_configs: - targets: @@ -287,6 +288,11 @@ To use an external Prometheus server: - 1.1.1.1:9236 ``` + WARNING: + The `gitlab-rails` job in the snippet assumes that GitLab is reachable through HTTPS. If your + deployment doesn't use HTTPS, the job configuration is adapted to use the `http` scheme and port + 80. + 1. Reload the Prometheus server. ## Viewing performance metrics diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 2a2e9f05312..a0170e6c4ef 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -20,31 +20,31 @@ file system performance, see ## Gitaly and NFS deprecation -Starting with GitLab version 14.0, support for NFS to store Git repository data will be deprecated. Technical customer support and engineering support will be available for the 14.x releases. Engineering will fix bugs and security vulnerabilities consistent with our [release and maintenance policy](../policy/maintenance.md#security-releases). +Starting with GitLab version 14.0, support for NFS to store Git repository data is deprecated. Technical customer support and engineering support is available for the 14.x releases. Engineering is fixing bugs and security vulnerabilities consistent with our [release and maintenance policy](../policy/maintenance.md#security-releases). -At the end of the 14.12 milestone (tenatively June 22nd, 2022) technical and engineering support for using NFS to store Git repository data will be officially at end-of-life. There will be no product changes or troubleshooting provided via Engineering, Security or Paid Support channels. +At the end of the 14.12 milestone (tentatively June 22nd, 2022) technical and engineering support for using NFS to store Git repository data will be officially at end-of-life. There will be no product changes or troubleshooting provided via Engineering, Security or Paid Support channels. For those customers still running earlier versions of GitLab, [our support eligibility and maintenance policy applies](https://about.gitlab.com/support/statement-of-support.html#version-support). -For the 14.x releases, we will continue to help with Git related tickets from customers running one or more Gitaly servers with its data stored on NFS. Examples may include: +For the 14.x releases, we continue to help with Git related tickets from customers running one or more Gitaly servers with its data stored on NFS. Examples may include: - Performance issues or timeouts accessing Git data - Commits or branches vanish - GitLab intermittently returns the wrong Git data (such as reporting that a repository has no branches) -Assistance will be limited to activities like: +Assistance is limited to activities like: - Verifying developers' workflow uses features like protected branches - Reviewing GitLab event data from the database to advise if it looks like a force push over-wrote branches - Verifying that NFS client mount options match our [documented recommendations](#mount-options) - Analyzing the GitLab Workhorse and Rails logs, and determining that `500` errors being seen in the environment are caused by slow responses from Gitaly -GitLab support will be unable to continue with the investigation if: +GitLab support is unable to continue with the investigation if: - The date of the request is on or after the release of GitLab version 15.0, and - Support Engineers and Management determine that all reasonable non-NFS root causes have been exhausted -If the issue is reproducible, or if it happens intermittently but regularly, GitLab Support will investigate providing the issue reproduces without the use of NFS. In order to reproduce without NFS, the affected repositories should be migrated to a different Gitaly shard, such as Gitaly cluster or a standalone Gitaly VM, backed with block storage. +If the issue is reproducible, or if it happens intermittently but regularly, GitLab Support can investigate providing the issue reproduces without the use of NFS. In order to reproduce without NFS, the affected repositories should be migrated to a different Gitaly shard, such as Gitaly cluster or a standalone Gitaly VM, backed with block storage. ### Why remove NFS for Git repository data @@ -438,7 +438,7 @@ the file system access GitLab requires. Workloads where many small files are wri a serialized manner, like `git`, are not well suited to cloud-based file systems. If you do choose to use these, avoid storing GitLab log files (for example, those in `/var/log/gitlab`) -there because this will also affect performance. We recommend that the log files be +there because this also affects performance. We recommend that the log files be stored on a local volume. For more details on the experience of using a cloud-based file systems with GitLab, @@ -447,12 +447,12 @@ see this [Commit Brooklyn 2019 video](https://youtu.be/K6OS8WodRBQ?t=313). ### Avoid using CephFS and GlusterFS GitLab strongly recommends against using CephFS and GlusterFS. -These distributed file systems are not well-suited for the GitLab input/output access patterns because Git uses many small files and access times and file locking times to propagate will make Git activity very slow. +These distributed file systems are not well-suited for the GitLab input/output access patterns because Git uses many small files and access times and file locking times to propagate makes Git activity very slow. ### Avoid using PostgreSQL with NFS GitLab strongly recommends against running your PostgreSQL database -across NFS. The GitLab support team will not be able to assist on performance issues related to +across NFS. The GitLab support team is not able to assist on performance issues related to this configuration. Additionally, this configuration is specifically warned against in the diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 8576b429213..c6490e365a5 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -281,6 +281,9 @@ The service account must have permission to access the bucket. Learn more in Google's [Cloud Storage authentication documentation](https://cloud.google.com/storage/docs/authentication). +NOTE: +Bucket encryption with the [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and will result in [ETag mismatch errors](#etag-mismatch). + ##### Google example (consolidated form) For Omnibus installations, this is an example of the `connection` setting: @@ -354,7 +357,7 @@ gitlab_rails['object_store']['connection'] = { 'provider' => 'AzureRM', 'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>', 'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>', - 'azure_storage_domain' => '<AZURE STORAGE DOMAIN>', + 'azure_storage_domain' => '<AZURE STORAGE DOMAIN>' } ``` @@ -682,6 +685,8 @@ With the consolidated object configuration and instance profile, Workhorse has S3 credentials so that it can compute the `Content-MD5` header. This eliminates the need to compare ETag headers returned from the S3 server. +Encrypting buckets with GCS' [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and will result in ETag mismatch errors. + ### Using Amazon instance profiles Instead of supplying AWS access and secret keys in object storage diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 02cb7ad0bca..1c9b98041dc 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -263,9 +263,9 @@ This sets the concurrency (number of threads) for the Sidekiq process. ## Modify the check interval -To modify the check interval for the additional Sidekiq processes: +To modify `sidekiq-cluster`'s health check interval for the additional Sidekiq processes: -1. Edit `/etc/gitlab/gitlab.rb` and add: +1. Edit `/etc/gitlab/gitlab.rb` and add (the value can be any integer number of seconds): ```ruby sidekiq['interval'] = 5 @@ -273,8 +273,6 @@ To modify the check interval for the additional Sidekiq processes: 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -This tells the additional processes how often to check for enqueued jobs. - ## Troubleshoot using the CLI WARNING: diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 9cf7ac18c81..84e6dca1f2b 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -42,7 +42,12 @@ To move repositories into a [Gitaly Cluster](../gitaly/index.md#gitaly-cluster) WARNING: Repositories can be **permanently deleted** by a call to `/projects/:project_id/repository_storage_moves` that attempts to move a project already stored in a Gitaly Cluster back into that cluster. -See [this issue for more details](https://gitlab.com/gitlab-org/gitaly/-/issues/3752). +See [this issue for more details](https://gitlab.com/gitlab-org/gitaly/-/issues/3752). This was fixed in +GitLab 14.3.0 and backported to +[14.2.4](https://about.gitlab.com/releases/2021/09/17/gitlab-14-2-4-released/), +[14.1.6](https://about.gitlab.com/releases/2021/09/27/gitlab-14-1-6-released/), +[14.0.11](https://about.gitlab.com/releases/2021/09/27/gitlab-14-0-11-released/), and +[13.12.12](https://about.gitlab.com/releases/2021/09/22/gitlab-13-12-12-released/). Each repository is made read-only for the duration of the move. The repository is not writable until the move has completed. diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index f1f02b606f5..c7df8249ae4 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -113,7 +113,7 @@ is used when Puma is enabled. NOTE: Unlike Unicorn, the `puma['worker_timeout']` setting does not set the maximum request duration. -To change the worker timeout: +To change the worker timeout to 600 seconds: 1. Edit `/etc/gitlab/gitlab.rb`: diff --git a/doc/administration/package_information/deprecated_os.md b/doc/administration/package_information/deprecated_os.md index 7234d68e4b2..1f6fe0fce5d 100644 --- a/doc/administration/package_information/deprecated_os.md +++ b/doc/administration/package_information/deprecated_os.md @@ -1,83 +1,9 @@ --- -stage: Enablement -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +redirect_to: 'supported_os.md' +remove_date: '2022-02-18' --- -# OS Versions that are no longer supported **(FREE SELF)** +This document was moved to [another location](supported_os.md). -GitLab provides omnibus packages for operating systems only until their -EOL (End-Of-Life). After the EOL date of the OS, GitLab will stop releasing -official packages. The list of deprecated operating systems and the final GitLab -release for them can be found below: - -| OS Version | End Of Life | Last supported GitLab version | -| --------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Raspbian Wheezy | [May 2015](https://downloads.raspberrypi.org/raspbian/images/raspbian-2015-05-07/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_8.17&dist=debian%2Fwheezy) 8.17 | -| OpenSUSE 13.2 | [January 2017](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-9.1&dist=opensuse%2F13.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-9.1&dist=opensuse%2F13.2) 9.1 | -| Ubuntu 12.04 | [April 2017](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_9.1&dist=ubuntu%2Fprecise) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_9.1&dist=ubuntu%2Fprecise) 9.1 | -| OpenSUSE 42.1 | [May 2017](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-9.3&dist=opensuse%2F42.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-9.3&dist=opensuse%2F42.1) 9.3 | -| OpenSUSE 42.2 | [January 2018](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-10.4&dist=opensuse%2F42.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-10.4&dist=opensuse%2F42.2) 10.4 | -| Debian Wheezy | [May 2018](https://www.debian.org/News/2018/20180601) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_11.6&dist=debian%2Fwheezy) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_11.6&dist=debian%2Fwheezy) 11.6 | -| Raspbian Jessie | [May 2017](https://downloads.raspberrypi.org/raspbian/images/raspbian-2017-07-05/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_11.7&dist=debian%2Fjessie) 11.7 | -| Ubuntu 14.04 | [April 2019](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_11.10&dist=ubuntu%2Ftrusty) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_11.10&dist=ubuntu%2Ftrusty) 11.10 | -| OpenSUSE 42.3 | [July 2019](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-12.1&dist=opensuse%2F42.3) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-12.1&dist=opensuse%2F42.3) 12.1 | -| OpenSUSE 15.0 | [December 2019](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-12.5&dist=opensuse%2F15.0) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-12.5&dist=opensuse%2F15.0) 12.5 | -| Raspbian Stretch | [June 2020](https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-04-09/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_13.2&dist=raspbian%2Fstretch) 13.3 | -| Debian Jessie | [June 2020](https://www.debian.org/News/2020/20200709) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.2&dist=debian%2Fjessie) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.2&dist=debian%2Fjessie) 13.3 | -| CentOS 6 | [November 2020](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=13.6&filter=all&filter=all&dist=el%2F6) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=13.6&filter=all&filter=all&dist=el%2F6) 13.6 | -| OpenSUSE 15.1 | [November 2020](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-13.12&dist=opensuse%2F15.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-13.12&dist=opensuse%2F15.2) 13.12 | -| Ubuntu 16.04 | [April 2021](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.12&dist=ubuntu%2Fxenial) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.12&dist=ubuntu%2Fxenial) 13.12 | - -NOTE: -An exception to this deprecation policy is when we are unable to provide -packages for the next version of the operating system. The most common reason -for this our package repository provider, Packagecloud, not supporting newer -versions and hence we can't upload packages to it. - -## Update GitLab package sources after upgrading the OS - -After upgrading the Operating System (OS) as per its own documentation, -it may be necessary to also update the GitLab package source URL -in your package manager configuration. -If your package manager reports that no further updates are available, -although [new versions have been released](https://about.gitlab.com/releases/categories/releases/), repeat the -"Add the GitLab package repository" instructions -of the [Linux package install guide](https://about.gitlab.com/install/#content). -Future GitLab upgrades will now be fetched according to your upgraded OS. - -## Supported Operating Systems - -GitLab officially supports LTS versions of operating systems. While OSs like -Ubuntu have a clear distinction between LTS and non-LTS versions, there are -other OSs, openSUSE for example, that don't follow the LTS concept. Hence to -avoid confusion, the official policy is that at any point of time, all the -operating systems supported by GitLab are listed in the [installation -page](https://about.gitlab.com/install/). - -The following lists the currently supported OSs and their possible EOL dates. - -| OS Version | First supported GitLab version | Arch | OS EOL | Details | -| ---------------- | ------------------------------ | --------------- | ------------- | ------------------------------------------------------------ | -| CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | June 2024 | <https://wiki.centos.org/About/Product> | -| CentOS 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, aarch64 | Dec 2021 | <https://wiki.centos.org/About/Product> | -| Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | 2022 | <https://wiki.debian.org/DebianReleases#Production_Releases> | -| Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | TBD | <https://wiki.debian.org/DebianReleases#Production_Releases> | -| OpenSUSE 15.2 | GitLab CE / GitLab EE 13.11.0 | x86_64, aarch64 | Dec 2021 | <https://en.opensuse.org/Lifetime> | -| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | Nov 2022 | <https://en.opensuse.org/Lifetime> | -| SLES 12 | GitLab EE 9.0.0 | x86_64 | Oct 2027 | <https://www.suse.com/lifecycle/> | -| Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | April 2023 | <https://wiki.ubuntu.com/Releases> | -| Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | April 2025 | <https://wiki.ubuntu.com/Releases> | -| Raspbian Buster | GitLab CE 12.2.0 | armhf | 2022 | <https://wiki.debian.org/DebianReleases#Production_Releases> | - -### Packages for ARM64 - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/issues/27) in GitLab 13.4. - -GitLab provides arm64/aarch64 packages for some supported operating systems. -You can see if your operating system architecture is supported in the table -above. - -WARNING: -There are currently still some [known issues and limitation](https://gitlab.com/groups/gitlab-org/-/epics/4397) -running GitLab on ARM. +<!-- This redirect file can be deleted after <2022-02-18>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/package_information/deprecation_policy.md b/doc/administration/package_information/deprecation_policy.md index d45c2ea3127..905de387dcb 100644 --- a/doc/administration/package_information/deprecation_policy.md +++ b/doc/administration/package_information/deprecation_policy.md @@ -16,18 +16,18 @@ setup, various configuration requires removal. ### Policy -The Omnibus GitLab package will retain configuration for at least **one major** -version. We cannot guarantee that deprecated configuration -will be available in the next major release. See [example](#example) for more details. +The Omnibus GitLab package retains configuration for at least **one major** +version. We can't guarantee that deprecated configuration +is available in the next major release. See [example](#example) for more details. ### Notice -If the configuration becomes obsolete, we will announce the deprecation: +If the configuration becomes obsolete, we announce the deprecation: - via release blog post on `https://about.gitlab.com/blog/`. The blog post item - will contain the deprecation notice together with the target removal date. + contains the deprecation notice together with the target removal date. - via installation/reconfigure output (if applicable). -- via official documentation on `https://docs.gitlab.com/`. The documentation update will contain the corrected syntax (if applicable) or a date of configuration removal. +- via official documentation on `https://docs.gitlab.com/`. The documentation update contains the corrected syntax (if applicable) or a date of configuration removal. ### Procedure @@ -82,16 +82,16 @@ The final comment in the issue **has to have**: ## Example -User configuration available in `/etc/gitlab/gitlab.rb` was introduced in GitLab version 10.0, `gitlab_rails['configuration'] = true`. In GitLab version 10.4.0, a new change was introduced that requires rename of this configuration option. New configuration option is `gitlab_rails['better_configuration'] = true`. Development team will translate the old configuration into new one -and trigger a deprecation procedure. +User configuration available in `/etc/gitlab/gitlab.rb` was introduced in GitLab version 10.0, `gitlab_rails['configuration'] = true`. In GitLab version 10.4.0, a new change was introduced that requires rename of this configuration option. New configuration option is `gitlab_rails['better_configuration'] = true`. Development team translates the old configuration into a new one +and triggers a deprecation procedure. This means that these two configuration -options will both be valid through GitLab version 10. In other words, +options are valid through GitLab version 10. In other words, if you still have `gitlab_rails['configuration'] = true` set in GitLab 10.8.0 -the feature will continue working the same way as if you had `gitlab_rails['better_configuration'] = true` set. -However, setting the old version of configuration will print out a deprecation +the feature continues working the same way as if you had `gitlab_rails['better_configuration'] = true` set. +However, setting the old version of the configuration prints out a deprecation notice at the end of installation/upgrade/reconfigure run. -With GitLab 11, `gitlab_rails['configuration'] = true` will no longer work and you will have to manually change the configuration in `/etc/gitlab/gitlab.rb` to the new valid configuration. +In GitLab 11, `gitlab_rails['configuration'] = true` no longer works and you must manually change the configuration in `/etc/gitlab/gitlab.rb` to the new valid configuration. **Note** If this configuration option is sensitive and can put integrity of the installation or -data in danger, installation/upgrade will be aborted. +data in danger,the installation or upgrade is aborted. diff --git a/doc/administration/package_information/index.md b/doc/administration/package_information/index.md index 12f3274ecab..ab4b1edfa30 100644 --- a/doc/administration/package_information/index.md +++ b/doc/administration/package_information/index.md @@ -18,7 +18,7 @@ The released package versions are in the format `MAJOR.MINOR.PATCH-EDITION.OMNIB |-------------------|---------|---------| | MAJOR.MINOR.PATCH | The GitLab version this corresponds to. | 13.3.0 | | EDITION | The edition of GitLab this corresponds to. | ee | -| OMNIBUS_RELEASE | The Omnibus GitLab release. Usually, this will be 0. This is incremented if we need to build a new package without changing the GitLab version. | 0 | +| OMNIBUS_RELEASE | The Omnibus GitLab release. Usually, this is 0. This is incremented if we need to build a new package without changing the GitLab version. | 0 | ## Licenses @@ -27,7 +27,7 @@ See [licensing](licensing.md) ## Defaults The Omnibus GitLab package requires various configuration to get the components -in working order. If the configuration is not provided, the package will use +in working order. If the configuration is not provided, the package uses the default values assumed in the package. These defaults are noted in the package [defaults document](defaults.md). @@ -59,8 +59,8 @@ accidental overwrite of user configuration provided in `/etc/gitlab/gitlab.rb`. New configuration options are noted in the [`gitlab.rb.template` file](https://gitlab.com/gitlab-org/omnibus-gitlab/raw/master/files/gitlab-config-template/gitlab.rb.template). -The Omnibus GitLab package also provides convenience command which will -compare the existing user configuration with the latest version of the +The Omnibus GitLab package also provides convenience command which +compares the existing user configuration with the latest version of the template contained in the package. To view a diff between your configuration file and the latest version, run: @@ -76,7 +76,7 @@ characters on each line. ## Init system detection -Omnibus GitLab will attempt to query the underlaying system in order to +Omnibus GitLab attempts to query the underlaying system in order to check which init system it uses. This manifests itself as a `WARNING` during the `sudo gitlab-ctl reconfigure` run. diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md new file mode 100644 index 00000000000..fcc2fef3e63 --- /dev/null +++ b/doc/administration/package_information/supported_os.md @@ -0,0 +1,90 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Supported operating systems **(FREE SELF)** + +GitLab officially supports LTS versions of operating systems. While OSs like +Ubuntu have a clear distinction between LTS and non-LTS versions, there are +other OSs, openSUSE for example, that don't follow the LTS concept. Hence to +avoid confusion, the official policy is that at any point of time, all the +operating systems supported by GitLab are listed in the [installation +page](https://about.gitlab.com/install/). + +The following lists the currently supported OSs and their possible EOL dates. + +| OS Version | First supported GitLab version | Arch | OS EOL | Details | +| ---------------- | ------------------------------ | --------------- | ------------- | ------------------------------------------------------------ | +| CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | June 2024 | <https://wiki.centos.org/About/Product> | +| CentOS 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, aarch64 | Dec 2021 | <https://wiki.centos.org/About/Product> | +| Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | 2022 | <https://wiki.debian.org/LTS> | +| Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | 2024 | <https://wiki.debian.org/LTS> | +| Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | 2026 | <https://wiki.debian.org/LTS> | +| OpenSUSE 15.2 | GitLab CE / GitLab EE 13.11.0 | x86_64, aarch64 | Dec 2021 | <https://en.opensuse.org/Lifetime> | +| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | Nov 2022 | <https://en.opensuse.org/Lifetime> | +| SLES 12 | GitLab EE 9.0.0 | x86_64 | Oct 2027 | <https://www.suse.com/lifecycle/> | +| Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | April 2023 | <https://wiki.ubuntu.com/Releases> | +| Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | April 2025 | <https://wiki.ubuntu.com/Releases> | +| Raspbian Buster | GitLab CE 12.2.0 | armhf | 2022 | <https://wiki.debian.org/DebianReleases#Production_Releases> | + +NOTE: +CentOS 8 will be EOL on December 31, 2021. In GitLab 14.5 and later, +[CentOS builds work in AlmaLinux](https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/954#note_730198505). +We will officially support all distributions that are binary compatible with Red Hat Enterprise Linux. +This gives users a path forward for their CentOS 8 builds at its end of life. + +## Update GitLab package sources after upgrading the OS + +After upgrading the Operating System (OS) as per its own documentation, +it may be necessary to also update the GitLab package source URL +in your package manager configuration. +If your package manager reports that no further updates are available, +although [new versions have been released](https://about.gitlab.com/releases/categories/releases/), repeat the +"Add the GitLab package repository" instructions +of the [Linux package install guide](https://about.gitlab.com/install/#content). +Future GitLab upgrades will now be fetched according to your upgraded OS. + +## Packages for ARM64 + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/issues/27) in GitLab 13.4. + +GitLab provides arm64/aarch64 packages for some supported operating systems. +You can see if your operating system architecture is supported in the table +above. + +WARNING: +There are currently still some [known issues and limitation](https://gitlab.com/groups/gitlab-org/-/epics/4397) +running GitLab on ARM. + +## OS Versions that are no longer supported + +GitLab provides omnibus packages for operating systems only until their +EOL (End-Of-Life). After the EOL date of the OS, GitLab will stop releasing +official packages. The list of deprecated operating systems and the final GitLab +release for them can be found below: + +| OS Version | End Of Life | Last supported GitLab version | +| --------------- | ---------------------------------------------------------------------------------- | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Raspbian Wheezy | [May 2015](https://downloads.raspberrypi.org/raspbian/images/raspbian-2015-05-07/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_8.17&dist=debian%2Fwheezy) 8.17 | +| OpenSUSE 13.2 | [January 2017](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-9.1&dist=opensuse%2F13.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-9.1&dist=opensuse%2F13.2) 9.1 | +| Ubuntu 12.04 | [April 2017](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_9.1&dist=ubuntu%2Fprecise) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_9.1&dist=ubuntu%2Fprecise) 9.1 | +| OpenSUSE 42.1 | [May 2017](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-9.3&dist=opensuse%2F42.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-9.3&dist=opensuse%2F42.1) 9.3 | +| OpenSUSE 42.2 | [January 2018](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-10.4&dist=opensuse%2F42.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-10.4&dist=opensuse%2F42.2) 10.4 | +| Debian Wheezy | [May 2018](https://www.debian.org/News/2018/20180601) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_11.6&dist=debian%2Fwheezy) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_11.6&dist=debian%2Fwheezy) 11.6 | +| Raspbian Jessie | [May 2017](https://downloads.raspberrypi.org/raspbian/images/raspbian-2017-07-05/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_11.7&dist=debian%2Fjessie) 11.7 | +| Ubuntu 14.04 | [April 2019](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_11.10&dist=ubuntu%2Ftrusty) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_11.10&dist=ubuntu%2Ftrusty) 11.10 | +| OpenSUSE 42.3 | [July 2019](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-12.1&dist=opensuse%2F42.3) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-12.1&dist=opensuse%2F42.3) 12.1 | +| OpenSUSE 15.0 | [December 2019](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-12.5&dist=opensuse%2F15.0) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-12.5&dist=opensuse%2F15.0) 12.5 | +| Raspbian Stretch | [June 2020](https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-04-09/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_13.2&dist=raspbian%2Fstretch) 13.3 | +| Debian Jessie | [June 2020](https://www.debian.org/News/2020/20200709) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.2&dist=debian%2Fjessie) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.2&dist=debian%2Fjessie) 13.3 | +| CentOS 6 | [November 2020](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=13.6&filter=all&filter=all&dist=el%2F6) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=13.6&filter=all&filter=all&dist=el%2F6) 13.6 | +| OpenSUSE 15.1 | [November 2020](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-13.12&dist=opensuse%2F15.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-13.12&dist=opensuse%2F15.2) 13.12 | +| Ubuntu 16.04 | [April 2021](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.12&dist=ubuntu%2Fxenial) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.12&dist=ubuntu%2Fxenial) 13.12 | + +NOTE: +An exception to this deprecation policy is when we are unable to provide +packages for the next version of the operating system. The most common reason +for this our package repository provider, PackageCloud, not supporting newer +versions and hence we can't upload packages to it. diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 7e711bb5740..0877fe510de 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -436,7 +436,7 @@ To configure the `s3` storage driver in Omnibus: ``` If using with an [AWS S3 VPC endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html), - then set `regionendpoint` to your VPC endpoint address and set `path_style` to false: + then set `regionendpoint` to your VPC endpoint address and set `pathstyle` to false: ```ruby registry['storage'] = { @@ -446,7 +446,7 @@ To configure the `s3` storage driver in Omnibus: 'bucket' => 'your-s3-bucket', 'region' => 'your-s3-region', 'regionendpoint' => 'your-s3-vpc-endpoint', - 'path_style' => false + 'pathstyle' => false } } ``` @@ -454,7 +454,7 @@ To configure the `s3` storage driver in Omnibus: - `regionendpoint` is only required when configuring an S3 compatible service such as MinIO, or when using an AWS S3 VPC Endpoint. - `your-s3-bucket` should be the name of a bucket that exists, and can't include subdirectories. - - `path_style` should be set to true to use `host/bucket_name/object` style paths instead of + - `pathstyle` should be set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. [Set to false for AWS S3](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/). 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -484,6 +484,12 @@ storage: #### Migrate to object storage without downtime +WARNING: +Using [AWS DataSync](https://aws.amazon.com/datasync/) +to copy the registry data to or between S3 buckets creates invalid metadata objects in the bucket. +For additional details, see [Tags with an empty name](#tags-with-an-empty-name). +To move data to and between S3 buckets, the AWS CLI `sync` operation is recommended. + To migrate storage without stopping the Container Registry, set the Container Registry to read-only mode. On large instances, this may require the Container Registry to be in read-only mode for a while. During this time, @@ -860,7 +866,7 @@ To remove image tags by running the cleanup policy, run the following commands i # Numeric ID of the project whose container registry should be cleaned up P = <project_id> -# Numeric ID of a developer, maintainer or owner in that project +# Numeric ID of a user with Developer, Maintainer, or Owner role for the project U = <user_id> # Get required details / objects @@ -873,7 +879,7 @@ project.container_repositories.find_each do |repo| puts repo.attributes # Start the tag cleanup - puts Projects::ContainerRepository::CleanupTagsService.new(project, user, policy.attributes.except("created_at", "updated_at")).execute(repo) + puts Projects::ContainerRepository::CleanupTagsService.new(repo, user, policy.attributes.except("created_at", "updated_at")).execute() end ``` @@ -888,7 +894,7 @@ GitLab offers a set of APIs to manipulate the Container Registry and aid the pro of removing unused tags. Currently, this is exposed using the API, but in the future, these controls should migrate to the GitLab interface. -Project maintainers can +Users who have the [Maintainer role](../../user/permissions.md) for the project can [delete Container Registry tags in bulk](../../api/container_registry.md#delete-registry-repository-tags-in-bulk) periodically based on their own criteria, however, this alone does not recycle data, it only unlinks tags from manifests and image blobs. To recycle the Container @@ -1072,6 +1078,19 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin You may want to add the `-m` flag to [remove untagged manifests and unreferenced layers](#removing-untagged-manifests-and-unreferenced-layers). +### Stop garbage collection + +If you anticipate stopping garbage collection, you should manually run garbage collection as +described in [Performing garbage collection without downtime](#performing-garbage-collection-without-downtime). +You can then stop garbage collection by pressing <kbd>Control</kbd>+<kbd>C</kbd>. + +Otherwise, interrupting `gitlab-ctl` could leave your registry service in a down state. In this +case, you must find the [garbage collection process](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-ctl-commands/registry_garbage_collect.rb#L26-35) +itself on the system so that the `gitlab-ctl` command can bring the registry service back up again. + +Also, there's no way to save progress or results during the mark phase of the process. Only once +blobs start being deleted is anything permanent done. + ## Configuring GitLab and Registry to run on separate nodes (Omnibus GitLab) By default, package assumes that both services are running on the same node. @@ -1080,28 +1099,28 @@ is necessary for Registry and GitLab. ### Configuring Registry -Below you will find configuration options you should set in `/etc/gitlab/gitlab.rb`, +Below you can find configuration options you should set in `/etc/gitlab/gitlab.rb`, for Registry to run separately from GitLab: - `registry['registry_http_addr']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L50). Needs to be reachable by web server (or LB). - `registry['token_realm']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L53). Specifies the endpoint to use to perform authentication, usually the GitLab URL. This endpoint needs to be reachable by user. - `registry['http_secret']`, [random string](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L32). A random piece of data used to sign state that may be stored with the client to protect against tampering. -- `registry['internal_key']`, default [automatically generated](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/recipes/gitlab-rails.rb#L113-119). Contents of the key that GitLab uses to sign the tokens. They key gets created on the Registry server, but it won't be used there. -- `gitlab_rails['registry_key_path']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/recipes/gitlab-rails.rb#L35). This is the path where `internal_key` contents will be written to disk. +- `registry['internal_key']`, default [automatically generated](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/recipes/gitlab-rails.rb#L113-119). Contents of the key that GitLab uses to sign the tokens. They key gets created on the Registry server, but it is not used there. +- `gitlab_rails['registry_key_path']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/recipes/gitlab-rails.rb#L35). This is the path where `internal_key` contents are written to disk. - `registry['internal_certificate']`, default [automatically generated](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/registry/recipes/enable.rb#L60-66). Contents of the certificate that GitLab uses to sign the tokens. - `registry['rootcertbundle']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/registry/recipes/enable.rb#L60). Path to certificate. This is the path where `internal_certificate` - contents will be written to disk. + contents are written to disk. - `registry['health_storagedriver_enabled']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-7-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L88). Configure whether health checks on the configured storage driver are enabled. - `gitlab_rails['registry_issuer']`, [default value](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/attributes/default.rb#L153). This setting needs to be set the same between Registry and GitLab. ### Configuring GitLab -Below you will find configuration options you should set in `/etc/gitlab/gitlab.rb`, +Below you can find configuration options you should set in `/etc/gitlab/gitlab.rb`, for GitLab to run separately from Registry: -- `gitlab_rails['registry_enabled']`, must be set to `true`. This setting will - signal to GitLab that it should allow Registry API requests. +- `gitlab_rails['registry_enabled']`, must be set to `true`. This setting + signals to GitLab that it should allow Registry API requests. - `gitlab_rails['registry_api_url']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L52). This is the Registry URL used internally that users do not need to interact with, `registry['registry_http_addr']` with scheme. - `gitlab_rails['registry_host']`, eg. `registry.gitlab.example`. Registry endpoint without the scheme, the address that gets shown to the end user. - `gitlab_rails['registry_port']`. Registry endpoint port, visible to the end user. @@ -1257,7 +1276,7 @@ Check which files are in use: enabled: true host: gitlab.company.com port: 4567 - api_url: http://127.0.0.1:5000 # internal address to the registry, will be used by GitLab to directly communicate with API + api_url: http://127.0.0.1:5000 # internal address to the registry, is used by GitLab to directly communicate with API path: /var/opt/gitlab/gitlab-rails/shared/registry --> key: /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key issuer: omnibus-gitlab-issuer @@ -1501,6 +1520,28 @@ The most straightforward option is to pull those images and push them once again using a Docker client version above v1.12. Docker converts images automatically before pushing them to the registry. Once done, all your v1 images should now be available as v2 images. +### Tags with an empty name + +If using [AWS DataSync](https://aws.amazon.com/datasync/) +to copy the registry data to or between S3 buckets, an empty metadata object is created in the root +path of each container repository in the destination bucket. This causes the registry to interpret +such files as a tag that appears with no name in the GitLab UI and API. For more information, see +[this issue](https://gitlab.com/gitlab-org/container-registry/-/issues/341). + +To fix this you can do one of two things: + +- Use the AWS CLI [`rm`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/rm.html) + command to remove the empty objects from the root of **each** affected repository. Pay special + attention to the trailing `/` and make sure **not** to use the `--recursive` option: + + ```shell + aws s3 rm s3://<bucket>/docker/registry/v2/repositories/<path to repository>/ + ``` + +- Use the AWS CLI [`sync`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html) + command to copy the registry data to a new bucket and configure the registry to use it. This + leaves the empty objects behind. + ### Advanced Troubleshooting We use a concrete example to illustrate how to diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 90f2d9127fe..eea4964efbe 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -218,8 +218,8 @@ We recommend using the [consolidated object storage settings](../object_storage. ### Migrating local packages to object storage -After [configuring the object storage](#using-object-storage), you may use the -following task to migrate existing packages from the local storage to the remote one. +After [configuring the object storage](#using-object-storage), use the following task to +migrate existing packages from the local storage to the remote storage. The processing is done in a background worker and requires **no downtime**. For Omnibus GitLab: @@ -234,11 +234,13 @@ For installations from source: RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:packages:migrate ``` -You can optionally track progress and verify that all packages migrated successfully. +You can optionally track progress and verify that all packages migrated successfully using the +[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -From the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) -(`sudo gitlab-psql -d gitlabhq_production` for Omnibus GitLab), verify that `objectstg` below (where -`file_store=2`) has the count of all packages: +- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. + +Verify `objectstg` below (where `store=2`) has count of all packages: ```shell gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM packages_package_files; @@ -247,3 +249,9 @@ total | filesystem | objectstg ------+------------+----------- 34 | 0 | 34 ``` + +Verify that there are no files on disk in the `packages` folder: + +```shell +sudo find /var/opt/gitlab/gitlab-rails/shared/packages -type f | grep -v tmp | wc -l +``` diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 163eb5388b6..f3ad474771c 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -56,11 +56,11 @@ Before proceeding with the Pages configuration, you must: | `gitlab.example.com` | `pages.example.com` | **{check-circle}** Yes | 1. Configure a **wildcard DNS record**. -1. (Optional) Have a **wildcard certificate** for that domain if you decide to +1. Optional. Have a **wildcard certificate** for that domain if you decide to serve Pages under HTTPS. -1. (Optional but recommended) Enable [Shared runners](../../ci/runners/index.md) +1. Optional but recommended. Enable [Shared runners](../../ci/runners/index.md) so that your users don't have to bring their own. -1. (Only for custom domains) Have a **secondary IP**. +1. For custom domains, have a **secondary IP**. NOTE: If your GitLab instance and the Pages daemon are deployed in a private network or behind a firewall, your GitLab Pages websites are only accessible to devices/users that have access to the private network. @@ -144,7 +144,8 @@ The Pages daemon doesn't listen to the outside world. 1. Set the external URL for GitLab Pages in `/etc/gitlab/gitlab.rb`: ```ruby - pages_external_url 'http://example.io' + external_url "http://gitlab.example.com" # external_url here is only for reference + pages_external_url "http://pages.example.com" # not a subdomain of external_url ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -169,7 +170,8 @@ outside world. 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby - pages_external_url 'https://example.io' + external_url "https://gitlab.example.com" # external_url here is only for reference + pages_external_url "https://pages.example.com" # not a subdomain of external_url pages_nginx['redirect_http_to_https'] = true ``` @@ -256,7 +258,6 @@ control over how the Pages daemon runs and serves content in your environment. | `pages_path` | The directory on disk where pages are stored, defaults to `GITLAB-RAILS/shared/pages`. | | **`pages_nginx[]`** | | | `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). | -| `FF_ENABLE_REDIRECTS` | Feature flag to enable/disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#feature-flag-for-redirects) for more information. | | `FF_ENABLE_PLACEHOLDERS` | Feature flag to enable/disable rewrites (disabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#feature-flag-for-rewrites) for more information. | | `use_legacy_storage` | Temporarily-introduced parameter allowing to use legacy domain configuration source and storage. [Removed in 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166). | | `rate_limit_source_ip` | Rate limit per source IP in number of requests per second. Set to `0` to disable this feature. | @@ -288,7 +289,8 @@ world. Custom domains are supported, but no TLS. 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby - pages_external_url "http://example.io" + external_url "http://gitlab.example.com" # external_url here is only for reference + pages_external_url "http://pages.example.com" # not a subdomain of external_url nginx['listen_addresses'] = ['192.0.2.1'] # The primary IP of the GitLab instance pages_nginx['enable'] = false gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001:db8::2]:80'] # The secondary IPs for the GitLab Pages daemon @@ -318,7 +320,8 @@ world. Custom domains and TLS are supported. 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby - pages_external_url "https://example.io" + external_url "https://gitlab.example.com" # external_url here is only for reference + pages_external_url "https://pages.example.com" # not a subdomain of external_url nginx['listen_addresses'] = ['192.0.2.1'] # The primary IP of the GitLab instance pages_nginx['enable'] = false gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001:db8::2]:80'] # The secondary IPs for the GitLab Pages daemon @@ -795,7 +798,7 @@ Incorrect configuration of these values may result in intermittent or persistent errors, or the Pages Daemon serving old content. NOTE: -Expiry, interval and timeout flags use [Golang's duration formatting](https://golang.org/pkg/time/#ParseDuration). +Expiry, interval and timeout flags use [Golang's duration formatting](https://pkg.go.dev/time#ParseDuration). A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as `300ms`, `1.5h` or `2h45m`. Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. @@ -1055,11 +1058,11 @@ Source-IP rate limits are enforced using the following: gitlab_pages['rate_limit_source_ip_burst'] = 600 ``` -1. To reject requests that exceed the specified limits, enable the `FF_ENABLE_RATE_LIMITER` feature flag in +1. To reject requests that exceed the specified limits, enable the `FF_ENFORCE_IP_RATE_LIMITS` feature flag in `/etc/gitlab/gitlab.rb`: ```ruby - gitlab_pages['env'] = {'FF_ENABLE_RATE_LIMITER' => 'true'} + gitlab_pages['env'] = {'FF_ENFORCE_IP_RATE_LIMITS' => 'true'} ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1281,8 +1284,8 @@ in all of your GitLab Pages instances. ### 500 error with `securecookie: failed to generate random iv` and `Failed to save the session` -This problem most likely results from an [out-dated operating system](../package_information/deprecated_os.md). -The [Pages daemon uses the `securecookie` library](https://gitlab.com/search?group_id=9970&project_id=734943&repository_ref=master&scope=blobs&search=securecookie&snippets=false) to get random strings via [`crypto/rand` in Go](https://golang.org/pkg/crypto/rand/#pkg-variables). +This problem most likely results from an [out-dated operating system](../package_information/supported_os.md#os-versions-that-are-no-longer-supported). +The [Pages daemon uses the `securecookie` library](https://gitlab.com/search?group_id=9970&project_id=734943&repository_ref=master&scope=blobs&search=securecookie&snippets=false) to get random strings via [`crypto/rand` in Go](https://pkg.go.dev/crypto/rand#pkg-variables). This requires the `getrandom` system call or `/dev/urandom` to be available on the host OS. Upgrading to an [officially supported operating system](https://about.gitlab.com/install/) is recommended. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 3a277204d21..45e9dadd1cf 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -59,9 +59,9 @@ Before proceeding with the Pages configuration, make sure that: 1. You have installed the `zip` and `unzip` packages in the same server that GitLab is installed since they are needed to compress and decompress the Pages artifacts. -1. (Optional) You have a **wildcard certificate** for the Pages domain if you +1. Optional. You have a **wildcard certificate** for the Pages domain if you decide to serve Pages (`*.example.io`) under HTTPS. -1. (Optional but recommended) You have configured and enabled the [shared runners](../../ci/runners/index.md) +1. Optional but recommended. You have configured and enabled the [shared runners](../../ci/runners/index.md) so that your users don't have to bring their own. ### DNS configuration diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md new file mode 100644 index 00000000000..b83820dd0b6 --- /dev/null +++ b/doc/administration/postgresql/database_load_balancing.md @@ -0,0 +1,234 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Database Load Balancing **(FREE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60894) from GitLab Premium to GitLab Free in 14.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334494) for Sidekiq in GitLab 14.1. + +With Database Load Balancing, read-only queries can be distributed across +multiple PostgreSQL nodes to increase performance. + +This functionality is provided natively in GitLab Rails and Sidekiq where +they can be configured to balance their database read queries in a round-robin approach, +without any external dependencies: + +```plantuml +@startuml +card "**Internal Load Balancer**" as ilb #9370DB +skinparam linetype ortho + +together { + collections "**GitLab Rails** x3" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 +} + +collections "**Consul** x3" as consul #e76a9b + +card "Database" as database { + collections "**PGBouncer x3**\n//Consul//" as pgbouncer #4EA7FF + + card "**PostgreSQL** //Primary//\n//Patroni//\n//PgBouncer//\n//Consul//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// **x2**\n//Patroni//\n//PgBouncer//\n//Consul//" as postgres_secondary #4EA7FF + + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]r-> postgres_secondary +} + +gitlab -[#32CD32]-> ilb +gitlab -[hidden]-> pgbouncer +gitlab .[#32CD32,norank]-> postgres_primary +gitlab .[#32CD32,norank]-> postgres_secondary + +sidekiq -[#ff8dd1]-> ilb +sidekiq -[hidden]-> pgbouncer +sidekiq .[#ff8dd1,norank]-> postgres_primary +sidekiq .[#ff8dd1,norank]-> postgres_secondary + +ilb -[#9370DB]-> pgbouncer + +consul -[#e76a9b]r-> pgbouncer +consul .[#e76a9b,norank]r-> postgres_primary +consul .[#e76a9b,norank]r-> postgres_secondary +@enduml +``` + +## Requirements to enable Database Load Balancing + +To enable Database Load Balancing, make sure that: + +- The HA PostgreSQL setup has one or more secondary nodes replicating the primary. +- Each PostgreSQL node is connected with the same credentials and on the same port. + +For Omnibus GitLab, you also need PgBouncer configured on each PostgreSQL node to pool +all load-balanced connections when [configuring a multi-node setup](replication_and_failover.md). + +## Configuring Database Load Balancing + +Database Load Balancing can be configured in one of two ways: + +- (Recommended) [Hosts](#hosts): a list of PostgreSQL hosts. +- [Service Discovery](#service-discovery): a DNS record that returns a list of PostgreSQL hosts. + +### Hosts + +To configure a list of hosts, add the `gitlab_rails['db_load_balancing']` setting into the +`gitlab.rb` file in the GitLab Rails / Sidekiq nodes for each environment you want to balance. + +For example, on an environment that has PostgreSQL running on the hosts `host1.example.com`, +`host2.example.com` and `host3.example.com` and reachable on the same port configured with +`gitlab_rails['db_port']`: + +1. On each GitLab Rails / Sidekiq node, edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] } + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Service Discovery + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883) in GitLab 11.0. + +Service discovery allows GitLab to automatically retrieve a list of PostgreSQL +hosts to use. It periodically +checks a DNS A record, using the IPs returned by this record as the addresses +for the secondaries. For service discovery to work, all you need is a DNS server +and an A record containing the IP addresses of your secondaries. + +When using Omnibus GitLab the provided [Consul](../consul.md) service works as +a DNS server and returns PostgreSQL addresses via the `postgresql-ha.service.consul` +record. For example: + +1. On each GitLab Rails / Sidekiq node, edit `/etc/gitlab/gitlab.rb` and add the following: + + ```ruby + gitlab_rails['db_load_balancing'] = { 'discover' => { + 'nameserver' => 'localhost' + 'record' => 'postgresql-ha.service.consul' + 'record_type' => 'A' + 'port' => '8600' + 'interval' => '60' + 'disconnect_timeout' => '120' + } + } + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +| Option | Description | Default | +|----------------------|---------------------------------------------------------------------------------------------------|-----------| +| `nameserver` | The nameserver to use for looking up the DNS record. | localhost | +| `record` | The record to look up. This option is required for service discovery to work. | | +| `record_type` | Optional record type to look up, this can be either A or SRV (GitLab 12.3 and later) | A | +| `port` | The port of the nameserver. | 8600 | +| `interval` | The minimum time in seconds between checking the DNS record. | 60 | +| `disconnect_timeout` | The time in seconds after which an old connection is closed, after the list of hosts was updated. | 120 | +| `use_tcp` | Lookup DNS resources using TCP instead of UDP | false | + +If `record_type` is set to `SRV`, then GitLab continues to use round-robin algorithm +and ignores the `weight` and `priority` in the record. Since SRV records usually +return hostnames instead of IPs, GitLab needs to look for the IPs of returned hostnames +in the additional section of the SRV response. If no IP is found for a hostname, GitLab +needs to query the configured `nameserver` for ANY record for each such hostname looking for A or AAAA +records, eventually dropping this hostname from rotation if it can't resolve its IP. + +The `interval` value specifies the _minimum_ time between checks. If the A +record has a TTL greater than this value, then service discovery honors said +TTL. For example, if the TTL of the A record is 90 seconds, then service +discovery waits at least 90 seconds before checking the A record again. + +When the list of hosts is updated, it might take a while for the old connections +to be terminated. The `disconnect_timeout` setting can be used to enforce an +upper limit on the time it takes to terminate all old database connections. + +### Handling Stale Reads **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526) in GitLab 10.3. + +To prevent reading from an outdated secondary the load balancer checks if it +is in sync with the primary. If the data is recent enough, the +secondary is used, otherwise it is ignored. To reduce the overhead of +these checks we only perform them at certain intervals. + +There are three configuration options that influence this behavior: + +| Option | Description | Default | +|------------------------------|----------------------------------------------------------------------------------------------------------------|------------| +| `max_replication_difference` | The amount of data (in bytes) a secondary is allowed to lag behind when it hasn't replicated data for a while. | 8 MB | +| `max_replication_lag_time` | The maximum number of seconds a secondary is allowed to lag behind before we stop using it. | 60 seconds | +| `replica_check_interval` | The minimum number of seconds we have to wait before checking the status of a secondary. | 60 seconds | + +The defaults should be sufficient for most users. + +To configure these options with a hosts list, use the following example: + +```ruby +gitlab_rails['db_load_balancing'] = { + 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] + 'max_replication_difference' => 16777216 # 16 MB + 'max_replication_lag_time' => 30 + 'replica_check_interval' => 30 +} +``` + +## Logging + +The load balancer logs various events in +[`database_load_balancing.log`](../logs.md#database_load_balancinglog), such as + +- When a host is marked as offline +- When a host comes back online +- When all secondaries are offline +- When a read is retried on a different host due to a query conflict + +The log is structured with each entry a JSON object containing at least: + +- An `event` field useful for filtering. +- A human-readable `message` field. +- Some event-specific metadata. For example, `db_host` +- Contextual information that is always logged. For example, `severity` and `time`. + +For example: + +```json +{"severity":"INFO","time":"2019-09-02T12:12:01.728Z","correlation_id":"abcdefg","event":"host_online","message":"Host came back online","db_host":"111.222.333.444","db_port":null,"tag":"rails.database_load_balancing","environment":"production","hostname":"web-example-1","fqdn":"gitlab.example.com","path":null,"params":null} +``` + +## Implementation Details + +### Balancing queries + +Read-only `SELECT` queries balance among all the given hosts. +Everything else (including transactions) executes on the primary. +Queries such as `SELECT ... FOR UPDATE` are also executed on the primary. + +### Prepared statements + +Prepared statements don't work well with load balancing and are disabled +automatically when load balancing is enabled. This shouldn't impact +response timings. + +### Primary sticking + +After a write has been performed, GitLab sticks to using the primary for a +certain period of time, scoped to the user that performed the write. GitLab +reverts back to using secondaries when they have either caught up, or after 30 +seconds. + +### Failover handling + +In the event of a failover or an unresponsive database, the load balancer +tries to use the next available host. If no secondaries are available the +operation is performed on the primary instead. + +If a connection error occurs while writing data, the +operation retries up to 3 times using an exponential back-off. + +When using load balancing, you should be able to safely restart a database server +without it immediately leading to errors being presented to the users. diff --git a/doc/administration/postgresql/img/pg_ha_architecture.png b/doc/administration/postgresql/img/pg_ha_architecture.png Binary files differdeleted file mode 100644 index 5d2a4a584bf..00000000000 --- a/doc/administration/postgresql/img/pg_ha_architecture.png +++ /dev/null diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index e5fef61540a..a666c1fab95 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -17,7 +17,7 @@ through `/etc/gitlab/gitlab.rb`. ## PgBouncer as part of a fault-tolerant GitLab installation -This content has been moved to a [new location](replication_and_failover.md#configuring-the-pgbouncer-node). +This content has been moved to a [new location](replication_and_failover.md#configure-pgbouncer-nodes). ## PgBouncer as part of a non-fault-tolerant GitLab installation diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 01fe4bf64ba..5777f35bfcf 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -19,13 +19,54 @@ replication and failover for GitLab. ## Architecture The Omnibus GitLab recommended configuration for a PostgreSQL cluster with -replication and failover requires: +replication failover requires: + +- A minimum of three PostgreSQL nodes. +- A minimum of three Consul server nodes. +- A minimum of three PgBouncer nodes that track and handle primary database reads and writes. + - An internal load balancer (TCP) to balance requests between the PgBouncer nodes. +- [Database Load Balancing](database_load_balancing.md) enabled. + - A local PgBouncer service configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. + +```plantuml +@startuml +card "**Internal Load Balancer**" as ilb #9370DB +skinparam linetype ortho + +together { + collections "**GitLab Rails** x3" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 +} + +collections "**Consul** x3" as consul #e76a9b -- A minimum of three database nodes. -- A minimum of three `Consul` server nodes. -- A minimum of one `pgbouncer` service node, but it's recommended to have one per database node. An internal load balancer (TCP) is required when there is more than one `pgbouncer` service node. +card "Database" as database { + collections "**PGBouncer x3**\n//Consul//" as pgbouncer #4EA7FF + + card "**PostgreSQL** //Primary//\n//Patroni//\n//PgBouncer//\n//Consul//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// **x2**\n//Patroni//\n//PgBouncer//\n//Consul//" as postgres_secondary #4EA7FF + + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]r-> postgres_secondary +} - +gitlab -[#32CD32]-> ilb +gitlab -[hidden]-> pgbouncer +gitlab .[#32CD32,norank]-> postgres_primary +gitlab .[#32CD32,norank]-> postgres_secondary + +sidekiq -[#ff8dd1]-> ilb +sidekiq -[hidden]-> pgbouncer +sidekiq .[#ff8dd1,norank]-> postgres_primary +sidekiq .[#ff8dd1,norank]-> postgres_secondary + +ilb -[#9370DB]-> pgbouncer + +consul -[#e76a9b]r-> pgbouncer +consul .[#e76a9b,norank]r-> postgres_primary +consul .[#e76a9b,norank]r-> postgres_secondary +@enduml +``` You also need to take into consideration the underlying network topology, making sure you have redundant connectivity between all Database and GitLab instances @@ -38,13 +79,14 @@ shipped with Omnibus GitLab, and thus Patroni becomes mandatory for replication ### Database node -Each database node runs three services: +Each database node runs four services: - `PostgreSQL`: The database itself. - `Patroni`: Communicates with other Patroni services in the cluster and handles failover when issues with the leader server occurs. The failover procedure consists of: - Selecting a new leader for the cluster. - Promoting the new node to leader. - Instructing remaining servers to follow the new leader node. +- `PgBouncer`: A local pooler for the node. Used for _read_ queries as part of [Database Load Balancing](database_load_balancing.md). - `Consul` agent: To communicate with Consul cluster which stores the current Patroni state. The agent monitors the status of each node in the database cluster and tracks its health in a service definition on the Consul cluster. ### Consul server node @@ -62,8 +104,26 @@ Each PgBouncer node runs two services: Each service in the package comes with a set of [default ports](../package_information/defaults.md#ports). You may need to make specific firewall rules for the connections listed below: +There are several connection flows in this setup: + +- [Primary](#primary) +- [Database Load Balancing](#database-load-balancing) +- [Replication](#replication) + +#### Primary + - Application servers connect to either PgBouncer directly via its [default port](../package_information/defaults.md) or via a configured Internal Load Balancer (TCP) that serves multiple PgBouncers. -- PgBouncer connects to the primary database servers [PostgreSQL default port](../package_information/defaults.md) +- PgBouncer connects to the primary database server's [PostgreSQL default port](../package_information/defaults.md). + +#### Database Load Balancing + +For read queries against data that haven't been recently changed and are up to date on all database nodes: + +- Application servers connect to the local PgBouncer service via its [default port](../package_information/defaults.md) on each database node in a round-robin approach. +- Local PgBouncer connects to the local database server's [PostgreSQL default port](../package_information/defaults.md). + +#### Replication + - Patroni actively manages the running PostgreSQL processes and configuration. - PostgreSQL secondaries connect to the primary database servers [PostgreSQL default port](../package_information/defaults.md) - Consul servers and agents connect to each others [Consul default ports](../package_information/defaults.md) @@ -203,8 +263,8 @@ repmgr-specific configuration as well. Especially, make sure that you remove `po Here is an example: ```ruby -# Disable all components except Patroni and Consul -roles(['patroni_role']) +# Disable all components except Patroni, PgBouncer and Consul +roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -245,6 +305,15 @@ patroni['allowlist'] = %w(XXX.XXX.XXX.XXX/YY 127.0.0.1/32) # Replace XXX.XXX.XXX.XXX/YY with Network Address postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY 127.0.0.1/32) +# Local PgBouncer service for Database Load Balancing +pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "PGBOUNCER_USERNAME", + password: 'PGBOUNCER_PASSWORD_HASH' + } +} + # Replace placeholders: # # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z @@ -342,7 +411,7 @@ You can use different certificates and keys for both API server and client on di However, the CA certificate (`patroni['tls_ca_file']`), TLS certificate verification (`patroni['tls_verify']`), and client TLS authentication mode (`patroni['tls_client_mode']`), must each have the same value on all nodes. -### Configuring the PgBouncer node +### Configure PgBouncer nodes 1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. @@ -480,6 +549,7 @@ attributes set, but the following need to be set. gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = 'POSTGRESQL_USER_PASSWORD' gitlab_rails['auto_migrate'] = false + gitlab_rails['db_load_balancing'] = { 'hosts' => ['POSTGRESQL_NODE_1', 'POSTGRESQL_NODE_2', 'POSTGRESQL_NODE_3'] } ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -595,8 +665,8 @@ An internal load balancer (TCP) is then required to be setup to serve each PgBou On database nodes edit `/etc/gitlab/gitlab.rb`: ```ruby -# Disable all components except Patroni and Consul -roles(['patroni_role']) +# Disable all components except Patroni, PgBouncer and Consul +roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -616,6 +686,15 @@ patroni['postgresql']['max_wal_senders'] = 7 patroni['allowlist'] = = %w(10.6.0.0/16 127.0.0.1/32) postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16 127.0.0.1/32) +# Local PgBouncer service for Database Load Balancing +pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "pgbouncer", + password: '771a8625958a529132abe6f1a4acb19c' + } +} + # Configure the Consul agent consul['services'] = %w(postgresql) consul['configuration'] = { @@ -650,115 +729,6 @@ After deploying the configuration follow these steps: gitlab-rake gitlab:db:configure ``` -### Example minimal setup - -This example uses 3 PostgreSQL servers, and 1 application node (with PgBouncer setup alongside). - -It differs from the [recommended setup](#example-recommended-setup) by moving the Consul servers into the same servers we use for PostgreSQL. -The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#manual-failover-procedure-for-patroni) procedures in addition to [Consul outage recovery](../consul.md#outage-recovery) on the same set of machines. - -In this example, we start with all servers on the same 10.6.0.0/16 private network range; they can connect to each freely other on those addresses. - -Here is a list and description of each machine and the assigned IP: - -- `10.6.0.21`: PostgreSQL 1 -- `10.6.0.22`: PostgreSQL 2 -- `10.6.0.23`: PostgreSQL 3 -- `10.6.0.31`: GitLab application - -All passwords are set to `toomanysecrets`. Please do not use this password or derived hashes. - -The `external_url` for GitLab is `http://gitlab.example.com` - -After the initial configuration, if a failover occurs, the PostgresSQL leader node changes to one of the available secondaries until it is failed back. - -#### Example minimal configuration for database servers - -On database nodes edit `/etc/gitlab/gitlab.rb`: - -```ruby -# Disable all components except Patroni and Consul -roles(['patroni_role']) - -# PostgreSQL configuration -postgresql['listen_address'] = '0.0.0.0' -postgresql['hot_standby'] = 'on' -postgresql['wal_level'] = 'replica' - -# Disable automatic database migrations -gitlab_rails['auto_migrate'] = false - -# Configure the Consul agent -consul['services'] = %w(postgresql) - -postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' -postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' - -# Sets `max_replication_slots` to double the number of database nodes. -# Patroni uses one extra slot per node when initiating the replication. -patroni['postgresql']['max_replication_slots'] = 6 - -patroni['username'] = 'PATRONI_API_USERNAME' -patroni['password'] = 'PATRONI_API_PASSWORD' - -# Set `max_wal_senders` to one more than the number of replication slots in the cluster. -# This is used to prevent replication from using up all of the -# available database connections. -patroni['postgresql']['max_wal_senders'] = 7 - -patroni['allowlist'] = = %w(10.6.0.0/16 127.0.0.1/32) -postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16 127.0.0.1/32) - -consul['configuration'] = { - server: true, - retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) -} -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -#### Example minimal configuration for application server - -On the server edit `/etc/gitlab/gitlab.rb`: - -```ruby -external_url 'http://gitlab.example.com' - -gitlab_rails['db_host'] = '127.0.0.1' -gitlab_rails['db_port'] = 6432 -gitlab_rails['db_password'] = 'toomanysecrets' -gitlab_rails['auto_migrate'] = false - -postgresql['enable'] = false -pgbouncer['enable'] = true -consul['enable'] = true - -# Configure PgBouncer -pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - -# Configure Consul agent -consul['watchers'] = %w(postgresql) - -pgbouncer['users'] = { - 'gitlab-consul': { - password: '5e0e3263571e3704ad655076301d6ebe' - }, - 'pgbouncer': { - password: '771a8625958a529132abe6f1a4acb19c' - } -} - -consul['configuration'] = { - retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) -} -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -#### Example minimal setup manual steps - -The manual steps for this configuration are the same as for the [example recommended setup](#example-recommended-setup-manual-steps). - ## Patroni NOTE: @@ -791,7 +761,7 @@ Run `gitlab-ctl patroni members` to query Patroni for a summary of the cluster s To verify the status of replication: ```shell -echo 'select * from pg_stat_wal_receiver\x\g\x \n select * from pg_stat_replication\x\g\x' | gitlab-psql +echo -e 'select * from pg_stat_wal_receiver\x\g\x \n select * from pg_stat_replication\x\g\x' | gitlab-psql ``` The same command can be run on all three database servers. It returns any information @@ -1047,7 +1017,7 @@ Here are a few key facts that you must consider before upgrading PostgreSQL: configured replication method (`pg_basebackup` is the only available option). It might take some time for replica to catch up with the leader, depending on the size of your database. -- An overview of the upgrade procedure is outlined in [Patoni's documentation](https://patroni.readthedocs.io/en/latest/existing_data.html#major-upgrade-of-postgresql-version). +- An overview of the upgrade procedure is outlined in [Patroni's documentation](https://patroni.readthedocs.io/en/latest/existing_data.html#major-upgrade-of-postgresql-version). You can still use `gitlab-ctl pg-upgrade` which implements this procedure with a few adjustments. Considering these, you should carefully plan your PostgreSQL upgrade: diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index d770361864e..950b508ab0c 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -268,7 +268,7 @@ sudo -u git -H bundle exec rake gitlab:tcp_check[example.com,80] RAILS_ENV=produ GitLab uses a shared lock mechanism: `ExclusiveLease` to prevent simultaneous operations in a shared resource. An example is running periodic garbage collection on repositories. -In very specific situations, a operation locked by an Exclusive Lease can fail without +In very specific situations, an operation locked by an Exclusive Lease can fail without releasing the lock. If you can't wait for it to expire, you can run this task to manually clear it. diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 017565e1b39..912cf260a03 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -13,7 +13,7 @@ uses to organize the Git data. ## List projects and attachments -The following Rake tasks will list the projects and attachments that are +The following Rake tasks lists the projects and attachments that are available on legacy and hashed storage. ### On legacy storage @@ -82,8 +82,8 @@ GitLab 14.0 eliminates support for legacy storage. If you're on GitLab The option to choose between hashed and legacy storage in the admin area has been disabled. -This task must be run on any machine that has Rails/Sidekiq configured and will -schedule all your existing projects and attachments associated with it to be +This task must be run on any machine that has Rails/Sidekiq configured, and the task +schedules all your existing projects and attachments associated with it to be migrated to the **Hashed** storage type: - **Omnibus installation** @@ -112,7 +112,7 @@ To monitor the progress in GitLab: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. Watch how long the `hashed_storage:hashed_storage_project_migrate` queue - will take to finish. After it reaches zero, you can confirm every project + takes to finish. After it reaches zero, you can confirm every project has been migrated by running the commands above. If you find it necessary, you can run the previous migration script again to schedule missing projects. @@ -160,12 +160,12 @@ sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 ``` You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. -On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish. +On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process takes to finish. After it reaches zero, you can confirm every project has been rolled back by running the commands above. If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks. Any error or warning is logged in Sidekiq's log file. -If you have a Geo setup, the rollback will not be reflected automatically +If you have a Geo setup, the rollback is not reflected automatically on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove the remaining repositories from the special `@hashed/` folder manually. diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index 0628e351b63..aec75f0b302 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -42,6 +42,28 @@ gitlab-rake "gitlab:uploads:migrate:all" sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate:all ``` +You can optionally track progress and verify that all packages migrated successfully using the +[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): + +- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. + +Verify `objectstg` below (where `store=2`) has count of all artifacts: + +```shell +gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads; + +total | filesystem | objectstg +------+------------+----------- + 2409 | 0 | 2409 +``` + +Verify that there are no files on disk in the `uploads` folder: + +```shell +sudo find /var/opt/gitlab/gitlab-rails/uploads -type f | grep -v tmp | wc -l +``` + ### Individual Rake tasks If you already ran the [all-in-one Rake task](#all-in-one-rake-task), there is no need to run these diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index 2fbcb2a62e7..b7e8397dd95 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -16,7 +16,7 @@ The configuration for doing so depends on your desired outcome. ## Make the repositories read-only -The first thing you'll want to accomplish is to ensure that no changes can be +The first thing you want to accomplish is to ensure that no changes can be made to your repositories. There's two ways you can accomplish that: - Either stop Puma to make the internal API unreachable: @@ -46,7 +46,7 @@ made to your repositories. There's two ways you can accomplish that: ## Shut down the GitLab UI If you don't mind shutting down the GitLab UI, then the easiest approach is to -stop `sidekiq` and `puma`, and you'll effectively ensure that no +stop `sidekiq` and `puma`, and you effectively ensure that no changes can be made to GitLab: ```shell @@ -63,7 +63,7 @@ sudo gitlab-ctl start puma ## Make the database read-only -If you want to allow users to use the GitLab UI, then you'll need to ensure that +If you want to allow users to use the GitLab UI, then you need to ensure that the database is read-only: 1. Take a [GitLab backup](../raketasks/backup_restore.md) @@ -113,7 +113,7 @@ the database is read-only: sudo gitlab-ctl restart postgresql ``` -When you're ready to revert the read-only state, you'll need to remove the added +When you're ready to revert the read-only state, you need to remove the added lines in `/etc/gitlab/gitlab.rb`, and reconfigure GitLab and restart PostgreSQL: ```shell diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index 6ab3d55e06a..f4aab9d7b7f 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -20,6 +20,18 @@ Before proceeding with the troubleshooting below, check your firewall rules: - Connect to other Sentinel machines via TCP in `26379` - Connect to the Redis machines via TCP in `6379` +## Basic Redis activity check + +Start Redis troubleshooting with a basic Redis activity check: + +1. Open a terminal on your GitLab server. +1. Run `gitlab-redis-cli --stat` and observe the output while it runs. +1. Go to your GitLab UI and browse to a handful of pages. Any page works, like + group or project overviews, issues, files in repositories, and so on. +1. Check the `stat` output again and verify that the values for `keys`, `clients`, + `requests`, and `connections` increases as you browse. If the numbers go up, + basic Redis functionality is working and GitLab can connect to it. + ## Troubleshooting Redis replication You can check if everything is correct by connecting to each server using diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 9c3c33e1fa8..fa8dfdf667b 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -12,6 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 10,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Estimated Costs:** [GCP](https://cloud.google.com/products/calculator#id=e77713f6-dc0b-4bb3-bcef-cea904ac8efd) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Performance tested daily with the [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance)**: > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS @@ -37,7 +38,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -49,6 +50,8 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 10k +skinparam linetype ortho + card "**External Load Balancer**" as elb #6a9be7 card "**Internal Load Balancer**" as ilb #9370DB @@ -73,8 +76,8 @@ card "Gitaly Cluster" as gitaly_cluster { card "Database" as database { collections "**PGBouncer** x3" as pgbouncer #4EA7FF - card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF - collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + card "**PostgreSQL** //Primary//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// x2" as postgres_secondary #4EA7FF pgbouncer -[#4EA7FF]-> postgres_primary postgres_primary .[#4EA7FF]> postgres_secondary @@ -83,31 +86,38 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 + + redis_cache -[hidden]-> redis_persistent } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]--> monitor +elb -[#6a9be7,norank]--> monitor -gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis +gitlab -[#32CD32,norank]--> ilb +gitlab -[#32CD32]r-> object_storage +gitlab -[#32CD32]----> redis +gitlab .[#32CD32]----> database gitlab -[hidden]-> monitor gitlab -[hidden]-> consul -sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis +sidekiq -[#ff8dd1,norank]--> ilb +sidekiq -[#ff8dd1]r-> object_storage +sidekiq -[#ff8dd1]----> redis +sidekiq .[#ff8dd1]----> database sidekiq -[hidden]-> monitor sidekiq -[hidden]-> consul -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden]--> redis +ilb -[hidden]u-> consul +ilb -[hidden]u-> monitor consul .[#e76a9b]u-> gitlab consul .[#e76a9b]u-> sidekiq -consul .[#e76a9b]> monitor +consul .[#e76a9b]r-> monitor consul .[#e76a9b]-> database consul .[#e76a9b]-> gitaly_cluster consul .[#e76a9b,norank]--> redis @@ -124,21 +134,34 @@ monitor .[#7FFFD4,norank]u--> elb @enduml ``` -The Google Cloud Platform (GCP) architectures were built and tested using the +## Requirements + +Before starting, you should take note of the following requirements / guidance for this reference architecture. + +### Supported CPUs + +This reference architecture was built and tested on Google Cloud Platform (GCP) using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -Due to better performance and availability, for data objects (such as LFS, -uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended. +### Supported infrastructure + +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. + +Be aware of the following specific call outs: -It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and +- [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. +- [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. + +### Praefect PostgreSQL + +It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) +can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). ## Setup components @@ -471,14 +494,15 @@ run: node-exporter: (pid 30093) 76833s; run: log: (pid 29663) 76855s ## Configure PostgreSQL -In this section, you'll be guided through configuring an external PostgreSQL database -to be used with GitLab. +In this section, you'll be guided through configuring a highly available PostgreSQL +cluster to be used with GitLab. ### Provide your own PostgreSQL instance If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. For example, AWS offers a managed Relational -Database Service (RDS) that runs PostgreSQL. +managed service for PostgreSQL. + +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -488,12 +512,25 @@ If you use a cloud-managed service, or provide your own PostgreSQL: needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) + with multiple read replicas is recommended. See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for further configuration steps. ### Standalone PostgreSQL using Omnibus GitLab +The recommended Omnibus GitLab configuration for a PostgreSQL cluster with +replication and failover requires: + +- A minimum of three PostgreSQL nodes. +- A minimum of three Consul server nodes. +- A minimum of three PgBouncer nodes that track and handle primary database reads and writes. + - An [internal load balancer](#configure-the-internal-load-balancer) (TCP) to balance requests between the PgBouncer nodes. +- [Database Load Balancing](../postgresql/database_load_balancing.md) enabled. + + A local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. + The following IPs will be used as an example: - `10.6.0.21`: PostgreSQL primary @@ -548,8 +585,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except Patroni and Consul - roles(['patroni_role']) + # Disable all components except Patroni, PgBouncer and Consul + roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -594,6 +631,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) + # Local PgBouncer service for Database Load Balancing + pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "pgbouncer", + password: '<pgbouncer_password_hash>' + } + } + # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' @@ -654,9 +700,11 @@ If the 'State' column for any node doesn't say "running", check the </a> </div> -## Configure PgBouncer +### Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer +for tracking and handling reads/writes to the primary database. -Now that the PostgreSQL servers are all set up, let's configure PgBouncer. The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -1216,6 +1264,15 @@ There are many third-party solutions for PostgreSQL HA. The solution selected mu - A static IP for all connections that doesn't change on failover. - [`LISTEN`](https://www.postgresql.org/docs/12/sql-listen.html) SQL functionality must be supported. +NOTE: +With a third-party setup, it's possible to colocate Praefect's database on the same server as +the main [GitLab](#provide-your-own-postgresql-instance) database as a convenience unless +you are using Geo, where separate database instances are required for handling replication correctly. +In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be +minimal. + +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). + Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1671,8 +1728,8 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1797,6 +1854,8 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -2120,8 +2179,7 @@ cluster alongside your instance, read how to ## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) -are recommended over NFS wherever possible for improved performance. If you intend -to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). +are recommended over NFS wherever possible for improved performance. See how to [configure NFS](../nfs.md). @@ -2200,7 +2258,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -2212,6 +2270,7 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 10k +skinparam linetype ortho card "Kubernetes via Helm Charts" as kubernetes { card "**External Load Balancer**" as elb #6a9be7 @@ -2221,7 +2280,6 @@ card "Kubernetes via Helm Charts" as kubernetes { collections "**Sidekiq** x4" as sidekiq #ff8dd1 } - card "**Prometheus + Grafana**" as monitor #7FFFD4 card "**Supporting Services**" as support } @@ -2249,37 +2307,33 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 + + redis_cache -[hidden]-> redis_persistent } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]-> monitor +elb -[hidden]-> sidekiq elb -[hidden]-> support gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis -gitlab -[hidden]--> consul +gitlab -[#32CD32]r--> object_storage +gitlab -[#32CD32,norank]----> redis +gitlab -[#32CD32]----> database sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis -sidekiq -[hidden]--> consul - -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +sidekiq -[#ff8dd1]r--> object_storage +sidekiq -[#ff8dd1,norank]----> redis +sidekiq .[#ff8dd1]----> database -consul .[#e76a9b]-> database -consul .[#e76a9b]-> gitaly_cluster -consul .[#e76a9b,norank]--> redis +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden,norank]--> redis -monitor .[#7FFFD4]> consul -monitor .[#7FFFD4]-> database -monitor .[#7FFFD4]-> gitaly_cluster -monitor .[#7FFFD4,norank]--> redis -monitor .[#7FFFD4]> ilb -monitor .[#7FFFD4,norank]u--> elb +consul .[#e76a9b]--> database +consul .[#e76a9b,norank]--> gitaly_cluster +consul .[#e76a9b]--> redis @enduml ``` diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index 5488d8d33a6..ed6fbe84a48 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -29,13 +29,64 @@ many organizations. | Up to 500 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Up to 1,000 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -The Google Cloud Platform (GCP) architectures were built and tested using the +```plantuml +@startuml 1k +card "**Prometheus + Grafana**" as monitor #7FFFD4 +package "GitLab Single Server" as gitlab-single-server { +together { + card "**GitLab Rails**" as gitlab #32CD32 + card "**Gitaly**" as gitaly #FF8C00 + card "**PostgreSQL**" as postgres #4EA7FF + card "**Redis**" as redis #FF6347 + card "**Sidekiq**" as sidekiq #ff8dd1 +} +card "Local Storage" as local_storage #white +} + +gitlab -[#32CD32]--> gitaly +gitlab -[#32CD32]--> postgres +gitlab -[#32CD32]--> redis +gitlab -[#32CD32]--> sidekiq +gitaly -[#32CD32]--> local_storage +postgres -[#32CD32]--> local_storage +sidekiq -[#32CD32]--> local_storage +gitlab -[#32CD32]--> local_storage + +monitor .[#7FFFD4]u-> gitlab +monitor .[#7FFFD4]u-> sidekiq +monitor .[#7FFFD4]-> postgres +monitor .[#7FFFD4]-> gitaly +monitor .[#7FFFD4,norank]--> redis + +@enduml +``` + +The diagram above shows that while GitLab can be installed on a single server, it is internally composed of multiple services. As a GitLab instance is scaled, each of these services are broken out and independently scaled according to the demands placed on them. In some cases PaaS can be leveraged for some services (e.g. Cloud Object Storage for some file systems). For the sake of redundancy some of the services become clusters of nodes storing the same data. In a horizontal configuration of GitLab there are various ancillary services required to coordinate clusters or discover of resources (e.g. PgBouncer for Postgres connection management, Consul for Prometheus end point discovery). + +## Requirements + +Before starting, you should take note of the following requirements / guidance for this reference architecture. + +### Supported CPUs + +This reference architecture was built and tested on Google Cloud Platform (GCP) using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). +### Supported infrastructure + +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. + +Be aware of the following specific call outs: + +- [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. +- [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. + +### Swap + In addition to the stated configurations, we recommend having at least 2 GB of swap on your server, even if you currently have enough available memory. Having swap helps to reduce the chance of errors occurring if your available memory diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 25cafbe667b..24b3350bd75 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -12,6 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 25,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Estimated Costs:** [GCP](https://cloud.google.com/products/calculator#id=925386e1-c01c-4c0a-8d7d-ebde1824b7b0) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: > - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS @@ -37,7 +38,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -49,6 +50,8 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 25k +skinparam linetype ortho + card "**External Load Balancer**" as elb #6a9be7 card "**Internal Load Balancer**" as ilb #9370DB @@ -73,8 +76,8 @@ card "Gitaly Cluster" as gitaly_cluster { card "Database" as database { collections "**PGBouncer** x3" as pgbouncer #4EA7FF - card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF - collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + card "**PostgreSQL** //Primary//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// x2" as postgres_secondary #4EA7FF pgbouncer -[#4EA7FF]-> postgres_primary postgres_primary .[#4EA7FF]> postgres_secondary @@ -83,31 +86,38 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 + + redis_cache -[hidden]-> redis_persistent } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]--> monitor +elb -[#6a9be7,norank]--> monitor -gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis +gitlab -[#32CD32,norank]--> ilb +gitlab -[#32CD32]r-> object_storage +gitlab -[#32CD32]----> redis +gitlab .[#32CD32]----> database gitlab -[hidden]-> monitor gitlab -[hidden]-> consul -sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis +sidekiq -[#ff8dd1,norank]--> ilb +sidekiq -[#ff8dd1]r-> object_storage +sidekiq -[#ff8dd1]----> redis +sidekiq .[#ff8dd1]----> database sidekiq -[hidden]-> monitor sidekiq -[hidden]-> consul -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden]--> redis +ilb -[hidden]u-> consul +ilb -[hidden]u-> monitor consul .[#e76a9b]u-> gitlab consul .[#e76a9b]u-> sidekiq -consul .[#e76a9b]> monitor +consul .[#e76a9b]r-> monitor consul .[#e76a9b]-> database consul .[#e76a9b]-> gitaly_cluster consul .[#e76a9b,norank]--> redis @@ -124,21 +134,34 @@ monitor .[#7FFFD4,norank]u--> elb @enduml ``` -The Google Cloud Platform (GCP) architectures were built and tested using the +## Requirements + +Before starting, you should take note of the following requirements / guidance for this reference architecture. + +### Supported CPUs + +This reference architecture was built and tested on Google Cloud Platform (GCP) using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -Due to better performance and availability, for data objects (such as LFS, -uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended. +### Supported infrastructure + +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. -It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and +Be aware of the following specific call outs: + +- [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. +- [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. + +### Praefect PostgreSQL + +It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) +can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). ## Setup components @@ -474,14 +497,15 @@ run: node-exporter: (pid 30093) 76833s; run: log: (pid 29663) 76855s ## Configure PostgreSQL -In this section, you'll be guided through configuring an external PostgreSQL database -to be used with GitLab. +In this section, you'll be guided through configuring a highly available PostgreSQL +cluster to be used with GitLab. ### Provide your own PostgreSQL instance If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. For example, AWS offers a managed Relational -Database Service (RDS) that runs PostgreSQL. +managed service for PostgreSQL. + +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -491,12 +515,25 @@ If you use a cloud-managed service, or provide your own PostgreSQL: needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) + with multiple read replicas is recommended. See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for further configuration steps. ### Standalone PostgreSQL using Omnibus GitLab +The recommended Omnibus GitLab configuration for a PostgreSQL cluster with +replication and failover requires: + +- A minimum of three PostgreSQL nodes. +- A minimum of three Consul server nodes. +- A minimum of three PgBouncer nodes that track and handle primary database reads and writes. + - An [internal load balancer](#configure-the-internal-load-balancer) (TCP) to balance requests between the PgBouncer nodes. +- [Database Load Balancing](../postgresql/database_load_balancing.md) enabled. + + A local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. + The following IPs will be used as an example: - `10.6.0.21`: PostgreSQL primary @@ -551,8 +588,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except Patroni and Consul - roles(['patroni_role']) + # Disable all components except Patroni, PgBouncer and Consul + roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -597,6 +634,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) + # Local PgBouncer service for Database Load Balancing + pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "pgbouncer", + password: '<pgbouncer_password_hash>' + } + } + # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' @@ -657,9 +703,11 @@ If the 'State' column for any node doesn't say "running", check the </a> </div> -## Configure PgBouncer +### Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer +for tracking and handling reads/writes to the primary database. -Now that the PostgreSQL servers are all set up, let's configure PgBouncer. The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -1222,7 +1270,14 @@ There are many third-party solutions for PostgreSQL HA. The solution selected mu - A static IP for all connections that doesn't change on failover. - [`LISTEN`](https://www.postgresql.org/docs/12/sql-listen.html) SQL functionality must be supported. -Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). +NOTE: +With a third-party setup, it's possible to colocate Praefect's database on the same server as +the main [GitLab](#provide-your-own-postgresql-instance) database as a convenience unless +you are using Geo, where separate database instances are required for handling replication correctly. +In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be +minimal. + +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1677,8 +1732,8 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1805,6 +1860,8 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -2126,8 +2183,7 @@ cluster alongside your instance, read how to ## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) -are recommended over NFS wherever possible for improved performance. If you intend -to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). +are recommended over NFS wherever possible for improved performance. See how to [configure NFS](../nfs.md). @@ -2200,7 +2256,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -2212,16 +2268,16 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 25k +skinparam linetype ortho card "Kubernetes via Helm Charts" as kubernetes { card "**External Load Balancer**" as elb #6a9be7 together { - collections "**Webservice** x7" as gitlab #32CD32 + collections "**Webservice** x4" as gitlab #32CD32 collections "**Sidekiq** x4" as sidekiq #ff8dd1 } - card "**Prometheus + Grafana**" as monitor #7FFFD4 card "**Supporting Services**" as support } @@ -2249,37 +2305,33 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 + + redis_cache -[hidden]-> redis_persistent } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]-> monitor +elb -[hidden]-> sidekiq elb -[hidden]-> support gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis -gitlab -[hidden]--> consul +gitlab -[#32CD32]r--> object_storage +gitlab -[#32CD32,norank]----> redis +gitlab -[#32CD32]----> database sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis -sidekiq -[hidden]--> consul - -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +sidekiq -[#ff8dd1]r--> object_storage +sidekiq -[#ff8dd1,norank]----> redis +sidekiq .[#ff8dd1]----> database -consul .[#e76a9b]-> database -consul .[#e76a9b]-> gitaly_cluster -consul .[#e76a9b,norank]--> redis +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden,norank]--> redis -monitor .[#7FFFD4]> consul -monitor .[#7FFFD4]-> database -monitor .[#7FFFD4]-> gitaly_cluster -monitor .[#7FFFD4,norank]--> redis -monitor .[#7FFFD4]> ilb -monitor .[#7FFFD4,norank]u--> elb +consul .[#e76a9b]--> database +consul .[#e76a9b,norank]--> gitaly_cluster +consul .[#e76a9b]--> redis @enduml ``` diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index e619294704f..f72c0877ddb 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -13,6 +13,7 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 2,000 > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). +> - **Estimated Costs:** [GCP](https://cloud.google.com/products/calculator#id=84d11491-d72a-493c-a16e-650931faa658) > - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Performance tested daily with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: > - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS @@ -27,10 +28,10 @@ For a full list of reference architectures, see | GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run as reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run as reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -41,6 +42,8 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 2k +skinparam linetype ortho + card "**External Load Balancer**" as elb #6a9be7 collections "**GitLab Rails** x3" as gitlab #32CD32 @@ -67,17 +70,27 @@ monitor .[#7FFFD4,norank]u--> elb @enduml ``` -The Google Cloud Platform (GCP) architectures were built and tested using the +## Requirements + +Before starting, you should take note of the following requirements / guidance for this reference architecture. + +### Supported CPUs + +This reference architecture was built and tested on Google Cloud Platform (GCP) using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -Due to better performance and availability, for data objects (such as LFS, -uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +### Supported infrastructure + +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. + +Be aware of the following specific call outs: + +- [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. +- [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. ## Setup components @@ -100,8 +113,7 @@ To set up GitLab and its components to accommodate up to 2,000 users: more advanced code search across your entire GitLab instance. 1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). + storage. ## Configure the external load balancer @@ -232,8 +244,9 @@ to be used with GitLab. ### Provide your own PostgreSQL instance If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. For example, AWS offers a managed relational -database service (RDS) that runs PostgreSQL. +managed service for PostgreSQL. + +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -958,8 +971,7 @@ cluster alongside your instance, read how to For improved performance, [object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly), are recommended over using NFS whenever -possible. However, if you intend to use GitLab Pages, -[you must use NFS](troubleshooting.md#gitlab-pages-requires-nfs). +possible. See how to [configure NFS](../nfs.md). @@ -1028,7 +1040,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> @@ -1038,6 +1050,7 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 2k +skinparam linetype ortho card "Kubernetes via Helm Charts" as kubernetes { card "**External Load Balancer**" as elb #6a9be7 @@ -1045,10 +1058,8 @@ card "Kubernetes via Helm Charts" as kubernetes { together { collections "**Webservice** x3" as gitlab #32CD32 collections "**Sidekiq** x2" as sidekiq #ff8dd1 + card "**Supporting Services**" as support } - - card "**Prometheus + Grafana**" as monitor #7FFFD4 - card "**Supporting Services**" as support } card "**Gitaly**" as gitaly #FF8C00 @@ -1057,7 +1068,6 @@ card "**Redis**" as redis #FF6347 cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]--> monitor gitlab -[#32CD32]--> gitaly gitlab -[#32CD32]--> postgres @@ -1066,14 +1076,8 @@ gitlab -[#32CD32]--> redis sidekiq -[#ff8dd1]--> gitaly sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> postgres -sidekiq -[#ff8dd1]---> redis - -monitor .[#7FFFD4]u-> gitlab -monitor .[#7FFFD4]-> gitaly -monitor .[#7FFFD4]-> postgres -monitor .[#7FFFD4,norank]--> redis -monitor .[#7FFFD4,norank]u--> elb +sidekiq -[#ff8dd1]--> postgres +sidekiq -[#ff8dd1]--> redis @enduml ``` diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 9332ae8d271..c788a73753b 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -22,6 +22,7 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 3,000 > - **High Availability:** Yes, although [Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution +> - **Estimated Costs:** [GCP](https://cloud.google.com/products/calculator/#id=ac4838e6-9c40-4a36-ac43-6d1bc1843e08) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: > - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS @@ -42,11 +43,11 @@ For a full list of reference architectures, see | GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -58,6 +59,8 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 3k +skinparam linetype ortho + card "**External Load Balancer**" as elb #6a9be7 card "**Internal Load Balancer**" as ilb #9370DB @@ -66,7 +69,10 @@ together { collections "**Sidekiq** x4" as sidekiq #ff8dd1 } -card "**Prometheus + Grafana**" as monitor #7FFFD4 +together { + card "**Prometheus + Grafana**" as monitor #7FFFD4 + collections "**Consul** x3" as consul #e76a9b +} card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 @@ -79,47 +85,45 @@ card "Gitaly Cluster" as gitaly_cluster { card "Database" as database { collections "**PGBouncer** x3" as pgbouncer #4EA7FF - card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF - collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + card "**PostgreSQL** //Primary//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// x2" as postgres_secondary #4EA7FF pgbouncer -[#4EA7FF]-> postgres_primary postgres_primary .[#4EA7FF]> postgres_secondary } -card "**Consul + Sentinel**" as consul_sentinel { - collections "**Consul** x3" as consul #e76a9b - collections "**Redis Sentinel** x3" as sentinel #e6e727 -} - card "Redis" as redis { collections "**Redis** x3" as redis_nodes #FF6347 - - redis_nodes <.[#FF6347]- sentinel } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]--> monitor +elb -[#6a9be7,norank]--> monitor -gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis +gitlab -[#32CD32,norank]--> ilb +gitlab -[#32CD32]r-> object_storage +gitlab -[#32CD32]----> redis +gitlab .[#32CD32]----> database gitlab -[hidden]-> monitor gitlab -[hidden]-> consul -sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis +sidekiq -[#ff8dd1,norank]--> ilb +sidekiq -[#ff8dd1]r-> object_storage +sidekiq -[#ff8dd1]----> redis +sidekiq .[#ff8dd1]----> database sidekiq -[hidden]-> monitor sidekiq -[hidden]-> consul -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden]--> redis +ilb -[hidden]u-> consul +ilb -[hidden]u-> monitor consul .[#e76a9b]u-> gitlab consul .[#e76a9b]u-> sidekiq -consul .[#e76a9b]> monitor +consul .[#e76a9b]r-> monitor consul .[#e76a9b]-> database consul .[#e76a9b]-> gitaly_cluster consul .[#e76a9b,norank]--> redis @@ -136,27 +140,34 @@ monitor .[#7FFFD4,norank]u--> elb @enduml ``` -The Google Cloud Platform (GCP) architectures were built and tested using the +## Requirements + +Before starting, you should take note of the following requirements / guidance for this reference architecture. + +### Supported CPUs + +This reference architecture was built and tested on Google Cloud Platform (GCP) using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -Due to better performance and availability, for data objects (such as LFS, -uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +### Supported infrastructure + +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. -[Praefect requires its own database server](../gitaly/praefect.md#postgresql), -and a third-party PostgreSQL database solution is required to achieve full -high availability. Although we hope to offer a built-in solution for these -restrictions in the future, you can set up a non-HA PostgreSQL server by using -Omnibus GitLab (which the previous specifications reflect). Refer to the -following issues for more information: +Be aware of the following specific call outs: -- [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) -- [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) +- [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. +- [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. + +### Praefect PostgreSQL + +It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and +that to achieve full High Availability a third-party PostgreSQL database solution will be required. +We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server +can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). ## Setup components @@ -184,8 +195,7 @@ To set up GitLab and its components to accommodate up to 3,000 users: more advanced code search across your entire GitLab instance. 1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). + storage. The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -769,14 +779,15 @@ run: sentinel: (pid 30098) 76832s; run: log: (pid 29704) 76850s ## Configure PostgreSQL -In this section, you'll be guided through configuring an external PostgreSQL database -to be used with GitLab. +In this section, you'll be guided through configuring a highly available PostgreSQL +cluster to be used with GitLab. ### Provide your own PostgreSQL instance If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. For example, AWS offers a managed Relational -Database Service (RDS) that runs PostgreSQL. +managed service for PostgreSQL. + +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -786,12 +797,25 @@ If you use a cloud-managed service, or provide your own PostgreSQL: needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) + with multiple read replicas is recommended. See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for further configuration steps. ### Standalone PostgreSQL using Omnibus GitLab +The recommended Omnibus GitLab configuration for a PostgreSQL cluster with +replication and failover requires: + +- A minimum of three PostgreSQL nodes. +- A minimum of three Consul server nodes. +- A minimum of three PgBouncer nodes that track and handle primary database reads and writes. + - An [internal load balancer](#configure-the-internal-load-balancer) (TCP) to balance requests between the PgBouncer nodes. +- [Database Load Balancing](../postgresql/database_load_balancing.md) enabled. + + A local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. + The following IPs will be used as an example: - `10.6.0.31`: PostgreSQL primary @@ -846,8 +870,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except Patroni and Consul - roles(['patroni_role']) + # Disable all components except Patroni, PgBouncer and Consul + roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -892,6 +916,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) + # Local PgBouncer service for Database Load Balancing + pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "pgbouncer", + password: '<pgbouncer_password_hash>' + } + } + # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' @@ -952,9 +985,11 @@ If the 'State' column for any node doesn't say "running", check the </a> </div> -## Configure PgBouncer +### Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer +for tracking and handling reads/writes to the primary database. -Now that the PostgreSQL servers are all set up, let's configure PgBouncer. The following IPs will be used as an example: - `10.6.0.21`: PgBouncer 1 @@ -1175,7 +1210,14 @@ There are many third-party solutions for PostgreSQL HA. The solution selected mu - A static IP for all connections that doesn't change on failover. - [`LISTEN`](https://www.postgresql.org/docs/12/sql-listen.html) SQL functionality must be supported. -Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). +NOTE: +With a third-party setup, it's possible to colocate Praefect's database on the same server as +the main [GitLab](#provide-your-own-postgresql-instance) database as a convenience unless +you are using Geo, where separate database instances are required for handling replication correctly. +In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be +minimal. + +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1613,8 +1655,8 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.31', '10.6.0.32', '10.6.0.33'] } # PostgreSQL IPs + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1773,6 +1815,8 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.31', '10.6.0.32', '10.6.0.33'] } # PostgreSQL IPs + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -2074,8 +2118,7 @@ cluster alongside your instance, read how to ## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) -are recommended over NFS wherever possible for improved performance. If you intend -to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). +are recommended over NFS wherever possible for improved performance. See how to [configure NFS](../nfs.md). @@ -2102,7 +2145,7 @@ but with smaller performance requirements, several modifications can be consider - Lowering node specs: Depending on your user count, you can lower all suggested node specs as desired. However, it's recommended that you don't go lower than the [general requirements](../../install/requirements.md). - Combining select nodes: Some nodes can be combined to reduce complexity at the cost of some performance: - GitLab Rails and Sidekiq: Sidekiq nodes can be removed and the component instead enabled on the GitLab Rails nodes. - - PostgreSQL and PgBouncer: PgBouncer nodes can be removed and the component instead enabled on PostgreSQL with the Internal Load Balancer pointing to them instead. + - PostgreSQL and PgBouncer: PgBouncer nodes could be removed and instead be enabled on PostgreSQL nodes with the Internal Load Balancer pointing to them. However, to enable [Database Load Balancing](../postgresql/database_load_balancing.md), a separate PgBouncer array is still required. - Reducing the node counts: Some node types do not need consensus and can run with fewer nodes (but more than one for redundancy). This will also lead to reduced performance. - GitLab Rails and Sidekiq: Stateless services don't have a minimum node count. Two are enough for redundancy. - Gitaly and Praefect: A quorum is not strictly necessary. Two Gitaly nodes and two Praefect nodes are enough for redundancy. @@ -2171,7 +2214,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -2183,25 +2226,21 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 3k +skinparam linetype ortho card "Kubernetes via Helm Charts" as kubernetes { card "**External Load Balancer**" as elb #6a9be7 together { - collections "**Webservice** x2" as gitlab #32CD32 - collections "**Sidekiq** x3" as sidekiq #ff8dd1 + collections "**Webservice** x4" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 } - card "**Prometheus + Grafana**" as monitor #7FFFD4 card "**Supporting Services**" as support } card "**Internal Load Balancer**" as ilb #9370DB - -card "**Consul + Sentinel**" as consul_sentinel { - collections "**Consul** x3" as consul #e76a9b - collections "**Redis Sentinel** x3" as sentinel #e6e727 -} +collections "**Consul** x3" as consul #e76a9b card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 @@ -2221,41 +2260,33 @@ card "Database" as database { postgres_primary .[#4EA7FF]> postgres_secondary } -card "Redis" as redis { +card "redis" as redis { collections "**Redis** x3" as redis_nodes #FF6347 - - redis_nodes <.[#FF6347]- sentinel } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]-> monitor +elb -[hidden]-> sidekiq elb -[hidden]-> support gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis -gitlab -[hidden]--> consul +gitlab -[#32CD32]r--> object_storage +gitlab -[#32CD32,norank]----> redis +gitlab -[#32CD32]----> database sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis -sidekiq -[hidden]--> consul - -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +sidekiq -[#ff8dd1]r--> object_storage +sidekiq -[#ff8dd1,norank]----> redis +sidekiq .[#ff8dd1]----> database -consul .[#e76a9b]-> database -consul .[#e76a9b]-> gitaly_cluster -consul .[#e76a9b,norank]--> redis +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden,norank]--> redis -monitor .[#7FFFD4]> consul -monitor .[#7FFFD4]-> database -monitor .[#7FFFD4]-> gitaly_cluster -monitor .[#7FFFD4,norank]--> redis -monitor .[#7FFFD4]> ilb -monitor .[#7FFFD4,norank]u--> elb +consul .[#e76a9b]--> database +consul .[#e76a9b,norank]--> gitaly_cluster +consul .[#e76a9b]--> redis @enduml ``` diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index bbdf798d9ad..4f576fc1c19 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -12,6 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 50,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Estimated Costs:** [GCP](https://cloud.google.com/products/calculator/#id=8006396b-88ee-40cd-a1c8-77cdefa4d3c8) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: > - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS @@ -37,7 +38,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -49,6 +50,8 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 50k +skinparam linetype ortho + card "**External Load Balancer**" as elb #6a9be7 card "**Internal Load Balancer**" as ilb #9370DB @@ -73,8 +76,8 @@ card "Gitaly Cluster" as gitaly_cluster { card "Database" as database { collections "**PGBouncer** x3" as pgbouncer #4EA7FF - card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF - collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + card "**PostgreSQL** //Primary//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// x2" as postgres_secondary #4EA7FF pgbouncer -[#4EA7FF]-> postgres_primary postgres_primary .[#4EA7FF]> postgres_secondary @@ -83,31 +86,38 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 + + redis_cache -[hidden]-> redis_persistent } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]--> monitor +elb -[#6a9be7,norank]--> monitor -gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis +gitlab -[#32CD32,norank]--> ilb +gitlab -[#32CD32]r-> object_storage +gitlab -[#32CD32]----> redis +gitlab .[#32CD32]----> database gitlab -[hidden]-> monitor gitlab -[hidden]-> consul -sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis +sidekiq -[#ff8dd1,norank]--> ilb +sidekiq -[#ff8dd1]r-> object_storage +sidekiq -[#ff8dd1]----> redis +sidekiq .[#ff8dd1]----> database sidekiq -[hidden]-> monitor sidekiq -[hidden]-> consul -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden]--> redis +ilb -[hidden]u-> consul +ilb -[hidden]u-> monitor consul .[#e76a9b]u-> gitlab consul .[#e76a9b]u-> sidekiq -consul .[#e76a9b]> monitor +consul .[#e76a9b]r-> monitor consul .[#e76a9b]-> database consul .[#e76a9b]-> gitaly_cluster consul .[#e76a9b,norank]--> redis @@ -124,21 +134,34 @@ monitor .[#7FFFD4,norank]u--> elb @enduml ``` -The Google Cloud Platform (GCP) architectures were built and tested using the +## Requirements + +Before starting, you should take note of the following requirements / guidance for this reference architecture. + +### Supported CPUs + +This reference architecture was built and tested on Google Cloud Platform (GCP) using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -Due to better performance and availability, for data objects (such as LFS, -uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended. +### Supported infrastructure + +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. + +Be aware of the following specific call outs: -It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and +- [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. +- [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. + +### Praefect PostgreSQL + +It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) +can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). ## Setup components @@ -480,14 +503,15 @@ run: node-exporter: (pid 30093) 76833s; run: log: (pid 29663) 76855s ## Configure PostgreSQL -In this section, you'll be guided through configuring an external PostgreSQL database -to be used with GitLab. +In this section, you'll be guided through configuring a highly available PostgreSQL +cluster to be used with GitLab. ### Provide your own PostgreSQL instance If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. For example, AWS offers a managed Relational -Database Service (RDS) that runs PostgreSQL. +managed service for PostgreSQL. + +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -497,12 +521,25 @@ If you use a cloud-managed service, or provide your own PostgreSQL: needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) + with multiple read replicas is recommended. See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for further configuration steps. ### Standalone PostgreSQL using Omnibus GitLab +The recommended Omnibus GitLab configuration for a PostgreSQL cluster with +replication and failover requires: + +- A minimum of three PostgreSQL nodes. +- A minimum of three Consul server nodes. +- A minimum of three PgBouncer nodes that track and handle primary database reads and writes. + - An [internal load balancer](#configure-the-internal-load-balancer) (TCP) to balance requests between the PgBouncer nodes. +- [Database Load Balancing](../postgresql/database_load_balancing.md) enabled. + + A local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. + The following IPs will be used as an example: - `10.6.0.21`: PostgreSQL primary @@ -557,8 +594,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except Patroni and Consul - roles(['patroni_role']) + # Disable all components except Patroni, PgBouncer and Consul + roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -604,6 +641,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) + # Local PgBouncer service for Database Load Balancing + pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "pgbouncer", + password: '<pgbouncer_password_hash>' + } + } + # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' @@ -664,9 +710,11 @@ If the 'State' column for any node doesn't say "running", check the </a> </div> -## Configure PgBouncer +### Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer +for tracking and handling reads/writes to the primary database. -Now that the PostgreSQL servers are all set up, let's configure PgBouncer. The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -891,7 +939,7 @@ a node and change its status from primary to replica (and vice versa). package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the priimary node in the previous section by replacing `redis_master_node` with `redis_replica_node`: +1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the primary node in the previous section by replacing `redis_master_node` with `redis_replica_node`: ```ruby # Specify server role as 'redis_replica_role' with Sentinel and enable Consul agent @@ -1229,6 +1277,15 @@ There are many third-party solutions for PostgreSQL HA. The solution selected mu - A static IP for all connections that doesn't change on failover. - [`LISTEN`](https://www.postgresql.org/docs/12/sql-listen.html) SQL functionality must be supported. +NOTE: +With a third-party setup, it's possible to colocate Praefect's database on the same server as +the main [GitLab](#provide-your-own-postgresql-instance) database as a convenience unless +you are using Geo, where separate database instances are required for handling replication correctly. +In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be +minimal. + +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). + Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1684,8 +1741,8 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1819,6 +1876,8 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -2140,8 +2199,7 @@ cluster alongside your instance, read how to ## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) -are recommended over NFS wherever possible for improved performance. If you intend -to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). +are recommended over NFS wherever possible for improved performance. See how to [configure NFS](../nfs.md). @@ -2214,7 +2272,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -2226,16 +2284,16 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 50k +skinparam linetype ortho card "Kubernetes via Helm Charts" as kubernetes { card "**External Load Balancer**" as elb #6a9be7 together { - collections "**Webservice** x16" as gitlab #32CD32 + collections "**Webservice** x4" as gitlab #32CD32 collections "**Sidekiq** x4" as sidekiq #ff8dd1 } - card "**Prometheus + Grafana**" as monitor #7FFFD4 card "**Supporting Services**" as support } @@ -2263,37 +2321,33 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 + + redis_cache -[hidden]-> redis_persistent } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]-> monitor +elb -[hidden]-> sidekiq elb -[hidden]-> support gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis -gitlab -[hidden]--> consul +gitlab -[#32CD32]r--> object_storage +gitlab -[#32CD32,norank]----> redis +gitlab -[#32CD32]----> database sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis -sidekiq -[hidden]--> consul - -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +sidekiq -[#ff8dd1]r--> object_storage +sidekiq -[#ff8dd1,norank]----> redis +sidekiq .[#ff8dd1]----> database -consul .[#e76a9b]-> database -consul .[#e76a9b]-> gitaly_cluster -consul .[#e76a9b,norank]--> redis +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden,norank]--> redis -monitor .[#7FFFD4]> consul -monitor .[#7FFFD4]-> database -monitor .[#7FFFD4]-> gitaly_cluster -monitor .[#7FFFD4,norank]--> redis -monitor .[#7FFFD4]> ilb -monitor .[#7FFFD4,norank]u--> elb +consul .[#e76a9b]--> database +consul .[#e76a9b,norank]--> gitaly_cluster +consul .[#e76a9b]--> redis @enduml ``` diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index a1921f50e4e..92950806cfb 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -19,6 +19,7 @@ costly-to-operate environment by using the > - **Supported users (approximate):** 5,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Estimated Costs:** [GCP](https://cloud.google.com/products/calculator/#id=8742e8ea-c08f-4e0a-b058-02f3a1c38a2f) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: > - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS @@ -39,11 +40,11 @@ costly-to-operate environment by using the | GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | `F16s v2`| | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -55,6 +56,8 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 5k +skinparam linetype ortho + card "**External Load Balancer**" as elb #6a9be7 card "**Internal Load Balancer**" as ilb #9370DB @@ -63,7 +66,10 @@ together { collections "**Sidekiq** x4" as sidekiq #ff8dd1 } -card "**Prometheus + Grafana**" as monitor #7FFFD4 +together { + card "**Prometheus + Grafana**" as monitor #7FFFD4 + collections "**Consul** x3" as consul #e76a9b +} card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 @@ -76,47 +82,45 @@ card "Gitaly Cluster" as gitaly_cluster { card "Database" as database { collections "**PGBouncer** x3" as pgbouncer #4EA7FF - card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF - collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + card "**PostgreSQL** //Primary//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// x2" as postgres_secondary #4EA7FF pgbouncer -[#4EA7FF]-> postgres_primary postgres_primary .[#4EA7FF]> postgres_secondary } -card "**Consul + Sentinel**" as consul_sentinel { - collections "**Consul** x3" as consul #e76a9b - collections "**Redis Sentinel** x3" as sentinel #e6e727 -} - card "Redis" as redis { collections "**Redis** x3" as redis_nodes #FF6347 - - redis_nodes <.[#FF6347]- sentinel } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]--> monitor +elb -[#6a9be7,norank]--> monitor -gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis +gitlab -[#32CD32,norank]--> ilb +gitlab -[#32CD32]r-> object_storage +gitlab -[#32CD32]----> redis +gitlab .[#32CD32]----> database gitlab -[hidden]-> monitor gitlab -[hidden]-> consul -sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis +sidekiq -[#ff8dd1,norank]--> ilb +sidekiq -[#ff8dd1]r-> object_storage +sidekiq -[#ff8dd1]----> redis +sidekiq .[#ff8dd1]----> database sidekiq -[hidden]-> monitor sidekiq -[hidden]-> consul -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden]--> redis +ilb -[hidden]u-> consul +ilb -[hidden]u-> monitor consul .[#e76a9b]u-> gitlab consul .[#e76a9b]u-> sidekiq -consul .[#e76a9b]> monitor +consul .[#e76a9b]r-> monitor consul .[#e76a9b]-> database consul .[#e76a9b]-> gitaly_cluster consul .[#e76a9b,norank]--> redis @@ -133,22 +137,34 @@ monitor .[#7FFFD4,norank]u--> elb @enduml ``` -The Google Cloud Platform (GCP) architectures were built and tested using the +## Requirements + +Before starting, you should take note of the following requirements / guidance for this reference architecture. + +### Supported CPUs + +This reference architecture was built and tested on Google Cloud Platform (GCP) using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -Due to better performance and availability, for data objects (such as LFS, -uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +### Supported infrastructure + +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. + +Be aware of the following specific call outs: + +- [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. +- [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. + +### Praefect PostgreSQL -It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and +It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) +can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). ## Setup components @@ -760,14 +776,15 @@ run: sentinel: (pid 30098) 76832s; run: log: (pid 29704) 76850s ## Configure PostgreSQL -In this section, you'll be guided through configuring an external PostgreSQL database -to be used with GitLab. +In this section, you'll be guided through configuring a highly available PostgreSQL +cluster to be used with GitLab. ### Provide your own PostgreSQL instance If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. For example, AWS offers a managed Relational -Database Service (RDS) that runs PostgreSQL. +managed service for PostgreSQL. + +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -777,12 +794,25 @@ If you use a cloud-managed service, or provide your own PostgreSQL: needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) + with multiple read replicas is recommended. See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for further configuration steps. ### Standalone PostgreSQL using Omnibus GitLab +The recommended Omnibus GitLab configuration for a PostgreSQL cluster with +replication and failover requires: + +- A minimum of three PostgreSQL nodes. +- A minimum of three Consul server nodes. +- A minimum of three PgBouncer nodes that track and handle primary database reads and writes. + - An [internal load balancer](#configure-the-internal-load-balancer) (TCP) to balance requests between the PgBouncer nodes. +- [Database Load Balancing](../postgresql/database_load_balancing.md) enabled. + + A local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. + The following IPs will be used as an example: - `10.6.0.31`: PostgreSQL primary @@ -837,8 +867,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except Patroni and Consul - roles(['patroni_role']) + # Disable all components except Patroni, PgBouncer and Consul + roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -883,6 +913,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) + # Local PgBouncer service for Database Load Balancing + pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "pgbouncer", + password: '<pgbouncer_password_hash>' + } + } + # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' @@ -943,9 +982,11 @@ If the 'State' column for any node doesn't say "running", check the </a> </div> -## Configure PgBouncer +### Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer +for tracking and handling reads/writes to the primary database. -Now that the PostgreSQL servers are all set up, let's configure PgBouncer. The following IPs will be used as an example: - `10.6.0.21`: PgBouncer 1 @@ -1167,7 +1208,14 @@ There are many third-party solutions for PostgreSQL HA. The solution selected mu - A static IP for all connections that doesn't change on failover. - [`LISTEN`](https://www.postgresql.org/docs/12/sql-listen.html) SQL functionality must be supported. -Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). +NOTE: +With a third-party setup, it's possible to colocate Praefect's database on the same server as +the main [GitLab](#provide-your-own-postgresql-instance) database as a convenience unless +you are using Geo, where separate database instances are required for handling replication correctly. +In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be +minimal. + +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1604,8 +1652,8 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.31', '10.6.0.32', '10.6.0.33'] } # PostgreSQL IPs + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1764,6 +1812,8 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.31', '10.6.0.32', '10.6.0.33'] } # PostgreSQL IPs + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -2068,8 +2118,7 @@ cluster alongside your instance, read how to ## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) -are recommended over NFS wherever possible for improved performance. If you intend -to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). +are recommended over NFS wherever possible for improved performance. See how to [configure NFS](../nfs.md). @@ -2141,7 +2190,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -2153,25 +2202,21 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 5k +skinparam linetype ortho card "Kubernetes via Helm Charts" as kubernetes { card "**External Load Balancer**" as elb #6a9be7 together { - collections "**Webservice** x5" as gitlab #32CD32 - collections "**Sidekiq** x3" as sidekiq #ff8dd1 + collections "**Webservice** x4" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 } - card "**Prometheus + Grafana**" as monitor #7FFFD4 card "**Supporting Services**" as support } card "**Internal Load Balancer**" as ilb #9370DB - -card "**Consul + Sentinel**" as consul_sentinel { - collections "**Consul** x3" as consul #e76a9b - collections "**Redis Sentinel** x3" as sentinel #e6e727 -} +collections "**Consul** x3" as consul #e76a9b card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 @@ -2191,41 +2236,33 @@ card "Database" as database { postgres_primary .[#4EA7FF]> postgres_secondary } -card "Redis" as redis { +card "redis" as redis { collections "**Redis** x3" as redis_nodes #FF6347 - - redis_nodes <.[#FF6347]- sentinel } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]-> monitor +elb -[hidden]-> sidekiq elb -[hidden]-> support gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis -gitlab -[hidden]--> consul +gitlab -[#32CD32]r--> object_storage +gitlab -[#32CD32,norank]----> redis +gitlab -[#32CD32]----> database sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis -sidekiq -[hidden]--> consul - -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +sidekiq -[#ff8dd1]r--> object_storage +sidekiq -[#ff8dd1,norank]----> redis +sidekiq .[#ff8dd1]----> database -consul .[#e76a9b]-> database -consul .[#e76a9b]-> gitaly_cluster -consul .[#e76a9b,norank]--> redis +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden,norank]--> redis -monitor .[#7FFFD4]> consul -monitor .[#7FFFD4]-> database -monitor .[#7FFFD4]-> gitaly_cluster -monitor .[#7FFFD4,norank]--> redis -monitor .[#7FFFD4]> ilb -monitor .[#7FFFD4,norank]u--> elb +consul .[#e76a9b]--> database +consul .[#e76a9b,norank]--> gitaly_cluster +consul .[#e76a9b]--> redis @enduml ``` diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 4d95a61176b..6bf35ba6e22 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -166,7 +166,7 @@ that can also be promoted in case of disaster. ## Deviating from the suggested reference architectures -As a general rule of thumb, the further away you move from the Reference Architectures, +As a general guideline, the further away you move from the Reference Architectures, the harder it will be get support for it. With any deviation, you're introducing a layer of complexity that will add challenges to finding out where potential issues might lie. @@ -191,3 +191,36 @@ The reference architectures for user counts [3,000](3k_users.md) and up support In the specific case you have the requirement to achieve HA but have a lower user count, select modifications to the [3,000 user](3k_users.md) architecture are supported. For more details, [refer to this section in the architecture's documentation](3k_users.md#supported-modifications-for-lower-user-counts-ha). + +## Testing process and results + +The [Quality Engineering - Enablement team](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/) does regular smoke and performance tests for the reference architectures to ensure they remain compliant. + +In this section, we detail some of the process as well as the results. + +Note the following about the testing process: + +- Testing occurs against all main reference architectures and cloud providers in an automated and ad-hoc fashion. + This is achieved through two tools built by the team: + - The [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit) for building the environments. + - The [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) for performance testing. +- Network latency on the test environments between components on all Cloud Providers were measured at <5ms. Note that this is shared as an observation and not as an implicit recommendation. +- We aim to have a "test smart" approach where architectures tested have a good range that can also apply to others. Testing focuses on 10k Omnibus on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids. +- Testing is done publicly and all results are shared. + +Τhe following table details the testing done against the reference architectures along with the frequency and results. + +| Reference Architecture | Tests Run<sup>1</sup> | +|------------------------|----------------------------------------------------------------------------------------------------------------------| +| 1k | [Omnibus - Daily (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)<sup>2</sup> | +| 2k | [Omnibus - Daily (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)<sup>2</sup> | +| 3k | [Omnibus - Weekly (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)<sup>2</sup> | +| 5k | [Omnibus - Weekly (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)<sup>2</sup> | +| 10k | [Omnibus - Daily (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)<sup>2</sup><br/>[Omnibus - Ad-Hoc (GCP, AWS, Azure)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k)<br/><br/>[Cloud Native Hybrid - Ad-Hoc (GCP, AWS)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k-Cloud-Native-Hybrid) | +| 25k | [Omnibus - Weekly (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)<sup>2</sup><br/>[Omnibus - Ad-Hoc (Azure)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/25k) | +| 50k | [Omnibus - Weekly (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)<sup>2</sup><br/>[Omnibus - Ad-Hoc (AWS)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/50k) | + +Note that: + +1. The list above is non exhaustive. Additional testing is continuously evaluated and iterated on, and the table is updated regularly. +1. The Omnibus reference architectures are VM-based only and testing has shown that they perform similarly on equivalently specced hardware regardless of Cloud Provider or if run on premises. diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index aabf4809b4a..c8c13fca59d 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -20,15 +20,14 @@ with the Fog library that GitLab uses. Symptoms include: 411 Length Required ``` -### GitLab Pages requires NFS +### GitLab Pages can use object storage -If you intend to use [GitLab Pages](../../user/project/pages/index.md), this currently requires -[NFS](../nfs.md). There is [work in progress](https://gitlab.com/groups/gitlab-org/-/epics/3901) -to remove this dependency. In the future, GitLab Pages will use -object storage. +If you intend to use [GitLab Pages](../../user/project/pages/index.md), you can +[configure object storage](../pages/index.md#using-object-storage). +NFS is still available if you prefer. -The dependency on disk storage also prevents Pages being deployed using the -[GitLab Helm chart](https://gitlab.com/groups/gitlab-org/-/epics/4283). +The [GitLab Pages Helm chart](https://docs.gitlab.com/charts/charts/gitlab/gitlab-pages/) is also available +for Kubernetes deployments. ### Incremental logging is required for CI to use object storage diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index a85f678fe95..f33d494f638 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -101,10 +101,10 @@ To look up a project's hash path using a Rails console: #### From hashed path to project name -Administrators can look up a project's name from its hashed storage path using: +Administrators can look up a project's name from its hashed storage path using: - A Rails console. -- The `config` file in the `*.git` directory. +- The `config` file in the `*.git` directory. To look up a project's name using the Rails console: diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 388ae74f207..582ffc9dc9c 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -130,6 +130,28 @@ For GitLab 13.8 and earlier versions, you can use a workaround for the Rake task end ``` +You can optionally track progress and verify that all packages migrated successfully using the +[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): + +- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. + +Verify `objectstg` below (where `store=2`) has count of all states: + +```shell +gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM terraform_states; + +total | filesystem | objectstg +------+------------+----------- + 15 | 0 | 15 +``` + +Verify that there are no files on disk in the `terraform_state` folder: + +```shell +sudo find /var/opt/gitlab/gitlab-rails/shared/terraform_state -type f | wc -l +``` + ### S3-compatible connection settings See [the available connection settings for different providers](object_storage.md#connection-settings). @@ -162,11 +184,7 @@ See [the available connection settings for different providers](object_storage.m ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local states to the object storage (GitLab 13.9 and later): - - ```shell - gitlab-rake gitlab:terraform_states:migrate - ``` +1. [Migrate any existing local states to the object storage](#migrate-to-object-storage) **In installations from source:** @@ -187,8 +205,4 @@ See [the available connection settings for different providers](object_storage.m ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -1. Migrate any existing local states to the object storage (GitLab 13.9 and later): - - ```shell - sudo -u git -H bundle exec rake gitlab:terraform_states:migrate RAILS_ENV=production - ``` +1. [Migrate any existing local states to the object storage](#migrate-to-object-storage) diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index cfce3b94554..c45938ecd3f 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -4,7 +4,7 @@ group: Global Search info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Troubleshooting Elasticsearch +# Troubleshooting Elasticsearch **(PREMIUM SELF)** To install and configure Elasticsearch, and for common and known issues, visit the [administrator documentation](../../integration/elasticsearch.md). diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 87f514a2fdd..ccfa93d9bc8 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -97,14 +97,15 @@ Rails.cache.instance_variable_get(:@data).keys ## Profile a page ```ruby +url = '<url/of/the/page>' + # Before 11.6.0 logger = Logger.new($stdout) -admin_token = User.find_by_username('ADMIN_USERNAME').personal_access_tokens.first.token -app.get("URL/?private_token=#{admin_token}") +admin_token = User.find_by_username('<admin-username>').personal_access_tokens.first.token +app.get("#{url}/?private_token=#{admin_token}") # From 11.6.0 -admin = User.find_by_username('ADMIN_USERNAME') -url = "/url/goes/here" +admin = User.find_by_username('<admin-username>') Gitlab::Profiler.with_user(admin) { app.get(url) } ``` @@ -112,8 +113,8 @@ Gitlab::Profiler.with_user(admin) { app.get(url) } ```ruby logger = Logger.new($stdout) -admin = User.find_by_username('ADMIN_USERNAME') -Gitlab::Profiler.profile('URL', logger: logger, user: admin) +admin = User.find_by_username('<admin-username>') +Gitlab::Profiler.profile('<url/of/the/page>', logger: logger, user: admin) ``` ## Time an operation @@ -414,12 +415,14 @@ p.create_wiki ### creates the wiki project on the filesystem ### In case of issue boards not loading properly and it's getting time out. We need to call the Issue Rebalancing service to fix this ```ruby -p = Project.find_by_full_path('PROJECT PATH') +p = Project.find_by_full_path('<username-or-group>/<project-name>') Issues::RelativePositionRebalancingService.new(p.root_namespace.all_projects).execute ``` -## Imports / Exports +## Imports and exports + +### Import a project ```ruby # Find the project and get the error @@ -462,18 +465,19 @@ Clear the cache: sudo gitlab-rake cache:clear ``` -### Export a repository +### Export a project It's typically recommended to export a project through [the web interface](../../user/project/settings/import_export.md#export-a-project-and-its-data) or through [the API](../../api/project_import_export.md). In situations where this is not working as expected, it may be preferable to export a project directly via the Rails console: ```ruby -user = User.find_by_username('USERNAME') -project = Project.find_by_full_path('PROJECT_PATH') +user = User.find_by_username('<username>') +# Sufficient permissions needed +# Read https://docs.gitlab.com/ee/user/permissions.html#project-members-permissions + +project = Project.find_by_full_path('<username-or-group>/<project-name') Projects::ImportExport::ExportService.new(project, user).execute ``` -If the project you wish to export is available at `https://gitlab.example.com/baltig/pipeline-templates`, the value to use for `PROJECT_PATH` would be `baltig/pipeline-templates`. - If this all runs successfully, you see an output like the following before being returned to the Rails console prompt: ```ruby @@ -482,6 +486,11 @@ If this all runs successfully, you see an output like the following before being The exported project is located within a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. +If this fails, [enable verbose logging](navigating_gitlab_via_rails_console.md#looking-up-database-persisted-objects), +repeat the above procedure after, +and report the output to +[GitLab Support](https://about.gitlab.com/support/). + ## Repository ### Search sequence of pushes to a repository @@ -586,7 +595,7 @@ User.active.count User.billable.count # The historical max on the instance as of the past year -::HistoricalData.max_historical_user_count +::HistoricalData.max_historical_user_count(from: 1.year.ago.beginning_of_day, to: Time.current.end_of_day) ``` Using cURL and jq (up to a max 100, see the [pagination docs](../../api/index.md#pagination)): @@ -618,7 +627,7 @@ users.count # If that count looks sane: # You can either block the users: -users.each { |user| user.block! } +users.each { |user| user.blocked? ? nil : user.block! } # Or you can delete them: # need 'current user' (your user) for auditing purposes @@ -782,7 +791,7 @@ end emails = [email1, email2] emails.each do |e| - delete_bad_scim(e,'GROUPPATH') + delete_bad_scim(e,'<group-path>') end ``` @@ -815,28 +824,28 @@ conflicting_permanent_redirects.destroy_all ### Close a merge request properly (if merged but still marked as open) ```ruby -p = Project.find_by_full_path('<full/path/to/project>') -m = p.merge_requests.find_by(iid: <iid>) u = User.find_by_username('<username>') -MergeRequests::PostMergeService.new(p, u).execute(m) +p = Project.find_by_full_path('<namespace/project>') +m = p.merge_requests.find_by(iid: <iid>) +MergeRequests::PostMergeService.new(project: p, current_user: u).execute(m) ``` ### Delete a merge request ```ruby u = User.find_by_username('<username>') -p = Project.find_by_full_path('<group>/<project>') -m = p.merge_requests.find_by(iid: <IID>) -Issuable::DestroyService.new(m.project, u).execute(m) +p = Project.find_by_full_path('<namespace/project>') +m = p.merge_requests.find_by(iid: <iid>) +Issuable::DestroyService.new(project: m.project, current_user: u).execute(m) ``` ### Rebase manually ```ruby -p = Project.find_by_full_path('<project_path>') -m = project.merge_requests.find_by(iid: ) u = User.find_by_username('<username>') -MergeRequests::RebaseService.new(m.target_project, u).execute(m) +p = Project.find_by_full_path('<namespace/project>') +m = p.merge_requests.find_by(iid: <iid>) +MergeRequests::RebaseService.new(project: m.target_project, current_user: u).execute(m) ``` ## CI @@ -1255,6 +1264,9 @@ registry.replicator.send(:sync_repository) ## Generate Service Ping +The [Service Ping Guide](../../development/service_ping/index.md) in our developer documentation +has more information about Service Ping. + ### Generate or get the cached Service Ping ```ruby @@ -1277,6 +1289,12 @@ Generates Service Ping data in JSON format. rake gitlab:usage_data:generate ``` +Generates Service Ping data in YAML format: + +```shell +rake gitlab:usage_data:dump_sql_in_yaml +``` + ### Generate and send Service Ping Prints the metrics saved in `conversational_development_index_metrics`. diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index 9e9ef492ebd..d052688363c 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -72,6 +72,10 @@ Self-managed instance example:  +Setting the username for the newly provisioned users when assigning them the SCIM app: + + + ## OneLogin Application details: diff --git a/doc/administration/troubleshooting/img/okta_setting_username.png b/doc/administration/troubleshooting/img/okta_setting_username.png Binary files differnew file mode 100644 index 00000000000..c413b9d3a27 --- /dev/null +++ b/doc/administration/troubleshooting/img/okta_setting_username.png diff --git a/doc/administration/troubleshooting/img/sidekiq_flamegraph.png b/doc/administration/troubleshooting/img/sidekiq_flamegraph.png Binary files differnew file mode 100644 index 00000000000..89d6e8da3ce --- /dev/null +++ b/doc/administration/troubleshooting/img/sidekiq_flamegraph.png diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index 57d64a2323e..91db321295d 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w At the heart of GitLab is a web application [built using the Ruby on Rails framework](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build-gitlab/). Thanks to this, we also get access to the amazing tools built right into Rails. -In this guide, we'll introduce the [Rails console](../operations/rails_console.md#starting-a-rails-console-session) +This guide introduces the [Rails console](../operations/rails_console.md#starting-a-rails-console-session) and the basics of interacting with your GitLab instance from the command line. WARNING: @@ -19,7 +19,7 @@ or destroying production data. If you would like to explore the Rails console with no consequences, you are strongly advised to do so in a test environment. This guide is targeted at GitLab system administrators who are troubleshooting -a problem or need to retrieve some data that can only be done through direct +a problem or must retrieve some data that can only be done through direct access of the GitLab application. Basic knowledge of Ruby is needed (try [this 30-minute tutorial](https://try.ruby-lang.org/) for a quick introduction). Rails experience is helpful to have but not a must. @@ -29,7 +29,7 @@ Rails experience is helpful to have but not a must. Your type of GitLab installation determines how [to start a rails console](../operations/rails_console.md). -The following code examples will all take place inside the Rails console and also +The following code examples take place inside the Rails console and also assume an Omnibus GitLab installation. ## Active Record objects @@ -37,7 +37,7 @@ assume an Omnibus GitLab installation. ### Looking up database-persisted objects Under the hood, Rails uses [Active Record](https://guides.rubyonrails.org/active_record_basics.html), -an object-relational mapping system, to read, write and map application objects +an object-relational mapping system, to read, write, and map application objects to the PostgreSQL database. These mappings are handled by Active Record models, which are Ruby classes defined in a Rails app. For GitLab, the model classes can be found at `/opt/gitlab/embedded/service/gitlab-rails/app/models`. @@ -144,7 +144,7 @@ NoMethodError (undefined method `username' for #<ActiveRecord::Relation [#<User Did you mean? by_username ``` -We need to retrieve the single object from the collection by using the `.first` +Let's retrieve the single object from the collection by using the `.first` method to get the first item in the collection: ```ruby @@ -164,7 +164,7 @@ Record, please see the [Active Record Query Interface documentation](https://gui ### Modifying Active Record objects In the previous section, we learned about retrieving database records using -Active Record. Now, we'll learn how to write changes to the database. +Active Record. Now, let's learn how to write changes to the database. First, let's retrieve the `root` user: @@ -195,7 +195,7 @@ a background job to deliver an email notification. This is an example of an -- code which is designated to run in response to events in the Active Record object life cycle. This is also why using the Rails console is preferred when direct changes to data is necessary as changes made via direct database queries -will not trigger these callbacks. +do not trigger these callbacks. It's also possible to update attributes in a single line: @@ -265,8 +265,8 @@ user.save!(validate: false) This is not recommended, as validations are usually put in place to ensure the integrity and consistency of user-provided data. -A validation error will prevent the entire object from being saved to -the database. We'll see a little of this in the next section. If you're getting +A validation error prevents the entire object from being saved to +the database. You can see a little of this in the section below. If you're getting a mysterious red banner in the GitLab UI when submitting a form, this can often be the fastest way to get to the root of the problem. @@ -336,7 +336,7 @@ user.activate user.state ``` -Earlier, we mentioned that a validation error will prevent the entire object +Earlier, we mentioned that a validation error prevents the entire object from being saved to the database. Let's see how this can have unexpected interactions: @@ -455,7 +455,7 @@ Ci::Build.find(66124) ``` The pipeline and job ID numbers increment globally across your GitLab -instance, so there's no need to use an internal ID attribute to look them up, +instance, so there's no requirement to use an internal ID attribute to look them up, unlike with issues or merge requests. **Get the current application settings object:** diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 7a8ac8c3dbe..a606a3712ba 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -85,6 +85,27 @@ several `WARN` level messages. Here's an example of a single thread's backtrace: In some cases Sidekiq may be hung and unable to respond to the `TTIN` signal. Move on to other troubleshooting methods if this happens. +## Ruby profiling with `rbspy` + +[rbspy](https://rbspy.github.io) is an easy to use and low-overhead Ruby profiler that can be used to create +flamegraph-style diagrams of CPU usage by Ruby processes. + +No changes to GitLab are required to use it and it has no dependencies. To install it: + +1. Download the binary from the [`rbspy` releases page](https://github.com/rbspy/rbspy/releases). +1. Make the binary executable. + +To profile a Sidekiq worker for one minute, run: + +```shell +sudo ./rbspy record --pid <sidekiq_pid> --duration 60 --file /tmp/sidekiq_profile.svg +``` + + + +In this example of a flamegraph generated by `rbspy`, almost all of the Sidekiq process's time is spent in `rev_parse`, a native C +function in Rugged. In the stack, we can see `rev_parse` is being called by the `ExpirePipelineCacheWorker`. + ## Process profiling with `perf` Linux has a process profiling tool called `perf` that is helpful when a certain @@ -358,3 +379,17 @@ has number of drawbacks, as mentioned in [Why Ruby's Timeout is dangerous (and T > - in any of your code, regardless of whether it could have possibly raised an exception before > > Nobody writes code to defend against an exception being raised on literally any line. That's not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that's unlikely :) + +## Disable Rugged + +Calls into Rugged, Ruby bindings for `libgit2`, [lock the Sidekiq processes's GVL](https://silverhammermba.github.io/emberb/c/#c-in-ruby-threads), +blocking all jobs on that worker from proceeding. If Rugged calls performed by Sidekiq are slow, this can cause significant delays in +background task processing. + +By default, Rugged is used when Git repository data is stored on local storage or on an NFS mount. +[Using Rugged is recommened when using NFS](../nfs.md#improving-nfs-performance-with-gitlab), but if +you are using local storage, disabling Rugged can improve Sidekiq performance: + +```shell +sudo gitlab-rake gitlab:features:disable_rugged +``` diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 3bafbed4b3f..3a0c6a30cde 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -2,13 +2,12 @@ stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Finding relevant log entries with a correlation ID **(FREE SELF)** -In GitLab 11.6 and later, a unique request tracking ID, known as the "correlation ID" has been -logged by the GitLab instance for most requests. Each individual request to GitLab gets +GitLab instances log a unique request tracking ID (known as the +"correlation ID") for most requests. Each individual request to GitLab gets its own correlation ID, which then gets logged in each GitLab component's logs for that request. This makes it easier to trace behavior in a distributed system. Without this ID it can be difficult or @@ -147,7 +146,7 @@ First, enable the **Developer Tools** panel. See [Getting the correlation ID in After developer tools have been enabled, obtain a session cookie as follows: 1. Visit <https://gitlab.com> while logged in. -1. (Optional) Select **Fetch/XHR** request filter in the **Developer Tools** panel. This step is described for Google Chrome developer tools and is not strictly necessary, it just makes it easier to find the correct request. +1. Optional. Select **Fetch/XHR** request filter in the **Developer Tools** panel. This step is described for Google Chrome developer tools and is not strictly necessary, it just makes it easier to find the correct request. 1. Select the `results?request_id=<some-request-id>` request on the left hand side. 1. The session cookie is displayed under the `Request Headers` section of the `Headers` panel. Right-click on the cookie value and select `Copy value`. diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 15ef024647c..55c3f85bfb9 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -51,12 +51,6 @@ _The uploads are stored by default in ## Using object storage **(FREE SELF)** -> **Notes:** -> -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3867) in GitLab 10.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17358) from GitLab Premium to GitLab Free in 10.7. -> - Since version 11.1, we support direct_upload to S3. - If you don't want to use the local disk where GitLab is installed to store the uploads, you can use an object storage provider like AWS S3 instead. This configuration relies on valid AWS credentials to be configured already. @@ -112,24 +106,7 @@ _The uploads are stored by default in ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). -1. Optional: Verify all files migrated properly. - From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) - (`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `store=2`) has count of all artifacts: - - ```shell - gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads; - - total | filesystem | objectstg - ------+------------+----------- - 2409 | 0 | 2409 - ``` - - Verify no files on disk in `artifacts` folder: - - ```shell - sudo find /var/opt/gitlab/gitlab-rails/uploads -type f | grep -v tmp | wc -l - ``` +1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). **In installations from source:** @@ -153,22 +130,6 @@ _The uploads are stored by default in 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). -1. Optional: Verify all files migrated properly. - From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: - - ```shell - gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads; - - total | filesystem | objectstg - ------+------------+----------- - 2409 | 0 | 2409 - ``` - - Verify no files on disk in `artifacts` folder: - - ```shell - sudo find /var/opt/gitlab/gitlab-rails/uploads -type f | grep -v tmp | wc -l - ``` #### OpenStack example @@ -195,23 +156,6 @@ _The uploads are stored by default in 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). -1. Optional: Verify all files migrated properly. - From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) - (`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `store=2`) has count of all artifacts: - - ```shell - gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads; - - total | filesystem | objectstg - ------+------------+----------- - 2409 | 0 | 2409 - ``` - - Verify no files on disk in `artifacts` folder: - - ```shell - sudo find /var/opt/gitlab/gitlab-rails/uploads -type f | grep -v tmp | wc -l - ``` --- @@ -243,19 +187,3 @@ _The uploads are stored by default in 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). -1. Optional: Verify all files migrated properly. - From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: - - ```shell - gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads; - - total | filesystem | objectstg - ------+------------+----------- - 2409 | 0 | 2409 - ``` - - Verify no files on disk in `artifacts` folder: - - ```shell - sudo find /var/opt/gitlab/gitlab-rails/uploads -type f | grep -v tmp | wc -l - ``` diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md index d496ecbca5b..f489cb780ef 100644 --- a/doc/api/api_resources.md +++ b/doc/api/api_resources.md @@ -22,160 +22,159 @@ See also: The following API resources are available in the project context: -| Resource | Available endpoints | -|:--------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Access requests](access_requests.md) | `/projects/:id/access_requests` (also available for groups) | -| [Access tokens](resource_access_tokens.md) | `/projects/:id/access_tokens` | -| [Award emoji](award_emoji.md) | `/projects/:id/issues/.../award_emoji`, `/projects/:id/merge_requests/.../award_emoji`, `/projects/:id/snippets/.../award_emoji` | -| [Branches](branches.md) | `/projects/:id/repository/branches/`, `/projects/:id/repository/merged_branches` | -| [Commits](commits.md) | `/projects/:id/repository/commits`, `/projects/:id/statuses` | -| [Container Registry](container_registry.md) | `/projects/:id/registry/repositories` | -| [Custom attributes](custom_attributes.md) | `/projects/:id/custom_attributes` (also available for groups and users) | -| [Dependencies](dependencies.md) **(ULTIMATE)** | `/projects/:id/dependencies` | -| [Deploy keys](deploy_keys.md) | `/projects/:id/deploy_keys` (also available standalone) | -| [Freeze Periods](freeze_periods.md) | `/projects/:id/freeze_periods` | -| [Debian distributions](packages/debian_project_distributions.md) | `/projects/:id/debian_distributions` (also available for groups) | -| [Deployments](deployments.md) | `/projects/:id/deployments` | -| [Discussions](discussions.md) (threaded comments) | `/projects/:id/issues/.../discussions`, `/projects/:id/snippets/.../discussions`, `/projects/:id/merge_requests/.../discussions`, `/projects/:id/commits/.../discussions` (also available for groups) | -| [Environments](environments.md) | `/projects/:id/environments` | -| [Error Tracking](error_tracking.md) | `/projects/:id/error_tracking/settings` | -| [Events](events.md) | `/projects/:id/events` (also available for users and standalone) | -| [Feature Flags](feature_flags.md) | `/projects/:id/feature_flags` | -| [Feature Flag User Lists](feature_flag_user_lists.md) | `/projects/:id/feature_flags_user_lists` | -| [Integrations](integrations.md) | `/projects/:id/integrations` | -| [Invitations](invitations.md) | `/projects/:id/invitations` (also available for groups) | -| [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | -| [Issues Statistics](issues_statistics.md) | `/projects/:id/issues_statistics` (also available for groups and standalone) | -| [Issue boards](boards.md) | `/projects/:id/boards` | -| [Issue links](issue_links.md). | `/projects/:id/issues/.../links` | -| [Iterations](iterations.md) **(PREMIUM)** | `/projects/:id/iterations` (also available for groups) | -| [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | -| [Labels](labels.md) | `/projects/:id/labels` | -| [Managed licenses](managed_licenses.md) **(ULTIMATE)** | `/projects/:id/managed_licenses` | -| [Members](members.md) | `/projects/:id/members` (also available for groups) | -| [Merge request approvals](merge_request_approvals.md) **(PREMIUM)** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | -| [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | -| [Merge trains](merge_trains.md) | `/projects/:id/merge_trains` | -| [Notes](notes.md) (comments) | `/projects/:id/issues/.../notes`, `/projects/:id/snippets/.../notes`, `/projects/:id/merge_requests/.../notes` (also available for groups) | -| [Notification settings](notification_settings.md) | `/projects/:id/notification_settings` (also available for groups and standalone) | -| [Packages](packages.md) | `/projects/:id/packages` | -| [Pages domains](pages_domains.md) | `/projects/:id/pages` (also available standalone) | -| [Pipelines](pipelines.md) | `/projects/:id/pipelines` | -| [Pipeline schedules](pipeline_schedules.md) | `/projects/:id/pipeline_schedules` | -| [Pipeline triggers](pipeline_triggers.md) | `/projects/:id/triggers` | -| [Projects](projects.md) including setting Webhooks | `/projects`, `/projects/:id/hooks` (also available for users) | -| [Project badges](project_badges.md) | `/projects/:id/badges` | -| [Project clusters](project_clusters.md) | `/projects/:id/clusters` | -| [Project-level variables](project_level_variables.md) | `/projects/:id/variables` | -| [Project import/export](project_import_export.md) | `/projects/:id/export`, `/projects/import`, `/projects/:id/import` | -| [Project milestones](milestones.md) | `/projects/:id/milestones` | -| [Project snippets](project_snippets.md) | `/projects/:id/snippets` | -| [Project templates](project_templates.md) | `/projects/:id/templates` | -| [Project vulnerabilities](project_vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/templates` | -| [Protected environments](protected_environments.md) | `/projects/:id/protected_environments` | -| [Protected branches](protected_branches.md) | `/projects/:id/protected_branches` | -| [Protected tags](protected_tags.md) | `/projects/:id/protected_tags` | -| [Releases](releases/index.md) | `/projects/:id/releases` | -| [Release links](releases/links.md) | `/projects/:id/releases/.../assets/links` | -| [Remote mirrors](remote_mirrors.md) | `/projects/:id/remote_mirrors` | -| [Repositories](repositories.md) | `/projects/:id/repository` | -| [Repository files](repository_files.md) | `/projects/:id/repository/files` | -| [Repository submodules](repository_submodules.md) | `/projects/:id/repository/submodules` | -| [Resource label events](resource_label_events.md) | `/projects/:id/issues/.../resource_label_events`, `/projects/:id/merge_requests/.../resource_label_events` (also available for groups) | -| [Runners](runners.md) | `/projects/:id/runners` (also available standalone) | -| [Search](search.md) | `/projects/:id/search` (also available for groups and standalone) | -| [Services](services.md) (renamed to [Integrations](integrations.md)) | `/projects/:id/services` | -| [Tags](tags.md) | `/projects/:id/repository/tags` | -| [User-starred metrics dashboards](metrics_user_starred_dashboards.md ) | `/projects/:id/metrics/user_starred_dashboards` | -| [Visual Review discussions](visual_review_discussions.md) **(PREMIUM)** | `/projects/:id/merge_requests/:merge_request_id/visual_review_discussions` | -| [Vulnerabilities](vulnerabilities.md) **(ULTIMATE)** | `/vulnerabilities/:id` | -| [Vulnerability exports](vulnerability_exports.md) **(ULTIMATE)** | `/projects/:id/vulnerability_exports` | -| [Vulnerability findings](vulnerability_findings.md) **(ULTIMATE)** | `/projects/:id/vulnerability_findings` | -| [Project wikis](wikis.md) | `/projects/:id/wikis` | +| Resource | Available endpoints | +|:------------------------------------------------------------------------|:--------------------| +| [Access requests](access_requests.md) | `/projects/:id/access_requests` (also available for groups) | +| [Access tokens](resource_access_tokens.md) | `/projects/:id/access_tokens` | +| [Award emoji](award_emoji.md) | `/projects/:id/issues/.../award_emoji`, `/projects/:id/merge_requests/.../award_emoji`, `/projects/:id/snippets/.../award_emoji` | +| [Branches](branches.md) | `/projects/:id/repository/branches/`, `/projects/:id/repository/merged_branches` | +| [Commits](commits.md) | `/projects/:id/repository/commits`, `/projects/:id/statuses` | +| [Container Registry](container_registry.md) | `/projects/:id/registry/repositories` | +| [Custom attributes](custom_attributes.md) | `/projects/:id/custom_attributes` (also available for groups and users) | +| [Debian distributions](packages/debian_project_distributions.md) | `/projects/:id/debian_distributions` (also available for groups) | +| [Dependencies](dependencies.md) **(ULTIMATE)** | `/projects/:id/dependencies` | +| [Deploy keys](deploy_keys.md) | `/projects/:id/deploy_keys` (also available standalone) | +| [Deployments](deployments.md) | `/projects/:id/deployments` | +| [Discussions](discussions.md) (threaded comments) | `/projects/:id/issues/.../discussions`, `/projects/:id/snippets/.../discussions`, `/projects/:id/merge_requests/.../discussions`, `/projects/:id/commits/.../discussions` (also available for groups) | +| [Environments](environments.md) | `/projects/:id/environments` | +| [Error Tracking](error_tracking.md) | `/projects/:id/error_tracking/settings` | +| [Events](events.md) | `/projects/:id/events` (also available for users and standalone) | +| [Feature Flag User Lists](feature_flag_user_lists.md) | `/projects/:id/feature_flags_user_lists` | +| [Feature Flags](feature_flags.md) | `/projects/:id/feature_flags` | +| [Freeze Periods](freeze_periods.md) | `/projects/:id/freeze_periods` | +| [Integrations](integrations.md) (Formerly "services") | `/projects/:id/integrations` | +| [Invitations](invitations.md) | `/projects/:id/invitations` (also available for groups) | +| [Issue boards](boards.md) | `/projects/:id/boards` | +| [Issue links](issue_links.md) | `/projects/:id/issues/.../links` | +| [Issues Statistics](issues_statistics.md) | `/projects/:id/issues_statistics` (also available for groups and standalone) | +| [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | +| [Iterations](iterations.md) **(PREMIUM)** | `/projects/:id/iterations` (also available for groups) | +| [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | +| [Labels](labels.md) | `/projects/:id/labels` | +| [Managed licenses](managed_licenses.md) **(ULTIMATE)** | `/projects/:id/managed_licenses` | +| [Members](members.md) | `/projects/:id/members` (also available for groups) | +| [Merge request approvals](merge_request_approvals.md) **(PREMIUM)** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | +| [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | +| [Merge trains](merge_trains.md) | `/projects/:id/merge_trains` | +| [Notes](notes.md) (comments) | `/projects/:id/issues/.../notes`, `/projects/:id/snippets/.../notes`, `/projects/:id/merge_requests/.../notes` (also available for groups) | +| [Notification settings](notification_settings.md) | `/projects/:id/notification_settings` (also available for groups and standalone) | +| [Packages](packages.md) | `/projects/:id/packages` | +| [Pages domains](pages_domains.md) | `/projects/:id/pages` (also available standalone) | +| [Pipeline schedules](pipeline_schedules.md) | `/projects/:id/pipeline_schedules` | +| [Pipeline triggers](pipeline_triggers.md) | `/projects/:id/triggers` | +| [Pipelines](pipelines.md) | `/projects/:id/pipelines` | +| [Project badges](project_badges.md) | `/projects/:id/badges` | +| [Project clusters](project_clusters.md) | `/projects/:id/clusters` | +| [Project import/export](project_import_export.md) | `/projects/:id/export`, `/projects/import`, `/projects/:id/import` | +| [Project milestones](milestones.md) | `/projects/:id/milestones` | +| [Project snippets](project_snippets.md) | `/projects/:id/snippets` | +| [Project templates](project_templates.md) | `/projects/:id/templates` | +| [Project vulnerabilities](project_vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/templates` | +| [Project wikis](wikis.md) | `/projects/:id/wikis` | +| [Project-level variables](project_level_variables.md) | `/projects/:id/variables` | +| [Projects](projects.md) including setting Webhooks | `/projects`, `/projects/:id/hooks` (also available for users) | +| [Protected branches](protected_branches.md) | `/projects/:id/protected_branches` | +| [Protected environments](protected_environments.md) | `/projects/:id/protected_environments` | +| [Protected tags](protected_tags.md) | `/projects/:id/protected_tags` | +| [Release links](releases/links.md) | `/projects/:id/releases/.../assets/links` | +| [Releases](releases/index.md) | `/projects/:id/releases` | +| [Remote mirrors](remote_mirrors.md) | `/projects/:id/remote_mirrors` | +| [Repositories](repositories.md) | `/projects/:id/repository` | +| [Repository files](repository_files.md) | `/projects/:id/repository/files` | +| [Repository submodules](repository_submodules.md) | `/projects/:id/repository/submodules` | +| [Resource label events](resource_label_events.md) | `/projects/:id/issues/.../resource_label_events`, `/projects/:id/merge_requests/.../resource_label_events` (also available for groups) | +| [Runners](runners.md) | `/projects/:id/runners` (also available standalone) | +| [Search](search.md) | `/projects/:id/search` (also available for groups and standalone) | +| [Tags](tags.md) | `/projects/:id/repository/tags` | +| [User-starred metrics dashboards](metrics_user_starred_dashboards.md ) | `/projects/:id/metrics/user_starred_dashboards` | +| [Visual Review discussions](visual_review_discussions.md) **(PREMIUM)** | `/projects/:id/merge_requests/:merge_request_id/visual_review_discussions` | +| [Vulnerabilities](vulnerabilities.md) **(ULTIMATE)** | `/vulnerabilities/:id` | +| [Vulnerability exports](vulnerability_exports.md) **(ULTIMATE)** | `/projects/:id/vulnerability_exports` | +| [Vulnerability findings](vulnerability_findings.md) **(ULTIMATE)** | `/projects/:id/vulnerability_findings` | ## Group resources The following API resources are available in the group context: -| Resource | Available endpoints | -|:-----------------------------------------------------------------|:---------------------------------------------------------------------------------| -| [Access requests](access_requests.md) | `/groups/:id/access_requests/` (also available for projects) | -| [Custom attributes](custom_attributes.md) | `/groups/:id/custom_attributes` (also available for projects and users) | -| [Debian distributions](packages/debian_group_distributions.md) | `/groups/:id/-/packages/debian` (also available for projects) | -| [Discussions](discussions.md) (threaded comments) **(ULTIMATE)** | `/groups/:id/epics/.../discussions` (also available for projects) | -| [Epic issues](epic_issues.md) **(ULTIMATE)** | `/groups/:id/epics/.../issues` | -| [Epic links](epic_links.md) **(ULTIMATE)** | `/groups/:id/epics/.../epics` | -| [Epics](epics.md) **(ULTIMATE)** | `/groups/:id/epics` | -| [Groups](groups.md) | `/groups`, `/groups/.../subgroups` | -| [Group badges](group_badges.md) | `/groups/:id/badges` | -| [Group issue boards](group_boards.md) | `/groups/:id/boards` | -| [Group iterations](group_iterations.md) **(PREMIUM)** | `/groups/:id/iterations` (also available for projects) | -| [Group labels](group_labels.md) | `/groups/:id/labels` | -| [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | -| [Group milestones](group_milestones.md) | `/groups/:id/milestones` | -| [Invitations](invitations.md) | `/groups/:id/invitations` (also available for projects) | -| [Issues](issues.md) | `/groups/:id/issues` (also available for projects and standalone) | -| [Issues Statistics](issues_statistics.md) | `/groups/:id/issues_statistics` (also available for projects and standalone) | -| [Members](members.md) | `/groups/:id/members` (also available for projects) | -| [Merge requests](merge_requests.md) | `/groups/:id/merge_requests` (also available for projects and standalone) | -| [Notes](notes.md) (comments) | `/groups/:id/epics/.../notes` (also available for projects) | +| Resource | Available endpoints | +|:-----------------------------------------------------------------|:--------------------| +| [Access requests](access_requests.md) | `/groups/:id/access_requests/` (also available for projects) | +| [Custom attributes](custom_attributes.md) | `/groups/:id/custom_attributes` (also available for projects and users) | +| [Debian distributions](packages/debian_group_distributions.md) | `/groups/:id/-/packages/debian` (also available for projects) | +| [Discussions](discussions.md) (threaded comments) **(ULTIMATE)** | `/groups/:id/epics/.../discussions` (also available for projects) | +| [Epic issues](epic_issues.md) **(ULTIMATE)** | `/groups/:id/epics/.../issues` | +| [Epic links](epic_links.md) **(ULTIMATE)** | `/groups/:id/epics/.../epics` | +| [Epics](epics.md) **(ULTIMATE)** | `/groups/:id/epics` | +| [Groups](groups.md) | `/groups`, `/groups/.../subgroups` | +| [Group badges](group_badges.md) | `/groups/:id/badges` | +| [Group issue boards](group_boards.md) | `/groups/:id/boards` | +| [Group iterations](group_iterations.md) **(PREMIUM)** | `/groups/:id/iterations` (also available for projects) | +| [Group labels](group_labels.md) | `/groups/:id/labels` | +| [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | +| [Group milestones](group_milestones.md) | `/groups/:id/milestones` | +| [Group wikis](group_wikis.md) **(PREMIUM)** | `/groups/:id/wikis` | +| [Invitations](invitations.md) | `/groups/:id/invitations` (also available for projects) | +| [Issues](issues.md) | `/groups/:id/issues` (also available for projects and standalone) | +| [Issues Statistics](issues_statistics.md) | `/groups/:id/issues_statistics` (also available for projects and standalone) | +| [Members](members.md) | `/groups/:id/members` (also available for projects) | +| [Merge requests](merge_requests.md) | `/groups/:id/merge_requests` (also available for projects and standalone) | +| [Notes](notes.md) (comments) | `/groups/:id/epics/.../notes` (also available for projects) | | [Notification settings](notification_settings.md) | `/groups/:id/notification_settings` (also available for projects and standalone) | -| [Resource label events](resource_label_events.md) | `/groups/:id/epics/.../resource_label_events` (also available for projects) | -| [Search](search.md) | `/groups/:id/search` (also available for projects and standalone) | -| [Group wikis](group_wikis.md) **(PREMIUM)** | `/groups/:id/wikis` | +| [Resource label events](resource_label_events.md) | `/groups/:id/epics/.../resource_label_events` (also available for projects) | +| [Search](search.md) | `/groups/:id/search` (also available for projects and standalone) | ## Standalone resources The following API resources are available outside of project and group contexts (including `/users`): -| Resource | Available endpoints | -|:---------------------------------------------------|:------------------------------------------------------------------------| -| [Instance-level CI/CD variables](instance_level_ci_variables.md) **(FREE SELF)** | `/admin/ci/variables` | -| [Sidekiq queues administration](admin_sidekiq_queues.md) **(FREE SELF)** | `/admin/sidekiq/queues/:queue_name` | -| [Appearance](appearance.md) **(FREE SELF)** | `/application/appearance` | -| [Applications](applications.md) | `/applications` | -| [Audit Events](audit_events.md) **(PREMIUM SELF)** | `/audit_events` | -| [Avatar](avatar.md) | `/avatar` | -| [Broadcast messages](broadcast_messages.md) | `/broadcast_messages` | -| [Code snippets](snippets.md) | `/snippets` | -| [Custom attributes](custom_attributes.md) | `/users/:id/custom_attributes` (also available for groups and projects) | -| [Deploy keys](deploy_keys.md) | `/deploy_keys` (also available for projects) | -| [Events](events.md) | `/events`, `/users/:id/events` (also available for projects) | -| [Feature flags](features.md) | `/features` | -| [Geo Nodes](geo_nodes.md) **(PREMIUM SELF)** | `/geo_nodes` | -| [Group Activity Analytics](group_activity_analytics.md) | `/analytics/group_activity/{issues_count | merge_requests_count | new_members_count }` | -| [Group repository storage moves](group_repository_storage_moves.md) **(PREMIUM SELF)** | `/group_repository_storage_moves` | -| [Import repository from GitHub](import.md) | `/import/github` | -| [Instance clusters](instance_clusters.md) **(FREE SELF)** | `/admin/clusters` | -| [Issues](issues.md) | `/issues` (also available for groups and projects) | -| [Issues Statistics](issues_statistics.md) | `/issues_statistics` (also available for groups and projects) | -| [Jobs](jobs.md) | `/job` | -| [Keys](keys.md) | `/keys` | -| [License](license.md) **(FREE SELF)** | `/license` | -| [Markdown](markdown.md) | `/markdown` | -| [Merge requests](merge_requests.md) | `/merge_requests` (also available for groups and projects) | -| [Metrics dashboard annotations](metrics_dashboard_annotations.md) | `/environments/:id/metrics_dashboard/annotations`, `/clusters/:id/metrics_dashboard/annotations` | -| [Namespaces](namespaces.md) | `/namespaces` | -| [Notification settings](notification_settings.md) | `/notification_settings` (also available for groups and projects) | -| [Pages domains](pages_domains.md) | `/pages/domains` (also available for projects) | -| [Plan limits](plan_limits.md) | `/application/plan_limits` | -| [Personal access tokens](personal_access_tokens.md) | `/personal_access_tokens` | -| [Projects](projects.md) | `/users/:id/projects` (also available for projects) | +| Resource | Available endpoints | +|:----------------------------------------------------------------------------------------|:--------------------| +| [Appearance](appearance.md) **(FREE SELF)** | `/application/appearance` | +| [Applications](applications.md) | `/applications` | +| [Audit Events](audit_events.md) **(PREMIUM SELF)** | `/audit_events` | +| [Avatar](avatar.md) | `/avatar` | +| [Broadcast messages](broadcast_messages.md) | `/broadcast_messages` | +| [Code snippets](snippets.md) | `/snippets` | +| [Custom attributes](custom_attributes.md) | `/users/:id/custom_attributes` (also available for groups and projects) | +| [Deploy keys](deploy_keys.md) | `/deploy_keys` (also available for projects) | +| [Events](events.md) | `/events`, `/users/:id/events` (also available for projects) | +| [Feature flags](features.md) | `/features` | +| [Geo Nodes](geo_nodes.md) **(PREMIUM SELF)** | `/geo_nodes` | +| [Group Activity Analytics](group_activity_analytics.md) | `/analytics/group_activity/{issues_count}` | +| [Group repository storage moves](group_repository_storage_moves.md) **(PREMIUM SELF)** | `/group_repository_storage_moves` | +| [Import repository from GitHub](import.md) | `/import/github` | +| [Instance clusters](instance_clusters.md) **(FREE SELF)** | `/admin/clusters` | +| [Instance-level CI/CD variables](instance_level_ci_variables.md) **(FREE SELF)** | `/admin/ci/variables` | +| [Issues Statistics](issues_statistics.md) | `/issues_statistics` (also available for groups and projects) | +| [Issues](issues.md) | `/issues` (also available for groups and projects) | +| [Jobs](jobs.md) | `/job` | +| [Keys](keys.md) | `/keys` | +| [License](license.md) **(FREE SELF)** | `/license` | +| [Markdown](markdown.md) | `/markdown` | +| [Merge requests](merge_requests.md) | `/merge_requests` (also available for groups and projects) | +| [Metrics dashboard annotations](metrics_dashboard_annotations.md) | `/environments/:id/metrics_dashboard/annotations`, `/clusters/:id/metrics_dashboard/annotations` | +| [Namespaces](namespaces.md) | `/namespaces` | +| [Notification settings](notification_settings.md) | `/notification_settings` (also available for groups and projects) | +| [Pages domains](pages_domains.md) | `/pages/domains` (also available for projects) | +| [Personal access tokens](personal_access_tokens.md) | `/personal_access_tokens` | +| [Plan limits](plan_limits.md) | `/application/plan_limits` | | [Project repository storage moves](project_repository_storage_moves.md) **(FREE SELF)** | `/project_repository_storage_moves` | -| [Runners](runners.md) | `/runners` (also available for projects) | -| [Search](search.md) | `/search` (also available for groups and projects) | -| [Settings](settings.md) **(FREE SELF)** | `/application/settings` | +| [Projects](projects.md) | `/users/:id/projects` (also available for projects) | +| [Runners](runners.md) | `/runners` (also available for projects) | +| [Search](search.md) | `/search` (also available for groups and projects) | +| [Service Data](usage_data.md) | `/usage_data` (For GitLab instance [Administrator](../user/permissions.md) users only) | +| [Settings](settings.md) **(FREE SELF)** | `/application/settings` | +| [Sidekiq metrics](sidekiq_metrics.md) **(FREE SELF)** | `/sidekiq` | +| [Sidekiq queues administration](admin_sidekiq_queues.md) **(FREE SELF)** | `/admin/sidekiq/queues/:queue_name` | | [Snippet repository storage moves](snippet_repository_storage_moves.md) **(FREE SELF)** | `/snippet_repository_storage_moves` | -| [Statistics](statistics.md) | `/application/statistics` | -| [Sidekiq metrics](sidekiq_metrics.md) **(FREE SELF)** | `/sidekiq` | -| [Suggestions](suggestions.md) | `/suggestions` | -| [System hooks](system_hooks.md) | `/hooks` | -| [To-dos](todos.md) | `/todos` | -| [Topics](topics.md) | `/topics` | -| [Service Data](usage_data.md) | `/usage_data` (For GitLab instance [Administrator](../user/permissions.md) users only) | -| [Users](users.md) | `/users` | -| [Validate `.gitlab-ci.yml` file](lint.md) | `/lint` | -| [Version](version.md) | `/version` | +| [Statistics](statistics.md) | `/application/statistics` | +| [Suggestions](suggestions.md) | `/suggestions` | +| [System hooks](system_hooks.md) | `/hooks` | +| [To-dos](todos.md) | `/todos` | +| [Topics](topics.md) | `/topics` | +| [Users](users.md) | `/users` | +| [Validate `.gitlab-ci.yml` file](lint.md) | `/lint` | +| [Version](version.md) | `/version` | ## Templates API resources diff --git a/doc/api/audit_events.md b/doc/api/audit_events.md index 3ffcae04791..888323f9362 100644 --- a/doc/api/audit_events.md +++ b/doc/api/audit_events.md @@ -277,7 +277,7 @@ Example response: "entity_id": 7, "entity_type": "Project", "details": { - "change": "prevent merge request approval from reviewers", + "change": "prevent merge request approval from committers", "from": "", "to": "true", "author_name": "Administrator", @@ -312,7 +312,7 @@ Example response: ### Retrieve a specific project audit event -Only available to project maintainers or owners. +Only available to users with at least the [Maintainer role](../user/permissions.md) for the project. ```plaintext GET /projects/:id/audit_events/:audit_event_id @@ -336,7 +336,7 @@ Example response: "entity_id": 7, "entity_type": "Project", "details": { - "change": "prevent merge request approval from reviewers", + "change": "prevent merge request approval from committers", "from": "", "to": "true", "author_name": "Administrator", diff --git a/doc/api/branches.md b/doc/api/branches.md index 7b9354f3264..4e2a0306845 100644 --- a/doc/api/branches.md +++ b/doc/api/branches.md @@ -40,14 +40,14 @@ Example response: ```json [ { - "name": "master", + "name": "main", "merged": false, "protected": true, "default": true, "developers_can_push": false, "developers_can_merge": false, "can_push": true, - "web_url": "https://gitlab.example.com/my-group/my-project/-/tree/master", + "web_url": "https://gitlab.example.com/my-group/my-project/-/tree/main", "commit": { "author_email": "john@example.com", "author_name": "John Smith", @@ -89,7 +89,7 @@ Parameters: Example request: ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/branches/master" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/branches/main" ``` Example response: @@ -103,7 +103,7 @@ Example response: "developers_can_push": false, "developers_can_merge": false, "can_push": true, - "web_url": "https://gitlab.example.com/my-group/my-project/-/tree/master", + "web_url": "https://gitlab.example.com/my-group/my-project/-/tree/main", "commit": { "author_email": "john@example.com", "author_name": "John Smith", @@ -151,7 +151,7 @@ Parameters: Example request: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/branches?branch=newbranch&ref=master" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/branches?branch=newbranch&ref=main" ``` Example response: diff --git a/doc/api/deploy_tokens.md b/doc/api/deploy_tokens.md index c7189586230..0f2b9a13a3f 100644 --- a/doc/api/deploy_tokens.md +++ b/doc/api/deploy_tokens.md @@ -165,8 +165,8 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \ ## Group deploy tokens -Group maintainers and owners can list group deploy -tokens. Only group owners can create and delete group deploy tokens. +Users with at least the [Maintainer role](../user/permissions.md) for the group can list group deploy +tokens. Only group Owners can create and delete group deploy tokens. ### List group deploy tokens diff --git a/doc/api/discussions.md b/doc/api/discussions.md index 18b74e1450f..5f750df4a48 100644 --- a/doc/api/discussions.md +++ b/doc/api/discussions.md @@ -987,7 +987,7 @@ addition by referencing line 157 in the *new* file: ```shell curl --request POST --header "PRIVATE-TOKEN: [ACCESS_TOKEN]"\ --form "note=This is brilliant!" --form "path=hello.rb"\ - --form "line=157" --form "line_type=old"\ + --form "line=157" --form "line_type=new"\ "https://gitlab.com/api/v4/projects/47/repository/commits/<COMMIT_ID>/comments" ``` diff --git a/doc/api/dora4_project_analytics.md b/doc/api/dora4_project_analytics.md index f69c918c6e2..c202b075c42 100644 --- a/doc/api/dora4_project_analytics.md +++ b/doc/api/dora4_project_analytics.md @@ -25,9 +25,6 @@ GET /projects/:id/analytics/deployment_frequency?environment=:environment&from=: | Attribute | Type | Required | Description | |--------------|--------|----------|-----------------------| | `id` | string | yes | The ID of the project | - -| Parameter | Type | Required | Description | -|--------------|--------|----------|-----------------------| | `environment`| string | yes | The name of the environment to filter by | | `from` | string | yes | Datetime range to start from, inclusive, ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`) | | `to` | string | no | Datetime range to end at, exclusive, ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`) | diff --git a/doc/api/epics.md b/doc/api/epics.md index 263cfe5806e..6d572303460 100644 --- a/doc/api/epics.md +++ b/doc/api/epics.md @@ -49,6 +49,8 @@ NOTE: ## List epics for a group +> `parent_iid` and `_links[parent]` in response were [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/347527) in GitLab 14.6. + Gets all epics of the requested group and its subgroups. ```plaintext @@ -62,7 +64,7 @@ GET /groups/:id/epics?state=opened | ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | | `author_id` | integer | no | Return epics created by the given user `id` | -| `labels` | string | no | Return epics matching a comma separated list of labels names. Label names from the epic group or a parent group can be used | +| `labels` | string | no | Return epics matching a comma-separated list of labels names. Label names from the epic group or a parent group can be used | | `with_labels_details` | boolean | no | If `true`, response returns more details for each label in labels field: `:name`, `:color`, `:description`, `:description_html`, `:text_color`. Default is `false`. Available in [GitLab 12.7](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21413) and later | | `order_by` | string | no | Return epics ordered by `created_at`, `updated_at`, or `title` fields. Default is `created_at` | | `sort` | string | no | Return epics sorted in `asc` or `desc` order. Default is `desc` | @@ -75,6 +77,7 @@ GET /groups/:id/epics?state=opened | `include_ancestor_groups` | boolean | no | Include epics from the requested group's ancestors. Default is `false` | | `include_descendant_groups` | boolean | no | Include epics from the requested group's descendants. Default is `true` | | `my_reaction_emoji` | string | no | Return epics reacted by the authenticated user by the given emoji. `None` returns epics not given a reaction. `Any` returns epics given at least one reaction. Available in [GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31479) and later | +| `not` | Hash | no | Return epics that do not match the parameters supplied. Accepts: `author_id` and `labels`. Available in [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/347525) and later | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/epics" @@ -89,6 +92,7 @@ Example response: "iid": 4, "group_id": 7, "parent_id": 23, + "parent_iid": 3, "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", "state": "opened", @@ -128,7 +132,8 @@ Example response: "_links":{ "self": "http://gitlab.example.com/api/v4/groups/7/epics/4", "epic_issues": "http://gitlab.example.com/api/v4/groups/7/epics/4/issues", - "group":"http://gitlab.example.com/api/v4/groups/7" + "group":"http://gitlab.example.com/api/v4/groups/7", + "parent":"http://gitlab.example.com/api/v4/groups/7/epics/3" } }, { @@ -136,6 +141,7 @@ Example response: "iid": 35, "group_id": 17, "parent_id": 19, + "parent_iid": 1, "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", "state": "opened", @@ -174,7 +180,8 @@ Example response: "_links":{ "self": "http://gitlab.example.com/api/v4/groups/17/epics/35", "epic_issues": "http://gitlab.example.com/api/v4/groups/17/epics/35/issues", - "group":"http://gitlab.example.com/api/v4/groups/17" + "group":"http://gitlab.example.com/api/v4/groups/17", + "parent":"http://gitlab.example.com/api/v4/groups/17/epics/1" } } ] @@ -182,6 +189,8 @@ Example response: ## Single epic +> `parent_iid` and `_links[parent]` in response were [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/347527) in GitLab 14.6. + Gets a single epic ```plaintext @@ -204,6 +213,8 @@ Example response: "id": 30, "iid": 5, "group_id": 7, + "parent_id": null, + "parent_iid": null, "title": "Ea cupiditate dolores ut vero consequatur quasi veniam voluptatem et non.", "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", "state": "opened", @@ -243,13 +254,16 @@ Example response: "_links":{ "self": "http://gitlab.example.com/api/v4/groups/7/epics/5", "epic_issues": "http://gitlab.example.com/api/v4/groups/7/epics/5/issues", - "group":"http://gitlab.example.com/api/v4/groups/7" + "group":"http://gitlab.example.com/api/v4/groups/7", + "parent": null } } ``` ## New epic +> `parent_iid` and `_links[parent]` in response were [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/347527) in GitLab 14.6. + Creates a new epic. NOTE: @@ -265,7 +279,7 @@ POST /groups/:id/epics | ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | | `title` | string | yes | The title of the epic | -| `labels` | string | no | The comma separated list of labels | +| `labels` | string | no | The comma-separated list of labels | | `description` | string | no | The description of the epic. Limited to 1,048,576 characters. | | `confidential` | boolean | no | Whether the epic should be confidential | | `created_at` | string | no | When the epic was created. Date time string, ISO 8601 formatted, for example `2016-03-11T03:45:40Z` . Requires administrator or project/group owner privileges ([available](https://gitlab.com/gitlab-org/gitlab/-/issues/255309) in GitLab 13.5 and later) | @@ -276,7 +290,7 @@ POST /groups/:id/epics | `parent_id` | integer/string | no | The ID of a parent epic (in GitLab 11.11 and later) | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/epics?title=Epic&description=Epic%20description" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/epics?title=Epic&description=Epic%20description&parent_id=29" ``` Example response: @@ -286,6 +300,8 @@ Example response: "id": 33, "iid": 6, "group_id": 7, + "parent_id": 29, + "parent_iid": 4, "title": "Epic", "description": "Epic description", "state": "opened", @@ -325,13 +341,16 @@ Example response: "_links":{ "self": "http://gitlab.example.com/api/v4/groups/7/epics/6", "epic_issues": "http://gitlab.example.com/api/v4/groups/7/epics/6/issues", - "group":"http://gitlab.example.com/api/v4/groups/7" + "group":"http://gitlab.example.com/api/v4/groups/7", + "parent": "http://gitlab.example.com/api/v4/groups/7/epics/4" } } ``` ## Update epic +> `parent_iid` and `_links[parent]` in response were [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/347527) in GitLab 14.6. + Updates an epic. NOTE: @@ -371,6 +390,8 @@ Example response: "id": 33, "iid": 6, "group_id": 7, + "parent_id": null, + "parent_iid": null, "title": "New Title", "description": "Epic description", "state": "opened", diff --git a/doc/api/error_tracking.md b/doc/api/error_tracking.md index 203c1a23996..c62d33f82f4 100644 --- a/doc/api/error_tracking.md +++ b/doc/api/error_tracking.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Error Tracking project settings The project settings API allows you to retrieve the [Error Tracking](../operations/error_tracking.md) -settings for a project. Only for project maintainers. +settings for a project. Only for users with [Maintainer role](../user/permissions.md) for the project. ### Get Error Tracking settings @@ -41,7 +41,8 @@ Example response: ### Enable or disable the Error Tracking project settings -The API allows you to enable or disable the Error Tracking settings for a project. Only for project maintainers. +The API allows you to enable or disable the Error Tracking settings for a project. Only for users with the +[Maintainer role](../user/permissions.md) for the project. ```plaintext PATCH /projects/:id/error_tracking/settings @@ -73,7 +74,8 @@ Example response: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68384) in GitLab 14.3. -For [integrated error tracking](https://gitlab.com/gitlab-org/gitlab/-/issues/329596) feature. Only for project maintainers. +For [integrated error tracking](https://gitlab.com/gitlab-org/gitlab/-/issues/329596) feature. Only for users with the +[Maintainer role](../user/permissions.md) for the project. ### List project client keys diff --git a/doc/api/feature_flag_user_lists.md b/doc/api/feature_flag_user_lists.md index 4306167603d..46ed44d8808 100644 --- a/doc/api/feature_flag_user_lists.md +++ b/doc/api/feature_flag_user_lists.md @@ -71,7 +71,7 @@ POST /projects/:id/feature_flags_user_lists | ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `name` | string | yes | The name of the feature flag. | -| `user_xids` | string | yes | A comma separated list of user IDs. | +| `user_xids` | string | yes | A comma-separated list of user IDs. | ```shell curl "https://gitlab.example.com/api/v4/projects/1/feature_flags_user_lists" \ @@ -143,7 +143,7 @@ PUT /projects/:id/feature_flags_user_lists/:iid | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `iid` | integer/string | yes | The internal ID of the project's feature flag user list. | | `name` | string | no | The name of the feature flag. | -| `user_xids` | string | no | A comma separated list of user IDs. | +| `user_xids` | string | no | A comma-separated list of user IDs. | ```shell curl "https://gitlab.example.com/api/v4/projects/1/feature_flags_user_lists/1" \ diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md index fb821824dd1..3952a87e698 100644 --- a/doc/api/geo_nodes.md +++ b/doc/api/geo_nodes.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Geo Nodes API **(PREMIUM SELF)** -To interact with Geo node endpoints, you need to authenticate yourself as an +To interact with Geo node endpoints, you must authenticate yourself as an administrator. ## Create a new Geo node @@ -26,7 +26,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://primary.example.com/ | Attribute | Type | Required | Description | | ----------------------------| ------- | -------- | -----------------------------------------------------------------| -| `primary` | boolean | no | Specifying whether this node will be primary. Defaults to false. | +| `primary` | boolean | no | Specifying whether this node should be primary. Defaults to false. | | `enabled` | boolean | no | Flag indicating if the Geo node is enabled. Defaults to true. | | `name` | string | yes | The unique identifier for the Geo node. Must match `geo_node_name` if it is set in `gitlab.rb`, otherwise it must match `external_url` | | `url` | string | yes | The user-facing URL for the Geo node. | @@ -35,11 +35,11 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://primary.example.com/ | `repos_max_capacity` | integer | no | Control the maximum concurrency of repository backfill for this secondary node. Defaults to 25. | | `verification_max_capacity` | integer | no | Control the maximum concurrency of repository verification for this node. Defaults to 100. | | `container_repositories_max_capacity` | integer | no | Control the maximum concurrency of container repository sync for this node. Defaults to 10. | -| `sync_object_storage` | boolean | no | Flag indicating if the secondary Geo node will replicate blobs in Object Storage. Defaults to false. | +| `sync_object_storage` | boolean | no | Flag indicating if the secondary Geo node should replicate blobs in Object Storage. Defaults to false. | | `selective_sync_type` | string | no | Limit syncing to only specific groups or shards. Valid values: `"namespaces"`, `"shards"`, or `null`. | | `selective_sync_shards` | array | no | The repository storage for the projects synced if `selective_sync_type` == `shards`. | | `selective_sync_namespace_ids` | array | no | The IDs of groups that should be synced, if `selective_sync_type` == `namespaces`. | -| `minimum_reverification_interval` | integer | no | The interval (in days) in which the repository verification is valid. Once expired, it will be reverified. This has no effect when set on a secondary node. | +| `minimum_reverification_interval` | integer | no | The interval (in days) in which the repository verification is valid. Once expired, it is reverified. This has no effect when set on a secondary node. | Example response: @@ -199,11 +199,11 @@ PUT /geo_nodes/:id | `repos_max_capacity` | integer | no | Control the maximum concurrency of repository backfill for this secondary node. | | `verification_max_capacity` | integer | no | Control the maximum concurrency of verification for this node. | | `container_repositories_max_capacity` | integer | no | Control the maximum concurrency of container repository sync for this node. | -| `sync_object_storage` | boolean | no | Flag indicating if the secondary Geo node will replicate blobs in Object Storage. | +| `sync_object_storage` | boolean | no | Flag indicating if the secondary Geo node should replicate blobs in Object Storage. | | `selective_sync_type` | string | no | Limit syncing to only specific groups or shards. Valid values: `"namespaces"`, `"shards"`, or `null`. | | `selective_sync_shards` | array | no | The repository storage for the projects synced if `selective_sync_type` == `shards`. | | `selective_sync_namespace_ids` | array | no | The IDs of groups that should be synced, if `selective_sync_type` == `namespaces`. | -| `minimum_reverification_interval` | integer | no | The interval (in days) in which the repository verification is valid. Once expired, it will be reverified. This has no effect when set on a secondary node. | +| `minimum_reverification_interval` | integer | no | The interval (in days) in which the repository verification is valid. Once expired, it is reverified. This has no effect when set on a secondary node. | Example response: @@ -241,7 +241,7 @@ Example response: Removes the Geo node. NOTE: -Only a Geo primary node will accept this request. +Only a Geo primary node accepts this request. ```plaintext DELETE /geo_nodes/:id @@ -307,11 +307,18 @@ Example response: "health_status": "Healthy", "missing_oauth_application": false, "db_replication_lag_seconds": null, - "lfs_objects_count": 0, + "lfs_objects_count": 5, + "lfs_objects_checksum_total_count": 5, + "lfs_objects_checksummed_count": 5, + "lfs_objects_checksum_failed_count": 0, "lfs_objects_synced_count": null, "lfs_objects_failed_count": null, - "lfs_objects_synced_missing_on_primary_count": 0, + "lfs_objects_registry_count": null, + "lfs_objects_verification_total_count": null, + "lfs_objects_verified_count": null, + "lfs_objects_verification_failed_count": null, "lfs_objects_synced_in_percentage": "0.00%", + "lfs_objects_verified_in_percentage": "0.00%", "job_artifacts_count": 2, "job_artifacts_synced_count": null, "job_artifacts_failed_count": null, @@ -453,6 +460,13 @@ Example response: "uploads_failed_count": 0, "uploads_registry_count": null, "uploads_synced_in_percentage": "0.00%", + "uploads_checksum_total_count": 5, + "uploads_checksummed_count": 5, + "uploads_checksum_failed_count": null, + "uploads_verification_total_count": null, + "uploads_verified_count": null, + "uploads_verification_failed_count": null, + "uploads_verified_in_percentage": "0.00%", }, { "geo_node_id": 2, @@ -461,11 +475,18 @@ Example response: "health_status": "Healthy", "missing_oauth_application": false, "db_replication_lag_seconds": 0, - "lfs_objects_count": 0, - "lfs_objects_synced_count": 0, - "lfs_objects_failed_count": 0, - "lfs_objects_synced_missing_on_primary_count": 0, + "lfs_objects_count": 5, + "lfs_objects_checksum_total_count": 5, + "lfs_objects_checksummed_count": 5, + "lfs_objects_checksum_failed_count": 0, + "lfs_objects_synced_count": null, + "lfs_objects_failed_count": null, + "lfs_objects_registry_count": null, + "lfs_objects_verification_total_count": null, + "lfs_objects_verified_count": null, + "lfs_objects_verification_failed_count": null, "lfs_objects_synced_in_percentage": "0.00%", + "lfs_objects_verified_in_percentage": "0.00%", "job_artifacts_count": 2, "job_artifacts_synced_count": 1, "job_artifacts_failed_count": 1, @@ -595,6 +616,13 @@ Example response: "uploads_failed_count": 0, "uploads_registry_count": null, "uploads_synced_in_percentage": "0.00%", + "uploads_checksum_total_count": 5, + "uploads_checksummed_count": 5, + "uploads_checksum_failed_count": null, + "uploads_verification_total_count": null, + "uploads_verified_count": null, + "uploads_verification_failed_count": null, + "uploads_verified_in_percentage": "0.00%", } ] ``` @@ -619,11 +647,18 @@ Example response: "health_status": "Healthy", "missing_oauth_application": false, "db_replication_lag_seconds": 0, - "lfs_objects_count": 0, - "lfs_objects_synced_count": 0, - "lfs_objects_failed_count": 0, - "lfs_objects_synced_missing_on_primary_count": 0, + "lfs_objects_count": 5, + "lfs_objects_checksum_total_count": 5, + "lfs_objects_checksummed_count": 5, + "lfs_objects_checksum_failed_count": 0, + "lfs_objects_synced_count": null, + "lfs_objects_failed_count": null, + "lfs_objects_registry_count": null, + "lfs_objects_verification_total_count": null, + "lfs_objects_verified_count": null, + "lfs_objects_verification_failed_count": null, "lfs_objects_synced_in_percentage": "0.00%", + "lfs_objects_verified_in_percentage": "0.00%", "job_artifacts_count": 2, "job_artifacts_synced_count": 1, "job_artifacts_failed_count": 1, @@ -734,6 +769,13 @@ Example response: "uploads_failed_count": 0, "uploads_registry_count": null, "uploads_synced_in_percentage": "0.00%", + "uploads_checksum_total_count": 5, + "uploads_checksummed_count": 5, + "uploads_checksum_failed_count": null, + "uploads_verification_total_count": null, + "uploads_verified_count": null, + "uploads_verification_failed_count": null, + "uploads_verified_in_percentage": "0.00%", } ``` diff --git a/doc/api/graphql/index.md b/doc/api/graphql/index.md index 3523276bdf5..bcaa5930faf 100644 --- a/doc/api/graphql/index.md +++ b/doc/api/graphql/index.md @@ -6,27 +6,42 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GraphQL API **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19008) in GitLab 11.0 (enabled by feature flag `graphql`). -> - [Always enabled](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30444) in GitLab 12.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19008) in GitLab 11.0 [with a flag](../../administration/feature_flags.md) named `graphql`. +> - [Enabled](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30444) in GitLab 12.1. -## Getting Started +[GraphQL](https://graphql.org/) is a query language for APIs. You can use it to +request the exact data you need, and therefore limit the number of requests you need. -For those new to the GitLab GraphQL API, see -[Getting started with GitLab GraphQL API](getting_started.md). +GraphQL data is arranged in types, so your client can use +[client-side GraphQL libraries](https://graphql.org/code/#graphql-clients) +to consume the API and avoid manual parsing. -### Quick Reference +There are no fixed endpoints and no data model, so you can add +to the API without creating [breaking changes](../../development/contributing/#breaking-changes). +This enables us to have a [versionless API](https://graphql.org/learn/best-practices/#versioning). -- The GitLab GraphQL API endpoint is located at `/api/graphql`. -- Get an [introduction to GraphQL from graphql.org](https://graphql.org/). -- GitLab supports a wide range of resources, listed in the [GraphQL API Reference](reference/index.md). +## Vision + +We want the GraphQL API to be the **primary** means of interacting +programmatically with GitLab. To achieve this, it needs full coverage - anything +possible in the REST API should also be possible in the GraphQL API. -### Examples +To help us meet this vision, the frontend should use GraphQL in preference to +the REST API for new features. -To work with sample queries that pull data from public projects on GitLab.com, -see the menu options in the left-hand -documentation menu, under API > GraphQL at `https://docs.gitlab.com/ee/api/graphql/`. +There are no plans to deprecate the REST API. To reduce the technical burden of +supporting two APIs in parallel, they should share implementations as much as +possible. -The [Getting started](getting_started.md) page includes different methods to customize GraphQL queries. +## Work with GraphQL + +If you're new to the GitLab GraphQL API, see [Get started with GitLab GraphQL API](getting_started.md). + +You can view the available resources in the [GraphQL API reference](reference/index.md). +The reference is automatically generated from the GitLab GraphQL schema and +written to a Markdown file. + +The GitLab GraphQL API endpoint is located at `/api/graphql`. ### GraphiQL @@ -34,104 +49,102 @@ Explore the GraphQL API using the interactive [GraphiQL explorer](https://gitlab or on your self-managed GitLab instance on `https://<your-gitlab-site.com>/-/graphql-explorer`. -See the [GitLab GraphQL overview](getting_started.md#graphiql) for more information about the GraphiQL Explorer. +For more information, see [GraphiQL](getting_started.md#graphiql). -## What is GraphQL? +### View GraphQL examples -[GraphQL](https://graphql.org/) is a query language for APIs that -allows clients to request exactly the data they need, making it -possible to get all required data in a limited number of requests. +You can work with sample queries that pull data from public projects on GitLab.com: -The GraphQL data (fields) can be described in the form of types, -allowing clients to use [client-side GraphQL -libraries](https://graphql.org/code/#graphql-clients) to consume the -API and avoid manual parsing. +- [Create an audit report](audit_report.md) +- [Identify issue boards](sample_issue_boards.md) +- [Query users](users_example.md) +- [Use custom emojis](custom_emoji.md) -Since there's no fixed endpoints and data model, new abilities can be -added to the API without creating [breaking changes](../../development/contributing/#breaking-changes). This allows us to -have a versionless API as described in [the GraphQL -documentation](https://graphql.org/learn/best-practices/#versioning). +The [get started](getting_started.md) page includes different methods to customize GraphQL queries. -## Vision +### Update the GraphQL API reference -We want the GraphQL API to be the **primary** means of interacting -programmatically with GitLab. To achieve this, it needs full coverage - anything -possible in the REST API should also be possible in the GraphQL API. +If you change the GraphQL schema, create a merge request to get your changes approved. +To generate the required documentation and schema, see +[Rake tasks for developers](../../development/rake_tasks.md#update-graphql-documentation-and-schema-definitions). -To help us meet this vision, the frontend should use GraphQL in preference to -the REST API for new features. - -There are no plans to deprecate the REST API. To reduce the technical burden of -supporting two APIs in parallel, they should share implementations as much as -possible. +Run the commands using the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit/). ## Breaking changes -The GitLab GraphQL API is [versionless](https://graphql.org/learn/best-practices/#versioning) and -changes are made to the API in a way that maintains backwards-compatibility. +The GitLab GraphQL API is [versionless](https://graphql.org/learn/best-practices/#versioning) and changes to the API are primarily backward-compatible. -Occasionally GitLab needs to change the GraphQL API in a way that is not backwards-compatible. -These changes include the removal or renaming of fields, arguments or other parts of the schema. +However, GitLab sometimes changes the GraphQL API in a way that is not backward-compatible. These changes are considered breaking changes, and +can include removing or renaming fields, arguments, or other parts of the schema. +When creating a breaking change, GitLab follows a [deprecation and removal process](#deprecation-and-removal-process). -In these situations, GitLab follows a [Deprecation and removal process](#deprecation-and-removal-process) -where the deprecated part of the schema is supported for a period of time before being removed. +Learn more about [breaking changes](../../development/contributing/#breaking-changes). -There are some changes which are explicitly [not considered breaking](../../development/contributing/#breaking-changes). +Fields behind a feature flag and disabled by default do not follow the deprecation and removal process, and can be removed at any time without notice. -Clients should familiarize themselves with the process to avoid breaking changes affecting their integrations. +To avoid having a breaking change affect your integrations, you should +familiarize yourself with the deprecation and removal process. WARNING: -While GitLab will make all attempts to follow the [deprecation and removal process](#deprecation-and-removal-process), -GitLab may on very rare occasions need to make immediate breaking changes to the GraphQL API to patch critical security or performance -concerns and where the deprecation process would be considered to pose significant risk. +GitLab makes all attempts to follow the [deprecation and removal process](#deprecation-and-removal-process). +On rare occasions, GitLab might make immediate breaking changes to the GraphQL +API to patch critical security or performance concerns if the deprecation +process would pose significant risk. -NOTE: -Fields behind a feature flag and disabled by default are exempt from the deprecation process, -and can be removed at any time without notice. +### Deprecation and removal process -### Deprecation and Removal process +The deprecation and removal process for the GitLab GraphQL API aligns with the wider GitLab +[deprecation process](https://about.gitlab.com/handbook/product/gitlab-the-product/#breaking-changes-deprecations-and-removing-features). -Parts of the schema marked for removal from the GitLab GraphQL API are first **deprecated** but still available -for at least six releases, and then **removed entirely**. -Removals occur at `X.0` and `X.6` releases. +Parts of the schema marked for removal from the GitLab GraphQL API are first +[deprecated](https://about.gitlab.com/handbook/product/gitlab-the-product/#deprecation) +but still available for at least six releases. They are then [removed](https://about.gitlab.com/handbook/product/gitlab-the-product/#removal) +entirely during the next `XX.0` major release. -The process is as follows: +Items are marked as deprecated in: -1. The item is marked as deprecated in the schema. It will be displayed as deprecated in the - [GraphQL API Reference](reference/index.md) and in introspection queries. -1. Removals are announced at least one release prior in the [Deprecations](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations) - section of the release post (at or prior to `X.11` and `X.5` releases). -1. Items meeting criteria are removed in `X.0` or `X.6` and added to: +- The [schema](https://spec.graphql.org/October2021/#sec--deprecated). +- The [GraphQL API reference](reference/index.md). +- The [deprecation feature removal schedule](../../update/deprecations.md), which is linked from release posts. +- Introspection queries of the GraphQL API. - - The [Removals](https://about.gitlab.com/handbook/marketing/blog/release-posts/#removals) section of the Release Post. - - The [Removed items page](removed_items.md). +NOTE: +If you use the GraphQL API, we recommend you remove the deprecated schema from your GraphQL +API calls as soon as possible to avoid experiencing breaking changes. -This gives consumers of the GraphQL API a minimum of six months to update their GraphQL queries. +The deprecation message provides an alternative for the deprecated schema item, +if applicable. -When an item is deprecated or removed, an alternative is provided if available. +#### Deprecation example -**Example:** +The following fields are deprecated in different minor releases, but both +removed in GitLab 14.0: -A field marked as deprecated in `12.7` can be used until its removal in `13.6`. +| Field deprecated in | Reason | +| ------------------- | --- | +| 12.7 | GitLab traditionally has 12 minor releases per major release. To ensure the field is available for 6 more releases, it is removed in the 14.0 major release (and not 13.0). | +| 13.6 | The removal in 14.0 allows for 6 months of availability. | ### List of removed items -View the [fields, enums, and other items we removed](removed_items.md) from the GraphQL API. +View the [list of items removed](removed_items.md) in previous releases. ## Available queries The GraphQL API includes the following queries at the root level: -1. `project` : Project information, with many of its associations such as issues and merge requests. -1. `group` : Basic group information and epics **(ULTIMATE)** are currently supported. -1. `user` : Information about a particular user. -1. `namespace` : Within a namespace it is also possible to fetch `projects`. -1. `currentUser`: Information about the currently logged in user. -1. `users`: Information about a collection of users. -1. `metaData`: Metadata about GitLab and the GraphQL API. -1. `snippets`: Snippets visible to the currently logged in user. - -New associations and root level objects are constantly being added. +Query | Description +--------------|------------ +`project` | Project information and many of its associations, such as issues and merge requests. +`group` | Basic group information and epics. +`user` | Information about a particular user. +`namespace` | The namespace and the `projects` in it. +`currentUser` | Information about the signed-in user. +`users` | Information about a collection of users. +`metaData` | Metadata about GitLab and the GraphQL API. +`snippets` | Snippets visible to the signed-in user. + +New associations and root level objects are regularly added. See the [GraphQL API Reference](reference/index.md) for up-to-date information. Root-level queries are defined in @@ -149,41 +162,33 @@ library GitLab uses on the backend. The following limits apply to the GitLab GraphQL API. -### Max page size - -By default, connections return at most `100` records ("nodes") per page, -and this limit applies to most connections in the API. Particular connections -may have different max page size limits that are higher or lower. +Limit | Default +---------------------|--------------------------------------------------------------------- +Max page size | 100 records (nodes) per page. Applies to most connections in the API. Particular connections may have different max page size limits that are higher or lower. +[Max query complexity](#max-query-complexity) | `200` for unauthenticated requests and `250` for authenticated requests. +Request timeout | 30 seconds. ### Max query complexity The GitLab GraphQL API scores the _complexity_ of a query. Generally, larger -queries will have a higher complexity score. This limit is designed to protect +queries have a higher complexity score. This limit is designed to protect the API from performing queries that could negatively impact its overall performance. -The complexity of a single query is limited to a maximum of: - -- `200` for unauthenticated requests. -- `250` for authenticated requests. +You can [query](getting_started.md#query-complexity) the complexity score of a query +and the limit for the request. -The complexity score of a query and limit for the request [can be queried for](getting_started.md#query-complexity). +If a query exceeds the complexity limit, an error message response is +returned. -If a query exceeds the complexity limit an error message response will -be returned. - -In general, each field in a query will add `1` to the complexity score, although -this can be higher or lower for particular fields. Sometimes the addition of +In general, each field in a query adds `1` to the complexity score, although +this can be higher or lower for particular fields. Sometimes, adding certain arguments may also increase the complexity of a query. NOTE: The complexity limits may be revised in future, and additionally, the complexity of a query may be altered. -### Request timeout - -Requests time out at 30 seconds. - -### Spam +## Spam GraphQL mutations can be detected as spam. If this happens, a [GraphQL top-level error](https://spec.graphql.org/June2018/#sec-Errors) is raised. For example: @@ -208,11 +213,11 @@ GraphQL mutations can be detected as spam. If this happens, a } ``` -If mutation is detected as potential spam and a CAPTCHA service is configured: +If a mutation is detected as potential spam and a CAPTCHA service is configured: -- The `captchaSiteKey` should be used to obtain a CAPTCHA response value using the appropriate CAPTCHA API. +- Use the `captchaSiteKey` to obtain a CAPTCHA response value using the appropriate CAPTCHA API. Only [Google reCAPTCHA v2](https://developers.google.com/recaptcha/docs/display) is supported. -- The request can be resubmitted with the `X-GitLab-Captcha-Response` and `X-GitLab-Spam-Log-Id` headers set. +- Resubmit the request with the `X-GitLab-Captcha-Response` and `X-GitLab-Spam-Log-Id` headers set. ```json { @@ -235,17 +240,3 @@ If mutation is detected as potential spam and a CAPTCHA service is configured: } } ``` - -## Reference - -The GitLab GraphQL reference [is available](reference/index.md). - -It is automatically generated from the GitLab GraphQL schema and embedded in a Markdown file. - -## Generate updates for documentation - -If you've changed the GraphQL schema, you should set up an MR to gain approval of your changes. -To generate the required documentation and schema, follow the instructions given in the -[Rake tasks for developers](../../development/rake_tasks.md#update-graphql-documentation-and-schema-definitions) page. - -Be sure to run these commands using the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit/). diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 905b6d418a9..123e65162d8 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -370,6 +370,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="queryrunnersactive"></a>`active` | [`Boolean`](#boolean) | Filter runners by active (true) or paused (false) status. | | <a id="queryrunnerssearch"></a>`search` | [`String`](#string) | Filter by full token or partial text in description field. | | <a id="queryrunnerssort"></a>`sort` | [`CiRunnerSort`](#cirunnersort) | Sort order of results. | | <a id="queryrunnersstatus"></a>`status` | [`CiRunnerStatus`](#cirunnerstatus) | Filter runners by status. | @@ -397,6 +398,16 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="querysnippetstype"></a>`type` | [`TypeEnum`](#typeenum) | Type of snippet. | | <a id="querysnippetsvisibility"></a>`visibility` | [`VisibilityScopesEnum`](#visibilityscopesenum) | Visibility of the snippet. | +### `Query.subscriptionFutureEntries` + +Fields related to entries in future subscriptions. + +Returns [`SubscriptionFutureEntryConnection`](#subscriptionfutureentryconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + ### `Query.timelogs` Find timelogs visible to the current user. @@ -500,6 +511,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="queryvulnerabilitiesclusteragentid"></a>`clusterAgentId` | [`[ClustersAgentID!]`](#clustersagentid) | Filter vulnerabilities by `cluster_agent_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | | <a id="queryvulnerabilitiesclusterid"></a>`clusterId` | [`[ClustersClusterID!]`](#clustersclusterid) | Filter vulnerabilities by `cluster_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | | <a id="queryvulnerabilitieshasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked issues. | | <a id="queryvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | @@ -676,9 +688,9 @@ Input type: `ApiFuzzingCiConfigurationCreateInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationapifuzzingciconfigurationcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationapifuzzingciconfigurationcreateconfigurationyaml"></a>`configurationYaml` | [`String`](#string) | A YAML snippet that can be inserted into the project's `.gitlab-ci.yml` to set up API fuzzing scans. | +| <a id="mutationapifuzzingciconfigurationcreateconfigurationyaml"></a>`configurationYaml` **{warning-solid}** | [`String`](#string) | **Deprecated:** The configuration snippet is now generated client-side. Deprecated in 14.6. | | <a id="mutationapifuzzingciconfigurationcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationapifuzzingciconfigurationcreategitlabciyamleditpath"></a>`gitlabCiYamlEditPath` | [`String`](#string) | Location at which the project's `.gitlab-ci.yml` file can be edited in the browser. | +| <a id="mutationapifuzzingciconfigurationcreategitlabciyamleditpath"></a>`gitlabCiYamlEditPath` **{warning-solid}** | [`String`](#string) | **Deprecated:** The configuration snippet is now generated client-side. Deprecated in 14.6. | ### `Mutation.awardEmojiAdd` @@ -2846,7 +2858,7 @@ Input type: `IssueSetCrmContactsInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationissuesetcrmcontactsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationissuesetcrmcontactscrmcontactids"></a>`crmContactIds` | [`[CustomerRelationsContactID!]!`](#customerrelationscontactid) | Customer relations contact IDs to set. Replaces existing contacts by default. | +| <a id="mutationissuesetcrmcontactscontactids"></a>`contactIds` | [`[CustomerRelationsContactID!]!`](#customerrelationscontactid) | Customer relations contact IDs to set. Replaces existing contacts by default. | | <a id="mutationissuesetcrmcontactsiid"></a>`iid` | [`String!`](#string) | IID of the issue to mutate. | | <a id="mutationissuesetcrmcontactsoperationmode"></a>`operationMode` | [`MutationOperationMode`](#mutationoperationmode) | Changes the operation mode. Defaults to REPLACE. | | <a id="mutationissuesetcrmcontactsprojectpath"></a>`projectPath` | [`ID!`](#id) | Project the issue to mutate is in. | @@ -3757,7 +3769,7 @@ Input type: `ProjectSetComplianceFrameworkInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationprojectsetcomplianceframeworkclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationprojectsetcomplianceframeworkcomplianceframeworkid"></a>`complianceFrameworkId` | [`ComplianceManagementFrameworkID`](#compliancemanagementframeworkid) | ID of the compliance framework to assign to the project. | +| <a id="mutationprojectsetcomplianceframeworkcomplianceframeworkid"></a>`complianceFrameworkId` | [`ComplianceManagementFrameworkID`](#compliancemanagementframeworkid) | ID of the compliance framework to assign to the project. Set to `null` to unset. | | <a id="mutationprojectsetcomplianceframeworkprojectid"></a>`projectId` | [`ProjectID!`](#projectid) | ID of the project to change the compliance framework of. | #### Fields @@ -5457,6 +5469,30 @@ The edge type for [`CiStage`](#cistage). | <a id="cistageedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="cistageedgenode"></a>`node` | [`CiStage`](#cistage) | The item at the end of the edge. | +#### `ClusterAgentActivityEventConnection` + +The connection type for [`ClusterAgentActivityEvent`](#clusteragentactivityevent). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="clusteragentactivityeventconnectioncount"></a>`count` | [`Int!`](#int) | Total count of collection. | +| <a id="clusteragentactivityeventconnectionedges"></a>`edges` | [`[ClusterAgentActivityEventEdge]`](#clusteragentactivityeventedge) | A list of edges. | +| <a id="clusteragentactivityeventconnectionnodes"></a>`nodes` | [`[ClusterAgentActivityEvent]`](#clusteragentactivityevent) | A list of nodes. | +| <a id="clusteragentactivityeventconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `ClusterAgentActivityEventEdge` + +The edge type for [`ClusterAgentActivityEvent`](#clusteragentactivityevent). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="clusteragentactivityeventedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="clusteragentactivityeventedgenode"></a>`node` | [`ClusterAgentActivityEvent`](#clusteragentactivityevent) | The item at the end of the edge. | + #### `ClusterAgentConnection` The connection type for [`ClusterAgent`](#clusteragent). @@ -5767,6 +5803,7 @@ The connection type for [`DastProfile`](#dastprofile). | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="dastprofileconnectioncount"></a>`count` | [`Int!`](#int) | Total count of collection. | | <a id="dastprofileconnectionedges"></a>`edges` | [`[DastProfileEdge]`](#dastprofileedge) | A list of edges. | | <a id="dastprofileconnectionnodes"></a>`nodes` | [`[DastProfile]`](#dastprofile) | A list of nodes. | | <a id="dastprofileconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | @@ -6476,6 +6513,29 @@ The edge type for [`JiraProject`](#jiraproject). | <a id="jiraprojectedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="jiraprojectedgenode"></a>`node` | [`JiraProject`](#jiraproject) | The item at the end of the edge. | +#### `JobNeedUnionConnection` + +The connection type for [`JobNeedUnion`](#jobneedunion). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="jobneedunionconnectionedges"></a>`edges` | [`[JobNeedUnionEdge]`](#jobneedunionedge) | A list of edges. | +| <a id="jobneedunionconnectionnodes"></a>`nodes` | [`[JobNeedUnion]`](#jobneedunion) | A list of nodes. | +| <a id="jobneedunionconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `JobNeedUnionEdge` + +The edge type for [`JobNeedUnion`](#jobneedunion). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="jobneedunionedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="jobneedunionedgenode"></a>`node` | [`JobNeedUnion`](#jobneedunion) | The item at the end of the edge. | + #### `LabelConnection` The connection type for [`Label`](#label). @@ -7540,6 +7600,29 @@ The edge type for [`Submodule`](#submodule). | <a id="submoduleedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="submoduleedgenode"></a>`node` | [`Submodule`](#submodule) | The item at the end of the edge. | +#### `SubscriptionFutureEntryConnection` + +The connection type for [`SubscriptionFutureEntry`](#subscriptionfutureentry). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="subscriptionfutureentryconnectionedges"></a>`edges` | [`[SubscriptionFutureEntryEdge]`](#subscriptionfutureentryedge) | A list of edges. | +| <a id="subscriptionfutureentryconnectionnodes"></a>`nodes` | [`[SubscriptionFutureEntry]`](#subscriptionfutureentry) | A list of nodes. | +| <a id="subscriptionfutureentryconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `SubscriptionFutureEntryEdge` + +The edge type for [`SubscriptionFutureEntry`](#subscriptionfutureentry). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="subscriptionfutureentryedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="subscriptionfutureentryedgenode"></a>`node` | [`SubscriptionFutureEntry`](#subscriptionfutureentry) | The item at the end of the edge. | + #### `TerraformStateConnection` The connection type for [`TerraformState`](#terraformstate). @@ -8339,6 +8422,7 @@ Represents an epic on an issue board. | <a id="boardepicdownvotes"></a>`downvotes` | [`Int!`](#int) | Number of downvotes the epic has received. | | <a id="boardepicduedate"></a>`dueDate` | [`Time`](#time) | Due date of the epic. | | <a id="boardepicduedatefixed"></a>`dueDateFixed` | [`Time`](#time) | Fixed due date of the epic. | +| <a id="boardepicduedatefrominheritedsource"></a>`dueDateFromInheritedSource` | [`Time`](#time) | Inherited due date of the epic from child epics or milestones. | | <a id="boardepicduedatefrommilestones"></a>`dueDateFromMilestones` | [`Time`](#time) | Inherited due date of the epic from milestones. | | <a id="boardepicduedateisfixed"></a>`dueDateIsFixed` | [`Boolean`](#boolean) | Indicates if the due date has been manually set. | | <a id="boardepicevents"></a>`events` | [`EventConnection`](#eventconnection) | List of events associated with the object. (see [Connections](#connections)) | @@ -8358,6 +8442,7 @@ Represents an epic on an issue board. | <a id="boardepicrelativeposition"></a>`relativePosition` | [`Int`](#int) | Relative position of the epic in the epic tree. | | <a id="boardepicstartdate"></a>`startDate` | [`Time`](#time) | Start date of the epic. | | <a id="boardepicstartdatefixed"></a>`startDateFixed` | [`Time`](#time) | Fixed start date of the epic. | +| <a id="boardepicstartdatefrominheritedsource"></a>`startDateFromInheritedSource` | [`Time`](#time) | Inherited start date of the epic from child epics or milestones. | | <a id="boardepicstartdatefrommilestones"></a>`startDateFromMilestones` | [`Time`](#time) | Inherited start date of the epic from milestones. | | <a id="boardepicstartdateisfixed"></a>`startDateIsFixed` | [`Boolean`](#boolean) | Indicates if the start date has been manually set. | | <a id="boardepicstate"></a>`state` | [`EpicState!`](#epicstate) | State of the epic. | @@ -8557,7 +8642,7 @@ Represents the total number of issues and their weights for a particular day. | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="cibuildneedid"></a>`id` | [`ID!`](#id) | ID of the job we need to complete. | +| <a id="cibuildneedid"></a>`id` | [`ID!`](#id) | ID of the BuildNeed. | | <a id="cibuildneedname"></a>`name` | [`String`](#string) | Name of the job we need to complete. | ### `CiConfig` @@ -8661,6 +8746,7 @@ Represents the total number of issues and their weights for a particular day. | <a id="cijobneeds"></a>`needs` | [`CiBuildNeedConnection`](#cibuildneedconnection) | References to builds that must complete before the jobs run. (see [Connections](#connections)) | | <a id="cijobpipeline"></a>`pipeline` | [`Pipeline`](#pipeline) | Pipeline the job belongs to. | | <a id="cijobplayable"></a>`playable` | [`Boolean!`](#boolean) | Indicates the job can be played. | +| <a id="cijobpreviousstagejobsorneeds"></a>`previousStageJobsOrNeeds` | [`JobNeedUnionConnection`](#jobneedunionconnection) | Jobs that must complete before the job runs. Returns `BuildNeed`, which is the needed jobs if the job uses the `needs` keyword, or the previous stage jobs otherwise. (see [Connections](#connections)) | | <a id="cijobqueuedat"></a>`queuedAt` | [`Time`](#time) | When the job was enqueued and marked as pending. | | <a id="cijobqueuedduration"></a>`queuedDuration` | [`Duration`](#duration) | How long the job was enqueued before starting. | | <a id="cijobrefname"></a>`refName` | [`String`](#string) | Ref name of the job. | @@ -8723,7 +8809,7 @@ Represents the total number of issues and their weights for a particular day. | ---- | ---- | ----------- | | <a id="cirunneraccesslevel"></a>`accessLevel` | [`CiRunnerAccessLevel!`](#cirunneraccesslevel) | Access level of the runner. | | <a id="cirunneractive"></a>`active` | [`Boolean!`](#boolean) | Indicates the runner is allowed to receive jobs. | -| <a id="cirunneradminurl"></a>`adminUrl` | [`String`](#string) | Admin URL of the runner. Only available for adminstrators. | +| <a id="cirunneradminurl"></a>`adminUrl` | [`String`](#string) | Admin URL of the runner. Only available for administrators. | | <a id="cirunnercontactedat"></a>`contactedAt` | [`Time`](#time) | Last contact from the runner. | | <a id="cirunnerdescription"></a>`description` | [`String`](#string) | Description of the runner. | | <a id="cirunnerid"></a>`id` | [`CiRunnerID!`](#cirunnerid) | ID of the runner. | @@ -8738,11 +8824,24 @@ Represents the total number of issues and their weights for a particular day. | <a id="cirunnerrununtagged"></a>`runUntagged` | [`Boolean!`](#boolean) | Indicates the runner is able to run untagged jobs. | | <a id="cirunnerrunnertype"></a>`runnerType` | [`CiRunnerType!`](#cirunnertype) | Type of the runner. | | <a id="cirunnershortsha"></a>`shortSha` | [`String`](#string) | First eight characters of the runner's token used to authenticate new job requests. Used as the runner's unique ID. | -| <a id="cirunnerstatus"></a>`status` | [`CiRunnerStatus!`](#cirunnerstatus) | Status of the runner. | | <a id="cirunnertaglist"></a>`tagList` | [`[String!]`](#string) | Tags associated with the runner. | | <a id="cirunneruserpermissions"></a>`userPermissions` | [`RunnerPermissions!`](#runnerpermissions) | Permissions for the current user on the resource. | | <a id="cirunnerversion"></a>`version` | [`String`](#string) | Version of the runner. | +#### Fields with arguments + +##### `CiRunner.status` + +Status of the runner. + +Returns [`CiRunnerStatus!`](#cirunnerstatus). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="cirunnerstatuslegacymode"></a>`legacyMode` **{warning-solid}** | [`String`](#string) | **Deprecated** in 14.6. Will be removed in 15.0. From that release onward, the field will behave as if legacyMode is null. | + ### `CiStage` #### Fields @@ -8773,6 +8872,7 @@ GitLab CI/CD configuration template. | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="clusteragentactivityevents"></a>`activityEvents` | [`ClusterAgentActivityEventConnection`](#clusteragentactivityeventconnection) | Recent activity for the cluster agent. (see [Connections](#connections)) | | <a id="clusteragentconnections"></a>`connections` | [`ConnectedAgentConnection`](#connectedagentconnection) | Active connections for the cluster agent. (see [Connections](#connections)) | | <a id="clusteragentcreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp the cluster agent was created. | | <a id="clusteragentcreatedbyuser"></a>`createdByUser` | [`UserCore`](#usercore) | User object, containing information about the person who created the agent. | @@ -8783,6 +8883,18 @@ GitLab CI/CD configuration template. | <a id="clusteragentupdatedat"></a>`updatedAt` | [`Time`](#time) | Timestamp the cluster agent was updated. | | <a id="clusteragentwebpath"></a>`webPath` | [`String`](#string) | Web path of the cluster agent. | +### `ClusterAgentActivityEvent` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="clusteragentactivityeventagenttoken"></a>`agentToken` | [`ClusterAgentToken`](#clusteragenttoken) | Agent token associated with the event. | +| <a id="clusteragentactivityeventkind"></a>`kind` | [`String`](#string) | Type of event. | +| <a id="clusteragentactivityeventlevel"></a>`level` | [`String`](#string) | Severity of the event. | +| <a id="clusteragentactivityeventrecordedat"></a>`recordedAt` | [`Time`](#time) | Timestamp the event was recorded. | +| <a id="clusteragentactivityeventuser"></a>`user` | [`UserCore`](#usercore) | User associated with the event. | + ### `ClusterAgentToken` #### Fields @@ -9008,10 +9120,28 @@ Details of a container repository. | <a id="containerrepositorydetailspath"></a>`path` | [`String!`](#string) | Path of the container repository. | | <a id="containerrepositorydetailsproject"></a>`project` | [`Project!`](#project) | Project of the container registry. | | <a id="containerrepositorydetailsstatus"></a>`status` | [`ContainerRepositoryStatus`](#containerrepositorystatus) | Status of the container repository. | -| <a id="containerrepositorydetailstags"></a>`tags` | [`ContainerRepositoryTagConnection`](#containerrepositorytagconnection) | Tags of the container repository. (see [Connections](#connections)) | | <a id="containerrepositorydetailstagscount"></a>`tagsCount` | [`Int!`](#int) | Number of tags associated with this image. | | <a id="containerrepositorydetailsupdatedat"></a>`updatedAt` | [`Time!`](#time) | Timestamp when the container repository was updated. | +#### Fields with arguments + +##### `ContainerRepositoryDetails.tags` + +Tags of the container repository. + +Returns [`ContainerRepositoryTagConnection`](#containerrepositorytagconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="containerrepositorydetailstagsname"></a>`name` | [`String`](#string) | Search by tag name. | +| <a id="containerrepositorydetailstagssort"></a>`sort` | [`ContainerRepositoryTagSort`](#containerrepositorytagsort) | Sort tags by these criteria. | + ### `ContainerRepositoryTag` A tag from a container repository. @@ -9765,6 +9895,7 @@ Represents an epic. | <a id="epicdownvotes"></a>`downvotes` | [`Int!`](#int) | Number of downvotes the epic has received. | | <a id="epicduedate"></a>`dueDate` | [`Time`](#time) | Due date of the epic. | | <a id="epicduedatefixed"></a>`dueDateFixed` | [`Time`](#time) | Fixed due date of the epic. | +| <a id="epicduedatefrominheritedsource"></a>`dueDateFromInheritedSource` | [`Time`](#time) | Inherited due date of the epic from child epics or milestones. | | <a id="epicduedatefrommilestones"></a>`dueDateFromMilestones` | [`Time`](#time) | Inherited due date of the epic from milestones. | | <a id="epicduedateisfixed"></a>`dueDateIsFixed` | [`Boolean`](#boolean) | Indicates if the due date has been manually set. | | <a id="epicevents"></a>`events` | [`EventConnection`](#eventconnection) | List of events associated with the object. (see [Connections](#connections)) | @@ -9784,6 +9915,7 @@ Represents an epic. | <a id="epicrelativeposition"></a>`relativePosition` | [`Int`](#int) | Relative position of the epic in the epic tree. | | <a id="epicstartdate"></a>`startDate` | [`Time`](#time) | Start date of the epic. | | <a id="epicstartdatefixed"></a>`startDateFixed` | [`Time`](#time) | Fixed start date of the epic. | +| <a id="epicstartdatefrominheritedsource"></a>`startDateFromInheritedSource` | [`Time`](#time) | Inherited start date of the epic from child epics or milestones. | | <a id="epicstartdatefrommilestones"></a>`startDateFromMilestones` | [`Time`](#time) | Inherited start date of the epic from milestones. | | <a id="epicstartdateisfixed"></a>`startDateIsFixed` | [`Boolean`](#boolean) | Indicates if the start date has been manually set. | | <a id="epicstate"></a>`state` | [`EpicState!`](#epicstate) | State of the epic. | @@ -10843,6 +10975,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="grouprunnersactive"></a>`active` | [`Boolean`](#boolean) | Filter runners by active (true) or paused (false) status. | | <a id="grouprunnersmembership"></a>`membership` | [`RunnerMembershipFilter`](#runnermembershipfilter) | Control which runners to include in the results. | | <a id="grouprunnerssearch"></a>`search` | [`String`](#string) | Filter by full token or partial text in description field. | | <a id="grouprunnerssort"></a>`sort` | [`CiRunnerSort`](#cirunnersort) | Sort order of results. | @@ -10886,6 +11019,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="groupvulnerabilitiesclusteragentid"></a>`clusterAgentId` | [`[ClustersAgentID!]`](#clustersagentid) | Filter vulnerabilities by `cluster_agent_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | | <a id="groupvulnerabilitiesclusterid"></a>`clusterId` | [`[ClustersClusterID!]`](#clustersclusterid) | Filter vulnerabilities by `cluster_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | | <a id="groupvulnerabilitieshasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked issues. | | <a id="groupvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | @@ -12388,7 +12522,7 @@ Represents a package in the Package Registry. Note that this type is in beta and | <a id="packagemetadata"></a>`metadata` | [`PackageMetadata`](#packagemetadata) | Package metadata. | | <a id="packagename"></a>`name` | [`String!`](#string) | Name of the package. | | <a id="packagepackagetype"></a>`packageType` | [`PackageTypeEnum!`](#packagetypeenum) | Package type. | -| <a id="packagepipelines"></a>`pipelines` | [`PipelineConnection`](#pipelineconnection) | Pipelines that built the package. (see [Connections](#connections)) | +| <a id="packagepipelines"></a>`pipelines` **{warning-solid}** | [`PipelineConnection`](#pipelineconnection) | **Deprecated** in 14.6. Due to scalability concerns, this field is going to be removed. | | <a id="packageproject"></a>`project` | [`Project!`](#project) | Project where the package is stored. | | <a id="packagestatus"></a>`status` | [`PackageStatus!`](#packagestatus) | Package status. | | <a id="packagetags"></a>`tags` | [`PackageTagConnection`](#packagetagconnection) | Package tags. (see [Connections](#connections)) | @@ -12450,7 +12584,7 @@ Represents a package details in the Package Registry. Note that this type is in | <a id="packagedetailstypename"></a>`name` | [`String!`](#string) | Name of the package. | | <a id="packagedetailstypepackagefiles"></a>`packageFiles` | [`PackageFileConnection`](#packagefileconnection) | Package files. (see [Connections](#connections)) | | <a id="packagedetailstypepackagetype"></a>`packageType` | [`PackageTypeEnum!`](#packagetypeenum) | Package type. | -| <a id="packagedetailstypepipelines"></a>`pipelines` | [`PipelineConnection`](#pipelineconnection) | Pipelines that built the package. (see [Connections](#connections)) | +| <a id="packagedetailstypepipelines"></a>`pipelines` **{warning-solid}** | [`PipelineConnection`](#pipelineconnection) | **Deprecated** in 14.6. Due to scalability concerns, this field is going to be removed. | | <a id="packagedetailstypeproject"></a>`project` | [`Project!`](#project) | Project where the package is stored. | | <a id="packagedetailstypestatus"></a>`status` | [`PackageStatus!`](#packagestatus) | Package status. | | <a id="packagedetailstypetags"></a>`tags` | [`PackageTagConnection`](#packagetagconnection) | Package tags. (see [Connections](#connections)) | @@ -12862,6 +12996,7 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectservicedeskenabled"></a>`serviceDeskEnabled` | [`Boolean`](#boolean) | Indicates if the project has service desk enabled. | | <a id="projectsharedrunnersenabled"></a>`sharedRunnersEnabled` | [`Boolean`](#boolean) | Indicates if shared runners are enabled for the project. | | <a id="projectsnippetsenabled"></a>`snippetsEnabled` | [`Boolean`](#boolean) | Indicates if Snippets are enabled for the current user. | +| <a id="projectsquashcommittemplate"></a>`squashCommitTemplate` | [`String`](#string) | Template used to create squash commit message in merge requests. | | <a id="projectsquashreadonly"></a>`squashReadOnly` | [`Boolean!`](#boolean) | Indicates if `squashReadOnly` is enabled. | | <a id="projectsshurltorepo"></a>`sshUrlToRepo` | [`String`](#string) | URL to connect to the project via SSH. | | <a id="projectstarcount"></a>`starCount` | [`Int!`](#int) | Number of times the project has been starred. | @@ -13602,7 +13737,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="projectscanexecutionpoliciesactionscantypes"></a>`actionScanTypes` | [`[SecurityReportTypeEnum!]`](#securityreporttypeenum) | Filters policies by the action scan type. Only these scan types are supported: `dast`, `secret_detection`, `cluster_image_scanning`, `container_scanning`. | +| <a id="projectscanexecutionpoliciesactionscantypes"></a>`actionScanTypes` | [`[SecurityReportTypeEnum!]`](#securityreporttypeenum) | Filters policies by the action scan type. Only these scan types are supported: `dast`, `secret_detection`, `cluster_image_scanning`, `container_scanning`, `sast`. | ##### `Project.sentryDetailedError` @@ -13698,6 +13833,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="projectvulnerabilitiesclusteragentid"></a>`clusterAgentId` | [`[ClustersAgentID!]`](#clustersagentid) | Filter vulnerabilities by `cluster_agent_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | | <a id="projectvulnerabilitiesclusterid"></a>`clusterId` | [`[ClustersClusterID!]`](#clustersclusterid) | Filter vulnerabilities by `cluster_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | | <a id="projectvulnerabilitieshasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked issues. | | <a id="projectvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | @@ -14066,7 +14202,9 @@ Returns [`Tree`](#tree). | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="repositoryblobcancurrentuserpushtobranch"></a>`canCurrentUserPushToBranch` | [`Boolean`](#boolean) | Whether the current user can push to the branch. | | <a id="repositoryblobcanmodifyblob"></a>`canModifyBlob` | [`Boolean`](#boolean) | Whether the current user can modify the blob. | +| <a id="repositoryblobcodeowners"></a>`codeOwners` | [`[UserCore!]`](#usercore) | List of code owners for the blob. | | <a id="repositoryblobeditblobpath"></a>`editBlobPath` | [`String`](#string) | Web path to edit the blob in the old-style editor. | | <a id="repositoryblobexternalstorageurl"></a>`externalStorageUrl` | [`String`](#string) | Web path to download the raw blob via external storage, if enabled. | | <a id="repositoryblobfiletype"></a>`fileType` | [`String`](#string) | Expected format of the blob based on the extension. | @@ -14660,6 +14798,23 @@ Represents the Geo sync and verification state of a snippet repository. | <a id="submoduletype"></a>`type` | [`EntryType!`](#entrytype) | Type of tree entry. | | <a id="submoduleweburl"></a>`webUrl` | [`String`](#string) | Web URL for the sub-module. | +### `SubscriptionFutureEntry` + +Represents an entry from the future subscriptions. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="subscriptionfutureentrycompany"></a>`company` | [`String`](#string) | Company of the licensee. | +| <a id="subscriptionfutureentryemail"></a>`email` | [`String`](#string) | Email of the licensee. | +| <a id="subscriptionfutureentryexpiresat"></a>`expiresAt` | [`Date`](#date) | Date when the license expires. | +| <a id="subscriptionfutureentryname"></a>`name` | [`String`](#string) | Name of the licensee. | +| <a id="subscriptionfutureentryplan"></a>`plan` | [`String!`](#string) | Name of the subscription plan. | +| <a id="subscriptionfutureentrystartsat"></a>`startsAt` | [`Date`](#date) | Date when the license started. | +| <a id="subscriptionfutureentrytype"></a>`type` | [`String!`](#string) | Type of license the subscription will yield. | +| <a id="subscriptionfutureentryusersinlicensecount"></a>`usersInLicenseCount` | [`Int`](#int) | Number of paid user seats. | + ### `TaskCompletionStatus` Completion status of tasks. @@ -15235,6 +15390,7 @@ Represents a vulnerability. | <a id="vulnerabilityconfirmedat"></a>`confirmedAt` | [`Time`](#time) | Timestamp of when the vulnerability state was changed to confirmed. | | <a id="vulnerabilityconfirmedby"></a>`confirmedBy` | [`UserCore`](#usercore) | User that confirmed the vulnerability. | | <a id="vulnerabilitydescription"></a>`description` | [`String`](#string) | Description of the vulnerability. | +| <a id="vulnerabilitydescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | <a id="vulnerabilitydetails"></a>`details` | [`[VulnerabilityDetail!]!`](#vulnerabilitydetail) | Details of the vulnerability. | | <a id="vulnerabilitydetectedat"></a>`detectedAt` | [`Time!`](#time) | Timestamp of when the vulnerability was first detected. | | <a id="vulnerabilitydiscussions"></a>`discussions` | [`DiscussionConnection!`](#discussionconnection) | All discussions on this noteable. (see [Connections](#connections)) | @@ -15505,6 +15661,19 @@ Represents a link related to a vulnerability. | <a id="vulnerabilitylinkname"></a>`name` | [`String`](#string) | Name of the link. | | <a id="vulnerabilitylinkurl"></a>`url` | [`String!`](#string) | URL of the link. | +### `VulnerabilityLocationClusterImageScanning` + +Represents the location of a vulnerability found by a cluster image scan. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilitylocationclusterimagescanningdependency"></a>`dependency` | [`VulnerableDependency`](#vulnerabledependency) | Dependency containing the vulnerability. | +| <a id="vulnerabilitylocationclusterimagescanningimage"></a>`image` | [`String`](#string) | Name of the vulnerable container image. | +| <a id="vulnerabilitylocationclusterimagescanningkubernetesresource"></a>`kubernetesResource` | [`VulnerableKubernetesResource`](#vulnerablekubernetesresource) | Kubernetes resource which uses the vulnerable container image. | +| <a id="vulnerabilitylocationclusterimagescanningoperatingsystem"></a>`operatingSystem` | [`String`](#string) | Operating system that runs on the vulnerable container image. | + ### `VulnerabilityLocationContainerScanning` Represents the location of a vulnerability found by a container security scan. @@ -15655,6 +15824,21 @@ Represents a vulnerable dependency. Used in vulnerability location data. | <a id="vulnerabledependencypackage"></a>`package` | [`VulnerablePackage`](#vulnerablepackage) | Package associated with the vulnerable dependency. | | <a id="vulnerabledependencyversion"></a>`version` | [`String`](#string) | Version of the vulnerable dependency. | +### `VulnerableKubernetesResource` + +Represents a vulnerable Kubernetes resource. Used in vulnerability location data. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerablekubernetesresourceagent"></a>`agent` | [`ClusterAgent`](#clusteragent) | Kubernetes Agent which performed the scan. | +| <a id="vulnerablekubernetesresourceclusterid"></a>`clusterId` | [`ClustersClusterID`](#clustersclusterid) | ID of the Cluster integration which was used to perform the scan. | +| <a id="vulnerablekubernetesresourcecontainername"></a>`containerName` | [`String!`](#string) | Name of the container that had its image scanned. | +| <a id="vulnerablekubernetesresourcekind"></a>`kind` | [`String!`](#string) | Kind of the Kubernetes resource. | +| <a id="vulnerablekubernetesresourcename"></a>`name` | [`String!`](#string) | Name of the Kubernetes resource. | +| <a id="vulnerablekubernetesresourcenamespace"></a>`namespace` | [`String!`](#string) | Kubernetes namespace which the resource resides in. | + ### `VulnerablePackage` Represents a vulnerable package. Used in vulnerability dependency data. @@ -15894,11 +16078,13 @@ Values for sorting runners. | Value | Description | | ----- | ----------- | -| <a id="cirunnerstatusactive"></a>`ACTIVE` | A runner that is not paused. | -| <a id="cirunnerstatusnot_connected"></a>`NOT_CONNECTED` | A runner that has never contacted this instance. | -| <a id="cirunnerstatusoffline"></a>`OFFLINE` | A runner that has not contacted this instance within the last 2 hours. | -| <a id="cirunnerstatusonline"></a>`ONLINE` | A runner that contacted this instance within the last 2 hours. | -| <a id="cirunnerstatuspaused"></a>`PAUSED` | A runner that is paused. | +| <a id="cirunnerstatusactive"></a>`ACTIVE` **{warning-solid}** | **Deprecated** in 14.6. Use CiRunnerType.active instead. | +| <a id="cirunnerstatusnever_contacted"></a>`NEVER_CONTACTED` | Runner that has never contacted this instance. Set legacyMode to null to utilize this value. Will replace NOT_CONNECTED starting in 15.0. | +| <a id="cirunnerstatusnot_connected"></a>`NOT_CONNECTED` **{warning-solid}** | **Deprecated** in 14.6. Use NEVER_CONTACTED instead. NEVER_CONTACTED will have a slightly different scope starting in 15.0, with STALE being returned instead after 3 months of no contact. | +| <a id="cirunnerstatusoffline"></a>`OFFLINE` **{warning-solid}** | **Deprecated** in 14.6. This field will have a slightly different scope starting in 15.0, with STALE being returned after a certain period offline. | +| <a id="cirunnerstatusonline"></a>`ONLINE` | Runner that contacted this instance within the last 2 hours. | +| <a id="cirunnerstatuspaused"></a>`PAUSED` **{warning-solid}** | **Deprecated** in 14.6. Use CiRunnerType.active instead. | +| <a id="cirunnerstatusstale"></a>`STALE` | Runner that has not contacted this instance within the last 3 months. Only available if legacyMode is null. Will be a possible return value starting in 15.0. | ### `CiRunnerType` @@ -16014,6 +16200,15 @@ Status of a container repository. | <a id="containerrepositorystatusdelete_failed"></a>`DELETE_FAILED` | Delete Failed status. | | <a id="containerrepositorystatusdelete_scheduled"></a>`DELETE_SCHEDULED` | Delete Scheduled status. | +### `ContainerRepositoryTagSort` + +Values for sorting tags. + +| Value | Description | +| ----- | ----------- | +| <a id="containerrepositorytagsortname_asc"></a>`NAME_ASC` | Ordered by name in ascending order. | +| <a id="containerrepositorytagsortname_desc"></a>`NAME_DESC` | Ordered by name in descending order. | + ### `DastProfileCadenceUnit` Unit for the duration of Dast Profile Cadence. @@ -17021,7 +17216,6 @@ Name of the feature that the callout is for. | <a id="usercalloutfeaturenameenumcanary_deployment"></a>`CANARY_DEPLOYMENT` | Callout feature name for canary_deployment. | | <a id="usercalloutfeaturenameenumcloud_licensing_subscription_activation_banner"></a>`CLOUD_LICENSING_SUBSCRIPTION_ACTIVATION_BANNER` | Callout feature name for cloud_licensing_subscription_activation_banner. | | <a id="usercalloutfeaturenameenumcluster_security_warning"></a>`CLUSTER_SECURITY_WARNING` | Callout feature name for cluster_security_warning. | -| <a id="usercalloutfeaturenameenumcustomize_homepage"></a>`CUSTOMIZE_HOMEPAGE` | Callout feature name for customize_homepage. | | <a id="usercalloutfeaturenameenumeoa_bronze_plan_banner"></a>`EOA_BRONZE_PLAN_BANNER` | Callout feature name for eoa_bronze_plan_banner. | | <a id="usercalloutfeaturenameenumfeature_flags_new_version"></a>`FEATURE_FLAGS_NEW_VERSION` | Callout feature name for feature_flags_new_version. | | <a id="usercalloutfeaturenameenumgcp_signup_offer"></a>`GCP_SIGNUP_OFFER` | Callout feature name for gcp_signup_offer. | @@ -17048,6 +17242,7 @@ Name of the feature that the callout is for. | <a id="usercalloutfeaturenameenumtwo_factor_auth_recovery_settings_check"></a>`TWO_FACTOR_AUTH_RECOVERY_SETTINGS_CHECK` | Callout feature name for two_factor_auth_recovery_settings_check. | | <a id="usercalloutfeaturenameenumultimate_trial"></a>`ULTIMATE_TRIAL` | Callout feature name for ultimate_trial. | | <a id="usercalloutfeaturenameenumunfinished_tag_cleanup_callout"></a>`UNFINISHED_TAG_CLEANUP_CALLOUT` | Callout feature name for unfinished_tag_cleanup_callout. | +| <a id="usercalloutfeaturenameenumverification_reminder"></a>`VERIFICATION_REMINDER` | Callout feature name for verification_reminder. | | <a id="usercalloutfeaturenameenumweb_ide_alert_dismissed"></a>`WEB_IDE_ALERT_DISMISSED` | Callout feature name for web_ide_alert_dismissed. | | <a id="usercalloutfeaturenameenumweb_ide_ci_environments_guidance"></a>`WEB_IDE_CI_ENVIRONMENTS_GUIDANCE` | Callout feature name for web_ide_ci_environments_guidance. | @@ -17777,6 +17972,13 @@ One of: - [`Issue`](#issue) - [`MergeRequest`](#mergerequest) +#### `JobNeedUnion` + +One of: + +- [`CiBuildNeed`](#cibuildneed) +- [`CiJob`](#cijob) + #### `NoteableType` Represents an object that supports notes. @@ -17825,6 +18027,7 @@ Represents a vulnerability location. The fields with data will depend on the vul One of: +- [`VulnerabilityLocationClusterImageScanning`](#vulnerabilitylocationclusterimagescanning) - [`VulnerabilityLocationContainerScanning`](#vulnerabilitylocationcontainerscanning) - [`VulnerabilityLocationCoverageFuzzing`](#vulnerabilitylocationcoveragefuzzing) - [`VulnerabilityLocationDast`](#vulnerabilitylocationdast) @@ -18288,9 +18491,11 @@ Field that are available while modifying the custom mapping attributes for an HT | <a id="boardissueinputassigneeusername"></a>`assigneeUsername` | [`[String]`](#string) | Filter by assignee username. | | <a id="boardissueinputassigneewildcardid"></a>`assigneeWildcardId` | [`AssigneeWildcardId`](#assigneewildcardid) | Filter by assignee wildcard. Incompatible with assigneeUsername. | | <a id="boardissueinputauthorusername"></a>`authorUsername` | [`String`](#string) | Filter by author username. | +| <a id="boardissueinputconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter by confidentiality. | | <a id="boardissueinputepicid"></a>`epicId` | [`EpicID`](#epicid) | Filter by epic ID. Incompatible with epicWildcardId. | | <a id="boardissueinputepicwildcardid"></a>`epicWildcardId` | [`EpicWildcardId`](#epicwildcardid) | Filter by epic ID wildcard. Incompatible with epicId. | | <a id="boardissueinputiids"></a>`iids` | [`[String!]`](#string) | List of IIDs of issues. For example `["1", "2"]`. | +| <a id="boardissueinputiterationcadenceid"></a>`iterationCadenceId` | [`[IterationsCadenceID!]`](#iterationscadenceid) | Filter by a list of iteration cadence IDs. | | <a id="boardissueinputiterationid"></a>`iterationId` | [`[IterationID!]`](#iterationid) | Filter by a list of iteration IDs. Incompatible with iterationWildcardId. | | <a id="boardissueinputiterationtitle"></a>`iterationTitle` | [`String`](#string) | Filter by iteration title. | | <a id="boardissueinputiterationwildcardid"></a>`iterationWildcardId` | [`IterationWildcardId`](#iterationwildcardid) | Filter by iteration ID wildcard. | @@ -18413,6 +18618,7 @@ Input type for DastSiteProfile authentication. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="epicfiltersauthorusername"></a>`authorUsername` | [`String`](#string) | Filter by author username. | +| <a id="epicfiltersconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter by confidentiality. | | <a id="epicfilterslabelname"></a>`labelName` | [`[String]`](#string) | Filter by label name. | | <a id="epicfiltersmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. Wildcard values "NONE" and "ANY" are supported. | | <a id="epicfiltersnot"></a>`not` | [`NegatedEpicBoardIssueInput`](#negatedepicboardissueinput) | Negated epic arguments. | diff --git a/doc/api/graphql/removed_items.md b/doc/api/graphql/removed_items.md index 0048148ab11..f70add0de45 100644 --- a/doc/api/graphql/removed_items.md +++ b/doc/api/graphql/removed_items.md @@ -1,10 +1,10 @@ --- -stage: Plan -group: Product Planning +stage: Ecosystem +group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GraphQL API removed items +# GraphQL API removed items **(FREE)** GraphQL is a versionless API, unlike the REST API. Occasionally, items have to be updated or removed from the GraphQL API. @@ -38,6 +38,8 @@ Fields [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63293) in ## GitLab 13.6 +Prior to GitLab 14.0, deprecated items could be removed in `XX.6` releases. + Fields [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44866) in GitLab 13.6: | Field name | GraphQL type | Deprecated in | Use instead | diff --git a/doc/api/group_import_export.md b/doc/api/group_import_export.md index 212a62516f1..dbdc2c3669e 100644 --- a/doc/api/group_import_export.md +++ b/doc/api/group_import_export.md @@ -4,7 +4,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Group Import/Export API **(FREE)** +# Group import/export API **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20353) in GitLab 12.8. diff --git a/doc/api/group_labels.md b/doc/api/group_labels.md index 04a619c8fa9..96b8a162e34 100644 --- a/doc/api/group_labels.md +++ b/doc/api/group_labels.md @@ -77,7 +77,7 @@ GET /groups/:id/labels/:label_id | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user. | | `label_id` | integer or string | yes | The ID or title of a group's label. | | `include_ancestor_groups` | boolean | no | Include ancestor groups. Defaults to `true`. | | `include_descendant_groups` | boolean | no | Include descendant groups. Defaults to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259024) in GitLab 13.6 | @@ -152,7 +152,7 @@ PUT /groups/:id/labels/:label_id | Attribute | Type | Required | Description | | ------------- | ------- | -------- | ---------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | | `label_id` | integer or string | yes | The ID or title of a group's label. | | `new_name` | string | no | The new name of the label | | `color` | string | no | The color of the label given in 6-digit hex notation with leading '#' sign (for example, #FFAABB) or one of the [CSS color names](https://developer.mozilla.org/en-US/docs/Web/CSS/color_value#Color_keywords) | @@ -193,7 +193,7 @@ DELETE /groups/:id/labels/:label_id | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | | `label_id` | integer or string | yes | The ID or title of a group's label. | ```shell @@ -214,7 +214,7 @@ POST /groups/:id/labels/:label_id/subscribe | Attribute | Type | Required | Description | | ---------- | ----------------- | -------- | ------------------------------------ | -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | | `label_id` | integer or string | yes | The ID or title of a group's label. | ```shell @@ -250,7 +250,7 @@ POST /groups/:id/labels/:label_id/unsubscribe | Attribute | Type | Required | Description | | ---------- | ----------------- | -------- | ------------------------------------ | -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | | `label_id` | integer or string | yes | The ID or title of a group's label. | ```shell diff --git a/doc/api/groups.md b/doc/api/groups.md index 5faa63585c1..13dea42f3c6 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -275,24 +275,24 @@ GET /groups/:id/projects Parameters: -| Attribute | Type | Required | Description | -| ----------------------------- | -------------- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | -| `archived` | boolean | no | Limit by archived status | -| `visibility` | string | no | Limit by visibility `public`, `internal`, or `private` | -| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, `similarity` (1), or `last_activity_at` fields. Default is `created_at` | -| `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` | -| `search` | string | no | Return list of authorized projects matching the search criteria | -| `simple` | boolean | no | Return only the ID, URL, name, and path of each project | -| `owned` | boolean | no | Limit by projects owned by the current user | -| `starred` | boolean | no | Limit by projects starred by the current user | -| `with_issues_enabled` | boolean | no | Limit by projects with issues feature enabled. Default is `false` | -| `with_merge_requests_enabled` | boolean | no | Limit by projects with merge requests feature enabled. Default is `false` | -| `with_shared` | boolean | no | Include projects shared to this group. Default is `true` | -| `include_subgroups` | boolean | no | Include projects in subgroups of this group. Default is `false` | -| `min_access_level` | integer | no | Limit to projects where current user has at least this [access level](members.md#valid-access-levels) | -| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (administrators only) | -| `with_security_reports` | boolean | no | **(ULTIMATE)** Return only projects that have security reports artifacts present in any of their builds. This means "projects with security reports enabled". Default is `false` | +| Attribute | Type | Required | Description | +| -------------------------------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | +| `archived` | boolean | no | Limit by archived status | +| `visibility` | string | no | Limit by visibility `public`, `internal`, or `private` | +| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, `similarity` (1), or `last_activity_at` fields. Default is `created_at` | +| `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` | +| `search` | string | no | Return list of authorized projects matching the search criteria | +| `simple` | boolean | no | Return only the ID, URL, name, and path of each project | +| `owned` | boolean | no | Limit by projects owned by the current user | +| `starred` | boolean | no | Limit by projects starred by the current user | +| `with_issues_enabled` | boolean | no | Limit by projects with issues feature enabled. Default is `false` | +| `with_merge_requests_enabled` | boolean | no | Limit by projects with merge requests feature enabled. Default is `false` | +| `with_shared` | boolean | no | Include projects shared to this group. Default is `true` | +| `include_subgroups` | boolean | no | Include projects in subgroups of this group. Default is `false` | +| `min_access_level` | integer | no | Limit to projects where current user has at least this [access level](members.md#valid-access-levels) | +| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (administrators only) | +| `with_security_reports` **(ULTIMATE)** | boolean | no | Return only projects that have security reports artifacts present in any of their builds. This means "projects with security reports enabled". Default is `false` | 1. Order by similarity: Orders the results by a similarity score calculated from the provided `search` URL parameter. When using `order_by=similarity`, the `sort` parameter is ignored. When the `search` @@ -783,38 +783,38 @@ POST /groups Parameters: -| Attribute | Type | Required | Description | -| ------------------------------------ | ------- | -------- | ----------- | -| `name` | string | yes | The name of the group. | -| `path` | string | yes | The path of the group. | -| `description` | string | no | The group's description. | -| `membership_lock` | boolean | no | **(PREMIUM)** Prevent adding new members to project membership within this group. | -| `visibility` | string | no | The group's visibility. Can be `private`, `internal`, or `public`. | -| `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group. | -| `require_two_factor_authentication` | boolean | no | Require all users in this group to setup Two-factor authentication. | -| `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | -| `project_creation_level` | string | no | Determine if developers can create projects in the group. Can be `noone` (No one), `maintainer` (Maintainers), or `developer` (Developers + Maintainers). | -| `auto_devops_enabled` | boolean | no | Default to Auto DevOps pipeline for all projects within this group. | -| `subgroup_creation_level` | string | no | Allowed to [create subgroups](../user/group/subgroups/index.md#creating-a-subgroup). Can be `owner` (Owners), or `maintainer` (Maintainers). | -| `emails_disabled` | boolean | no | Disable email notifications | -| `avatar` | mixed | no | Image file for avatar of the group. [Introduced in GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/36681) | -| `mentions_disabled` | boolean | no | Disable the capability of a group from getting mentioned | -| `lfs_enabled` | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group. | -| `request_access_enabled` | boolean | no | Allow users to request member access. | -| `parent_id` | integer | no | The parent group ID for creating nested group. | -| `default_branch_protection` | integer | no | See [Options for `default_branch_protection`](#options-for-default_branch_protection). Default to the global level default branch protection setting. | -| `shared_runners_minutes_limit` | integer | no | **(PREMIUM SELF)** Pipeline minutes quota for this group (included in plan). Can be `nil` (default; inherit system default), `0` (unlimited) or `> 0` | -| `extra_shared_runners_minutes_limit` | integer | no | **(PREMIUM SELF)** Extra pipeline minutes quota for this group (purchased in addition to the minutes included in the plan). | +| Attribute | Type | Required | Description | +| ------------------------------------------------------- | ------- | -------- | ----------- | +| `name` | string | yes | The name of the group. | +| `path` | string | yes | The path of the group. | +| `description` | string | no | The group's description. | +| `membership_lock` **(PREMIUM)** | boolean | no | Prevent adding new members to projects within this group. | +| `visibility` | string | no | The group's visibility. Can be `private`, `internal`, or `public`. | +| `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group. | +| `require_two_factor_authentication` | boolean | no | Require all users in this group to setup Two-factor authentication. | +| `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | +| `project_creation_level` | string | no | Determine if developers can create projects in the group. Can be `noone` (No one), `maintainer` (users with the Maintainer role), or `developer` (users with the Developer or Maintainer role). | +| `auto_devops_enabled` | boolean | no | Default to Auto DevOps pipeline for all projects within this group. | +| `subgroup_creation_level` | string | no | Allowed to [create subgroups](../user/group/subgroups/index.md#creating-a-subgroup). Can be `owner` (Owners), or `maintainer` (users with the Maintainer role). | +| `emails_disabled` | boolean | no | Disable email notifications | +| `avatar` | mixed | no | Image file for avatar of the group. [Introduced in GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/36681) | +| `mentions_disabled` | boolean | no | Disable the capability of a group from getting mentioned | +| `lfs_enabled` | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group. | +| `request_access_enabled` | boolean | no | Allow users to request member access. | +| `parent_id` | integer | no | The parent group ID for creating nested group. | +| `default_branch_protection` | integer | no | See [Options for `default_branch_protection`](#options-for-default_branch_protection). Default to the global level default branch protection setting. | +| `shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Pipeline minutes quota for this group (included in plan). Can be `nil` (default; inherit system default), `0` (unlimited) or `> 0` | +| `extra_shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Extra pipeline minutes quota for this group (purchased in addition to the minutes included in the plan). | ### Options for `default_branch_protection` -The `default_branch_protection` attribute determines whether developers and maintainers can push to the applicable [default branch](../user/project/repository/branches/default.md), as described in the following table: +The `default_branch_protection` attribute determines whether users with the Developer or [Maintainer role](../user/permissions.md) can push to the applicable [default branch](../user/project/repository/branches/default.md), as described in the following table: | Value | Description | |-------|-------------------------------------------------------------------------------------------------------------| -| `0` | No protection. Developers and maintainers can: <br>- Push new commits<br>- Force push changes<br>- Delete the branch | -| `1` | Partial protection. Developers and maintainers can: <br>- Push new commits | -| `2` | Full protection. Only maintainers can: <br>- Push new commits | +| `0` | No protection. Users with the Developer or Maintainer role can: <br>- Push new commits<br>- Force push changes<br>- Delete the branch | +| `1` | Partial protection. Users with the Developer or Maintainer role can: <br>- Push new commits | +| `2` | Full protection. Only users with the Maintainer role can: <br>- Push new commits | ## New Subgroup @@ -850,6 +850,32 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ "https://gitlab.example.com/api/v4/groups/4/projects/56" ``` +## Transfer a group to a new parent group / Turn a subgroup to a top-level group + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23831) in GitLab 14.6. + +Transfer a group to a new parent group or turn a subgroup to a top-level group. Available to administrators and users: + +- With the Owner role for the group to transfer. +- With permission to [create a subgroup](../user/group/subgroups/index.md#creating-a-subgroup) in the new parent group if transferring a group. +- With [permission to create a top-level group](../administration/user_settings.md#prevent-users-from-creating-top-level-groups) if turning a subgroup into a top-level group. + +```plaintext +POST /groups/:id/transfer +``` + +Parameters: + +| Attribute | Type | Required | Description | +| ------------ | -------------- | -------- | ----------- | +| `id` | integer | yes | ID of the group to transfer. | +| `group_id` | integer | no | ID of the new parent group. When not specified, the group to transfer is instead turned into a top-level group. | + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/4/transfer?group_id=7" +``` + ## Update group Updates the project group. Only available to group owners and administrators. @@ -858,32 +884,32 @@ Updates the project group. Only available to group owners and administrators. PUT /groups/:id ``` -| Attribute | Type | Required | Description | -| ------------------------------------------ | ------- | -------- | ----------- | -| `id` | integer | yes | The ID of the group. | -| `name` | string | no | The name of the group. | -| `path` | string | no | The path of the group. | -| `description` | string | no | The description of the group. | -| `membership_lock` | boolean | no | **(PREMIUM)** Prevent adding new members to project membership within this group. | -| `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group. | -| `visibility` | string | no | The visibility level of the group. Can be `private`, `internal`, or `public`. | -| `require_two_factor_authentication` | boolean | no | Require all users in this group to setup Two-factor authentication. | -| `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | -| `project_creation_level` | string | no | Determine if developers can create projects in the group. Can be `noone` (No one), `maintainer` (Maintainers), or `developer` (Developers + Maintainers). | -| `auto_devops_enabled` | boolean | no | Default to Auto DevOps pipeline for all projects within this group. | -| `subgroup_creation_level` | string | no | Allowed to [create subgroups](../user/group/subgroups/index.md#creating-a-subgroup). Can be `owner` (Owners), or `maintainer` (Maintainers). | -| `emails_disabled` | boolean | no | Disable email notifications | -| `avatar` | mixed | no | Image file for avatar of the group. [Introduced in GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/36681) | -| `mentions_disabled` | boolean | no | Disable the capability of a group from getting mentioned | -| `lfs_enabled` (optional) | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group. | -| `request_access_enabled` | boolean | no | Allow users to request member access. | -| `default_branch_protection` | integer | no | See [Options for `default_branch_protection`](#options-for-default_branch_protection). | -| `file_template_project_id` | integer | no | **(PREMIUM)** The ID of a project to load custom file templates from. | -| `shared_runners_minutes_limit` | integer | no | **(PREMIUM SELF)** Pipeline minutes quota for this group (included in plan). Can be `nil` (default; inherit system default), `0` (unlimited) or `> 0` | -| `extra_shared_runners_minutes_limit` | integer | no | **(PREMIUM SELF)** Extra pipeline minutes quota for this group (purchased in addition to the minutes included in the plan). | -| `prevent_forking_outside_group` | boolean | no | **(PREMIUM)** When enabled, users can **not** fork projects from this group to external namespaces -| `shared_runners_setting` | string | no | See [Options for `shared_runners_setting`](#options-for-shared_runners_setting). Enable or disable shared runners for a group's subgroups and projects. | -| `prevent_sharing_groups_outside_hierarchy` | boolean | no | See [Prevent group sharing outside the group hierarchy](../user/group/index.md#prevent-group-sharing-outside-the-group-hierarchy). This attribute is only available on top-level groups. [Introduced in GitLab 14.1](https://gitlab.com/gitlab-org/gitlab/-/issues/333721) | +| Attribute | Type | Required | Description | +| ------------------------------------------------------- | ------- | -------- | ----------- | +| `id` | integer | yes | The ID of the group. | +| `name` | string | no | The name of the group. | +| `path` | string | no | The path of the group. | +| `description` | string | no | The description of the group. | +| `membership_lock` **(PREMIUM)** | boolean | no | Prevent adding new members to projects within this group. | +| `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group. | +| `visibility` | string | no | The visibility level of the group. Can be `private`, `internal`, or `public`. | +| `require_two_factor_authentication` | boolean | no | Require all users in this group to setup Two-factor authentication. | +| `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | +| `project_creation_level` | string | no | Determine if developers can create projects in the group. Can be `noone` (No one), `maintainer` (users with the Maintainer role), or `developer` (users with the Developer or Maintainer role). | +| `auto_devops_enabled` | boolean | no | Default to Auto DevOps pipeline for all projects within this group. | +| `subgroup_creation_level` | string | no | Allowed to [create subgroups](../user/group/subgroups/index.md#creating-a-subgroup). Can be `owner` (Owners), or `maintainer` (users with the Maintainer role). | +| `emails_disabled` | boolean | no | Disable email notifications | +| `avatar` | mixed | no | Image file for avatar of the group. [Introduced in GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/36681) | +| `mentions_disabled` | boolean | no | Disable the capability of a group from getting mentioned | +| `lfs_enabled` (optional) | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group. | +| `request_access_enabled` | boolean | no | Allow users to request member access. | +| `default_branch_protection` | integer | no | See [Options for `default_branch_protection`](#options-for-default_branch_protection). | +| `file_template_project_id` **(PREMIUM)** | integer | no | The ID of a project to load custom file templates from. | +| `shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Pipeline minutes quota for this group (included in plan). Can be `nil` (default; inherit system default), `0` (unlimited) or `> 0` | +| `extra_shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Extra pipeline minutes quota for this group (purchased in addition to the minutes included in the plan). | +| `prevent_forking_outside_group` **(PREMIUM)** | boolean | no | When enabled, users can **not** fork projects from this group to external namespaces +| `shared_runners_setting` | string | no | See [Options for `shared_runners_setting`](#options-for-shared_runners_setting). Enable or disable shared runners for a group's subgroups and projects. | +| `prevent_sharing_groups_outside_hierarchy` | boolean | no | See [Prevent group sharing outside the group hierarchy](../user/group/index.md#prevent-group-sharing-outside-the-group-hierarchy). This attribute is only available on top-level groups. [Introduced in GitLab 14.1](https://gitlab.com/gitlab-org/gitlab/-/issues/333721) | NOTE: The `projects` and `shared_projects` attributes in the response are deprecated and [scheduled for removal in API v5](https://gitlab.com/gitlab-org/gitlab/-/issues/213797). diff --git a/doc/api/import.md b/doc/api/import.md index 18c0eb04fff..1baea5d1500 100644 --- a/doc/api/import.md +++ b/doc/api/import.md @@ -4,7 +4,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Import API +# Import API **(FREE)** ## Import repository from GitHub diff --git a/doc/api/index.md b/doc/api/index.md index 7b599b6ae0a..a4e7a893618 100644 --- a/doc/api/index.md +++ b/doc/api/index.md @@ -8,10 +8,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w Use the GitLab APIs to automate GitLab. -You can also use a partial [OpenAPI definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/api/openapi/openapi.yaml), -to test the API directly from the GitLab user interface. -Contributions are welcome. - ## REST API A REST API is available in GitLab. @@ -19,6 +15,10 @@ Usage instructions are below. For a list of the available resources and their endpoints, see [REST API resources](api_resources.md). +You can also use a partial [OpenAPI definition](openapi/openapi_interactive.md), +to test the API directly from the GitLab user interface. +Contributions are welcome. + <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an introduction and basic steps, see [How to make GitLab API calls](https://www.youtube.com/watch?v=0LsMC3ZiXkA). @@ -43,11 +43,6 @@ There were some patenting and licensing concerns with GraphQL. However, these have been resolved to our satisfaction. The reference implementations were re-licensed under MIT, and the OWF license used for the GraphQL specification. -When GraphQL is fully implemented, GitLab: - -- Can delete controller-specific endpoints. -- Will no longer maintain two different APIs. - ## Compatibility guidelines The HTTP API is versioned with a single number, which is currently `4`. This number @@ -214,7 +209,7 @@ Impersonation tokens are a type of [personal access token](../user/profile/perso They can be created only by an administrator, and are used to authenticate with the API as a specific user. -Use impersonation tokens an alternative to: +Use impersonation tokens as an alternative to: - The user's password or one of their personal access tokens. - The [Sudo](#sudo) feature. The user's or administrator's password or token @@ -283,7 +278,7 @@ message with a status code of `403`: ``` If an access token without the `sudo` scope is provided, an error message is -be returned with a status code of `403`: +returned with a status code of `403`: ```json { diff --git a/doc/api/instance_clusters.md b/doc/api/instance_clusters.md index 85046388275..f29ac5cd7f2 100644 --- a/doc/api/instance_clusters.md +++ b/doc/api/instance_clusters.md @@ -97,7 +97,6 @@ Example response: ... } ] - ``` ## Get a single instance cluster @@ -184,7 +183,6 @@ curl --header "Private-Token:<your_access_token>" "http://gitlab.example.com/api -H "Accept:application/json" \ -H "Content-Type:application/json" \ -X POST --data '{"name":"cluster-3", "environment_scope":"production", "platform_kubernetes_attributes":{"api_url":"https://example.com", "token":"12345", "ca_cert":"-----BEGIN CERTIFICATE-----qpoeiXXZafCM0ZDJkZjM...-----END CERTIFICATE-----"}}' - ``` Example response: @@ -255,7 +253,6 @@ Example request: curl --header "Private-Token: <your_access_token>" "http://gitlab.example.com/api/v4/admin/clusters/9" \ -H "Content-Type:application/json" \ -X PUT --data '{"name":"update-cluster-name", "platform_kubernetes_attributes":{"api_url":"https://new-example.com","token":"new-token"}}' - ``` Example response: @@ -290,7 +287,6 @@ Example response: "management_project": null, "project": null } - ``` ## Delete instance cluster diff --git a/doc/api/invitations.md b/doc/api/invitations.md index 80a05f8ea0d..0bf9d106404 100644 --- a/doc/api/invitations.md +++ b/doc/api/invitations.md @@ -42,9 +42,8 @@ POST /projects/:id/invitations | `access_level` | integer | yes | A valid access level | | `expires_at` | string | no | A date string in the format YEAR-MONTH-DAY | | `invite_source` | string | no | The source of the invitation that starts the member creation process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/327120). | -| `areas_of_focus` | string | no | Areas the inviter wants the member to focus upon. | -| `tasks_to_be_done` | array of strings | no | Tasks the inviter wants the member to focus on. The tasks are added as issues to a specified project. The possible values are: `ci`, `code` and `issues`. If specified, requires `tasks_project_id`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69299) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `invite_members_for_task`. Disabled by default. | -| `tasks_project_id` | integer | no | The project ID in which to create the task issues. If specified, requires `tasks_to_be_done`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69299) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `invite_members_for_task`. Disabled by default. | +| `tasks_to_be_done` | array of strings | no | Tasks the inviter wants the member to focus on. The tasks are added as issues to a specified project. The possible values are: `ci`, `code` and `issues`. If specified, requires `tasks_project_id`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69299) in GitLab 14.6 | +| `tasks_project_id` | integer | no | The project ID in which to create the task issues. If specified, requires `tasks_to_be_done`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69299) in GitLab 14.6 | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ diff --git a/doc/api/issues_statistics.md b/doc/api/issues_statistics.md index a760424f6a2..11f24d94763 100644 --- a/doc/api/issues_statistics.md +++ b/doc/api/issues_statistics.md @@ -49,7 +49,7 @@ GET /issues_statistics?confidential=true | `created_before` | datetime | no | Return issues created on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `updated_after` | datetime | no | Return issues updated on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `updated_before` | datetime | no | Return issues updated on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `confidential` | boolean | no | Filter confidential or public issues. | +| `confidential` | boolean | no | Filter confidential or public issues. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/issues_statistics" @@ -105,7 +105,7 @@ GET /groups/:id/issues_statistics?confidential=true | `created_before` | datetime | no | Return issues created on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `updated_after` | datetime | no | Return issues updated on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `updated_before` | datetime | no | Return issues updated on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `confidential` | boolean | no | Filter confidential or public issues. | +| `confidential` | boolean | no | Filter confidential or public issues. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/4/issues_statistics" @@ -161,7 +161,7 @@ GET /projects/:id/issues_statistics?confidential=true | `created_before` | datetime | no | Return issues created on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `updated_after` | datetime | no | Return issues updated on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `updated_before` | datetime | no | Return issues updated on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `confidential` | boolean | no | Filter confidential or public issues. | +| `confidential` | boolean | no | Filter confidential or public issues. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/4/issues_statistics" diff --git a/doc/api/job_artifacts.md b/doc/api/job_artifacts.md index 6d8c256d5aa..7c7847bf368 100644 --- a/doc/api/job_artifacts.md +++ b/doc/api/job_artifacts.md @@ -252,6 +252,7 @@ Example response: "finished_at": "2016-01-11T10:15:10.506Z", "duration": 97.0, "status": "failed", + "failure_reason": "script_failure", "tag": false, "web_url": "https://example.com/foo/bar/-/jobs/42", "user": null diff --git a/doc/api/jobs.md b/doc/api/jobs.md index 2a07e2d92c5..8dcd898b8c3 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -71,6 +71,7 @@ Example of response "runner": null, "stage": "test", "status": "failed", + "failure_reason": "script_failure", "tag": false, "web_url": "https://example.com/foo/bar/-/jobs/7", "user": { @@ -126,6 +127,7 @@ Example of response "runner": null, "stage": "test", "status": "failed", + "failure_reason": "stuck_or_timeout_failure", "tag": false, "web_url": "https://example.com/foo/bar/-/jobs/6", "user": { @@ -207,6 +209,7 @@ Example of response "runner": null, "stage": "test", "status": "failed", + "failure_reason": "stuck_or_timeout_failure", "tag": false, "web_url": "https://example.com/foo/bar/-/jobs/6", "user": { @@ -271,6 +274,7 @@ Example of response "runner": null, "stage": "test", "status": "failed", + "failure_reason": "script_failure", "tag": false, "web_url": "https://example.com/foo/bar/-/jobs/7", "user": { @@ -443,6 +447,7 @@ Example of response "runner": null, "stage": "test", "status": "failed", + "failure_reason": "script_failure", "tag": false, "web_url": "https://example.com/foo/bar/-/jobs/8", "user": { @@ -465,12 +470,12 @@ Example of response } ``` -## Get Kubernetes Agents by `CI_JOB_TOKEN` **(PREMIUM)** +## Get GitLab Agent by `CI_JOB_TOKEN` **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/324269) in GitLab 13.11. -Retrieve the job that generated the `CI_JOB_TOKEN`, along with a list of allowed GitLab -Kubernetes Agents. +Retrieve the job that generated the `CI_JOB_TOKEN`, along with a list of allowed +[agents](../user/clusters/agent/index.md). ```plaintext GET /job/allowed_agents @@ -801,6 +806,10 @@ Example of response } ``` +NOTE: +You can't delete archived jobs with the API, but you can +[delete job artifacts and logs from jobs completed before a specific date](../administration/job_artifacts.md#delete-job-artifacts-and-logs-from-jobs-completed-before-a-specific-date) + ## Play a job Triggers a manual action to start a job. diff --git a/doc/api/labels.md b/doc/api/labels.md index a8cb56f1573..5de227c8e1f 100644 --- a/doc/api/labels.md +++ b/doc/api/labels.md @@ -119,7 +119,7 @@ GET /projects/:id/labels/:label_id | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `label_id` | integer or string | yes | The ID or title of a project's label. | | `include_ancestor_groups` | boolean | no | Include ancestor groups. Defaults to `true`. | @@ -195,7 +195,7 @@ DELETE /projects/:id/labels/:label_id | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `label_id` | integer or string | yes | The ID or title of a group's label. | ```shell @@ -216,7 +216,7 @@ PUT /projects/:id/labels/:label_id | Attribute | Type | Required | Description | | --------------- | ------- | --------------------------------- | ------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `label_id` | integer or string | yes | The ID or title of a group's label. | | `new_name` | string | yes if `color` is not provided | The new name of the label | | `color` | string | yes if `new_name` is not provided | The color of the label given in 6-digit hex notation with leading '#' sign (for example, #FFAABB) or one of the [CSS color names](https://developer.mozilla.org/en-US/docs/Web/CSS/color_value#Color_keywords) | @@ -265,7 +265,7 @@ PUT /projects/:id/labels/:label_id/promote | Attribute | Type | Required | Description | | --------------- | ------- | --------------------------------- | ------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `label_id` | integer or string | yes | The ID or title of a group's label. | ```shell @@ -303,7 +303,7 @@ POST /projects/:id/labels/:label_id/subscribe | Attribute | Type | Required | Description | | ---------- | ----------------- | -------- | ------------------------------------ | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `label_id` | integer or string | yes | The ID or title of a project's label | ```shell @@ -341,7 +341,7 @@ POST /projects/:id/labels/:label_id/unsubscribe | Attribute | Type | Required | Description | | ---------- | ----------------- | -------- | ------------------------------------ | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `label_id` | integer or string | yes | The ID or title of a project's label | ```shell diff --git a/doc/api/managed_licenses.md b/doc/api/managed_licenses.md index 3967fe52a03..f2626574bf0 100644 --- a/doc/api/managed_licenses.md +++ b/doc/api/managed_licenses.md @@ -6,6 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Managed Licenses API **(ULTIMATE)** +WARNING: +"approval" and "blacklisted" approval statuses are deprecated and scheduled to be changed to "allowed" and "denied" in GitLab 15.0. + ## List managed licenses Get all managed licenses for a given project. @@ -78,10 +81,10 @@ POST /projects/:id/managed_licenses | ------------- | ------- | -------- | ---------------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `name` | string | yes | The name of the managed license | -| `approval_status` | string | yes | The approval status. "approved" or "blacklisted" | +| `approval_status` | string | yes | The approval status of the license. "allowed" or "denied". "blacklisted" and "approved" are deprecated. | ```shell -curl --data "name=MIT&approval_status=blacklisted" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses" +curl --data "name=MIT&approval_status=denied" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses" ``` Example response: @@ -125,10 +128,10 @@ PATCH /projects/:id/managed_licenses/:managed_license_id | --------------- | ------- | --------------------------------- | ------------------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `managed_license_id` | integer/string | yes | The ID or URL-encoded name of the license belonging to the project | -| `approval_status` | string | yes | The approval status. "approved" or "blacklisted" | +| `approval_status` | string | yes | The approval status of the license. "allowed" or "denied". "blacklisted" and "approved" are deprecated. | ```shell -curl --request PATCH --data "approval_status=blacklisted" \ +curl --request PATCH --data "approval_status=denied" \ --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses/6" ``` diff --git a/doc/api/members.md b/doc/api/members.md index ce276487f21..bc476980d2d 100644 --- a/doc/api/members.md +++ b/doc/api/members.md @@ -65,7 +65,8 @@ Example response: "web_url": "http://192.168.1.8:3000/root", "expires_at": "2012-10-22T14:13:35Z", "access_level": 30, - "group_saml_identity": null + "group_saml_identity": null, + "membership_state": "active" }, { "id": 2, @@ -81,7 +82,8 @@ Example response: "extern_uid":"ABC-1234567890", "provider": "group_saml", "saml_provider_id": 10 - } + }, + "membership_state": "active" } ] ``` @@ -107,6 +109,7 @@ GET /projects/:id/members/all | `id` | integer/string | yes | The ID or [URL-encoded path of the project or group](index.md#namespaced-path-encoding) owned by the authenticated user | | `query` | string | no | A query string to search for members | | `user_ids` | array of integers | no | Filter the results on the given user IDs | +| `state` | string | no | Filter results by member state, one of `awaiting`, `active` or `created` **(PREMIUM)** | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/members/all" @@ -126,7 +129,8 @@ Example response: "web_url": "http://192.168.1.8:3000/root", "expires_at": "2012-10-22T14:13:35Z", "access_level": 30, - "group_saml_identity": null + "group_saml_identity": null, + "membership_state": "active" }, { "id": 2, @@ -142,7 +146,8 @@ Example response: "extern_uid":"ABC-1234567890", "provider": "group_saml", "saml_provider_id": 10 - } + }, + "membership_state": "active" }, { "id": 3, @@ -153,7 +158,8 @@ Example response: "web_url": "http://192.168.1.8:3000/root", "expires_at": "2012-11-22T14:13:35Z", "access_level": 30, - "group_saml_identity": null + "group_saml_identity": null, + "membership_state": "active" } ] ``` @@ -191,7 +197,8 @@ Example response: "email": "john@example.com", "created_at": "2012-10-22T14:13:35Z", "expires_at": null, - "group_saml_identity": null + "group_saml_identity": null, + "membership_state": "active" } ``` @@ -229,7 +236,8 @@ Example response: "access_level": 30, "email": "john@example.com", "expires_at": null, - "group_saml_identity": null + "group_saml_identity": null, + "membership_state": "active" } ``` @@ -421,7 +429,6 @@ POST /projects/:id/members | `access_level` | integer | yes | A valid access level | | `expires_at` | string | no | A date string in the format `YEAR-MONTH-DAY` | | `invite_source` | string | no | The source of the invitation that starts the member creation process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/327120). | -| `areas_of_focus` | string | no | Areas the inviter wants the member to focus upon. | | `tasks_to_be_done` | array of strings | no | Tasks the inviter wants the member to focus on. The tasks are added as issues to a specified project. The possible values are: `ci`, `code` and `issues`. If specified, requires `tasks_project_id`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69299) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `invite_members_for_task`. Disabled by default. | | `tasks_project_id` | integer | no | The project ID in which to create the task issues. If specified, requires `tasks_to_be_done`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69299) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `invite_members_for_task`. Disabled by default. | @@ -589,26 +596,26 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/:id/members/:user_id" ``` -## Approve a member for a group +## Approve a member for a group -Approves a pending user for a group and its subgroups and projects. +Approves a pending user for a group and its subgroups and projects. ```plaintext -PUT /groups/:id/members/:user_id/approve +PUT /groups/:id/members/:member_id/approve ``` | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the root group](index.md#namespaced-path-encoding) owned by the authenticated user | -| `user_id` | integer | yes | The user ID of the member | +| `member_id` | integer | yes | The ID of the member | Example request: ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/members/:user_id/approve" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/members/:member_id/approve" ``` -## Approve all pending members for a group +## Approve all pending members for a group Approves all pending users for a group and its subgroups and projects. @@ -626,6 +633,65 @@ Example request: curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/members/approve_all" ``` +## List pending members of a group and its subgroups and projects + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332596) in GitLab 14.6. + +For a group and its subgroups and projects, get a list of all members in an `awaiting` state and those who are invited but do not have a GitLab account. + +This request returns all matching group and project members from all groups and projects in the root group's hierarchy. + +When the member is an invited user that has not signed up for a GitLab account yet, the invited email address is returned. + +This API endpoint works on top-level groups only. It does not work on subgroups. + +This API endpoint requires permission to administer members for the group. + +This API endpoint takes [pagination](index.md#pagination) parameters `page` and `per_page` to restrict the list of members. + +```plaintext +GET /groups/:id/pending_members +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/pending_members" +``` + +Example response: + +```json +[ + { + "id": 168, + "name": "Alex Garcia", + "username": "alex_garcia", + "email": "alex@example.com", + "avatar_url": "http://example.com/uploads/user/avatar/1/cd8.jpeg", + "web_url": "http://example.com/alex_garcia", + "approved": false, + "invited": false + }, + { + "id": 169, + "email": "sidney@example.com", + "avatar_url": "http://gravatar.com/../e346561cd8.jpeg", + "approved": false, + "invited": true + }, + { + "id": 170, + "email": "zhang@example.com", + "avatar_url": "http://gravatar.com/../e32131cd8.jpeg", + "approved": true, + "invited": true + } +] +``` + ## Give a group access to a project See [share project with group](projects.md#share-project-with-group) diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index 4ede95ea189..b6021d494fd 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -294,7 +294,12 @@ POST /projects/:id/approval_rules | `rule_type` | string | no | The type of rule. `any_approver` is a pre-configured default rule with `approvals_required` at `0`. Other rules are `regular`. | `user_ids` | Array | no | The ids of users as approvers | | `group_ids` | Array | no | The ids of groups as approvers | -| `protected_branch_ids` | Array | no | **(PREMIUM)** The ids of protected branches to scope the rule by. To identify the ID, [use the API](protected_branches.md#list-protected-branches). | +| `protected_branch_ids` | Array | no | The IDs of protected branches to scope the rule by. To identify the ID, [use the API](protected_branches.md#list-protected-branches). | +| `report_type` | string | no | The report type required when the rule type is `report_approver`. The supported report types are: `vulnerability`, `license_scanning`, `code_coverage`. | +| `scanners` | Array | no | The security scanners the `Vulnerability-Check` approval rule considers. The supported scanners are: `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing`. Defaults to all supported scanners. | +| `severity_levels` | Array | no | The severity levels the `Vulnerability-Check` approval rule considers. The supported severity levels are: `info`, `unknown`, `low`, `medium`, `high`, `critical`. Defaults to `unknown`, `high`, and `critical`. | +| `vulnerabilities_allowed` | integer | no | The number of vulnerabilities allowed for the `Vulnerability-Check` approval rule. Defaults to `0`. | +| `vulnerability_states` | Array | no | The vulnerability states the `Vulnerability-Check` approval rule considers. The supported vulnerability states are: `newly_detected` (default), `detected`, `confirmed`, `resolved`, `dismissed`. | ```json { @@ -417,7 +422,11 @@ PUT /projects/:id/approval_rules/:approval_rule_id | `approvals_required` | integer | yes | The number of required approvals for this rule | | `user_ids` | Array | no | The ids of users as approvers | | `group_ids` | Array | no | The ids of groups as approvers | -| `protected_branch_ids` | Array | no | **(PREMIUM)** The ids of protected branches to scope the rule by. To identify the ID, [use the API](protected_branches.md#list-protected-branches). | +| `protected_branch_ids` | Array | no | The IDs of protected branches to scope the rule by. To identify the ID, [use the API](protected_branches.md#list-protected-branches). | +| `scanners` | Array | no | The security scanners the `Vulnerability-Check` approval rule considers. The supported scanners are: `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing`. Defaults to all supported scanners. | +| `severity_levels` | Array | no | The severity levels the `Vulnerability-Check` approval rule considers. The supported severity levels are: `info`, `unknown`, `low`, `medium`, `high`, `critical`. Defaults to `unknown`, `high`, and `critical`. | +| `vulnerabilities_allowed` | integer | no | The number of vulnerabilities allowed for the `Vulnerability-Check` approval rule. Defaults to `0`. | +| `vulnerability_states` | Array | no | The vulnerability states the `Vulnerability-Check` approval rule considers. The supported vulnerability states are: `newly_detected` (default), `detected`, `confirmed`, `resolved`, `dismissed`. | ```json { diff --git a/doc/api/merge_request_context_commits.md b/doc/api/merge_request_context_commits.md index b40d67ab4e3..9984c5abb70 100644 --- a/doc/api/merge_request_context_commits.md +++ b/doc/api/merge_request_context_commits.md @@ -17,8 +17,10 @@ GET /projects/:id/merge_requests/:merge_request_iid/context_commits Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `merge_request_iid` (required) - The internal ID of the merge request +| Attribute | Type | Required | Description | +|---------------------|---------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `merge_request_iid` | integer | yes | The internal ID of the merge request | ```json [ @@ -49,8 +51,10 @@ POST /projects/:id/merge_requests/:merge_request_iid/context_commits Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `merge_request_iid` (required) - The internal ID of the merge request +| Attribute | Type | Required | Description | +|---------------------|---------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `merge_request_iid` | integer | yes | The internal ID of the merge request | ```plaintext POST /projects/:id/merge_requests/ @@ -88,9 +92,8 @@ DELETE /projects/:id/merge_requests/:merge_request_iid/context_commits Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `merge_request_iid` (required) - The internal ID of the merge request - -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `commits` | string array | yes | The context commits' SHA | +| Attribute | Type | Required | Description | +|---------------------|--------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `merge_request_iid` | integer | yes | The internal ID of the merge request | +| `commits` | string array | yes | The context commits' SHA | diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 98af228a064..fa713558684 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -1,15 +1,11 @@ --- stage: Create group: Code Review -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference, api +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Merge requests API **(FREE)** -> - `author_id`, `author_username`, and `assignee_id` were [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13060) in GitLab 9.5. -> - `my_reaction_emoji` was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14016) in GitLab 10.0. -> - For the `scope` attribute, `created-by-me` and `assigned-to-me` were [deprecated](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18935) in favor of `created_by_me` and `assigned_to_me` in GitLab 11.0. > - `with_labels_details` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21413) in GitLab 12.7. > - `author_username` and `author_username` were [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13060) in GitLab 12.10. > - `reference` was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20354) in GitLab 12.10 in favour of `references`. @@ -52,8 +48,6 @@ still apply. ## List merge requests -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13060) in GitLab 9.5. - Get all merge requests the authenticated user has access to. By default it returns only merge requests created by the current user. To get all merge requests, use parameter `scope=all`. @@ -85,7 +79,7 @@ Parameters: | `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc`. | | `milestone` | string | no | Return merge requests for a specific milestone. `None` returns merge requests with no milestone. `Any` returns merge requests that have an assigned milestone. | | `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request. | -| `labels` | string | no | Return merge requests matching a comma separated list of labels. `None` lists all merge requests with no labels. `Any` lists all merge requests with at least one label. `No+Label` (Deprecated) lists all merge requests with no labels. Predefined names are case-insensitive. | +| `labels` | string | no | Return merge requests matching a comma-separated list of labels. `None` lists all merge requests with no labels. `Any` lists all merge requests with at least one label. `No+Label` (Deprecated) lists all merge requests with no labels. Predefined names are case-insensitive. | | `with_labels_details` | boolean | no | If `true`, response returns more details for each label in labels field: `:name`, `:color`, `:description`, `:description_html`, `:text_color`. Default is `false`. | | `with_merge_status_recheck` | boolean | no | If `true`, this projection requests (but does not guarantee) that the `merge_status` field be recalculated asynchronously. Default is `false`. | | `created_after` | datetime | no | Return merge requests created on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | @@ -272,7 +266,7 @@ Parameters: | `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc`. | | `milestone` | string | no | Return merge requests for a specific milestone. `None` returns merge requests with no milestone. `Any` returns merge requests that have an assigned milestone. | | `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request. | -| `labels` | string | no | Return merge requests matching a comma separated list of labels. `None` lists all merge requests with no labels. `Any` lists all merge requests with at least one label. `No+Label` (Deprecated) lists all merge requests with no labels. Predefined names are case-insensitive. | +| `labels` | string | no | Return merge requests matching a comma-separated list of labels. `None` lists all merge requests with no labels. `Any` lists all merge requests with at least one label. `No+Label` (Deprecated) lists all merge requests with no labels. Predefined names are case-insensitive. | | `with_labels_details` | boolean | no | If `true`, response returns more details for each label in labels field: `:name`, `:color`, `:description`, `:description_html`, `:text_color`. Default is `false`. | | `with_merge_status_recheck` | boolean | no | If `true`, this projection requests (but does not guarantee) that the `merge_status` field be recalculated asynchronously. Default is `false`. | | `created_after` | datetime | no | Return merge requests created on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | @@ -459,7 +453,7 @@ Parameters: | `sort` | string | no | Return merge requests sorted in `asc` or `desc` order. Default is `desc`. | | `milestone` | string | no | Return merge requests for a specific milestone. `None` returns merge requests with no milestone. `Any` returns merge requests that have an assigned milestone. | | `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request. | -| `labels` | string | no | Return merge requests matching a comma separated list of labels. `None` lists all merge requests with no labels. `Any` lists all merge requests with at least one label. `No+Label` (Deprecated) lists all merge requests with no labels. Predefined names are case-insensitive. | +| `labels` | string | no | Return merge requests matching a comma-separated list of labels. `None` lists all merge requests with no labels. `Any` lists all merge requests with at least one label. `No+Label` (Deprecated) lists all merge requests with no labels. Predefined names are case-insensitive. | | `with_labels_details` | boolean | no | If `true`, response returns more details for each label in labels field: `:name`, `:color`, `:description`, `:description_html`, `:text_color`. Default is `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21413) in GitLab 12.7.| | `with_merge_status_recheck` | boolean | no | If `true`, this projection requests (but does not guarantee) that the `merge_status` field be recalculated asynchronously. Default is `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31890) in GitLab 13.0. | | `created_after` | datetime | no | Return merge requests created on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). | @@ -467,14 +461,14 @@ Parameters: | `updated_after` | datetime | no | Return merge requests updated on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). | | `updated_before` | datetime | no | Return merge requests updated on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). | | `scope` | string | no | Return merge requests for the given scope: `created_by_me`, `assigned_to_me` or `all`. | -| `author_id` | integer | no | Returns merge requests created by the given user `id`. Mutually exclusive with `author_username`. _([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13060) in GitLab 9.5)_. | +| `author_id` | integer | no | Returns merge requests created by the given user `id`. Mutually exclusive with `author_username`. | | `author_username` | string | no | Returns merge requests created by the given `username`. Mutually exclusive with `author_id`. _([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13060) in GitLab 12.10)_. | -| `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. _([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13060) in GitLab 9.5)_. | +| `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. | | `approver_ids` **(PREMIUM)** | integer array | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | | `approved_by_ids` **(PREMIUM)** | integer array | no | Returns merge requests which have been approved by all the users with the given `id`s (Max: 5). `None` returns merge requests with no approvals. `Any` returns merge requests with an approval. | | `reviewer_id` | integer | no | Returns merge requests which have the user as a [reviewer](../user/project/merge_requests/getting_started.md#reviewer) with the given user `id`. `None` returns merge requests with no reviewers. `Any` returns merge requests with any reviewer. Mutually exclusive with `reviewer_username`. | | `reviewer_username` | string | no | Returns merge requests which have the user as a [reviewer](../user/project/merge_requests/getting_started.md#reviewer) with the given `username`. `None` returns merge requests with no reviewers. `Any` returns merge requests with any reviewer. Mutually exclusive with `reviewer_id`. | -| `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14016) in GitLab 10.0)_. | +| `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. | | `source_branch` | string | no | Return merge requests with the given source branch. | | `target_branch` | string | no | Return merge requests with the given target branch. | | `search` | string | no | Search merge requests against their `title` and `description`. | @@ -976,8 +970,6 @@ Parameters: ## List MR pipelines -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/15454) in GitLab 10.5. - Get a list of merge request pipelines. ```plaintext diff --git a/doc/api/milestones.md b/doc/api/milestones.md index 84b4e2fe39d..3c1e09eaace 100644 --- a/doc/api/milestones.md +++ b/doc/api/milestones.md @@ -27,7 +27,7 @@ Parameters: | Attribute | Type | Required | Description | | ---------------------------- | ------ | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `iids[]` | integer array | optional | Return only the milestones having the given `iid` (Note: ignored if `include_parent_milestones` is set as `true`) | | `state` | string | optional | Return only `active` or `closed` milestones | | `title` | string | optional | Return only the milestones having the given `title` | @@ -68,8 +68,10 @@ GET /projects/:id/milestones/:milestone_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `milestone_id` (required) - The ID of the project's milestone +| Attribute | Type | Required | Description | +|----------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `milestone_id` | integer | yes | The ID of the project's milestone | ## Create new milestone @@ -81,11 +83,13 @@ POST /projects/:id/milestones Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `title` (required) - The title of a milestone -- `description` (optional) - The description of the milestone -- `due_date` (optional) - The due date of the milestone -- `start_date` (optional) - The start date of the milestone +| Attribute | Type | Required | Description | +|---------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `title` | string | yes | The title of a milestone | +| `description` | string | no | The description of the milestone | +| `due_date` | string | no | The due date of the milestone | +| `start_date` | string | no | The start date of the milestone | ## Edit milestone @@ -97,13 +101,15 @@ PUT /projects/:id/milestones/:milestone_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `milestone_id` (required) - The ID of a project milestone -- `title` (optional) - The title of a milestone -- `description` (optional) - The description of a milestone -- `due_date` (optional) - The due date of the milestone -- `start_date` (optional) - The start date of the milestone -- `state_event` (optional) - The state event of the milestone (close or activate) +| Attribute | Type | Required | Description | +|----------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `milestone_id` | integer | yes | The ID of the project's milestone | +| `title` | string | no | The title of a milestone | +| `description` | string | no | The description of the milestone | +| `due_date` | string | no | The due date of the milestone | +| `start_date` | string | no | The start date of the milestone | +| `state_event` | string | no | The state event of the milestone (close or activate) | ## Delete project milestone @@ -115,8 +121,10 @@ DELETE /projects/:id/milestones/:milestone_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `milestone_id` (required) - The ID of the project's milestone +| Attribute | Type | Required | Description | +|----------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `milestone_id` | integer | yes | The ID of the project's milestone | ## Get all issues assigned to a single milestone @@ -128,8 +136,10 @@ GET /projects/:id/milestones/:milestone_id/issues Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `milestone_id` (required) - The ID of a project milestone +| Attribute | Type | Required | Description | +|----------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `milestone_id` | integer | yes | The ID of the project's milestone | ## Get all merge requests assigned to a single milestone @@ -141,8 +151,10 @@ GET /projects/:id/milestones/:milestone_id/merge_requests Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `milestone_id` (required) - The ID of a project milestone +| Attribute | Type | Required | Description | +|----------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `milestone_id` | integer | yes | The ID of the project's milestone | ## Promote project milestone to a group milestone @@ -156,8 +168,10 @@ POST /projects/:id/milestones/:milestone_id/promote Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `milestone_id` (required) - The ID of a project milestone +| Attribute | Type | Required | Description | +|----------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `milestone_id` | integer | yes | The ID of the project's milestone | ## Get all burndown chart events for a single milestone **(PREMIUM)** @@ -172,5 +186,7 @@ GET /projects/:id/milestones/:milestone_id/burndown_events Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `milestone_id` (required) - The ID of a project milestone +| Attribute | Type | Required | Description | +|----------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `milestone_id` | integer | yes | The ID of the project's milestone | diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index 03aefaf4380..9a52b0983a7 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -4,7 +4,7 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Namespaces API +# Namespaces API **(FREE)** Usernames and group names fall under a special category called [namespaces](../user/group/index.md#namespaces). diff --git a/doc/api/notes.md b/doc/api/notes.md index 6a3db0a2aab..879ffaca191 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -54,7 +54,7 @@ GET /projects/:id/issues/:issue_iid/notes?sort=asc&order_by=updated_at | Attribute | Type | Required | Description | | ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | `issue_iid` | integer | yes | The IID of an issue | `sort` | string | no | Return issue notes sorted in `asc` or `desc` order. Default is `desc` | `order_by` | string | no | Return issue notes ordered by `created_at` or `updated_at` fields. Default is `created_at` @@ -120,9 +120,11 @@ GET /projects/:id/issues/:issue_iid/notes/:note_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) -- `issue_iid` (required) - The IID of a project issue -- `note_id` (required) - The ID of an issue note +| Attribute | Type | Required | Description | +|-------------|----------------|----------|---------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `issue_iid` | integer | yes | The IID of a project issue | +| `note_id` | integer | yes | The ID of an issue note | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/issues/11/notes/1" @@ -138,11 +140,13 @@ POST /projects/:id/issues/:issue_iid/notes Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) -- `issue_iid` (required) - The IID of an issue -- `body` (required) - The content of a note. Limited to 1,000,000 characters. -- `confidential` (optional) - The confidential flag of a note. Default is false. -- `created_at` (optional) - Date time string, ISO 8601 formatted. Example: `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) +| Attribute | Type | Required | Description | +|----------------|----------------|----------|------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | +| `issue_iid` | integer | yes | The IID of an issue. | +| `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | +| `confidential` | boolean | no | The confidential flag of a note. Default is false. | +| `created_at` | string | no | Date time string, ISO 8601 formatted. Example: `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/issues/11/notes?body=note" @@ -158,11 +162,13 @@ PUT /projects/:id/issues/:issue_iid/notes/:note_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). -- `issue_iid` (required) - The IID of an issue. -- `note_id` (required) - The ID of a note. -- `body` (optional) - The content of a note. Limited to 1,000,000 characters. -- `confidential` (optional) - The confidential flag of a note. +| Attribute | Type | Required | Description | +|----------------|----------------|----------|----------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | +| `issue_iid` | integer | yes | The IID of an issue. | +| `note_id` | integer | yes | The ID of a note. | +| `body` | string | no | The content of a note. Limited to 1,000,000 characters. | +| `confidential` | boolean | no | The confidential flag of a note. | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/issues/11/notes?body=note" @@ -180,7 +186,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | | `issue_iid` | integer | yes | The IID of an issue | | `note_id` | integer | yes | The ID of a note | @@ -203,7 +209,7 @@ GET /projects/:id/snippets/:snippet_id/notes?sort=asc&order_by=updated_at | Attribute | Type | Required | Description | | ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | `snippet_id` | integer | yes | The ID of a project snippet | `sort` | string | no | Return snippet notes sorted in `asc` or `desc` order. Default is `desc` | `order_by` | string | no | Return snippet notes ordered by `created_at` or `updated_at` fields. Default is `created_at` @@ -222,9 +228,11 @@ GET /projects/:id/snippets/:snippet_id/notes/:note_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) -- `snippet_id` (required) - The ID of a project snippet -- `note_id` (required) - The ID of a snippet note +| Attribute | Type | Required | Description | +|--------------|----------------|----------|---------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `snippet_id` | integer | yes | The ID of a project snippet | +| `note_id` | integer | yes | The ID of a snippet note | ```json { @@ -260,10 +268,12 @@ POST /projects/:id/snippets/:snippet_id/notes Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) -- `snippet_id` (required) - The ID of a snippet -- `body` (required) - The content of a note. Limited to 1,000,000 characters. -- `created_at` (optional) - Date time string, ISO 8601 formatted. Example: `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) +| Attribute | Type | Required | Description | +|--------------|----------------|----------|------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `snippet_id` | integer | yes | The ID of a snippet | +| `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | +| `created_at` | string | no | Date time string, ISO 8601 formatted. Example: `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/snippet/11/notes?body=note" @@ -279,10 +289,12 @@ PUT /projects/:id/snippets/:snippet_id/notes/:note_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) -- `snippet_id` (required) - The ID of a snippet -- `note_id` (required) - The ID of a note -- `body` (required) - The content of a note. Limited to 1,000,000 characters. +| Attribute | Type | Required | Description | +|--------------|----------------|----------|------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `snippet_id` | integer | yes | The ID of a snippet | +| `note_id` | integer | yes | The ID of a snippet note | +| `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/snippets/11/notes?body=note" @@ -300,7 +312,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | | `snippet_id` | integer | yes | The ID of a snippet | | `note_id` | integer | yes | The ID of a note | @@ -321,7 +333,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/notes?sort=asc&order_by=upda | Attribute | Type | Required | Description | | ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | `merge_request_iid` | integer | yes | The IID of a project merge request | `sort` | string | no | Return merge request notes sorted in `asc` or `desc` order. Default is `desc` | `order_by` | string | no | Return merge request notes ordered by `created_at` or `updated_at` fields. Default is `created_at` @@ -340,9 +352,11 @@ GET /projects/:id/merge_requests/:merge_request_iid/notes/:note_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) -- `merge_request_iid` (required) - The IID of a project merge request -- `note_id` (required) - The ID of a merge request note +| Attribute | Type | Required | Description | +|---------------------|----------------|----------|---------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `merge_request_iid` | integer | yes | The IID of a project merge request | +| `note_id` | integer | yes | The ID of a merge request note | ```json { @@ -383,10 +397,12 @@ POST /projects/:id/merge_requests/:merge_request_iid/notes Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) -- `merge_request_iid` (required) - The IID of a merge request -- `body` (required) - The content of a note. Limited to 1,000,000 characters. -- `created_at` (optional) - Date time string, ISO 8601 formatted. Example: `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) +| Attribute | Type | Required | Description | +|---------------------|----------------|----------|------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `merge_request_iid` | integer | yes | The IID of a project merge request | +| `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | +| `created_at` | string | no | Date time string, ISO 8601 formatted. Example: `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | ### Modify existing merge request note @@ -398,10 +414,12 @@ PUT /projects/:id/merge_requests/:merge_request_iid/notes/:note_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) -- `merge_request_iid` (required) - The IID of a merge request -- `note_id` (required) - The ID of a note -- `body` (required) - The content of a note. Limited to 1,000,000 characters. +| Attribute | Type | Required | Description | +|---------------------|----------------|----------|---------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `merge_request_iid` | integer | yes | The IID of a project merge request | +| `note_id` | integer | no | The ID of a note | +| `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/merge_requests/11/notes?body=note" @@ -419,7 +437,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of a merge request | | `note_id` | integer | yes | The ID of a note | @@ -440,7 +458,7 @@ GET /groups/:id/epics/:epic_id/notes?sort=asc&order_by=updated_at | Attribute | Type | Required | Description | | ------------------- | ---------------- | ---------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | | `epic_id` | integer | yes | The ID of a group epic | | `sort` | string | no | Return epic notes sorted in `asc` or `desc` order. Default is `desc` | | `order_by` | string | no | Return epic notes ordered by `created_at` or `updated_at` fields. Default is `created_at` | @@ -461,7 +479,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | -------------- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | | `epic_id` | integer | yes | The ID of an epic | | `note_id` | integer | yes | The ID of a note | @@ -502,7 +520,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | -------------- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | | `epic_id` | integer | yes | The ID of an epic | | `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | @@ -522,7 +540,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | -------------- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | | `epic_id` | integer | yes | The ID of an epic | | `note_id` | integer | yes | The ID of a note | | `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | @@ -543,7 +561,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | -------------- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | | `epic_id` | integer | yes | The ID of an epic | | `note_id` | integer | yes | The ID of a note | diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md index 8a8a54a753a..778c229e3c8 100644 --- a/doc/api/oauth2.md +++ b/doc/api/oauth2.md @@ -98,7 +98,7 @@ Before starting the flow, generate the `STATE`, the `CODE_VERIFIER` and the `COD This page asks the user to approve the request from the app to access their account based on the scopes specified in `REQUESTED_SCOPES`. The user is then redirected back to the specified `REDIRECT_URI`. The [scope parameter](https://github.com/doorkeeper-gem/doorkeeper/wiki/Using-Scopes#requesting-particular-scopes) - is a space separated list of scopes associated with the user. + is a space-separated list of scopes associated with the user. For example,`scope=read_user+profile` requests the `read_user` and `profile` scopes. The redirect includes the authorization `code`, for example: @@ -126,7 +126,7 @@ Before starting the flow, generate the `STATE`, the `CODE_VERIFIER` and the `COD "created_at": 1607635748 } ``` - + 1. To retrieve a new `access_token`, use the `refresh_token` parameter. Refresh tokens may be used even after the `access_token` itself expires. This request: - Invalidates the existing `access_token` and `refresh_token`. @@ -178,7 +178,7 @@ be used as a CSRF token. This page asks the user to approve the request from the app to access their account based on the scopes specified in `REQUESTED_SCOPES`. The user is then redirected back to the specified `REDIRECT_URI`. The [scope parameter](https://github.com/doorkeeper-gem/doorkeeper/wiki/Using-Scopes#requesting-particular-scopes) - is a space separated list of scopes associated with the user. + is a space-separated list of scopes associated with the user. For example,`scope=read_user+profile` requests the `read_user` and `profile` scopes. The redirect includes the authorization `code`, for example: @@ -206,7 +206,7 @@ be used as a CSRF token. "created_at": 1607635748 } ``` - + 1. To retrieve a new `access_token`, use the `refresh_token` parameter. Refresh tokens may be used even after the `access_token` itself expires. This request: - Invalidates the existing `access_token` and `refresh_token`. @@ -266,7 +266,7 @@ https://gitlab.example.com/oauth/authorize?client_id=APP_ID&redirect_uri=REDIREC This prompts the user to approve the applications access to their account based on the scopes specified in `REQUESTED_SCOPES` and then redirect back to the `REDIRECT_URI` you provided. The [scope parameter](https://github.com/doorkeeper-gem/doorkeeper/wiki/Using-Scopes#requesting-particular-scopes) - is a space separated list of scopes you want to have access to (for example, `scope=read_user+profile` + is a space-separated list of scopes you want to have access to (for example, `scope=read_user+profile` would request `read_user` and `profile` scopes). The redirect includes a fragment with `access_token` as well as token details in GET parameters, for example: diff --git a/doc/api/packages/nuget.md b/doc/api/packages/nuget.md index aee3a4fe542..f25a3a25754 100644 --- a/doc/api/packages/nuget.md +++ b/doc/api/packages/nuget.md @@ -287,12 +287,13 @@ Example response: Returns metadata for a specific package version: ```plaintext -GET <route-prefix>/metadata/:package_name/index +GET <route-prefix>/metadata/:package_name/:package_version ``` -| Attribute | Type | Required | Description | -| -------------- | ------ | -------- | ----------- | -| `package_name` | string | yes | The name of the package. | +| Attribute | Type | Required | Description | +| ----------------- | ------ | -------- | ----------- | +| `package_name` | string | yes | The name of the package. | +| `package_version` | string | yes | The version of the package. | ```shell curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/nuget/metadata/MyNuGetPkg/1.3.0.17" diff --git a/doc/api/personal_access_tokens.md b/doc/api/personal_access_tokens.md index 9c9551a5103..b51866fe9b1 100644 --- a/doc/api/personal_access_tokens.md +++ b/doc/api/personal_access_tokens.md @@ -40,8 +40,9 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a "scopes": [ "api" ], - "active": true, "user_id": 24, + "last_used_at": "2021-10-06T17:58:37.550Z", + "active": true, "expires_at": null } ] @@ -61,8 +62,9 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a "scopes": [ "api" ], - "active": true, "user_id": 3, + "last_used_at": "2021-10-06T17:58:37.550Z", + "active": true, "expires_at": null } ] diff --git a/doc/api/pipelines.md b/doc/api/pipelines.md index 4dddbbc7826..d850113f9b6 100644 --- a/doc/api/pipelines.md +++ b/doc/api/pipelines.md @@ -15,6 +15,8 @@ Read more on [pagination](index.md#pagination). ## List project pipelines +> `iid` in response [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342223) in GitLab 14.6. + List pipelines in a project. Child pipelines are not included in the results, but you can [get child pipeline](pipelines.md#get-a-single-pipeline) individually. @@ -50,7 +52,7 @@ Example of response "iid": 12, "project_id": 1, "status": "pending", - "soure": "push", + "source": "push", "ref": "new-pipeline", "sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "web_url": "https://example.com/foo/bar/pipelines/47", @@ -62,7 +64,7 @@ Example of response "iid": 13, "project_id": 1, "status": "pending", - "soure": "web", + "source": "web", "ref": "new-pipeline", "sha": "eb94b618fb5865b26e80fdd8ae531b7a63ad851a", "web_url": "https://example.com/foo/bar/pipelines/48", @@ -74,6 +76,8 @@ Example of response ## Get a single pipeline +> `iid` in response [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342223) in GitLab 14.6. + Get one pipeline from a project. You can also get a single [child pipeline](../ci/pipelines/parent_child_pipelines.md). @@ -267,6 +271,8 @@ Sample response: ## Create a new pipeline +> `iid` in response [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342223) in GitLab 14.6. + ```plaintext POST /projects/:id/pipeline ``` @@ -275,7 +281,7 @@ POST /projects/:id/pipeline |-------------|---------|----------|---------------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `ref` | string | yes | Reference to commit | -| `variables` | array | no | An array containing the variables available in the pipeline, matching the structure `[{ 'key': 'UPLOAD_TO_S3', 'variable_type': 'file', 'value': 'true' }]` | +| `variables` | array | no | An array containing the variables available in the pipeline, matching the structure `[{ 'key': 'UPLOAD_TO_S3', 'variable_type': 'file', 'value': 'true' }, {'key': 'TEST', 'value': 'test variable'}]`. If `variable_type` is excluded, it defaults to `env_var`. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/pipeline?ref=main" @@ -316,6 +322,8 @@ Example of response ## Retry jobs in a pipeline +> `iid` in response [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342223) in GitLab 14.6. + ```plaintext POST /projects/:id/pipelines/:pipeline_id/retry ``` diff --git a/doc/api/project_badges.md b/doc/api/project_badges.md index 2e4ab0e2b8c..c6f979c1643 100644 --- a/doc/api/project_badges.md +++ b/doc/api/project_badges.md @@ -11,7 +11,7 @@ type: reference, api ## Placeholder tokens -Badges support placeholders that are replaced in real time in both the link and image URL. The allowed placeholders are: +Badges support placeholders that are replaced in real-time in both the link and image URL. The allowed placeholders are: <!-- vale gitlab.Spelling = NO --> diff --git a/doc/api/project_clusters.md b/doc/api/project_clusters.md index d2a574b5cbd..129b064c650 100644 --- a/doc/api/project_clusters.md +++ b/doc/api/project_clusters.md @@ -384,7 +384,6 @@ Example response: } } } - ``` ## Delete project cluster diff --git a/doc/api/project_import_export.md b/doc/api/project_import_export.md index 92b1558551d..39c68041725 100644 --- a/doc/api/project_import_export.md +++ b/doc/api/project_import_export.md @@ -225,6 +225,7 @@ The passed override parameters take precedence over all values defined in the ex ```shell curl --request POST \ --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ --url "https://gitlab.example.com/api/v4/projects/remote-import" \ --data '{"url":"https://remoteobject/file?token=123123","path":"remote-project"}' ``` @@ -293,6 +294,7 @@ The `failed_relations` array is capped to 100 items. "path_with_namespace": "gitlab-org/gitlab-test", "created_at": "2017-08-29T04:36:44.383Z", "import_status": "started", + "import_type": "github", "correlation_id": "mezklWso3Za", "failed_relations": [ { @@ -301,8 +303,58 @@ The `failed_relations` array is capped to 100 items. "exception_class": "RuntimeError", "exception_message": "A failure occurred", "source": "custom error context", - "relation_name": "merge_requests" + "relation_name": "merge_requests", + "line_number": 0 } ] } ``` + +When importing from GitHub, the a `stats` field lists how many objects were already fetched from +GitHub and how many were already imported: + +```json +{ + "id": 1, + "description": "Itaque perspiciatis minima aspernatur corporis consequatur.", + "name": "Gitlab Test", + "name_with_namespace": "Gitlab Org / Gitlab Test", + "path": "gitlab-test", + "path_with_namespace": "gitlab-org/gitlab-test", + "created_at": "2017-08-29T04:36:44.383Z", + "import_status": "started", + "import_type": "github", + "correlation_id": "mezklWso3Za", + "failed_relations": [ + { + "id": 42, + "created_at": "2020-04-02T14:48:59.526Z", + "exception_class": "RuntimeError", + "exception_message": "A failure occurred", + "source": "custom error context", + "relation_name": "merge_requests", + "line_number": 0 + } + ], + "stats": { + "fetched": { + "diff_note": 19, + "issue": 3, + "label": 1, + "note": 3, + "pull_request": 2, + "pull_request_merged_by": 1, + "pull_request_review": 16 + }, + "imported": { + "diff_note": 19, + "issue": 3, + "label": 1, + "note": 3, + "pull_request": 2, + "pull_request_merged_by": 1, + "pull_request_review": 16 + } + } +} +``` diff --git a/doc/api/project_level_variables.md b/doc/api/project_level_variables.md index a2c2da9065f..2251b0fc7fd 100644 --- a/doc/api/project_level_variables.md +++ b/doc/api/project_level_variables.md @@ -15,7 +15,7 @@ Get list of a project's variables. GET /projects/:id/variables ``` -| Attribute | Type | required | Description | +| Attribute | Type | Required | Description | | --------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | @@ -52,7 +52,7 @@ Get the details of a project's specific variable. GET /projects/:id/variables/:key ``` -| Attribute | Type | required | Description | +| Attribute | Type | Required | Description | | --------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `key` | string | yes | The `key` of a variable | @@ -81,7 +81,7 @@ Create a new variable. POST /projects/:id/variables ``` -| Attribute | Type | required | Description | +| Attribute | Type | Required | Description | | ------------------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `key` | string | yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | @@ -115,7 +115,7 @@ Update a project's variable. PUT /projects/:id/variables/:key ``` -| Attribute | Type | required | Description | +| Attribute | Type | Required | Description | | ------------------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `key` | string | yes | The `key` of a variable | @@ -150,7 +150,7 @@ Remove a project's variable. DELETE /projects/:id/variables/:key ``` -| Attribute | Type | required | Description | +| Attribute | Type | Required | Description | | --------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `key` | string | yes | The `key` of a variable | diff --git a/doc/api/project_snippets.md b/doc/api/project_snippets.md index 0ac2297e3c1..569270e5de1 100644 --- a/doc/api/project_snippets.md +++ b/doc/api/project_snippets.md @@ -36,7 +36,9 @@ GET /projects/:id/snippets Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user +| Attribute | Type | Required | Description | +|-----------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | ## Single snippet @@ -48,8 +50,10 @@ GET /projects/:id/snippets/:snippet_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `snippet_id` (required) - The ID of a project's snippet +| Attribute | Type | Required | Description | +|--------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `snippet_id` | integer | yes | The ID of a project's snippet | ```json { @@ -85,7 +89,7 @@ Parameters: | Attribute | Type | Required | Description | |:------------------|:----------------|:---------|:----------------------------------------------------------------------------------------------------------------| -| `id` | integer | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `title` | string | yes | Title of a snippet | | `file_name` | string | no | Deprecated: Use `files` instead. Name of a snippet file | | `content` | string | no | Deprecated: Use `files` instead. Content of a snippet | @@ -132,7 +136,7 @@ Parameters: | Attribute | Type | Required | Description | |:----------------------|:----------------|:---------|:----------------------------------------------------------------------------------------------------------------| -| `id` | integer | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `snippet_id` | integer | yes | The ID of a project's snippet | | `title` | string | no | Title of a snippet | | `file_name` | string | no | Deprecated: Use `files` instead. Name of a snippet file | @@ -183,8 +187,10 @@ DELETE /projects/:id/snippets/:snippet_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `snippet_id` (required) - The ID of a project's snippet +| Attribute | Type | Required | Description | +|:-------------|:---------------|:---------|:----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `snippet_id` | integer | yes | The ID of a project's snippet | Example request: @@ -203,8 +209,10 @@ GET /projects/:id/snippets/:snippet_id/raw Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `snippet_id` (required) - The ID of a project's snippet +| Attribute | Type | Required | Description | +|:-------------|:---------------|:---------|:----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `snippet_id` | integer | yes | The ID of a project's snippet | Example request: @@ -223,10 +231,12 @@ GET /projects/:id/snippets/:snippet_id/files/:ref/:file_path/raw Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user -- `snippet_id` (required) - The ID of a project's snippet -- `ref` (required) - The name of a branch, tag or commit, such as `main` -- `file_path` (required) - The URL-encoded path to the file, such as `snippet%2Erb` +| Attribute | Type | Required | Description | +|:-------------|:---------------|:---------|:----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `snippet_id` | integer | yes | The ID of a project's snippet | +| `ref` | string | yes | The name of a branch, tag or commit e.g. master | +| `file_path` | string | yes | The URL-encoded path to the file, e.g. snippet%2Erb | Example request: @@ -245,10 +255,10 @@ Available only for users with the Administrator [role](../user/permissions.md). GET /projects/:id/snippets/:snippet_id/user_agent_detail ``` -| Attribute | Type | Required | Description | -|---------------|---------|----------|--------------------------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of a project](index.md#namespaced-path-encoding). | -| `snippet_id` | Integer | yes | The ID of a snippet | +| Attribute | Type | Required | Description | +|--------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `snippet_id` | Integer | yes | The ID of a snippet | Example request: diff --git a/doc/api/project_statistics.md b/doc/api/project_statistics.md index c69e41a423e..c0a1227b295 100644 --- a/doc/api/project_statistics.md +++ b/doc/api/project_statistics.md @@ -21,7 +21,7 @@ GET /projects/:id/statistics | Attribute | Type | Required | Description | | ---------- | ------ | -------- | ----------- | -| `id` | integer / string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | Example response: diff --git a/doc/api/project_templates.md b/doc/api/project_templates.md index d4af0e8d78d..2ec30c80a6b 100644 --- a/doc/api/project_templates.md +++ b/doc/api/project_templates.md @@ -33,7 +33,7 @@ GET /projects/:id/templates/:type | Attribute | Type | Required | Description | | ---------- | ------ | -------- | ----------- | -| `id` | integer / string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | | `type` | string | yes | The type of the template. Accepted values are: `dockerfiles`, `gitignores`, `gitlab_ci_ymls`, `licenses`, `issues`, `merge_requests` | Example response (licenses): @@ -99,7 +99,7 @@ GET /projects/:id/templates/:type/:name | Attribute | Type | Required | Description | | ---------- | ------ | -------- | ----------- | -| `id` | integer / string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | | `type` | string | yes| The type of the template. One of: `dockerfiles`, `gitignores`, `gitlab_ci_ymls`, `licenses`, `issues`, or `merge_requests`. | | `name` | string | yes | The key of the template, as obtained from the collection endpoint | | `source_template_project_id` | integer | no | The project ID where a given template is being stored. This is useful when multiple templates from different projects have the same name. If multiple templates have the same name, the match from `closest ancestor` is returned if `source_template_project_id` is not specified | diff --git a/doc/api/project_vulnerabilities.md b/doc/api/project_vulnerabilities.md index 7ba359587f6..1267f748633 100644 --- a/doc/api/project_vulnerabilities.md +++ b/doc/api/project_vulnerabilities.md @@ -10,9 +10,11 @@ type: reference, api > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10242) in GitLab 12.6. WARNING: -This API is in an alpha stage and considered unstable. +This API is in the process of being deprecated and considered unstable. The response payload may be subject to change or breakage -across GitLab releases. +across GitLab releases. Please use the +[GraphQL API](graphql/reference/index.md#queryvulnerabilities) +instead. Every API call to vulnerabilities must be [authenticated](index.md#authentication). diff --git a/doc/api/projects.md b/doc/api/projects.md index d19019c9597..65911567f87 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -183,6 +183,7 @@ When the user is authenticated and `simple` is not set this returns something li "autoclose_referenced_issues": true, "suggestion_commit_message": null, "merge_commit_template": null, + "squash_commit_template": null, "marked_for_deletion_at": "2020-04-03", // Deprecated and will be removed in API v5 in favor of marked_for_deletion_on "marked_for_deletion_on": "2020-04-03", "statistics": { @@ -300,6 +301,7 @@ When the user is authenticated and `simple` is not set this returns something li "autoclose_referenced_issues": true, "suggestion_commit_message": null, "merge_commit_template": null, + "squash_commit_template": null, "statistics": { "commit_count": 12, "storage_size": 2066080, @@ -369,6 +371,9 @@ Keyset pagination supports only `order_by=id`. Other sorting options aren't avai Get a list of visible projects owned by the given user. When accessed without authentication, only public projects are returned. +NOTE: +Only the projects in the user's (specified in `user_id`) namespace are returned. Projects owned by the user in any group or subgroups are not returned. + This endpoint supports [keyset pagination](index.md#keyset-based-pagination) for selected `order_by` options. @@ -467,6 +472,7 @@ GET /users/:user_id/projects "autoclose_referenced_issues": true, "suggestion_commit_message": null, "merge_commit_template": null, + "squash_commit_template": null, "marked_for_deletion_at": "2020-04-03", // Deprecated and will be removed in API v5 in favor of marked_for_deletion_on "marked_for_deletion_on": "2020-04-03", "statistics": { @@ -584,6 +590,7 @@ GET /users/:user_id/projects "autoclose_referenced_issues": true, "suggestion_commit_message": null, "merge_commit_template": null, + "squash_commit_template": null, "statistics": { "commit_count": 12, "storage_size": 2066080, @@ -711,6 +718,7 @@ Example response: "autoclose_referenced_issues": true, "suggestion_commit_message": null, "merge_commit_template": null, + "squash_commit_template": null, "statistics": { "commit_count": 37, "storage_size": 1038090, @@ -823,6 +831,7 @@ Example response: "autoclose_referenced_issues": true, "suggestion_commit_message": null, "merge_commit_template": null, + "squash_commit_template": null, "statistics": { "commit_count": 12, "storage_size": 2066080, @@ -991,6 +1000,7 @@ GET /projects/:id "autoclose_referenced_issues": true, "suggestion_commit_message": null, "merge_commit_template": null, + "squash_commit_template": null, "marked_for_deletion_at": "2020-04-03", // Deprecated and will be removed in API v5 in favor of marked_for_deletion_on "marked_for_deletion_on": "2020-04-03", "compliance_frameworks": [ "sox" ], @@ -1062,7 +1072,7 @@ If the project is a fork, and you provide a valid token to authenticate, the "ssh_url_to_repo":"git@gitlab.com:gitlab-org/gitlab-foss.git", "http_url_to_repo":"https://gitlab.com/gitlab-org/gitlab-foss.git", "web_url":"https://gitlab.com/gitlab-org/gitlab-foss", - "avatar_url":"https://assets.gitlab-static.net/uploads/-/system/project/avatar/13083/logo-extra-whitespace.png", + "avatar_url":"https://gitlab.com/uploads/-/system/project/avatar/13083/logo-extra-whitespace.png", "license_url": "https://gitlab.com/gitlab-org/gitlab/-/blob/master/LICENSE", "license": { "key": "mit", @@ -1228,8 +1238,10 @@ POST /projects | `jobs_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | | `lfs_enabled` | boolean | **{dotted-circle}** No | Enable LFS. | | `merge_method` | string | **{dotted-circle}** No | Set the [merge method](#project-merge-method) used. | +| `merge_pipelines_enabled` | boolean | **{dotted-circle}** No | Enable or disable merge pipelines. | | `merge_requests_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `merge_requests_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | +| `merge_trains_enabled` | boolean | **{dotted-circle}** No | Enable or disable merge trains. | | `mirror_trigger_builds` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirroring triggers builds. | | `mirror` **(PREMIUM)** | boolean | **{dotted-circle}** No | Enables pull mirroring in a project. | | `namespace_id` | integer | **{dotted-circle}** No | Namespace for the new project (defaults to the current user's namespace). | @@ -1252,7 +1264,7 @@ POST /projects | `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | | `squash_option` | string | **{dotted-circle}** No | One of `never`, `always`, `default_on`, or `default_off`. | | `tag_list` | array | **{dotted-circle}** No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | -| `template_name` | string | **{dotted-circle}** No | When used without `use_custom_template`, name of a [built-in project template](../user/project/working_with_projects.md#built-in-templates). When used with `use_custom_template`, name of a custom project template. | +| `template_name` | string | **{dotted-circle}** No | When used without `use_custom_template`, name of a [built-in project template](../user/project/working_with_projects.md#create-a-project-from-a-built-in-template). When used with `use_custom_template`, name of a custom project template. | | `template_project_id` **(PREMIUM)** | integer | **{dotted-circle}** No | When used with `use_custom_template`, project ID of a custom project template. This is preferable to using `template_name` since `template_name` may be ambiguous. | | `topics` | array | **{dotted-circle}** No | The list of topics for a project; put array of topics, that should be finally assigned to a project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | | `use_custom_template` **(PREMIUM)** | boolean | **{dotted-circle}** No | Use either custom [instance](../user/admin_area/custom_project_templates.md) or [group](../user/group/custom_project_templates.md) (with `group_with_project_templates_id`) project template. | @@ -1304,6 +1316,7 @@ POST /projects/user/:user_id | `jobs_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | | `lfs_enabled` | boolean | **{dotted-circle}** No | Enable LFS. | | `merge_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | +| `squash_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | | `merge_method` | string | **{dotted-circle}** No | Set the [merge method](#project-merge-method) used. | | `merge_requests_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `merge_requests_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | @@ -1331,7 +1344,7 @@ POST /projects/user/:user_id | `squash_option` | string | **{dotted-circle}** No | One of `never`, `always`, `default_on`, or `default_off`. | | `suggestion_commit_message` | string | **{dotted-circle}** No | The commit message used to apply merge request [suggestions](../user/project/merge_requests/reviews/suggestions.md). | | `tag_list` | array | **{dotted-circle}** No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | -| `template_name` | string | **{dotted-circle}** No | When used without `use_custom_template`, name of a [built-in project template](../user/project/working_with_projects.md#built-in-templates). When used with `use_custom_template`, name of a custom project template. | +| `template_name` | string | **{dotted-circle}** No | When used without `use_custom_template`, name of a [built-in project template](../user/project/working_with_projects.md#create-a-project-from-a-built-in-template). When used with `use_custom_template`, name of a custom project template. | | `topics` | array | **{dotted-circle}** No | The list of topics for the project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | | `use_custom_template` **(PREMIUM)** | boolean | **{dotted-circle}** No | Use either custom [instance](../user/admin_area/custom_project_templates.md) or [group](../user/group/custom_project_templates.md) (with `group_with_project_templates_id`) project template. | | `visibility` | string | **{dotted-circle}** No | See [project visibility level](#project-visibility-level). | @@ -1350,6 +1363,17 @@ where `password` is a public access key with the `api` scope enabled. PUT /projects/:id ``` +For example, to toggle the setting for +[shared runners on a GitLab.com project](../ci/runners/index.md): + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <your-token>" \ + --url 'https://gitlab.com/api/v4/projects/<your-project-ID>' \ + --data "shared_runners_enabled=true" # to turn off: "shared_runners_enabled=false" +``` + +Supported attributes: + | Attribute | Type | Required | Description | |-------------------------------------------------------------|----------------|------------------------|-------------| | `allow_merge_on_skipped_pipeline` | boolean | **{dotted-circle}** No | Set whether or not merge requests can be merged with skipped jobs. | @@ -1383,8 +1407,10 @@ PUT /projects/:id | `lfs_enabled` | boolean | **{dotted-circle}** No | Enable LFS. | | `merge_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | | `merge_method` | string | **{dotted-circle}** No | Set the [merge method](#project-merge-method) used. | +| `merge_pipelines_enabled` | boolean | **{dotted-circle}** No | Enable or disable merge pipelines. | | `merge_requests_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `merge_requests_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | +| `merge_trains_enabled` | boolean | **{dotted-circle}** No | Enable or disable merge trains. | | `mirror_overwrites_diverged_branches` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirror overwrites diverged branches. | | `mirror_trigger_builds` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirroring triggers builds. | | `mirror_user_id` **(PREMIUM)** | integer | **{dotted-circle}** No | User responsible for all the activity surrounding a pull mirror event. _(administrators only)_ | @@ -1397,8 +1423,9 @@ PUT /projects/:id | `packages_enabled` | boolean | **{dotted-circle}** No | Enable or disable packages repository feature. | | `pages_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled`, or `public`. | | `requirements_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled` or `public` | -| `restrict_user_defined_variables` | boolean | **{dotted-circle}** No | Allow only maintainers to pass user-defined variables when triggering a pipeline. For example when the pipeline is triggered in the UI, with the API, or by a trigger token. | +| `restrict_user_defined_variables` | boolean | **{dotted-circle}** No | Allow only users with the [Maintainer role](../user/permissions.md) to pass user-defined variables when triggering a pipeline. For example when the pipeline is triggered in the UI, with the API, or by a trigger token. | | `path` | string | **{dotted-circle}** No | Custom repository name for the project. By default generated based on name. | +| `printing_merge_request_link_enabled` | boolean | **{dotted-circle}** No | Show link to create/view merge request when pushing from the command line. | | `public_builds` | boolean | **{dotted-circle}** No | If `true`, jobs can be viewed by non-project members. | | `remove_source_branch_after_merge` | boolean | **{dotted-circle}** No | Enable `Delete source branch` option by default for all new merge requests. | | `repository_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | @@ -1410,6 +1437,7 @@ PUT /projects/:id | `show_default_award_emojis` | boolean | **{dotted-circle}** No | Show default award emojis. | | `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | +| `squash_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | | `squash_option` | string | **{dotted-circle}** No | One of `never`, `always`, `default_on`, or `default_off`. | | `suggestion_commit_message` | string | **{dotted-circle}** No | The commit message used to apply merge request suggestions. | | `tag_list` | array | **{dotted-circle}** No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | @@ -2077,9 +2105,13 @@ This endpoint: - Deletes a project including all associated resources (including issues and merge requests). +- In [GitLab 12.6](https://gitlab.com/gitlab-org/gitlab/-/issues/32935) and later, on + [Premium or higher](https://about.gitlab.com/pricing/) tiers, + [delayed project deletion](../user/project/settings/index.md#delayed-project-deletion) + is applied if enabled. - From [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/issues/220382) on [Premium or higher](https://about.gitlab.com/pricing/) tiers, group - administrators can [configure](../user/group/index.md#enable-delayed-project-removal) + administrators can [configure](../user/group/index.md#enable-delayed-project-deletion) projects within a group to be deleted after a delayed period. When enabled, actual deletion happens after the number of days specified in the [default deletion delay](../user/admin_area/settings/visibility_and_access_controls.md#default-deletion-delay). @@ -2087,7 +2119,7 @@ This endpoint: WARNING: The default behavior of [Delayed Project deletion](https://gitlab.com/gitlab-org/gitlab/-/issues/32935) in GitLab 12.6 was changed to [Immediate deletion](https://gitlab.com/gitlab-org/gitlab/-/issues/220382) -in GitLab 13.2, as discussed in [Enable delayed project removal](../user/group/index.md#enable-delayed-project-removal). +in GitLab 13.2, as discussed in [Enable delayed project deletion](../user/group/index.md#enable-delayed-project-deletion). ```plaintext DELETE /projects/:id @@ -2477,8 +2509,8 @@ GET /projects/:id/push_rule { "id": 1, "project_id": 3, - "commit_message_regex": "Fixes \d+\..*", - "commit_message_negative_regex": "ssh\:\/\/", + "commit_message_regex": "Fixes \\d+\\..*", + "commit_message_negative_regex": "ssh\\:\\/\\/", "branch_name_regex": "", "deny_delete_tag": false, "created_at": "2012-10-12T17:04:47Z", diff --git a/doc/api/protected_branches.md b/doc/api/protected_branches.md index a75090c90d5..d17341759ad 100644 --- a/doc/api/protected_branches.md +++ b/doc/api/protected_branches.md @@ -197,18 +197,18 @@ POST /projects/:id/protected_branches curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/protected_branches?name=*-stable&push_access_level=30&merge_access_level=30&unprotect_access_level=40" ``` -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the branch or wildcard | -| `push_access_level` | string | no | Access levels allowed to push (defaults: `40`, Maintainer role) | -| `merge_access_level` | string | no | Access levels allowed to merge (defaults: `40`, Maintainer role) | -| `unprotect_access_level` | string | no | Access levels allowed to unprotect (defaults: `40`, Maintainer role) | -| `allow_force_push` | boolean | no | Allow all users with push access to force push. (default: `false`) | -| `allowed_to_push` | array | no | **(PREMIUM)** Array of access levels allowed to push, with each described by a hash | -| `allowed_to_merge` | array | no | **(PREMIUM)** Array of access levels allowed to merge, with each described by a hash | -| `allowed_to_unprotect` | array | no | **(PREMIUM)** Array of access levels allowed to unprotect, with each described by a hash | -| `code_owner_approval_required` | boolean | no | **(PREMIUM)** Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/code_owners.md). (defaults: false) | +| Attribute | Type | Required | Description | +| -------------------------------------------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the branch or wildcard | +| `push_access_level` | string | no | Access levels allowed to push (defaults: `40`, Maintainer role) | +| `merge_access_level` | string | no | Access levels allowed to merge (defaults: `40`, Maintainer role) | +| `unprotect_access_level` | string | no | Access levels allowed to unprotect (defaults: `40`, Maintainer role) | +| `allow_force_push` | boolean | no | Allow all users with push access to force push. (default: `false`) | +| `allowed_to_push` **(PREMIUM)** | array | no | Array of access levels allowed to push, with each described by a hash | +| `allowed_to_merge` **(PREMIUM)** | array | no | Array of access levels allowed to merge, with each described by a hash | +| `allowed_to_unprotect` **(PREMIUM)** | array | no | Array of access levels allowed to unprotect, with each described by a hash | +| `code_owner_approval_required` **(PREMIUM)** | boolean | no | Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/code_owners.md). (defaults: false) | Example response: @@ -402,7 +402,7 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `name` | string | yes | The name of the branch | -## Require code owner approvals for a single branch +## Require code owner approvals for a single branch **(PREMIUM)** Update the "code owner approval required" option for the given protected branch. @@ -411,11 +411,11 @@ PATCH /projects/:id/protected_branches/:name ``` ```shell -curl --request PATCH --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/protected_branches/feature-branch" +curl --request PATCH --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/protected_branches/feature-branch?code_owner_approval_required=true" ``` -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the branch | -| `code_owner_approval_required` | boolean | no | **(PREMIUM)** Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/code_owners.md). (defaults: false)| +| Attribute | Type | Required | Description | +| -------------------------------------------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the branch | +| `code_owner_approval_required` | boolean | no | Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/code_owners.md). (defaults: false)| diff --git a/doc/api/protected_environments.md b/doc/api/protected_environments.md index 82bb1e55e77..c7de4c504a4 100644 --- a/doc/api/protected_environments.md +++ b/doc/api/protected_environments.md @@ -117,13 +117,13 @@ Example response: ```json { - "name":"production", - "deploy_access_levels":[ + "name": "production", + "deploy_access_levels": [ { - "access_level":40, - "access_level_description":"protected-access-group", - "user_id":null, - "group_id":9899826 + "access_level": 40, + "access_level_description": "protected-access-group", + "user_id": null, + "group_id": 9899826 } ] } diff --git a/doc/api/repositories.md b/doc/api/repositories.md index e93ffbc5e72..ec5c97e5b25 100644 --- a/doc/api/repositories.md +++ b/doc/api/repositories.md @@ -28,7 +28,7 @@ Supported attributes: | Attribute | Type | Required | Description | | :---------- | :------------- | :------- | :---------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | | `path` | string | no | The path inside repository. Used to get content of subdirectories. | | `ref` | string | no | The name of a repository branch or tag or if not given the default branch. | | `recursive` | boolean | no | Boolean value used to get a recursive tree (false by default). | @@ -104,7 +104,7 @@ Supported attributes: | Attribute | Type | Required | Description | | :-------- | :------------- | :------- | :---------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | | `sha` | string | yes | The blob SHA. | ## Raw blob content @@ -146,7 +146,7 @@ Supported attributes: | Attribute | Type | Required | Description | |:------------|:---------------|:---------|:----------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | | `sha` | string | no | The commit SHA to download. A tag, branch reference, or SHA can be used. This defaults to the tip of the default branch if not specified. | | `path` | string | no | The subpath of the repository to download. This defaults to the whole repository (empty string). | @@ -169,7 +169,7 @@ Supported attributes: | Attribute | Type | Required | Description | | :--------- | :------------- | :------- | :---------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | | `from` | string | yes | The commit SHA or branch name. | | `to` | string | yes | The commit SHA or branch name. | | `from_project_id` | integer | no | The ID to compare from | @@ -231,7 +231,7 @@ Supported attributes: | Attribute | Type | Required | Description | | :--------- | :------------- | :------- | :---------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | | `order_by` | string | no | Return contributors ordered by `name`, `email`, or `commits` (orders by commit date) fields. Default is `commits`. | | `sort` | string | no | Return contributors sorted in `asc` or `desc` order. Default is `asc`. | @@ -263,7 +263,7 @@ GET /projects/:id/repository/merge_base | Attribute | Type | Required | Description | | --------- | -------------- | -------- | ------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | | `refs` | array | yes | The refs to find the common ancestor of, multiple refs can be passed | Example request: @@ -291,7 +291,7 @@ Example response: } ``` -## Generate changelog data +## Add changelog data to a changelog file > [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/351) in GitLab 13.9. @@ -373,26 +373,26 @@ If the last tag is `v0.9.0` and the default branch is `main`, the range of commi included in this example is `v0.9.0..main`: ```shell -curl --header "PRIVATE-TOKEN: token" --data "version=1.0.0" "https://gitlab.com/api/v4/projects/42/repository/changelog" +curl --request POST --header "PRIVATE-TOKEN: token" --data "version=1.0.0" "https://gitlab.com/api/v4/projects/42/repository/changelog" ``` To generate the data on a different branch, specify the `branch` parameter. This command generates data from the `foo` branch: ```shell -curl --header "PRIVATE-TOKEN: token" --data "version=1.0.0&branch=foo" "https://gitlab.com/api/v4/projects/42/repository/changelog" +curl --request POST --header "PRIVATE-TOKEN: token" --data "version=1.0.0&branch=foo" "https://gitlab.com/api/v4/projects/42/repository/changelog" ``` To use a different trailer, use the `trailer` parameter: ```shell -curl --header "PRIVATE-TOKEN: token" --data "version=1.0.0&trailer=Type" "https://gitlab.com/api/v4/projects/42/repository/changelog" +curl --request POST --header "PRIVATE-TOKEN: token" --data "version=1.0.0&trailer=Type" "https://gitlab.com/api/v4/projects/42/repository/changelog" ``` To store the results in a different file, use the `file` parameter: ```shell -curl --header "PRIVATE-TOKEN: token" --data "version=1.0.0&file=NEWS" "https://gitlab.com/api/v4/projects/42/repository/changelog" +curl --request POST --header "PRIVATE-TOKEN: token" --data "version=1.0.0&file=NEWS" "https://gitlab.com/api/v4/projects/42/repository/changelog" ``` ### How it works @@ -689,7 +689,7 @@ The following capture groups are optional: - `pre`: If set, the tag is ignored. Ignoring `pre` tags ensures release candidate tags and other pre-release tags are not considered when determining the range of commits to generate a changelog for. -- `meta`: (Optional) Specifies build metadata. +- `meta`: Optional. Specifies build metadata. Using this information, GitLab builds a map of Git tags and their release versions. It then determines what the latest tag is, based on the version @@ -707,3 +707,39 @@ tag_regex: '^version-(?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)$' To test if your regular expression is working, you can use websites such as [regex101](https://regex101.com/). If the regular expression syntax is invalid, an error is produced when generating a changelog. + +## Generate changelog data + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345934) in GitLab 14.6. + +Generate changelog data based on commits in a repository, without committing +them to a changelog file. + +Works exactly like `POST /projects/:id/repository/changelog`, except the changelog +data isn't committed to any changelog file. + +```plaintext +GET /projects/:id/repository/changelog +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +| :-------- | :------- | :--------- | :---------- | +| `version` | string | yes | The version to generate the changelog for. The format must follow [semantic versioning](https://semver.org/). | +| `from` | string | no | The start of the range of commits (as a SHA) to use for generating the changelog. This commit itself isn't included in the list. | +| `to` | string | no | The end of the range of commits (as a SHA) to use for the changelog. This commit _is_ included in the list. Defaults to the branch specified in the `branch` attribute. | +| `date` | datetime | no | The date and time of the release, ISO 8601 formatted. Example: `2016-03-11T03:45:40Z`. Defaults to the current time. | +| `trailer` | string | no | The Git trailer to use for including commits, defaults to `Changelog`. | + +```shell +curl --header "PRIVATE-TOKEN: token" "https://gitlab.com/api/v4/projects/42/repository/changelog?version=1.0.0" +``` + +Example Response: + +```json +{ + "notes": "## 1.0.0 (2021-11-17)\n\n### feature (2 changes)\n\n- [Title 2](namespace13/project13@ad608eb642124f5b3944ac0ac772fecaf570a6bf) ([merge request](namespace13/project13!2))\n- [Title 1](namespace13/project13@3c6b80ff7034fa0d585314e1571cc780596ce3c8) ([merge request](namespace13/project13!1))\n" +} +``` diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index cc210eacd49..fd024240e90 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -35,6 +35,12 @@ GET /projects/:id/repository/files/:file_path curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb?ref=master" ``` +| Attribute | Type | Required | Description | +|-------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `file_path` | string | yes | URL encoded full path to new file. Ex. `lib%2Fclass%2Erb`. | +| `ref` | string | yes | The name of branch, tag or commit | + Example response: ```json @@ -52,11 +58,6 @@ Example response: } ``` -Parameters: - -- `file_path` (required) - URL encoded full path to new file. Ex. lib%2Fclass%2Erb -- `ref` (required) - The name of branch, tag or commit - NOTE: `blob_id` is the blob SHA, see [repositories - Get a blob from repository](repositories.md#get-a-blob-from-repository) @@ -95,6 +96,12 @@ Allows you to receive blame information. Each blame range contains lines and cor GET /projects/:id/repository/files/:file_path/blame ``` +| Attribute | Type | Required | Description | +|-------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `file_path` | string | yes | URL encoded full path to new file. Ex. `lib%2Fclass%2Erb`. | +| `ref` | string | yes | The name of branch, tag or commit | + ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/path%2Fto%2Ffile.rb/blame?ref=master" ``` @@ -127,11 +134,6 @@ Example response: ] ``` -Parameters: - -- `file_path` (required) - URL encoded full path to new file. Ex. lib%2Fclass%2Erb -- `ref` (required) - The name of branch, tag or commit - NOTE: `HEAD` method return just file metadata as in [Get file from repository](repository_files.md#get-file-from-repository). @@ -162,15 +164,16 @@ X-Gitlab-Size: 1476 GET /projects/:id/repository/files/:file_path/raw ``` +| Attribute | Type | Required | Description | +|-------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `file_path` | string | yes | URL encoded full path to new file. Ex. `lib%2Fclass%2Erb`. | +| `ref` | string | yes | The name of branch, tag or commit. Default is the `HEAD` of the project. | + ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb/raw?ref=master" ``` -Parameters: - -- `file_path` (required) - URL encoded full path to new file, such as lib%2Fclass%2Erb. -- `ref` (optional) - The name of branch, tag or commit. Default is the `HEAD` of the project. - NOTE: Like [Get file from repository](repository_files.md#get-file-from-repository) you can use `HEAD` to get just file metadata. @@ -182,6 +185,18 @@ This allows you to create a single file. For creating multiple files with a sing POST /projects/:id/repository/files/:file_path ``` +| Attribute | Type | Required | Description | +|------------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `file_path` | string | yes | URL encoded full path to new file. Ex. `lib%2Fclass%2Erb`. | +| `branch` | string | yes | Name of the branch | +| `start_branch` | string | no | Name of the branch to start the new commit from | +| `encoding` | string | no | Change encoding to `base64`. Default is `text`. | +| `author_email` | string | no | Specify the commit author's email address | +| `author_name` | string | no | Specify the commit author's name | +| `content` | string | yes | File content | +| `commit_message` | string | yes | Commit message | + ```shell curl --request POST --header 'PRIVATE-TOKEN: <your_access_token>' \ --header "Content-Type: application/json" \ @@ -199,17 +214,6 @@ Example response: } ``` -Parameters: - -- `file_path` (required) - URL encoded full path to new file. Ex. lib%2Fclass%2Erb -- `branch` (required) - Name of the branch -- `start_branch` (optional) - Name of the branch to start the new commit from -- `encoding` (optional) - Change encoding to `base64`. Default is `text`. -- `author_email` (optional) - Specify the commit author's email address -- `author_name` (optional) - Specify the commit author's name -- `content` (required) - File content -- `commit_message` (required) - Commit message - ## Update existing file in repository This allows you to update a single file. For updating multiple files with a single request see the [commits API](commits.md#create-a-commit-with-multiple-files-and-actions). @@ -218,6 +222,19 @@ This allows you to update a single file. For updating multiple files with a sing PUT /projects/:id/repository/files/:file_path ``` +| Attribute | Type | Required | Description | +|------------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `file_path` | string | yes | URL encoded full path to new file. Ex. `lib%2Fclass%2Erb`. | +| `branch` | string | yes | Name of the branch | +| `start_branch` | string | no | Name of the branch to start the new commit from | +| `encoding` | string | no | Change encoding to `base64`. Default is `text`. | +| `author_email` | string | no | Specify the commit author's email address | +| `author_name` | string | no | Specify the commit author's name | +| `content` | string | yes | File content | +| `commit_message` | string | yes | Commit message | +| `last_commit_id` | string | no | Last known file commit ID | + ```shell curl --request PUT --header 'PRIVATE-TOKEN: <your_access_token>' \ --header "Content-Type: application/json" \ @@ -235,18 +252,6 @@ Example response: } ``` -Parameters: - -- `file_path` (required) - URL encoded full path to new file. Ex. lib%2Fclass%2Erb -- `branch` (required) - Name of the branch -- `start_branch` (optional) - Name of the branch to start the new commit from -- `encoding` (optional) - Change encoding to `base64`. Default is `text`. -- `author_email` (optional) - Specify the commit author's email address -- `author_name` (optional) - Specify the commit author's name -- `content` (required) - New file content -- `commit_message` (required) - Commit message -- `last_commit_id` (optional) - Last known file commit ID - If the commit fails for any reason we return a 400 error with a non-specific error message. Possible causes for a failed commit include: @@ -265,6 +270,17 @@ This allows you to delete a single file. For deleting multiple files with a sing DELETE /projects/:id/repository/files/:file_path ``` +| Attribute | Type | Required | Description | +|------------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `file_path` | string | yes | URL encoded full path to new file. Ex. `lib%2Fclass%2Erb`. | +| `branch` | string | yes | Name of the branch | +| `start_branch` | string | no | Name of the branch to start the new commit from | +| `author_email` | string | no | Specify the commit author's email address. | +| `author_name` | string | no | Specify the commit author's name. | +| `commit_message` | string | yes | Commit message. | +| `last_commit_id` | string | no | Last known file commit ID. | + ```shell curl --request DELETE --header 'PRIVATE-TOKEN: <your_access_token>' \ --header "Content-Type: application/json" \ @@ -272,13 +288,3 @@ curl --request DELETE --header 'PRIVATE-TOKEN: <your_access_token>' \ "commit_message": "delete file"}' \ "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" ``` - -Parameters: - -- `file_path` (required) - URL encoded full path to new file. Ex. lib%2Fclass%2Erb -- `branch` (required) - Name of the branch -- `start_branch` (optional) - Name of the branch to start the new commit from -- `author_email` (optional) - Specify the commit author's email address -- `author_name` (optional) - Specify the commit author's name -- `commit_message` (required) - Commit message -- `last_commit_id` (optional) - Last known file commit ID diff --git a/doc/api/resource_access_tokens.md b/doc/api/resource_access_tokens.md index fa1e7aace9a..90e9769b896 100644 --- a/doc/api/resource_access_tokens.md +++ b/doc/api/resource_access_tokens.md @@ -58,7 +58,7 @@ POST projects/:id/access_tokens |-----------|---------|----------|---------------------| | `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | | `name` | String | yes | The name of the project access token | -| `scopes` | `Array[String]` | yes | [List of scopes](../user/project/settings/project_access_tokens.md#limiting-scopes-of-a-project-access-token) | +| `scopes` | `Array[String]` | yes | [List of scopes](../user/project/settings/project_access_tokens.md#scopes-for-a-project-access-token) | | `access_level` | Integer | no | A valid access level. Default value is 40 (Maintainer). Other allowed values are 10 (Guest), 20 (Reporter), and 30 (Developer). | | `expires_at` | Date | no | The token expires at midnight UTC on that date | diff --git a/doc/api/resource_groups.md b/doc/api/resource_groups.md index ce4fa33d7f2..237ba6e7d72 100644 --- a/doc/api/resource_groups.md +++ b/doc/api/resource_groups.md @@ -2,12 +2,11 @@ stage: Release group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: concepts, howto --- -# Resource Groups API +# Resource group API **(FREE)** -You can read more about [controling the job concurrency with resource groups](../ci/resource_groups/index.md). +You can read more about [controlling the job concurrency with resource groups](../ci/resource_groups/index.md). ## Get a specific resource group diff --git a/doc/api/services.md b/doc/api/services.md deleted file mode 100644 index 7587e53c9db..00000000000 --- a/doc/api/services.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'integrations.md' -remove_date: '2021-11-09' ---- - -This file was moved to [another location](integrations.md). - -<!-- This redirect file can be deleted after <2021-11-09>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/api/settings.md b/doc/api/settings.md index dd32c882860..e953990c091 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -228,7 +228,7 @@ listed in the descriptions of the relevant settings. | `after_sign_up_text` | string | no | Text shown to the user after signing up. | | `akismet_api_key` | string | required by: `akismet_enabled` | API key for Akismet spam protection. | | `akismet_enabled` | boolean | no | (**If enabled, requires:** `akismet_api_key`) Enable or disable Akismet spam protection. | -| `allow_group_owners_to_manage_ldap` | boolean | no | **(PREMIUM)** Set to `true` to allow group owners to manage LDAP. | +| `allow_group_owners_to_manage_ldap` **(PREMIUM)** | boolean | no | Set to `true` to allow group owners to manage LDAP. | | `allow_local_requests_from_hooks_and_services` | boolean | no | (Deprecated: Use `allow_local_requests_from_web_hooks_and_services` instead) Allow requests to the local network from hooks and services. | | `allow_local_requests_from_system_hooks` | boolean | no | Allow requests to the local network from system hooks. | | `allow_local_requests_from_web_hooks_and_services` | boolean | no | Allow requests to the local network from web hooks and services. | @@ -242,21 +242,21 @@ listed in the descriptions of the relevant settings. | `auto_devops_domain` | string | no | Specify a domain to use by default for every project's Auto Review Apps and Auto Deploy stages. | | `auto_devops_enabled` | boolean | no | Enable Auto DevOps for projects by default. It automatically builds, tests, and deploys applications based on a predefined CI/CD configuration. | | `automatic_purchased_storage_allocation` | boolean | no | Enabling this permits automatic allocation of purchased storage in a namespace. | -| `check_namespace_plan` | boolean | no | **(PREMIUM)** Enabling this makes only licensed EE features available to projects if the project namespace's plan includes the feature or if the project is public. | +| `check_namespace_plan` **(PREMIUM)** | boolean | no | Enabling this makes only licensed EE features available to projects if the project namespace's plan includes the feature or if the project is public. | | `commit_email_hostname` | string | no | Custom hostname (for private commit emails). | | `container_registry_token_expire_delay` | integer | no | Container Registry token duration in minutes. | | `deactivate_dormant_users` | boolean | no | Enable [automatic deactivation of dormant users](../user/admin_area/moderate_users.md#automatically-deactivate-dormant-users). | | `default_artifacts_expire_in` | string | no | Set the default expiration time for each job's artifacts. | | `default_branch_name` | string | no | [Instance-level custom initial branch name](../user/project/repository/branches/default.md#instance-level-custom-initial-branch-name) ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225258) in GitLab 13.2). | -| `default_branch_protection` | integer | no | Determine if developers can push to the default branch. Can take: `0` _(not protected, both developers and maintainers can push new commits and force push)_, `1` _(partially protected, developers and maintainers can push new commits, but cannot force push)_ or `2` _(fully protected, developers cannot push new commits, but maintainers can; no one can force push)_ as a parameter. Default is `2`. | +| `default_branch_protection` | integer | no | Determine if developers can push to the default branch. Can take: `0` _(not protected, both users with the Developer role or Maintainer role can push new commits and force push)_, `1` _(partially protected, users with the Developer role or Maintainer role can push new commits, but cannot force push)_ or `2` _(fully protected, users with the Developer or Maintainer role cannot push new commits, but users with the Developer or Maintainer role can; no one can force push)_ as a parameter. Default is `2`. | | `default_ci_config_path` | string | no | Default CI/CD configuration file and path for new projects (`.gitlab-ci.yml` if not set). | | `default_group_visibility` | string | no | What visibility level new groups receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | | `default_project_creation` | integer | no | Default project creation protection. Can take: `0` _(No one)_, `1` _(Maintainers)_ or `2` _(Developers + Maintainers)_| | `default_project_visibility` | string | no | What visibility level new projects receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | | `default_projects_limit` | integer | no | Project limit per user. Default is `100000`. | | `default_snippet_visibility` | string | no | What visibility level new snippets receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | -| `delayed_project_deletion` | boolean | no | **(PREMIUM SELF)** Enable delayed project deletion by default in new groups. Default is `false`. | -| `deletion_adjourned_period` | integer | no | **(PREMIUM SELF)** The number of days to wait before deleting a project or group that is marked for deletion. Value must be between 0 and 90. +| `delayed_project_deletion` **(PREMIUM SELF)** | boolean | no | Enable delayed project deletion by default in new groups. Default is `false`. | +| `deletion_adjourned_period` **(PREMIUM SELF)** | integer | no | The number of days to wait before deleting a project or group that is marked for deletion. Value must be between 0 and 90. | `diff_max_patch_bytes` | integer | no | Maximum [diff patch size](../user/admin_area/diff_limits.md), in bytes. | | `diff_max_files` | integer | no | Maximum [files in a diff](../user/admin_area/diff_limits.md). | | `diff_max_lines` | integer | no | Maximum [lines in a diff](../user/admin_area/diff_limits.md). | @@ -273,23 +273,23 @@ listed in the descriptions of the relevant settings. | `eks_account_id` | string | no | Amazon account ID. | | `eks_integration_enabled` | boolean | no | Enable integration with Amazon EKS. | | `eks_secret_access_key` | string | no | AWS IAM secret access key. | -| `elasticsearch_aws_access_key` | string | no | **(PREMIUM)** AWS IAM access key. | -| `elasticsearch_aws_region` | string | no | **(PREMIUM)** The AWS region the Elasticsearch domain is configured. | -| `elasticsearch_aws_secret_access_key` | string | no | **(PREMIUM)** AWS IAM secret access key. | -| `elasticsearch_aws` | boolean | no | **(PREMIUM)** Enable the use of AWS hosted Elasticsearch. | -| `elasticsearch_indexed_field_length_limit` | integer | no | **(PREMIUM)** Maximum size of text fields to index by Elasticsearch. 0 value means no limit. This does not apply to repository and wiki indexing. | -| `elasticsearch_indexed_file_size_limit_kb` | integer | no | **(PREMIUM)** Maximum size of repository and wiki files that are indexed by Elasticsearch. | -| `elasticsearch_indexing` | boolean | no | **(PREMIUM)** Enable Elasticsearch indexing. | -| `elasticsearch_limit_indexing` | boolean | no | **(PREMIUM)** Limit Elasticsearch to index certain namespaces and projects. | -| `elasticsearch_max_bulk_concurrency` | integer | no | **(PREMIUM)** Maximum concurrency of Elasticsearch bulk requests per indexing operation. This only applies to repository indexing operations. | -| `elasticsearch_max_bulk_size_mb` | integer | no | **(PREMIUM)** Maximum size of Elasticsearch bulk indexing requests in MB. This only applies to repository indexing operations. | -| `elasticsearch_namespace_ids` | array of integers | no | **(PREMIUM)** The namespaces to index via Elasticsearch if `elasticsearch_limit_indexing` is enabled. | -| `elasticsearch_project_ids` | array of integers | no | **(PREMIUM)** The projects to index via Elasticsearch if `elasticsearch_limit_indexing` is enabled. | -| `elasticsearch_search` | boolean | no | **(PREMIUM)** Enable Elasticsearch search. | -| `elasticsearch_url` | string | no | **(PREMIUM)** The URL to use for connecting to Elasticsearch. Use a comma-separated list to support cluster (for example, `http://localhost:9200, http://localhost:9201"`). | -| `elasticsearch_username` | string | no | **(PREMIUM)** The `username` of your Elasticsearch instance. | -| `elasticsearch_password` | string | no | **(PREMIUM)** The password of your Elasticsearch instance. | -| `email_additional_text` | string | no | **(PREMIUM)** Additional text added to the bottom of every email for legal/auditing/compliance reasons. | +| `elasticsearch_aws_access_key` **(PREMIUM)** | string | no | AWS IAM access key. | +| `elasticsearch_aws_region` **(PREMIUM)** | string | no | The AWS region the Elasticsearch domain is configured. | +| `elasticsearch_aws_secret_access_key` **(PREMIUM)** | string | no | AWS IAM secret access key. | +| `elasticsearch_aws` **(PREMIUM)** | boolean | no | Enable the use of AWS hosted Elasticsearch. | +| `elasticsearch_indexed_field_length_limit` **(PREMIUM)** | integer | no | Maximum size of text fields to index by Elasticsearch. 0 value means no limit. This does not apply to repository and wiki indexing. | +| `elasticsearch_indexed_file_size_limit_kb` **(PREMIUM)** | integer | no | Maximum size of repository and wiki files that are indexed by Elasticsearch. | +| `elasticsearch_indexing` **(PREMIUM)** | boolean | no | Enable Elasticsearch indexing. | +| `elasticsearch_limit_indexing` **(PREMIUM)** | boolean | no | Limit Elasticsearch to index certain namespaces and projects. | +| `elasticsearch_max_bulk_concurrency` **(PREMIUM)** | integer | no | Maximum concurrency of Elasticsearch bulk requests per indexing operation. This only applies to repository indexing operations. | +| `elasticsearch_max_bulk_size_mb` **(PREMIUM)** | integer | no | Maximum size of Elasticsearch bulk indexing requests in MB. This only applies to repository indexing operations. | +| `elasticsearch_namespace_ids` **(PREMIUM)** | array of integers | no | The namespaces to index via Elasticsearch if `elasticsearch_limit_indexing` is enabled. | +| `elasticsearch_project_ids` **(PREMIUM)** | array of integers | no | The projects to index via Elasticsearch if `elasticsearch_limit_indexing` is enabled. | +| `elasticsearch_search` **(PREMIUM)** | boolean | no | Enable Elasticsearch search. | +| `elasticsearch_url` **(PREMIUM)** | string | no | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support cluster (for example, `http://localhost:9200, http://localhost:9201"`). | +| `elasticsearch_username` **(PREMIUM)** | string | no | The `username` of your Elasticsearch instance. | +| `elasticsearch_password` **(PREMIUM)** | string | no | The password of your Elasticsearch instance. | +| `email_additional_text` **(PREMIUM)** | string | no | Additional text added to the bottom of every email for legal/auditing/compliance reasons. | | `email_author_in_body` | boolean | no | Some email servers do not support overriding the email sender name. Enable this option to include the name of the author of the issue, merge request or comment in the email body instead. | | `enabled_git_access_protocol` | string | no | Enabled protocols for Git access. Allowed values are: `ssh`, `http`, and `nil` to allow both protocols. | | `enforce_namespace_storage_limit` | boolean | no | Enabling this permits enforcement of namespace storage limits. | @@ -302,13 +302,13 @@ listed in the descriptions of the relevant settings. | `external_authorization_service_timeout` | float | required by:<br>`external_authorization_service_enabled` | The timeout after which an authorization request is aborted, in seconds. When a request times out, access is denied to the user. (min: 0.001, max: 10, step: 0.001). | | `external_authorization_service_url` | string | required by:<br>`external_authorization_service_enabled` | URL to which authorization requests are directed. | | `external_pipeline_validation_service_url` | string | no | URL to use for pipeline validation requests. | -| `external_pipeline_validation_service_token` | string | no | (Optional) Token to include as the `X-Gitlab-Token` header in requests to the URL in `external_pipeline_validation_service_url`. | +| `external_pipeline_validation_service_token` | string | no | Optional. Token to include as the `X-Gitlab-Token` header in requests to the URL in `external_pipeline_validation_service_url`. | | `external_pipeline_validation_service_timeout` | integer | no | How long to wait for a response from the pipeline validation service. Assumes `OK` if it times out. | -| `file_template_project_id` | integer | no | **(PREMIUM)** The ID of a project to load custom file templates from. | +| `file_template_project_id` **(PREMIUM)** | integer | no | The ID of a project to load custom file templates from. | | `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday, `1` for Monday, and `6` for Saturday. | -| `geo_node_allowed_ips` | string | yes | **(PREMIUM)** Comma-separated list of IPs and CIDRs of allowed secondary nodes. For example, `1.1.1.1, 2.2.2.0/24`. | -| `geo_status_timeout` | integer | no | **(PREMIUM)** The amount of seconds after which a request to get a secondary node status times out. | -| `git_two_factor_session_expiry` | integer | no | **(PREMIUM)** Maximum duration (in minutes) of a session for Git operations when 2FA is enabled. | +| `geo_node_allowed_ips` **(PREMIUM)** | string | yes | Comma-separated list of IPs and CIDRs of allowed secondary nodes. For example, `1.1.1.1, 2.2.2.0/24`. | +| `geo_status_timeout` **(PREMIUM)** | integer | no | The amount of seconds after which a request to get a secondary node status times out. | +| `git_two_factor_session_expiry` **(PREMIUM)** | integer | no | Maximum duration (in minutes) of a session for Git operations when 2FA is enabled. | | `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for Git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | | `gitaly_timeout_fast` | integer | no | Gitaly fast operation timeout, in seconds. Some Gitaly operations are expected to be fast. If they exceed this threshold, there may be a problem with a storage shard and 'failing fast' can help maintain the stability of the GitLab instance. Set to `0` to disable timeouts. | | `gitaly_timeout_medium` | integer | no | Medium Gitaly timeout, in seconds. This should be a value between the Fast and the Default timeout. Set to `0` to disable timeouts. | @@ -319,7 +319,7 @@ listed in the descriptions of the relevant settings. | `help_page_hide_commercial_content` | boolean | no | Hide marketing-related entries from help. | | `help_page_support_url` | string | no | Alternate support URL for help page and help dropdown. | | `help_page_text` | string | no | Custom text displayed on the help page. | -| `help_text` | string | no | **(PREMIUM)** GitLab server administrator information. | +| `help_text` **(PREMIUM)** | string | no | GitLab server administrator information. | | `hide_third_party_offers` | boolean | no | Do not display offers from third parties in GitLab. | | `home_page_url` | string | no | Redirect to this URL when not logged in. | | `housekeeping_bitmaps_enabled` | boolean | required by: `housekeeping_enabled` | Enable Git pack file bitmap creation. | @@ -336,20 +336,21 @@ listed in the descriptions of the relevant settings. | `local_markdown_version` | integer | no | Increase this value when any cached Markdown should be invalidated. | | `mailgun_signing_key` | string | no | The Mailgun HTTP webhook signing key for receiving events from webhook. | | `mailgun_events_enabled` | boolean | no | Enable Mailgun event receiver. | -| `maintenance_mode_message` | string | no | **(PREMIUM)** Message displayed when instance is in maintenance mode. | -| `maintenance_mode` | boolean | no | **(PREMIUM)** When instance is in maintenance mode, non-administrative users can sign in with read-only access and make read-only API requests. | +| `maintenance_mode_message` **(PREMIUM)** | string | no | Message displayed when instance is in maintenance mode. | +| `maintenance_mode` **(PREMIUM)** | boolean | no | When instance is in maintenance mode, non-administrative users can sign in with read-only access and make read-only API requests. | | `max_artifacts_size` | integer | no | Maximum artifacts size in MB. | | `max_attachment_size` | integer | no | Limit attachment size in MB. | | `max_import_size` | integer | no | Maximum import size in MB. 0 for unlimited. Default = 0 (unlimited) [Modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50MB to 0 in GitLab 13.8. | | `max_pages_size` | integer | no | Maximum size of pages repositories in MB. | -| `max_personal_access_token_lifetime` | integer | no | **(ULTIMATE SELF)** Maximum allowable lifetime for personal access tokens in days. | +| `max_personal_access_token_lifetime` **(ULTIMATE SELF)** | integer | no | Maximum allowable lifetime for personal access tokens in days. | +| `max_ssh_key_lifetime` **(ULTIMATE SELF)** | integer | no | Maximum allowable lifetime for SSH keys in days. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1007) in GitLab 14.6. | | `metrics_method_call_threshold` | integer | no | A method call is only tracked when it takes longer than the given amount of milliseconds. | | `mirror_available` | boolean | no | Allow repository mirroring to configured by project Maintainers. If disabled, only Administrators can configure repository mirroring. | -| `mirror_capacity_threshold` | integer | no | **(PREMIUM)** Minimum capacity to be available before scheduling more mirrors preemptively. | -| `mirror_max_capacity` | integer | no | **(PREMIUM)** Maximum number of mirrors that can be synchronizing at the same time. | -| `mirror_max_delay` | integer | no | **(PREMIUM)** Maximum time (in minutes) between updates that a mirror can have when scheduled to synchronize. | -| `npm_package_requests_forwarding` | boolean | no | **(PREMIUM)** Use npmjs.org as a default remote repository when the package is not found in the GitLab Package Registry for npm. | -| `pypi_package_requests_forwarding` | boolean | no | **(PREMIUM)** Use pypi.org as a default remote repository when the package is not found in the GitLab Package Registry for PyPI. | +| `mirror_capacity_threshold` **(PREMIUM)** | integer | no | Minimum capacity to be available before scheduling more mirrors preemptively. | +| `mirror_max_capacity` **(PREMIUM)** | integer | no | Maximum number of mirrors that can be synchronizing at the same time. | +| `mirror_max_delay` **(PREMIUM)** | integer | no | Maximum time (in minutes) between updates that a mirror can have when scheduled to synchronize. | +| `npm_package_requests_forwarding` **(PREMIUM)** | boolean | no | Use npmjs.org as a default remote repository when the package is not found in the GitLab Package Registry for npm. | +| `pypi_package_requests_forwarding` **(PREMIUM)** | boolean | no | Use pypi.org as a default remote repository when the package is not found in the GitLab Package Registry for PyPI. | | `outbound_local_requests_whitelist` | array of strings | no | Define a list of trusted domains or IP addresses to which local requests are allowed when local requests for hooks and services are disabled. | `pages_domain_verification_enabled` | boolean | no | Require users to prove ownership of custom domains. Domain verification is an essential security measure for public GitLab sites. Users are required to demonstrate they control a domain before it is enabled. | | `password_authentication_enabled_for_git` | boolean | no | Enable authentication for Git over HTTP(S) via a GitLab account password. Default is `true`. | @@ -364,7 +365,7 @@ listed in the descriptions of the relevant settings. | `project_export_enabled` | boolean | no | Enable project export. | | `prometheus_metrics_enabled` | boolean | no | Enable Prometheus metrics. | | `protected_ci_variables` | boolean | no | CI/CD variables are protected by default. | -| `pseudonymizer_enabled` | boolean | no | **(PREMIUM)** When enabled, GitLab runs a background job that produces pseudonymized CSVs of the GitLab database to upload to your configured object storage directory. +| `pseudonymizer_enabled` **(PREMIUM)** | boolean | no | When enabled, GitLab runs a background job that produces pseudonymized CSVs of the GitLab database to upload to your configured object storage directory. | `push_event_activities_limit` | integer | no | Number of changes (branches or tags) in a single push to determine whether individual push events or bulk push events are created. [Bulk push events are created](../user/admin_area/settings/push_event_activities_limit.md) if it surpasses that value. | | `push_event_hooks_limit` | integer | no | Number of changes (branches or tags) in a single push to determine whether webhooks and services fire or not. Webhooks and services aren't submitted if it surpasses that value. | | `rate_limiting_response_text` | string | no | When rate limiting is enabled via the `throttle_*` settings, send this plain text response when a rate limit is exceeded. 'Retry later' is sent if this is blank. | @@ -374,7 +375,7 @@ listed in the descriptions of the relevant settings. | `recaptcha_site_key` | string | required by: `recaptcha_enabled` | Site key for reCAPTCHA. | | `receive_max_input_size` | integer | no | Maximum push size (MB). | | `repository_checks_enabled` | boolean | no | GitLab periodically runs `git fsck` in all project and wiki repositories to look for silent disk corruption issues. | -| `repository_size_limit` | integer | no | **(PREMIUM)** Size limit per repository (MB) | +| `repository_size_limit` **(PREMIUM)** | integer | no | Size limit per repository (MB) | | `repository_storages_weighted` | hash of strings to integers | no | (GitLab 13.1 and later) Hash of names of taken from `gitlab.yml` to [weights](../administration/repository_storage_paths.md#configure-where-new-repositories-are-stored). New projects are created in one of these stores, chosen by a weighted random selection. | | `repository_storages` | array of strings | no | (GitLab 13.0 and earlier) List of names of enabled storage paths, taken from `gitlab.yml`. New projects are created in one of these stores, chosen at random. | | `require_admin_approval_after_user_signup` | boolean | no | When enabled, any user that signs up for an account using the registration form is placed under a **Pending approval** state and has to be explicitly [approved](../user/admin_area/moderate_users.md) by an administrator. | @@ -384,7 +385,7 @@ listed in the descriptions of the relevant settings. | `send_user_confirmation_email` | boolean | no | Send confirmation email on sign-up. | | `session_expire_delay` | integer | no | Session duration in minutes. GitLab restart is required to apply changes. | | `shared_runners_enabled` | boolean | no | (**If enabled, requires:** `shared_runners_text` and `shared_runners_minutes`) Enable shared runners for new projects. | -| `shared_runners_minutes` | integer | required by: `shared_runners_enabled` | **(PREMIUM)** Set the maximum number of pipeline minutes that a group can use on shared runners per month. | +| `shared_runners_minutes` **(PREMIUM)** | integer | required by: `shared_runners_enabled` | Set the maximum number of pipeline minutes that a group can use on shared runners per month. | | `shared_runners_text` | string | required by: `shared_runners_enabled` | Shared runners text. | | `sidekiq_job_limiter_mode` | string | no | `track` or `compress`. Sets the behavior for [Sidekiq job size limits](../user/admin_area/settings/sidekiq_job_limits.md). Default: 'compress'. | | `sidekiq_job_limiter_compression_threshold_bytes` | integer | no | The threshold in bytes at which Sidekiq jobs are compressed before being stored in Redis. Default: 100 000 bytes (100KB). | @@ -392,10 +393,10 @@ listed in the descriptions of the relevant settings. | `sign_in_text` | string | no | Text on the login page. | | `signin_enabled` | string | no | (Deprecated: Use `password_authentication_enabled_for_web` instead) Flag indicating if password authentication is enabled for the web interface. | | `signup_enabled` | boolean | no | Enable registration. Default is `true`. | -| `slack_app_enabled` | boolean | no | **(PREMIUM)** (**If enabled, requires:** `slack_app_id`, `slack_app_secret` and `slack_app_secret`) Enable Slack app. | -| `slack_app_id` | string | required by: `slack_app_enabled` | **(PREMIUM)** The app ID of the Slack-app. | -| `slack_app_secret` | string | required by: `slack_app_enabled` | **(PREMIUM)** The app secret of the Slack-app. | -| `slack_app_verification_token` | string | required by: `slack_app_enabled` | **(PREMIUM)** The verification token of the Slack-app. | +| `slack_app_enabled` **(PREMIUM)** | boolean | no | (**If enabled, requires:** `slack_app_id`, `slack_app_secret` and `slack_app_secret`) Enable Slack app. | +| `slack_app_id` **(PREMIUM)** | string | required by: `slack_app_enabled` | The app ID of the Slack-app. | +| `slack_app_secret` **(PREMIUM)** | string | required by: `slack_app_enabled` | The app secret of the Slack-app. | +| `slack_app_verification_token` **(PREMIUM)** | string | required by: `slack_app_enabled` | The verification token of the Slack-app. | | `snippet_size_limit` | integer | no | Max snippet content size in **bytes**. Default: 52428800 Bytes (50MB).| | `snowplow_app_id` | string | no | The Snowplow site name / application ID. (for example, `gitlab`) | | `snowplow_collector_hostname` | string | required by: `snowplow_enabled` | The Snowplow collector hostname. (for example, `snowplow.trx.gitlab.net`) | @@ -404,9 +405,9 @@ listed in the descriptions of the relevant settings. | `sourcegraph_enabled` | boolean | no | Enables Sourcegraph integration. Default is `false`. **If enabled, requires** `sourcegraph_url`. | | `sourcegraph_public_only` | boolean | no | Blocks Sourcegraph from being loaded on private and internal projects. Default is `true`. | | `sourcegraph_url` | string | required by: `sourcegraph_enabled` | The Sourcegraph instance URL for integration. | -| `spam_check_endpoint_enabled` | boolean | no | Enables Spam Check via external API endpoint. Default is `false`. | -| `spam_check_endpoint_url` | string | no | URL of the external Spam Check service endpoint. | -| `spam_check_api_key` | string | no | The API key used by GitLab for accessing the Spam Check service endpoint. | +| `spam_check_endpoint_enabled` | boolean | no | Enables spam checking using external Spam Check API endpoint. Default is `false`. | +| `spam_check_endpoint_url` | string | no | URL of the external Spamcheck service endpoint. Valid URI schemes are `grpc` or `tls`. Specifying `tls` forces communication to be encrypted.| +| `spam_check_api_key` | string | no | API key used by GitLab for accessing the Spam Check service endpoint. | | `suggest_pipeline_enabled` | boolean | no | Enable pipeline suggestion banner. | | `terminal_max_session_time` | integer | no | Maximum time for web terminal websocket connection (in seconds). Set to `0` for unlimited time. | | `terms` | text | required by: `enforce_terms` | (**Required by:** `enforce_terms`) Markdown content for the ToS. | diff --git a/doc/api/status_checks.md b/doc/api/status_checks.md index 0c72ee37348..c0dba71bdc5 100644 --- a/doc/api/status_checks.md +++ b/doc/api/status_checks.md @@ -145,6 +145,6 @@ PUT /projects/:id/external_status_checks/:check_id | `external_url` | string | no | URL of external status check resource | | `protected_branch_ids` | `array<Integer>` | no | IDs of protected branches to scope the rule by | -## Related links +## Related topics - [External status checks](../user/project/merge_requests/status_checks.md). diff --git a/doc/api/topics.md b/doc/api/topics.md index 5e9e1b8fc12..538b9af9374 100644 --- a/doc/api/topics.md +++ b/doc/api/topics.md @@ -161,7 +161,7 @@ curl --request PUT \ --data "name=topic1" \ --header "PRIVATE-TOKEN: <your_access_token>" \ "https://gitlab.example.com/api/v4/topics/1" - +``` Example response: @@ -188,3 +188,18 @@ curl --request PUT \ "https://gitlab.example.com/api/v4/topics/1" \ --form "avatar=@/tmp/example.png" ``` + +### Remove a topic avatar + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/348148) in GitLab 14.6. + +To remove a topic avatar, use a blank value for the `avatar` attribute. + +Example request: + +```shell +curl --request PUT \ + --data "avatar=" \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/topics/1" +``` diff --git a/doc/api/users.md b/doc/api/users.md index d8effc4d38f..292dc411e5b 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -269,7 +269,9 @@ GET /users/:id Parameters: -- `id` (required) - The ID of a user +| Attribute | Type | Required | Description | +|-----------|---------|----------|------------------| +| `id` | integer | yes | The ID of a user | ```json { @@ -303,7 +305,9 @@ GET /users/:id Parameters: -- `id` (required) - The ID of a user +| Attribute | Type | Required | Description | +|-----------|---------|----------|------------------| +| `id` | integer | yes | The ID of a user | Example Responses: @@ -509,8 +513,10 @@ DELETE /users/:id/identities/:provider Parameters: -- `id` (required) - The ID of the user -- `provider` (required) - External provider name +| Attribute | Type | Required | Description | +|------------|---------|----------|------------------------| +| `id` | integer | yes | The ID of a user | +| `provider` | string | yes | External provider name | ## User deletion @@ -523,10 +529,10 @@ DELETE /users/:id Parameters: -- `id` (required) - The ID of the user -- `hard_delete` (optional) - If true, contributions that would usually be - [moved to the ghost user](../user/profile/account/delete_account.md#associated-records) - are deleted instead, as well as groups owned solely by this user. +| Attribute | Type | Required | Description | +|---------------|---------|----------|----------------------------------------------| +| `id` | integer | yes | The ID of a user | +| `hard_delete` | boolean | no | If true, contributions that would usually be [moved to the ghost user](../user/profile/account/delete_account.md#associated-records) are deleted instead, as well as groups owned solely by this user. | ## List current user (for normal users) @@ -576,14 +582,16 @@ GET /user ## List current user (for admins) -Parameters: - -- `sudo` (optional) - the ID of a user to make the call in their place - ```plaintext GET /user ``` +Parameters: + +| Attribute | Type | Required | Description | +|-----------|---------|----------|--------------------------------------------------| +| `sudo` | integer | no | the ID of a user to make the call in their place | + ```json { "id": 1, @@ -936,7 +944,9 @@ GET /user/keys/:key_id Parameters: -- `key_id` (required) - The ID of an SSH key +| Attribute | Type | Required | Description | +|-----------|--------|----------|----------------------| +| `key_id` | string | yes | The ID of an SSH key | ```json { @@ -957,9 +967,11 @@ POST /user/keys Parameters: -- `title` (required) - new SSH key's title -- `key` (required) - new SSH key -- `expires_at` (optional) - The expiration date of the SSH key in ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`) +| Attribute | Type | Required | Description | +|--------------|--------|----------|--------------------------------------------------------------------------------| +| `title` | string | yes | new SSH key's title | +| `key` | string | yes | new SSH key | +| `expires_at` | string | no | The expiration date of the SSH key in ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`) | ```json { @@ -995,10 +1007,12 @@ POST /users/:id/keys Parameters: -- `id` (required) - ID of specified user -- `title` (required) - new SSH key's title -- `key` (required) - new SSH key -- `expires_at` (optional) - The expiration date of the SSH key in ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`) +| Attribute | Type | Required | Description | +|--------------|---------|----------|--------------------------------------------------------------------------------| +| `id` | integer | yes | ID of specified user | +| `title` | string | yes | new SSH key's title | +| `key` | string | yes | new SSH key | +| `expires_at` | string | no | The expiration date of the SSH key in ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`) | NOTE: This also adds an audit event, as described in [audit instance events](../administration/audit_events.md#instance-events). **(PREMIUM)** @@ -1014,7 +1028,9 @@ DELETE /user/keys/:key_id Parameters: -- `key_id` (required) - SSH key ID +| Attribute | Type | Required | Description | +|-----------|---------|----------|-------------| +| `key_id` | integer | yes | SSH key ID | ## Delete SSH key for given user @@ -1026,8 +1042,10 @@ DELETE /users/:id/keys/:key_id Parameters: -- `id` (required) - ID of specified user -- `key_id` (required) - SSH key ID +| Attribute | Type | Required | Description | +|-----------|---------|----------|----------------------| +| `id` | integer | yes | ID of specified user | +| `key_id` | integer | yes | SSH key ID | ## List all GPG keys @@ -1092,8 +1110,8 @@ POST /user/gpg_keys Parameters: | Attribute | Type | Required | Description | -| --------- | ------ | -------- | --------------- | -| key | string | yes | The new GPG key | +|-----------|--------|----------|-----------------| +| `key` | string | yes | The new GPG key | ```shell curl --data "key=-----BEGIN PGP PUBLIC KEY BLOCK-----\r\n\r\nxsBNBFV..." \ @@ -1288,7 +1306,9 @@ GET /users/:id/emails Parameters: -- `id` (required) - ID of specified user +| Attribute | Type | Required | Description | +|-----------|---------|----------|----------------------| +| `id` | integer | yes | ID of specified user | ## Single email @@ -1300,7 +1320,9 @@ GET /user/emails/:email_id Parameters: -- `email_id` (required) - email ID +| Attribute | Type | Required | Description | +|------------|---------|----------|-------------| +| `email_id` | integer | yes | Email ID | ```json { @@ -1320,7 +1342,9 @@ POST /user/emails Parameters: -- `email` (required) - email address +| Attribute | Type | Required | Description | +|-----------|--------|----------|-------------| +| `email` | string | yes | Email address | ```json { @@ -1353,9 +1377,11 @@ POST /users/:id/emails Parameters: -- `id` (required) - ID of specified user -- `email` (required) - email address -- `skip_confirmation` (optional) - Skip confirmation and assume email is verified - true or false (default) +| Attribute | Type | Required | Description | +|---------------------|---------|----------|---------------------------------------------------------------------------| +| `id` | string | yes | ID of specified user | +| `email` | string | yes | Email address | +| `skip_confirmation` | boolean | no | Skip confirmation and assume email is verified - true or false (default) | ## Delete email for current user @@ -1368,7 +1394,9 @@ DELETE /user/emails/:email_id Parameters: -- `email_id` (required) - email ID +| Attribute | Type | Required | Description | +|------------|---------|----------|-------------| +| `email_id` | integer | yes | Email ID | ## Delete email for given user @@ -1380,8 +1408,10 @@ DELETE /users/:id/emails/:email_id Parameters: -- `id` (required) - ID of specified user -- `email_id` (required) - email ID +| Attribute | Type | Required | Description | +|------------|---------|----------|----------------------| +| `id` | integer | yes | ID of specified user | +| `email_id` | integer | yes | Email ID | ## Block user @@ -1393,7 +1423,9 @@ POST /users/:id/block Parameters: -- `id` (required) - ID of specified user +| Attribute | Type | Required | Description | +|------------|---------|----------|----------------------| +| `id` | integer | yes | ID of specified user | Returns: @@ -1413,7 +1445,9 @@ POST /users/:id/unblock Parameters: -- `id` (required) - ID of specified user +| Attribute | Type | Required | Description | +|------------|---------|----------|----------------------| +| `id` | integer | yes | ID of specified user | Returns `201 OK` on success, `404 User Not Found` is user cannot be found or `403 Forbidden` when trying to unblock a user blocked by LDAP synchronization. @@ -1430,7 +1464,9 @@ POST /users/:id/deactivate Parameters: -- `id` (required) - ID of specified user +| Attribute | Type | Required | Description | +|------------|---------|----------|----------------------| +| `id` | integer | yes | ID of specified user | Returns: @@ -1453,7 +1489,9 @@ POST /users/:id/activate Parameters: -- `id` (required) - ID of specified user +| Attribute | Type | Required | Description | +|------------|---------|----------|----------------------| +| `id` | integer | yes | ID of specified user | Returns: @@ -1572,7 +1610,9 @@ POST /users/:id/approve Parameters: -- `id` (required) - ID of specified user +| Attribute | Type | Required | Description | +|------------|---------|----------|----------------------| +| `id` | integer | yes | ID of specified user | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/42/approve" @@ -1583,6 +1623,7 @@ Returns: - `201 Created` on success. - `404 User Not Found` if user cannot be found. - `403 Forbidden` if the user cannot be approved because they are blocked by an administrator or by LDAP synchronization. +- `409 Conflict` if the user has been deactivated. Example Responses: @@ -1731,10 +1772,6 @@ It revokes an impersonation token. DELETE /users/:user_id/impersonation_tokens/:impersonation_token_id ``` -```shell -curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/42/impersonation_tokens/1" -``` - Parameters: | Attribute | Type | Required | Description | @@ -1742,6 +1779,10 @@ Parameters: | `user_id` | integer | yes | The ID of the user | | `impersonation_token_id` | integer | yes | The ID of the impersonation token | +```shell +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/42/impersonation_tokens/1" +``` + ## Create a personal access token **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17176) in GitLab 13.6. diff --git a/doc/api/v3_to_v4.md b/doc/api/v3_to_v4.md index 3fba95c1fb3..da269073905 100644 --- a/doc/api/v3_to_v4.md +++ b/doc/api/v3_to_v4.md @@ -11,7 +11,7 @@ The GitLab API v3 was [removed](https://gitlab.com/gitlab-org/gitlab-foss/-/issu For information about the current version of the GitLab API, read the [API documentation](index.md). -## Related links +## Related topics - [GitLab v3 API documentation](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-16-stable/doc/api/index.md) - [Migration guide](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/11-0-stable/doc/api/v3_to_v4.md) from diff --git a/doc/architecture/blueprints/ci_scale/index.md b/doc/architecture/blueprints/ci_scale/index.md index 3e9fbc534d5..092f8a7119a 100644 --- a/doc/architecture/blueprints/ci_scale/index.md +++ b/doc/architecture/blueprints/ci_scale/index.md @@ -5,7 +5,7 @@ comments: false description: 'Improve scalability of GitLab CI/CD' --- -# Next CI/CD scale target: 20M builds per day by 2024 +# CI/CD Scaling ## Summary @@ -20,13 +20,8 @@ store all the builds in PostgreSQL in `ci_builds` table, and because we are creating more than [2 million builds each day on GitLab.com](https://docs.google.com/spreadsheets/d/17ZdTWQMnTHWbyERlvj1GA7qhw_uIfCoI5Zfrrsh95zU), we are reaching database limits that are slowing our development velocity down. -On February 1st, 2021, a billionth CI/CD job was created and the number of -builds is growing exponentially. We will run out of the available primary keys -for builds before December 2021 unless we improve the database model used to -store CI/CD data. - -We expect to see 20M builds created daily on GitLab.com in the first half of -2024. +On February 1st, 2021, GitLab.com surpassed 1 billion CI/CD builds created and the number of +builds continues to grow exponentially.  @@ -60,8 +55,8 @@ that have the same problem. Primary keys problem will be tackled by our Database Team. -Status: As of October 2021 the primary keys in CI tables have been migrated to -big integers. +**Status**: As of October 2021 the primary keys in CI tables have been migrated +to big integers. ### The table is too large @@ -84,6 +79,14 @@ seem fine in the development environment may not work on GitLab.com. The difference in the dataset size between the environments makes it difficult to predict the performance of even the most simple queries. +Team members and the wider community members are struggling to contribute the +Verify area, because we restricted the possibility of extending `ci_builds` +even further. Our static analysis tools prevent adding more columns to this +table. Adding new queries is unpredictable because of the size of the dataset +and the amount of queries executed using the table. This significantly hinders +the development velocity and contributes to incidents on the production +environment. + We also expect a significant, exponential growth in the upcoming years. One of the forecasts done using [Facebook's @@ -94,6 +97,10 @@ sustain in upcoming years.  +**Status**: As of October 2021 we reduced the growth rate of `ci_builds` table +by writing build options and variables to `ci_builds_metadata` table. We plan +to ship further improvements that will be described in a separate blueprint. + ### Queuing mechanisms are using the large table Because of how large the table is, mechanisms that we use to build queues of @@ -114,8 +121,8 @@ table that will accelerate SQL queries used to build queues](https://gitlab.com/gitlab-org/gitlab/-/issues/322766) and we want to explore them. -Status: the new architecture [has been implemented on GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/5909#note_680407908). - +**Status**: As of October 2021 the new architecture [has been implemented on +GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/5909#note_680407908). The following epic tracks making it generally available: [Make the new pending builds architecture generally available]( https://gitlab.com/groups/gitlab-org/-/epics/6954). @@ -136,17 +143,8 @@ columns, tables, partitions or database shards. Effort to improve background migrations will be owned by our Database Team. -Status: In progress. - -### Development velocity is negatively affected - -Team members and the wider community members are struggling to contribute the -Verify area, because we restricted the possibility of extending `ci_builds` -even further. Our static analysis tools prevent adding more columns to this -table. Adding new queries is unpredictable because of the size of the dataset -and the amount of queries executed using the table. This significantly hinders -the development velocity and contributes to incidents on the production -environment. +**Status**: In progress. We plan to ship further improvements that will be +described in a separate architectural blueprint. ## Proposal @@ -157,32 +155,34 @@ First, we want to focus on things that are urgently needed right now. We need to fix primary keys overflow risk and unblock other teams that are working on database partitioning and sharding. -We want to improve situation around bottlenecks that are known already, like -queuing mechanisms using the large table and things that are holding other -teams back. +We want to improve known bottlenecks, like +builds queuing mechanisms that is using the large table, and other things that +are holding other teams back. Extending CI/CD metrics is important to get a better sense of how the system performs and to what growth should we expect. This will make it easier for us to identify bottlenecks and perform more advanced capacity planning. -As we work on first iterations we expect our Database Sharding team and -Database Scalability Working Group to make progress on patterns we will be able -to use to partition the large CI/CD dataset. We consider the strong time-decay -effect, related to the diminishing importance of pipelines with time, as an -opportunity we might want to seize. +Next step is to better understand how we can leverage strong time-decay +characteristic of CI/CD data. This might help us to partition CI/CD dataset to +reduce the size of CI/CD database tables. ## Iterations Work required to achieve our next CI/CD scaling target is tracked in the -[GitLab CI/CD 20M builds per day scaling -target](https://gitlab.com/groups/gitlab-org/-/epics/5745) epic. +[CI/CD Scaling](https://gitlab.com/groups/gitlab-org/-/epics/5745) epic. + +1. ✓ Migrate primary keys to big integers on GitLab.com. +1. ✓ Implement the new architecture of builds queuing on GitLab.com. +1. Make the new builds queuing architecture generally available. +1. Partition CI/CD data using time-decay pattern. ## Status |-------------|--------------| | Created at | 21.01.2021 | | Approved at | 26.04.2021 | -| Updated at | 28.10.2021 | +| Updated at | 06.12.2021 | Status: In progress. @@ -215,6 +215,7 @@ Domain experts: | Area | Who |------------------------------|------------------------| | Domain Expert / Verify | Fabio Pitino | +| Domain Expert / Verify | Marius Bobin | | Domain Expert / Database | Jose Finotto | | Domain Expert / PostgreSQL | Nikolay Samokhvalov | diff --git a/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md b/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md index 60ddfe8ce02..e545e8844ec 100644 --- a/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md +++ b/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md @@ -28,7 +28,7 @@ might be useful to understand why it is important, and what is the roadmap. ## How GitLab Pages Works GitLab Pages is a daemon designed to serve static content, written in -[Go](https://golang.org/). +[Go](https://go.dev/). Initially, GitLab Pages has been designed to store static content on a local shared block storage (NFS) in a hierarchical group > project directory diff --git a/doc/architecture/blueprints/consolidating_groups_and_projects/index.md b/doc/architecture/blueprints/consolidating_groups_and_projects/index.md index 53357220755..345160dc77f 100644 --- a/doc/architecture/blueprints/consolidating_groups_and_projects/index.md +++ b/doc/architecture/blueprints/consolidating_groups_and_projects/index.md @@ -133,7 +133,7 @@ The initial iteration will provide a framework to house features under `Namespac 1. **Conceptual model**: What are the current and future state conceptual models of these features ([see object modeling for designers](https://hpadkisson.medium.com/object-modeling-for-designers-an-introduction-7871bdcf8baf))? These should be documented in Pajamas (example: [Merge Requests](https://design.gitlab.com/objects/merge-request)). 1. **Merge conflicts**: What inconsistencies are there across project, group, and admin levels? How might these be addressed? For an example of how we rationalized this for labels, please see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338820). -1. **Inheritence & information flow**: How is information inherited across our container hierarchy currently? How might this be impacted if complying with the new [inheritence behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/343316) framework? +1. **Inheritance & information flow**: How is information inherited across our container hierarchy currently? How might this be impacted if complying with the new [inheritance behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/343316) framework? 1. **Settings**: Where can settings for this feature be found currently? How will these be impacted by `Namespaces`? 1. **Access**: Who can access this feature and is that impacted by the new container structure? Are there any role or privacy considerations? 1. **Tier**: Is there any tier functionality that is differentiated by projects and groups? diff --git a/doc/architecture/blueprints/container_registry_metadata_database/index.md b/doc/architecture/blueprints/container_registry_metadata_database/index.md index 7bbaefb8e1e..a38a8727dc4 100644 --- a/doc/architecture/blueprints/container_registry_metadata_database/index.md +++ b/doc/architecture/blueprints/container_registry_metadata_database/index.md @@ -18,7 +18,7 @@ For GitLab.com and for GitLab customers, the Container Registry is a critical co ## Current Architecture -The Container Registry is a single [Go](https://golang.org/) application. Its only dependency is the storage backend on which images and metadata are stored. +The Container Registry is a single [Go](https://go.dev/) application. Its only dependency is the storage backend on which images and metadata are stored. ```mermaid graph LR @@ -146,7 +146,7 @@ The interaction between the registry and its clients, including GitLab Rails and ### Database -Following the GitLab [Go standards and style guidelines](../../../development/go_guide), no ORM is used to manage the database, only the [`database/sql`](https://golang.org/pkg/database/sql/) package from the Go standard library, a PostgreSQL driver ([`lib/pq`](https://pkg.go.dev/github.com/lib/pq?tab=doc)) and raw SQL queries, over a TCP connection pool. +Following the GitLab [Go standards and style guidelines](../../../development/go_guide), no ORM is used to manage the database, only the [`database/sql`](https://pkg.go.dev/database/sql) package from the Go standard library, a PostgreSQL driver ([`lib/pq`](https://pkg.go.dev/github.com/lib/pq?tab=doc)) and raw SQL queries, over a TCP connection pool. The design and development of the registry database adhere to the GitLab [database guidelines](../../../development/database/). Being a Go application, the required tooling to support the database will have to be developed, such as for running database migrations. diff --git a/doc/architecture/blueprints/gitlab_to_kubernetes_communication/index.md b/doc/architecture/blueprints/gitlab_to_kubernetes_communication/index.md index fb71707c146..754988487de 100644 --- a/doc/architecture/blueprints/gitlab_to_kubernetes_communication/index.md +++ b/doc/architecture/blueprints/gitlab_to_kubernetes_communication/index.md @@ -9,7 +9,7 @@ description: 'GitLab to Kubernetes communication' # GitLab to Kubernetes communication **(FREE)** The goal of this document is to define how GitLab can communicate with Kubernetes -and in-cluster services through the GitLab Kubernetes Agent. +and in-cluster services through the GitLab Agent. ## Challenges @@ -48,7 +48,7 @@ are stored on the GitLab side and this is yet another security concern for our c For more discussion on these issues, read [issue #212810](https://gitlab.com/gitlab-org/gitlab/-/issues/212810). -## GitLab Kubernetes Agent epic +## GitLab Agent epic To address these challenges and provide some new features, the Configure group is building an active in-cluster component that inverts the @@ -62,12 +62,12 @@ The customer does not need to provide any credentials to GitLab, and is in full control of what permissions the agent has. For more information, visit the -[GitLab Kubernetes Agent repository](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) or +[GitLab Agent repository](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) or [the epic](https://gitlab.com/groups/gitlab-org/-/epics/3329). ### Request routing -Agents connect to the server-side component called GitLab Kubernetes Agent Server +Agents connect to the server-side component called GitLab Agent Server (`gitlab-kas`) and keep an open connection that waits for commands. The difficulty with the approach is in routing requests from GitLab to the correct agent. Each cluster may contain multiple logical agents, and each may be running as multiple diff --git a/doc/architecture/blueprints/object_storage/index.md b/doc/architecture/blueprints/object_storage/index.md new file mode 100644 index 00000000000..a79374d60bd --- /dev/null +++ b/doc/architecture/blueprints/object_storage/index.md @@ -0,0 +1,220 @@ +--- +stage: none +group: unassigned +comments: false +description: 'Object storage: direct_upload consolidation - architecture blueprint.' +--- + +# Object storage: `direct_upload` consolidation + +## Abstract + +GitLab stores three classes of user data: database records, Git +repositories, and user-uploaded files (which are referred to as +file storage throughout the blueprint). + +The user and contributor experience for our file +storage has room for significant improvement: + +- Initial GitLab setup experience requires creation and setup of 13 + buckets, instead of just 1. +- Features using file storage require contributors to think about both local + storage and object storage, which leads to friction and + complexity. This often results in broken features and security issues. +- Contributors who work on file storage often also have to write code + for Workhorse, Omnibus, and cloud native GitLab (CNG). + +## Problem definition + +Object storage is a fundamental component of GitLab, providing the +underlying implementation for shared, distributed, highly-available +(HA) file storage. + +Over time, we have built support for object storage across the +application, solving specific problems in a [multitude of +iterations](https://about.gitlab.com/company/team/structure/working-groups/object-storage/#company-efforts-on-uploads). This +has led to increased complexity across the board, from development +(new features and bug fixes) to installation: + +- New GitLab installations require the creation and configuration of + several object storage buckets instead of just one, as each group of + features requires its own. This has an impact on the installation + experience and new feature adoption, and takes us further away from + boring solutions. +- The release of cloud native GitLab required the removal of NFS + shared storage and the development of direct upload, a feature that + was expanded, milestone after milestone, to several type of uploads, + but never enabled globally. +- Today, GitLab supports both local storage and object storage. Local + storage only works on single box installations or with a NFS, which + [we no longer recommend](../../../administration/nfs.md) to our + users and is no longer in use on GitLab.com. +- Understanding all the moving parts and the flow is extremely + complicated: we have CarrierWave, Fog, Golang S3/Azure SDKs, all + being used, and that complicates testing as well. +- Fog and CarrierWave are not maintained to the level of the native + SDKs (for example, AWS S3 SDK), so we have to maintain or monkey + patch those tools to support requested customer features + (for example, [issue #242245](https://gitlab.com/gitlab-org/gitlab/-/issues/242245)) + that would normally be "free". +- In many cases, we copy around object storage files needlessly + (for example, [issue #285597](https://gitlab.com/gitlab-org/gitlab/-/issues/285597)). + Large files (LFS, packages, and so on) are slow to finalize or don't work + at all as a result. + +## Improvements over the current situation + +The following is a brief description of the main directions we can take to +remove the pain points affecting our object storage implementation. + +This is also available as [a YouTube +video](https://youtu.be/X9V_w8hsM8E) recorded for the [Object Storage +Working +Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/). + +### Simplify GitLab architecture by shipping MinIO + +In the beginning, object storage support was a Premium feature, not +part of our CE distribution. Because of that, we had to support both +local storage and object storage. + +With local storage, there is the assumption of a shared storage +between components. This can be achieved by having a single box +installation, without HA, or with a NFS, which [we no longer +recommend](../../../administration/nfs.md). + +We have a testing gap on object storage. It also requires Workhorse +and MinIO, which are not present in our pipelines, so too much is +replaced by a mock implementation. Furthermore, the presence of a +shared disk, both in CI and in local development, often hides broken +implementations until we deploy on an HA environment. + +Shipping MinIO as part of the product will reduce the differences +between a cloud and a local installation, standardizing our file +storage on a single technology. + +The removal of local disk operations will reduce the complexity of +development as well as mitigate several security attack vectors as +we no longer write user-provided data on the local storage. + +It will also reduce human errors as we will always run a local object +storage in development mode and any local file disk access should +raise a red flag during the merge request review. + +This effort is described in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6099). + +### Enable direct upload by default on every upload + +Because every group of features requires its own bucket, we don't have +direct upload enabled everywhere. Contributing a new upload requires +coding it in both Ruby on Rails and Go. + +Implementing a new feature that does not yet have a dedicated bucket +requires the developer to also create a merge request in Omnibus +and CNG, as well as coordinate with SREs to configure the new bucket +for our own environments. + +This also slows down feature adoptions, because our users need to +reconfigure GitLab and prepare a new bucket in their +infrastructure. It also makes the initial installation more complex +feature after feature. + +Implementing a direct upload by default, with a +[consolidated object storage configuration](../../../administration/object_storage.md#consolidated-object-storage-configuration) +will reduce the number of merge requests needed to ship a new feature +from four to only one. It will also remove the need for SRE +intervention as the bucket will always be the same. + +This will simplify our development and review processes, as well as +the GitLab configuration file. And every user will immediately have +access to new features without infrastructure chores. + +### Simplify object storage code + +Our implementation is built on top of a 3rd-party framework where +every object storage client is a 3rd-party library. Unfortunately some +of them are unmaintained. [We have customers who cannot push 5GB Git +LFS objects](https://gitlab.com/gitlab-org/gitlab/-/issues/216442), +but with such a vital feature implemented in 3rd-party libraries we +are slowed down in fixing it, and we also rely on external maintainers +to merge and release fixes. + +Before the introduction of direct upload, using the +[CarrierWave](https://github.com/carrierwaveuploader/carrierwave) +library, _"a gem that provides a simple and extremely flexible way to +upload files from Ruby applications."_, was the boring solution. +However this is no longer our use-case, as we upload files from +Workhorse, and we had to [patch CarrierWave's +internals](https://gitlab.com/gitlab-org/gitlab/-/issues/285597#note_452696638) +to support direct upload. + +A brief proposal covering CarrierWave removal and a new streamlined +internal upload API is described +[in this issue comment](https://gitlab.com/gitlab-org/gitlab/-/issues/213288#note_325358026). + +Ideally, we wouldn't need to duplicate object storage clients in Go +and Ruby. By removing CarrierWave, we can make use of the officially +supported native clients when the provider S3 compatibility level is +not sufficient. + +## Iterations + +In this section we list some possible iterations. This is not +intended to be the final roadmap, but is a conversation started for the +Object Storage Working Group. + +1. Create a new catchall bucket and a unified internal API for + authorization without CarrierWave. +1. Ship MinIO with Omnibus (CNG images already include it). +1. Expand GitLab-QA to cover all the supported configurations. +1. Deprecate local disk access. +1. Deprecate configurations with multiple buckets. +1. Implement a bucket-to-bucket migration. +1. Migrate the current CarrierWave uploads to the new implementation. +1. On the next major release: Remove support for local disk access and + configurations with multiple buckets. + +### Benefits of the current iteration plan + +The current plan is designed to provide tangible benefits from the +first step. + +With the introduction of the catchall bucket, every upload currently +not subject to direct upload will get its benefits, and new features +could be shipped with a single merge request. + +Shipping MinIO with Omnibus will allow us to default new installations +to object storage, and Omnibus could take care of creating +buckets. This will simplify HA installation outside of Kubernetes. + +Then we can migrate each CarrierWave uploader to the new +implementation, up to a point where GitLab installation will only +require one bucket. + +## Additional reading materials + +- [Uploads development documentation: The problem description](../../../development/uploads.md#the-problem-description). +- [Speed up the monolith, building a smart reverse proxy in Go](https://archive.fosdem.org/2020/schedule/event/speedupmonolith/): a presentation explaining a bit of workhorse history and the challenge we faced in releasing the first cloud-native installation. +- [Object Storage improvements epic](https://gitlab.com/groups/gitlab-org/-/epics/483). +- We are moving to GraphQL API, but [we do not support direct upload](https://gitlab.com/gitlab-org/gitlab/-/issues/280819). + +## Who + +Proposal: + +<!-- vale gitlab.Spelling = NO --> + +| Role | Who | +|--------------------------------|-------------------------| +| Author | Alessio Caiazza | +| Architecture Evolution Coach | Gerardo Lopez-Fernandez | +| Engineering Leader | Marin Jankovski | +| Domain Expert / Object storage | Stan Hu | +| Domain Expert / Security | Joern Schneeweisz | + +DRIs: + +The DRI for this blueprint is the [Object Storage Working +Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/). + +<!-- vale gitlab.Spelling = YES --> diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index 57cbf387115..491454aed28 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -27,7 +27,7 @@ can't link to files outside it. ### Cache -- Define cache per job by using the `cache:` keyword. Otherwise it is disabled. +- Define cache per job by using the `cache` keyword. Otherwise it is disabled. - Subsequent pipelines can use the cache. - Subsequent jobs in the same pipeline can use the cache, if the dependencies are identical. - Different projects cannot share the cache. diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md index fbfcdcbf64f..7bc138d083d 100644 --- a/doc/ci/ci_cd_for_external_repos/index.md +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -9,33 +9,34 @@ type: index, howto >[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4642) in GitLab 10.6. -GitLab CI/CD can be used with: +INFO: +Get external repo access and more by upgrading to GitLab Ultimate. +[Try a free 30-day trial now](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-ci-cd-external-docs). -- [GitHub](github_integration.md). -- [Bitbucket Cloud](bitbucket_integration.md). -- Any other Git server. +GitLab CI/CD can be used with [GitHub](github_integration.md), [Bitbucket Cloud](bitbucket_integration.md), or any other +Git server. Instead of moving your entire project to GitLab, you can connect your external repository to get the benefits of GitLab CI/CD. Connecting an external repository sets up [repository mirroring](../../user/project/repository/mirror/index.md) -and create a lightweight project with issues, merge requests, wiki, and +and creates a lightweight project with issues, merge requests, wiki, and snippets disabled. These features [can be re-enabled later](../../user/project/settings/index.md#sharing-and-permissions). +## Connect to an external repository + To connect to an external repository: <!-- vale gitlab.Spelling = NO --> -1. On the top menu, select **Projects > Create new project**. +1. On the top bar, select **Menu > Projects > Create new project**. 1. Select **Run CI/CD for external repository**. 1. Select **GitHub** or **Repo by URL**. 1. Complete the fields. <!-- vale gitlab.Spelling = YES --> - - ## Pipelines for external pull requests > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/65139) in GitLab 12.3. diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index f26a678962a..abf43390834 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -66,9 +66,9 @@ as quickly as possible. ## Usage -Relationships are defined between jobs using the [`needs:` keyword](../yaml/index.md#needs). +Relationships are defined between jobs using the [`needs` keyword](../yaml/index.md#needs). -Note that `needs:` also works with the [parallel](../yaml/index.md#parallel) keyword, +Note that `needs` also works with the [parallel](../yaml/index.md#parallel) keyword, giving you powerful options for parallelization within your pipeline. ## Limitations @@ -87,7 +87,7 @@ are certain use cases that you may need to work around. For more information, ch The needs visualization makes it easier to visualize the relationships between dependent jobs in a DAG. This graph displays all the jobs in a pipeline that need or are needed by other jobs. Jobs with no relationships are not displayed in this view. -To see the needs visualization, click on the **Needs** tab when viewing a pipeline that uses the `needs:` keyword. +To see the needs visualization, click on the **Needs** tab when viewing a pipeline that uses the `needs` keyword.  diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index f6a6e892177..3a05e9aa7d9 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -548,7 +548,7 @@ kubectl create configmap docker-daemon --namespace gitlab-runner --from-file /tm ``` NOTE: -Make sure to use the namespace that the GitLab Runner Kubernetes executor uses +Make sure to use the namespace that the Kubernetes executor for GitLab Runner uses to create job pods in. After the ConfigMap is created, you can update the `config.toml` diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index 78f30b29e06..ca7b01edf39 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -119,17 +119,17 @@ The other pipelines don't get the protected variable. You can also We recommend that you use protected variables on protected environments to make sure that the secrets aren't exposed unintentionally. You can also define production secrets on the [runner side](../runners/configure_runners.md#prevent-runners-from-revealing-sensitive-information). -This prevents other maintainers from reading the secrets and makes sure that the runner only runs on -protected branches. +This prevents other users with the [Maintainer role](../../user/permissions.md) from reading the secrets and makes sure +that the runner only runs on protected branches. For more information, see [pipeline security](../pipelines/index.md#pipeline-security-on-protected-branches). ## Separate project for deployments -All project maintainers have access to production secrets. If you need to limit the number of users +All users with the Maintainer role for the project have access to production secrets. If you need to limit the number of users that can deploy to a production environment, you can create a separate project and configure a new permission model that isolates the CD permissions from the original project and prevents the -original project's maintainers from accessing the production secret and CD configuration. You can +original users with the Maintainer role for the project from accessing the production secret and CD configuration. You can connect the CD project to your development projects by using [multi-project pipelines](../pipelines/multi_project_pipelines.md). ## Protect `gitlab-ci.yml` from change diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 87d7b9f9e30..561507cab97 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -256,7 +256,7 @@ GitLab supports the [dotenv (`.env`)](https://github.com/bkeepers/dotenv) file f and expands the `environment:url` value with variables defined in the `.env` file. To use this feature, specify the -[`artifacts:reports:dotenv`](../yaml/index.md#artifactsreportsdotenv) keyword in `.gitlab-ci.yml`. +[`artifacts:reports:dotenv`](../yaml/artifacts_reports.md#artifactsreportsdotenv) keyword in `.gitlab-ci.yml`. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Set dynamic URLs after a job finished](https://youtu.be/70jDXtOf4Ig). @@ -294,7 +294,7 @@ As soon as the `review` job finishes, GitLab updates the `review/your-branch-nam environment's URL. It parses the `deploy.env` report artifact, registers a list of variables as runtime-created, uses it for expanding `environment:url: $DYNAMIC_ENVIRONMENT_URL` and sets it to the environment URL. -You can also specify a static part of the URL at `environment:url:`, such as +You can also specify a static part of the URL at `environment:url`, such as `https://$DYNAMIC_ENVIRONMENT_URL`. If the value of `DYNAMIC_ENVIRONMENT_URL` is `example.com`, the final result is `https://example.com`. @@ -303,7 +303,7 @@ The assigned URL for the `review/your-branch-name` environment is visible in the Note the following: - `stop_review` doesn't generate a dotenv report artifact, so it doesn't recognize the - `DYNAMIC_ENVIRONMENT_URL` environment variable. Therefore you shouldn't set `environment:url:` in the + `DYNAMIC_ENVIRONMENT_URL` environment variable. Therefore you shouldn't set `environment:url` in the `stop_review` job. - If the environment URL isn't valid (for example, the URL is malformed), the system doesn't update the environment URL. @@ -447,6 +447,63 @@ try to check out the code after the branch is deleted. Read more in the [`.gitlab-ci.yml` reference](../yaml/index.md#environmenton_stop). +#### Stop an environment when another job is finished + +You can set an environment to stop when another job is finished. + +In your `.gitlab-ci.yml` file, specify in the [`on_stop`](../yaml/index.md#environmenton_stop) +keyword the name of the job that stops the environment. + +The following example shows a `review_app` job that calls a `stop_review_app` job after the first +job is finished. The `stop_review_app` is triggered based on what is defined under `when`. In this +case, it is set to `manual`, so it needs a +[manual action](../jobs/job_control.md#create-a-job-that-must-be-run-manually) +from the GitLab UI to run. + +Both jobs must have the same rules or only/except configuration. +In this example, if the configuration is not identical: + +- The `stop_review_app` job might not be included in all pipelines that include the `review_app` job. +- It is not possible to trigger the `action: stop` to stop the environment automatically. + +Also in the example, `GIT_STRATEGY` is set to `none`. If the +`stop_review_app` job is [automatically triggered](../environments/index.md#stop-an-environment), +the runner won't try to check out the code after the branch is deleted. + +The example also overwrites global variables. If your `stop` `environment` job depends +on global variables, use [anchor variables](../yaml/yaml_optimization.md#yaml-anchors-for-variables) when you set the `GIT_STRATEGY` +to change the job without overriding the global variables. + +The `stop_review_app` job **must** have the following keywords defined: + +- `when`, defined at either: + - [The job level](../yaml/index.md#when). + - [In a rules clause](../yaml/index.md#rules). If you use `rules` and `when: manual`, you should + also set [`allow_failure: true`](../yaml/index.md#allow_failure) so the pipeline can complete + even if the job doesn't run. +- `environment:name` +- `environment:action` + +```yaml +review_app: + stage: deploy + script: make deploy-app + environment: + name: review/$CI_COMMIT_REF_SLUG + url: https://$CI_ENVIRONMENT_SLUG.example.com + on_stop: stop_review_app + +stop_review_app: + stage: deploy + variables: + GIT_STRATEGY: none + script: make delete-app + when: manual + environment: + name: review/$CI_COMMIT_REF_SLUG + action: stop +``` + #### Stop an environment after a certain time period > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20956) in GitLab 12.8. @@ -716,10 +773,11 @@ fetch = +refs/environments/*:refs/remotes/origin/environments/* ### Archive Old Deployments -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73628) in GitLab 14.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73628) in GitLab 14.5. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.6. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `deployments_archive`. On GitLab.com, this feature will be rolled out gradually. +On self-managed GitLab, by default this feature is available. To hide the feature per project or for your entire instance, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `deployments_archive`. On GitLab.com, this feature is available. When a new deployment happens in your project, GitLab creates [a special Git-ref to the deployment](#check-out-deployments-locally). @@ -878,11 +936,6 @@ To ensure the `action: stop` can always run when needed, you can: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21182) in GitLab 14.4. -FLAG: -On self-managed GitLab, by default this bug fix is available. To hide the bug fix per project, -ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `surface_environment_creation_failure`. -On GitLab.com, this bug fix is available. - If your project is configured to [create a dynamic environment](#create-a-dynamic-environment), you might encounter this error because the dynamically generated parameter can't be used for creating an environment. diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index 55b63dd090d..57fd72863c1 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -79,7 +79,7 @@ Alternatively, you can use the API to protect an environment: $ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ --data "user_id=3222377&access_level=20" "https://gitlab.com/api/v4/groups/9899826/members" - {"id":3222377,"name":"Sean Carroll","username":"sfcarroll","state":"active","avatar_url":"https://assets.gitlab-static.net/uploads/-/system/user/avatar/3222377/avatar.png","web_url":"https://gitlab.com/sfcarroll","access_level":20,"created_at":"2020-10-26T17:37:50.309Z","expires_at":null} + {"id":3222377,"name":"Sean Carroll","username":"sfcarroll","state":"active","avatar_url":"https://gitlab.com/uploads/-/system/user/avatar/3222377/avatar.png","web_url":"https://gitlab.com/sfcarroll","access_level":20,"created_at":"2020-10-26T17:37:50.309Z","expires_at":null} ``` 1. Use the API to add the group to the project as a reporter: @@ -194,16 +194,15 @@ and are protected at the same time. In an enterprise organization, with thousands of projects under a single group, ensuring that all of the [project-level protected environments](#protecting-environments) are properly configured is not a scalable solution. For example, a developer -might gain privileged access to a higher environment when they are added as a -maintainer to a new project. Group-level protected environments can be a solution -in this situation. +might gain privileged access to a higher environment when they are given the [Maintainer role](../../user/permissions.md) +for a new project. Group-level protected environments can be a solution in this situation. To maximize the effectiveness of group-level protected environments, [group-level memberships](../../user/group/index.md) must be correctly configured: -- Operators should be assigned the [maintainer role](../../user/permissions.md) - (or above) to the top-level group. They can maintain CI/CD configurations for +- Operators should be given at least the [Maintainer role](../../user/permissions.md) + for the top-level group. They can maintain CI/CD configurations for the higher environments (such as production) in the group-level settings page, which includes group-level protected environments, [group-level runners](../runners/runners_scope.md#group-runners), and @@ -211,9 +210,9 @@ configured: configurations are inherited to the child projects as read-only entries. This ensures that only operators can configure the organization-wide deployment ruleset. -- Developers should be assigned the [developer role](../../user/permissions.md) - (or below) at the top-level group, or explicitly assigned to a child project - as maintainers. They do *NOT* have access to the CI/CD configurations in the +- Developers should be given no more than the [Developer role](../../user/permissions.md) + for the top-level group, or explicitly given the [Maintainer role](../../user/permissions.md) for a child project + They do *NOT* have access to the CI/CD configurations in the top-level group, so operators can ensure that the critical configuration won't be accidentally changed by the developers. - For sub-groups and child projects: @@ -225,7 +224,7 @@ configured: environment configurations exist, to run a deployment job, the user must be allowed in **both** rulesets. - In a project or a subgroup of the top-level group, developers can be - safely assigned the Maintainer role to tune their lower environments (such + safely assigned the [Maintainer role](../../user/permissions.md) to tune their lower environments (such as `testing`). Having this configuration in place: diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 238c76b2c3c..1141583df3f 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -22,8 +22,7 @@ This tutorial assumes you are familiar with GitLab CI/CD and Vault. To follow along, you must have: - An account on GitLab. -- A running Vault server and access to it is required to configure authentication and create roles - and policies. For HashiCorp Vaults, this can be the Open Source or Enterprise version. +- Access to a running Vault server (at least v1.2.0) to configure authentication and to create roles and policies. For HashiCorp Vaults, this can be the Open Source or Enterprise version. NOTE: You must replace the `vault.example.com` URL below with the URL of your Vault server, and `gitlab.example.com` with the URL of your GitLab instance. diff --git a/doc/ci/examples/semantic-release.md b/doc/ci/examples/semantic-release.md index eb758218f17..c74af852a9a 100644 --- a/doc/ci/examples/semantic-release.md +++ b/doc/ci/examples/semantic-release.md @@ -56,7 +56,7 @@ default: { echo "@${CI_PROJECT_ROOT_NAMESPACE}:registry=${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/npm/" echo "${CI_API_V4_URL#https?}/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=\${CI_JOB_TOKEN}" - } | tee --append .npmrc + } | tee -a .npmrc cache: key: ${CI_COMMIT_REF_SLUG} paths: diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index dc5faf0188e..2a002b8fb9f 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -59,7 +59,7 @@ To make submodules work correctly in CI/CD jobs: variables: GIT_SUBMODULE_STRATEGY: recursive ``` - + If you use the [`CI_JOB_TOKEN`](jobs/ci_job_token.md) to clone a submodule in a pipeline job, the user executing the job must be assigned to a role that has [permission](../user/permissions.md#gitlab-cicd-permissions) to trigger a pipeline diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index b6a3011a3d6..532a0dffbce 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -4,7 +4,7 @@ group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab CI/CD job token +# GitLab CI/CD job token **(FREE)** When a pipeline job is about to run, GitLab generates a unique token and injects it as the [`CI_JOB_TOKEN` predefined variable](../variables/predefined_variables.md). @@ -61,11 +61,7 @@ tries to steal tokens from other jobs. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328553) in GitLab 14.1. [Deployed behind the `:ci_scoped_job_token` feature flag](../../user/feature_flags.md), disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/332272) in GitLab 14.4. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, -ask an administrator to [disable the `ci_scoped_job_token` flag](../../administration/feature_flags.md). -On GitLab.com, this feature is available. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/332272) in GitLab 14.6. You can limit the access scope of a project's CI/CD job token to increase the job token's security. A job token might give extra permissions that aren't necessary @@ -95,7 +91,7 @@ The job token scope is only for controlling access to private projects. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Token Access**. 1. Toggle **Limit CI_JOB_TOKEN access** to enabled. -1. (Optional) Add existing projects to the token's access scope. The user adding a +1. Optional. Add existing projects to the token's access scope. The user adding a project must have the [maintainer role](../../user/permissions.md) in both projects. There is [a proposal](https://gitlab.com/groups/gitlab-org/-/epics/3559) to improve @@ -121,7 +117,7 @@ trigger_pipeline: ``` If you use the `CI_PIPELINE_SOURCE` [predefined CI/CD variable](../variables/predefined_variables.md) -in a pipeline triggered this way, [the value is `pipeline` (not `triggered`)](../triggers/index.md#authentication-tokens). +in a pipeline triggered this way, [the value is `pipeline` (not `triggered`)](../triggers/index.md#configure-cicd-jobs-to-run-in-triggered-pipelines). ## Download an artifact from a different pipeline **(PREMIUM)** diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index 0f92ae5ca49..596df34b5c2 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -79,7 +79,7 @@ job: - In **all other cases**, the job is added to the pipeline, with `when: on_success`. WARNING: -If you use a `when:` clause as the final rule (not including `when: never`), two +If you use a `when` clause as the final rule (not including `when: never`), two simultaneous pipelines may start. Both push pipelines and merge request pipelines can be triggered by the same event (a push to the source branch for an open merge request). See how to [prevent duplicate pipelines](#avoid-duplicate-pipelines) @@ -153,7 +153,7 @@ To avoid duplicate pipelines, you can: - Use [`workflow`](../yaml/index.md#workflow) to specify which types of pipelines can run. - Rewrite the rules to run the job only in very specific cases, - and avoid a final `when:` rule: + and avoid a final `when` rule: ```yaml job: @@ -225,7 +225,7 @@ check the value of the `$CI_PIPELINE_SOURCE` variable: | `pipeline` | For [multi-project pipelines](../pipelines/multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api), or the [`trigger`](../yaml/index.md#trigger) keyword. | | `push` | For pipelines triggered by a `git push` event, including for branches and tags. | | `schedule` | For [scheduled pipelines](../pipelines/schedules.md). | -| `trigger` | For pipelines created by using a [trigger token](../triggers/index.md#authentication-tokens). | +| `trigger` | For pipelines created by using a [trigger token](../triggers/index.md#configure-cicd-jobs-to-run-in-triggered-pipelines). | | `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | | `webide` | For pipelines created by using the [WebIDE](../../user/project/web_ide/index.md). | @@ -335,7 +335,7 @@ to control when to add jobs to pipelines. In the following example, `job` runs only for: - Git tags -- [Triggers](../triggers/index.md#authentication-tokens) +- [Triggers](../triggers/index.md#configure-cicd-jobs-to-run-in-triggered-pipelines) - [Scheduled pipelines](../pipelines/schedules.md) ```yaml @@ -480,8 +480,8 @@ All files are considered to have changed when a scheduled pipeline runs. If you use multiple keywords with `only` or `except`, the keywords are evaluated as a single conjoined expression. That is: -- `only:` includes the job if **all** of the keys have at least one condition that matches. -- `except:` excludes the job if **any** of the keys have at least one condition that matches. +- `only` includes the job if **all** of the keys have at least one condition that matches. +- `except` excludes the job if **any** of the keys have at least one condition that matches. With `only`, individual keys are logically joined by an `AND`. A job is added to the pipeline if the following is true: @@ -634,7 +634,7 @@ timed rollout 10%: start_in: 30 minutes ``` -To stop the active timer of a delayed job, click the **{time-out}** (**Unschedule**) button. +To stop the active timer of a delayed job, select **Unschedule** (**{time-out}**). This job can no longer be scheduled to run automatically. You can, however, execute the job manually. To start a delayed job immediately, select **Play** (**{play}**). diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index 76e34df1f8c..f3044a03e04 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -259,5 +259,5 @@ For very active repositories with a large number of references and files, you ca must be configured per-repository. The pack-objects cache also automatically works for forks. On GitLab.com, where the pack-objects cache is enabled on all Gitaly servers, we found that we no longer need a pre-clone step for `gitlab-org/gitlab` development. - Optimize your CI/CD jobs by seeding repository data in a pre-clone step with the - [`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) of GitLab Runner. See the - [Runner Cloud for Linux](../runners/runner_cloud/linux_runner_cloud.md#pre-clone-script) for more details. + [`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) of GitLab Runner. See + [SaaS runners on Linux](../runners/saas/linux_saas_runner.md#pre-clone-script) for details. diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index 5343af16489..eb302b9ed7f 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -37,7 +37,7 @@ For an MR, the values of these metrics from the feature branch are compared to t ## How to set it up -Add a job that creates a [metrics report](yaml/index.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. +Add a job that creates a [metrics report](yaml/artifacts_reports.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. For example: diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index c2c06375d7b..ef6f28e36e5 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -78,7 +78,7 @@ There are some high level differences between the products worth mentioning: - on [schedule](../pipelines/schedules.md) - from the [GitLab UI](../pipelines/index.md#run-a-pipeline-manually) - by [API call](../triggers/index.md) - - by [webhook](../triggers/index.md#triggering-a-pipeline-from-a-webhook) + - by [webhook](../triggers/index.md#use-a-webhook) - by [ChatOps](../chatops/index.md) - You can control which jobs run in which cases, depending on how they are triggered, @@ -146,15 +146,15 @@ as well. Jenkins Pipelines are based on [Groovy](https://groovy-lang.org/), so the pipeline specification is written as code. GitLab works a bit differently, we use the more highly structured [YAML](https://yaml.org/) format, which -places scripting elements inside of `script:` blocks separate from the pipeline specification itself. +places scripting elements inside of `script` blocks separate from the pipeline specification itself. This is a strength of GitLab, in that it helps keep the learning curve much simpler to get up and running and avoids some of the problem of unconstrained complexity which can make your Jenkinsfile hard to understand and manage. That said, we do of course still value DRY (don't repeat yourself) principles and want to ensure that -behaviors of your jobs can be codified once and applied as needed. You can use the `extends:` syntax to -[reuse configuration in your jobs](../yaml/index.md#extends), and `include:` can +behaviors of your jobs can be codified once and applied as needed. You can use the `extends` syntax to +[reuse configuration in your jobs](../yaml/index.md#extends), and `include` can be used to [reuse pipeline configurations](../yaml/index.md#include) in pipelines in different projects: @@ -174,7 +174,7 @@ rspec: ## Artifact publishing Artifacts may work a bit differently than you've used them with Jenkins. In GitLab, any job can define -a set of artifacts to be saved by using the `artifacts:` keyword. This can be configured to point to a file +a set of artifacts to be saved by using the `artifacts` keyword. This can be configured to point to a file or set of files that can then be persisted from job to job. Read more on our detailed [artifacts documentation](../pipelines/job_artifacts.md): @@ -271,7 +271,7 @@ default: GitLab CI/CD also lets you define stages, but is a little bit more free-form to configure. The GitLab [`stages` keyword](../yaml/index.md#stages) is a top level setting that enumerates the list of stages, but you are not required to nest individual jobs underneath the `stages` section. Any job defined in the `.gitlab-ci.yml` can be made a part of any stage through use of the -[`stage:` keyword](../yaml/index.md#stage). +[`stage` keyword](../yaml/index.md#stage). Note that, unless otherwise specified, every pipeline is instantiated with a `build`, `test`, and `deploy` stage which are run in that order. Jobs that have no `stage` defined are placed by default in the `test` stage. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index 7ecee5508ef..e47b6dddc5f 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -48,7 +48,171 @@ is used. If you run two types of pipelines (like branch and scheduled) for the same ref, the pipeline that finishes later creates the job artifact. -For more examples, view the [keyword reference for the `.gitlab-ci.yml` file](../yaml/index.md#artifacts). +To disable artifact passing, define the job with empty [dependencies](../yaml/index.md#dependencies): + +```yaml +job: + stage: build + script: make build + dependencies: [] +``` + +You may want to create artifacts only for tagged releases to avoid filling the +build server storage with temporary build artifacts. For example, use [`rules`](../yaml/index.md#rules) +to create artifacts only for tags: + +```yaml +default-job: + script: + - mvn test -U + rules: + - if: $CI_COMMIT_BRANCH + +release-job: + script: + - mvn package -U + artifacts: + paths: + - target/*.war + rules: + - if: $CI_COMMIT_TAG +``` + +You can use wildcards for directories too. For example, if you want to get all the +files inside the directories that end with `xyz`: + +```yaml +job: + artifacts: + paths: + - path/*xyz/* +``` + +### Use CI/CD variables to define the artifacts name + +You can use [CI/CD variables](../variables/index.md) to dynamically define the +artifacts file's name. + +For example, to create an archive with a name of the current job: + +```yaml +job: + artifacts: + name: "$CI_JOB_NAME" + paths: + - binaries/ +``` + +To create an archive with a name of the current branch or tag including only +the binaries directory: + +```yaml +job: + artifacts: + name: "$CI_COMMIT_REF_NAME" + paths: + - binaries/ +``` + +If your branch-name contains forward slashes +(for example `feature/my-feature`) it's advised to use `$CI_COMMIT_REF_SLUG` +instead of `$CI_COMMIT_REF_NAME` for proper naming of the artifact. + +To create an archive with a name of the current job and the current branch or +tag including only the binaries directory: + +```yaml +job: + artifacts: + name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME" + paths: + - binaries/ +``` + +To create an archive with a name of the current [stage](../yaml/index.md#stages) and branch name: + +```yaml +job: + artifacts: + name: "$CI_JOB_STAGE-$CI_COMMIT_REF_NAME" + paths: + - binaries/ +``` + +If you use **Windows Batch** to run your shell scripts you must replace +`$` with `%`: + +```yaml +job: + artifacts: + name: "%CI_JOB_STAGE%-%CI_COMMIT_REF_NAME%" + paths: + - binaries/ +``` + +If you use **Windows PowerShell** to run your shell scripts you must replace +`$` with `$env:`: + +```yaml +job: + artifacts: + name: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_NAME" + paths: + - binaries/ +``` + +### Exclude files from job artifacts + +Use [`artifacts:exclude`](../yaml/index.md#artifactsexclude) to prevent files from +being added to an artifacts archive. + +For example, to store all files in `binaries/`, but not `*.o` files located in +subdirectories of `binaries/`. + +```yaml +artifacts: + paths: + - binaries/ + exclude: + - binaries/**/*.o +``` + +Unlike [`artifacts:paths`](../yaml/index.md#artifactspaths), `exclude` paths are not recursive. +To exclude all of the contents of a directory, match them explicitly rather +than matching the directory itself. + +For example, to store all files in `binaries/` but nothing located in the `temp/` subdirectory: + +```yaml +artifacts: + paths: + - binaries/ + exclude: + - binaries/temp/**/* +``` + +### Add untracked files to artifacts + +Use [`artifacts:untracked`](../yaml/index.md#artifactsuntracked) to add all Git untracked +files as artifacts (along with the paths defined in [`artifacts:paths`](../yaml/index.md#artifactspaths)). + +Save all Git untracked files and files in `binaries`: + +```yaml +artifacts: + untracked: true + paths: + - binaries/ +``` + +Save all untracked files but [exclude](../yaml/index.md#artifactsexclude) `*.txt`: + +```yaml +artifacts: + untracked: true + exclude: + - "*.txt" +``` ## Download job artifacts @@ -103,6 +267,35 @@ To delete a job: 1. On the top right of the job's log, select **Erase job log** (**{remove}**). 1. On the confirmation dialog, select **OK**. +## Expose job artifacts in the merge request UI + +Use the [`artifacts:expose_as`](../yaml/index.md#artifactsexpose_as) keyword to expose +[job artifacts](../pipelines/job_artifacts.md) in the [merge request](../../user/project/merge_requests/index.md) UI. + +For example, to match a single file: + +```yaml +test: + script: ["echo 'test' > file.txt"] + artifacts: + expose_as: 'artifact 1' + paths: ['file.txt'] +``` + +With this configuration, GitLab adds a link **artifact 1** to the relevant merge request +that points to `file.txt`. To access the link, select **View exposed artifact** +below the pipeline graph in the merge request overview. + +An example that matches an entire directory: + +```yaml +test: + script: ["mkdir test && echo 'test' > test/file.txt"] + artifacts: + expose_as: 'artifact 1' + paths: ['test/'] +``` + ## Retrieve job artifacts for other projects To retrieve a job artifact from a different project, you might need to use a @@ -182,6 +375,14 @@ job artifacts are deleted. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/229936) in GitLab 13.4. > - [Made optional with a CI/CD setting](https://gitlab.com/gitlab-org/gitlab/-/issues/241026) in GitLab 13.8. +By default artifacts are always kept for the most recent successful pipeline for +each ref. This means that the latest artifacts do not immediately expire according +to the `expire_in` specification. + +If a new pipeline for the same ref completes successfully, the previous pipeline's +artifacts are deleted according to the `expire_in` configuration. The artifacts +of the new pipeline are kept automatically. + Keeping the latest artifacts can use a large amount of storage space in projects with a lot of jobs or large artifacts. If the latest artifacts are not needed in a project, you can disable this behavior to save space: @@ -194,9 +395,6 @@ a project, you can disable this behavior to save space: You can disable this behavior for all projects on a self-managed instance in the [instance's CI/CD settings](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). -When you disable the feature, the latest artifacts do not immediately expire. -A new pipeline must run before the latest artifacts can expire and be deleted. - ## Troubleshooting job artifacts ### Error message `No files to upload` diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md index 119633d38e2..85e5b62b0c4 100644 --- a/doc/ci/pipelines/merge_request_pipelines.md +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -112,11 +112,11 @@ C: - merge_requests ``` -- `A` and `B` always run, because they get the `only:` rule to execute in all cases. +- `A` and `B` always run, because they get the `only` rule to execute in all cases. - `C` only runs for merge requests. It doesn't run for any pipeline except a merge request pipeline. -In this example, you don't have to add the `only:` rule to all of your jobs to make +In this example, you don't have to add the `only` rule to all of your jobs to make them always run. You can use this format to set up a Review App, which helps to save resources. diff --git a/doc/ci/pipelines/merge_trains.md b/doc/ci/pipelines/merge_trains.md index 6074909a887..593cdb68b3f 100644 --- a/doc/ci/pipelines/merge_trains.md +++ b/doc/ci/pipelines/merge_trains.md @@ -11,6 +11,10 @@ last_update: 2019-07-03 > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9186) in GitLab 12.0. > - [Squash and merge](../../user/project/merge_requests/squash_and_merge.md) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13001) in GitLab 12.6. +INFO: +Get merge trains and more in GitLab Ultimate. +[Try a free 30-day trial now](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-ci-cd-external-docs). + For more information about why you might want to use merge trains, read [How merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). When [pipelines for merged results](pipelines_for_merged_results.md) are @@ -35,7 +39,8 @@ If the pipeline for the merge request at the front of the train completes succes the changes are merged into the target branch, and the other pipelines continue to run. -To add a merge request to a merge train, you need [permissions](../../user/permissions.md) to push to the target branch. +To add a merge request to a merge train, you need [permissions](../../user/permissions.md) to merge or push to the +target branch. Each merge train can run a maximum of **twenty** pipelines in parallel. If more than twenty merge requests are added to the merge train, the merge requests diff --git a/doc/ci/pipelines/multi_project_pipelines.md b/doc/ci/pipelines/multi_project_pipelines.md index 30b3bc2e277..8a83e7e31f4 100644 --- a/doc/ci/pipelines/multi_project_pipelines.md +++ b/doc/ci/pipelines/multi_project_pipelines.md @@ -213,7 +213,7 @@ In the upstream pipeline: ``` 1. Set the `test` job in the downstream pipeline to inherit the variables from the `build_vars` - job in the upstream project with `needs:`. The `test` job inherits the variables in the + job in the upstream project with `needs`. The `test` job inherits the variables in the `dotenv` report and it can access `BUILD_VERSION` in the script: ```yaml diff --git a/doc/ci/pipelines/parent_child_pipelines.md b/doc/ci/pipelines/parent_child_pipelines.md index 64f4160c963..5e4b707a38c 100644 --- a/doc/ci/pipelines/parent_child_pipelines.md +++ b/doc/ci/pipelines/parent_child_pipelines.md @@ -31,10 +31,9 @@ set of concurrently running child pipelines, but within the same project: - Child pipelines still execute each of their jobs according to a stage sequence, but would be free to continue forward through their stages without waiting for unrelated jobs in the parent pipeline to finish. -- The configuration is split up into smaller child pipeline configurations, which are +- The configuration is split up into smaller child pipeline configurations. Each child pipeline contains only relevant steps which are easier to understand. This reduces the cognitive load to understand the overall configuration. - Imports are done at the child pipeline level, reducing the likelihood of collisions. -- Each pipeline has only relevant steps, making it easier to understand what's going on. Child pipelines work well with other GitLab CI/CD features: @@ -43,7 +42,7 @@ Child pipelines work well with other GitLab CI/CD features: - Since the parent pipeline in `.gitlab-ci.yml` and the child pipeline run as normal pipelines, they can have their own behaviors and sequencing in relation to triggers. -See the [`trigger:`](../yaml/index.md#trigger) keyword documentation for full details on how to +See the [`trigger`](../yaml/index.md#trigger) keyword documentation for full details on how to include the child pipeline configuration. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> @@ -85,7 +84,7 @@ microservice_a: file: '/path/to/child-pipeline.yml' ``` -The maximum number of entries that are accepted for `trigger:include:` is three. +The maximum number of entries that are accepted for `trigger:include` is three. Similar to [multi-project pipelines](multi_project_pipelines.md#mirror-status-of-a-triggered-pipeline-in-the-trigger-job), we can set the parent pipeline to depend on the status of the child pipeline upon completion: diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index 1b23727b142..3ff22a16900 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -211,7 +211,7 @@ trigger_b: ``` Example child `a` pipeline configuration, located in `/a/.gitlab-ci.yml`, making -use of the DAG `needs:` keyword: +use of the DAG `needs` keyword: ```yaml stages: @@ -240,7 +240,7 @@ deploy_a: ``` Example child `b` pipeline configuration, located in `/b/.gitlab-ci.yml`, making -use of the DAG `needs:` keyword: +use of the DAG `needs` keyword: ```yaml stages: diff --git a/doc/ci/pipelines/pipelines_for_merged_results.md b/doc/ci/pipelines/pipelines_for_merged_results.md index 2acef9be557..718519faf48 100644 --- a/doc/ci/pipelines/pipelines_for_merged_results.md +++ b/doc/ci/pipelines/pipelines_for_merged_results.md @@ -10,6 +10,10 @@ last_update: 2019-07-03 > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7380) in GitLab 11.10. +INFO: +Get these pipelines and more in GitLab Ultimate. +[Try a free 30-day trial now](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-ci-cd-external-docs). + When you submit a merge request, you are requesting to merge changes from a source branch into a target branch. By default, the CI pipeline runs jobs against the source branch. diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index a8ecb5e0d74..cf470836e32 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -134,7 +134,7 @@ For example: - `my/path/.my-custom-file.yml@mygroup/another-project` - `my/path/.my-custom-file.yml@mygroup/another-project:refname` -If the configuration file is in a separate project, you can more set more granular permissions. For example: +If the configuration file is in a separate project, you can set more granular permissions. For example: - Create a public project to host the configuration file. - Give write permissions on the project only to users who are allowed to edit the file. @@ -267,7 +267,7 @@ when merging a merge request would cause the project's test coverage to decline. Follow these steps to enable the `Coverage-Check` MR approval rule: -1. Set up a [`coverage:`](../yaml/index.md#coverage) regular expression for all jobs you want to include in the overall coverage value. +1. Set up a [`coverage`](../yaml/index.md#coverage) regular expression for all jobs you want to include in the overall coverage value. 1. Go to your project and select **Settings > General**. 1. Expand **Merge request approvals**. 1. Select **Enable** next to the `Coverage-Check` approval rule. diff --git a/doc/ci/resource_groups/index.md b/doc/ci/resource_groups/index.md index 25dacc9c437..d31fb5561e9 100644 --- a/doc/ci/resource_groups/index.md +++ b/doc/ci/resource_groups/index.md @@ -52,7 +52,7 @@ deploy: ``` With this configuration, the safety on the deployments is assured while you -can still run `build` jobs concurrently for maximizing the pipeline efficency. +can still run `build` jobs concurrently for maximizing the pipeline efficiency. ## Requirements diff --git a/doc/ci/runners/build_cloud/linux_build_cloud.md b/doc/ci/runners/build_cloud/linux_build_cloud.md index e8bad31c821..2892a30cd2e 100644 --- a/doc/ci/runners/build_cloud/linux_build_cloud.md +++ b/doc/ci/runners/build_cloud/linux_build_cloud.md @@ -1,9 +1,9 @@ --- -redirect_to: '../runner_cloud/linux_runner_cloud.md' +redirect_to: '../saas/linux_saas_runner.md' remove_date: '2022-02-05' --- -This document was moved to [another location](../runner_cloud/linux_runner_cloud.md). +This document was moved to [another location](../saas/linux_saas_runner.md). <!-- This redirect file can be deleted after 2022-02-05. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/build_cloud/macos/environment.md b/doc/ci/runners/build_cloud/macos/environment.md index aaef0d07098..a534e87cc34 100644 --- a/doc/ci/runners/build_cloud/macos/environment.md +++ b/doc/ci/runners/build_cloud/macos/environment.md @@ -1,9 +1,9 @@ --- -redirect_to: '../../runner_cloud/macos/environment.md' +redirect_to: '../../saas/macos/environment.md' remove_date: '2022-02-05' --- -This document was moved to [another location](../../runner_cloud/macos/environment.md). +This document was moved to [another location](../../saas/macos/environment.md). <!-- This redirect file can be deleted after 2022-02-05. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/build_cloud/macos_build_cloud.md b/doc/ci/runners/build_cloud/macos_build_cloud.md index e478f93f34c..50b7e0cfb79 100644 --- a/doc/ci/runners/build_cloud/macos_build_cloud.md +++ b/doc/ci/runners/build_cloud/macos_build_cloud.md @@ -1,9 +1,9 @@ --- -redirect_to: '../runner_cloud/macos_runner_cloud.md' +redirect_to: '../saas/macos_saas_runner.md' remove_date: '2022-02-05' --- -This document was moved to [another location](../runner_cloud/macos_runner_cloud.md). +This document was moved to [another location](../saas/macos_saas_runner.md). <!-- This redirect file can be deleted after 2022-02-05. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/build_cloud/windows_build_cloud.md b/doc/ci/runners/build_cloud/windows_build_cloud.md index 8d57ecf27ed..fb64938eb9f 100644 --- a/doc/ci/runners/build_cloud/windows_build_cloud.md +++ b/doc/ci/runners/build_cloud/windows_build_cloud.md @@ -1,9 +1,9 @@ --- -redirect_to: '../runner_cloud/windows_runner_cloud.md' +redirect_to: '../saas/windows_saas_runner.md' remove_date: '2022-02-05' --- -This document was moved to [another location](../runner_cloud/windows_runner_cloud.md). +This document was moved to [another location](../saas/windows_saas_runner.md). <!-- This redirect file can be deleted after 2022-02-05. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index 9e30f4dbf4d..b2885262e9d 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -297,6 +297,7 @@ globally or for individual jobs: - [`TRANSFER_METER_FREQUENCY`](#artifact-and-cache-settings) (artifact/cache meter update frequency) - [`ARTIFACT_COMPRESSION_LEVEL`](#artifact-and-cache-settings) (artifact archiver compression level) - [`CACHE_COMPRESSION_LEVEL`](#artifact-and-cache-settings) (cache archiver compression level) +- [`CACHE_REQUEST_TIMEOUT`](#artifact-and-cache-settings) (cache request timeout) You can also use variables to configure how many times a runner [attempts certain stages of job execution](#job-stages-attempts). @@ -637,7 +638,10 @@ For [GitLab Pages](../../user/project/pages/index.md) to serve should use the `ARTIFACT_COMPRESSION_LEVEL: fastest` setting, as only uncompressed zip archives support this feature. -A meter can also be enabled to provide the rate of transfer for uploads and downloads. +A meter can be enabled to provide the rate of transfer for uploads and downloads. + +You can set a maximum time for cache upload and download with the `CACHE_REQUEST_TIMEOUT` setting. +This setting can be useful when slow cache uploads substantially increase the duration of your job. ```yaml variables: @@ -649,6 +653,9 @@ variables: # Use no compression for caches CACHE_COMPRESSION_LEVEL: "fastest" + + # Set maximum duration of cache upload and download + CACHE_REQUEST_TIMEOUT: 5 ``` | Variable | Description | @@ -656,3 +663,4 @@ variables: | `TRANSFER_METER_FREQUENCY` | Specify how often to print the meter's transfer rate. It can be set to a duration (for example, `1s` or `1m30s`). A duration of `0` disables the meter (default). When a value is set, the pipeline shows a progress meter for artifact and cache uploads and downloads. | | `ARTIFACT_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | | `CACHE_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | +| `CACHE_REQUEST_TIMEOUT` | Configure the maximum duration of cache upload and download operations for a single job in minutes. Default is `10` minutes. | diff --git a/doc/ci/runners/index.md b/doc/ci/runners/index.md index d408bc46609..b4e9fe818cf 100644 --- a/doc/ci/runners/index.md +++ b/doc/ci/runners/index.md @@ -5,12 +5,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# GitLab Runner Cloud **(FREE)** +# Runner SaaS **(FREE SAAS)** -If you are using self-managed GitLab or you want to use your own runners on GitLab.com, you can +If you are using self-managed GitLab or you use GitLab.com but want to use your own runners, you can [install and configure your own runners](https://docs.gitlab.com/runner/install/). -If you are using GitLab SaaS (GitLab.com), your CI jobs automatically run on runners in the GitLab Runner Cloud. +If you are using GitLab SaaS (GitLab.com), your CI jobs automatically run on runners provided by GitLab. No configuration is required. Your jobs can run on: - [Linux runners](build_cloud/linux_build_cloud.md). diff --git a/doc/ci/runners/runner_cloud/linux_runner_cloud.md b/doc/ci/runners/runner_cloud/linux_runner_cloud.md index d0fedfcabb2..2892a30cd2e 100644 --- a/doc/ci/runners/runner_cloud/linux_runner_cloud.md +++ b/doc/ci/runners/runner_cloud/linux_runner_cloud.md @@ -1,186 +1,9 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../saas/linux_saas_runner.md' +remove_date: '2022-02-05' --- -# Runner Cloud for Linux **(FREE)** +This document was moved to [another location](../saas/linux_saas_runner.md). -Runner Cloud runners for Linux run in autoscale mode and are powered by Google Cloud Platform. - -Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each job, thus maximizing security. These shared runners are available on GitLab.com. - -GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier. - -All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, Google COS and the latest Docker Engine -installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default -region of the VMs is US East1. -Each instance is used only for one job. This ensures that any sensitive data left on the system can't be accessed by other people's CI/CD jobs. - -NOTE: -The final disk space your jobs can use will be less than 25GB. Some disk space allocated to the instance will be occupied by the operating system, the Docker image, and a copy of your cloned repository. - -The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times. - -Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`), -**time out after 3 hours**, regardless of the timeout configured in a -project. Check the issues [#4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [#4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. - -Below are the runners' settings. - -| Setting | GitLab.com | Default | -| ----------- | ----------------- | ---------- | -| Executor | `docker+machine` | - | -| Default Docker image | `ruby:2.5` | - | -| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` | - -These runners share a [distributed cache](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) through use of a Google Cloud Storage (GCS) bucket. Cache contents not updated within the last 14 days are automatically removed through use of an [object lifecycle management policy](https://cloud.google.com/storage/docs/lifecycle). - -## Pre-clone script - -Cloud runners for Linux provide a way to run commands in a CI -job before the runner attempts to run `git init` and `git fetch` to -download a GitLab repository. The -[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) -can be used for: - -- Seeding the build directory with repository data -- Sending a request to a server -- Downloading assets from a CDN -- Any other commands that must run before the `git init` - -To use this feature, define a [CI/CD variable](../../../ci/variables/index.md#custom-cicd-variables) called -`CI_PRE_CLONE_SCRIPT` that contains a bash script. - -NOTE: -The `CI_PRE_CLONE_SCRIPT` variable does not work on Windows runners. - -### Pre-clone script example - -This example was used in the `gitlab-org/gitlab` project until November 2021. -The project no longer uses this optimization because the [pack-objects cache](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache) -lets Gitaly serve the full CI/CD fetch traffic. See [Git fetch caching](../../../development/pipelines.md#git-fetch-caching). - -The `CI_PRE_CLONE_SCRIPT` was defined as a project CI/CD variable: - -```shell -( - echo "Downloading archived master..." - wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz - - if [ ! -f /tmp/gitlab.tar.gz ]; then - echo "Repository cache not available, cloning a new directory..." - exit - fi - - rm -rf $CI_PROJECT_DIR - echo "Extracting tarball into $CI_PROJECT_DIR..." - mkdir -p $CI_PROJECT_DIR - cd $CI_PROJECT_DIR - tar xzf /tmp/gitlab.tar.gz - rm -f /tmp/gitlab.tar.gz - chmod a+w $CI_PROJECT_DIR -) -``` - -The first step of the script downloads `gitlab-master.tar.gz` from Google Cloud Storage. -There was a [GitLab CI/CD job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/5fb40526c8c8aaafc5f92eab36d5bbddaca3893d/.gitlab/ci/cache-repo.gitlab-ci.yml) -that was responsible for keeping that archive up-to-date. Every two hours on a scheduled pipeline, -it did the following: - -1. Create a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. -1. Save the data as a `.tar.gz`. -1. Upload it into the Google Cloud Storage bucket. - -When a job ran with this configuration, the output looked similar to: - -```shell -$ eval "$CI_PRE_CLONE_SCRIPT" -Downloading archived master... -Extracting tarball into /builds/gitlab-org/gitlab... -Fetching changes... -Reinitialized existing Git repository in /builds/gitlab-org/gitlab/.git/ -``` - -The `Reinitialized existing Git repository` message shows that -the pre-clone step worked. The runner runs `git init`, which -overwrites the Git configuration with the appropriate settings to fetch -from the GitLab repository. - -`CI_REPO_CACHE_CREDENTIALS` must contain the Google Cloud service account -JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. - -Note that this bucket should be located in the same continent as the -runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). - -## `config.toml` - -The full contents of our `config.toml` are: - -NOTE: -Settings that are not public are shown as `X`. - -**Google Cloud Platform** - -```toml -concurrent = X -check_interval = 1 -metrics_server = "X" -sentry_dsn = "X" - -[[runners]] - name = "docker-auto-scale" - request_concurrency = X - url = "https://gitlab.com/" - token = "SHARED_RUNNER_TOKEN" - pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\"" - executor = "docker+machine" - environment = [ - "DOCKER_DRIVER=overlay2", - "DOCKER_TLS_CERTDIR=" - ] - limit = X - [runners.docker] - image = "ruby:2.5" - privileged = true - volumes = [ - "/certs/client", - "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. - ] - [runners.machine] - IdleCount = 50 - IdleTime = 3600 - MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. - MachineName = "srm-%s" - MachineDriver = "google" - MachineOptions = [ - "google-project=PROJECT", - "google-disk-size=25", - "google-machine-type=n1-standard-1", - "google-username=core", - "google-tags=gitlab-com,srm", - "google-use-internal-ip", - "google-zone=us-east1-d", - "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928 - "google-machine-image=PROJECT/global/images/IMAGE", - "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. - "engine-opt=fixed-cidr-v6=fc00::/7", - "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 - ] - [[runners.machine.autoscaling]] - Periods = ["* * * * * sat,sun *"] - Timezone = "UTC" - IdleCount = 70 - IdleTime = 3600 - [[runners.machine.autoscaling]] - Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"] - Timezone = "UTC" - IdleCount = 700 - IdleTime = 3600 - [runners.cache] - Type = "gcs" - Shared = true - [runners.cache.gcs] - CredentialsFile = "/path/to/file" - BucketName = "bucket-name" -``` +<!-- This redirect file can be deleted after 2022-02-05. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/runner_cloud/macos/environment.md b/doc/ci/runners/runner_cloud/macos/environment.md index ddefad775c1..37ad21c28fc 100644 --- a/doc/ci/runners/runner_cloud/macos/environment.md +++ b/doc/ci/runners/runner_cloud/macos/environment.md @@ -1,43 +1,9 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../saas/macos/environment.md' +remove_date: '2022-02-05' --- -# VM instances and images for Runner Cloud for macOS **(FREE)** +This document was moved to [another location](../../saas/macos/environment.md). -When you use the Runner Cloud for macOS: - -- Each of your jobs runs in a newly provisioned VM, which is dedicated to the specific job. -- The VM is active only for the duration of the job and immediately deleted. - -## VM types - -The virtual machine where your job runs has `sudo` access with no password. -For the Beta, there is only one available machine type, `gbc-macos-large`. - -| Instance type | vCPUS | Memory (GB) | -| --------- | --- | ------- | -| `gbc-macos-large` | 4 | 10 | - -## VM images - -You can execute your build on one of the following images. -You specify this image in your `.gitlab-ci.yml` file. - -Each image is running a specific version of macOS and Xcode. - -| VM image | Included software | -|---------------------------|--------------------| -| macos-10.13-xcode-7 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | -| macos-10.13-xcode-8 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | -| macos-10.13-xcode-9 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | -| macos-10.14-xcode-10 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/mojave.yml> | -| macos-10.15-xcode-11 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/catalina.yml> | -| macos-11-xcode-12 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/big-sur.yml> | - -### Image update policy - -- Support for new macOS versions is planned. -- Additional details on the support policy and image update release process are documented - [in this project](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/55bf59c8fa88712960afff2bf6ecc5daa879a8f5/docs/overview.md#os-images). +<!-- This redirect file can be deleted after 2022-02-05. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
\ No newline at end of file diff --git a/doc/ci/runners/runner_cloud/macos_runner_cloud.md b/doc/ci/runners/runner_cloud/macos_runner_cloud.md index 332284fa8c1..50b7e0cfb79 100644 --- a/doc/ci/runners/runner_cloud/macos_runner_cloud.md +++ b/doc/ci/runners/runner_cloud/macos_runner_cloud.md @@ -1,62 +1,9 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../saas/macos_saas_runner.md' +remove_date: '2022-02-05' --- -# Runner Cloud for macOS (Beta) **(FREE SAAS)** +This document was moved to [another location](../saas/macos_saas_runner.md). -The Runner Cloud for macOS Beta provides on-demand runners integrated with GitLab SaaS [CI/CD](../../../ci/index.md). -Use these runners to build, test, and deploy apps for the Apple ecosystem (macOS, iOS, tvOS). You can take advantage -of all the capabilities of the GitLab single DevOps platform and not have to manage or operate a -build environment. - -Cloud runners for macOS are in [Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) -and shouldn't be relied upon for mission-critical production jobs. - -## Quickstart - -To start using Runner Cloud for macOS Beta, you must submit an access request [issue](https://gitlab.com/gitlab-com/macos-buildcloud-runners-beta/-/issues/new?issuable_template=beta_access_request). After your -access has been granted and your build environment configured, you must configure your -`.gitlab-ci.yml` pipeline file: - -1. Add a `.gitlab-ci.yml` file to your project repository. -1. Specify the [image](macos/environment.md#vm-images) you want to use. -1. Commit a change to your repository. - -The runners automatically run your build. - -## Example `.gitlab-ci.yml` file - -The following sample `.gitlab-ci.yml` file shows how to start using the runners for macOS: - -```yaml -.macos_buildcloud_runners: - tags: - - shared-macos-amd64 - image: macos-11-xcode-12 - -stages: - - build - - test - -before_script: - - echo "started by ${GITLAB_USER_NAME}" - -build: - extends: - - .macos_buildcloud_runners - stage: build - script: - - echo "running scripts in the build job" - -test: - extends: - - .macos_buildcloud_runners - stage: test - script: - - echo "running scripts in the test job" -``` - -NOTE: -During the Beta period, the architecture of this solution will change. Rather than the jobs running on a specific VM instance, they will run on an ephemeral VM instance that is created by an autoscaling instance, known as the Runner Manager. We will notify all Beta participants of any downtime required to do this work. +<!-- This redirect file can be deleted after 2022-02-05. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/runner_cloud/windows_runner_cloud.md b/doc/ci/runners/runner_cloud/windows_runner_cloud.md index ef4d4076c91..fb64938eb9f 100644 --- a/doc/ci/runners/runner_cloud/windows_runner_cloud.md +++ b/doc/ci/runners/runner_cloud/windows_runner_cloud.md @@ -1,155 +1,9 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../saas/windows_saas_runner.md' +remove_date: '2022-02-05' --- -# Runner Cloud for Windows (beta) **(FREE)** +This document was moved to [another location](../saas/windows_saas_runner.md). -Runner Cloud runners for Windows are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) -and shouldn't be used for production workloads. - -During this beta period, the [shared runner pipeline quota](../../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) -applies for groups and projects in the same manner as Linux runners. This may -change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834). - -Windows runners on GitLab.com autoscale by launching virtual machines on -the Google Cloud Platform. This solution uses an -[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md) -developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). -Windows runners execute your CI/CD jobs on `n1-standard-2` instances with -2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in -the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/main/cookbooks/preinstalled-software/README.md). - -We want to keep iterating to get Windows runners in a stable state and -[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). -You can follow our work towards this goal in the -[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162). - -## Configuration - -The full contents of our `config.toml` are: - -NOTE: -Settings that aren't public are shown as `X`. - -```toml -concurrent = X -check_interval = 3 - -[[runners]] - name = "windows-runner" - url = "https://gitlab.com/" - token = "TOKEN" - executor = "custom" - builds_dir = "C:\\GitLab-Runner\\builds" - cache_dir = "C:\\GitLab-Runner\\cache" - shell = "powershell" - [runners.custom] - config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"] - prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"] - run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"] - cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"] -``` - -The full contents of our `autoscaler/config.toml` are: - -```toml -Provider = "gcp" -Executor = "winrm" -OS = "windows" -LogLevel = "info" -LogFormat = "text" -LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log" -VMTag = "windows" - -[GCP] - ServiceAccountFile = "PATH" - Project = "some-project-df9323" - Zone = "us-east1-c" - MachineType = "n1-standard-2" - Image = "IMAGE" - DiskSize = 50 - DiskType = "pd-standard" - Subnetwork = "default" - Network = "default" - Tags = ["TAGS"] - Username = "gitlab_runner" - -[WinRM] - MaximumTimeout = 3600 - ExecutionMaxRetries = 0 - -[ProviderCache] - Enabled = true - Directory = "C:\\GitLab-Runner\\autoscaler\\machines" -``` - -## Example `.gitlab-ci.yml` file - -Below is a sample `.gitlab-ci.yml` file that shows how to start using the runners for Windows: - -```yaml -.shared_windows_runners: - tags: - - shared-windows - - windows - - windows-1809 - -stages: - - build - - test - -before_script: - - Set-Variable -Name "time" -Value (date -Format "%H:%m") - - echo ${time} - - echo "started by ${GITLAB_USER_NAME}" - -build: - extends: - - .shared_windows_runners - stage: build - script: - - echo "running scripts in the build job" - -test: - extends: - - .shared_windows_runners - stage: test - script: - - echo "running scripts in the test job" -``` - -## Limitations and known issues - -- All the limitations mentioned in our [beta - definition](https://about.gitlab.com/handbook/product/#beta). -- The average provisioning time for a new Windows VM is 5 minutes. - This means that you may notice slower build start times - on the Windows runner fleet during the beta. In a future - release we intend to update the autoscaler to enable - the pre-provisioning of virtual machines. This is intended to significantly reduce - the time it takes to provision a VM on the Windows fleet. You can - follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). -- The Windows runner fleet may be unavailable occasionally - for maintenance or updates. -- The Windows runner virtual machine instances do not use the - GitLab Docker executor. This means that you can't specify - [`image`](../../../ci/yaml/index.md#image) or [`services`](../../../ci/yaml/index.md#services) in - your pipeline configuration. -- For the beta release, we have included a set of software packages in - the base VM image. If your CI job requires additional software that's - not included in this list, then you must add installation - commands to [`before_script`](../../../ci/yaml/index.md#before_script) or [`script`](../../../ci/yaml/index.md#script) to install the required - software. Note that each job runs on a new VM instance, so the - installation of additional software packages needs to be repeated for - each job in your pipeline. -- The job may stay in a pending state for longer than the - Linux runners. -- There is the possibility that we introduce breaking changes which will - require updates to pipelines that are using the Windows runner - fleet. +<!-- This redirect file can be deleted after 2022-02-05. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/saas/linux_saas_runner.md b/doc/ci/runners/saas/linux_saas_runner.md new file mode 100644 index 00000000000..4d1e628b8e7 --- /dev/null +++ b/doc/ci/runners/saas/linux_saas_runner.md @@ -0,0 +1,188 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# SaaS runners on Linux **(FREE SAAS)** + +SaaS runners on Linux are autoscaled ephemeral Google Cloud Platform virtual machines. + +Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each job, thus maximizing security. These shared runners are available on GitLab.com. + +GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier. + +All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, Google COS and the latest Docker Engine +installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default +region of the VMs is US East1. +Each instance is used only for one job. This ensures that any sensitive data left on the system can't be accessed by other people's CI/CD jobs. + +NOTE: +The final disk space your jobs can use will be less than 25GB. Some disk space allocated to the instance will be occupied by the operating system, the Docker image, and a copy of your cloned repository. + +The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times. + +Jobs handled by shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`) +**time out after 3 hours**, regardless of the timeout configured in a +project. Check issue [#4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [#4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. + +Jobs handled by shared runners on Windows and macOS on GitLab.com **time out after 1 hour** while this service is in the Beta stage. + +Below are the runners' settings. + +| Setting | GitLab.com | Default | +| ----------- | ----------------- | ---------- | +| Executor | `docker+machine` | - | +| Default Docker image | `ruby:2.5` | - | +| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` | + +These runners share a [distributed cache](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) through use of a Google Cloud Storage (GCS) bucket. Cache contents not updated within the last 14 days are automatically removed through use of an [object lifecycle management policy](https://cloud.google.com/storage/docs/lifecycle). + +## Pre-clone script + +With SaaS runners on Linux, you can run commands in a CI +job before the runner attempts to run `git init` and `git fetch` to +download a GitLab repository. The +[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +can be used for: + +- Seeding the build directory with repository data +- Sending a request to a server +- Downloading assets from a CDN +- Any other commands that must run before the `git init` + +To use this feature, define a [CI/CD variable](../../../ci/variables/index.md#custom-cicd-variables) called +`CI_PRE_CLONE_SCRIPT` that contains a bash script. + +NOTE: +The `CI_PRE_CLONE_SCRIPT` variable does not work on Windows runners. + +### Pre-clone script example + +This example was used in the `gitlab-org/gitlab` project until November 2021. +The project no longer uses this optimization because the [pack-objects cache](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache) +lets Gitaly serve the full CI/CD fetch traffic. See [Git fetch caching](../../../development/pipelines.md#git-fetch-caching). + +The `CI_PRE_CLONE_SCRIPT` was defined as a project CI/CD variable: + +```shell +( + echo "Downloading archived master..." + wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz + + if [ ! -f /tmp/gitlab.tar.gz ]; then + echo "Repository cache not available, cloning a new directory..." + exit + fi + + rm -rf $CI_PROJECT_DIR + echo "Extracting tarball into $CI_PROJECT_DIR..." + mkdir -p $CI_PROJECT_DIR + cd $CI_PROJECT_DIR + tar xzf /tmp/gitlab.tar.gz + rm -f /tmp/gitlab.tar.gz + chmod a+w $CI_PROJECT_DIR +) +``` + +The first step of the script downloads `gitlab-master.tar.gz` from Google Cloud Storage. +There was a [GitLab CI/CD job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/5fb40526c8c8aaafc5f92eab36d5bbddaca3893d/.gitlab/ci/cache-repo.gitlab-ci.yml) +that was responsible for keeping that archive up-to-date. Every two hours on a scheduled pipeline, +it did the following: + +1. Create a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. +1. Save the data as a `.tar.gz`. +1. Upload it into the Google Cloud Storage bucket. + +When a job ran with this configuration, the output looked similar to: + +```shell +$ eval "$CI_PRE_CLONE_SCRIPT" +Downloading archived master... +Extracting tarball into /builds/gitlab-org/gitlab... +Fetching changes... +Reinitialized existing Git repository in /builds/gitlab-org/gitlab/.git/ +``` + +The `Reinitialized existing Git repository` message shows that +the pre-clone step worked. The runner runs `git init`, which +overwrites the Git configuration with the appropriate settings to fetch +from the GitLab repository. + +`CI_REPO_CACHE_CREDENTIALS` must contain the Google Cloud service account +JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. + +Note that this bucket should be located in the same continent as the +runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). + +## `config.toml` + +The full contents of our `config.toml` are: + +NOTE: +Settings that are not public are shown as `X`. + +**Google Cloud Platform** + +```toml +concurrent = X +check_interval = 1 +metrics_server = "X" +sentry_dsn = "X" + +[[runners]] + name = "docker-auto-scale" + request_concurrency = X + url = "https://gitlab.com/" + token = "SHARED_RUNNER_TOKEN" + pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\"" + executor = "docker+machine" + environment = [ + "DOCKER_DRIVER=overlay2", + "DOCKER_TLS_CERTDIR=" + ] + limit = X + [runners.docker] + image = "ruby:2.5" + privileged = true + volumes = [ + "/certs/client", + "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. + ] + [runners.machine] + IdleCount = 50 + IdleTime = 3600 + MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. + MachineName = "srm-%s" + MachineDriver = "google" + MachineOptions = [ + "google-project=PROJECT", + "google-disk-size=25", + "google-machine-type=n1-standard-1", + "google-username=core", + "google-tags=gitlab-com,srm", + "google-use-internal-ip", + "google-zone=us-east1-d", + "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928 + "google-machine-image=PROJECT/global/images/IMAGE", + "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. + "engine-opt=fixed-cidr-v6=fc00::/7", + "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 + ] + [[runners.machine.autoscaling]] + Periods = ["* * * * * sat,sun *"] + Timezone = "UTC" + IdleCount = 70 + IdleTime = 3600 + [[runners.machine.autoscaling]] + Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"] + Timezone = "UTC" + IdleCount = 700 + IdleTime = 3600 + [runners.cache] + Type = "gcs" + Shared = true + [runners.cache.gcs] + CredentialsFile = "/path/to/file" + BucketName = "bucket-name" +``` diff --git a/doc/ci/runners/saas/macos/environment.md b/doc/ci/runners/saas/macos/environment.md new file mode 100644 index 00000000000..3332eab9b44 --- /dev/null +++ b/doc/ci/runners/saas/macos/environment.md @@ -0,0 +1,43 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# VM instances and images for SaaS runners on macOS **(FREE SAAS)** + +When you use SaaS runners on macOS: + +- Each of your jobs runs in a newly provisioned VM, which is dedicated to the specific job. +- The VM is active only for the duration of the job and immediately deleted. + +## VM types + +The virtual machine where your job runs has `sudo` access with no password. +For the Beta, there is only one available machine type, `gbc-macos-large`. + +| Instance type | vCPUS | Memory (GB) | +| --------- | --- | ------- | +| `gbc-macos-large` | 4 | 10 | + +## VM images + +You can execute your build on one of the following images. +You specify this image in your `.gitlab-ci.yml` file. + +Each image is running a specific version of macOS and Xcode. + +| VM image | Included software | +|---------------------------|--------------------| +| macos-10.13-xcode-7 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | +| macos-10.13-xcode-8 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | +| macos-10.13-xcode-9 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | +| macos-10.14-xcode-10 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/mojave.yml> | +| macos-10.15-xcode-11 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/catalina.yml> | +| macos-11-xcode-12 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/big-sur.yml> | + +### Image update policy + +- Support for new macOS versions is planned. +- Additional details on the support policy and image update release process are documented + [in this project](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/55bf59c8fa88712960afff2bf6ecc5daa879a8f5/docs/overview.md#os-images). diff --git a/doc/ci/runners/saas/macos_saas_runner.md b/doc/ci/runners/saas/macos_saas_runner.md new file mode 100644 index 00000000000..40c4deb51aa --- /dev/null +++ b/doc/ci/runners/saas/macos_saas_runner.md @@ -0,0 +1,63 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# SaaS runners on macOS (Beta) **(FREE SAAS)** + +SaaS runners on macOS provide an on-demand macOS build environment integrated with +GitLab SaaS [CI/CD](../../../ci/index.md). +Use these runners to build, test, and deploy apps for the Apple ecosystem (macOS, iOS, tvOS). You can take advantage +of all the capabilities of the GitLab single DevOps platform and not have to manage or operate a +build environment. + +SaaS runners on macOS are in [Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +and shouldn't be relied upon for mission-critical production jobs. + +## Quickstart + +To start using SaaS runners on macOS, you must submit an access request [issue](https://gitlab.com/gitlab-com/macos-buildcloud-runners-beta/-/issues/new?issuable_template=beta_access_request). After your +access has been granted and your build environment configured, you must configure your +`.gitlab-ci.yml` pipeline file: + +1. Add a `.gitlab-ci.yml` file to your project repository. +1. Specify the [image](macos/environment.md#vm-images) you want to use. +1. Commit a change to your repository. + +The runners automatically run your build. + +## Example `.gitlab-ci.yml` file + +The following sample `.gitlab-ci.yml` file shows how to start using the SaaS runners on macOS: + +```yaml +.macos_saas_runners: + tags: + - shared-macos-amd64 + image: macos-11-xcode-12 + +stages: + - build + - test + +before_script: + - echo "started by ${GITLAB_USER_NAME}" + +build: + extends: + - .macos_saas_runners + stage: build + script: + - echo "running scripts in the build job" + +test: + extends: + - .macos_saas_runners + stage: test + script: + - echo "running scripts in the test job" +``` + +NOTE: +During the Beta period, the architecture of this solution will change. Rather than the jobs running on a specific VM instance, they will run on an ephemeral VM instance that is created by an autoscaling instance, known as the Runner Manager. We will notify all Beta participants of any downtime required to do this work. diff --git a/doc/ci/runners/saas/windows_saas_runner.md b/doc/ci/runners/saas/windows_saas_runner.md new file mode 100644 index 00000000000..87ee542fb14 --- /dev/null +++ b/doc/ci/runners/saas/windows_saas_runner.md @@ -0,0 +1,155 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# SaaS runners on Windows (beta) **(FREE SAAS)** + +SaaS runners on Windows are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +and shouldn't be used for production workloads. + +During this beta period, the [shared runner pipeline quota](../../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) +applies for groups and projects in the same manner as Linux runners. This may +change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834). + +Windows runners on GitLab.com autoscale by launching virtual machines on +the Google Cloud Platform. This solution uses an +[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md) +developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). +Windows runners execute your CI/CD jobs on `n1-standard-2` instances with +2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in +the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/main/cookbooks/preinstalled-software/README.md). + +We want to keep iterating to get Windows runners in a stable state and +[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). +You can follow our work towards this goal in the +[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162). + +## Configuration + +The full contents of our `config.toml` are: + +NOTE: +Settings that aren't public are shown as `X`. + +```toml +concurrent = X +check_interval = 3 + +[[runners]] + name = "windows-runner" + url = "https://gitlab.com/" + token = "TOKEN" + executor = "custom" + builds_dir = "C:\\GitLab-Runner\\builds" + cache_dir = "C:\\GitLab-Runner\\cache" + shell = "powershell" + [runners.custom] + config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"] + prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"] + run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"] + cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"] +``` + +The full contents of our `autoscaler/config.toml` are: + +```toml +Provider = "gcp" +Executor = "winrm" +OS = "windows" +LogLevel = "info" +LogFormat = "text" +LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log" +VMTag = "windows" + +[GCP] + ServiceAccountFile = "PATH" + Project = "some-project-df9323" + Zone = "us-east1-c" + MachineType = "n1-standard-2" + Image = "IMAGE" + DiskSize = 50 + DiskType = "pd-standard" + Subnetwork = "default" + Network = "default" + Tags = ["TAGS"] + Username = "gitlab_runner" + +[WinRM] + MaximumTimeout = 3600 + ExecutionMaxRetries = 0 + +[ProviderCache] + Enabled = true + Directory = "C:\\GitLab-Runner\\autoscaler\\machines" +``` + +## Example `.gitlab-ci.yml` file + +Below is a sample `.gitlab-ci.yml` file that shows how to start using the runners for Windows: + +```yaml +.shared_windows_runners: + tags: + - shared-windows + - windows + - windows-1809 + +stages: + - build + - test + +before_script: + - Set-Variable -Name "time" -Value (date -Format "%H:%m") + - echo ${time} + - echo "started by ${GITLAB_USER_NAME}" + +build: + extends: + - .shared_windows_runners + stage: build + script: + - echo "running scripts in the build job" + +test: + extends: + - .shared_windows_runners + stage: test + script: + - echo "running scripts in the test job" +``` + +## Limitations and known issues + +- All the limitations mentioned in our [beta + definition](https://about.gitlab.com/handbook/product/#beta). +- The average provisioning time for a new Windows VM is 5 minutes. + This means that you may notice slower build start times + on the Windows runner fleet during the beta. In a future + release we intend to update the autoscaler to enable + the pre-provisioning of virtual machines. This is intended to significantly reduce + the time it takes to provision a VM on the Windows fleet. You can + follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). +- The Windows runner fleet may be unavailable occasionally + for maintenance or updates. +- The Windows runner virtual machine instances do not use the + GitLab Docker executor. This means that you can't specify + [`image`](../../../ci/yaml/index.md#image) or [`services`](../../../ci/yaml/index.md#services) in + your pipeline configuration. +- For the beta release, we have included a set of software packages in + the base VM image. If your CI job requires additional software that's + not included in this list, then you must add installation + commands to [`before_script`](../../../ci/yaml/index.md#before_script) or [`script`](../../../ci/yaml/index.md#script) to install the required + software. Note that each job runs on a new VM instance, so the + installation of additional software packages needs to be repeated for + each job in your pipeline. +- The job may stay in a pending state for longer than the + Linux runners. +- There is the possibility that we introduce breaking changes which will + require updates to pipelines that are using the Windows runner + fleet. diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index 4d42bc69df8..c0a763c80f0 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -53,6 +53,7 @@ and supports multiple secrets engines. To configure your Vault server: +1. Ensure your Vault server is running on version 1.2.0 or higher. 1. Enable the authentication method by running these commands. They provide your Vault server the [JSON Web Key Set](https://tools.ietf.org/html/rfc7517) (JWKS) endpoint for your GitLab instance, so Vault can fetch the public signing key and verify the JSON Web Token (JWT) when authenticating: @@ -85,10 +86,10 @@ To configure your Vault server: to provide details about your Vault server: - `VAULT_SERVER_URL` - The URL of your Vault server, such as `https://vault.example.com:8200`. Required. - - `VAULT_AUTH_ROLE` - (Optional) The role to use when attempting to authenticate. + - `VAULT_AUTH_ROLE` - Optional. The role to use when attempting to authenticate. If no role is specified, Vault uses the [default role](https://www.vaultproject.io/api/auth/jwt#default_role) specified when the authentication method was configured. - - `VAULT_AUTH_PATH` - (Optional) The path where the authentication method is mounted, default is `jwt`. + - `VAULT_AUTH_PATH` - Optional. The path where the authentication method is mounted, default is `jwt`. NOTE: Support for providing these values in the user interface [is tracked in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/218677). diff --git a/doc/ci/services/gitlab.md b/doc/ci/services/gitlab.md index 5ac66846ab7..689ce884ae4 100644 --- a/doc/ci/services/gitlab.md +++ b/doc/ci/services/gitlab.md @@ -28,7 +28,7 @@ NOTE: Variables set in the GitLab UI are not passed down to the service containers. [Learn more](../variables/index.md#). -Then, commands in `script:` sections in your `.gitlab-ci.yml` file can access the API at `http://gitlab/api/v4`. +Then, commands in `script` sections in your `.gitlab-ci.yml` file can access the API at `http://gitlab/api/v4`. For more information about why `gitlab` is used for the `Host`, see [How services are linked to the job](../docker/using_docker_images.md#extended-docker-configuration-options). diff --git a/doc/ci/test_cases/index.md b/doc/ci/test_cases/index.md index 384bfc10779..4c840125d24 100644 --- a/doc/ci/test_cases/index.md +++ b/doc/ci/test_cases/index.md @@ -11,6 +11,10 @@ type: reference > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233479) in GitLab 13.6. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/241983) in GitLab 13.7. +INFO: +Create test cases in GitLab Ultimate. +[Try it free for 30 days](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=u-test-cases-docs). + Test cases in GitLab can help your teams create testing scenarios in their existing development platform. Now your Implementation and Testing teams can collaborate better, as they no longer have to diff --git a/doc/ci/triggers/img/triggers_page.png b/doc/ci/triggers/img/triggers_page.png Binary files differdeleted file mode 100644 index 7dc8f91cf7e..00000000000 --- a/doc/ci/triggers/img/triggers_page.png +++ /dev/null diff --git a/doc/ci/triggers/index.md b/doc/ci/triggers/index.md index afcf8ae629a..d3ac1de7c3b 100644 --- a/doc/ci/triggers/index.md +++ b/doc/ci/triggers/index.md @@ -5,131 +5,121 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: tutorial --- -# Triggering pipelines through the API **(FREE)** +# Trigger pipelines by using the API **(FREE)** -Triggers can be used to force a pipeline rerun of a specific `ref` (branch or -tag) with an API call. +To trigger a pipeline for a specific branch or tag, you can use an API call +to the [pipeline triggers API endpoint](../../api/pipeline_triggers.md). -## Authentication tokens +When authenticating with the API, you can use: -The following methods of authentication are supported: +- A [trigger token](#create-a-trigger-token) to trigger a branch or tag pipeline. +- A [CI/CD job token](../jobs/ci_job_token.md) to trigger a [multi-project pipeline](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api). -- Trigger tokens: A unique trigger token can be obtained when [adding a new trigger](#adding-a-new-trigger). -- [CI job tokens](../jobs/ci_job_token.md). +## Create a trigger token -If using the `$CI_PIPELINE_SOURCE` [predefined CI/CD variable](../variables/predefined_variables.md) -to limit which jobs run in a pipeline, the value could be either `pipeline` or `trigger`, -depending on which trigger method is used. +You can trigger a pipeline for a branch or tag by generating a trigger token and using it +to authenticate an API call. The token impersonates a user's project access and permissions. -| `$CI_PIPELINE_SOURCE` value | Trigger method | -|-----------------------------|----------------| -| `pipeline` | Using the `trigger:` keyword in the CI/CD configuration file, or using the trigger API with `$CI_JOB_TOKEN`. | -| `trigger` | Using the trigger API using a generated trigger token | +Prerequisite: -This also applies when using the `pipelines` or `triggers` keywords with the legacy [`only/except` basic syntax](../yaml/index.md#only--except). +- You must have at least the [Maintainer role](../../user/permissions.md) for the project. -## Adding a new trigger +To create a trigger token: -Go to your -**Settings > CI/CD** under **Triggers** to add a new trigger. The **Add trigger** button creates -a new token which you can then use to trigger a rerun of this -particular project's pipeline. - -Every new trigger you create, gets assigned a different token which you can -then use inside your scripts or `.gitlab-ci.yml`. You also have a nice -overview of the time the triggers were last used. - - +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Pipeline triggers**. +1. Enter a description and select **Add trigger**. + - You can view and copy the full token for all triggers you have created. + - You can only see the first 4 characters for tokens created by other project members. WARNING: -Passing plain text tokens in public projects is a security issue. Potential -attackers can impersonate the user that exposed their trigger token publicly in -their `.gitlab-ci.yml` file. Use [CI/CD variables](../variables/index.md) -to protect trigger tokens. +It is a security risk to save tokens in plain text in public projects. Potential +attackers could use a trigger token exposed in the `.gitlab-ci.yml` file to impersonate +the user that created the token. Use [masked CI/CD variables](../variables/index.md#mask-a-cicd-variable) +to improve the security of trigger tokens. -## Revoking a trigger +## Trigger a pipeline -You can revoke a trigger any time by going at your project's -**Settings > CI/CD** under **Triggers** and hitting the **Revoke** button. -The action is irreversible. +After you [create a trigger token](#create-a-trigger-token), you can use it to trigger +pipelines with a tool that can access the API, or a webhook. -## Triggering a pipeline +### Use cURL -To trigger a pipeline you need to send a `POST` request to the GitLab API endpoint: +You can use cURL to trigger pipelines with the [pipeline triggers API endpoint](../../api/pipeline_triggers.md). +For example: -```plaintext -POST /projects/:id/trigger/pipeline -``` +- Use a multiline cURL command: -The required parameters are the [trigger's `token`](#authentication-tokens) -and the Git `ref` on which the trigger is performed. Valid refs are -branches or tags. The `:id` of a project can be found by -[querying the API](../../api/projects.md) or by visiting the **CI/CD** -settings page which provides self-explanatory examples. + ```shell + curl --request POST \ + --form token=<token> \ + --formref=<ref_name> \ + "https://gitlab.example.com/api/v4/projects/<project_id>/trigger/pipeline" + ``` -When a rerun of a pipeline is triggered, jobs are labeled as `triggered` in -**CI/CD > Jobs**. +- Use cURL and pass the `<token>` and `<ref_name>` in the query string: -You can see which trigger caused a job to run by visiting the single job page. -A part of the trigger's token is exposed in the UI as you can see from the image -below. + ```shell + curl --request POST \ + "https://gitlab.example.com/api/v4/projects/<project_id>/trigger/pipeline?token=<token>&ref=<ref_name>" + ``` - - -By using cURL you can trigger a pipeline rerun with minimal effort, for example: - -```shell -curl --request POST \ - --form token=TOKEN \ - --form ref=main \ - "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" -``` +In each example, replace: -In this case, the pipeline for the project with ID `9` runs on the `main` branch. +- The URL with `https://gitlab.com` or the URL of your instance. +- `<token>` with your trigger token. +- `<ref_name>` with a branch or tag name, like `main`. +- `<project_id>` with your project ID, like `123456`. The project ID is displayed + at the top of every project's landing page. -Alternatively, you can pass the `token` and `ref` arguments in the query string: +### Use a CI/CD job -```shell -curl --request POST \ - "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline?token=TOKEN&ref=main" -``` +You can use a CI/CD job with a triggers token to trigger pipelines when another pipeline +runs. -You can also benefit by using triggers in your `.gitlab-ci.yml`. Let's say that -you have two projects, A and B, and you want to trigger a pipeline on the `main` -branch of project B whenever a tag on project A is created. This is the job you -need to add in project A's `.gitlab-ci.yml`: +For example, to trigger a pipeline on the `main` branch of `project-B` when a tag +is created in `project-A`, add the following job to project A's `.gitlab-ci.yml` file: ```yaml trigger_pipeline: stage: deploy script: - - 'curl --request POST --form token=TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline"' + - 'curl --fail --request POST --form token=$MY_TRIGGER_TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/123456/trigger/pipeline"' rules: - if: $CI_COMMIT_TAG ``` -This means that whenever a new tag is pushed on project A, the job runs and the -`trigger_pipeline` job is executed, triggering the pipeline for project B. The -`stage: deploy` ensures that this job runs only after all jobs with -`stage: test` complete successfully. +In this example: -NOTE: -You [cannot use the API to start `when:manual` trigger jobs](https://gitlab.com/gitlab-org/gitlab/-/issues/284086). +- `1234` is the project ID for `project-B`. The project ID is displayed at the top + of every project's landing page. +- The [`rules`](../yaml/index.md#rules) cause the job to run every time a tag is added to `project-A`. +- `MY_TRIGGER_TOKEN` is a [masked CI/CD variables](../variables/index.md#mask-a-cicd-variable) + that contains the trigger token. -## Triggering a pipeline from a webhook +### Use a webhook -To trigger a job from a webhook of another project you need to add the following -webhook URL for Push and Tag events (change the project ID, ref and token): +To trigger a pipeline from another project's webhook, use a webhook URL like the following +for push and tag events: ```plaintext https://gitlab.example.com/api/v4/projects/9/ref/main/trigger/pipeline?token=TOKEN ``` -You should pass `ref` as part of the URL, to take precedence over `ref` from -the webhook body that designates the branch ref that fired the trigger in the -source repository. Be sure to URL-encode `ref` if it contains slashes. +Replace: + +- The URL with `https://gitlab.com` or the URL of your instance. +- `<token>` with your trigger token. +- `<ref_name>` with a branch or tag name, like `main`. +- `<project_id>` with your project ID, like `123456`. The project ID is displayed + at the top of the project's landing page. + +The `ref` in the URL takes precedence over the `ref` in the webhook payload. The +payload `ref` is the branch that fired the trigger in the source repository. +You must URL-encode `ref` if it contains slashes. -### Using webhook payload in the triggered pipeline +#### Use a webhook payload > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31197) in GitLab 13.9. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321027) in GitLab 13.11. @@ -139,94 +129,68 @@ the `TRIGGER_PAYLOAD` [predefined CI/CD variable](../variables/predefined_variab The payload is exposed as a [file-type variable](../variables/index.md#cicd-variable-types), so you can access the data with `cat $TRIGGER_PAYLOAD` or a similar command. -## Making use of trigger variables +### Pass CI/CD variables in the API call -You can pass any number of arbitrary variables in the trigger API call and they -are available in GitLab CI/CD so that they can be used in your `.gitlab-ci.yml` -file. The parameter is of the form: +You can pass any number of [CI/CD variables](../variables/index.md) in the trigger API call. +These variables have the [highest precedence](../variables/index.md#cicd-variable-precedence), +and override all variables with the same name. -```plaintext -variables[key]=value +The parameter is of the form `variables[key]=value`, for example: + +```shell +curl --request POST \ + --form token=TOKEN \ + --form ref=main \ + --form "variables[UPLOAD_TO_S3]=true" \ + "https://gitlab.example.com/api/v4/projects/123456/trigger/pipeline" ``` -This information is also exposed in the UI. _Values_ are only viewable by users with the Owner and Maintainer role. +CI/CD variables in triggered pipelines display on each job's page, but only +users with the Owner and Maintainer role can view the values.  -Using trigger variables can be proven useful for a variety of reasons: - -- Identifiable jobs. Since the variable is exposed in the UI you can know - why the pipeline was triggered if you pass a variable that explains the - purpose. -- Conditional job processing. You can have conditional jobs that run whenever - a certain variable is present. +## Revoke a trigger token -Consider the following `.gitlab-ci.yml` where we set three -[stages](../yaml/index.md#stages) and the `upload_package` job is run only -when all jobs from the test and build stages pass. When the `UPLOAD_TO_S3` -variable is non-zero, `make upload` is run. +To revoke a trigger token: -```yaml -stages: - - test - - build - - package - -run_tests: - stage: test - script: - - make test - -build_package: - stage: build - script: - - make build - -upload_package: - stage: package - script: - - if [ -n "${UPLOAD_TO_S3}" ]; then make upload; fi -``` +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Pipeline triggers**. +1. To the left of the trigger token you want to revoke, select **Revoke** (**{remove}**). -You can then trigger a pipeline while you pass the `UPLOAD_TO_S3` variable -and the script of the `upload_package` job is run: +A revoked trigger token cannot be added back. -```shell -curl --request POST \ - --form token=TOKEN \ - --form ref=main \ - --form "variables[UPLOAD_TO_S3]=true" \ - "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" -``` +## Configure CI/CD jobs to run in triggered pipelines -Trigger variables have the [highest priority](../variables/index.md#cicd-variable-precedence) -of all types of variables. +To [configure when to run jobs](../jobs/job_control.md) in triggered pipelines: -## Using cron to trigger nightly pipelines +- Use [`rules`](../yaml/index.md#rules) with the `$CI_PIPELINE_SOURCE` [predefined CI/CD variable](../variables/predefined_variables.md). +- Use [`only`/`except`](../yaml/index.md#onlyrefs--exceptrefs) keywords. -Whether you craft a script or just run cURL directly, you can trigger jobs -in conjunction with cron. The example below triggers a job on the `main` branch -of project with ID `9` every night at `00:30`: +| `$CI_PIPELINE_SOURCE` value | `only`/`except` keywords | Trigger method | +|-----------------------------|--------------------------|---------------------| +| `trigger` | `triggers` | In pipelines triggered with the [pipeline triggers API](../../api/pipeline_triggers.md) by using a [trigger token](#create-a-trigger-token). | +| `pipeline` | `pipelines` | In [multi-project pipelines](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api) triggered with the [pipeline triggers API](../../api/pipeline_triggers.md) by using the [`$CI_JOB_TOKEN`](../jobs/ci_job_token.md), or by using the [`trigger`](../yaml/index.md#trigger) keyword in the CI/CD configuration file. | -```shell -30 0 * * * curl --request POST --form token=TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" -``` +Additionally, the `$CI_PIPELINE_TRIGGERED` predefined CI/CD variable is set to `true` +in pipelines triggered with a trigger token. -This behavior can also be achieved through the GitLab UI with -[pipeline schedules](../pipelines/schedules.md). +## See which trigger token was used -## Legacy triggers +You can see which trigger caused a job to run by visiting the single job page. +A part of the trigger's token displays on the right of the page, under the job details: -Old triggers, created before GitLab 9.0 are marked as legacy. + -Triggers with the legacy label do not have an associated user and only have -access to the current project. They are considered deprecated and might be -removed with one of the future versions of GitLab. +In pipelines triggered with a trigger token, jobs are labeled as `triggered` in +**CI/CD > Jobs**. ## Troubleshooting -### '404 not found' when triggering a pipeline +### `404 not found` when triggering a pipeline A response of `{"message":"404 Not Found"}` when triggering a pipeline might be caused -by using a Personal Access Token instead of a trigger token. [Add a new trigger](#adding-a-new-trigger) -and use that token to authenticate when triggering a pipeline. +by using a [personal access token](../../user/profile/personal_access_tokens.md) +instead of a trigger token. [Create a new trigger token](#create-a-trigger-token) +and use it instead of the personal access token. diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 037e8d3497d..4d550f6da13 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -291,7 +291,7 @@ Pipeline configuration warnings are shown when you: ### "Job may allow multiple pipelines to run for a single action" warning -When you use [`rules`](yaml/index.md#rules) with a `when:` clause without an `if:` +When you use [`rules`](yaml/index.md#rules) with a `when` clause without an `if` clause, multiple pipelines may run. Usually this occurs when you push a commit to a branch that has an open merge request associated with it. diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index e758fbc91dd..55fd8c1eb49 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -41,7 +41,7 @@ Consider the following workflow: ## How it works First, GitLab Runner uploads all [JUnit report format XML files](https://www.ibm.com/docs/en/adfz/developer-for-zos/14.1.0?topic=formats-junit-xml-format) -as [artifacts](yaml/index.md#artifactsreportsjunit) to GitLab. Then, when you visit a merge request, GitLab starts +as [artifacts](yaml/artifacts_reports.md#artifactsreportsjunit) to GitLab. Then, when you visit a merge request, GitLab starts comparing the head and base branch's JUnit report format XML files, where: - The base branch is the target branch (usually the default branch). @@ -77,7 +77,7 @@ If a test failed in the project's default branch in the last 14 days, a message ## How to set it up To enable the Unit test reports in merge requests, you need to add -[`artifacts:reports:junit`](yaml/index.md#artifactsreportsjunit) +[`artifacts:reports:junit`](yaml/artifacts_reports.md#artifactsreportsjunit) in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). @@ -377,7 +377,7 @@ GitLab does not parse very [large nodes](https://nokogiri.org/tutorials/parsing_ > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0 behind the `:junit_pipeline_screenshots_view` feature flag, disabled by default. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/216979) in GitLab 13.12. -Upload your screenshots as [artifacts](yaml/index.md#artifactsreportsjunit) to GitLab. If JUnit +Upload your screenshots as [artifacts](yaml/artifacts_reports.md#artifactsreportsjunit) to GitLab. If JUnit report format XML files contain an `attachment` tag, GitLab parses the attachment. Note that: - The `attachment` tag **must** contain the relative path to `$CI_PROJECT_DIR` of the screenshots you uploaded. For diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index a0c8dbd4e4a..acc3489143a 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -166,10 +166,10 @@ To add or update variables in the project settings: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Environment scope**: (Optional) `All`, or specific [environments](../environments/index.md). - - **Protect variable** (Optional): If selected, the variable is only available + - **Environment scope**: Optional. `All`, or specific [environments](../environments/index.md). + - **Protect variable** Optional. If selected, the variable is only available in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** is masked + - **Mask variable** Optional. If selected, the variable's **Value** is masked in job logs. The variable fails to save if the value does not meet the [masking requirements](#mask-a-cicd-variable). @@ -208,10 +208,10 @@ To add a group variable: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Environment scope** (Optional): `All`, or specific [environments](#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** - - **Protect variable** (Optional): If selected, the variable is only available + - **Environment scope** Optional. `All`, or specific [environments](#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** + - **Protect variable** Optional. If selected, the variable is only available in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** is masked + - **Mask variable** Optional. If selected, the variable's **Value** is masked in job logs. The variable fails to save if the value does not meet the [masking requirements](#mask-a-cicd-variable). @@ -248,9 +248,9 @@ To add an instance variable: 10,000 characters is allowed. This is also bounded by the limits of the selected runner operating system. In GitLab 13.0 to 13.2, 700 characters is allowed. - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Protect variable** (Optional): If selected, the variable is only available + - **Protect variable** Optional. If selected, the variable is only available in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** is not shown + - **Mask variable** Optional. If selected, the variable's **Value** is not shown in job logs. The variable is not saved if the value does not meet the [masking requirements](#mask-a-cicd-variable). ### CI/CD variable types @@ -293,6 +293,11 @@ Use the variables in a job script like this: kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KUBE_CA_PEM" ``` +WARNING: +Be careful when assigning the value of a file variable to another variable. The other +variable takes the content of the file as its value, **not** the path to the file. +See [issue 29407](https://gitlab.com/gitlab-org/gitlab/-/issues/29407) for more details. + An alternative to `File` type variables is to: - Read the value of a CI/CD variable (`variable` type). @@ -554,7 +559,7 @@ These variables cannot be used as CI/CD variables to configure a pipeline, but they can be used in job scripts. 1. In the job script, save the variable as a `.env` file. -1. Save the `.env` file as an [`artifacts:reports:dotenv`](../yaml/index.md#artifactsreportsdotenv) +1. Save the `.env` file as an [`artifacts:reports:dotenv`](../yaml/artifacts_reports.md#artifactsreportsdotenv) artifact. 1. Set a job in a later stage to receive the artifact by using the [`dependencies`](../yaml/index.md#dependencies) or the [`needs`](../yaml/index.md#needs) keywords. @@ -607,7 +612,7 @@ which variables take precedence. The order of precedence for variables is (from highest to lowest): -1. [Trigger variables](../triggers/index.md#making-use-of-trigger-variables), +1. [Trigger variables](../triggers/index.md#pass-cicd-variables-in-the-api-call), [scheduled pipeline variables](../pipelines/schedules.md#using-variables), and [manual pipeline run variables](#override-a-variable-when-running-a-pipeline-manually). 1. Project [variables](#custom-cicd-variables). @@ -641,7 +646,7 @@ You can override the value of a variable when you: 1. Create a pipeline by using [the API](../../api/pipelines.md#create-a-new-pipeline). 1. Run a job manually in the UI. 1. Use [push options](../../user/project/push_options.md#push-options-for-gitlab-cicd). -1. Trigger a pipeline by using [the API](../triggers/index.md#making-use-of-trigger-variables). +1. Trigger a pipeline by using [the API](../triggers/index.md#pass-cicd-variables-in-the-api-call). 1. Pass variables to a downstream pipeline [by using the `variable` keyword](../pipelines/multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-the-variables-keyword) or [by using variable inheritance](../pipelines/multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-variable-inheritance). diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 45fa1994342..c06ca6878c4 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -14,7 +14,9 @@ Some variables are only available with more recent versions of [GitLab Runner](h You can [output the values of all variables available for a job](index.md#list-all-environment-variables) with a `script` command. -There are also [Kubernetes-specific deployment variables](../../user/project/clusters/deploy_to_cluster.md#deployment-variables). +There are also [Kubernetes-specific deployment variables (deprecated)](../../user/project/clusters/deploy_to_cluster.md#deployment-variables). + +There are also a number of [variables you can use to configure runner behavior](../runners/configure_runners.md#configure-runner-behavior-with-variables) globally or for individual jobs. | Variable | GitLab | Runner | Description | |------------------------------------------|--------|--------|-------------| @@ -75,7 +77,7 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `CI_PAGES_URL` | 11.8 | all | The URL for a GitLab Pages site. Always a subdomain of `CI_PAGES_DOMAIN`. | | `CI_PIPELINE_ID` | 8.10 | all | The instance-level ID of the current pipeline. This ID is unique across all projects on the GitLab instance. | | `CI_PIPELINE_IID` | 11.0 | all | The project-level IID (internal ID) of the current pipeline. This ID is unique only within the current project. | -| `CI_PIPELINE_SOURCE` | 10.0 | all | How the pipeline was triggered. Can be `push`, `web`, `schedule`, `api`, `external`, `chat`, `webide`, `merge_request_event`, `external_pull_request_event`, `parent_pipeline`, [`trigger`, or `pipeline`](../triggers/index.md#authentication-tokens). | +| `CI_PIPELINE_SOURCE` | 10.0 | all | How the pipeline was triggered. Can be `push`, `web`, `schedule`, `api`, `external`, `chat`, `webide`, `merge_request_event`, `external_pull_request_event`, `parent_pipeline`, [`trigger`, or `pipeline`](../triggers/index.md#configure-cicd-jobs-to-run-in-triggered-pipelines). | | `CI_PIPELINE_TRIGGERED` | all | all | `true` if the job was [triggered](../triggers/index.md). | | `CI_PIPELINE_URL` | 11.1 | 0.5 | The URL for the pipeline details. | | `CI_PIPELINE_CREATED_AT` | 13.10 | all | The UTC datetime when the pipeline was created, in [ISO 8601](https://tools.ietf.org/html/rfc3339#appendix-A) format. | @@ -122,7 +124,7 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `GITLAB_USER_ID` | 8.12 | all | The ID of the user who started the job. | | `GITLAB_USER_LOGIN` | 10.0 | all | The username of the user who started the job. | | `GITLAB_USER_NAME` | 10.0 | all | The name of the user who started the job. | -| `TRIGGER_PAYLOAD` | 13.9 | all | The webhook payload. Only available when a pipeline is [triggered with a webhook](../triggers/index.md#using-webhook-payload-in-the-triggered-pipeline). | +| `TRIGGER_PAYLOAD` | 13.9 | all | The webhook payload. Only available when a pipeline is [triggered with a webhook](../triggers/index.md#use-a-webhook-payload). | ## Predefined variables for merge request pipelines diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md new file mode 100644 index 00000000000..cd38cf58c71 --- /dev/null +++ b/doc/ci/yaml/artifacts_reports.md @@ -0,0 +1,304 @@ +--- +stage: Verify +group: Testing +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GitLab CI/CD artifacts reports types **(FREE)** + +Use [`artifacts:reports`](index.md#artifactsreports) to: + +- Collect test reports, code quality reports, security reports, and other artifacts generated by included templates in + jobs. +- Some of these reports are used to display information in: + - Merge requests. + - Pipeline views. + - [Security dashboards](../../user/application_security/security_dashboard/index.md). + +The test reports are collected regardless of the job results (success or failure). +You can use [`artifacts:expire_in`](index.md#artifactsexpire_in) to set up an expiration +date for their artifacts. + +Some `artifacts:reports` types can be generated by multiple jobs in the same pipeline, and used by merge request or +pipeline features from each job. + +To be able to browse the report output files, include the [`artifacts:paths`](index.md#artifactspaths) keyword. + +NOTE: +Combined reports in parent pipelines using [artifacts from child pipelines](index.md#needspipelinejob) is +not supported. Track progress on adding support in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215725). + +## `artifacts:reports:accessibility` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39425) in GitLab 12.8. + +The `accessibility` report uses [pa11y](https://pa11y.org/) to report on the accessibility impact +of changes introduced in merge requests. + +GitLab can display the results of one or more reports in the merge request +[accessibility widget](../../user/project/merge_requests/accessibility_testing.md#accessibility-merge-request-widget). + +For more information, see [Accessibility testing](../../user/project/merge_requests/accessibility_testing.md). + +## `artifacts:reports:api_fuzzing` **(ULTIMATE)** + +> - Introduced in GitLab 13.4. +> - Requires GitLab Runner 13.4 or later. + +The `api_fuzzing` report collects [API Fuzzing bugs](../../user/application_security/api_fuzzing/index.md) +as artifacts. + +GitLab can display the results of one or more reports in: + +- The merge request [security widget](../../user/application_security/api_fuzzing/index.md#view-details-of-an-api-fuzzing-vulnerability). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [security dashboard](../../user/application_security/api_fuzzing/index.md#security-dashboard). + +## `artifacts:reports:browser_performance` **(PREMIUM)** + +> [Name changed](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) from `artifacts:reports:performance` in GitLab 14.0. + +The `browser_performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) +as artifacts. + +GitLab can display the results of one report in the merge request +[browser performance testing widget](../../user/project/merge_requests/browser_performance_testing.md#how-browser-performance-testing-works). + +GitLab cannot display the combined results of multiple `browser_performance` reports. + +## `artifacts:reports:cluster_image_scanning` **(ULTIMATE)** + +> - Introduced in GitLab 14.1. +> - Requires GitLab Runner 14.1 and above. + +The `cluster_image_scanning` report collects `CLUSTER_IMAGE_SCANNING` vulnerabilities. The collected +`CLUSTER_IMAGE_SCANNING` report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). + +## `artifacts:reports:cobertura` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. + +The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). +The collected Cobertura coverage reports upload to GitLab as an artifact. + +GitLab can display the results of one or more reports in the merge request +[diff annotations](../../user/project/merge_requests/test_coverage_visualization.md). + +Cobertura was originally developed for Java, but there are many third-party ports for other languages such as +JavaScript, Python, and Ruby. + +## `artifacts:reports:codequality` + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. + +The `codequality` report collects [code quality issues](../../user/project/merge_requests/code_quality.md). The +collected code quality report uploads to GitLab as an artifact. + +GitLab can display the results of: + +- One or more reports in the merge request [code quality widget](../../user/project/merge_requests/code_quality.md#code-quality-widget). +- Only one report in: + - The merge request [diff annotations](../../user/project/merge_requests/code_quality.md#code-quality-in-diff-view). + Track progress on adding support for multiple reports in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328257). + - The [full report](../metrics_reports.md). Track progress on adding support for multiple reports in + [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/9014). + +## `artifacts:reports:container_scanning` **(ULTIMATE)** + +The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md). +The collected Container Scanning report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The merge request [container scanning widget](../../user/application_security/container_scanning/index.md). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). + +## `artifacts:reports:coverage_fuzzing` **(ULTIMATE)** + +> - Introduced in GitLab 13.4. +> - Requires GitLab Runner 13.4 or later. + +The `coverage_fuzzing` report collects [coverage fuzzing bugs](../../user/application_security/coverage_fuzzing/index.md). +The collected coverage fuzzing report uploads to GitLab as an artifact. +GitLab can display the results of one or more reports in: + +- The merge request [coverage fuzzing widget](../../user/application_security/coverage_fuzzing/index.md#interacting-with-the-vulnerabilities). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). + +## `artifacts:reports:dast` **(ULTIMATE)** + +The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md). The collected DAST +report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The merge request [security widget](../../user/application_security/dast/index.md#view-details-of-a-vulnerability-detected-by-dast). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). + +## `artifacts:reports:dependency_scanning` **(ULTIMATE)** + +The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md). +The collected Dependency Scanning report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The merge request [dependency scanning widget](../../user/application_security/dependency_scanning/index.md#overview). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +- The [dependency list](../../user/application_security/dependency_list/). + +## `artifacts:reports:dotenv` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17066) in GitLab 12.9. + +The `dotenv` report collects a set of environment variables as artifacts. + +The collected variables are registered as runtime-created variables of the job, +which you can use to [set dynamic environment URLs after a job finishes](../environments/index.md#set-dynamic-environment-urls-after-a-job-finishes). + +If duplicate environment variables are present in a `dotenv` report: + +- In GitLab 14.6 and later, the last one specified is used. +- In GitLab 14.5 and earlier, an error occurs. + +The exceptions to the [original dotenv rules](https://github.com/motdotla/dotenv#rules) are: + +- The variable key can contain only letters, digits, and underscores (`_`). +- The maximum size of the `.env` file is 5 KB. + This limit [can be changed on self-managed instances](../../administration/instance_limits.md#limit-dotenv-file-size). +- On GitLab.com, [the maximum number of inherited variables](../../user/gitlab_com/index.md#gitlab-cicd) + is 50 for Free, 100 for Premium and 150 for Ultimate. The default for + self-managed instances is 150, and can be changed by changing the + `dotenv_variables` [application limit](../../administration/instance_limits.md#limit-dotenv-variables). +- Variable substitution in the `.env` file is not supported. +- The `.env` file can't have empty lines or comments (starting with `#`). +- Key values in the `env` file cannot have spaces or newline characters (`\n`), including when using single or double quotes. +- Quote escaping during parsing (`key = 'value'` -> `{key: "value"}`) is not supported. + +## `artifacts:reports:junit` + +The `junit` report collects [JUnit report format XML files](https://www.ibm.com/docs/en/adfz/developer-for-zos/14.1.0?topic=formats-junit-xml-format). +The collected Unit test reports upload to GitLab as an artifact. Although JUnit was originally developed in Java, there +are many third-party ports for other languages such as JavaScript, Python, and Ruby. + +See [Unit test reports](../unit_test_reports.md) for more details and examples. +Below is an example of collecting a JUnit report format XML file from Ruby's RSpec test tool: + +```yaml +rspec: + stage: test + script: + - bundle install + - rspec --format RspecJunitFormatter --out rspec.xml + artifacts: + reports: + junit: rspec.xml +``` + +GitLab can display the results of one or more reports in: + +- The merge request [code quality widget](../../ci/unit_test_reports.md#how-it-works). +- The [full report](../../ci/unit_test_reports.md#viewing-unit-test-reports-on-gitlab). + +Some JUnit tools export to multiple XML files. You can specify multiple test report paths in a single job to +concatenate them into a single file. Use either: + +- A filename pattern (`junit: rspec-*.xml`). +- an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`). +- A Combination of both (`junit: [rspec.xml, test-results/TEST-*.xml]`). + +## `artifacts:reports:license_scanning` **(ULTIMATE)** + +> Introduced in GitLab 12.8. + +The License Compliance report collects [Licenses](../../user/compliance/license_compliance/index.md). The License +Compliance report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The merge request [license compliance widget](../../user/compliance/license_compliance/index.md). +- The [license list](../../user/compliance/license_compliance/index.md#license-list). + +## `artifacts:reports:load_performance` **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35260) in GitLab 13.2. +> - Requires GitLab Runner 11.5 and above. + +The `load_performance` report collects [Load Performance Testing metrics](../../user/project/merge_requests/load_performance_testing.md). +The report is uploaded to GitLab as an artifact. + +GitLab can display the results of only one report in the merge request +[load testing widget](../../user/project/merge_requests/load_performance_testing.md#how-load-performance-testing-works). + +GitLab cannot display the combined results of multiple `load_performance` reports. + +## `artifacts:reports:metrics` **(PREMIUM)** + +The `metrics` report collects [Metrics](../metrics_reports.md). The collected Metrics report uploads to GitLab as an +artifact. + +GitLab can display the results of one or more reports in the merge request +[metrics reports widget](../../ci/metrics_reports.md#metrics-reports). + +## `artifacts:reports:requirements` **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. + +The `requirements` report collects `requirements.json` files. The collected Requirements report uploads to GitLab as an +artifact and existing [requirements](../../user/project/requirements/index.md) are marked as Satisfied. + +GitLab can display the results of one or more reports in the +[project requirements](../../user/project/requirements/index.md#view-a-requirement). + +## `artifacts:reports:sast` + +> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) from GitLab Ultimate to GitLab Free in 13.3. + +The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md). The collected SAST +report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The merge request [SAST widget](../../user/application_security/sast/index.md#static-application-security-testing-sast). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). + +## `artifacts:reports:secret_detection` + +> - Introduced in GitLab 13.1. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/222788) to GitLab Free in 13.3. +> - Requires GitLab Runner 11.5 and above. + +The `secret-detection` report collects [detected secrets](../../user/application_security/secret_detection/index.md). +The collected Secret Detection report is uploaded to GitLab. + +GitLab can display the results of one or more reports in: + +- The merge request [secret scanning widget](../../user/application_security/secret_detection/index.md). +- The [pipeline **Security** tab](../../user/application_security/index.md#view-security-scan-information-in-the-pipeline-security-tab). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). + +## `artifacts:reports:terraform` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. +> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. + +The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/iac/mr_integration.md#configure-terraform-report-artifacts). +The collected Terraform plan report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in the merge request +[terraform widget](../../user/infrastructure/iac/mr_integration.md#output-terraform-plan-information-into-a-merge-request). + +For more information, see [Output `terraform plan` information into a merge request](../../user/infrastructure/iac/mr_integration.md). diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index 5e2eb53a0ea..3c94ddb3c14 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -238,7 +238,7 @@ In `include` sections in your `.gitlab-ci.yml` file, you can use: In GitLab 14.5 and later, you can also use: -- [Trigger variables](../triggers/index.md#making-use-of-trigger-variables). +- [Trigger variables](../triggers/index.md#pass-cicd-variables-in-the-api-call). - [Scheduled pipeline variables](../pipelines/schedules.md#using-variables). - [Manual pipeline run variables](../variables/index.md#override-a-variable-when-running-a-pipeline-manually). - Pipeline [predefined variables](../variables/predefined_variables.md). diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index 5702c7a7dfd..ed05ef08d02 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -16,6 +16,8 @@ This document lists the configuration options for your GitLab `.gitlab-ci.yml` f When you are editing your `.gitlab-ci.yml` file, you can validate it with the [CI Lint](../lint.md) tool. +If you are editing this page, make sure you follow the [CI/CD YAML reference style guide](../../development/cicd/cicd_reference_documentation_guide.md). + ## Keywords A GitLab CI/CD pipeline configuration includes: @@ -25,9 +27,9 @@ A GitLab CI/CD pipeline configuration includes: | Keyword | Description | |-------------------------|:------------| | [`default`](#default) | Custom default values for job keywords. | + | [`include`](#include) | Import configuration from other YAML files. | | [`stages`](#stages) | The names and order of the pipeline stages. | | [`workflow`](#workflow) | Control what types of pipeline run. | - | [`include`](#include) | Import configuration from other YAML files. | - [Jobs](../jobs/index.md) configured with [job keywords](#job-keywords): @@ -73,7 +75,7 @@ or import additional pipeline configuration. ### `default` You can set global defaults for some keywords. Jobs that do not define one or more -of the listed keywords use the value defined in the `default:` section. +of the listed keywords use the value defined in the `default` section. **Keyword type**: Global keyword. @@ -90,7 +92,7 @@ of the listed keywords use the value defined in the `default:` section. - [`tags`](#tags) - [`timeout`](#timeout) -**Example of `default`:** +**Example of `default`**: ```yaml default: @@ -106,7 +108,7 @@ rspec 2.7: In this example, `ruby:3.0` is the default `image` value for all jobs in the pipeline. The `rspec 2.7` job does not use the default, because it overrides the default with -a job-specific `image:` section: +a job-specific `image` section: **Additional details**: @@ -116,179 +118,6 @@ a job-specific `image:` section: takes precedence and is not replaced by the default. - Control inheritance of default keywords in jobs with [`inherit:default`](#inheritdefault). -### `stages` - -Use `stages` to define stages that contain groups of jobs. Use [`stage`](#stage) -in a job to configure the job to run in a specific stage. - -If `stages` is not defined in the `.gitlab-ci.yml` file, the default pipeline stages are: - -- [`.pre`](#stage-pre) -- `build` -- `test` -- `deploy` -- [`.post`](#stage-post) - -The order of the items in `stages` defines the execution order for jobs: - -- Jobs in the same stage run in parallel. -- Jobs in the next stage run after the jobs from the previous stage complete successfully. - -**Keyword type**: Global keyword. - -**Example of `stages`:** - -```yaml -stages: - - build - - test - - deploy -``` - -In this example: - -1. All jobs in `build` execute in parallel. -1. If all jobs in `build` succeed, the `test` jobs execute in parallel. -1. If all jobs in `test` succeed, the `deploy` jobs execute in parallel. -1. If all jobs in `deploy` succeed, the pipeline is marked as `passed`. - -If any job fails, the pipeline is marked as `failed` and jobs in later stages do not -start. Jobs in the current stage are not stopped and continue to run. - -**Additional details**: - -- If a job does not specify a [`stage`](#stage), the job is assigned the `test` stage. -- If a stage is defined but no jobs use it, the stage is not visible in the pipeline, - which can help [compliance pipeline configurations](../../user/project/settings/index.md#compliance-pipeline-configuration): - - Stages can be defined in the compliance configuration but remain hidden if not used. - - The defined stages become visible when developers use them in job definitions. - -**Related topics**: - -- To make a job start earlier and ignore the stage order, use the [`needs`](#needs) keyword. - -### `workflow` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29654) in GitLab 12.5 - -Use [`workflow`](workflow.md) to control pipeline behavior. - -**Related topics**: - -- [`workflow: rules` examples](workflow.md#workflow-rules-examples) -- [Switch between branch pipelines and merge request pipelines](workflow.md#switch-between-branch-pipelines-and-merge-request-pipelines) - -#### `workflow:rules` - -The `rules` keyword in `workflow` is similar to [`rules:` defined in jobs](#rules), -but controls whether or not a whole pipeline is created. - -When no rules evaluate to true, the pipeline does not run. - -**Possible inputs**: You can use some of the same keywords as job-level [`rules`](#rules): - -- [`rules: if`](#rulesif). -- [`rules: changes`](#ruleschanges). -- [`rules: exists`](#rulesexists). -- [`when`](#when), can only be `always` or `never` when used with `workflow`. -- [`variables`](#workflowrulesvariables). - -**Example of `workflow:rules`:** - -```yaml -workflow: - rules: - - if: $CI_COMMIT_MESSAGE =~ /-draft$/ - when: never - - if: $CI_PIPELINE_SOURCE == "merge_request_event" - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH -``` - -In this example, pipelines run if the commit message does not have `-drafts` in it -and the pipeline is for either: - -- A merge request -- The default branch. - -**Additional details**: - -- If your rules match both branch pipelines (other than the default branch) and merge request pipelines, - [duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) can occur. - -**Related topics**: - -- You can use the [`workflow:rules` templates](workflow.md#workflowrules-templates) to import - a preconfigured `workflow: rules` entry. -- [Common `if` clauses for `workflow:rules`](workflow.md#common-if-clauses-for-workflowrules). - -#### `workflow:rules:variables` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294232) in GitLab 13.11. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/300997) in GitLab 14.1. - -You can use [`variables`](#variables) in `workflow:rules:` to define variables for -specific pipeline conditions. - -When the condition matches, the variable is created and can be used by all jobs -in the pipeline. If the variable is already defined at the global level, the `workflow` -variable takes precedence and overrides the global variable. - -**Keyword type**: Global keyword. - -**Possible inputs**: Variable name and value pairs: - -- The name can use only numbers, letters, and underscores (`_`). -- The value must be a string. - -**Example of `workflow:rules:variables`:** - -```yaml -variables: - DEPLOY_VARIABLE: "default-deploy" - -workflow: - rules: - - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH - variables: - DEPLOY_VARIABLE: "deploy-production" # Override globally-defined DEPLOY_VARIABLE - - if: $CI_COMMIT_REF_NAME =~ /feature/ - variables: - IS_A_FEATURE: "true" # Define a new variable. - - when: always # Run the pipeline in other cases - -job1: - variables: - DEPLOY_VARIABLE: "job1-default-deploy" - rules: - - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH - variables: # Override DEPLOY_VARIABLE defined - DEPLOY_VARIABLE: "job1-deploy-production" # at the job level. - - when: on_success # Run the job in other cases - script: - - echo "Run script with $DEPLOY_VARIABLE as an argument" - - echo "Run another script if $IS_A_FEATURE exists" - -job2: - script: - - echo "Run script with $DEPLOY_VARIABLE as an argument" - - echo "Run another script if $IS_A_FEATURE exists" -``` - -When the branch is the default branch: - -- job1's `DEPLOY_VARIABLE` is `job1-deploy-production`. -- job2's `DEPLOY_VARIABLE` is `deploy-production`. - -When the branch is `feature`: - -- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`, and `IS_A_FEATURE` is `true`. -- job2's `DEPLOY_VARIABLE` is `default-deploy`, and `IS_A_FEATURE` is `true`. - -When the branch is something else: - -- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`. -- job2's `DEPLOY_VARIABLE` is `default-deploy`. - ### `include` > [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/42861) to GitLab Free in 11.4. @@ -372,8 +201,10 @@ use `include:file`. You can use `include:file` in combination with `include:proj **Keyword type**: Global keyword. -**Possible inputs**: A full path, relative to the root directory (`/`). -The YAML file must have the extension `.yml` or `.yaml`. +**Possible inputs**: + +- A full path, relative to the root directory (`/`). The YAML file must have the + extension `.yml` or `.yaml`. **Example of `include:file`**: @@ -428,10 +259,10 @@ Use `include:remote` with a full URL to include a file from a different location **Keyword type**: Global keyword. -**Possible inputs**: A public URL accessible by an HTTP/HTTPS `GET` request. -Authentication with the remote URL is not supported. +**Possible inputs**: -The YAML file must have the extension `.yml` or `.yaml`. +- A public URL accessible by an HTTP/HTTPS `GET` request. Authentication with the + remote URL is not supported. The YAML file must have the extension `.yml` or `.yaml`. **Example of `include:remote`**: @@ -454,7 +285,9 @@ Use `include:template` to include [`.gitlab-ci.yml` templates](https://gitlab.co **Keyword type**: Global keyword. -**Possible inputs**: [`.gitlab-ci.yml` templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). +**Possible inputs**: + +- [`.gitlab-ci.yml` templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). **Example of `include:template`**: @@ -477,224 +310,189 @@ include: - All [nested includes](includes.md#use-nested-includes) are executed only with the permission of the user, so it's possible to use `project`, `remote`, or `template` includes. -## Job keywords +### `stages` -The following topics explain how to use keywords to configure CI/CD pipelines. +Use `stages` to define stages that contain groups of jobs. Use [`stage`](#stage) +in a job to configure the job to run in a specific stage. -### `image` +If `stages` is not defined in the `.gitlab-ci.yml` file, the default pipeline stages are: -Use `image` to specify a Docker image that the job runs in. +- [`.pre`](#stage-pre) +- `build` +- `test` +- `deploy` +- [`.post`](#stage-post) -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +The order of the items in `stages` defines the execution order for jobs: -**Possible inputs**: The name of the image, including the registry path if needed, in one of these formats: +- Jobs in the same stage run in parallel. +- Jobs in the next stage run after the jobs from the previous stage complete successfully. -- `<image-name>` (Same as using `<image-name>` with the `latest` tag) -- `<image-name>:<tag>` -- `<image-name>@<digest>` +**Keyword type**: Global keyword. -**Example of `image`**: +**Example of `stages`**: ```yaml -default: - image: ruby:3.0 - -rspec: - script: bundle exec rspec - -rspec 2.7: - image: registry.example.com/my-group/my-project/ruby:2.7 - script: bundle exec rspec +stages: + - build + - test + - deploy ``` -In this example, the `ruby:3.0` image is the default for all jobs in the pipeline. -The `rspec 2.7` job does not use the default, because it overrides the default with -a job-specific `image:` section. +In this example: -**Related topics**: +1. All jobs in `build` execute in parallel. +1. If all jobs in `build` succeed, the `test` jobs execute in parallel. +1. If all jobs in `test` succeed, the `deploy` jobs execute in parallel. +1. If all jobs in `deploy` succeed, the pipeline is marked as `passed`. -- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). +If any job fails, the pipeline is marked as `failed` and jobs in later stages do not +start. Jobs in the current stage are not stopped and continue to run. -#### `image:name` +**Additional details**: -The name of the Docker image that the job runs in. Similar to [`image:`](#image) used by itself. +- If a job does not specify a [`stage`](#stage), the job is assigned the `test` stage. +- If a stage is defined but no jobs use it, the stage is not visible in the pipeline, + which can help [compliance pipeline configurations](../../user/project/settings/index.md#compliance-pipeline-configuration): + - Stages can be defined in the compliance configuration but remain hidden if not used. + - The defined stages become visible when developers use them in job definitions. -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +**Related topics**: -**Possible inputs**: The name of the image, including the registry path if needed, in one of these formats: +- To make a job start earlier and ignore the stage order, use the [`needs`](#needs) keyword. -- `<image-name>` (Same as using `<image-name>` with the `latest` tag) -- `<image-name>:<tag>` -- `<image-name>@<digest>` +### `workflow` -**Example of `image:name`**: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29654) in GitLab 12.5 -```yaml -image: - name: "registry.example.com/my/image:latest" -``` +Use [`workflow`](workflow.md) to control pipeline behavior. **Related topics**: -- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). +- [`workflow: rules` examples](workflow.md#workflow-rules-examples) +- [Switch between branch pipelines and merge request pipelines](workflow.md#switch-between-branch-pipelines-and-merge-request-pipelines) -#### `image:entrypoint` +#### `workflow:rules` -Command or script to execute as the container's entry point. +The `rules` keyword in `workflow` is similar to [`rules` defined in jobs](#rules), +but controls whether or not a whole pipeline is created. -When the Docker container is created, the `entrypoint` is translated to the Docker `--entrypoint` option. -The syntax is similar to the [Dockerfile `ENTRYPOINT` directive](https://docs.docker.com/engine/reference/builder/#entrypoint), -where each shell token is a separate string in the array. +When no rules evaluate to true, the pipeline does not run. -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +**Possible inputs**: You can use some of the same keywords as job-level [`rules`](#rules): -**Possible inputs**: A string. +- [`rules: if`](#rulesif). +- [`rules: changes`](#ruleschanges). +- [`rules: exists`](#rulesexists). +- [`when`](#when), can only be `always` or `never` when used with `workflow`. +- [`variables`](#workflowrulesvariables). -**Example of `image:entrypoint`**: +**Example of `workflow:rules`**: ```yaml -image: - name: super/sql:experimental - entrypoint: [""] +workflow: + rules: + - if: $CI_COMMIT_MESSAGE =~ /-draft$/ + when: never + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` -**Related topics**: - -- [Override the entrypoint of an image](../docker/using_docker_images.md#override-the-entrypoint-of-an-image). - -#### `services` - -Use `services` to specify an additional Docker image to run scripts in. The [`services` image](../services/index.md) is linked -to the image specified in the [`image`](#image) keyword. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: The name of the services image, including the registry path if needed, in one of these formats: - -- `<image-name>` (Same as using `<image-name>` with the `latest` tag) -- `<image-name>:<tag>` -- `<image-name>@<digest>` +In this example, pipelines run if the commit message does not have `-drafts` in it +and the pipeline is for either: -**Example of `services`**: +- A merge request +- The default branch. -```yaml -default: - image: - name: ruby:2.6 - entrypoint: ["/bin/bash"] +**Additional details**: - services: - - name: my-postgres:11.7 - alias: db-postgres - entrypoint: ["/usr/local/bin/db-postgres"] - command: ["start"] +- If your rules match both branch pipelines (other than the default branch) and merge request pipelines, + [duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) can occur. - before_script: - - bundle install +**Related topics**: -test: - script: - - bundle exec rake spec -``` +- You can use the [`workflow:rules` templates](workflow.md#workflowrules-templates) to import + a preconfigured `workflow: rules` entry. +- [Common `if` clauses for `workflow:rules`](workflow.md#common-if-clauses-for-workflowrules). -In this example, the job launches a Ruby container. Then, from that container, the job launches -another container that's running PostgreSQL. Then the job then runs scripts -in that container. +#### `workflow:rules:variables` -**Related topics**: +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294232) in GitLab 13.11. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/300997) in GitLab 14.1. -- [Available settings for `services`](../services/index.md#available-settings-for-services). -- [Define `services` in the `.gitlab-ci.yml` file](../services/index.md#define-services-in-the-gitlab-ciyml-file). -- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). -- [Use Docker to build Docker images](../docker/using_docker_build.md). +You can use [`variables`](#variables) in `workflow:rules` to define variables for +specific pipeline conditions. -### `script` +When the condition matches, the variable is created and can be used by all jobs +in the pipeline. If the variable is already defined at the global level, the `workflow` +variable takes precedence and overrides the global variable. -Use `script` to specify commands for the runner to execute. +**Keyword type**: Global keyword. -All jobs except [trigger jobs](#trigger) require a `script` keyword. +**Possible inputs**: Variable name and value pairs: -**Keyword type**: Job keyword. You can use it only as part of a job. +- The name can use only numbers, letters, and underscores (`_`). +- The value must be a string. -**Possible inputs**: An array including: +**Example of `workflow:rules:variables`**: -- Single line commands. -- Long commands [split over multiple lines](script.md#split-long-commands). -- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). +```yaml +variables: + DEPLOY_VARIABLE: "default-deploy" -**Example of `script`:** +workflow: + rules: + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH + variables: + DEPLOY_VARIABLE: "deploy-production" # Override globally-defined DEPLOY_VARIABLE + - if: $CI_COMMIT_REF_NAME =~ /feature/ + variables: + IS_A_FEATURE: "true" # Define a new variable. + - when: always # Run the pipeline in other cases -```yaml job1: - script: "bundle exec rspec" + variables: + DEPLOY_VARIABLE: "job1-default-deploy" + rules: + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH + variables: # Override DEPLOY_VARIABLE defined + DEPLOY_VARIABLE: "job1-deploy-production" # at the job level. + - when: on_success # Run the job in other cases + script: + - echo "Run script with $DEPLOY_VARIABLE as an argument" + - echo "Run another script if $IS_A_FEATURE exists" job2: script: - - uname -a - - bundle exec rspec + - echo "Run script with $DEPLOY_VARIABLE as an argument" + - echo "Run another script if $IS_A_FEATURE exists" ``` -**Additional details**: - -- When you use [these special characters in `script`](script.md#use-special-characters-with-script), you must use single quotes (`'`) or double quotes (`"`) . - -**Related topics**: - -- You can [ignore non-zero exit codes](script.md#ignore-non-zero-exit-codes). -- [Use color codes with `script`](script.md#add-color-codes-to-script-output) - to make job logs easier to review. -- [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) - to simplify job log output. - -#### `before_script` - -Use `before_script` to define an array of commands that should run before each job's -`script` commands, but after [artifacts](#artifacts) are restored. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: An array including: +When the branch is the default branch: -- Single line commands. -- Long commands [split over multiple lines](script.md#split-long-commands). -- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). +- job1's `DEPLOY_VARIABLE` is `job1-deploy-production`. +- job2's `DEPLOY_VARIABLE` is `deploy-production`. -**Example of `before_script`:** +When the branch is `feature`: -```yaml -job: - before_script: - - echo "Execute this command before any 'script:' commands." - script: - - echo "This command executes after the job's 'before_script' commands." -``` +- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`, and `IS_A_FEATURE` is `true`. +- job2's `DEPLOY_VARIABLE` is `default-deploy`, and `IS_A_FEATURE` is `true`. -**Additional details**: +When the branch is something else: -- Scripts you specify in `before_script` are concatenated with any scripts you specify - in the main [`script`](#script). The combined scripts execute together in a single shell. +- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`. +- job2's `DEPLOY_VARIABLE` is `default-deploy`. -**Related topics**: +## Job keywords -- [Use `before_script` with `default`](script.md#set-a-default-before_script-or-after_script-for-all-jobs) - to define a default array of commands that should run before the `script` commands in all jobs. -- You can [ignore non-zero exit codes](script.md#ignore-non-zero-exit-codes). -- [Use color codes with `before_script`](script.md#add-color-codes-to-script-output) - to make job logs easier to review. -- [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) - to simplify job log output. +The following topics explain how to use keywords to configure CI/CD pipelines. -#### `after_script` +### `after_script` Use `after_script` to define an array of commands that run after each job, including failed jobs. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +[`default` section](#default). **Possible inputs**: An array including: @@ -702,7 +500,7 @@ Use `after_script` to define an array of commands that run after each job, inclu - Long commands [split over multiple lines](script.md#split-long-commands). - [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). -**Example of `after_script`:** +**Example of `after_script`**: ```yaml job: @@ -740,1168 +538,922 @@ If a job times out or is cancelled, the `after_script` commands do not execute. - [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) to simplify job log output. -### `stage` +### `allow_failure` -Use `stage` to define which [stage](#stages) a job runs in. Jobs in the same -`stage` can execute in parallel (see **Additional details**). +Use `allow_failure` to determine whether a pipeline should continue running when a job fails. -If `stage` is not defined, the job uses the `test` stage by default. +- To let the pipeline continue running subsequent jobs, use `allow_failure: true`. +- To stop the pipeline from running subsequent jobs, use `allow_failure: false`. + +When jobs are allowed to fail (`allow_failure: true`) an orange warning (**{status_warning}**) +indicates that a job failed. However, the pipeline is successful and the associated commit +is marked as passed with no warnings. + +This same warning is displayed when: + +- All other jobs in the stage are successful. +- All other jobs in the pipeline are successful. + +The default value for `allow_failure` is: + +- `true` for [manual jobs](../jobs/job_control.md#create-a-job-that-must-be-run-manually). +- `false` for manual jobs that also use [`rules`](#rules). +- `false` in all other cases. **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: An array including any number of stage names. Stage names can be: +**Possible inputs**: -- The [default stages](#stages). -- User-defined stages. +- `true` or `false`. -**Example of `stage`**: +**Example of `allow_failure`**: ```yaml -stages: - - build - - test - - deploy - job1: - stage: build + stage: test script: - - echo "This job compiles code." + - execute_script_1 job2: stage: test script: - - echo "This job tests the compiled code. It runs when the build stage completes." + - execute_script_2 + allow_failure: true job3: - script: - - echo "This job also runs in the test stage". - -job4: stage: deploy script: - - echo "This job deploys the code. It runs when the test stage completes." + - deploy_to_staging ``` -**Additional details**: +In this example, `job1` and `job2` run in parallel: -- Jobs can run in parallel if they run on different runners. -- If you have only one runner, jobs can run in parallel if the runner's - [`concurrent` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) - is greater than `1`. +- If `job1` fails, jobs in the `deploy` stage do not start. +- If `job2` fails, jobs in the `deploy` stage can still start. -#### `stage: .pre` +**Additional details**: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31441) in GitLab 12.4. +- You can use `allow_failure` as a subkey of [`rules`](#rulesallow_failure). +- You can use `allow_failure: false` with a manual job to create a [blocking manual job](../jobs/job_control.md#types-of-manual-jobs). + A blocked pipeline does not run any jobs in later stages until the manual job + is started and completes successfully. -Use the `.pre` stage to make a job run at the start of a pipeline. `.pre` is -always the first stage in a pipeline. User-defined stages execute after `.pre`. -You do not have to define `.pre` in [`stages`](#stages). +#### `allow_failure:exit_codes` -You must have a job in at least one stage other than `.pre` or `.post`. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/273157) in GitLab 13.8. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292024) in GitLab 13.9. -**Keyword type**: You can only use it with a job's `stage` keyword. +Use `allow_failure:exit_codes` to control when a job should be +allowed to fail. The job is `allow_failure: true` for any of the listed exit codes, +and `allow_failure` false for any other exit code. -**Example of `stage: .pre`**: +**Keyword type**: Job keyword. You can use it only as part of a job. -```yaml -stages: - - build - - test +**Possible inputs**: -job1: - stage: build - script: - - echo "This job runs in the build stage." +- A single exit code. +- An array of exit codes. -first-job: - stage: .pre +**Example of `allow_failure`**: + +```yaml +test_job_1: script: - - echo "This job runs in the .pre stage, before all other stages." + - echo "Run a script that results in exit code 1. This job fails." + - exit 1 + allow_failure: + exit_codes: 137 -job2: - stage: test +test_job_2: script: - - echo "This job runs in the test stage." + - echo "Run a script that results in exit code 137. This job is allowed to fail." + - exit 137 + allow_failure: + exit_codes: + - 137 + - 255 ``` -#### `stage: .post` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31441) in GitLab 12.4. - -Use the `.post` stage to make a job run at the end of a pipeline. `.post` -is always the last stage in a pipeline. User-defined stages execute before `.post`. -You do not have to define `.post` in [`stages`](#stages). - -You must have a job in at least one stage other than `.pre` or `.post`. - -**Keyword type**: You can only use it with a job's `stage` keyword. - -**Example of `stage: .post`**: - -```yaml -stages: - - build - - test - -job1: - stage: build - script: - - echo "This job runs in the build stage." +### `artifacts` -last-job: - stage: .post - script: - - echo "This job runs in the .post stage, after all other stages." +Use `artifacts` to specify which files to save as [job artifacts](../pipelines/job_artifacts.md). +Job artifacts are a list of files and directories that are +attached to the job when it [succeeds, fails, or always](#artifactswhen). -job2: - stage: test - script: - - echo "This job runs in the test stage." -``` +The artifacts are sent to GitLab after the job finishes. They are +available for download in the GitLab UI if the size is smaller than the +the [maximum artifact size](../../user/gitlab_com/index.md#gitlab-cicd). -### `extends` +By default, jobs in later stages automatically download all the artifacts created +by jobs in earlier stages. You can control artifact download behavior in jobs with +[`dependencies`](#dependencies). -Use `extends` to reuse configuration sections. It's an alternative to [YAML anchors](yaml_optimization.md#anchors) -and is a little more flexible and readable. +When using the [`needs`](#needs) keyword, jobs can only download +artifacts from the jobs defined in the `needs` configuration. -**Keyword type**: Job keyword. You can use it only as part of a job. +Job artifacts are only collected for successful jobs by default, and +artifacts are restored after [caches](#cache). -**Possible inputs:** +[Read more about artifacts](../pipelines/job_artifacts.md). -- The name of another job in the pipeline. -- A list (array) of names of other jobs in the pipeline. +#### `artifacts:exclude` -**Example of `extends`:** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15122) in GitLab 13.1 +> - Requires GitLab Runner 13.1 -```yaml -.tests: - script: rake test - stage: test - only: - refs: - - branches +Use `artifacts:exclude` to prevent files from being added to an artifacts archive. -rspec: - extends: .tests - script: rake rspec - only: - variables: - - $RSPEC -``` +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -In this example, the `rspec` job uses the configuration from the `.tests` template job. -When creating the pipeline, GitLab: +**Possible inputs**: -- Performs a reverse deep merge based on the keys. -- Merges the `.tests` content with the `rspec` job. -- Doesn't merge the values of the keys. +- An array of file paths, relative to the project directory. +- You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) or + [`doublestar.PathMatch`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#PathMatch) patterns. -The result is this `rspec` job: +**Example of `artifacts:exclude`**: ```yaml -rspec: - script: rake rspec - stage: test - only: - refs: - - branches - variables: - - $RSPEC +artifacts: + paths: + - binaries/ + exclude: + - binaries/**/*.o ``` -**Additional details:** +This example stores all files in `binaries/`, but not `*.o` files located in +subdirectories of `binaries/`. -- In GitLab 12.0 and later, you can use multiple parents for `extends`. -- The `extends` keyword supports up to eleven levels of inheritance, but you should - avoid using more than three levels. -- In the example above, `.tests` is a [hidden job](../jobs/index.md#hide-jobs), - but you can extend configuration from regular jobs as well. +**Additional details**: -**Related topics:** +- `artifacts:exclude` paths are not searched recursively. +- Files matched by [`artifacts:untracked`](#artifactsuntracked) can be excluded using + `artifacts:exclude` too. -- [Reuse configuration sections by using `extends`](yaml_optimization.md#use-extends-to-reuse-configuration-sections). -- Use `extends` to reuse configuration from [included configuration files](yaml_optimization.md#use-extends-and-include-together). - -### `rules` +**Related topics**: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3. +- [Exclude files from job artifacts](../pipelines/job_artifacts.md#exclude-files-from-job-artifacts). -Use `rules` to include or exclude jobs in pipelines. +#### `artifacts:expire_in` -Rules are evaluated when the pipeline is created, and evaluated *in order* -until the first match. When a match is found, the job is either included or excluded from the pipeline, -depending on the configuration. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) in GitLab 13.0 behind a disabled feature flag, the latest job artifacts are kept regardless of expiry time. +> - [Made default behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/229936) in GitLab 13.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241026) in GitLab 13.8, keeping latest job artifacts can be disabled at the project level. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276583) in GitLab 13.9, keeping latest job artifacts can be disabled instance-wide. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321323) in GitLab 13.12, the latest pipeline artifacts are kept regardless of expiry time. -You cannot use dotenv variables created in job scripts in rules, because rules are evaluated before any jobs run. +Use `expire_in` to specify how long [job artifacts](../pipelines/job_artifacts.md) are stored before +they expire and are deleted. The `expire_in` setting does not affect: -`rules` replaces [`only/except`](#only--except) and they can't be used together -in the same job. If you configure one job to use both keywords, the GitLab returns -a `key may not be used with rules` error. +- Artifacts from the latest job, unless keeping the latest job artifacts is: + - [Disabled at the project level](../pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). + - [Disabled instance-wide](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). +- [Pipeline artifacts](../pipelines/pipeline_artifacts.md). You can't specify an expiration date for + pipeline artifacts. See [When pipeline artifacts are deleted](../pipelines/pipeline_artifacts.md#when-pipeline-artifacts-are-deleted) + for more information. -`rules` accepts an array of rules defined with: +After their expiry, artifacts are deleted hourly by default (using a cron job), and are not +accessible anymore. -- `if` -- `changes` -- `exists` -- `allow_failure` -- `variables` -- `when` +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -You can combine multiple keywords together for [complex rules](../jobs/job_control.md#complex-rules). +**Possible inputs**: The expiry time. If no unit is provided, the time is in seconds. +Valid values include: -The job is added to the pipeline: +- `'42'` +- `42 seconds` +- `3 mins 4 sec` +- `2 hrs 20 min` +- `2h20min` +- `6 mos 1 day` +- `47 yrs 6 mos and 4d` +- `3 weeks and 2 days` +- `never` -- If an `if`, `changes`, or `exists` rule matches and also has `when: on_success` (default), - `when: delayed`, or `when: always`. -- If a rule is reached that is only `when: on_success`, `when: delayed`, or `when: always`. +**Example of `artifacts:expire_in`**: -The job is not added to the pipeline: +```yaml +job: + artifacts: + expire_in: 1 week +``` -- If no rules match. -- If a rule matches and has `when: never`. +**Additional details**: -You can use [`!reference` tags](yaml_optimization.md#reference-tags) to [reuse `rules` configuration](../jobs/job_control.md#reuse-rules-in-different-jobs) -in different jobs. +- The expiration time period begins when the artifact is uploaded and stored on GitLab. + If the expiry time is not defined, it defaults to the [instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration). +- To override the expiration date and protect artifacts from being automatically deleted: + - Select **Keep** on the job page. + - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/22761), set the value of + `expire_in` to `never`. -#### `rules:if` +#### `artifacts:expose_as` -Use `rules:if` clauses to specify when to add a job to a pipeline: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15018) in GitLab 12.5. -- If an `if` statement is true, add the job to the pipeline. -- If an `if` statement is true, but it's combined with `when: never`, do not add the job to the pipeline. -- If no `if` statements are true, do not add the job to the pipeline. +Use the `artifacts:expose_as` keyword to +[expose job artifacts in the merge request UI](../pipelines/job_artifacts.md#expose-job-artifacts-in-the-merge-request-ui). -`if:` clauses are evaluated based on the values of [predefined CI/CD variables](../variables/predefined_variables.md) -or [custom CI/CD variables](../variables/index.md#custom-cicd-variables). +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -**Keyword type**: Job-specific and pipeline-specific. You can use it as part of a job -to configure the job behavior, or with [`workflow`](#workflow) to configure the pipeline behavior. +**Possible inputs**: -**Possible inputs**: A [CI/CD variable expression](../jobs/job_control.md#cicd-variable-expressions). +- The name to display in the merge request UI for the artifacts download link. + Must be combined with [`artifacts:paths`](#artifactspaths). -**Example of `rules:if`**: +**Example of `artifacts:expose_as`**: ```yaml -job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != $CI_DEFAULT_BRANCH' - when: never - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/' - when: manual - allow_failure: true - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' +test: + script: ["echo 'test' > file.txt"] + artifacts: + expose_as: 'artifact 1' + paths: ['file.txt'] ``` **Additional details**: -- If a rule matches and has no `when` defined, the rule uses the `when` - defined for the job, which defaults to `on_success` if not defined. -- You can define `when` once per rule, or once at the job-level, which applies to - all rules. You can't mix `when` at the job-level with `when` in rules. -- Unlike variables in [`script`](../variables/index.md#use-cicd-variables-in-job-scripts) - sections, variables in rules expressions are always formatted as `$VARIABLE`. - - You can use `rules:if` with `include` to [conditionally include other configuration files](includes.md#use-rules-with-include). +- If `artifacts:paths` uses [CI/CD variables](../variables/index.md), the artifacts do not display in the UI. +- A maximum of 10 job artifacts per merge request can be exposed. +- Glob patterns are unsupported. +- If a directory is specified and there is more than one file in the directory, + the link is to the job [artifacts browser](../pipelines/job_artifacts.md#download-job-artifacts). +- If [GitLab Pages](../../administration/pages/index.md) is enabled, GitLab automatically + renders the artifacts when the artifacts is a single file with one of these extensions: + - `.html` or `.htm` + - `.txt` + - `.json` + - `.xml` + - `.log` **Related topics**: -- [Common `if` expressions for `rules`](../jobs/job_control.md#common-if-clauses-for-rules). -- [Avoid duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines). - -#### `rules:changes` - -Use `rules:changes` to specify when to add a job to a pipeline by checking for changes -to specific files. - -WARNING: -You should use `rules: changes` only with **branch pipelines** or **merge request pipelines**. -You can use `rules: changes` with other pipeline types, but `rules: changes` always -evaluates to true when there is no Git `push` event. Tag pipelines, scheduled pipelines, -and so on do **not** have a Git `push` event associated with them. A `rules: changes` job -is **always** added to those pipelines if there is no `if:` that limits the job to -branch or merge request pipelines. - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: An array of file paths. In GitLab 13.6 and later, -[file paths can include variables](../jobs/job_control.md#variables-in-ruleschanges). - -**Example of `rules:changes`**: - -```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - changes: - - Dockerfile - when: manual - allow_failure: true -``` +- [Expose job artifacts in the merge request UI](../pipelines/job_artifacts.md#expose-job-artifacts-in-the-merge-request-ui). -- If the pipeline is a merge request pipeline, check `Dockerfile` for changes. -- If `Dockerfile` has changed, add the job to the pipeline as a manual job, and the pipeline - continues running even if the job is not triggered (`allow_failure: true`). -- If `Dockerfile` has not changed, do not add job to any pipeline (same as `when: never`). +#### `artifacts:name` -**Additional details**: +Use the `artifacts:name` keyword to define the name of the created artifacts +archive. You can specify a unique name for every archive. -- `rules: changes` works the same way as [`only: changes` and `except: changes`](#onlychanges--exceptchanges). -- You can use `when: never` to implement a rule similar to [`except:changes`](#onlychanges--exceptchanges). -- `changes` resolves to `true` if any of the matching files are changed (an `OR` operation). - -#### `rules:exists` +If not defined, the default name is `artifacts`, which becomes `artifacts.zip` when downloaded. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -Use `exists` to run a job when certain files exist in the repository. +**Possible inputs**: -**Keyword type**: Job keyword. You can use it only as part of a job. +- The name of the artifacts archive. Can use [CI/CD variables](../variables/index.md). Must be combined with + [`artifacts:paths`](#artifactspaths). -**Possible inputs**: An array of file paths. Paths are relative to the project directory (`$CI_PROJECT_DIR`) -and can't directly link outside it. File paths can use glob patterns. +**Example of `artifacts:name`**: -**Example of `rules:exists`**: +To create an archive with a name of the current job: ```yaml job: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - exists: - - Dockerfile + artifacts: + name: "job1-artifacts-file" + paths: + - binaries/ ``` -`job` runs if a `Dockerfile` exists anywhere in the repository. - -**Additional details**: - -- Glob patterns are interpreted with Ruby [`File.fnmatch`](https://docs.ruby-lang.org/en/2.7.0/File.html#method-c-fnmatch) - with the flags `File::FNM_PATHNAME | File::FNM_DOTMATCH | File::FNM_EXTGLOB`. -- For performance reasons, GitLab matches a maximum of 10,000 `exists` patterns or - file paths. After the 10,000th check, rules with patterned globs always match. - In other words, the `exists` rule always assumes a match in projects with more - than 10,000 files. -- `exists` resolves to `true` if any of the listed files are found (an `OR` operation). +**Related topics**: -#### `rules:allow_failure` +- [Use CI/CD variables to define the artifacts name](../pipelines/job_artifacts.md#use-cicd-variables-to-define-the-artifacts-name). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30235) in GitLab 12.8. +#### `artifacts:paths` -Use [`allow_failure: true`](#allow_failure) in `rules:` to allow a job to fail -without stopping the pipeline. +Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly +link outside it. -You can also use `allow_failure: true` with a manual job. The pipeline continues -running without waiting for the result of the manual job. `allow_failure: false` -combined with `when: manual` in rules causes the pipeline to wait for the manual -job to run before continuing. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -**Keyword type**: Job keyword. You can use it only as part of a job. +**Possible inputs**: -**Possible inputs**: `true` or `false`. Defaults to `false` if not defined. +- An array of file paths, relative to the project directory. +- You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) + patterns and: + - In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), + [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). + - In GitLab Runner 12.10 and earlier, [`filepath.Match`](https://pkg.go.dev/path/filepath#Match). -**Example of `rules:allow_failure`**: +**Example of `artifacts:paths`**: ```yaml job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH' - when: manual - allow_failure: true + artifacts: + paths: + - binaries/ + - .config ``` -If the rule matches, then the job is a manual job with `allow_failure: true`. +This example creates an artifact with `.config` and all the files in the `binaries` directory. **Additional details**: -- The rule-level `rules:allow_failure` overrides the job-level [`allow_failure`](#allow_failure), - and only applies when the specific rule triggers the job. - -#### `rules:variables` +- If not used with [`artifacts:name`](#artifactsname) defined, the artifacts file + is named `artifacts`, which becomes `artifacts.zip` when downloaded. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209864) in GitLab 13.7. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289803) in GitLab 13.10. +**Related topics**: -Use [`variables`](#variables) in `rules:` to define variables for specific conditions. +- To restrict which jobs a specific job fetches artifacts from, see [`dependencies`](#dependencies). +- [Create job artifacts](../pipelines/job_artifacts.md#create-job-artifacts). -**Keyword type**: Job-specific. You can use it only as part of a job. - -**Possible inputs**: A hash of variables in the format `VARIABLE-NAME: value`. +#### `artifacts:public` -**Example of `rules:variables`**: +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 +> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's recommended for production use. -```yaml -job: - variables: - DEPLOY_VARIABLE: "default-deploy" - rules: - - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH - variables: # Override DEPLOY_VARIABLE defined - DEPLOY_VARIABLE: "deploy-production" # at the job level. - - if: $CI_COMMIT_REF_NAME =~ /feature/ - variables: - IS_A_FEATURE: "true" # Define a new variable. - script: - - echo "Run script with $DEPLOY_VARIABLE as an argument" - - echo "Run another script if $IS_A_FEATURE exists" -``` +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `non_public_artifacts`. On +GitLab.com, this feature is not available. -### `only` / `except` +Use `artifacts:public` to determine whether the job artifacts should be +publicly available. -NOTE: -`only` and `except` are not being actively developed. [`rules`](#rules) is the preferred -keyword to control when to add jobs to pipelines. +When `artifacts:public` is `true` (default), the artifacts in +public pipelines are available for download by anonymous and guest users. -You can use `only` and `except` to control when to add jobs to pipelines. +To deny read access for anonymous and guest users to artifacts in public +pipelines, set `artifacts:public` to `false`: -- Use `only` to define when a job runs. -- Use `except` to define when a job **does not** run. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -Four keywords can be used with `only` and `except`: +**Possible inputs**: -- [`refs`](#onlyrefs--exceptrefs) -- [`variables`](#onlyvariables--exceptvariables) -- [`changes`](#onlychanges--exceptchanges) -- [`kubernetes`](#onlykubernetes--exceptkubernetes) +- `true` (default if not defined) or `false`. -See [specify when jobs run with `only` and `except`](../jobs/job_control.md#specify-when-jobs-run-with-only-and-except) -for more details and examples. +**Example of `artifacts:paths`**: -#### `only:refs` / `except:refs` +```yaml +job: + artifacts: + public: false +``` -Use the `only:refs` and `except:refs` keywords to control when to add jobs to a -pipeline based on branch names or pipeline types. +#### `artifacts:reports` -**Keyword type**: Job keyword. You can use it only as part of a job. +Use [`artifacts:reports`](artifacts_reports.md) to collect artifacts generated by +included templates in jobs. -**Possible inputs**: An array including any number of: +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -- Branch names, for example `main` or `my-feature-branch`. -- [Regular expressions](../jobs/job_control.md#only--except-regex-syntax) - that match against branch names, for example `/^feature-.*/`. -- The following keywords: +**Possible inputs**: - | **Value** | **Description** | - | -------------------------|-----------------| - | `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | - | `branches` | When the Git reference for a pipeline is a branch. | - | `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | - | `external` | When you use CI services other than GitLab. | - | `external_pull_requests` | When an external pull request on GitHub is created or updated (See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests)). | - | `merge_requests` | For pipelines created when a merge request is created or updated. Enables [merge request pipelines](../pipelines/merge_request_pipelines.md), [merged results pipelines](../pipelines/pipelines_for_merged_results.md), and [merge trains](../pipelines/merge_trains.md). | - | `pipelines` | For [multi-project pipelines](../pipelines/multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api), or the [`trigger`](#trigger) keyword. | - | `pushes` | For pipelines triggered by a `git push` event, including for branches and tags. | - | `schedules` | For [scheduled pipelines](../pipelines/schedules.md). | - | `tags` | When the Git reference for a pipeline is a tag. | - | `triggers` | For pipelines created by using a [trigger token](../triggers/index.md#authentication-tokens). | - | `web` | For pipelines created by selecting **Run pipeline** in the GitLab UI, from the project's **CI/CD > Pipelines** section. | +- See list of available [artifacts reports types](artifacts_reports.md). -**Example of `only:refs` and `except:refs`**: +**Example of `artifacts:reports`**: ```yaml -job1: - script: echo - only: - - main - - /^issue-.*$/ - - merge_requests - -job2: - script: echo - except: - - main - - /^stable-branch.*$/ - - schedules +rspec: + stage: test + script: + - bundle install + - rspec --format RspecJunitFormatter --out rspec.xml + artifacts: + reports: + junit: rspec.xml ``` -**Additional details:** - -- Scheduled pipelines run on specific branches, so jobs configured with `only: branches` - run on scheduled pipelines too. Add `except: schedules` to prevent jobs with `only: branches` - from running on scheduled pipelines. -- `only` or `except` used without any other keywords are equivalent to `only: refs` - or `except: refs`. For example, the following two jobs configurations have the same - behavior: - - ```yaml - job1: - script: echo - only: - - branches - - job2: - script: echo - only: - refs: - - branches - ``` - -- If a job does not use `only`, `except`, or [`rules`](#rules), then `only` is set to `branches` - and `tags` by default. +**Additional details**: - For example, `job1` and `job2` are equivalent: +- Combining reports in parent pipelines using [artifacts from child pipelines](#needspipelinejob) is + not supported. Track progress on adding support in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215725). +- To be able to browse the report output files, include the [`artifacts:paths`](#artifactspaths) keyword. Please note that this will upload and store the artifact twice. +- The test reports are collected regardless of the job results (success or failure). + You can use [`artifacts:expire_in`](#artifactsexpire_in) to set up an expiration + date for artifacts reports. - ```yaml - job1: - script: echo 'test' +#### `artifacts:untracked` - job2: - script: echo 'test' - only: - - branches - - tags - ``` +Use `artifacts:untracked` to add all Git untracked files as artifacts (along +with the paths defined in `artifacts:paths`). `artifacts:untracked` ignores configuration +in the repository's `.gitignore` file. -#### `only:variables` / `except:variables` +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -Use the `only:variables` or `except:variables` keywords to control when to add jobs -to a pipeline, based on the status of [CI/CD variables](../variables/index.md). +**Possible inputs**: -**Keyword type**: Job keyword. You can use it only as part of a job. +- `true` or `false` (default if not defined). -**Possible inputs**: An array of [CI/CD variable expressions](../jobs/job_control.md#cicd-variable-expressions). +**Example of `artifacts:untracked`**: -**Example of `only:variables`**: +Save all Git untracked files: ```yaml -deploy: - script: cap staging deploy - only: - variables: - - $RELEASE == "staging" - - $STAGING +job: + artifacts: + untracked: true ``` **Related topics**: -- [`only:variables` and `except:variables` examples](../jobs/job_control.md#only-variables--except-variables-examples). - -#### `only:changes` / `except:changes` +- [Add untracked files to artifacts](../pipelines/job_artifacts.md#add-untracked-files-to-artifacts). -Use the `changes` keyword with `only` to run a job, or with `except` to skip a job, -when a Git push event modifies a file. - -Use `changes` in pipelines with the following refs: +#### `artifacts:when` -- `branches` -- `external_pull_requests` -- `merge_requests` (see additional details about [using `only:changes` with pipelines for merge requests](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests)) +Use `artifacts:when` to upload artifacts on job failure or despite the +failure. -**Keyword type**: Job keyword. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -**Possible inputs**: An array including any number of: +**Possible inputs**: -- Paths to files. -- Wildcard paths for single directories, for example `path/to/directory/*`, or a directory - and all its subdirectories, for example `path/to/directory/**/*`. -- Wildcard ([glob](https://en.wikipedia.org/wiki/Glob_(programming))) paths for all - files with the same extension or multiple extensions, for example `*.md` or `path/to/directory/*.{rb,py,sh}`. -- Wildcard paths to files in the root directory, or all directories, wrapped in double quotes. - For example `"*.json"` or `"**/*.json"`. +- `on_success` (default): Upload artifacts only when the job succeeds. +- `on_failure`: Upload artifacts only when the job fails. +- `always`: Always upload artifacts. For example, when + [uploading artifacts](../unit_test_reports.md#viewing-junit-screenshots-on-gitlab) + required to troubleshoot failing tests. -**Example of `only:changes`**: +**Example of `artifacts:when`**: ```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - only: - refs: - - branches - changes: - - Dockerfile - - docker/scripts/* - - dockerfiles/**/* - - more_scripts/*.{rb,py,sh} - - "**/*.json" +job: + artifacts: + when: on_failure ``` -**Additional details**: - -- `changes` resolves to `true` if any of the matching files are changed (an `OR` operation). -- If you use refs other than `branches`, `external_pull_requests`, or `merge_requests`, - `changes` can't determine if a given file is new or old and always returns `true`. -- If you use `only: changes` with other refs, jobs ignore the changes and always run. -- If you use `except: changes` with other refs, jobs ignore the changes and never run. - -**Related topics**: +### `before_script` -- [`only: changes` and `except: changes` examples](../jobs/job_control.md#onlychanges--exceptchanges-examples). -- If you use `changes` with [only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds), - you should [also use `only:merge_requests`](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests). -- Use `changes` with [new branches or tags *without* pipelines for merge requests](../jobs/job_control.md#use-onlychanges-without-pipelines-for-merge-requests). -- Use `changes` with [scheduled pipelines](../jobs/job_control.md#use-onlychanges-with-scheduled-pipelines). - -#### `only:kubernetes` / `except:kubernetes` +Use `before_script` to define an array of commands that should run before each job's +`script` commands, but after [artifacts](#artifacts) are restored. -Use `only:kubernetes` or `except:kubernetes` to control if jobs are added to the pipeline -when the Kubernetes service is active in the project. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -**Keyword type**: Job-specific. You can use it only as part of a job. +**Possible inputs**: An array including: -**Possible inputs**: The `kubernetes` strategy accepts only the `active` keyword. +- Single line commands. +- Long commands [split over multiple lines](script.md#split-long-commands). +- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). -**Example of `only:kubernetes`**: +**Example of `before_script`**: ```yaml -deploy: - only: - kubernetes: active +job: + before_script: + - echo "Execute this command before any 'script:' commands." + script: + - echo "This command executes after the job's 'before_script' commands." ``` -In this example, the `deploy` job runs only when the Kubernetes service is active -in the project. - -### `needs` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) in GitLab 12.2. -> - In GitLab 12.3, maximum number of jobs in `needs` array raised from five to 50. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30631) in GitLab 12.8, `needs: []` lets jobs start immediately. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) in GitLab 14.2, you can refer to jobs in the same stage as the job you are configuring. +**Additional details**: -Use `needs:` to execute jobs out-of-order. Relationships between jobs -that use `needs` can be visualized as a [directed acyclic graph](../directed_acyclic_graph/index.md). +- Scripts you specify in `before_script` are concatenated with any scripts you specify + in the main [`script`](#script). The combined scripts execute together in a single shell. -You can ignore stage ordering and run some jobs without waiting for others to complete. -Jobs in multiple stages can run concurrently. +**Related topics**: -**Keyword type**: Job keyword. You can use it only as part of a job. +- [Use `before_script` with `default`](script.md#set-a-default-before_script-or-after_script-for-all-jobs) + to define a default array of commands that should run before the `script` commands in all jobs. +- You can [ignore non-zero exit codes](script.md#ignore-non-zero-exit-codes). +- [Use color codes with `before_script`](script.md#add-color-codes-to-script-output) + to make job logs easier to review. +- [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) + to simplify job log output. -**Possible inputs**: +### `cache` -- An array of jobs. -- An empty array (`[]`), to set the job to start as soon as the pipeline is created. +Use `cache` to specify a list of files and directories to +cache between jobs. You can only use paths that are in the local working copy. -**Example of `needs`**: +Caching is shared between pipelines and jobs. Caches are restored before [artifacts](#artifacts). -```yaml -linux:build: - stage: build - script: echo "Building linux..." +Learn more about caches in [Caching in GitLab CI/CD](../caching/index.md). -mac:build: - stage: build - script: echo "Building mac..." +#### `cache:paths` -lint: - stage: test - needs: [] - script: echo "Linting..." +Use the `cache:paths` keyword to choose which files or directories to cache. -linux:rspec: - stage: test - needs: ["linux:build"] - script: echo "Running rspec on linux..." +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -mac:rspec: - stage: test - needs: ["mac:build"] - script: echo "Running rspec on mac..." +**Possible inputs**: -production: - stage: deploy - script: echo "Running production..." -``` +- An array of paths relative to the project directory (`$CI_PROJECT_DIR`). + You can use wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) + patterns: + - In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), + [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). + - In GitLab Runner 12.10 and earlier, + [`filepath.Match`](https://pkg.go.dev/path/filepath#Match). -This example creates four paths of execution: +**Example of `cache:paths`**: -- Linter: The `lint` job runs immediately without waiting for the `build` stage - to complete because it has no needs (`needs: []`). -- Linux path: The `linux:rspec` job runs as soon as the `linux:build` - job finishes, without waiting for `mac:build` to finish. -- macOS path: The `mac:rspec` jobs runs as soon as the `mac:build` - job finishes, without waiting for `linux:build` to finish. -- The `production` job runs as soon as all previous jobs finish: - `linux:build`, `linux:rspec`, `mac:build`, `mac:rspec`. +Cache all files in `binaries` that end in `.apk` and the `.config` file: -**Additional details**: +```yaml +rspec: + script: + - echo "This job uses a cache." + cache: + key: binaries-cache + paths: + - binaries/*.apk + - .config +``` -- The maximum number of jobs that a single job can have in the `needs:` array is limited: - - For GitLab.com, the limit is 50. For more information, see our - [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7541). - - For self-managed instances, the default limit is 50. This limit [can be changed](../../administration/cicd.md#set-the-needs-job-limit). -- If `needs:` refers to a job that uses the [`parallel`](#parallel) keyword, - it depends on all jobs created in parallel, not just one job. It also downloads - artifacts from all the parallel jobs by default. If the artifacts have the same - name, they overwrite each other and only the last one downloaded is saved. -- In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) you - can refer to jobs in the same stage as the job you are configuring. This feature is - enabled on GitLab.com and ready for production use. On self-managed [GitLab 14.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) - this feature is available by default. -- In GitLab 14.0 and older, you can only refer to jobs in earlier stages. Stages must be - explicitly defined for all jobs that use the `needs:` keyword, or are referenced - in a job's `needs:` section. -- In GitLab 13.9 and older, if `needs:` refers to a job that might not be added to - a pipeline because of `only`, `except`, or `rules`, the pipeline might fail to create. +**Related topics**: -#### `needs:artifacts` +- See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more + `cache:paths` examples. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.6. +#### `cache:key` -When a job uses `needs`, it no longer downloads all artifacts from previous stages -by default, because jobs with `needs` can start before earlier stages complete. With -`needs` you can only download artifacts from the jobs listed in the `needs:` configuration. +Use the `cache:key` keyword to give each cache a unique identifying key. All jobs +that use the same cache key use the same cache, including in different pipelines. -Use `artifacts: true` (default) or `artifacts: false` to control when artifacts are -downloaded in jobs that use `needs`. +If not set, the default key is `default`. All jobs with the `cache` keyword but +no `cache:key` share the `default` cache. -**Keyword type**: Job keyword. You can use it only as part of a job. Must be used with `needs:job`. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). **Possible inputs**: -- `true` (default) or `false`. +- A string. +- A [predefined variables](../variables/index.md). +- A combination of both. -**Example of `needs:artifacts`**: +**Example of `cache:key`**: ```yaml -test-job1: - stage: test - needs: - - job: build_job1 - artifacts: true - -test-job2: - stage: test - needs: - - job: build_job2 - artifacts: false - -test-job3: - needs: - - job: build_job1 - artifacts: true - - job: build_job2 - - build_job3 +cache-job: + script: + - echo "This job uses a cache." + cache: + key: binaries-cache-$CI_COMMIT_REF_SLUG + paths: + - binaries/ ``` -In this example: +**Additional details**: -- The `test-job1` job downloads the `build_job1` artifacts -- The `test-job2` job does not download the `build_job2` artifacts. -- The `test-job3` job downloads the artifacts from all three `build_jobs`, because - `artifacts:` is `true`, or defaults to `true`, for all three needed jobs. +- If you use **Windows Batch** to run your shell scripts you must replace + `$` with `%`. For example: `key: %CI_COMMIT_REF_SLUG%` +- The `cache:key` value can't contain: -**Additional details**: + - The `/` character, or the equivalent URI-encoded `%2F`. + - Only the `.` character (any number), or the equivalent URI-encoded `%2E`. -- In GitLab 12.6 and later, you can't combine the [`dependencies`](#dependencies) keyword - with `needs`. +- The cache is shared between jobs, so if you're using different + paths for different jobs, you should also set a different `cache:key`. + Otherwise cache content can be overwritten. -#### `needs:project` **(PREMIUM)** +**Related topics**: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.7. +- You can specify a [fallback cache key](../caching/index.md#use-a-fallback-cache-key) + to use if the specified `cache:key` is not found. +- You can [use multiple cache keys](../caching/index.md#use-multiple-caches) in a single job. +- See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more + `cache:key` examples. -Use `needs:project` to download artifacts from up to five jobs in other pipelines. -The artifacts are downloaded from the latest successful pipeline for the specified ref. +##### `cache:key:files` -If there is a pipeline running for the specified ref, a job with `needs:project` -does not wait for the pipeline to complete. Instead, the job downloads the artifact -from the latest pipeline that completed successfully. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab 12.5. -`needs:project` must be used with `job:`, `ref:`, and `artifacts:`. +Use the `cache:key:files` keyword to generate a new key when one or two specific files +change. `cache:key:files` lets you reuse some caches, and rebuild them less often, +which speeds up subsequent pipeline runs. -**Keyword type**: Job keyword. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). **Possible inputs**: -- `needs:project`: A full project path, including namespace and group. If the - project is in the same group or namespace, you can omit them from the `project:` - keyword. For example: `project: group/project-name` or `project: project-name`. -- `job`: The job to download artifacts from. -- `ref`: The ref to download artifacts from. -- `artifacts`: Must be `true` to download artifacts. +- An array of one or two file paths. -**Examples of `needs:project`**: +**Example of `cache:key:files`**: ```yaml -build_job: - stage: build +cache-job: script: - - ls -lhR - needs: - - project: namespace/group/project-name - job: build-1 - ref: main - artifacts: true + - echo "This job uses a cache." + cache: + key: + files: + - Gemfile.lock + - package.json + paths: + - vendor/ruby + - node_modules ``` -In this example, `build_job` downloads the artifacts from the latest successful `build-1` job -on the `main` branch in the `group/project-name` project. - -In GitLab 13.3 and later, you can use [CI/CD variables](../variables/index.md) in `needs:project`, -for example: - -```yaml -build_job: - stage: build - script: - - ls -lhR - needs: - - project: $CI_PROJECT_PATH - job: $DEPENDENCY_JOB_NAME - ref: $ARTIFACTS_DOWNLOAD_REF - artifacts: true -``` +This example creates a cache for Ruby and Node.js dependencies. The cache +is tied to the current versions of the `Gemfile.lock` and `package.json` files. When one of +these files changes, a new cache key is computed and a new cache is created. Any future +job runs that use the same `Gemfile.lock` and `package.json` with `cache:key:files` +use the new cache, instead of rebuilding the dependencies. **Additional details**: -- To download artifacts from a different pipeline in the current project, set `project:` - to be the same as the current project, but use a different ref than the current pipeline. - Concurrent pipelines running on the same ref could override the artifacts. -- The user running the pipeline must have at least the Reporter role for the group or project, - or the group/project must have public visibility. -- You can't use `needs:project` in the same job as [`trigger`](#trigger). -- When using `needs:project` to download artifacts from another pipeline, the job does not wait for - the needed job to complete. [Directed acyclic graph](../directed_acyclic_graph/index.md) - behavior is limited to jobs in the same pipeline. Make sure that the needed job in the other - pipeline completes before the job that needs it tries to download the artifacts. -- You can't download artifacts from jobs that run in [`parallel:`](#parallel). -- Support for [CI/CD variables](../variables/index.md) in `project`, `job`, and `ref` was - [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202093) in GitLab 13.3. - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235761) in GitLab 13.4. - -**Related topics**: - -- To download artifacts between [parent-child pipelines](../pipelines/parent_child_pipelines.md), - use [`needs:pipeline:job`](#needspipelinejob). +- The cache `key` is a SHA computed from the most recent commits +that changed each listed file. + If neither file is changed in any commits, the fallback key is `default`. -#### `needs:pipeline:job` +##### `cache:key:prefix` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/255983) in GitLab 13.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab 12.5. -A [child pipeline](../pipelines/parent_child_pipelines.md) can download artifacts from a job in -its parent pipeline or another child pipeline in the same parent-child pipeline hierarchy. +Use `cache:key:prefix` to combine a prefix with the SHA computed for [`cache:key:files`](#cachekeyfiles). -**Keyword type**: Job keyword. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). **Possible inputs**: -- `needs:pipeline`: A pipeline ID. Must be a pipeline present in the same parent-child pipeline hierarchy. -- `job:`: The job to download artifacts from. - -**Example of `needs:pipeline:job`**: - -- Parent pipeline (`.gitlab-ci.yml`): - - ```yaml - create-artifact: - stage: build - script: echo 'sample artifact' > artifact.txt - artifacts: - paths: [artifact.txt] - - child-pipeline: - stage: test - trigger: - include: child.yml - strategy: depend - variables: - PARENT_PIPELINE_ID: $CI_PIPELINE_ID - ``` +- A string +- A [predefined variables](../variables/index.md) +- A combination of both. -- Child pipeline (`child.yml`): +**Example of `cache:key:prefix`**: - ```yaml - use-artifact: - script: cat artifact.txt - needs: - - pipeline: $PARENT_PIPELINE_ID - job: create-artifact - ``` +```yaml +rspec: + script: + - echo "This rspec job uses a cache." + cache: + key: + files: + - Gemfile.lock + prefix: $CI_JOB_NAME + paths: + - vendor/ruby +``` -In this example, the `create-artifact` job in the parent pipeline creates some artifacts. -The `child-pipeline` job triggers a child pipeline, and passes the `CI_PIPELINE_ID` -variable to the child pipeline as a new `PARENT_PIPELINE_ID` variable. The child pipeline -can use that variable in `needs:pipeline` to download artifacts from the parent pipeline. +For example, adding a `prefix` of `$CI_JOB_NAME` causes the key to look like `rspec-feef9576d21ee9b6a32e30c5c79d0a0ceb68d1e5`. +If a branch changes `Gemfile.lock`, that branch has a new SHA checksum for `cache:key:files`. +A new cache key is generated, and a new cache is created for that key. If `Gemfile.lock` +is not found, the prefix is added to `default`, so the key in the example would be `rspec-default`. **Additional details**: -- The `pipeline` attribute does not accept the current pipeline ID (`$CI_PIPELINE_ID`). - To download artifacts from a job in the current pipeline, use [`needs`](#needsartifacts). - -#### `needs:optional` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30680) in GitLab 13.10. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 14.0. - -To need a job that sometimes does not exist in the pipeline, add `optional: true` -to the `needs` configuration. If not defined, `optional: false` is the default. +- If no file in `cache:key:files` is changed in any commits, the prefix is added to the `default` key. -Jobs that use [`rules`](#rules), [`only`, or `except`](#only--except), might -not always exist in a pipeline. When the pipeline is created, GitLab checks the `needs` -relationships before starting it. Without `optional: true`, needs relationships that -point to a job that does not exist stops the pipeline from starting and causes a pipeline -error similar to: +#### `cache:untracked` -- `'job1' job needs 'job2' job, but it was not added to the pipeline` +Use `untracked: true` to cache all files that are untracked in your Git repository: -**Keyword type**: Job keyword. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). **Possible inputs**: -- `job:`: The job to make optional. - `true` or `false` (default). -**Example of `needs:optional`**: +**Example of `cache:untracked`**: ```yaml -build: - stage: build - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - rspec: - stage: test - needs: - - job: build - optional: true + script: test + cache: + untracked: true ``` -In this example: +**Additional details**: -- When the branch is the default branch, the `build` job exists in the pipeline, and the `rspec` - job waits for it to complete before starting. -- When the branch is not the default branch, the `build` job does not exist in the pipeline. - The `rspec` job runs immediately (similar to `needs: []`) because its `needs` - relationship to the `build` job is optional. +- You can combine `cache:untracked` with `cache:paths` to cache all untracked files + as well as files in the configured paths. For example: -#### `needs:pipeline` + ```yaml + rspec: + script: test + cache: + untracked: true + paths: + - binaries/ + ``` -You can mirror the pipeline status from an upstream pipeline to a bridge job by -using the `needs:pipeline` keyword. The latest pipeline status from the default branch is -replicated to the bridge job. +#### `cache:when` -**Keyword type**: Job keyword. You can use it only as part of a job. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18969) in GitLab 13.5 and GitLab Runner v13.5.0. + +Use `cache:when` to define when to save the cache, based on the status of the job. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). **Possible inputs**: -- A full project path, including namespace and group. If the - project is in the same group or namespace, you can omit them from the `project:` - keyword. For example: `project: group/project-name` or `project: project-name`. +- `on_success` (default): Save the cache only when the job succeeds. +- `on_failure`: Save the cache only when the job fails. +- `always`: Always save the cache. -**Example of `needs:pipeline`**: +**Example of `cache:when`**: ```yaml -upstream_bridge: - stage: test - needs: - pipeline: other/project +rspec: + script: rspec + cache: + paths: + - rspec/ + when: 'always' ``` -**Additional details**: +This example stores the cache whether or not the job fails or succeeds. -- If you add the `job` keyword to `needs:pipeline`, the job no longer mirrors the - pipeline status. The behavior changes to [`needs:pipeline:job`](#needspipelinejob). +#### `cache:policy` -### `tags` +To change the upload and download behavior of a cache, use the `cache:policy` keyword. +By default, the job downloads the cache when the job starts, and uploads changes +to the cache when the job ends. This caching style is the `pull-push` policy (default). -> - A limit of 50 tags per job [enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338929) in GitLab 14.3. -> - A limit of 50 tags per job [enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/339855) in GitLab 14.3. +To set a job to only download the cache when the job starts, but never upload changes +when the job finishes, use `cache:policy:pull`. -Use `tags` to select a specific runner from the list of all runners that are -available for the project. +To set a job to only upload a cache when the job finishes, but never download the +cache when the job starts, use `cache:policy:push`. -When you register a runner, you can specify the runner's tags, for -example `ruby`, `postgres`, or `development`. To pick up and run a job, a runner must -be assigned every tag listed in the job. +Use the `pull` policy when you have many jobs executing in parallel that use the same cache. +This policy speeds up job execution and reduces load on the cache server. You can +use a job with the `push` policy to build the cache. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +[`default` section](#default). **Possible inputs**: -- An array of tag names. -- [CI/CD variables](../runners/configure_runners.md#use-cicd-variables-in-tags) in GitLab 14.1 and later. +- `pull` +- `push` +- `pull-push` (default) -**Example of `tags`**: +**Example of `cache:policy`**: ```yaml -job: - tags: - - ruby - - postgres +prepare-dependencies-job: + stage: build + cache: + key: gems + paths: + - vendor/bundle + policy: push + script: + - echo "This job only downloads dependencies and builds the cache." + - echo "Downloading dependencies..." + +faster-test-job: + stage: test + cache: + key: gems + paths: + - vendor/bundle + policy: pull + script: + - echo "This job script uses the cache, but does not update it." + - echo "Running tests..." ``` -In this example, only runners with *both* the `ruby` and `postgres` tags can run the job. +### `coverage` -**Additional details**: +Use `coverage` with a custom regular expression to configure how code coverage +is extracted from the job output. The coverage is shown in the UI if at least one +line in the job output matches the regular expression. -- In [GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/issues/338479) and later, - the number of tags must be less than `50`. +To extract the code coverage value in the matching line, GitLab uses this +regular expression: `\d+(\.\d+)?`. -**Related topics**: +**Possible inputs**: -- [Use tags to control which jobs a runner can run](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run). +- A regular expression. Must start and end with `/`. -### `allow_failure` +**Example of `coverage`**: -Use `allow_failure` to determine whether a pipeline should continue running when a job fails. +```yaml +job1: + script: rspec + coverage: '/Code coverage: \d+\.\d+/' +``` -- To let the pipeline continue running subsequent jobs, use `allow_failure: true`. -- To stop the pipeline from running subsequent jobs, use `allow_failure: false`. +In this example: -When jobs are allowed to fail (`allow_failure: true`) an orange warning (**{status_warning}**) -indicates that a job failed. However, the pipeline is successful and the associated commit -is marked as passed with no warnings. +1. GitLab checks the job log for a line that matches the regular expression. A line + like `Code coverage: 67.89` would match. +1. GitLab then checks the line to find a match to `\d+(\.\d+)?`. The sample matching + line above gives a code coverage of `67.89`. -This same warning is displayed when: +**Additional details**: -- All other jobs in the stage are successful. -- All other jobs in the pipeline are successful. +- If there is more than one matched line in the job output, the last line is used. +- Leading zeros are removed. +- Coverage output from [child pipelines](../pipelines/parent_child_pipelines.md) + is not recorded or displayed. Check [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/280818) + for more details. -The default value for `allow_failure` is: +### `dast_configuration` **(ULTIMATE)** -- `true` for [manual jobs](../jobs/job_control.md#create-a-job-that-must-be-run-manually). -- `false` for manual jobs that also use [`rules`](#rules). -- `false` in all other cases. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5981) in GitLab 14.1. -**Keyword type**: Job keyword. You can use it only as part of a job. +Use the `dast_configuration` keyword to specify a site profile and scanner profile to be used in a +CI/CD configuration. Both profiles must first have been created in the project. The job's stage must +be `dast`. -**Possible inputs**: `true` or `false`. +**Keyword type**: Job keyword. You can use only as part of a job. -**Example of `allow_failure`**: +**Possible inputs**: One each of `site_profile` and `scanner_profile`. + +- Use `site_profile` to specify the site profile to be used in the job. +- Use `scanner_profile` to specify the scanner profile to be used in the job. + +**Example of `dast_configuration`**: ```yaml -job1: - stage: test - script: - - execute_script_1 +stages: + - build + - dast -job2: - stage: test - script: - - execute_script_2 - allow_failure: true +include: + - template: DAST.gitlab-ci.yml -job3: - stage: deploy - script: - - deploy_to_staging +dast: + dast_configuration: + site_profile: "Example Co" + scanner_profile: "Quick Passive Test" ``` -In this example, `job1` and `job2` run in parallel: - -- If `job1` fails, jobs in the `deploy` stage do not start. -- If `job2` fails, jobs in the `deploy` stage can still start. +In this example, the `dast` job extends the `dast` configuration added with the `include` keyword +to select a specific site profile and scanner profile. **Additional details**: -- You can use `allow_failure` as a subkey of [`rules:`](#rulesallow_failure). -- You can use `allow_failure: false` with a manual job to create a [blocking manual job](../jobs/job_control.md#types-of-manual-jobs). - A blocked pipeline does not run any jobs in later stages until the manual job - is started and completes successfully. - -#### `allow_failure:exit_codes` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/273157) in GitLab 13.8. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292024) in GitLab 13.9. - -Use `allow_failure:exit_codes` to control when a job should be -allowed to fail. The job is `allow_failure: true` for any of the listed exit codes, -and `allow_failure` false for any other exit code. - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: - -- A single exit code. -- An array of exit codes. +- Settings contained in either a site profile or scanner profile take precedence over those + contained in the DAST template. -**Example of `allow_failure`**: +**Related topics**: -```yaml -test_job_1: - script: - - echo "Run a script that results in exit code 1. This job fails." - - exit 1 - allow_failure: - exit_codes: 137 +- [Site profile](../../user/application_security/dast/index.md#site-profile). +- [Scanner profile](../../user/application_security/dast/index.md#scanner-profile). -test_job_2: - script: - - echo "Run a script that results in exit code 137. This job is allowed to fail." - - exit 137 - allow_failure: - exit_codes: - - 137 - - 255 -``` +### `dependencies` -### `when` +Use the `dependencies` keyword to define a list of jobs to fetch [artifacts](#artifacts) from. +You can also set a job to download no artifacts at all. -Use `when` to configure the conditions for when jobs run. If not defined in a job, -the default value is `when: on_success`. +If you do not use `dependencies`, all artifacts from previous stages are passed to each job. **Keyword type**: Job keyword. You can use it only as part of a job. **Possible inputs**: -- `on_success` (default): Run the job only when all jobs in earlier stages succeed - or have `allow_failure: true`. -- `manual`: Run the job only when [triggered manually](../jobs/job_control.md#create-a-job-that-must-be-run-manually). -- `always`: Run the job regardless of the status of jobs in earlier stages. -- `on_failure`: Run the job only when at least one job in an earlier stage fails. -- `delayed`: [Delay the execution of a job](../jobs/job_control.md#run-a-job-after-a-delay) - for a specified duration. -- `never`: Don't run the job. +- The names of jobs to fetch artifacts from. +- An empty array (`[]`), to configure the job to not download any artifacts. -**Example of `when`**: +**Example of `dependencies`**: ```yaml -stages: - - build - - cleanup_build - - test - - deploy - - cleanup +build osx: + stage: build + script: make build:osx + artifacts: + paths: + - binaries/ -build_job: +build linux: stage: build - script: - - make build + script: make build:linux + artifacts: + paths: + - binaries/ -cleanup_build_job: - stage: cleanup_build - script: - - cleanup build when failed - when: on_failure +test osx: + stage: test + script: make test:osx + dependencies: + - build:osx -test_job: +test linux: stage: test - script: - - make test + script: make test:linux + dependencies: + - build:linux -deploy_job: +deploy: stage: deploy - script: - - make deploy - when: manual - -cleanup_job: - stage: cleanup - script: - - cleanup after jobs - when: always + script: make deploy ``` -In this example, the script: +In this example, two jobs have artifacts: `build osx` and `build linux`. When `test osx` is executed, +the artifacts from `build osx` are downloaded and extracted in the context of the build. +The same thing happens for `test linux` and artifacts from `build linux`. -1. Executes `cleanup_build_job` only when `build_job` fails. -1. Always executes `cleanup_job` as the last step in pipeline regardless of - success or failure. -1. Executes `deploy_job` when you run it manually in the GitLab UI. +The `deploy` job downloads artifacts from all previous jobs because of +the [stage](#stages) precedence. **Additional details**: -- In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201938), you - can use `when:manual` in the same job as [`trigger`](#trigger). In GitLab 13.4 and - earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. -- The default behavior of `allow_failure` changes to `true` with `when: manual`. - However, if you use `when: manual` with [`rules`](#rules), `allow_failure` defaults - to `false`. - -**Related topics**: - -- `when` can be used with [`rules`](#rules) for more dynamic job control. -- `when` can be used with [`workflow`](#workflow) to control when a pipeline can start. +- The job status does not matter. If a job fails or it's a manual job that isn't triggered, no error occurs. +- If the artifacts of a dependent job are [expired](#artifactsexpire_in) or + [deleted](../pipelines/job_artifacts.md#delete-job-artifacts), then the job fails. ### `environment` @@ -1999,23 +1551,19 @@ environment. Use the `action` keyword to specify jobs that prepare, start, or stop environments. -| **Value** | **Description** | -|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `start` | Default value. Indicates that job starts the environment. The deployment is created after the job starts. | -| `prepare` | Indicates that the job is only preparing the environment. It does not trigger deployments. [Read more about preparing environments](../environments/index.md#prepare-an-environment-without-creating-a-deployment). | -| `stop` | Indicates that job stops deployment. See the example below. | +**Keyword type**: Job keyword. You can use it only as part of a job. -Take for instance: +**Possible inputs**: One of the following keywords: -```yaml -review_app: - stage: deploy - script: make deploy-app - environment: - name: review/$CI_COMMIT_REF_SLUG - url: https://$CI_ENVIRONMENT_SLUG.example.com - on_stop: stop_review_app +| **Value** | **Description** | +|:----------|:----------------| +| `start` | Default value. Indicates that the job starts the environment. The deployment is created after the job starts. | +| `prepare` | Indicates that the job is only preparing the environment. It does not trigger deployments. [Read more about preparing environments](../environments/index.md#prepare-an-environment-without-creating-a-deployment). | +| `stop` | Indicates that the job stops a deployment. For more detail, read [Stop an environment](../environments/index.md#stop-an-environment). | + +**Example of `environment:action`**: +```yaml stop_review_app: stage: deploy variables: @@ -2027,39 +1575,6 @@ stop_review_app: action: stop ``` -In the above example, the `review_app` job deploys to the `review` -environment. A new `stop_review_app` job is listed under `on_stop`. -After the `review_app` job is finished, it triggers the -`stop_review_app` job based on what is defined under `when`. In this case, -it is set to `manual`, so it needs a [manual action](../jobs/job_control.md#create-a-job-that-must-be-run-manually) from -the GitLab UI to run. - -Also in the example, `GIT_STRATEGY` is set to `none`. If the -`stop_review_app` job is [automatically triggered](../environments/index.md#stop-an-environment), -the runner won't try to check out the code after the branch is deleted. - -The example also overwrites global variables. If your `stop` `environment` job depends -on global variables, use [anchor variables](yaml_optimization.md#yaml-anchors-for-variables) when you set the `GIT_STRATEGY` -to change the job without overriding the global variables. - -The `stop_review_app` job is **required** to have the following keywords defined: - -- `when`, defined at either: - - [The job level](#when). - - [In a rules clause](#rules). If you use `rules:` and `when: manual`, you should - also set [`allow_failure: true`](#allow_failure) so the pipeline can complete - even if the job doesn't run. -- `environment:name` -- `environment:action` - -Additionally, both jobs should have matching [`rules`](#only--except) -or [`only/except`](#only--except) configuration. - -In the examples above, if the configuration is not identical: - -- The `stop_review_app` job might not be included in all pipelines that include the `review_app` job. -- It is not possible to trigger the `action: stop` to stop the environment automatically. - #### `environment:auto_stop_in` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20956) in GitLab 12.8. @@ -2185,1211 +1700,869 @@ The common use case is to create dynamic environments for branches and use them as Review Apps. You can see an example that uses Review Apps at <https://gitlab.com/gitlab-examples/review-apps-nginx/>. -### `cache` - -Use `cache` to specify a list of files and directories to -cache between jobs. You can only use paths that are in the local working copy. - -Caching is shared between pipelines and jobs. Caches are restored before [artifacts](#artifacts). - -Learn more about caches in [Caching in GitLab CI/CD](../caching/index.md). - -#### `cache:paths` - -Use the `cache:paths` keyword to choose which files or directories to cache. +### `extends` -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +Use `extends` to reuse configuration sections. It's an alternative to [YAML anchors](yaml_optimization.md#anchors) +and is a little more flexible and readable. -**Possible inputs**: An array of paths relative to the project directory (`$CI_PROJECT_DIR`). -You can use wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) -patterns: +**Keyword type**: Job keyword. You can use it only as part of a job. -- In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), -[`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). -- In GitLab Runner 12.10 and earlier, -[`filepath.Match`](https://pkg.go.dev/path/filepath#Match). +**Possible inputs**: -**Example of `cache:paths`**: +- The name of another job in the pipeline. +- A list (array) of names of other jobs in the pipeline. -Cache all files in `binaries` that end in `.apk` and the `.config` file: +**Example of `extends`**: ```yaml +.tests: + script: rake test + stage: test + only: + refs: + - branches + rspec: - script: - - echo "This job uses a cache." - cache: - key: binaries-cache - paths: - - binaries/*.apk - - .config + extends: .tests + script: rake rspec + only: + variables: + - $RSPEC ``` -**Related topics**: - -- See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more - `cache:paths` examples. - -#### `cache:key` - -Use the `cache:key` keyword to give each cache a unique identifying key. All jobs -that use the same cache key use the same cache, including in different pipelines. - -If not set, the default key is `default`. All jobs with the `cache:` keyword but -no `cache:key` share the `default` cache. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: +In this example, the `rspec` job uses the configuration from the `.tests` template job. +When creating the pipeline, GitLab: -- A string. -- A [predefined variables](../variables/index.md). -- A combination of both. +- Performs a reverse deep merge based on the keys. +- Merges the `.tests` content with the `rspec` job. +- Doesn't merge the values of the keys. -**Example of `cache:key`**: +The result is this `rspec` job: ```yaml -cache-job: - script: - - echo "This job uses a cache." - cache: - key: binaries-cache-$CI_COMMIT_REF_SLUG - paths: - - binaries/ +rspec: + script: rake rspec + stage: test + only: + refs: + - branches + variables: + - $RSPEC ``` **Additional details**: -- If you use **Windows Batch** to run your shell scripts you must replace - `$` with `%`. For example: `key: %CI_COMMIT_REF_SLUG%` -- The `cache:key` value can't contain: - - - The `/` character, or the equivalent URI-encoded `%2F`. - - Only the `.` character (any number), or the equivalent URI-encoded `%2E`. - -- The cache is shared between jobs, so if you're using different - paths for different jobs, you should also set a different `cache:key`. - Otherwise cache content can be overwritten. +- In GitLab 12.0 and later, you can use multiple parents for `extends`. +- The `extends` keyword supports up to eleven levels of inheritance, but you should + avoid using more than three levels. +- In the example above, `.tests` is a [hidden job](../jobs/index.md#hide-jobs), + but you can extend configuration from regular jobs as well. **Related topics**: -- You can specify a [fallback cache key](../caching/index.md#use-a-fallback-cache-key) - to use if the specified `cache:key` is not found. -- You can [use multiple cache keys](../caching/index.md#use-multiple-caches) in a single job. -- See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more - `cache:key` examples. - -##### `cache:key:files` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab 12.5. - -Use the `cache:key:files` keyword to generate a new key when one or two specific files -change. `cache:key:files` lets you reuse some caches, and rebuild them less often, -which speeds up subsequent pipeline runs. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: An array of one or two file paths. - -**Example of `cache:key:files`**: - -```yaml -cache-job: - script: - - echo "This job uses a cache." - cache: - key: - files: - - Gemfile.lock - - package.json - paths: - - vendor/ruby - - node_modules -``` - -This example creates a cache for Ruby and Node.js dependencies. The cache -is tied to the current versions of the `Gemfile.lock` and `package.json` files. When one of -these files changes, a new cache key is computed and a new cache is created. Any future -job runs that use the same `Gemfile.lock` and `package.json` with `cache:key:files` -use the new cache, instead of rebuilding the dependencies. - -**Additional details**: - -- The cache `key` is a SHA computed from the most recent commits -that changed each listed file. - If neither file is changed in any commits, the fallback key is `default`. - -##### `cache:key:prefix` +- [Reuse configuration sections by using `extends`](yaml_optimization.md#use-extends-to-reuse-configuration-sections). +- Use `extends` to reuse configuration from [included configuration files](yaml_optimization.md#use-extends-and-include-together). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab 12.5. +### `image` -Use `cache:key:prefix` to combine a prefix with the SHA computed for [`cache:key:files`](#cachekeyfiles). +Use `image` to specify a Docker image that the job runs in. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +[`default` section](#default). -**Possible inputs**: +**Possible inputs**: The name of the image, including the registry path if needed, in one of these formats: -- A string -- A [predefined variables](../variables/index.md) -- A combination of both. +- `<image-name>` (Same as using `<image-name>` with the `latest` tag) +- `<image-name>:<tag>` +- `<image-name>@<digest>` -**Example of `cache:key:prefix`**: +**Example of `image`**: ```yaml +default: + image: ruby:3.0 + rspec: - script: - - echo "This rspec job uses a cache." - cache: - key: - files: - - Gemfile.lock - prefix: $CI_JOB_NAME - paths: - - vendor/ruby + script: bundle exec rspec + +rspec 2.7: + image: registry.example.com/my-group/my-project/ruby:2.7 + script: bundle exec rspec ``` -For example, adding a `prefix` of `$CI_JOB_NAME` causes the key to look like `rspec-feef9576d21ee9b6a32e30c5c79d0a0ceb68d1e5`. -If a branch changes `Gemfile.lock`, that branch has a new SHA checksum for `cache:key:files`. -A new cache key is generated, and a new cache is created for that key. If `Gemfile.lock` -is not found, the prefix is added to `default`, so the key in the example would be `rspec-default`. +In this example, the `ruby:3.0` image is the default for all jobs in the pipeline. +The `rspec 2.7` job does not use the default, because it overrides the default with +a job-specific `image` section. -**Additional details**: +**Related topics**: -- If no file in `cache:key:files` is changed in any commits, the prefix is added to the `default` key. +- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). -#### `cache:untracked` +#### `image:name` -Use `untracked: true` to cache all files that are untracked in your Git repository: +The name of the Docker image that the job runs in. Similar to [`image`](#image) used by itself. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +[`default` section](#default). -**Possible inputs**: `true` or `false` (default). +**Possible inputs**: The name of the image, including the registry path if needed, in one of these formats: -**Example of `cache:untracked`**: +- `<image-name>` (Same as using `<image-name>` with the `latest` tag) +- `<image-name>:<tag>` +- `<image-name>@<digest>` + +**Example of `image:name`**: ```yaml -rspec: - script: test - cache: - untracked: true +image: + name: "registry.example.com/my/image:latest" ``` -**Additional details**: - -- You can combine `cache:untracked` with `cache:paths` to cache all untracked files - as well as files in the configured paths. For example: +**Related topics**: - ```yaml - rspec: - script: test - cache: - untracked: true - paths: - - binaries/ - ``` +- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). -#### `cache:when` +#### `image:entrypoint` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18969) in GitLab 13.5 and GitLab Runner v13.5.0. +Command or script to execute as the container's entry point. -Use `cache:when` to define when to save the cache, based on the status of the job. +When the Docker container is created, the `entrypoint` is translated to the Docker `--entrypoint` option. +The syntax is similar to the [Dockerfile `ENTRYPOINT` directive](https://docs.docker.com/engine/reference/builder/#entrypoint), +where each shell token is a separate string in the array. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +[`default` section](#default). **Possible inputs**: -- `on_success` (default): Save the cache only when the job succeeds. -- `on_failure`: Save the cache only when the job fails. -- `always`: Always save the cache. +- A string. -**Example of `cache:when`**: +**Example of `image:entrypoint`**: ```yaml -rspec: - script: rspec - cache: - paths: - - rspec/ - when: 'always' +image: + name: super/sql:experimental + entrypoint: [""] ``` -This example stores the cache whether or not the job fails or succeeds. +**Related topics**: -#### `cache:policy` +- [Override the entrypoint of an image](../docker/using_docker_images.md#override-the-entrypoint-of-an-image). -To change the upload and download behavior of a cache, use the `cache:policy` keyword. -By default, the job downloads the cache when the job starts, and uploads changes -to the cache when the job ends. This caching style is the `pull-push` policy (default). +### `inherit` -To set a job to only download the cache when the job starts, but never upload changes -when the job finishes, use `cache:policy:pull`. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207484) in GitLab 12.9. -To set a job to only upload a cache when the job finishes, but never download the -cache when the job starts, use `cache:policy:push`. +Use `inherit` to [control inheritance of globally-defined defaults and variables](../jobs/index.md#control-the-inheritance-of-default-keywords-and-global-variables). -Use the `pull` policy when you have many jobs executing in parallel that use the same cache. -This policy speeds up job execution and reduces load on the cache server. You can -use a job with the `push` policy to build the cache. +#### `inherit:default` -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +Use `inherit:default` to control the inheritance of [default keywords](#default). + +**Keyword type**: Job keyword. You can use it only as part of a job. **Possible inputs**: -- `pull` -- `push` -- `pull-push` (default) +- `true` (default) or `false` to enable or disable the inheritance of all default keywords. +- A list of specific default keywords to inherit. -**Example of `cache:policy`**: +**Example of `inherit:default`**: ```yaml -prepare-dependencies-job: - stage: build - cache: - key: gems - paths: - - vendor/bundle - policy: push - script: - - echo "This job only downloads dependencies and builds the cache." - - echo "Downloading dependencies..." +default: + retry: 2 + image: ruby:3.0 + interruptible: true -faster-test-job: - stage: test - cache: - key: gems - paths: - - vendor/bundle - policy: pull - script: - - echo "This job script uses the cache, but does not update it." - - echo "Running tests..." +job1: + script: echo "This job does not inherit any default keywords." + inherit: + default: false + +job2: + script: echo "This job inherits only the two listed default keywords. It does not inherit 'interruptible'." + inherit: + default: + - retry + - image ``` -### `dependencies` +**Additional details**: -Use the `dependencies` keyword to define a list of jobs to fetch [artifacts](#artifacts) from. -You can also set a job to download no artifacts at all. +- You can also list default keywords to inherit on one line: `default: [keyword1, keyword2]` -If you do not use `dependencies`, all artifacts from previous stages are passed to each job. +#### `inherit:variables` + +Use `inherit:variables` to control the inheritance of [global variables](#variables) keywords. **Keyword type**: Job keyword. You can use it only as part of a job. **Possible inputs**: -- The names of jobs to fetch artifacts from. -- An empty array (`[]`), to configure the job to not download any artifacts. +- `true` (default) or `false` to enable or disable the inheritance of all global variables. +- A list of specific variables to inherit. -**Example of `dependencies`**: +**Example of `inherit:variables`**: ```yaml -build osx: - stage: build - script: make build:osx - artifacts: - paths: - - binaries/ - -build linux: - stage: build - script: make build:linux - artifacts: - paths: - - binaries/ - -test osx: - stage: test - script: make test:osx - dependencies: - - build:osx +variables: + VARIABLE1: "This is variable 1" + VARIABLE2: "This is variable 2" + VARIABLE3: "This is variable 3" -test linux: - stage: test - script: make test:linux - dependencies: - - build:linux +job1: + script: echo "This job does not inherit any global variables." + inherit: + variables: false -deploy: - stage: deploy - script: make deploy +job2: + script: echo "This job inherits only the two listed global variables. It does not inherit 'VARIABLE3'." + inherit: + variables: + - VARIABLE1 + - VARIABLE2 ``` -In this example, two jobs have artifacts: `build osx` and `build linux`. When `test osx` is executed, -the artifacts from `build osx` are downloaded and extracted in the context of the build. -The same thing happens for `test linux` and artifacts from `build linux`. - -The `deploy` job downloads artifacts from all previous jobs because of -the [stage](#stages) precedence. - **Additional details**: -- The job status does not matter. If a job fails or it's a manual job that isn't triggered, no error occurs. -- If the artifacts of a dependent job are [expired](#artifactsexpire_in) or - [deleted](../pipelines/job_artifacts.md#delete-job-artifacts), then the job fails. - -### `artifacts` - -Use `artifacts` to specify a list of files and directories that are -attached to the job when it [succeeds, fails, or always](#artifactswhen). - -The artifacts are sent to GitLab after the job finishes. They are -available for download in the GitLab UI if the size is not -larger than the [maximum artifact size](../../user/gitlab_com/index.md#gitlab-cicd). +- You can also list global variables to inherit on one line: `variables: [VARIABLE1, VARIABLE2]` -By default, jobs in later stages automatically download all the artifacts created -by jobs in earlier stages. You can control artifact download behavior in jobs with -[`dependencies`](#dependencies). +### `interruptible` -When using the [`needs`](#needs) keyword, jobs can only download -artifacts from the jobs defined in the `needs` configuration. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32022) in GitLab 12.3. -Job artifacts are only collected for successful jobs by default, and -artifacts are restored after [caches](#cache). +Use `interruptible` if a job should be canceled when a newer pipeline starts before the job completes. -[Read more about artifacts](../pipelines/job_artifacts.md). +This keyword is used with the [automatic cancellation of redundant pipelines](../pipelines/settings.md#auto-cancel-redundant-pipelines) +feature. When enabled, a running job with `interruptible: true` can be cancelled when +a new pipeline starts on the same branch. -#### `artifacts:exclude` +You can't cancel subsequent jobs after a job with `interruptible: false` starts. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15122) in GitLab 13.1 -> - Requires GitLab Runner 13.1 +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -`exclude` makes it possible to prevent files from being added to an artifacts -archive. +**Possible inputs**: -Similar to [`artifacts:paths`](#artifactspaths), `exclude` paths are relative -to the project directory. You can use Wildcards that use -[glob](https://en.wikipedia.org/wiki/Glob_(programming)) or -[`doublestar.PathMatch`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#PathMatch) patterns. +- `true` or `false` (default). -For example, to store all files in `binaries/`, but not `*.o` files located in -subdirectories of `binaries/`: +**Example of `interruptible`**: ```yaml -artifacts: - paths: - - binaries/ - exclude: - - binaries/**/*.o -``` +stages: + - stage1 + - stage2 + - stage3 -Unlike [`artifacts:paths`](#artifactspaths), `exclude` paths are not recursive. To exclude all of the contents of a directory, you can match them explicitly rather than matching the directory itself. +step-1: + stage: stage1 + script: + - echo "Can be canceled." + interruptible: true -For example, to store all files in `binaries/` but nothing located in the `temp/` subdirectory: +step-2: + stage: stage2 + script: + - echo "Can not be canceled." -```yaml -artifacts: - paths: - - binaries/ - exclude: - - binaries/temp/**/* +step-3: + stage: stage3 + script: + - echo "Because step-2 can not be canceled, this step can never be canceled, even though it's set as interruptible." + interruptible: true ``` -Files matched by [`artifacts:untracked`](#artifactsuntracked) can be excluded using -`artifacts:exclude` too. +In this example, a new pipeline causes a running pipeline to be: -#### `artifacts:expire_in` +- Canceled, if only `step-1` is running or pending. +- Not canceled, after `step-2` starts. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) in GitLab 13.0 behind a disabled feature flag, the latest job artifacts are kept regardless of expiry time. -> - [Made default behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/229936) in GitLab 13.4. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241026) in GitLab 13.8, keeping latest job artifacts can be disabled at the project level. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276583) in GitLab 13.9, keeping latest job artifacts can be disabled instance-wide. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321323) in GitLab 13.12, the latest pipeline artifacts are kept regardless of expiry time. +**Additional details**: -Use `expire_in` to specify how long [job artifacts](../pipelines/job_artifacts.md) are stored before -they expire and are deleted. The `expire_in` setting does not affect: +- Only set `interruptible: true` if the job can be safely canceled after it has started, + like a build job. Deployment jobs usually shouldn't be cancelled, to prevent partial deployments. +- To completely cancel a running pipeline, all jobs must have `interruptible: true`, + or `interruptible: false` jobs must not have started. -- Artifacts from the latest job, unless keeping the latest job artifacts is: - - [Disabled at the project level](../pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). - - [Disabled instance-wide](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). -- [Pipeline artifacts](../pipelines/pipeline_artifacts.md). You can't specify an expiration date for - pipeline artifacts. See [When pipeline artifacts are deleted](../pipelines/pipeline_artifacts.md#when-pipeline-artifacts-are-deleted) - for more information. +### `needs` -The value of `expire_in` is an elapsed time in seconds, unless a unit is provided. Valid values -include: +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) in GitLab 12.2. +> - In GitLab 12.3, maximum number of jobs in `needs` array raised from five to 50. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30631) in GitLab 12.8, `needs: []` lets jobs start immediately. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) in GitLab 14.2, you can refer to jobs in the same stage as the job you are configuring. -- `'42'` -- `42 seconds` -- `3 mins 4 sec` -- `2 hrs 20 min` -- `2h20min` -- `6 mos 1 day` -- `47 yrs 6 mos and 4d` -- `3 weeks and 2 days` -- `never` +Use `needs` to execute jobs out-of-order. Relationships between jobs +that use `needs` can be visualized as a [directed acyclic graph](../directed_acyclic_graph/index.md). -To expire artifacts one week after being uploaded: +You can ignore stage ordering and run some jobs without waiting for others to complete. +Jobs in multiple stages can run concurrently. -```yaml -job: - artifacts: - expire_in: 1 week -``` +**Keyword type**: Job keyword. You can use it only as part of a job. -The expiration time period begins when the artifact is uploaded and stored on GitLab. If the expiry -time is not defined, it defaults to the -[instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration) -(30 days by default). +**Possible inputs**: -To override the expiration date and protect artifacts from being automatically deleted: +- An array of jobs. +- An empty array (`[]`), to set the job to start as soon as the pipeline is created. -- Select **Keep** on the job page. -- [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/22761), set the value of - `expire_in` to `never`. +**Example of `needs`**: -After their expiry, artifacts are deleted hourly by default (using a cron job), and are not -accessible anymore. +```yaml +linux:build: + stage: build + script: echo "Building linux..." -#### `artifacts:expose_as` +mac:build: + stage: build + script: echo "Building mac..." -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15018) in GitLab 12.5. +lint: + stage: test + needs: [] + script: echo "Linting..." -Use the `expose_as` keyword to expose [job artifacts](../pipelines/job_artifacts.md) -in the [merge request](../../user/project/merge_requests/index.md) UI. +linux:rspec: + stage: test + needs: ["linux:build"] + script: echo "Running rspec on linux..." -For example, to match a single file: +mac:rspec: + stage: test + needs: ["mac:build"] + script: echo "Running rspec on mac..." -```yaml -test: - script: ["echo 'test' > file.txt"] - artifacts: - expose_as: 'artifact 1' - paths: ['file.txt'] +production: + stage: deploy + script: echo "Running production..." ``` -With this configuration, GitLab adds a link **artifact 1** to the relevant merge request -that points to `file1.txt`. To access the link, select **View exposed artifact** -below the pipeline graph in the merge request overview. - -An example that matches an entire directory: +This example creates four paths of execution: -```yaml -test: - script: ["mkdir test && echo 'test' > test/file.txt"] - artifacts: - expose_as: 'artifact 1' - paths: ['test/'] -``` +- Linter: The `lint` job runs immediately without waiting for the `build` stage + to complete because it has no needs (`needs: []`). +- Linux path: The `linux:rspec` job runs as soon as the `linux:build` + job finishes, without waiting for `mac:build` to finish. +- macOS path: The `mac:rspec` jobs runs as soon as the `mac:build` + job finishes, without waiting for `linux:build` to finish. +- The `production` job runs as soon as all previous jobs finish: + `linux:build`, `linux:rspec`, `mac:build`, `mac:rspec`. -Note the following: +**Additional details**: -- Artifacts do not display in the merge request UI when using variables to define the `artifacts:paths`. -- A maximum of 10 job artifacts per merge request can be exposed. -- Glob patterns are unsupported. -- If a directory is specified, the link is to the job [artifacts browser](../pipelines/job_artifacts.md#download-job-artifacts) if there is more than - one file in the directory. -- For exposed single file artifacts with `.html`, `.htm`, `.txt`, `.json`, `.xml`, - and `.log` extensions, if [GitLab Pages](../../administration/pages/index.md) is: - - Enabled, GitLab automatically renders the artifact. - - Not enabled, the file is displayed in the artifacts browser. +- The maximum number of jobs that a single job can have in the `needs` array is limited: + - For GitLab.com, the limit is 50. For more information, see our + [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7541). + - For self-managed instances, the default limit is 50. This limit [can be changed](../../administration/cicd.md#set-the-needs-job-limit). +- If `needs` refers to a job that uses the [`parallel`](#parallel) keyword, + it depends on all jobs created in parallel, not just one job. It also downloads + artifacts from all the parallel jobs by default. If the artifacts have the same + name, they overwrite each other and only the last one downloaded is saved. +- In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) you + can refer to jobs in the same stage as the job you are configuring. This feature is + enabled on GitLab.com and ready for production use. On self-managed [GitLab 14.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) + this feature is available by default. +- In GitLab 14.0 and older, you can only refer to jobs in earlier stages. Stages must be + explicitly defined for all jobs that use the `needs` keyword, or are referenced + in a job's `needs` section. +- In GitLab 13.9 and older, if `needs` refers to a job that might not be added to + a pipeline because of `only`, `except`, or `rules`, the pipeline might fail to create. -#### `artifacts:name` +#### `needs:artifacts` -Use the `name` directive to define the name of the created artifacts -archive. You can specify a unique name for every archive. The `artifacts:name` -variable can make use of any of the [predefined variables](../variables/index.md). -The default name is `artifacts`, which becomes `artifacts.zip` when you download it. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.6. -To create an archive with a name of the current job: +When a job uses `needs`, it no longer downloads all artifacts from previous stages +by default, because jobs with `needs` can start before earlier stages complete. With +`needs` you can only download artifacts from the jobs listed in the `needs` configuration. -```yaml -job: - artifacts: - name: "$CI_JOB_NAME" - paths: - - binaries/ -``` +Use `artifacts: true` (default) or `artifacts: false` to control when artifacts are +downloaded in jobs that use `needs`. -To create an archive with a name of the current branch or tag including only -the binaries directory: +**Keyword type**: Job keyword. You can use it only as part of a job. Must be used with `needs:job`. -```yaml -job: - artifacts: - name: "$CI_COMMIT_REF_NAME" - paths: - - binaries/ -``` +**Possible inputs**: -If your branch-name contains forward slashes -(for example `feature/my-feature`) it's advised to use `$CI_COMMIT_REF_SLUG` -instead of `$CI_COMMIT_REF_NAME` for proper naming of the artifact. +- `true` (default) or `false`. -To create an archive with a name of the current job and the current branch or -tag including only the binaries directory: +**Example of `needs:artifacts`**: ```yaml -job: - artifacts: - name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME" - paths: - - binaries/ -``` +test-job1: + stage: test + needs: + - job: build_job1 + artifacts: true -To create an archive with a name of the current [stage](#stages) and branch name: +test-job2: + stage: test + needs: + - job: build_job2 + artifacts: false -```yaml -job: - artifacts: - name: "$CI_JOB_STAGE-$CI_COMMIT_REF_NAME" - paths: - - binaries/ +test-job3: + needs: + - job: build_job1 + artifacts: true + - job: build_job2 + - build_job3 ``` ---- - -If you use **Windows Batch** to run your shell scripts you must replace -`$` with `%`: - -```yaml -job: - artifacts: - name: "%CI_JOB_STAGE%-%CI_COMMIT_REF_NAME%" - paths: - - binaries/ -``` +In this example: -If you use **Windows PowerShell** to run your shell scripts you must replace -`$` with `$env:`: +- The `test-job1` job downloads the `build_job1` artifacts +- The `test-job2` job does not download the `build_job2` artifacts. +- The `test-job3` job downloads the artifacts from all three `build_jobs`, because + `artifacts` is `true`, or defaults to `true`, for all three needed jobs. -```yaml -job: - artifacts: - name: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_NAME" - paths: - - binaries/ -``` +**Additional details**: -#### `artifacts:paths` +- In GitLab 12.6 and later, you can't combine the [`dependencies`](#dependencies) keyword + with `needs`. -Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly -link outside it. You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) -patterns and: +#### `needs:project` **(PREMIUM)** -- In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), - [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). -- In GitLab Runner 12.10 and earlier, [`filepath.Match`](https://pkg.go.dev/path/filepath#Match). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.7. -To restrict which jobs a specific job fetches artifacts from, see [dependencies](#dependencies). +Use `needs:project` to download artifacts from up to five jobs in other pipelines. +The artifacts are downloaded from the latest successful pipeline for the specified ref. -Send all files in `binaries` and `.config`: +If there is a pipeline running for the specified ref, a job with `needs:project` +does not wait for the pipeline to complete. Instead, the job downloads the artifact +from the latest pipeline that completed successfully. -```yaml -artifacts: - paths: - - binaries/ - - .config -``` +`needs:project` must be used with `job`, `ref`, and `artifacts`. -To disable artifact passing, define the job with empty [dependencies](#dependencies): +**Keyword type**: Job keyword. You can use it only as part of a job. -```yaml -job: - stage: build - script: make build - dependencies: [] -``` +**Possible inputs**: -You may want to create artifacts only for tagged releases to avoid filling the -build server storage with temporary build artifacts. +- `needs:project`: A full project path, including namespace and group. +- `job`: The job to download artifacts from. +- `ref`: The ref to download artifacts from. +- `artifacts`: Must be `true` to download artifacts. -Create artifacts only for tags (`default-job` doesn't create artifacts): +**Examples of `needs:project`**: ```yaml -default-job: - script: - - mvn test -U - rules: - - if: $CI_COMMIT_BRANCH - -release-job: +build_job: + stage: build script: - - mvn package -U - artifacts: - paths: - - target/*.war - rules: - - if: $CI_COMMIT_TAG -``` - -You can use wildcards for directories too. For example, if you want to get all the files inside the directories that end with `xyz`: - -```yaml -job: - artifacts: - paths: - - path/*xyz/* + - ls -lhR + needs: + - project: namespace/group/project-name + job: build-1 + ref: main + artifacts: true ``` -#### `artifacts:public` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 -> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's recommended for production use. - -Use `artifacts:public` to determine whether the job artifacts should be -publicly available. - -The default for `artifacts:public` is `true` which means that the artifacts in -public pipelines are available for download by anonymous and guest users: - -```yaml -artifacts: - public: true -``` +In this example, `build_job` downloads the artifacts from the latest successful `build-1` job +on the `main` branch in the `group/project-name` project. -To deny read access for anonymous and guest users to artifacts in public -pipelines, set `artifacts:public` to `false`: +In GitLab 13.3 and later, you can use [CI/CD variables](../variables/index.md) in `needs:project`, +for example: ```yaml -artifacts: - public: false +build_job: + stage: build + script: + - ls -lhR + needs: + - project: $CI_PROJECT_PATH + job: $DEPENDENCY_JOB_NAME + ref: $ARTIFACTS_DOWNLOAD_REF + artifacts: true ``` -#### `artifacts:reports` - -Use [`artifacts:reports`](#artifactsreports) to: - -- Collect test reports, code quality reports, security reports, and other artifacts generated by included templates in - jobs. -- Some of these reports are used to display information in: - - Merge requests. - - Pipeline views. - - [Security dashboards](../../user/application_security/security_dashboard/index.md). - -The test reports are collected regardless of the job results (success or failure). -You can use [`artifacts:expire_in`](#artifactsexpire_in) to set up an expiration -date for their artifacts. - -Some `artifacts:reports` types can be generated by multiple jobs in the same pipeline, and used by merge request or -pipeline features from each job. - -To be able to browse the report output files, include the [`artifacts:paths`](#artifactspaths) keyword. - -NOTE: -Combined reports in parent pipelines using [artifacts from child pipelines](#needspipelinejob) is -not supported. Track progress on adding support in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215725). - -##### `artifacts:reports:accessibility` **(FREE)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39425) in GitLab 12.8. - -The `accessibility` report uses [pa11y](https://pa11y.org/) to report on the accessibility impact -of changes introduced in merge requests. - -GitLab can display the results of one or more reports in the merge request -[accessibility widget](../../user/project/merge_requests/accessibility_testing.md#accessibility-merge-request-widget). - -For more information, see [Accessibility testing](../../user/project/merge_requests/accessibility_testing.md). - -##### `artifacts:reports:api_fuzzing` **(ULTIMATE)** - -> - Introduced in GitLab 13.4. -> - Requires GitLab Runner 13.4 or later. - -The `api_fuzzing` report collects [API Fuzzing bugs](../../user/application_security/api_fuzzing/index.md) -as artifacts. - -GitLab can display the results of one or more reports in: - -- The merge request [security widget](../../user/application_security/api_fuzzing/index.md#view-details-of-an-api-fuzzing-vulnerability). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). -- The [security dashboard](../../user/application_security/api_fuzzing/index.md#security-dashboard). - -##### `artifacts:reports:browser_performance` **(PREMIUM)** - -> [Name changed](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) from `artifacts:reports:performance` in GitLab 14.0. - -The `browser_performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) -as artifacts. - -GitLab can display the results of one report in the merge request -[browser performance testing widget](../../user/project/merge_requests/browser_performance_testing.md#how-browser-performance-testing-works). - -GitLab cannot display the combined results of multiple `browser_performance` reports. - -##### `artifacts:reports:cluster_image_scanning` **(ULTIMATE)** - -> - Introduced in GitLab 14.1. -> - Requires GitLab Runner 14.1 and above. - -The `cluster_image_scanning` report collects `CLUSTER_IMAGE_SCANNING` vulnerabilities. The collected -`CLUSTER_IMAGE_SCANNING` report uploads to GitLab as an artifact. - -GitLab can display the results of one or more reports in: - -- The [security dashboard](../../user/application_security/security_dashboard/index.md). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). - -##### `artifacts:reports:cobertura` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. - -The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). -The collected Cobertura coverage reports upload to GitLab as an artifact. - -GitLab can display the results of one or more reports in the merge request -[diff annotations](../../user/project/merge_requests/test_coverage_visualization.md). - -Cobertura was originally developed for Java, but there are many third-party ports for other languages such as -JavaScript, Python, and Ruby. - -##### `artifacts:reports:codequality` - -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. - -The `codequality` report collects [code quality issues](../../user/project/merge_requests/code_quality.md). The -collected code quality report uploads to GitLab as an artifact. +**Additional details**: -GitLab can display the results of: +- To download artifacts from a different pipeline in the current project, set `project` + to be the same as the current project, but use a different ref than the current pipeline. + Concurrent pipelines running on the same ref could override the artifacts. +- The user running the pipeline must have at least the Reporter role for the group or project, + or the group/project must have public visibility. +- You can't use `needs:project` in the same job as [`trigger`](#trigger). +- When using `needs:project` to download artifacts from another pipeline, the job does not wait for + the needed job to complete. [Directed acyclic graph](../directed_acyclic_graph/index.md) + behavior is limited to jobs in the same pipeline. Make sure that the needed job in the other + pipeline completes before the job that needs it tries to download the artifacts. +- You can't download artifacts from jobs that run in [`parallel`](#parallel). +- Support for [CI/CD variables](../variables/index.md) in `project`, `job`, and `ref` was + [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202093) in GitLab 13.3. + [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235761) in GitLab 13.4. -- One or more reports in the merge request [code quality widget](../../user/project/merge_requests/code_quality.md#code-quality-widget). -- Only one report in: - - The merge request [diff annotations](../../user/project/merge_requests/code_quality.md#code-quality-in-diff-view). - Track progress on adding support for multiple reports in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328257). - - The [full report](../metrics_reports.md). Track progress on adding support for multiple reports in - [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/9014). +**Related topics**: -##### `artifacts:reports:container_scanning` **(ULTIMATE)** +- To download artifacts between [parent-child pipelines](../pipelines/parent_child_pipelines.md), + use [`needs:pipeline:job`](#needspipelinejob). -The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md). -The collected Container Scanning report uploads to GitLab as an artifact. +#### `needs:pipeline:job` -GitLab can display the results of one or more reports in: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/255983) in GitLab 13.7. -- The merge request [container scanning widget](../../user/application_security/container_scanning/index.md). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +A [child pipeline](../pipelines/parent_child_pipelines.md) can download artifacts from a job in +its parent pipeline or another child pipeline in the same parent-child pipeline hierarchy. -##### `artifacts:reports:coverage_fuzzing` **(ULTIMATE)** +**Keyword type**: Job keyword. You can use it only as part of a job. -> - Introduced in GitLab 13.4. -> - Requires GitLab Runner 13.4 or later. +**Possible inputs**: -The `coverage_fuzzing` report collects [coverage fuzzing bugs](../../user/application_security/coverage_fuzzing/index.md). -The collected coverage fuzzing report uploads to GitLab as an artifact. -GitLab can display the results of one or more reports in: +- `needs:pipeline`: A pipeline ID. Must be a pipeline present in the same parent-child pipeline hierarchy. +- `job`: The job to download artifacts from. -- The merge request [coverage fuzzing widget](../../user/application_security/coverage_fuzzing/index.md#interacting-with-the-vulnerabilities). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). +**Example of `needs:pipeline:job`**: -##### `artifacts:reports:dast` **(ULTIMATE)** +- Parent pipeline (`.gitlab-ci.yml`): -The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md). The collected DAST -report uploads to GitLab as an artifact. + ```yaml + create-artifact: + stage: build + script: echo "sample artifact" > artifact.txt + artifacts: + paths: [artifact.txt] -GitLab can display the results of one or more reports in: + child-pipeline: + stage: test + trigger: + include: child.yml + strategy: depend + variables: + PARENT_PIPELINE_ID: $CI_PIPELINE_ID + ``` -- The merge request [security widget](../../user/application_security/dast/index.md#view-details-of-a-vulnerability-detected-by-dast). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- Child pipeline (`child.yml`): -##### `artifacts:reports:dependency_scanning` **(ULTIMATE)** + ```yaml + use-artifact: + script: cat artifact.txt + needs: + - pipeline: $PARENT_PIPELINE_ID + job: create-artifact + ``` -The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md). -The collected Dependency Scanning report uploads to GitLab as an artifact. +In this example, the `create-artifact` job in the parent pipeline creates some artifacts. +The `child-pipeline` job triggers a child pipeline, and passes the `CI_PIPELINE_ID` +variable to the child pipeline as a new `PARENT_PIPELINE_ID` variable. The child pipeline +can use that variable in `needs:pipeline` to download artifacts from the parent pipeline. -GitLab can display the results of one or more reports in: +**Additional details**: -- The merge request [dependency scanning widget](../../user/application_security/dependency_scanning/index.md#overview). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). -- The [dependency list](../../user/application_security/dependency_list/). +- The `pipeline` attribute does not accept the current pipeline ID (`$CI_PIPELINE_ID`). + To download artifacts from a job in the current pipeline, use [`needs`](#needsartifacts). -##### `artifacts:reports:dotenv` +#### `needs:optional` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17066) in GitLab 12.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30680) in GitLab 13.10. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 14.0. -The `dotenv` report collects a set of environment variables as artifacts. +To need a job that sometimes does not exist in the pipeline, add `optional: true` +to the `needs` configuration. If not defined, `optional: false` is the default. -The collected variables are registered as runtime-created variables of the job, -which is useful to [set dynamic environment URLs after a job finishes](../environments/index.md#set-dynamic-environment-urls-after-a-job-finishes). +Jobs that use [`rules`](#rules), [`only`, or `except`](#only--except), might +not always exist in a pipeline. When the pipeline is created, GitLab checks the `needs` +relationships before starting it. Without `optional: true`, needs relationships that +point to a job that does not exist stops the pipeline from starting and causes a pipeline +error similar to: -The exceptions to the [original dotenv rules](https://github.com/motdotla/dotenv#rules) are: +- `'job1' job needs 'job2' job, but it was not added to the pipeline` -- The variable key can contain only letters, digits, and underscores (`_`). -- The maximum size of the `.env` file is 5 KB. -- In GitLab 13.5 and older, the maximum number of inherited variables is 10. -- In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/247913), - the maximum number of inherited variables is 20. -- Variable substitution in the `.env` file is not supported. -- The `.env` file can't have empty lines or comments (starting with `#`). -- Key values in the `env` file cannot have spaces or newline characters (`\n`), including when using single or double quotes. -- Quote escaping during parsing (`key = 'value'` -> `{key: "value"}`) is not supported. +**Keyword type**: Job keyword. You can use it only as part of a job. -##### `artifacts:reports:junit` +**Possible inputs**: -The `junit` report collects [JUnit report format XML files](https://www.ibm.com/docs/en/adfz/developer-for-zos/14.1.0?topic=formats-junit-xml-format). -The collected Unit test reports upload to GitLab as an artifact. Although JUnit was originally developed in Java, there -are many third-party ports for other languages such as JavaScript, Python, and Ruby. +- `job`: The job to make optional. +- `true` or `false` (default). -See [Unit test reports](../unit_test_reports.md) for more details and examples. -Below is an example of collecting a JUnit report format XML file from Ruby's RSpec test tool: +**Example of `needs:optional`**: ```yaml +build: + stage: build + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + rspec: stage: test - script: - - bundle install - - rspec --format RspecJunitFormatter --out rspec.xml - artifacts: - reports: - junit: rspec.xml + needs: + - job: build + optional: true ``` -GitLab can display the results of one or more reports in: - -- The merge request [code quality widget](../../ci/unit_test_reports.md#how-it-works). -- The [full report](../../ci/unit_test_reports.md#viewing-unit-test-reports-on-gitlab). - -Some JUnit tools export to multiple XML files. You can specify multiple test report paths in a single job to -concatenate them into a single file. Use either: - -- A filename pattern (`junit: rspec-*.xml`). -- an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`). -- A Combination of both (`junit: [rspec.xml, test-results/TEST-*.xml]`). - -##### `artifacts:reports:license_scanning` **(ULTIMATE)** - -> Introduced in GitLab 12.8. - -The License Compliance report collects [Licenses](../../user/compliance/license_compliance/index.md). The License -Compliance report uploads to GitLab as an artifact. - -GitLab can display the results of one or more reports in: - -- The merge request [license compliance widget](../../user/compliance/license_compliance/index.md). -- The [license list](../../user/compliance/license_compliance/index.md#license-list). - -##### `artifacts:reports:load_performance` **(PREMIUM)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35260) in GitLab 13.2. -> - Requires GitLab Runner 11.5 and above. - -The `load_performance` report collects [Load Performance Testing metrics](../../user/project/merge_requests/load_performance_testing.md). -The report is uploaded to GitLab as an artifact. - -GitLab can display the results of only one report in the merge request -[load testing widget](../../user/project/merge_requests/load_performance_testing.md#how-load-performance-testing-works). - -GitLab cannot display the combined results of multiple `load_performance` reports. - -##### `artifacts:reports:metrics` **(PREMIUM)** +In this example: -The `metrics` report collects [Metrics](../metrics_reports.md). The collected Metrics report uploads to GitLab as an -artifact. +- When the branch is the default branch, the `build` job exists in the pipeline, and the `rspec` + job waits for it to complete before starting. +- When the branch is not the default branch, the `build` job does not exist in the pipeline. + The `rspec` job runs immediately (similar to `needs: []`) because its `needs` + relationship to the `build` job is optional. -GitLab can display the results of one or more reports in the merge request -[metrics reports widget](../../ci/metrics_reports.md#metrics-reports). +#### `needs:pipeline` -##### `artifacts:reports:requirements` **(ULTIMATE)** +You can mirror the pipeline status from an upstream pipeline to a bridge job by +using the `needs:pipeline` keyword. The latest pipeline status from the default branch is +replicated to the bridge job. -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. +**Keyword type**: Job keyword. You can use it only as part of a job. -The `requirements` report collects `requirements.json` files. The collected Requirements report uploads to GitLab as an -artifact and existing [requirements](../../user/project/requirements/index.md) are marked as Satisfied. +**Possible inputs**: -GitLab can display the results of one or more reports in the -[project requirements](../../user/project/requirements/index.md#view-a-requirement). +- A full project path, including namespace and group. If the + project is in the same group or namespace, you can omit them from the `project` + keyword. For example: `project: group/project-name` or `project: project-name`. -##### `artifacts:reports:sast` +**Example of `needs:pipeline`**: -> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) from GitLab Ultimate to GitLab Free in 13.3. +```yaml +upstream_bridge: + stage: test + needs: + pipeline: other/project +``` -The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md). The collected SAST -report uploads to GitLab as an artifact. +**Additional details**: -GitLab can display the results of one or more reports in: +- If you add the `job` keyword to `needs:pipeline`, the job no longer mirrors the + pipeline status. The behavior changes to [`needs:pipeline:job`](#needspipelinejob). -- The merge request [SAST widget](../../user/application_security/sast/index.md#static-application-security-testing-sast). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). +### `only` / `except` -##### `artifacts:reports:secret_detection` +NOTE: +`only` and `except` are not being actively developed. [`rules`](#rules) is the preferred +keyword to control when to add jobs to pipelines. -> - Introduced in GitLab 13.1. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/222788) to GitLab Free in 13.3. -> - Requires GitLab Runner 11.5 and above. +You can use `only` and `except` to control when to add jobs to pipelines. -The `secret-detection` report collects [detected secrets](../../user/application_security/secret_detection/index.md). -The collected Secret Detection report is uploaded to GitLab. +- Use `only` to define when a job runs. +- Use `except` to define when a job **does not** run. -GitLab can display the results of one or more reports in: +Four keywords can be used with `only` and `except`: -- The merge request [secret scanning widget](../../user/application_security/secret_detection/index.md). -- The [pipeline **Security** tab](../../user/application_security/index.md#view-security-scan-information-in-the-pipeline-security-tab). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- [`refs`](#onlyrefs--exceptrefs) +- [`variables`](#onlyvariables--exceptvariables) +- [`changes`](#onlychanges--exceptchanges) +- [`kubernetes`](#onlykubernetes--exceptkubernetes) -##### `artifacts:reports:terraform` +See [specify when jobs run with `only` and `except`](../jobs/job_control.md#specify-when-jobs-run-with-only-and-except) +for more details and examples. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. -> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. +#### `only:refs` / `except:refs` -The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/iac/mr_integration.md#configure-terraform-report-artifacts). -The collected Terraform plan report uploads to GitLab as an artifact. +Use the `only:refs` and `except:refs` keywords to control when to add jobs to a +pipeline based on branch names or pipeline types. -GitLab can display the results of one or more reports in the merge request -[terraform widget](../../user/infrastructure/iac/mr_integration.md#output-terraform-plan-information-into-a-merge-request). +**Keyword type**: Job keyword. You can use it only as part of a job. -For more information, see [Output `terraform plan` information into a merge request](../../user/infrastructure/iac/mr_integration.md). +**Possible inputs**: An array including any number of: -#### `artifacts:untracked` +- Branch names, for example `main` or `my-feature-branch`. +- [Regular expressions](../jobs/job_control.md#only--except-regex-syntax) + that match against branch names, for example `/^feature-.*/`. +- The following keywords: -Use `artifacts:untracked` to add all Git untracked files as artifacts (along -with the paths defined in `artifacts:paths`). `artifacts:untracked` ignores configuration -in the repository's `.gitignore` file. + | **Value** | **Description** | + | -------------------------|-----------------| + | `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | + | `branches` | When the Git reference for a pipeline is a branch. | + | `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | + | `external` | When you use CI services other than GitLab. | + | `external_pull_requests` | When an external pull request on GitHub is created or updated (See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests)). | + | `merge_requests` | For pipelines created when a merge request is created or updated. Enables [merge request pipelines](../pipelines/merge_request_pipelines.md), [merged results pipelines](../pipelines/pipelines_for_merged_results.md), and [merge trains](../pipelines/merge_trains.md). | + | `pipelines` | For [multi-project pipelines](../pipelines/multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api), or the [`trigger`](#trigger) keyword. | + | `pushes` | For pipelines triggered by a `git push` event, including for branches and tags. | + | `schedules` | For [scheduled pipelines](../pipelines/schedules.md). | + | `tags` | When the Git reference for a pipeline is a tag. | + | `triggers` | For pipelines created by using a [trigger token](../triggers/index.md#configure-cicd-jobs-to-run-in-triggered-pipelines). | + | `web` | For pipelines created by selecting **Run pipeline** in the GitLab UI, from the project's **CI/CD > Pipelines** section. | -Send all Git untracked files: +**Example of `only:refs` and `except:refs`**: ```yaml -artifacts: - untracked: true -``` - -Send all Git untracked files and files in `binaries`: +job1: + script: echo + only: + - main + - /^issue-.*$/ + - merge_requests -```yaml -artifacts: - untracked: true - paths: - - binaries/ +job2: + script: echo + except: + - main + - /^stable-branch.*$/ + - schedules ``` -Send all untracked files but [exclude](#artifactsexclude) `*.txt`: +**Additional details**: -```yaml -artifacts: - untracked: true - exclude: - - "*.txt" -``` +- Scheduled pipelines run on specific branches, so jobs configured with `only: branches` + run on scheduled pipelines too. Add `except: schedules` to prevent jobs with `only: branches` + from running on scheduled pipelines. +- `only` or `except` used without any other keywords are equivalent to `only: refs` + or `except: refs`. For example, the following two jobs configurations have the same + behavior: -#### `artifacts:when` + ```yaml + job1: + script: echo + only: + - branches -Use `artifacts:when` to upload artifacts on job failure or despite the -failure. + job2: + script: echo + only: + refs: + - branches + ``` -`artifacts:when` can be set to one of the following values: +- If a job does not use `only`, `except`, or [`rules`](#rules), then `only` is set to `branches` + and `tags` by default. -1. `on_success` (default): Upload artifacts only when the job succeeds. -1. `on_failure`: Upload artifacts only when the job fails. -1. `always`: Always upload artifacts. For example, when - [uploading artifacts](../unit_test_reports.md#viewing-junit-screenshots-on-gitlab) required to - troubleshoot failing tests. + For example, `job1` and `job2` are equivalent: -For example, to upload artifacts only when a job fails: + ```yaml + job1: + script: echo "test" -```yaml -job: - artifacts: - when: on_failure -``` + job2: + script: echo "test" + only: + - branches + - tags + ``` -### `coverage` +#### `only:variables` / `except:variables` -Use `coverage` with a custom regular expression to configure how code coverage -is extracted from the job output. The coverage is shown in the UI if at least one -line in the job output matches the regular expression. +Use the `only:variables` or `except:variables` keywords to control when to add jobs +to a pipeline, based on the status of [CI/CD variables](../variables/index.md). -To extract the code coverage value in the matching line, GitLab uses this -regular expression: `\d+(\.\d+)?`. +**Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: A regular expression. Must start and end with `/`. +**Possible inputs**: -**Example of `coverage`**: +- An array of [CI/CD variable expressions](../jobs/job_control.md#cicd-variable-expressions). + +**Example of `only:variables`**: ```yaml -job1: - script: rspec - coverage: '/Code coverage: \d+\.\d+/' +deploy: + script: cap staging deploy + only: + variables: + - $RELEASE == "staging" + - $STAGING ``` -In this example: - -1. GitLab checks the job log for a line that matches the regular expression. A line - like `Code coverage: 67.89` would match. -1. GitLab then checks the line to find a match to `\d+(\.\d+)?`. The sample matching - line above gives a code coverage of `67.89`. +**Related topics**: -**Additional details**: +- [`only:variables` and `except:variables` examples](../jobs/job_control.md#only-variables--except-variables-examples). -- If there is more than one matched line in the job output, the last line is used. -- Leading zeros are removed. -- Coverage output from [child pipelines](../pipelines/parent_child_pipelines.md) - is not recorded or displayed. Check [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/280818) - for more details. +#### `only:changes` / `except:changes` -### `dast_configuration` **(ULTIMATE)** +Use the `changes` keyword with `only` to run a job, or with `except` to skip a job, +when a Git push event modifies a file. -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5981) in GitLab 14.1. +Use `changes` in pipelines with the following refs: -Use the `dast_configuration` keyword to specify a site profile and scanner profile to be used in a -CI/CD configuration. Both profiles must first have been created in the project. The job's stage must -be `dast`. +- `branches` +- `external_pull_requests` +- `merge_requests` (see additional details about [using `only:changes` with pipelines for merge requests](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests)) -**Keyword type**: Job keyword. You can use only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: One each of `site_profile` and `scanner_profile`. +**Possible inputs**: An array including any number of: -- Use `site_profile` to specify the site profile to be used in the job. -- Use `scanner_profile` to specify the scanner profile to be used in the job. +- Paths to files. +- Wildcard paths for single directories, for example `path/to/directory/*`, or a directory + and all its subdirectories, for example `path/to/directory/**/*`. +- Wildcard ([glob](https://en.wikipedia.org/wiki/Glob_(programming))) paths for all + files with the same extension or multiple extensions, for example `*.md` or `path/to/directory/*.{rb,py,sh}`. +- Wildcard paths to files in the root directory, or all directories, wrapped in double quotes. + For example `"*.json"` or `"**/*.json"`. -**Example of `dast_configuration`**: +**Example of `only:changes`**: ```yaml -stages: - - build - - dast - -include: - - template: DAST.gitlab-ci.yml - -dast: - dast_configuration: - site_profile: "Example Co" - scanner_profile: "Quick Passive Test" +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + only: + refs: + - branches + changes: + - Dockerfile + - docker/scripts/* + - dockerfiles/**/* + - more_scripts/*.{rb,py,sh} + - "**/*.json" ``` -In this example, the `dast` job extends the `dast` configuration added with the `include:` keyword -to select a specific site profile and scanner profile. - **Additional details**: -- Settings contained in either a site profile or scanner profile take precedence over those - contained in the DAST template. +- `changes` resolves to `true` if any of the matching files are changed (an `OR` operation). +- If you use refs other than `branches`, `external_pull_requests`, or `merge_requests`, + `changes` can't determine if a given file is new or old and always returns `true`. +- If you use `only: changes` with other refs, jobs ignore the changes and always run. +- If you use `except: changes` with other refs, jobs ignore the changes and never run. **Related topics**: -- [Site profile](../../user/application_security/dast/index.md#site-profile). -- [Scanner profile](../../user/application_security/dast/index.md#scanner-profile). - -### `retry` +- [`only: changes` and `except: changes` examples](../jobs/job_control.md#onlychanges--exceptchanges-examples). +- If you use `changes` with [only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds), + you should [also use `only:merge_requests`](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests). +- Use `changes` with [new branches or tags *without* pipelines for merge requests](../jobs/job_control.md#use-onlychanges-without-pipelines-for-merge-requests). +- Use `changes` with [scheduled pipelines](../jobs/job_control.md#use-onlychanges-with-scheduled-pipelines). -Use `retry` to configure how many times a job is retried if it fails. -If not defined, defaults to `0` and jobs do not retry. +#### `only:kubernetes` / `except:kubernetes` -When a job fails, the job is processed up to two more times, until it succeeds or -reaches the maximum number of retries. +Use `only:kubernetes` or `except:kubernetes` to control if jobs are added to the pipeline +when the Kubernetes service is active in the project. -By default, all failure types cause the job to be retried. Use [`retry:when`](#retrywhen) -to select which failures to retry on. +**Keyword type**: Job-specific. You can use it only as part of a job. -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +**Possible inputs**: -**Possible inputs**: `0` (default), `1`, or `2`. +- The `kubernetes` strategy accepts only the `active` keyword. -**Example of `retry`**: +**Example of `only:kubernetes`**: ```yaml -test: - script: rspec - retry: 2 +deploy: + only: + kubernetes: active ``` -#### `retry:when` - -Use `retry:when` with `retry:max` to retry jobs for only specific failure cases. -`retry:max` is the maximum number of retries, like [`retry`](#retry), and can be -`0`, `1`, or `2`. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: A single failure type, or an array of one or more failure types: - -<!-- - If you change any of the values below, make sure to update the `RETRY_WHEN_IN_DOCUMENTATION` - array in `spec/lib/gitlab/ci/config/entry/retry_spec.rb`. - The test there makes sure that all documented - values are valid as a configuration option and therefore should always - stay in sync with this documentation. ---> - -- `always`: Retry on any failure (default). -- `unknown_failure`: Retry when the failure reason is unknown. -- `script_failure`: Retry when the script failed. -- `api_failure`: Retry on API failure. -- `stuck_or_timeout_failure`: Retry when the job got stuck or timed out. -- `runner_system_failure`: Retry if there is a runner system failure (for example, job setup failed). -- `runner_unsupported`: Retry if the runner is unsupported. -- `stale_schedule`: Retry if a delayed job could not be executed. -- `job_execution_timeout`: Retry if the script exceeded the maximum execution time set for the job. -- `archived_failure`: Retry if the job is archived and can't be run. -- `unmet_prerequisites`: Retry if the job failed to complete prerequisite tasks. -- `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner. -- `data_integrity_failure`: Retry if there is a structural integrity problem detected. +In this example, the `deploy` job runs only when the Kubernetes service is active +in the project. -**Example of `retry:when`** (single failure type): +### `pages` -```yaml -test: - script: rspec - retry: - max: 2 - when: runner_system_failure -``` +Use `pages` to define a [GitLab Pages](../../user/project/pages/index.md) job that +uploads static content to GitLab. The content is then published as a website. -If there is a failure other than a runner system failure, the job is not retried. +**Keyword type**: Job name. -**Example of `retry:when`** (array of failure types): +**Example of `pages`**: ```yaml -test: - script: rspec - retry: - max: 2 - when: - - runner_system_failure - - stuck_or_timeout_failure +pages: + stage: deploy + script: + - mkdir .public + - cp -r * .public + - mv .public public + artifacts: + paths: + - public + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` -**Related topics**: - -You can specify the number of [retry attempts for certain stages of job execution](../runners/configure_runners.md#job-stages-attempts) -using variables. - -### `timeout` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14887) in GitLab 12.3. - -Use `timeout` to configure a timeout for a specific job. If the job runs for longer -than the timeout, the job fails. - -The job-level timeout can be longer than the [project-level timeout](../pipelines/settings.md#set-a-limit-for-how-long-jobs-can-run). -but can't be longer than the [runner's timeout](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: A period of time written in natural language. For example, these are all equivalent: - -- `3600 seconds` -- `60 minutes` -- `one hour` +This example moves all files from the root of the project to the `public/` directory. +The `.public` workaround is so `cp` does not also copy `public/` to itself in an infinite loop. -**Example of `timeout`**: +**Additional details**: -```yaml -build: - script: build.sh - timeout: 3 hours 30 minutes +You must: -test: - script: rspec - timeout: 3h 30m -``` +- Place any static content in a `public/` directory. +- Define [`artifacts`](#artifacts) with a path to the `public/` directory. ### `parallel` @@ -3401,7 +2574,9 @@ Parallel jobs are named sequentially from `job_name 1/N` to `job_name N/N`. **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: A numeric value from `2` to `50`. +**Possible inputs**: + +- A numeric value from `2` to `50`. **Example of `parallel`**: @@ -3434,7 +2609,9 @@ Multiple runners must exist, or a single runner must be configured to run multip **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: A numeric value from `2` to `50`. +**Possible inputs**: + +- A numeric value from `2` to `50`. **Example of `parallel:matrix`**: @@ -3477,175 +2654,6 @@ deploystacks: [vultr, processing] - [Run a one-dimensional matrix of parallel jobs](../jobs/job_control.md#run-a-one-dimensional-matrix-of-parallel-jobs). - [Run a matrix of triggered parallel jobs](../jobs/job_control.md#run-a-matrix-of-parallel-trigger-jobs). -### `trigger` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in GitLab Premium 11.8. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. - -Use `trigger` to start a downstream pipeline that is either: - -- [A multi-project pipeline](../pipelines/multi_project_pipelines.md). -- [A child pipeline](../pipelines/parent_child_pipelines.md). - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: - -- For multi-project pipelines, path to the downstream project. -- For child pipelines, path to the child pipeline CI/CD configuration file. - -**Example of `trigger` for multi-project pipeline**: - -```yaml -rspec: - stage: test - script: bundle exec rspec - -staging: - stage: deploy - trigger: my/deployment -``` - -**Example of `trigger` for child pipelines**: - -```yaml -trigger_job: - trigger: - include: path/to/child-pipeline.yml -``` - -**Additional details**: - -- Jobs with `trigger` can only use a [limited set of keywords](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). - For example, you can't run commands with [`script`](#script), [`before_script`](#before_script), - or [`after_script`](#after_script). -- In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201938), you - can use [`when:manual`](#when) in the same job as `trigger`. In GitLab 13.4 and - earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. -- In [GitLab 13.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/197140/), you can - view which job triggered a downstream pipeline in the [pipeline graph](../pipelines/index.md#visualize-pipelines). - -**Related topics**: - -- [Multi-project pipeline configuration examples](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). -- [Child pipeline configuration examples](../pipelines/parent_child_pipelines.md#examples). -- To force a rebuild of a specific branch, tag, or commit, you can - [use an API call with a trigger token](../triggers/index.md). - The trigger token is different than the `trigger` keyword. - -#### `trigger:strategy` - -Use `trigger:strategy` to force the `trigger` job to wait for the downstream pipeline to complete -before it is marked as **success**. - -This behavior is different than the default, which is for the `trigger` job to be marked as -**success** as soon as the downstream pipeline is created. - -This setting makes your pipeline execution linear rather than parallel. - -**Example of `trigger:strategy`**: - -```yaml -trigger_job: - trigger: - include: path/to/child-pipeline.yml - strategy: depend -``` - -In this example, jobs from subsequent stages wait for the triggered pipeline to -successfully complete before starting. - -### `interruptible` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32022) in GitLab 12.3. - -Use `interruptible` if a job should be canceled when a newer pipeline starts before the job completes. - -This keyword is used with the [automatic cancellation of redundant pipelines](../pipelines/settings.md#auto-cancel-redundant-pipelines) -feature. When enabled, a running job with `interruptible: true` can be cancelled when -a new pipeline starts on the same branch. - -You can't cancel subsequent jobs after a job with `interruptible: false` starts. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: `true` or `false` (default). - -**Example of `interruptible`**: - -```yaml -stages: - - stage1 - - stage2 - - stage3 - -step-1: - stage: stage1 - script: - - echo "Can be canceled." - interruptible: true - -step-2: - stage: stage2 - script: - - echo "Can not be canceled." - -step-3: - stage: stage3 - script: - - echo "Because step-2 can not be canceled, this step can never be canceled, even though it's set as interruptible." - interruptible: true -``` - -In this example, a new pipeline causes a running pipeline to be: - -- Canceled, if only `step-1` is running or pending. -- Not canceled, after `step-2` starts. - -**Additional details**: - -- Only set `interruptible: true` if the job can be safely canceled after it has started, - like a build job. Deployment jobs usually shouldn't be cancelled, to prevent partial deployments. -- To completely cancel a running pipeline, all jobs must have `interruptible: true`, - or `interruptible: false` jobs must not have started. - -### `resource_group` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. - -Use `resource_group` to create a [resource group](../resource_groups/index.md) that -ensures a job is mutually exclusive across different pipelines for the same project. - -For example, if multiple jobs that belong to the same resource group are queued simultaneously, -only one of the jobs starts. The other jobs wait until the `resource_group` is free. - -Resource groups behave similar to semaphores in other programming languages. - -You can define multiple resource groups per environment. For example, -when deploying to physical devices, you might have multiple physical devices. Each device -can be deployed to, but only one deployment can occur per device at any given time. - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: Only letters, digits, `-`, `_`, `/`, `$`, `{`, `}`, `.`, and spaces. -It can't start or end with `/`. - -**Example of `resource_group`**: - -```yaml -deploy-to-production: - script: deploy - resource_group: production -``` - -In this example, two `deploy-to-production` jobs in two separate pipelines can never run at the same time. As a result, -you can ensure that concurrent deployments never happen to the production environment. - -**Related topics**: - -- [Pipeline-level concurrency control with cross-project/parent-child pipelines](../resource_groups/index.md#pipeline-level-concurrency-control-with-cross-projectparent-child-pipelines). - ### `release` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19298) in GitLab 13.2. @@ -3660,7 +2668,7 @@ you can use this image from the GitLab Container Registry: `registry.gitlab.com/ **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: The `release:` subkeys: +**Possible inputs**: The `release` subkeys: - [`tag_name`](#releasetag_name) - [`name`](#releasename) (optional) @@ -3679,7 +2687,7 @@ you can use this image from the GitLab Container Registry: `registry.gitlab.com/ rules: - if: $CI_COMMIT_TAG # Run this job when a tag is created manually script: - - echo 'Running the release job.' + - echo "Running the release job." release: name: 'Release $CI_COMMIT_TAG' description: 'Release created using the release-cli.' @@ -3697,7 +2705,7 @@ This example creates a release: ```yaml script: - - echo 'release job' + - echo "release job" ``` An [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/223856) exists to remove this requirement. @@ -3723,7 +2731,9 @@ New tags use the SHA associated with the pipeline. **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: A tag name. Can use [CI/CD variables](../variables/index.md). +**Possible inputs**: + +- A tag name. Can use [CI/CD variables](../variables/index.md). **Example of `release:tag_name`**: @@ -3735,7 +2745,7 @@ To create a release when a new tag is added to the project: ```yaml job: - script: echo 'Running the release job for the new tag.' + script: echo "Running the release job for the new tag." release: tag_name: $CI_COMMIT_TAG description: 'Release description' @@ -3748,7 +2758,7 @@ should **not** configure the job to run only for new tags. A semantic versioning ```yaml job: - script: echo 'Running the release job and creating a new tag.' + script: echo "Running the release job and creating a new tag." release: tag_name: ${MAJOR}_${MINOR}_${REVISION} description: 'Release description' @@ -3762,7 +2772,9 @@ The release name. If omitted, it is populated with the value of `release: tag_na **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: A text string. +**Possible inputs**: + +- A text string. **Example of `release:name`**: @@ -3849,6 +2861,405 @@ assets: link_type: 'other' # optional ``` +### `resource_group` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. + +Use `resource_group` to create a [resource group](../resource_groups/index.md) that +ensures a job is mutually exclusive across different pipelines for the same project. + +For example, if multiple jobs that belong to the same resource group are queued simultaneously, +only one of the jobs starts. The other jobs wait until the `resource_group` is free. + +Resource groups behave similar to semaphores in other programming languages. + +You can define multiple resource groups per environment. For example, +when deploying to physical devices, you might have multiple physical devices. Each device +can be deployed to, but only one deployment can occur per device at any given time. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- Only letters, digits, `-`, `_`, `/`, `$`, `{`, `}`, `.`, and spaces. + It can't start or end with `/`. + +**Example of `resource_group`**: + +```yaml +deploy-to-production: + script: deploy + resource_group: production +``` + +In this example, two `deploy-to-production` jobs in two separate pipelines can never run at the same time. As a result, +you can ensure that concurrent deployments never happen to the production environment. + +**Related topics**: + +- [Pipeline-level concurrency control with cross-project/parent-child pipelines](../resource_groups/index.md#pipeline-level-concurrency-control-with-cross-projectparent-child-pipelines). + +### `retry` + +Use `retry` to configure how many times a job is retried if it fails. +If not defined, defaults to `0` and jobs do not retry. + +When a job fails, the job is processed up to two more times, until it succeeds or +reaches the maximum number of retries. + +By default, all failure types cause the job to be retried. Use [`retry:when`](#retrywhen) +to select which failures to retry on. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). + +**Possible inputs**: + +- `0` (default), `1`, or `2`. + +**Example of `retry`**: + +```yaml +test: + script: rspec + retry: 2 +``` + +#### `retry:when` + +Use `retry:when` with `retry:max` to retry jobs for only specific failure cases. +`retry:max` is the maximum number of retries, like [`retry`](#retry), and can be +`0`, `1`, or `2`. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). + +**Possible inputs**: + +- A single failure type, or an array of one or more failure types: + +<!-- + If you change any of the values below, make sure to update the `RETRY_WHEN_IN_DOCUMENTATION` + array in `spec/lib/gitlab/ci/config/entry/retry_spec.rb`. + The test there makes sure that all documented + values are valid as a configuration option and therefore should always + stay in sync with this documentation. +--> + +- `always`: Retry on any failure (default). +- `unknown_failure`: Retry when the failure reason is unknown. +- `script_failure`: Retry when the script failed. +- `api_failure`: Retry on API failure. +- `stuck_or_timeout_failure`: Retry when the job got stuck or timed out. +- `runner_system_failure`: Retry if there is a runner system failure (for example, job setup failed). +- `runner_unsupported`: Retry if the runner is unsupported. +- `stale_schedule`: Retry if a delayed job could not be executed. +- `job_execution_timeout`: Retry if the script exceeded the maximum execution time set for the job. +- `archived_failure`: Retry if the job is archived and can't be run. +- `unmet_prerequisites`: Retry if the job failed to complete prerequisite tasks. +- `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner. +- `data_integrity_failure`: Retry if there is a structural integrity problem detected. + +**Example of `retry:when`** (single failure type): + +```yaml +test: + script: rspec + retry: + max: 2 + when: runner_system_failure +``` + +If there is a failure other than a runner system failure, the job is not retried. + +**Example of `retry:when`** (array of failure types): + +```yaml +test: + script: rspec + retry: + max: 2 + when: + - runner_system_failure + - stuck_or_timeout_failure +``` + +**Related topics**: + +You can specify the number of [retry attempts for certain stages of job execution](../runners/configure_runners.md#job-stages-attempts) +using variables. + +### `rules` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3. + +Use `rules` to include or exclude jobs in pipelines. + +Rules are evaluated when the pipeline is created, and evaluated *in order* +until the first match. When a match is found, the job is either included or excluded from the pipeline, +depending on the configuration. + +You cannot use dotenv variables created in job scripts in rules, because rules are evaluated before any jobs run. + +`rules` replaces [`only/except`](#only--except) and they can't be used together +in the same job. If you configure one job to use both keywords, the GitLab returns +a `key may not be used with rules` error. + +`rules` accepts an array of rules defined with: + +- `if` +- `changes` +- `exists` +- `allow_failure` +- `variables` +- `when` + +You can combine multiple keywords together for [complex rules](../jobs/job_control.md#complex-rules). + +The job is added to the pipeline: + +- If an `if`, `changes`, or `exists` rule matches and also has `when: on_success` (default), + `when: delayed`, or `when: always`. +- If a rule is reached that is only `when: on_success`, `when: delayed`, or `when: always`. + +The job is not added to the pipeline: + +- If no rules match. +- If a rule matches and has `when: never`. + +You can use [`!reference` tags](yaml_optimization.md#reference-tags) to [reuse `rules` configuration](../jobs/job_control.md#reuse-rules-in-different-jobs) +in different jobs. + +#### `rules:if` + +Use `rules:if` clauses to specify when to add a job to a pipeline: + +- If an `if` statement is true, add the job to the pipeline. +- If an `if` statement is true, but it's combined with `when: never`, do not add the job to the pipeline. +- If no `if` statements are true, do not add the job to the pipeline. + +`if` clauses are evaluated based on the values of [predefined CI/CD variables](../variables/predefined_variables.md) +or [custom CI/CD variables](../variables/index.md#custom-cicd-variables). + +**Keyword type**: Job-specific and pipeline-specific. You can use it as part of a job +to configure the job behavior, or with [`workflow`](#workflow) to configure the pipeline behavior. + +**Possible inputs**: + +- A [CI/CD variable expression](../jobs/job_control.md#cicd-variable-expressions). + +**Example of `rules:if`**: + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != $CI_DEFAULT_BRANCH' + when: never + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/' + when: manual + allow_failure: true + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' +``` + +**Additional details**: + +- If a rule matches and has no `when` defined, the rule uses the `when` + defined for the job, which defaults to `on_success` if not defined. +- You can define `when` once per rule, or once at the job-level, which applies to + all rules. You can't mix `when` at the job-level with `when` in rules. +- Unlike variables in [`script`](../variables/index.md#use-cicd-variables-in-job-scripts) + sections, variables in rules expressions are always formatted as `$VARIABLE`. + - You can use `rules:if` with `include` to [conditionally include other configuration files](includes.md#use-rules-with-include). + +**Related topics**: + +- [Common `if` expressions for `rules`](../jobs/job_control.md#common-if-clauses-for-rules). +- [Avoid duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines). + +#### `rules:changes` + +Use `rules:changes` to specify when to add a job to a pipeline by checking for changes +to specific files. + +WARNING: +You should use `rules: changes` only with **branch pipelines** or **merge request pipelines**. +You can use `rules: changes` with other pipeline types, but `rules: changes` always +evaluates to true when there is no Git `push` event. Tag pipelines, scheduled pipelines, +and so on do **not** have a Git `push` event associated with them. A `rules: changes` job +is **always** added to those pipelines if there is no `if` that limits the job to +branch or merge request pipelines. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- An array of file paths. In GitLab 13.6 and later, [file paths can include variables](../jobs/job_control.md#variables-in-ruleschanges). + +**Example of `rules:changes`**: + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + changes: + - Dockerfile + when: manual + allow_failure: true +``` + +- If the pipeline is a merge request pipeline, check `Dockerfile` for changes. +- If `Dockerfile` has changed, add the job to the pipeline as a manual job, and the pipeline + continues running even if the job is not triggered (`allow_failure: true`). +- If `Dockerfile` has not changed, do not add job to any pipeline (same as `when: never`). + +**Additional details**: + +- `rules: changes` works the same way as [`only: changes` and `except: changes`](#onlychanges--exceptchanges). +- You can use `when: never` to implement a rule similar to [`except:changes`](#onlychanges--exceptchanges). +- `changes` resolves to `true` if any of the matching files are changed (an `OR` operation). + +#### `rules:exists` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. + +Use `exists` to run a job when certain files exist in the repository. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- An array of file paths. Paths are relative to the project directory (`$CI_PROJECT_DIR`) + and can't directly link outside it. File paths can use glob patterns. + +**Example of `rules:exists`**: + +```yaml +job: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - exists: + - Dockerfile +``` + +`job` runs if a `Dockerfile` exists anywhere in the repository. + +**Additional details**: + +- Glob patterns are interpreted with Ruby [`File.fnmatch`](https://docs.ruby-lang.org/en/2.7.0/File.html#method-c-fnmatch) + with the flags `File::FNM_PATHNAME | File::FNM_DOTMATCH | File::FNM_EXTGLOB`. +- For performance reasons, GitLab matches a maximum of 10,000 `exists` patterns or + file paths. After the 10,000th check, rules with patterned globs always match. + In other words, the `exists` rule always assumes a match in projects with more + than 10,000 files. +- `exists` resolves to `true` if any of the listed files are found (an `OR` operation). + +#### `rules:allow_failure` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30235) in GitLab 12.8. + +Use [`allow_failure: true`](#allow_failure) in `rules` to allow a job to fail +without stopping the pipeline. + +You can also use `allow_failure: true` with a manual job. The pipeline continues +running without waiting for the result of the manual job. `allow_failure: false` +combined with `when: manual` in rules causes the pipeline to wait for the manual +job to run before continuing. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- `true` or `false`. Defaults to `false` if not defined. + +**Example of `rules:allow_failure`**: + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH' + when: manual + allow_failure: true +``` + +If the rule matches, then the job is a manual job with `allow_failure: true`. + +**Additional details**: + +- The rule-level `rules:allow_failure` overrides the job-level [`allow_failure`](#allow_failure), + and only applies when the specific rule triggers the job. + +#### `rules:variables` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209864) in GitLab 13.7. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289803) in GitLab 13.10. + +Use [`variables`](#variables) in `rules` to define variables for specific conditions. + +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: + +- A hash of variables in the format `VARIABLE-NAME: value`. + +**Example of `rules:variables`**: + +```yaml +job: + variables: + DEPLOY_VARIABLE: "default-deploy" + rules: + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH + variables: # Override DEPLOY_VARIABLE defined + DEPLOY_VARIABLE: "deploy-production" # at the job level. + - if: $CI_COMMIT_REF_NAME =~ /feature/ + variables: + IS_A_FEATURE: "true" # Define a new variable. + script: + - echo "Run script with $DEPLOY_VARIABLE as an argument" + - echo "Run another script if $IS_A_FEATURE exists" +``` + +### `script` + +Use `script` to specify commands for the runner to execute. + +All jobs except [trigger jobs](#trigger) require a `script` keyword. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: An array including: + +- Single line commands. +- Long commands [split over multiple lines](script.md#split-long-commands). +- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). + +**Example of `script`**: + +```yaml +job1: + script: "bundle exec rspec" + +job2: + script: + - uname -a + - bundle exec rspec +``` + +**Additional details**: + +- When you use [these special characters in `script`](script.md#use-special-characters-with-script), you must use single quotes (`'`) or double quotes (`"`) . + +**Related topics**: + +- You can [ignore non-zero exit codes](script.md#ignore-non-zero-exit-codes). +- [Use color codes with `script`](script.md#add-color-codes-to-script-output) + to make job logs easier to review. +- [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) + to simplify job log output. + ### `secrets` **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33014) in GitLab 13.4. @@ -3926,7 +3337,9 @@ the secret value directly in the variable. **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: `true` (default) or `false`. +**Possible inputs**: + +- `true` (default) or `false`. **Example of `secrets:file`**: @@ -3943,118 +3356,323 @@ job: - The `file` keyword is a setting for the CI/CD variable and must be nested under the CI/CD variable name, not in the `vault` section. -### `pages` +### `services` -Use `pages` to define a [GitLab Pages](../../user/project/pages/index.md) job that -uploads static content to GitLab. The content is then published as a website. +Use `services` to specify an additional Docker image to run scripts in. The [`services` image](../services/index.md) is linked +to the image specified in the [`image`](#image) keyword. -**Keyword type**: Job name. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -**Example of `pages`**: +**Possible inputs**: The name of the services image, including the registry path if needed, in one of these formats: + +- `<image-name>` (Same as using `<image-name>` with the `latest` tag) +- `<image-name>:<tag>` +- `<image-name>@<digest>` + +**Example of `services`**: ```yaml -pages: - stage: deploy +default: + image: + name: ruby:2.6 + entrypoint: ["/bin/bash"] + + services: + - name: my-postgres:11.7 + alias: db-postgres + entrypoint: ["/usr/local/bin/db-postgres"] + command: ["start"] + + before_script: + - bundle install + +test: script: - - mkdir .public - - cp -r * .public - - mv .public public - artifacts: - paths: - - public - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - bundle exec rake spec ``` -This example moves all files from the root of the project to the `public/` directory. -The `.public` workaround is so `cp` does not also copy `public/` to itself in an infinite loop. +In this example, the job launches a Ruby container. Then, from that container, the job launches +another container that's running PostgreSQL. Then the job then runs scripts +in that container. + +**Related topics**: + +- [Available settings for `services`](../services/index.md#available-settings-for-services). +- [Define `services` in the `.gitlab-ci.yml` file](../services/index.md#define-services-in-the-gitlab-ciyml-file). +- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). +- [Use Docker to build Docker images](../docker/using_docker_build.md). + +### `stage` + +Use `stage` to define which [stage](#stages) a job runs in. Jobs in the same +`stage` can execute in parallel (see **Additional details**). + +If `stage` is not defined, the job uses the `test` stage by default. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: An array including any number of stage names. Stage names can be: + +- The [default stages](#stages). +- User-defined stages. + +**Example of `stage`**: + +```yaml +stages: + - build + - test + - deploy + +job1: + stage: build + script: + - echo "This job compiles code." + +job2: + stage: test + script: + - echo "This job tests the compiled code. It runs when the build stage completes." + +job3: + script: + - echo "This job also runs in the test stage". + +job4: + stage: deploy + script: + - echo "This job deploys the code. It runs when the test stage completes." +``` **Additional details**: -You must: +- Jobs can run in parallel if they run on different runners. +- If you have only one runner, jobs can run in parallel if the runner's + [`concurrent` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) + is greater than `1`. -- Place any static content in a `public/` directory. -- Define [`artifacts`](#artifacts) with a path to the `public/` directory. +#### `stage: .pre` -### `inherit` +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31441) in GitLab 12.4. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207484) in GitLab 12.9. +Use the `.pre` stage to make a job run at the start of a pipeline. `.pre` is +always the first stage in a pipeline. User-defined stages execute after `.pre`. +You do not have to define `.pre` in [`stages`](#stages). -Use `inherit:` to [control inheritance of globally-defined defaults and variables](../jobs/index.md#control-the-inheritance-of-default-keywords-and-global-variables). +You must have a job in at least one stage other than `.pre` or `.post`. -#### `inherit:default` +**Keyword type**: You can only use it with a job's `stage` keyword. -Use `inherit:default` to control the inheritance of [default keywords](#default). +**Example of `stage: .pre`**: -**Keyword type**: Job keyword. You can use it only as part of a job. +```yaml +stages: + - build + - test -**Possible inputs**: +job1: + stage: build + script: + - echo "This job runs in the build stage." -- `true` (default) or `false` to enable or disable the inheritance of all default keywords. -- A list of specific default keywords to inherit. +first-job: + stage: .pre + script: + - echo "This job runs in the .pre stage, before all other stages." + +job2: + stage: test + script: + - echo "This job runs in the test stage." +``` + +#### `stage: .post` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31441) in GitLab 12.4. -**Example of `inherit:default`:** +Use the `.post` stage to make a job run at the end of a pipeline. `.post` +is always the last stage in a pipeline. User-defined stages execute before `.post`. +You do not have to define `.post` in [`stages`](#stages). + +You must have a job in at least one stage other than `.pre` or `.post`. + +**Keyword type**: You can only use it with a job's `stage` keyword. + +**Example of `stage: .post`**: ```yaml -default: - retry: 2 - image: ruby:3.0 - interruptible: true +stages: + - build + - test job1: - script: echo "This job does not inherit any default keywords." - inherit: - default: false + stage: build + script: + - echo "This job runs in the build stage." + +last-job: + stage: .post + script: + - echo "This job runs in the .post stage, after all other stages." job2: - script: echo "This job inherits only the two listed default keywords. It does not inherit 'interruptible'." - inherit: - default: - - retry - - image + stage: test + script: + - echo "This job runs in the test stage." ``` -**Additional details:** +### `tags` -- You can also list default keywords to inherit on one line: `default: [keyword1, keyword2]` +> - A limit of 50 tags per job [enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338929) in GitLab 14.3. +> - A limit of 50 tags per job [enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/339855) in GitLab 14.3. -#### `inherit:variables` +Use `tags` to select a specific runner from the list of all runners that are +available for the project. -Use `inherit:variables` to control the inheritance of [global variables](#variables) keywords. +When you register a runner, you can specify the runner's tags, for +example `ruby`, `postgres`, or `development`. To pick up and run a job, a runner must +be assigned every tag listed in the job. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). + +**Possible inputs**: + +- An array of tag names. +- [CI/CD variables](../runners/configure_runners.md#use-cicd-variables-in-tags) in GitLab 14.1 and later. + +**Example of `tags`**: + +```yaml +job: + tags: + - ruby + - postgres +``` + +In this example, only runners with *both* the `ruby` and `postgres` tags can run the job. + +**Additional details**: + +- In [GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/issues/338479) and later, + the number of tags must be less than `50`. + +**Related topics**: + +- [Use tags to control which jobs a runner can run](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run). + +### `timeout` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14887) in GitLab 12.3. + +Use `timeout` to configure a timeout for a specific job. If the job runs for longer +than the timeout, the job fails. + +The job-level timeout can be longer than the [project-level timeout](../pipelines/settings.md#set-a-limit-for-how-long-jobs-can-run). +but can't be longer than the [runner's timeout](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). + +**Possible inputs**: A period of time written in natural language. For example, these are all equivalent: + +- `3600 seconds` +- `60 minutes` +- `one hour` + +**Example of `timeout`**: + +```yaml +build: + script: build.sh + timeout: 3 hours 30 minutes + +test: + script: rspec + timeout: 3h 30m +``` + +### `trigger` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in GitLab Premium 11.8. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. + +Use `trigger` to start a downstream pipeline that is either: + +- [A multi-project pipeline](../pipelines/multi_project_pipelines.md). +- [A child pipeline](../pipelines/parent_child_pipelines.md). **Keyword type**: Job keyword. You can use it only as part of a job. **Possible inputs**: -- `true` (default) or `false` to enable or disable the inheritance of all global variables. -- A list of specific variables to inherit. +- For multi-project pipelines, path to the downstream project. +- For child pipelines, path to the child pipeline CI/CD configuration file. -**Example of `inherit:variables`:** +**Example of `trigger` for multi-project pipeline**: ```yaml -variables: - VARIABLE1: "This is variable 1" - VARIABLE2: "This is variable 2" - VARIABLE3: "This is variable 3" +rspec: + stage: test + script: bundle exec rspec -job1: - script: echo "This job does not inherit any global variables." - inherit: - variables: false +staging: + stage: deploy + trigger: my/deployment +``` -job2: - script: echo "This job inherits only the two listed global variables. It does not inherit 'VARIABLE3'." - inherit: - variables: - - VARIABLE1 - - VARIABLE2 +**Example of `trigger` for child pipelines**: + +```yaml +trigger_job: + trigger: + include: path/to/child-pipeline.yml ``` -**Additional details:** +**Additional details**: -- You can also list global variables to inherit on one line: `variables: [VARIABLE1, VARIABLE2]` +- Jobs with `trigger` can only use a [limited set of keywords](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). + For example, you can't run commands with [`script`](#script), [`before_script`](#before_script), + or [`after_script`](#after_script). +- You [cannot use the API to start `when:manual` trigger jobs](https://gitlab.com/gitlab-org/gitlab/-/issues/284086). +- In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201938), you + can use [`when:manual`](#when) in the same job as `trigger`. In GitLab 13.4 and + earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. +- In [GitLab 13.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/197140/), you can + view which job triggered a downstream pipeline in the [pipeline graph](../pipelines/index.md#visualize-pipelines). + +**Related topics**: + +- [Multi-project pipeline configuration examples](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). +- [Child pipeline configuration examples](../pipelines/parent_child_pipelines.md#examples). +- To run a pipeline for a specific branch, tag, or commit, you can use a [trigger token](../triggers/index.md) + to authenticate with the [pipeline triggers API](../../api/pipeline_triggers.md). + The trigger token is different than the `trigger` keyword. + +#### `trigger:strategy` + +Use `trigger:strategy` to force the `trigger` job to wait for the downstream pipeline to complete +before it is marked as **success**. + +This behavior is different than the default, which is for the `trigger` job to be marked as +**success** as soon as the downstream pipeline is created. + +This setting makes your pipeline execution linear rather than parallel. + +**Example of `trigger:strategy`**: + +```yaml +trigger_job: + trigger: + include: path/to/child-pipeline.yml + strategy: depend +``` + +In this example, jobs from subsequent stages wait for the triggered pipeline to +successfully complete before starting. -## `variables` +### `variables` [CI/CD variables](../variables/index.md) are configurable values that are passed to jobs. Use `variables` to create [custom variables](../variables/index.md#custom-cicd-variables). @@ -4071,10 +3689,11 @@ variable defined, the [job-level variable takes precedence](../variables/index.m **Possible inputs**: Variable name and value pairs: -- The name can use only numbers, letters, and underscores (`_`). +- The name can use only numbers, letters, and underscores (`_`). In some shells, + the first character must be a letter. - The value must be a string. -**Examples of `variables`:** +**Examples of `variables`**: ```yaml variables: @@ -4106,7 +3725,7 @@ deploy_review_job: automatically creates and makes available in the job. - You can [configure runner behavior with variables](../runners/configure_runners.md#configure-runner-behavior-with-variables). -### `variables:description` +#### `variables:description` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) in GitLab 13.7. @@ -4117,7 +3736,9 @@ Must be used with `value`, for the variable value. **Keyword type**: Global keyword. You cannot set job-level variables to be pre-filled when you run a pipeline manually. -**Possible inputs**: A string. +**Possible inputs**: + +- A string. **Example of `variables:description`**: @@ -4128,6 +3749,84 @@ variables: description: "The deployment target. Change this variable to 'canary' or 'production' if needed." ``` +### `when` + +Use `when` to configure the conditions for when jobs run. If not defined in a job, +the default value is `when: on_success`. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- `on_success` (default): Run the job only when all jobs in earlier stages succeed + or have `allow_failure: true`. +- `manual`: Run the job only when [triggered manually](../jobs/job_control.md#create-a-job-that-must-be-run-manually). +- `always`: Run the job regardless of the status of jobs in earlier stages. +- `on_failure`: Run the job only when at least one job in an earlier stage fails. +- `delayed`: [Delay the execution of a job](../jobs/job_control.md#run-a-job-after-a-delay) + for a specified duration. +- `never`: Don't run the job. + +**Example of `when`**: + +```yaml +stages: + - build + - cleanup_build + - test + - deploy + - cleanup + +build_job: + stage: build + script: + - make build + +cleanup_build_job: + stage: cleanup_build + script: + - cleanup build when failed + when: on_failure + +test_job: + stage: test + script: + - make test + +deploy_job: + stage: deploy + script: + - make deploy + when: manual + +cleanup_job: + stage: cleanup + script: + - cleanup after jobs + when: always +``` + +In this example, the script: + +1. Executes `cleanup_build_job` only when `build_job` fails. +1. Always executes `cleanup_job` as the last step in pipeline regardless of + success or failure. +1. Executes `deploy_job` when you run it manually in the GitLab UI. + +**Additional details**: + +- In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201938), you + can use `when:manual` in the same job as [`trigger`](#trigger). In GitLab 13.4 and + earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. +- The default behavior of `allow_failure` changes to `true` with `when: manual`. + However, if you use `when: manual` with [`rules`](#rules), `allow_failure` defaults + to `false`. + +**Related topics**: + +- `when` can be used with [`rules`](#rules) for more dynamic job control. +- `when` can be used with [`workflow`](#workflow) to control when a pipeline can start. + ## Deprecated keywords The following keywords are deprecated. @@ -4150,7 +3849,7 @@ Defining `image`, `services`, `cache`, `before_script`, and `after_script` globally is deprecated. Support could be removed from a future release. -Use [`default:`](#default) instead. For example: +Use [`default`](#default) instead. For example: ```yaml default: diff --git a/doc/ci/yaml/script.md b/doc/ci/yaml/script.md index c1b283ff10f..fdec0947df5 100644 --- a/doc/ci/yaml/script.md +++ b/doc/ci/yaml/script.md @@ -13,7 +13,7 @@ You can use special syntax in [`script`](index.md#script) sections to: - [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) to simplify job log output. -## Use special characters with `script:` +## Use special characters with `script` Sometimes, `script` commands must be wrapped in single or double quotes. For example, commands that contain a colon (`:`) must be wrapped in single quotes (`'`). @@ -101,7 +101,7 @@ WARNING: If multiple commands are combined into one command string, only the last command's failure or success is reported. [Failures from earlier commands are ignored due to a bug](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25394). -To work around this, run each command as a separate `script:` item, or add an `exit 1` +To work around this, run each command as a separate `script` item, or add an `exit 1` command to each command string. You can use the `|` (literal) YAML multiline block scalar indicator to write diff --git a/doc/ci/yaml/workflow.md b/doc/ci/yaml/workflow.md index 67ca1150553..332214638d8 100644 --- a/doc/ci/yaml/workflow.md +++ b/doc/ci/yaml/workflow.md @@ -4,7 +4,7 @@ group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab CI/CD `workflow` keyword +# GitLab CI/CD `workflow` keyword **(FREE)** Use the [`workflow`](index.md#workflow) keyword to control when pipelines are created. diff --git a/doc/development/adding_database_indexes.md b/doc/development/adding_database_indexes.md index 571d2f353d4..0e8e8289464 100644 --- a/doc/development/adding_database_indexes.md +++ b/doc/development/adding_database_indexes.md @@ -317,11 +317,11 @@ end You must test the database index changes locally before creating a merge request. -### Verify indexes created asynchronously +### Verify indexes created asynchronously Use the asynchronous index helpers on your local environment to test changes for creating an index: 1. Enable the feature flags by running `Feature.enable(:database_async_index_creation)` and `Feature.enable(:database_reindexing)` in the Rails console. 1. Run `bundle exec rails db:migrate` so that it creates an entry in the `postgres_async_indexes` table. 1. Run `bundle exec rails gitlab:db:reindex` so that the index is created asynchronously. -1. To verify the index, open the PostgreSQL console using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md) command `gdk psql` and run the command `\d <index_name>` to check that your newly created index exists. +1. To verify the index, open the PostgreSQL console using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md) command `gdk psql` and run the command `\d <index_name>` to check that your newly created index exists. diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 9a17ac4c813..afd745533c9 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -152,8 +152,8 @@ The `iid`, `title` and `description` are _scalar_ GraphQL types. `title` and `description` are regular `GraphQL::Types::String` types. Note that the old scalar types `GraphQL:ID`, `GraphQL::INT_TYPE`, `GraphQL::STRING_TYPE`, -and `GraphQL:BOOLEAN_TYPE` are no longer allowed. Please use `GraphQL::Types::ID`, -`GraphQL::Types::Int`, `GraphQL::Types::String`, and `GraphQL::Types::Boolean`. +`GraphQL:BOOLEAN_TYPE`, and `GraphQL::FLOAT_TYPE` are no longer allowed. Please use `GraphQL::Types::ID`, +`GraphQL::Types::Int`, `GraphQL::Types::String`, `GraphQL::Types::Boolean`, and `GraphQL::Types::Float`. When exposing a model through the GraphQL API, we do so by creating a new type in `app/graphql/types`. You can also declare custom GraphQL data types @@ -698,7 +698,10 @@ aware of the support. The documentation will mention that the old Global ID style is now deprecated. -See also [Aliasing and deprecating mutations](#aliasing-and-deprecating-mutations). +See also: + +- [Aliasing and deprecating mutations](#aliasing-and-deprecating-mutations). +- [How to filter Kibana for queries that used deprecated fields](graphql_guide/monitoring.md#queries-that-used-a-deprecated-field). ## Enums @@ -830,7 +833,7 @@ field :id, GraphQL::Types::ID, description: 'ID of the resource.' Descriptions of fields and arguments are viewable to users through: - The [GraphiQL explorer](#graphiql). -- The [static GraphQL API reference](../api/graphql/#reference). +- The [static GraphQL API reference](../api/graphql/reference/index.md). ### Description style guide @@ -2027,3 +2030,7 @@ elimination of laziness, where needed. For dealing with lazy values without forcing them, use `Gitlab::Graphql::Lazy.with_value`. + +## Monitoring GraphQL + +See the [Monitoring GraphQL](graphql_guide/monitoring.md) guide for tips on how to inspect logs of GraphQL requests and monitor the performance of your GraphQL queries. diff --git a/doc/development/application_slis/index.md b/doc/development/application_slis/index.md index 5bc6fffdb48..87c0bcfede5 100644 --- a/doc/development/application_slis/index.md +++ b/doc/development/application_slis/index.md @@ -180,7 +180,7 @@ alerts about the SLI in specified Slack channels. For more information, read the [alert routing documentation](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/uncategorized/alert-routing.md). In [this project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/614) we are extending this so alerts for SLIs with a `feature_category` -label in the souce metrics can also be routed. +label in the source metrics can also be routed. For any question, please don't hesitate to create an issue in [the Scalability issue tracker](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues) diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 38d0d5d7843..078eb5dd0de 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -126,7 +126,7 @@ graph LR Geo -- TCP 22 --> SSH Geo -- TCP 5432 --> PostgreSQL Runner{{GitLab Runner}} -- TCP 443 --> HTTP - K8sAgent{{GitLab Kubernetes Agent}} -- TCP 443 --> HTTP + K8sAgent{{GitLab Agent}} -- TCP 443 --> HTTP %% GitLab Application Suite subgraph GitLab @@ -157,7 +157,7 @@ graph LR Puma["Puma (GitLab Rails)"] Puma <--> Registry GitLabWorkhorse[GitLab Workhorse] <--> Puma - GitLabKas[GitLab Kubernetes Agent Server] --> GitLabWorkhorse + GitLabKas[GitLab Agent Server] --> GitLabWorkhorse GitLabPages[GitLab Pages] --> GitLabWorkhorse Mailroom Sidekiq @@ -349,7 +349,7 @@ Component statuses are linked to configuration documentation for each component. | [GitLab Exporter](#gitlab-exporter) | Generates a variety of GitLab metrics | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | CE & EE | | [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | ⚙ | ⚙ | ❌ | ❌ | ✅ | ❌ | ⚙ | EE Only | | [GitLab Pages](#gitlab-pages) | Hosts static websites | ⚙ | ⚙ | ❌ | ❌ | ✅ | ⚙ | ⚙ | CE & EE | -| [GitLab Kubernetes Agent](#gitlab-kubernetes-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | +| [GitLab Agent](#gitlab-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | | [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | ⚙ | ⚙ | ✅ | ⚙ | ✅ | ❌ | ❌ | CE & EE | | [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | ✅ | ✅ | ⚙ | ⤓ | ✅ | ❌ | ⚙ | CE & EE | | [GitLab self-monitoring: Jaeger](#jaeger) | View traces generated by the GitLab instance | ❌ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | CE & EE | @@ -499,14 +499,14 @@ Geo is a premium feature built to help speed up the development of distributed t GitLab Exporter is a process designed in house that allows us to export metrics about GitLab application internals to Prometheus. You can read more [in the project's README](https://gitlab.com/gitlab-org/gitlab-exporter). -#### GitLab Kubernetes Agent +#### GitLab Agent - [Project page](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) - Configuration: - [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) - [Charts](https://docs.gitlab.com/charts/charts/gitlab/kas/index.html) -[GitLab Kubernetes Agent](../user/clusters/agent/index.md) is an active in-cluster +The [GitLab Agent](../user/clusters/agent/index.md) is an active in-cluster component for solving GitLab and Kubernetes integration tasks in a secure and cloud-native way. diff --git a/doc/development/audit_event_guide/index.md b/doc/development/audit_event_guide/index.md index 65a7fa0ae90..ae2f9748178 100644 --- a/doc/development/audit_event_guide/index.md +++ b/doc/development/audit_event_guide/index.md @@ -176,3 +176,14 @@ deactivate B In addition to recording to the database, we also write these events to [a log file](../../administration/logs.md#audit_jsonlog). + +## Event streaming + +All events where the entity is a `Group` or `Project` are recorded in the audit log, and also streamed to one or more +[event streaming destinations](../../administration/audit_event_streaming.md). When the entity is a: + +- `Group`, events are streamed to the group's root ancestor's event streaming destinations. +- `Project`, events are streamed to the project's root ancestor's event streaming destinations. + +This feature is under heavy development. Follow the [parent epic](https://gitlab.com/groups/gitlab-org/-/epics/5925) for updates on feature +development. diff --git a/doc/development/backend/ruby_style_guide.md b/doc/development/backend/ruby_style_guide.md index 461bb9aafde..419db628b0d 100644 --- a/doc/development/backend/ruby_style_guide.md +++ b/doc/development/backend/ruby_style_guide.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w This is a GitLab-specific style guide for Ruby code. -Generally, if a style is not covered by [existing rubocop rules or styleguides](../contributing/style_guides.md#ruby-rails-rspec), it shouldn't be a blocker. +Generally, if a style is not covered by [existing Rubocop rules or style guides](../contributing/style_guides.md#ruby-rails-rspec), it shouldn't be a blocker. Before adding a new cop to enforce a given style, make sure to discuss it with your team. When the style is approved by a backend EM or by a BE staff eng, add a new section to this page to document the new rule. For every new guideline, add it in a new section and link the discussion from the section's diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index c93b5b448f0..4a18b2123da 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -165,7 +165,8 @@ big JSON blob) to column `bar` (containing a string). The process for this would roughly be as follows: 1. Release A: - 1. Create a migration class that perform the migration for a row with a given ID. + 1. Create a migration class that performs the migration for a row with a given ID. + You can use [background jobs tracking](#background-jobs-tracking) to simplify cleaning up. 1. Deploy the code for this release, this should include some code that will schedule jobs for newly created data (for example, using an `after_create` hook). 1. Schedule jobs for all existing rows in a post-deployment migration. It's @@ -174,8 +175,10 @@ roughly be as follows: 1. Release B: 1. Deploy code so that the application starts using the new column and stops scheduling jobs for newly created data. - 1. In a post-deployment migration use `finalize_background_migration` from - `BackgroundMigrationHelpers` to ensure no jobs remain. This helper will: + 1. In a post-deployment migration, finalize all jobs that have not succeeded by now. + If you used [background jobs tracking](#background-jobs-tracking) in release A, + you can use `finalize_background_migration` from `BackgroundMigrationHelpers` to ensure no jobs remain. + This helper will: 1. Use `Gitlab::BackgroundMigration.steal` to process any remaining jobs in Sidekiq. 1. Reschedule the migration to be run directly (that is, not through Sidekiq) @@ -510,12 +513,12 @@ See [`db/post_migrate/20210604070207_retry_backfill_traversal_ids.rb`](https://g ### Viewing failure error logs -After running a background migration, if any jobs have failed, you can view the logs in [Kibana](https://log.gprd.gitlab.net/goto/3afc1393447c401d7602c1874793e2f6). +After running a background migration, if any jobs have failed, you can view the logs in [Kibana](https://log.gprd.gitlab.net/goto/5f06a57f768c6025e1c65aefb4075694). View the production Sidekiq log and filter for: - `json.class: BackgroundMigrationWorker` - `json.job_status: fail` -- `json.args: <MyBackgroundMigrationClassName>` +- `json.meta.caller_id: <MyBackgroundMigrationClassName>` Looking at the `json.error_class`, `json.error_message` and `json.error_backtrace` values may be helpful in understanding why the jobs failed. diff --git a/doc/development/cascading_settings.md b/doc/development/cascading_settings.md index a85fc52d303..d04761400ac 100644 --- a/doc/development/cascading_settings.md +++ b/doc/development/cascading_settings.md @@ -194,7 +194,7 @@ This function should be imported and called in the [page-specific JavaScript](fe setting_locked: delayed_project_removal_locked, settings_path_helper: -> (locked_ancestor) { edit_group_path(locked_ancestor, anchor: 'js-permissions-settings') }, help_text: s_('Settings|Projects will be permanently deleted after a 7-day delay. Inherited by subgroups.') do - = s_('Settings|Enable delayed project removal') + = s_('Settings|Enable delayed project deletion') = render 'shared/namespaces/cascading_settings/enforcement_checkbox', attribute: :delayed_project_removal, group: @group, diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 2753257c941..7c4a600e1fa 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -101,7 +101,7 @@ EE: true - _Any_ contribution from a community member, no matter how small, **may** have a changelog entry regardless of these guidelines if the contributor wants one. - Any [GLEX experiment](experiment_guide/gitlab_experiment.md) changes **should not** have a changelog entry. -- [Removing](feature_flags/#changelog) a feature flag, when the new code is retained. +- [Modifying](feature_flags/#changelog) a feature flag (flag removal, default-on setting). ## Writing good changelog entries diff --git a/doc/development/cicd/cicd_reference_documentation_guide.md b/doc/development/cicd/cicd_reference_documentation_guide.md index aa3888cd866..e937220d208 100644 --- a/doc/development/cicd/cicd_reference_documentation_guide.md +++ b/doc/development/cicd/cicd_reference_documentation_guide.md @@ -6,17 +6,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w # CI/CD YAML reference style guide **(FREE)** -The CI/CD YAML reference uses a standard style to make it easier to use and update. +The [CI/CD YAML reference](../../ci/yaml/index.md) uses a standard style to make it easier to use and update. The reference information should be kept as simple as possible, and expanded details -and examples documented in a separate page. +and examples should be documented on other pages. ## YAML reference structure Every YAML keyword must have its own section in the reference. The sections should be nested so that the keywords follow a logical tree structure. For example: -```plaintext +```markdown ### `artifacts` #### `artifacts:name` #### `artifacts:paths` @@ -27,128 +27,127 @@ be nested so that the keywords follow a logical tree structure. For example: ## YAML reference style -Each keyword entry in the reference should use the following style: +Each keyword entry in the reference: -````markdown -### `keyword-name` - -> Version information - -Keyword description and main details. - -**Keyword type**: - -**Possible inputs**: +- Must have a simple introductory section. The introduction should give the fundamental + information needed to use the keyword. Advanced details and tasks should be in + feature pages, not the reference page. -**Example of `keyword-name`**: +- Must use the keyword name as the title, for example: -(optional) In this example... + ```markdown + ### `artifacts` + ``` -(optional) **Additional details**: +- Should include the following sections: + - [Keyword type](#keyword-type) + - [Possible inputs](#possible-inputs) + - [Example of `keyword-name`](#example-of-keyword-name) +- (Optional) Can also include the following sections when needed: + - [Additional details](#additional-details) + - [Related topics](#related-topics) -- List of extra details. +The keyword name must always be in backticks without a final `:`, like `artifacts`, not `artifacts:`. +If it is a subkey of another keyword, write out all the subkeys to the "parent" key the first time it +is used, like `artifacts:reports:dast`. Afterwards, you can use just the subkey alone, like `dast`. -(optional) **Related topics**: +## Keyword type -- List of links to topics related to the keyword. -```` - -- ``### `keyword-name` ``: The keyword name must always be in backticks. - If it is a subkey of another keyword, write out all the keywords, with each separated - by `:`, for example: `artifacts:reports:dast`. - -- ``> Version information``: The [version history details](../documentation/styleguide/index.md#version-text-in-the-version-history). - If the keyword is feature flagged, see the [feature flag documentation guide](../documentation/feature_flags.md) - as well. +The keyword can be either a job or global keyword. If it can be used in a `default` +section, make not of that as well, for example: -- `Keyword description and main details.`: A simple description of the keyword, and - how to use it. Additional use cases and benefits should be added to a page outside - the reference document. Link to that document in this section. +- `**Keyword type**: Global keyword.` +- `**Keyword type**: Job keyword. You can use it only as part of a job.` +- ``**Keyword type**: Job keyword. You can use it only as part of a job or in the [`default:` section](#default).`` -- `**Keyword type**:`: Most keywords are defined at the job level, like `script`, - or at the pipeline level, like `stages`. Add the appropriate line: +### Possible inputs - - `**Keyword type**: Job keyword. You can use it only as part of a job.` - - `**Keyword type**: Pipeline keyword. You cannot use it as part of a job.` +List all the possible inputs, and any extra details about the inputs, such as defaults +or changes due to different GitLab versions, for example: - If a keyword can be used at both the job and pipeline level, like `variables`, - explain it in detail instead of using the pre-written lines above. +```markdown +**Possible inputs**: -- `**Possible inputs**:`: Explain in detail which inputs the keyword can accept. - You can add the details in a sentence, paragraph, or list. +- `true` (default if not defined) or `false`. +``` -- ``**Example of `keyword-name`**:``: An example configuration that uses the keyword. - Do not add extra keywords that are not required to understand the behavior. +```markdown +**Possible inputs**: -- (optional) `In this example...`: If the example needs extra details, - add the clarification text below the example. +- A single exit code. +- An array of exit codes. +``` -- (optional) `**Additional details**:` If there are any caveats or extra details you - want to document along with the keyword, add each one as a list item here. +```markdown +**Possible inputs**: -- (optional) `**Related topics**:` If there are any other keywords or pages that - relate to this keyword, add these links as list items here. +- A string with the long description. +- The path to a file that contains the description. Introduced in [GitLab 13.7](https://gitlab.com/gitlab-org/release-cli/-/merge_requests/67). + - The file location must be relative to the project directory (`$CI_PROJECT_DIR`). + - If the file is a symbolic link, it must be in the `$CI_PROJECT_DIR`. + - The `./path/to/file` and filename can't contain spaces. +``` -### YAML reference style example +### Example of `keyword-name` -See the [`only:changes` / `except:changes`](../../ci/yaml/index.md#onlychanges--exceptchanges) -documentation for an example of the YAML reference style. The following example is a -shortened version of that documentation's Markdown: +An example of the keyword. Use the minimum number of other keywords necessary +to make the example valid. If the example needs explanation, add it after the example, +for example: ````markdown -#### `only:changes` / `except:changes` +**Example of `dast`**: -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19232) in GitLab 11.4. - -Use the `changes` keyword with `only` to run a job, or with `except` to skip a job, -when a Git push event modifies a file. +```yaml +stages: + - build + - dast -Use `changes` in pipelines with the following refs: +include: + - template: DAST.gitlab-ci.yml -- `branches` -- `external_pull_requests` -- `merge_requests` (see additional details about [using `only:changes` with pipelines for merge requests](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests)) +dast: + dast_configuration: + site_profile: "Example Co" + scanner_profile: "Quick Passive Test" +``` -**Keyword type**: Job keyword. You can use it only as part of a job. +In this example, the `dast` job extends the `dast` configuration added with the `include:` keyword +to select a specific site profile and scanner profile. +```` -**Possible inputs**: An array including any number of: +### Additional details -- Paths to files. -- Wildcard paths for single directories, for example `path/to/directory/*`, or a directory - and all its subdirectories, for example `path/to/directory/**/*`. +The additional details should be an unordered list of extra information that is +useful to know, but not important enough to put in the introduction. This information +can include changes introduced in different GitLab versions. For example: -**Example of `only:changes`**: +```markdown +**Additional details**: -```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - only: - refs: - - branches - changes: - - Dockerfile - - docker/scripts/* - - dockerfiles/**/* +- The expiration time period begins when the artifact is uploaded and stored on GitLab. + If the expiry time is not defined, it defaults to the [instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration). +- To override the expiration date and protect artifacts from being automatically deleted: + - Select **Keep** on the job page. + - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/22761), set the value of + `expire_in` to `never`. ``` -In this example, `docker build` only runs in branch pipelines, and only if at least one of -these files changed: +### Related topics -- `Dockerfile`. -- Any file in `docker/scripts` -- Any file in `dockerfiles/` or any of its subdirectories. +The related topics should be an unordered list of crosslinks to related pages, including: -**Additional details**: +- Specific tasks that you can accomplish with the keyword. +- Advanced examples of the keyword. +- Other related keywords that can be used together with this keyword. -- If you use refs other than `branches`, `external_pull_requests`, or `merge_requests`, - `changes` can't determine if a given file is new or old and always returns `true`. -- If you use `only: changes` with other refs, jobs ignore the changes and always run. -- If you use `except: changes` with other refs, jobs ignore the changes and never run. +For example: +```markdown **Related topics**: -- [`only: changes` and `except: changes` examples](../jobs/job_control.md#onlychanges--exceptchanges-examples). -- If you use `changes` with [only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds), - you should [also use `only:merge_requests`](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests). -- Use `changes` with [scheduled pipelines](../jobs/job_control.md#use-onlychanges-with-scheduled-pipelines). -```` +- You can specify a [fallback cache key](../caching/index.md#use-a-fallback-cache-key) + to use if the specified `cache:key` is not found. +- You can [use multiple cache keys](../caching/index.md#use-multiple-caches) in a single job. +- See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more + `cache:key` examples. +``` diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index 46442aa6106..b1252b86cc0 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -392,6 +392,6 @@ If you're unsure if it's secure or not, you must ask security experts for cross- After your CI/CD template MR is created and labeled with `ci::templates`, DangerBot suggests one reviewer and one maintainer that can review your code. When your merge -request is ready for review, please [mention](../../user/project/issues/issue_data_and_actions.md#mentions) +request is ready for review, please [mention](../../user/discussions/index.md#mentions) the reviewer and ask them to review your CI/CD template changes. See details in the merge request that added [a DangerBot task for CI/CD template MRs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44688). diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 7e797309a26..742d183e15c 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -381,7 +381,7 @@ first time. ### Requesting a review When you are ready to have your merge request reviewed, -you should request an initial review by assigning it to a reviewer from your group or team. +you should [request an initial review](../user/project/merge_requests/getting_started.md#reviewer) by selecting a reviewer from your group or team. However, you can also assign it to any reviewer. The list of reviewers can be found on [Engineering projects](https://about.gitlab.com/handbook/engineering/projects/) page. You can also use `workflow::ready for review` label. That means that your merge request is ready to be reviewed and any reviewer can pick it. It is recommended to use that label only if there isn't time pressure and make sure the merge request is assigned to a reviewer. diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index fe7dc52d077..ec05f75c709 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -129,7 +129,7 @@ The general flow of contributing to GitLab is: 1. In the merge request's description: - Ensure you provide complete and accurate information. - Review the provided checklist. -1. Assign the merge request (if possible) to, or [mention](../../user/project/issues/issue_data_and_actions.md#mentions), +1. Assign the merge request (if possible) to, or [mention](../../user/discussions/index.md#mentions), one of the [code owners](../../user/project/code_owners.md) for the relevant project, and explain that you are ready for review. diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 82fd62d8d79..cc6997e1a20 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -100,7 +100,7 @@ request is as follows: There isn't a way to know anything about our customers' data on their [self-managed instances](../../subscriptions/self_managed/index.md), so keep that in mind for any data implications with your merge request. - + 1. Merge requests **must** adhere to the [merge request performance guidelines](../merge_request_performance_guidelines.md). 1. For tests that use Capybara, read [how to write reliable, asynchronous integration tests](https://thoughtbot.com/blog/write-reliable-asynchronous-integration-tests-with-capybara). diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 754e6c7aec6..fdb6e99fdcd 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -171,7 +171,8 @@ we should track our progress through the exception list. When auto-generating the `.rubocop_todo.yml` exception list for a particular Cop, and more than 15 files are affected, we should add the exception list to -a different file, `.rubocop_manual_todo.yml`. +a different file in the directory `.rubocop_todo/`. For example, the configuration for the cop +`Gitlab/NamespacedClass` is in `.rubocop_todo/gitlab/namespaced_class.yml`. This ensures that our list isn't mistakenly removed by another auto generation of the `.rubocop_todo.yml`. This also allows us greater visibility into the exceptions @@ -184,19 +185,19 @@ bundle exec rake rubocop:todo:generate ``` You can then move the list from the freshly generated `.rubocop_todo.yml` for the Cop being actively -resolved and place it in the `.rubocop_manual_todo.yml`. In this scenario, do not commit auto generated -changes to the `.rubocop_todo.yml` as an `exclude limit` that is higher than 15 will make the -`.rubocop_todo.yml` hard to parse. +resolved and place it in the directory `.rubocop_todo/`. In this scenario, do not commit +auto-generated changes to the `.rubocop_todo.yml`, as an `exclude limit` that is higher than 15 +makes the `.rubocop_todo.yml` hard to parse. ### Reveal existing RuboCop exceptions To reveal existing RuboCop exceptions in the code that have been excluded via `.rubocop_todo.yml` and -`.rubocop_manual_todo.yml`, set the environment variable `REVEAL_RUBOCOP_TODO` to `1`. +`.rubocop_todo/**/*.yml`, set the environment variable `REVEAL_RUBOCOP_TODO` to `1`. This allows you to reveal existing RuboCop exceptions during your daily work cycle and fix them along the way. NOTE: -Permanent `Exclude`s should be defined in `.rubocop.yml` instead of `.rubocop_manual_todo.yml`. +Define permanent `Exclude`s in `.rubocop.yml` instead of `.rubocop_todo/**/*.yml`. ## Database migrations diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 829f6af76be..374e4e5de68 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -131,7 +131,7 @@ The main `Dangerfile` will then take care of adding the labels to the MR with a #### Shared rules and plugins If the rule or plugin you implement can be useful for other projects, think about -upstreaming them to the [`gitlab-org/gitlab-dangerfiles`](https://gitlab.com/gitlab-org/gitlab-dangerfiles) project. +upstreaming them to the [`gitlab-dangerfiles`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-dangerfiles) project. #### Enable Danger on a project diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index 157c1284512..c60989f225d 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -52,25 +52,40 @@ For this procedure to work, we must register which tables to clean up asynchrono ## Example migration and configuration -### Configure the model +### Configure the loose foreign key -First, tell the application that the `projects` table has a new loose foreign key. -You can do this in the `Project` model: +Loose foreign keys are defined in a YAML file. The configuration requires the +following information: -```ruby -class Project < ApplicationRecord - # ... +- Parent table name (`projects`) +- Child table name (`ci_pipelines`) +- The data cleanup method (`async_delete` or `async_nullify`) - include LooseForeignKey +The YAML file is located at `lib/gitlab/database/gitlab_loose_foreign_keys.yml`. The file groups +foreign key definitions by the name of the child table. The child table can have multiple loose +foreign key definitions, therefore we store them as an array. - loose_foreign_key :ci_pipelines, :project_id, on_delete: :async_delete # or async_nullify +Example definition: - # ... -end +```yaml +ci_pipelines: + - table: projects + column: project_id + on_delete: async_delete ``` -This instruction ensures the asynchronous cleanup process knows about the association, and the -how to do the cleanup. In this case, the associated `ci_pipelines` records are deleted. +If the `ci_pipelines` key is already present in the YAML file, then a new entry can be added +to the array: + +```yaml +ci_pipelines: + - table: projects + column: project_id + on_delete: async_delete + - table: another_table + column: another_id + on_delete: :async_nullify +``` ### Track record changes @@ -127,6 +142,19 @@ end At this point, the setup phase is concluded. The deleted `projects` records should be automatically picked up by the scheduled cleanup worker job. +## Testing + +The "`it has loose foreign keys`" shared example can be used to test the presence of the `ON DELETE` trigger and the +loose foreign key definitions. + +Simply add to the model test file: + +```ruby +it_behaves_like 'it has loose foreign keys' do + let(:factory_name) { :project } +end +``` + ## Caveats of loose foreign keys ### Record creation @@ -180,3 +208,90 @@ end NOTE: This example is unlikely in GitLab, because we usually look up the parent models to perform permission checks. + +## A note on `dependent: :destroy` and `dependent: :nullify` + +We considered using these Rails features as an alternative to foreign keys but there are several problems which include: + +1. These run on a different connection in the context of a transaction [which we do not allow](multiple_databases.md#removing-cross-database-transactions). +1. These can lead to severe performance degradation as we load all records from PostgreSQL, loop over them in Ruby, and call individual `DELETE` queries. +1. These can miss data as they only cover the case when the `destroy` method is called directly on the model. There are other cases including `delete_all` and cascading deletes from another parent table that could mean these are missed. + +## Risks of loose foreign keys and possible mitigations + +In general, the loose foreign keys architecture is eventually consistent and +the cleanup latency might lead to problems visible to GitLab users or +operators. We consider the tradeoff as acceptable, but there might be +cases where the problems are too frequent or too severe, and we must +implement a mitigation strategy. A general mitigation strategy might be to have +an "urgent" queue for cleanup of records that have higher impact with a delayed +cleanup. + +Below are some more specific examples of problems that might occur and how we +might mitigate them. In all the listed cases we might still consider the problem +described to be low risk and low impact, and in that case we would choose to not +implement any mitigation. + +### The record should be deleted but it shows up in a view + +This hypothetical example might happen with a foreign key like: + +```sql +ALTER TABLE ONLY vulnerability_occurrence_pipelines + ADD CONSTRAINT fk_rails_6421e35d7d FOREIGN KEY (pipeline_id) REFERENCES ci_pipelines(id) ON DELETE CASCADE; +``` + +In this example we expect to delete all associated `vulnerability_occurrence_pipelines` records +whenever we delete the `ci_pipelines` record associated with them. In this case +you might end up with some vulnerability page in GitLab which shows an occurrence +of a vulnerability. However, when you try to click a link to the pipeline, you get +a 404, because the pipeline is deleted. Then, when you navigate back you might find the +occurrence has disappeared too. + +**Mitigation** + +When rendering the vulnerability occurrences on the vulnerability page we could +try to load the corresponding pipeline and choose to skip displaying that +occurrence if pipeline is not found. + +### The deleted parent record is needed to render a view and causes a `500` error + +This hypothetical example might happen with a foreign key like: + +```sql +ALTER TABLE ONLY vulnerability_occurrence_pipelines + ADD CONSTRAINT fk_rails_6421e35d7d FOREIGN KEY (pipeline_id) REFERENCES ci_pipelines(id) ON DELETE CASCADE; +``` + +In this example we expect to delete all associated `vulnerability_occurrence_pipelines` records +whenever we delete the `ci_pipelines` record associated with them. In this case +you might end up with a vulnerability page in GitLab which shows an "occurrence" +of a vulnerability. However, when rendering the occurrence we try to load, for example, +`occurrence.pipeline.created_at`, which causes a 500 for the user. + +**Mitigation** + +When rendering the vulnerability occurrences on the vulnerability page we could +try to load the corresponding pipeline and choose to skip displaying that +occurrence if pipeline is not found. + +### The deleted parent record is accessed in a Sidekiq worker and causes a failed job + +This hypothetical example might happen with a foreign key like: + +```sql +ALTER TABLE ONLY vulnerability_occurrence_pipelines + ADD CONSTRAINT fk_rails_6421e35d7d FOREIGN KEY (pipeline_id) REFERENCES ci_pipelines(id) ON DELETE CASCADE; +``` + +In this example we expect to delete all associated `vulnerability_occurrence_pipelines` records +whenever we delete the `ci_pipelines` record associated with them. In this case +you might end up with a Sidekiq worker that is responsible for processing a +vulnerability and looping over all occurrences causing a Sidekiq job to fail if +it executes `occurrence.pipeline.created_at`. + +**Mitigation** + +When looping through the vulnerability occurrences in the Sidekiq worker, we +could try to load the corresponding pipeline and choose to skip processing that +occurrence if pipeline is not found. diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index a17ad798305..1338e83070f 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -15,10 +15,46 @@ To scale GitLab, the we are database for CI/CD tables was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64289) in GitLab 14.1. This feature is still under development, and is not ready for production use. +### Development setup + By default, GitLab is configured to use only one main database. To opt-in to use a main database, and CI database, modify the `config/database.yml` file to have a `main` and a `ci` database -configurations. For example, given a `config/database.yml` like below: +configurations. + +You can set this up using [GDK](#gdk-configuration) or by +[manually configuring `config/database.yml`](#manually-set-up-the-cicd-database). + +#### GDK configuration + +If you are using GDK, you can follow the following steps: + +1. On the GDK root directory, run: + + ```shell + gdk config set gitlab.rails.multiple_databases true + ``` + +1. Open your `gdk.yml`, and confirm that it has the following lines: + + ```yaml + gitlab: + rails: + multiple_databases: true + ``` + +1. Reconfigure GDK: + + ```shell + gdk reconfigure + ``` + +1. [Create the new CI/CD database](#create-the-new-database). + +#### Manually set up the CI/CD database + +You can manually edit `config/database.yml` to split the databases. +To do so, consider a `config/database.yml` file like the example below: ```yaml development: @@ -44,7 +80,7 @@ test: &test statement_timeout: 120s ``` -Edit the `config/database.yml` to look like this: +Edit it to split the databases into `main` and `ci`: ```yaml development: @@ -88,6 +124,25 @@ test: &test statement_timeout: 120s ``` +Next, [create the new CI/CD database](#create-the-new-database). + +#### Create the new database + +After configuring GitLab for the two databases, create the new CI/CD database: + +1. Create the new `ci:` database, load the DB schema into the `ci:` database, + and run any pending migrations: + + ```shell + bundle exec rails db:create db:schema:load:ci db:migrate + ``` + +1. Restart GDK: + + ```shell + gdk restart + ``` + <!-- NOTE: The `validate_cross_joins!` method in `spec/support/database/prevent_cross_joins.rb` references the following heading in the code, so if you make a change to this heading, make sure to update @@ -557,11 +612,31 @@ Don't hesitate to reach out to the [sharding group](https://about.gitlab.com/handbook/engineering/development/enablement/sharding/) for advice. +##### Avoid `dependent: :nullify` and `dependent: :destroy` across databases + +There may be cases where we want to use `dependent: :nullify` or `dependent: :destroy` +across databases. This is technically possible, but it's problematic because +these hooks run in the context of an outer transaction from the call to +`#destroy`, which creates a cross-database transaction and we are trying to +avoid that. Cross-database transactions caused this way could lead to confusing +outcomes when we switch to decomposed, because now you have some queries +happening outside the transaction and they may be partially applied while the +outer transaction fails, which could lead to surprising bugs. + +If you need to do some cleanup after a `destroy` you will need to choose +from some of the options above. If all you need to do is cleanup the child +records themselves from PostgreSQL then you could consider using ["loose foreign +keys"](loose_foreign_keys.md). + ## `config/database.yml` -GitLab will support running multiple databases in the future, for example to [separate tables for the continuous integration features](https://gitlab.com/groups/gitlab-org/-/epics/6167) from the main database. In order to prepare for this change, we [validate the structure of the configuration](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67877) in `database.yml` to ensure that only known databases are used. +GitLab is adding support to run multiple databases, for example to +[separate tables for the continuous integration features](https://gitlab.com/groups/gitlab-org/-/epics/6167) +from the main database. In order to prepare for this change, we +[validate the structure of the configuration](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67877) +in `database.yml` to ensure that only known databases are used. -Previously, the `config/database.yml` would look like this: +Previously, the `config/database.yml` looked like this: ```yaml production: @@ -571,15 +646,16 @@ production: ... ``` -With the support for many databases the support for this -syntax is deprecated and will be removed in [15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/338182). +With the support for many databases this +syntax is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/338182) +and will be removed in [15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/338182). The new `config/database.yml` needs to include a database name to define a database configuration. Only `main:` and `ci:` database -names are supported today. The `main:` needs to always be a first +names are supported. The `main:` database must always be a first entry in a hash. This change applies to decomposed and non-decomposed -change. If an invalidate or deprecated syntax is used the error -or warning will be printed during application start. +change. If an invalid or deprecated syntax is used the error +or warning is printed during application start. ```yaml # Non-decomposed database @@ -603,3 +679,15 @@ production: database: gitlabhq_production_ci ... ``` + +## Foreign keys that cross databases + +There are many places where we use foreign keys that reference across the two +databases. This is not possible to do with two separate PostgreSQL +databases, so we need to replicate the behavior we get from PostgreSQL in a +performant way. We can't, and shouldn't, try to replicate the data guarantees +given by PostgreSQL which prevent creating invalid references, but we still need a +way to replace cascading deletes so we don't end up with orphaned data +or records that point to nowhere, which might lead to bugs. As such we created +["loose foreign keys"](loose_foreign_keys.md) which is an asynchronous +process of cleaning up orphaned records. diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index 7c17a39746e..426d355bd82 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -115,7 +115,7 @@ Use these instructions for exploring the GitLab database while developing with t 1. **Use an SSL connection?** This depends on your installation. Options are: - **Use Secure Connection** - **Standard Connection** (default) - 1. **(Optional) The database to connect to**: `gitlabhq_development`. + 1. **Optional. The database to connect to**: `gitlabhq_development`. 1. **The display name for the database connection**: `gitlabhq_development`. Your database connection should now be displayed in the PostgreSQL Explorer pane and diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index f8ee29e6904..27c29a1ed7c 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -23,7 +23,9 @@ deprecated. A feature can be deprecated at any time, provided there is a viable alternative. -Deprecations should be announced via [release posts](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations). +Deprecations should be announced on the [Deprecated feature removal schedule](../../update/deprecations.md). + +For steps to create a deprecation entry, see [Deprecations](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations). ## When can a feature be removed/changed? diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 8fc6f2e2641..6a618e47211 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -58,7 +58,7 @@ FLAG: If needed, you can add this sentence: -`You should not use this feature for production environments.` +`The feature is not ready for production use.` ## Add version history text @@ -109,7 +109,7 @@ And, when the feature is done and fully available to all users: ```markdown > - Introduced in GitLab 13.7 [with a flag](../../administration/feature_flags.md) named `forti_token_cloud`. Disabled by default. -> - [Enabled on self-managed](https://gitlab.com/issue/etc) GitLab 13.8. +> - [Enabled on self-managed](https://gitlab.com/issue/etc) in GitLab 13.8. > - [Enabled on GitLab.com](https://gitlab.com/issue/etc) in GitLab 13.9. > - [Generally available](issue-link) in GitLab 14.0. [Feature flag <flag name>](issue-link) removed. ``` diff --git a/doc/development/documentation/img/manual_build_docs_v14_3.png b/doc/development/documentation/img/manual_build_docs_v14_3.png Binary files differdeleted file mode 100644 index e366a2f7ec4..00000000000 --- a/doc/development/documentation/img/manual_build_docs_v14_3.png +++ /dev/null diff --git a/doc/development/documentation/img/manual_build_docs_v14_6.png b/doc/development/documentation/img/manual_build_docs_v14_6.png Binary files differnew file mode 100644 index 00000000000..731bda3dd56 --- /dev/null +++ b/doc/development/documentation/img/manual_build_docs_v14_6.png diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index 03980a42381..5dc627c93e9 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -1,9 +1,6 @@ --- -type: reference, dev -stage: none -group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" -description: "Writing styles, markup, formatting, and other standards for the GitLab RESTful APIs." +info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-other-projects-and-subjects. +description: 'Writing styles, markup, formatting, and other standards for the GitLab RESTful APIs.' --- # RESTful API @@ -30,6 +27,10 @@ In the Markdown doc for a resource (AKA endpoint): - Every method must have a detailed [description of the parameters](#method-description). - Every method must have a cURL example. - Every method must have a response body (in JSON format). +- If an attribute is available only to higher level tiers than the other + parameters, add the appropriate inline [tier badge](styleguide/index.md#product-tier-badges). + Put the badge in the **Attribute** column, like the + `**(<tier>)**` code in the following template. ## API topic template @@ -49,12 +50,12 @@ METHOD /endpoint Supported attributes: -| Attribute | Type | Required | Description | -| :---------- | :------- | :--------------------- | :-------------------- | -| `attribute` | datatype | **{check-circle}** Yes | Detailed description. | -| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | -| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | -| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | +| Attribute | Type | Required | Description | +|:-------------------------|:---------|:-----------------------|:----------------------| +| `attribute` | datatype | **{check-circle}** Yes | Detailed description. | +| `attribute` **(<tier>)** | datatype | **{dotted-circle}** No | Detailed description. | +| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | +| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | Example request: @@ -83,20 +84,20 @@ always be in code blocks using backticks (`` ` ``). Sort the attributes in the table: first, required, then alphabetically. ```markdown -| Attribute | Type | Required | Description | -| :------------- | :------------ | :--------------------- | :--------------------------------------------------- | -| `user` | string | **{check-circle}** Yes | The GitLab username. | -| `assignee_ids` | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | -| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | +| Attribute | Type | Required | Description | +|:-----------------------------|:--------------|:-----------------------|:-----------------------------------------------------| +| `user` | string | **{check-circle}** Yes | The GitLab username. | +| `assignee_ids` **(PREMIUM)** | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | +| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | ``` Rendered example: -| Attribute | Type | Required | Description | -| :------------- | :------------ | :--------------------- | :--------------------------------------------------- | -| `user` | string | **{check-circle}** Yes | The GitLab username. | -| `assignee_ids` | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | -| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | +| Attribute | Type | Required | Description | +|:-----------------------------|:--------------|:-----------------------|:-----------------------------------------------------| +| `user` | string | **{check-circle}** Yes | The GitLab username. | +| `assignee_ids` **(PREMIUM)** | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | +| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | ## cURL commands @@ -109,7 +110,7 @@ Rendered example: username and password. | Methods | Description | -| :---------------------------------------------- | :----------------------------------------------------- | +|:------------------------------------------------|:-------------------------------------------------------| | `--header "PRIVATE-TOKEN: <your_access_token>"` | Use this method as is, whenever authentication needed. | | `--request POST` | Use this method when creating new objects | | `--request PUT` | Use this method when updating existing objects | diff --git a/doc/development/documentation/review_apps.md b/doc/development/documentation/review_apps.md index a5094ea87f0..4b58778a20c 100644 --- a/doc/development/documentation/review_apps.md +++ b/doc/development/documentation/review_apps.md @@ -26,7 +26,7 @@ to render and preview the documentation locally. If a merge request has documentation changes, use the `review-docs-deploy` manual job to deploy the documentation review app for your merge request. - + The `review-docs-deploy*` job triggers a cross project pipeline and builds the docs site with your changes. When the pipeline finishes, the review app URL diff --git a/doc/development/documentation/site_architecture/deployment_process.md b/doc/development/documentation/site_architecture/deployment_process.md index c038ee96dbf..25bc699c9d4 100644 --- a/doc/development/documentation/site_architecture/deployment_process.md +++ b/doc/development/documentation/site_architecture/deployment_process.md @@ -50,8 +50,8 @@ that stable documentation and deploys it to the registry. For example: - [13.12 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/395365202). - [12.10 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/395365405). -In particular, the [`image:docs-single` job](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4c18963fe0a414ad62f55b9e18f922588b2dd155/.gitlab-ci.yml#L655) in each pipeline -takes what is built, and pushes it to the [container registry](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635). +In particular, the [`image:docs-single` job](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4c18963fe0a414ad62f55b9e18f922588b2dd155/.gitlab-ci.yml#L655) in each pipeline runs automatically. +It takes what is built, and pushes it to the [container registry](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635). ```mermaid graph TD @@ -91,6 +91,7 @@ The [`image:docs-latest` job](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4 - Pulls the latest documentation from the default branches of the relevant upstream projects. - Pulls the Docker images previously built by the `image:docs-single` jobs. +- Must be run manually on a scheduled pipeline. For example, [a pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/399233948) containing the [`image:docs-latest` job](https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/1733948330): @@ -122,7 +123,8 @@ graph TD for it must be deployed to become available. The [`pages`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4c18963fe0a414ad62f55b9e18f922588b2dd155/.gitlab-ci.yml#L491) -job runs the necessary commands to combine: +job runs automatically when a pipeline runs on the default branch (`main`). +It runs the necessary commands to combine: - A very up-to-date build of the `gitlab-docs` site code. - The latest docs from the default branches of the upstream projects. diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index fac83af89f4..6ecffce01b4 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -88,7 +88,7 @@ Create an issue when you want to track bugs or future work. Prerequisites: -- You must have at least the Developer role for a project. +- You must have at least the Developer role for the project. To create an issue: diff --git a/doc/development/documentation/styleguide/img/callouts.png b/doc/development/documentation/styleguide/img/callouts.png Binary files differnew file mode 100644 index 00000000000..b84e9e269dc --- /dev/null +++ b/doc/development/documentation/styleguide/img/callouts.png diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 1382ec263f2..7f2e5a1ba26 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -58,8 +58,7 @@ it was originally composed for, if it is helpful to any of our audiences, we can include it. - If you use an image that has a separate source file (for example, a vector or - diagram format), link the image to the source file so that it may be reused or - updated by anyone. + diagram format), link the image to the source file so that anyone can update or reuse it. - Do not copy and paste content from other sources unless it is a limited quotation with the source cited. Typically it is better to either rephrase relevant information in your own words or link out to the other source. @@ -113,8 +112,9 @@ The more we reflexively add information to the documentation, the more the documentation helps others efficiently accomplish tasks and solve problems. If you have questions when considering, authoring, or editing documentation, ask -the Technical Writing team. They're available on Slack in `#docs` or in GitLab by mentioning the -writer for the applicable [DevOps stage](https://about.gitlab.com/handbook/product/categories/#devops-stages). +the Technical Writing team. They're available on Slack in `#docs` or in GitLab by +mentioning [the writer for](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) +the applicable [DevOps stage or group](https://about.gitlab.com/handbook/product/categories/#devops-stages). Otherwise, forge ahead with your best effort. It does not need to be perfect; the team is happy to review and improve upon your content. Review the [Documentation guidelines](index.md) before you begin your first documentation MR. @@ -787,10 +787,13 @@ This is overridden by the [documentation-specific punctuation rules](#punctuatio - When possible, avoid including words that might change in the future. Changing a heading changes its anchor URL, which affects other linked pages. - When introducing a new document, be careful for the headings to be - grammatically and syntactically correct. Mention an [assigned technical writer (TW)](https://about.gitlab.com/handbook/product/categories/) - for review. + grammatically and syntactically correct. Mention an [assigned technical writer (TW)](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) + for review, based upon the [product category](https://about.gitlab.com/handbook/product/categories/). This is to ensure that no document with wrong heading is going live without an audit, thus preventing dead links and redirection issues when corrected. +- Use the context provided by parent section headings. That is, don't repeat the parent heading's text in each + subsection's heading. +- Use articles and prepositions in headings where it would make sense in regular text. - Leave exactly one blank line before and after a heading. - Do not use links in headings. - Add the corresponding [product badge](#product-tier-badges) according to the tier the @@ -1053,7 +1056,9 @@ Guidance for each individual UI element is in [the word list](word_list.md). ### How to write navigation task steps -To be consistent, use this format when you write navigation steps in a task topic. +To be consistent, use these templates when you write navigation steps in a task topic. + +To open project settings: ```markdown 1. On the top bar, select **Menu > Projects** and find your project. @@ -1061,7 +1066,7 @@ To be consistent, use this format when you write navigation steps in a task topi 1. Expand **General pipelines**. ``` -Another example: +To open group settings: ```markdown 1. On the top bar, select **Menu > Groups** and find your group. @@ -1069,7 +1074,7 @@ Another example: 1. Expand **General pipelines**. ``` -An Admin Area example: +To open the Admin Area: ```markdown 1. On the top bar, select **Menu > Admin**. @@ -1081,6 +1086,15 @@ To select your avatar: 1. On the top bar, in the top right corner, select your avatar. ``` +To save the selection in some dropdown lists: + +```markdown +1. Go to your issue. +1. On the right sidebar, in the **Iteration** section, select **Edit**. +1. From the dropdown list, select the iteration to associate this issue with. +1. Select any area outside the dropdown list. +``` + ### Optional steps If a step is optional, start the step with the word `Optional` followed by a period. @@ -1150,6 +1164,17 @@ review app in the merge request. Make sure the image isn't blurry or overwhelmin - **Be consistent.** Coordinate screenshots with the other screenshots already on a documentation page for a consistent reading experience. +### Add callouts + +If you need to emphasize an area in a screenshot, use an arrow. + +- For color, use `#EE2604`. If you use the Preview application on macOS, this is the default red. +- For the line width, use 3 pt. If you use the Preview application on macOS, this is the third line in the list. +- Use the arrow style shown in the following image. +- If you have multiple arrows, make them parallel when possible. + + + ### Save the image - Resize any wide or tall screenshots if needed, but make sure the screenshot is @@ -1575,7 +1600,7 @@ users be aware of recent improvements or additions. The GitLab Technical Writing team determines which versions of documentation to display on this site based on the GitLab -[Statement of Support](https://about.gitlab.com/support/statement-of-support.html#we-support-the-current-major-version-and-the-two-previous-major-versions). +[Statement of Support](https://about.gitlab.com/support/statement-of-support.html#version-support). ### View older GitLab documentation versions @@ -1628,6 +1653,13 @@ This feature does something. This feature does something else. ``` +If you're documenting elements of a feature, start with the feature name or a gerund: + +```markdown +> - Notifications for expiring tokens [introduced](<link-to-issue>) in GitLab 11.3. +> - Creating an issue from an issue board [introduced](<link-to-issue>) in GitLab 13.1. +``` + If a feature is moved to another tier: ```markdown @@ -1781,7 +1813,9 @@ after the heading text. For example: # Heading title **(FREE)** ``` -Do not add tier badges inline with other text. The single source of truth for a feature should be the heading where the functionality is described. +Do not add tier badges inline with other text, except for [API attributes](../restful_api_styleguide.md). +The single source of truth for a feature should be the heading where the +functionality is described. #### Available product tier badges diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index 595dab09bf5..9c375379685 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -5,9 +5,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w description: 'Writing styles, markup, formatting, and other standards for GitLab Documentation.' --- -# A-Z word list +# Recommended word list -To help ensure consistency in the documentation, follow this guidance. +To help ensure consistency in the documentation, the Technical Writing team +recommends these wording choices. The GitLab handbook also maintains a list of +[top misused terms](https://about.gitlab.com/handbook/communication/top-misused-terms/). For guidance not on this page, we defer to these style guides: @@ -20,7 +22,7 @@ For guidance not on this page, we defer to these style guides: ## `@mention` Try to avoid **`@mention`**. Say **mention** instead, and consider linking to the -[mentions topic](../../../user/project/issues/issue_data_and_actions.md#mentions). +[mentions topic](../../../user/discussions/index.md#mentions). Don't use backticks. ## above @@ -31,9 +33,14 @@ Try to avoid using **above** when referring to an example or table in a document Do not use **above** when referring to versions of the product. Use [**later**](#later) instead. -- Do: In GitLab 14.4 and later... -- Do not: In GitLab 14.4 and above... -- Do not: In GitLab 14.4 and higher... +Use: + +- In GitLab 14.4 and later... + +Instead of: + +- In GitLab 14.4 and above... +- In GitLab 14.4 and higher... ## access level @@ -47,16 +54,21 @@ Capitalize these words when you refer to the UI. Otherwise use lowercase. Use **administrator** instead of **admin** when talking about a user's access level. Use lowercase unless you are referring to the **Admin** access level you select in the UI. -To view the administrator access type, in the GitLab UI, go to the Admin Area and select +To view the administrator access level, in the GitLab UI, go to the Admin Area and select **Users**. Then select **New user**.  An **administrator** is not a [role](#roles) or [permission](#permissions). -- Do: To do this thing, you must be an administrator. -- Do: To do this thing, you must have the administrator access level. -- Do not: To do this thing, you must have the Admin role. +Use: + +- To do this thing, you must be an administrator. +- To do this thing, you must have the administrator access level. + +Instead of: + +- To do this thing, you must have the Admin role. ## Admin Area @@ -65,10 +77,16 @@ This area of the UI says **Admin Area** at the top of the page and on the menu. ## allow, enable -Try to avoid **allow** and **enable**, unless you are talking about security-related features. For example: +Try to avoid **allow** and **enable**, unless you are talking about security-related features. -- Do: Use this feature to create a pipeline. -- Do not: This feature allows you to create a pipeline. +Use: + +- You can add a file to your repository. + +Instead of: + +- This feature allows you to add a file to your repository. +- This feature enables users to add files to their repository. This phrasing is more active and is from the user perspective, rather than the person who implemented the feature. [View details in the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/a/allow-allows). @@ -124,8 +142,13 @@ Use **text box** to refer to the UI field. Do not use **field** or **box**. For Don't use a descriptor with **button**. -- Do: Select **Run pipelines**. -- Do not: Select the **Run pipelines** button. +Use: + +- Select **Run pipelines**. + +Instead of: + +- Select the **Run pipelines** button. ## cannot, can not @@ -148,6 +171,13 @@ If you must refer to the checkbox, you can say it is selected or cleared. For ex - Ensure the **Protect environment** checkbox is cleared. - Ensure the **Protect environment** checkbox is selected. +## checkout, check out + +Use **check out** as a verb. For the Git command, use `checkout`. + +- Use `git checkout` to check out a branch locally. +- Check out the files you want to edit. + ## CI/CD CI/CD is always uppercase. No need to spell it out on first use. @@ -185,8 +215,8 @@ When writing about the Developer role: - Do not use the phrase, **if you are a developer** to mean someone who is assigned the Developer role. Instead, write it out. For example, **if you are assigned the Developer role**. - To describe a situation where the Developer role is the minimum required: - - Do: at least the Developer role - - Do not: the Developer role or higher + - Use: at least the Developer role + - Instead of: the Developer role or higher Do not use **Developer permissions**. A user who is assigned the Developer role has a set of associated permissions. @@ -208,8 +238,13 @@ For example: Use **earlier** when talking about version numbers. -- Do: In GitLab 14.1 and earlier. -- Do not: In GitLab 14.1 and lower. +Use: + +- In GitLab 14.1 and earlier. + +Instead of: + +- In GitLab 14.1 and lower. ## easily @@ -245,8 +280,13 @@ Use lowercase for **epic board**. Try to avoid **etc.**. Be as specific as you can. Do not use [**and so on**](#and-so-on) as a replacement. -- Do: You can update objects, like merge requests and issues. -- Do not: You can update objects, like merge requests, issues, etc. +Use: + +- You can update objects, like merge requests and issues. + +Instead of: + +- You can update objects, like merge requests, issues, etc. ## expand @@ -256,8 +296,13 @@ Use **expand** instead of **open** when you are talking about expanding or colla Use **box** instead of **field** or **text box**. -- Do: In the **Variable name** box, enter `my text`. -- Do not: In the **Variable name** field, enter `my text`. +Use: + +- In the **Variable name** box, enter `my text`. + +Instead of: + +- In the **Variable name** field, enter `my text`. However, you can make an exception when you are writing a task and you need to refer to all of the fields at once. For example: @@ -311,8 +356,8 @@ When writing about the Guest role: - Do not use the phrase, **if you are a guest** to mean someone who is assigned the Guest role. Instead, write it out. For example, **if you are assigned the Guest role**. - To describe a situation where the Guest role is the minimum required: - - Do: at least the Guest role - - Do not: the Guest role or higher + - Use: at least the Guest role + - Instead of: the Guest role or higher Do not use **Guest permissions**. A user who is assigned the Guest role has a set of associated permissions. @@ -328,16 +373,26 @@ Do not use **high availability** or **HA**. Instead, direct readers to the GitLa Do not use **higher** when talking about version numbers. -- Do: In GitLab 14.4 and later... -- Do not: In GitLab 14.4 and higher... -- Do not: In GitLab 14.4 and above... +Use: + +- In GitLab 14.4 and later... + +Instead of: + +- In GitLab 14.4 and higher... +- In GitLab 14.4 and above... ## hit Don't use **hit** to mean **press**. -- Do: Press **ENTER**. -- Do not: Hit the **ENTER** button. +Use: + +- Press **ENTER**. + +Instead of: + +- Hit the **ENTER** button. ## I @@ -369,13 +424,31 @@ Do not use **build** to be synonymous with **job**. A job is defined in the `.gi If you want to use **CI** with the word **job**, use **CI/CD job** rather than **CI job**. +## Kubernetes executor + +GitLab Runner can run jobs on a Kubernetes cluster. To do this, GitLab Runner uses the Kubernetes executor. + +When referring to this feature, use: + +- Kubernetes executor for GitLab Runner +- Kubernetes executor + +Do not use: + +- GitLab Runner Kubernetes executor, because this can infringe on the Kubernetes trademark. + ## later Use **later** when talking about version numbers. -- Do: In GitLab 14.1 and later... -- Do not: In GitLab 14.1 and higher... -- Do not: In GitLab 14.1 and above... +Use: + +- In GitLab 14.1 and later... + +Instead of: + +- In GitLab 14.1 and higher... +- In GitLab 14.1 and above... ## list @@ -390,8 +463,13 @@ Do not use **log in** or **log on**. Use [sign in](#sign-in) instead. If the use Do not use **lower** when talking about version numbers. -- Do: In GitLab 14.1 and earlier. -- Do not: In GitLab 14.1 and lower. +Use: + +- In GitLab 14.1 and earlier. + +Instead of: + +- In GitLab 14.1 and lower. ## Maintainer @@ -402,8 +480,8 @@ When writing about the Maintainer role: - Do not use the phrase, **if you are a maintainer** to mean someone who is assigned the Maintainer role. Instead, write it out. For example, **if you are assigned the Maintainer role**. - To describe a situation where the Maintainer role is the minimum required: - - Do: at least the Maintainer role - - Do not: the Maintainer role or higher + - Use: at least the Maintainer role + - Instead of: the Maintainer role or higher Do not use **Maintainer permissions**. A user who is assigned the Maintainer role has a set of associated permissions. @@ -446,8 +524,14 @@ Do not use **navigate**. Use **go** instead. For example: Try to avoid **needs to**, because it's wordy. Avoid **should** when you can be more specific. If something is required, use **must**. -- Do: You must set the variable. Or: Set the variable. -- Do not: You need to set the variable. +Use: + +- You must set the variable. +- Set the variable. + +Instead of: + +- You need to set the variable. **Should** is acceptable for recommended actions or items, or in cases where an event may not happen. For example: @@ -461,22 +545,63 @@ happen. For example: Do not use **note that** because it's wordy. -- Do: You can change the settings. -- Do not: Note that you can change the settings. +Use: + +- You can change the settings. + +Instead of: + +- Note that you can change the settings. ## on When documenting how to select high-level UI elements, use the word **on**. -- Do: `On the left sidebar...` +Use: + +- `On the left sidebar...` + +Instead of: + - Do not: `From the left sidebar...` or `In the left sidebar...` ## once The word **once** means **one time**. Don't use it to mean **after** or **when**. -- Do: When the process is complete... -- Do not: Once the process is complete... +Use: + +- When the process is complete... + +Instead of: + +- Once the process is complete... + +## only + +Put the word **only** next to the word it modifies. + +- You can create only private projects. + +In this example, **only** modifies the noun **projects**. The sentence means you can create one type of project--a private project. + +- You can only create private projects. + +In this example, **only** modifies the verb **create**. This sentence means that you can't perform other actions, +like deleting private projects, or adding users to them. + +## override + +Use **override** to indicate temporary replacement. + +For example, a value might be overridden when a job runs. The +original value does not change. + +## overwrite + +Use **overwrite** to indicate permanent replacement. + +For example, a log file might overwrite a log file of the same name. ## Owner @@ -518,8 +643,8 @@ When writing about the Reporter role: - Do not use the phrase, **if you are a reporter** to mean someone who is assigned the Reporter role. Instead, write it out. For example, **if you are assigned the Reporter role**. - To describe a situation where the Reporter role is the minimum required: - - Do: at least the Reporter role - - Do not: the Reporter role or higher + - Use: at least the Reporter role + - Instead of: the Reporter role or higher Do not use **Reporter permissions**. A user who is assigned the Reporter role has a set of associated permissions. @@ -541,8 +666,13 @@ Use lowercase for **runners**. These are the agents that run CI/CD jobs. See als Do not use **(s)** to make a word optionally plural. It can slow down comprehension. For example: -- Do: Select the jobs you want. -- Do not: Select the job(s) you want. +Use: + +- Select the jobs you want. + +Instead of: + +- Select the job(s) you want. If you can select multiples of something, then write the word as plural. @@ -564,7 +694,12 @@ into separate areas, refer to these areas as sections. We often think of expandable/collapsible areas as **sections**. When you refer to expanding or collapsing a section, don't include the word **section**. -- Do: Expand **Auto DevOps**. +Use: + +- Expand **Auto DevOps**. + +Instead of: + - Do not: Expand the **Auto DevOps** section. ## select @@ -597,8 +732,13 @@ Do not use **simply** or **simple**. If the user doesn't find the process to be The word **since** indicates a timeframe. For example, **Since 1984, Bon Jovi has existed**. Don't use **since** to mean **because**. -- Do: Because you have the Developer role, you can delete the widget. -- Do not: Since you have the Developer role, you can delete the widget. +Use: + +- Because you have the Developer role, you can delete the widget. + +Instead of: + +- Since you have the Developer role, you can delete the widget. ## slashes @@ -616,8 +756,13 @@ Use **subgroup** (no hyphen) instead of **sub-group**. ([Vale](../testing.md#val Do not use **that** when describing a noun. For example: -- Do: The file you save... -- Do not: The file **that** you save... +Use: + +- The file you save... + +Instead of: + +- The file **that** you save... See also [this, these, that, those](#this-these-that-those). @@ -636,8 +781,13 @@ Use **text box** instead of **field** or **box** when referring to the UI elemen Try to avoid **there is** and **there are**. These phrases hide the subject. -- Do: The bucket has holes. -- Do not: There are holes in the bucket. +Use: + +- The bucket has holes. + +Instead of: + +- There are holes in the bucket. ## they @@ -649,17 +799,17 @@ a gender-neutral pronoun. Always follow these words with a noun. For example: -- Do: **This setting** improves performance. -- Do not: **This** improves performance. +- Use: **This setting** improves performance. +- Instead of: **This** improves performance. -- Do: **These pants** are the best. -- Do not: **These** are the best. +- Use: **These pants** are the best. +- Instead of: **These** are the best. -- Do: **That droid** is the one you are looking for. -- Do not: **That** is the one you are looking for. +- Use: **That droid** is the one you are looking for. +- Instead of: **That** is the one you are looking for. -- Do: **Those settings** need to be configured. (Or even better, **Configure those settings.**) -- Do not: **Those** need to be configured. +- Use: **Those settings** need to be configured. (Or even better, **Configure those settings.**) +- Instead of: **Those** need to be configured. ## to-do item @@ -683,6 +833,19 @@ Do not use **type** if you can avoid it. Use **enter** instead. Do not use **useful**. If the user doesn't find the process to be useful, we lose their trust. ([Vale](../testing.md#vale) rule: [`Simplicity.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Simplicity.yml)) +## user, users + +When possible, address the reader directly, instead of calling them **users**. +Use the [second person](#you-your-yours), **you**, instead. + +Use: + +- You can configure a pipeline. + +Instead of: + +- Users can configure a pipeline. + ## utilize Do not use **utilize**. Use **use** instead. It's more succinct and easier for non-native English speakers to understand. @@ -700,8 +863,13 @@ Do not use Latin abbreviations. Use **with**, **through**, or **by using** inste Try to avoid **we** and focus instead on how the user can accomplish something in GitLab. -- Do: Use widgets when you have work you want to organize. -- Do not: We created a feature for you to add widgets. +Use: + +- Use widgets when you have work you want to organize. + +Instead of: + +- We created a feature for you to add widgets. One exception: You can use **we recommend** instead of **it is recommended** or **GitLab recommends**. ([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) @@ -709,5 +877,18 @@ One exception: You can use **we recommend** instead of **it is recommended** or Do not use **whitelist**. Another option is **allowlist**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) +## you, your, yours + +Use **you**, **your**, and **yours** instead of [**the user** and **the user's**](#user-users). +Documentation should be from the [point of view](https://design.gitlab.com/content/voice-tone#point-of-view) of the reader. + +Use: + +- You can configure a pipeline. + +Instead of: + +- Users can configure a pipeline. + <!-- vale on --> <!-- markdownlint-enable --> diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index 782cd3411b1..49ad51874e3 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -4,66 +4,44 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Documentation process +# How to update GitLab documentation -The process for creating and maintaining GitLab product documentation allows -anyone to contribute a merge request or create an issue for GitLab -documentation. - -Documentation updates relating to new features or feature enhancements must -use the [feature workflow process](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#for-a-product-change) described in the GitLab Handbook. - -## Who updates the docs? - -*Anyone* can contribute! You can create a merge request for documentation when: +Anyone can contribute to the GitLab documentation! You can create a merge request for documentation when: - You find errors or other room for improvement in existing documentation. - You have an idea for all-new documentation that would help a GitLab user or administrator to accomplish their work with GitLab. -## Documentation labels - -Regardless of the type of issue or merge request, certain labels are required when documentation -is added or updated. The following are added by the issue or merge request author: - -- An appropriate [type label](../contributing/issue_workflow.md#type-labels). -- The [stage label](../contributing/issue_workflow.md#stage-labels) and - [group label](../contributing/issue_workflow.md#group-labels). For example, `~devops::create` and - `~group::source code`. -- The `~documentation` [specialization label](../contributing/issue_workflow.md#specialization-labels). - -The following are also added by members of the Technical Writing team: - -- A documentation [scoped label](../../user/project/labels.md#scoped-labels) with the - `docs::` prefix. For example, `~docs::improvement`. -- The `~Technical Writing` [team label](../contributing/issue_workflow.md#team-labels). - -Documentation changes that are not associated with the release of a new or updated feature -do not take the `~"type::feature"` label, but still need the `~documentation` label. - -They may include: - -- Documentation created or updated to improve accuracy, completeness, ease of use, or any reason - other than a [feature change](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#for-a-product-change). -- Addressing gaps in existing documentation, or making improvements to existing documentation. -- Work on special projects related to the documentation. +If you are working on a feature or enhancement, use the +[feature workflow process described in the GitLab Handbook](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#for-a-product-change). ## How to update the docs -To update GitLab documentation: - -1. Either: - - Click the **Edit this Page** link at the bottom of any page on <https://docs.gitlab.com>. - - Navigate to one of the repositories and documentation paths listed on the - [GitLab Documentation guidelines](index.md) page. -1. Follow the described standards and processes listed on the page, including: - - The [Structure and template](structure.md) page. - - The [Style Guide](styleguide/index.md). - - The [Markdown Guide](https://about.gitlab.com/handbook/markdown-guide/). -1. Follow the [Merge Request Guidelines](../contributing/merge_request_workflow.md#merge-request-guidelines). - -NOTE: -Work in a fork if you do not have the Developer role in the GitLab project. +If you are not a GitLab team member, or do not have the Developer role for the GitLab repository, to update GitLab documentation: + +1. Select an issue you'd like to work on. + - You don't need an issue to open a merge request. + - For a Hackathon, in the issue, in a comment, mention the person who opened the issue and ask for the issue to be assigned to you. + To be fair to other contributors, if you see someone has already asked to work on the issue, choose another issue. + If you are looking for issues to work on and don't see any that suit you, you can always fix [Vale](testing.md#vale) issues. +1. Go to the [GitLab repository](https://gitlab.com/gitlab-org/gitlab). +1. In the top-right, select **Fork**. Forking makes a copy of the repository on GitLab.com. +1. In your fork, find the documentation page by going to the `\doc` directory. +1. If you know Git, make your changes and open a merge request. + If not, follow these steps: + 1. In the top right, select **Edit**, make the changes, and **Save**. + 1. From the left menu, select **Merge requests**. + 1. For the source branch, select your fork and branch. If you did not create a branch, select `master`. + For the target branch, select the [GitLab repository](https://gitlab.com/gitlab-org/gitlab) `master` branch. + 1. For the commit message, use 3-5 words, start with a capital letter, and do not end with a period. + 1. Select **Commit changes**. A merge request opens. + 1. Select the **Documentation** template. In the description, write a brief summary of the changes and link to the related issue, if there is one. + +If you need help while working on the page, view: + +- The [Style Guide](styleguide/index.md). +- The [Word list](styleguide/word_list.md) +- The [Markdown Guide](https://about.gitlab.com/handbook/markdown-guide/). ### Ask for help @@ -83,6 +61,22 @@ To identify someone who can help you: If you are a member of the GitLab Slack workspace, you can request help in `#docs`. +## Documentation labels + +When you author an issue or merge request, you must add these labels: + +- A [type label](../contributing/issue_workflow.md#type-labels). +- A [stage label](../contributing/issue_workflow.md#stage-labels) and [group label](../contributing/issue_workflow.md#group-labels). + For example, `~devops::create` and `~group::source code`. +- A `~documentation` [specialization label](../contributing/issue_workflow.md#specialization-labels). + +A member of the Technical Writing team adds these labels: + +- A [documentation scoped label](../../user/project/labels.md#scoped-labels) with the + `docs::` prefix. For example, `~docs::improvement`. +- The [`~Technical Writing` team label](../contributing/issue_workflow.md#team-labels). +- A type label: either `~"type::feature"` or `~"type::maintenance"`. + ### Reviewing and merging Anyone with the [Maintainer role](../../user/permissions.md) to the relevant GitLab project can @@ -130,9 +124,9 @@ immediately after merge by the developer or maintainer. For this, create an issue using the [Doc Review description template](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Doc%20Review) and link to it from the merged merge request that introduced the documentation change. -Circumstances, where a regular pre-merge Technical Writer review might be skipped, include: +Circumstances in which a regular pre-merge Technical Writer review might be skipped include: -- There is a short amount of time left before the milestone release. If less than three +- There is a short amount of time left before the milestone release. If fewer than three days are remaining, seek a post-merge review and ping the writer via Slack to ensure the review is completed as soon as possible. - The size of the change is small and you have a high degree of confidence diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 7f74d9660e9..70d7ea7c1a5 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -1079,6 +1079,48 @@ export default { - **EE extra HTML** - For the templates that have extra HTML in EE we should move it into a new component and use the `ee_else_ce` dynamic import +#### Testing modules using EE/CE aliases + +When writing Frontend tests, if the module under test imports other modules with `ee_else_ce/...` and these modules are also needed by the relevant test, then the relevant test **must** import these modules with `ee_else_ce/...`. This avoids unexpected EE or FOSS failures, and helps ensure the EE behaves like CE when it is unlicensed. + +For example: + +```vue +<script> +// ~/foo/component_under_test.vue + +import FriendComponent from 'ee_else_ce/components/friend.vue;' + +export default { + name: 'ComponentUnderTest', + components: { FriendComponent }. +} +</script> + +<template> + <friend-component /> +</template> +``` + +```javascript +// spec/frontend/foo/component_under_test_spec.js + +// ... +// because we referenced the component using ee_else_ce we have to do the same in the spec. +import Friend from 'ee_else_ce/components/friend.vue;' + +describe('ComponentUnderTest', () => { + const findFriend = () => wrapper.find(Friend); + + it('renders friend', () => { + // This would fail in CE if we did `ee/component...` + // and would fail in EE if we did `~/component...` + expect(findFriend().exists()).toBe(true); + }); +}); + +``` + ### Non Vue Files For regular JS files, the approach is similar. diff --git a/doc/development/event_tracking/backend.md b/doc/development/event_tracking/backend.md deleted file mode 100644 index 3931f0b35ab..00000000000 --- a/doc/development/event_tracking/backend.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' -remove_date: '2021-12-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after 2021-12-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/event_tracking/frontend.md b/doc/development/event_tracking/frontend.md deleted file mode 100644 index 3931f0b35ab..00000000000 --- a/doc/development/event_tracking/frontend.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' -remove_date: '2021-12-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after 2021-12-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/event_tracking/index.md b/doc/development/event_tracking/index.md deleted file mode 100644 index 3931f0b35ab..00000000000 --- a/doc/development/event_tracking/index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' -remove_date: '2021-12-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after 2021-12-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/experiment_guide/gitlab_experiment.md b/doc/development/experiment_guide/gitlab_experiment.md index af4512dcde0..288823bb41f 100644 --- a/doc/development/experiment_guide/gitlab_experiment.md +++ b/doc/development/experiment_guide/gitlab_experiment.md @@ -20,7 +20,7 @@ concepts which may seem confusing or advanced without understanding the underpin of how GitLab uses feature flags in development. One concept: GLEX supports experiments with multiple variants, which are sometimes referred to as A/B/n tests. -The [`gitlab-experiment` project](https://gitlab.com/gitlab-org/gitlab-experiment) +The [`gitlab-experiment` project](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) exists in a separate repository, so it can be shared across any GitLab property that uses Ruby. You should feel comfortable reading the documentation on that project as well if you want to dig into more advanced topics. @@ -394,26 +394,6 @@ You may be asked from time to time to track a specific record ID in experiments. The approach is largely up to the PM and engineer creating the implementation. No recommendations are provided here at this time. -### Record experiment subjects - -Snowplow tracking of identifiable users or groups is prohibited, but you can still -determine if an experiment is successful or not. We're allowed to record the ID of -a namespace, project or user in our database. Therefore, we can tell the experiment -to record their ID together with the assigned experiment variant in the -`experiment_subjects` database table for later analysis. - -For the recording to work, the experiment's context must include a `namespace`, -`group`, `project`, `user`, or `actor`. - -To record the experiment subject when you first assign a variant, call `record!` in -the experiment's block: - -```ruby -experiment(:pill_color, actor: current_user) do |e| - e.record! -end -``` - ## Test with RSpec This gem provides some RSpec helpers and custom matchers. These are in flux as of GitLab 13.10. diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index 1b1f756d4c0..9937cb2ebd1 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -38,13 +38,13 @@ being removed before the [experiment cleanup process](https://about.gitlab.com/h If, as a reviewer or maintainer, you find code that would usually fail review but is acceptable for now, mention your concerns with a note that there's no need to change the code. The author can then add a comment to this piece of code -and link to the issue that resolves the experiment. If the experiment is -successful and becomes part of the product, any follow up issues should be -addressed. +and link to the issue that resolves the experiment. The author or reviewer can add a link to this concern in the +experiment rollout issue under the `Experiment Successful Cleanup Concerns` section of the description. +If the experiment is successful and becomes part of the product, any items that appear under this section will be addressed. ## Implementing an experiment -[`GLEX`](https://gitlab.com/gitlab-org/gitlab-experiment) - or `Gitlab::Experiment`, the `gitlab-experiment` gem - is the preferred option for implementing an experiment in GitLab. +[`GLEX`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) - or `Gitlab::Experiment`, the `gitlab-experiment` gem - is the preferred option for implementing an experiment in GitLab. For more information, see [Implementing an A/B/n experiment using GLEX](gitlab_experiment.md). diff --git a/doc/development/fe_guide/event_tracking.md b/doc/development/fe_guide/event_tracking.md deleted file mode 100644 index 3931f0b35ab..00000000000 --- a/doc/development/fe_guide/event_tracking.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' -remove_date: '2021-12-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after 2021-12-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 44d43a32803..ed71f612061 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -171,6 +171,40 @@ import { getIdFromGraphQLId } from '~/graphql_shared/utils'; const primaryKeyId = getIdFromGraphQLId(data.id); ``` +**It is required** to query global `id` for every GraphQL type that has an `id` in the schema: + +```javascript +query allReleases(...) { + project(...) { + id // Project has an ID in GraphQL schema so should fetch it + releases(...) { + nodes { + // Release has no ID property in GraphQL schema + name + tagName + tagPath + assets { + count + links { + nodes { + id // Link has an ID in GraphQL schema so should fetch it + name + } + } + } + } + pageInfo { + // PageInfo no ID property in GraphQL schema + startCursor + hasPreviousPage + hasNextPage + endCursor + } + } + } +} +``` + ## Immutability and cache updates From Apollo version 3.0.0 all the cache updates need to be immutable. It needs to be replaced entirely @@ -808,11 +842,11 @@ export default { #### Polling and Performance -While the Apollo client has support for simple polling, for performance reasons, our [Etag-based caching](../polling.md) is preferred to hitting the database each time. +While the Apollo client has support for simple polling, for performance reasons, our [ETag-based caching](../polling.md) is preferred to hitting the database each time. -Once the backend is set up, there are a few changes to make on the frontend. +After the ETag resource is set up to be cached from backend, there are a few changes to make on the frontend. -First, get your resource ETag path from the backend. In the example of the pipelines graph, this is called the `graphql_resource_etag`. This will be used to create new headers to add to the Apollo context: +First, get your ETag resource from the backend, which should be in the form of a URL path. In the example of the pipelines graph, this is called the `graphql_resource_etag`, which is used to create new headers to add to the Apollo context: ```javascript /* pipelines/components/graph/utils.js */ @@ -847,7 +881,51 @@ apollo: { }, ``` -Then, because ETags depend on the request being a `GET` instead of GraphQL's usual `POST`, but our default link library does not support `GET` we need to let our default Apollo client know to use a different library. +Here, the apollo query is watching for changes in `graphqlResourceEtag`. If your ETag resource dynamically changes, you should make sure the resource you are sending in the query headers is also updated. To do this, you can store and update the ETag resource dynamically in the local cache. + +You can see an example of this in the pipeline status of the pipeline editor. The pipeline editor watches for changes in the latest pipeline. When the user creates a new commit, we update the pipeline query to poll for changes in the new pipeline. + +```graphql +# pipeline_etag.query.graphql + +query getPipelineEtag { + pipelineEtag @client +} +``` + +```javascript +/* pipeline_editor/components/header/pipeline_status.vue */ + +import getPipelineEtag from '~/pipeline_editor/graphql/queries/client/pipeline_etag.query.graphql'; + +apollo: { + pipelineEtag: { + query: getPipelineEtag, + }, + pipeline: { + context() { + return getQueryHeaders(this.pipelineEtag); + }, + query: getPipelineQuery, + pollInterval: POLL_INTERVAL, + }, +} + +/* pipeline_editor/components/commit/commit_section.vue */ + +await this.$apollo.mutate({ + mutation: commitCIFile, + update(store, { data }) { + const pipelineEtag = data?.commitCreate?.commit?.commitPipelinePath; + + if (pipelineEtag) { + store.writeQuery({ query: getPipelineEtag, data: { pipelineEtag } }); + } + }, +}); +``` + +ETags depend on the request being a `GET` instead of GraphQL's usual `POST`. Our default link library does not support `GET` requests, so we must let our default Apollo client know to use a different library. Keep in mind, this means your app cannot batch queries. ```javascript /* componentMountIndex.js */ @@ -862,10 +940,35 @@ const apolloProvider = new VueApollo({ }); ``` -Keep in mind, this means your app will not batch queries. +Finally, we can add a visibility check so that the component pauses polling when the browser tab is not active. This should lessen the request load on the page. + +```javascript +/* component.vue */ + +import { toggleQueryPollingByVisibility } from '~/pipelines/components/graph/utils'; + +export default { + mounted() { + toggleQueryPollingByVisibility(this.$apollo.queries.pipeline, POLL_INTERVAL); + }, +}; +``` + +You can use [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59672/) as a reference on how to fully implement ETag caching on the frontend. Once subscriptions are mature, this process can be replaced by using them and we can remove the separate link library and return to batching queries. +##### How to test ETag caching + +You can test that your implementation works by checking requests on the network tab. If there are no changes in your ETag resource, all polled requests should: + +- Be `GET` requests instead of `POST` requests. +- Have an HTTP status of `304` instead of `200`. + +Make sure that caching is not disabled in your developer tools when testing. + +If you are using Chrome and keep seeing `200` HTTP status codes, it might be this bug: [Developer tools show 200 instead of 304](https://bugs.chromium.org/p/chromium/issues/detail?id=1269602). In this case, inspect the response headers' source to confirm that the request was actually cached and did return with a `304` status code. + #### Subscriptions We use [subscriptions](https://www.apollographql.com/docs/react/data/subscriptions/) to receive real-time updates from GraphQL API via websockets. Currently, the number of existing subscriptions is limited, you can check a list of available ones in [GraphqiQL explorer](https://gitlab.com/-/graphql-explorer) diff --git a/doc/development/fe_guide/haml.md b/doc/development/fe_guide/haml.md index f905fdad77e..803bb89118c 100644 --- a/doc/development/fe_guide/haml.md +++ b/doc/development/fe_guide/haml.md @@ -59,8 +59,12 @@ When using the GitLab UI form builder, the following components are available fo NOTE: Currently only the listed components are available but more components are planned. +<!-- vale gitlab.Spelling = NO --> + #### gitlab_ui_checkbox_component +<!-- vale gitlab.Spelling = YES --> + [GitLab UI Docs](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-form-form-checkbox--default) | Argument | Description | Type | Required (default value) | @@ -73,8 +77,12 @@ Currently only the listed components are available but more components are plann | `unchecked_value` | Value when checkbox is unchecked. | `String` | `false` (`'0'`) | | `label_options` | Options that are passed to [Rails `label` method](https://api.rubyonrails.org/classes/ActionView/Helpers/FormBuilder.html#method-i-label). | `Hash` | `false` (`{}`) | +<!-- vale gitlab.Spelling = NO --> + #### gitlab_ui_radio_component +<!-- vale gitlab.Spelling = YES --> + [GitLab UI Docs](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-form-form-radio--default) | Argument | Description | Type | Required (default value) | diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 5d5d37e0398..b947d90cc11 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -73,7 +73,7 @@ component, is that you avoid the need to create a fixture or an HTML element in ##### provide/inject Vue supports dependency injection through [provide/inject](https://vuejs.org/v2/api/#provide-inject). -In the component the `inject` configuration accesses the values `provide` passes down. +In the component the `inject` configuration accesses the values `provide` passes down. This example of a Vue app initialization shows how the `provide` configuration passes a value from HAML to the component: ```javascript @@ -308,7 +308,7 @@ This folder holds all components that are specific to this new feature. If you need to use or create a component that is likely to be used somewhere else, please refer to `vue_shared/components`. -A good rule of thumb to know when you should create a component is to think if +A good guideline to know when you should create a component is to think if it could be reusable elsewhere. For example, tables are used in a quite amount of places across GitLab, a table @@ -336,7 +336,7 @@ In the [Vue documentation](https://vuejs.org/v2/api/#Options-Data) the Data func > The data object for the Vue instance. Vue recursively converts its properties into getter/setters to make it "reactive". The object must be plain: native objects such as browser API objects and -prototype properties are ignored. A rule of thumb is that data should just be data - it is not +prototype properties are ignored. A guideline is that data should just be data - it is not recommended to observe objects with their own stateful behavior. Based on the Vue guidance: diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index c6f480deb22..2b783eb21b7 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -31,7 +31,7 @@ Component's computed properties / methods or external helpers. **When to use** -If you are in a Vue app that doesn't use any event hub, try to avoid adding a new one unless absolutely necessary. For example, if you need a child component to react to its parent's event, it's preferred to pass a prop down. Then, use the watch property on that prop in the child component to create the desired side effect. +If you are in a Vue app that doesn't use any event hub, try to avoid adding a new one unless absolutely necessary. For example, if you need a child component to react to its parent's event, it's preferred to pass a prop down. Then, use the watch property on that prop in the child component to create the desired side effect. If you need cross-component communication (between different Vue apps), then perhaps introducing a hub is the right decision. diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index abb100c659e..4843b58c3fd 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -12,7 +12,8 @@ info: "See the Technical Writers assigned to Development Guidelines: https://abo To be able to turn on/off features behind feature flags in any of the GitLab Inc. provided environments such as staging and production, you need to have access to the [ChatOps](../chatops_on_gitlabcom.md) bot. The ChatOps bot -is currently running on the ops instance, which is different from <https://gitlab.com> or <https://dev.gitlab.org>. +is currently running on the ops instance, which is different from +[GitLab.com](https://gitlab.com) or [`dev.gitlab.org`](https://dev.gitlab.org). Follow the ChatOps document to [request access](../chatops_on_gitlabcom.md#requesting-access). @@ -35,12 +36,12 @@ This allows you to separate rolling out a feature from a deploy, making it easier to measure the impact of both separately. The GitLab feature library (using -[Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature -Flags process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) guide) supports rolling out changes to a percentage of +[Flipper](https://github.com/jnunemaker/flipper), and covered in the +[Feature Flags process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) guide) supports rolling out changes to a percentage of time to users. This in turn can be controlled using [GitLab ChatOps](../../ci/chatops/index.md). -For an up to date list of feature flag commands please see [the source -code](https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/commands/feature.rb). +For an up to date list of feature flag commands please see +[the source code](https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/commands/feature.rb). Note that all the examples in that file must be preceded by `/chatops run`. @@ -50,15 +51,16 @@ change feature flags or you do not [have access](#access). ### Enabling a feature for pre-production testing -As a first step in a feature rollout, you should enable the feature on <https://about.staging.gitlab.com> -and <https://dev.gitlab.org>. +As a first step in a feature rollout, you should enable the feature on +[`about.staging.gitlab.com`](https://about.staging.gitlab.com) +and [`dev.gitlab.org`](https://dev.gitlab.org). These two environments have different scopes. `dev.gitlab.org` is a production CE environment that has internal GitLab Inc. traffic and is used for some development and other related work. `staging.gitlab.com` has a smaller subset of GitLab.com database and repositories and does not have regular traffic. Staging is an EE instance and can give you -a (very) rough estimate of how your feature will look/behave on GitLab.com. +a (very) rough estimate of how your feature will look and behave on GitLab.com. Both of these instances are connected to Sentry so make sure you check the projects there for any exceptions while testing your feature after enabling the feature flag. @@ -97,7 +99,7 @@ Guidelines: #### Process When enabling a feature flag rollout, the system will automatically block the -chatops command from succeeding if there are active `"severity::1"` or `~"severity::2"` +ChatOps command from succeeding if there are active `"severity::1"` or `~"severity::2"` incidents or in-progress change issues, for example: ```shell @@ -227,7 +229,7 @@ Note, that if an actor based feature gate is present, switching the `default_enabled` attribute of the YAML definition from `false` to `true` will not have any effect. The feature gate must be deleted first. -For example, a feature flag is set via chatops: +For example, a feature flag is set via ChatOps: ```shell /chatops run feature set --project=gitlab-org/gitlab some_feature true @@ -265,7 +267,7 @@ To disable a feature flag that has been enabled for a specific project you can r You cannot selectively disable feature flags for a specific project/group/user without applying a [specific method of implementing](index.md#selectively-disable-by-actor) the feature flags. -If a feature flag is disabled via chatops, that will take precedence over the `default_enabled` value in the YML. In other words, you could have a feature enabled for on-premise installations but not for GitLab.com. +If a feature flag is disabled via ChatOps, that will take precedence over the `default_enabled` value in the YML. In other words, you could have a feature enabled for on-premise installations but not for GitLab.com. ### Feature flag change logging diff --git a/doc/development/features_inside_dot_gitlab.md b/doc/development/features_inside_dot_gitlab.md index 4631ab3a471..3a9decaad69 100644 --- a/doc/development/features_inside_dot_gitlab.md +++ b/doc/development/features_inside_dot_gitlab.md @@ -12,7 +12,7 @@ When implementing new features, please refer to these existing features to avoid - [Custom Dashboards](../operations/metrics/dashboards/index.md#add-a-new-dashboard-to-your-project): `.gitlab/dashboards/`. - [Issue Templates](../user/project/description_templates.md#create-an-issue-template): `.gitlab/issue_templates/`. - [Merge Request Templates](../user/project/description_templates.md#create-a-merge-request-template): `.gitlab/merge_request_templates/`. -- [GitLab Kubernetes Agents](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/configuration_repository.md#layout): `.gitlab/agents/`. +- [GitLab Agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/configuration_repository.md#layout): `.gitlab/agents/`. - [CODEOWNERS](../user/project/code_owners.md#set-up-code-owners): `.gitlab/CODEOWNERS`. - [Route Maps](../ci/review_apps/#route-maps): `.gitlab/route-map.yml`. - [Customize Auto DevOps Helm Values](../topics/autodevops/customize.md#customize-values-for-helm-chart): `.gitlab/auto-deploy-values.yaml`. diff --git a/doc/development/filtering_by_label.md b/doc/development/filtering_by_label.md index 6f9811f7e05..9e759744a1a 100644 --- a/doc/development/filtering_by_label.md +++ b/doc/development/filtering_by_label.md @@ -47,9 +47,9 @@ This is more complicated than is ideal. It makes the query construction more prone to errors (such as [issue #15557](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/15557)). -## Attempt A: WHERE EXISTS +## Attempt A: `WHERE EXISTS` -### Attempt A1: use multiple subqueries with WHERE EXISTS +### Attempt A1: use multiple subqueries with `WHERE EXISTS` In [issue #37137](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37137) and its associated [merge request](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14022), @@ -82,7 +82,7 @@ AND (EXISTS ( While this worked without schema changes, and did improve readability somewhat, it did not improve query performance. -### Attempt A2: use label IDs in the WHERE EXISTS clause +### Attempt A2: use label IDs in the `WHERE EXISTS` clause In [merge request #34503](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34503), we followed a similar approach to A1. But this time, we did a separate query to fetch the IDs of the labels used in the filter so that we avoid the `JOIN` in the `EXISTS` clause and filter directly by diff --git a/doc/development/go_guide/go_upgrade.md b/doc/development/go_guide/go_upgrade.md index 53f2d7d176a..889849799bc 100644 --- a/doc/development/go_guide/go_upgrade.md +++ b/doc/development/go_guide/go_upgrade.md @@ -32,7 +32,7 @@ Individual Golang projects need to support multiple Go versions because: - We must support the [official Omnibus GitLab Go version](#updating-go-version), which may be behind the latest minor release. - When Omnibus switches Go version, we still may need to support the old one for security backports. -These 3 requirements may easily be satisfied by keeping support for the [3 latest minor versions of Go](https://golang.org/dl/). +These 3 requirements may easily be satisfied by keeping support for the [3 latest minor versions of Go](https://go.dev/dl/). It is ok to drop support for the oldest Go version and support only the 2 latest releases, if this is enough to support backports to the last 3 minor GitLab releases. @@ -52,12 +52,12 @@ in case of a critical security release. We should always: - Use the same Go version for Omnibus GitLab and Cloud Native GitLab. -- Use a [supported version](https://golang.org/doc/devel/release#policy). +- Use a [supported version](https://go.dev/doc/devel/release#policy). - Use the most recent patch-level for that version to keep up with security fixes. Changing the version affects every project being compiled, so it's important to ensure that all projects have been updated to test against the new Go version -before changing the package builders to use it. Despite [Go's compatibility promise](https://golang.org/doc/go1compat), +before changing the package builders to use it. Despite [Go's compatibility promise](https://go.dev/doc/go1compat), changes between minor versions can expose bugs or cause problems in our projects. ### Upgrade process @@ -134,7 +134,7 @@ if you need help finding the correct person or labels: | GitLab Compose Kit | [Issuer Tracker](https://gitlab.com/gitlab-org/gitlab-compose-kit/-/issues) | | GitLab Container Registry | [Issue Tracker](https://gitlab.com/gitlab-org/container-registry) | | GitLab Elasticsearch Indexer | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/issues) | -| GitLab Kubernetes Agent (KAS) | [Issue Tracker](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues) | +| GitLab Agent Server (KAS) | [Issue Tracker](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues) | | GitLab Pages | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-pages/-/issues) | | GitLab Quality Images | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-build-images/-/issues) | | GitLab Shell | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-shell/-/issues) | diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 9bf8b7ef89a..a5661a77da3 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Go standards and style guidelines This document describes various guidelines and best practices for GitLab -projects using the [Go language](https://golang.org). +projects using the [Go language](https://go.dev/). ## Overview @@ -103,7 +103,7 @@ projects: - Use `goimports` before committing. [`goimports`](https://pkg.go.dev/golang.org/x/tools/cmd/goimports) is a tool that automatically formats Go source code using - [`Gofmt`](https://golang.org/cmd/gofmt/), in addition to formatting import lines, + [`Gofmt`](https://pkg.go.dev/cmd/gofmt), in addition to formatting import lines, adding missing ones and removing unreferenced ones. Most editors/IDEs allow you to run commands before/after saving a file, you can set it @@ -196,7 +196,7 @@ deploy a new pod, migrating the data automatically. ### Testing frameworks We should not use any specific library or framework for testing, as the -[standard library](https://golang.org/pkg/) provides already everything to get +[standard library](https://pkg.go.dev/std) provides already everything to get started. If there is a need for more sophisticated testing tools, the following external dependencies might be worth considering in case we decide to use a specific library or framework: @@ -279,7 +279,7 @@ to make the test output easily readable. ### Benchmarks Programs handling a lot of IO or complex operations should always include -[benchmarks](https://golang.org/pkg/testing/#hdr-Benchmarks), to ensure +[benchmarks](https://pkg.go.dev/testing#hdr-Benchmarks), to ensure performance consistency over time. ## Error handling @@ -435,7 +435,7 @@ The following are some style guidelines that are specific to the Secure Team. Use `goimports -local gitlab.com/gitlab-org` before committing. [`goimports`](https://pkg.go.dev/golang.org/x/tools/cmd/goimports) is a tool that automatically formats Go source code using -[`Gofmt`](https://golang.org/cmd/gofmt/), in addition to formatting import lines, +[`Gofmt`](https://pkg.go.dev/cmd/gofmt), in addition to formatting import lines, adding missing ones and removing unreferenced ones. By using the `-local gitlab.com/gitlab-org` option, `goimports` groups locally referenced packages separately from external ones. See @@ -452,7 +452,7 @@ If the scanner report is small, less than 35 lines, then feel free to [inline th #### Test Diffs -The [go-cmp]<https://github.com/google/go-cmp> package should be used when comparing large structs in tests. It makes it possible to output a specific diff where the two structs differ, rather than seeing the whole of both structs printed out in the test logs. Here is a small example: +The [go-cmp](https://github.com/google/go-cmp) package should be used when comparing large structs in tests. It makes it possible to output a specific diff where the two structs differ, rather than seeing the whole of both structs printed out in the test logs. Here is a small example: ```golang package main diff --git a/doc/development/graphql_guide/authorization.md b/doc/development/graphql_guide/authorization.md index d7edd01cda2..717a6d29fbc 100644 --- a/doc/development/graphql_guide/authorization.md +++ b/doc/development/graphql_guide/authorization.md @@ -43,6 +43,11 @@ such as short pages, which can expose the presence of confidential resources. See [`authorization_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/graphql/features/authorization_spec.rb) for examples of all the authorization schemes discussed here. +<!-- + NOTE: if you change this heading (or the location to this file), make sure to update + the referenced link in rubocop/cop/graphql/authorize_types.rb +--> + ## Type authorization Authorize a type by passing an ability to the `authorize` method. All diff --git a/doc/development/graphql_guide/index.md b/doc/development/graphql_guide/index.md index cc97e41df05..412825e06d3 100644 --- a/doc/development/graphql_guide/index.md +++ b/doc/development/graphql_guide/index.md @@ -21,3 +21,4 @@ feedback, and suggestions. - [GraphQL BatchLoader](batchloader.md): development documentation on the BatchLoader. - [GraphQL pagination](pagination.md): development documentation on pagination. - [GraphQL Pro](graphql_pro.md): information on our GraphQL Pro subscription. +- [GraphQL monitoring](monitoring.md): tips on how to use our monitoring tools to inspect GraphQL queries. diff --git a/doc/development/graphql_guide/monitoring.md b/doc/development/graphql_guide/monitoring.md new file mode 100644 index 00000000000..28d1a4a9046 --- /dev/null +++ b/doc/development/graphql_guide/monitoring.md @@ -0,0 +1,89 @@ +--- +stage: Ecosystem +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Monitoring GraphQL + +This page gives tips on how to analyze GraphQL data in our monitoring tools. +Please contribute your own tips to improve this document. + +## Kibana + +We use Kibana to filter GraphQL query logs. Sign in to [Kibana](https://log.gprd.gitlab.net/) +with a `@gitlab.com` email address. + +In Kibana we can inspect two kinds of GraphQL logs: + +- Logs of each GraphQL query executed within the request. +- Logs of the full request, which due to [query multiplexing](https://graphql-ruby.org/queries/multiplex.html) + may have executed multiple queries. + +### Logs of each GraphQL query + +In a [multiplex query](https://graphql-ruby.org/queries/multiplex.html), each individual query +is logged separately. We can use subcomponent filtering to inspect these logs. +[Visit Kibana with this filter enabled](https://log.gprd.gitlab.net/goto/a0da8c9a1e9c1f533a058b7d29d13956) +or set up the subcomponent filter using these steps: + +1. Add a filter: + 1. Filter: `json.subcomponent` + 1. Operator: `is` + 1. Value: `graphql_json` +1. Select **Refresh**. + +You can select Kibana fields from the **Available fields** section of the sidebar to +add columns to the log table, or [visit this view](https://log.gprd.gitlab.net/goto/5826d3d3affb41cac52e637ffc205905), +which already has a set of Kibana fields selected. Some relevant Kibana fields include: + +| Kibana field | Description | +| --- | --- | +| `json.operation_name` | The [operation name](https://graphql.org/learn/queries/#operation-name) used by the client. | +| `json.operation_fingerprint`| The [fingerprint](https://graphql-ruby.org/api-doc/1.12.20/GraphQL/Query#fingerprint-instance_method) of the query, used to recognize repeated queries over time. | +| `json.meta.caller_id` | Appears as `graphql:<operation_name>` for queries that came from the GitLab frontend, otherwise as `graphql:unknown`. Can be used to identify internal versus external queries. | +| `json.query_string` | The query string itself. | +| `json.is_mutation` | `true` when a mutation, `false` when not. | +| `json.query_analysis.used_fields` | List of GraphQL fields selected by the query. | +| `json.query_analysis.used_deprecated_fields` | List of deprecated GraphQL fields selected by the query. | +| `json.query_analysis.duration_s` | Duration of query execution in seconds. | +| `json.query_analysis.complexity` | The [complexity](../api_graphql_styleguide.md#max-complexity) score of the query. | + +#### Useful filters + +Combine the [subcomponent filter](#logs-of-each-graphql-query) with the following Kibana filters to further interrogate the query logs. + +##### Queries that used a particular field + +Filter logs by queries that used a particular field: + +1. Add a filter: + 1. Filter: `json.query_analysis.used_fields` + 1. Operator: `is` + 1. Value: `Type.myField`, where `Type.myField` is the type name and field name as it + appears in [our GraphQL reference documentation](../../api/graphql/reference/index.md). +1. Select **Refresh**. + +##### Queries that used a deprecated field + +Filter logs of queries that used a particular deprecated field by following the +[steps above](#queries-that-used-a-particular-field) but use the `json.graphql.used_deprecated_fields` +filter instead. + +### Logs of the full request + +The full request logs encompass log data for all [multiplexed queries](https://graphql-ruby.org/queries/multiplex.html) +in the request, as well as data from time spent outside of `GraphQLController#execute`. + +To see the full request logs, do **not** apply the `json.subcomponent` [filter](#logs-of-each-graphql-query), and instead: + +1. Add a filter: + 1. Filter: `json.meta.caller_id` + 1. Operator: `is` + 1. Value: `GraphqlController#execute` +1. Select **Refresh**. + +Some differences from the [query logs](#logs-of-each-graphql-query) described above: + +- Some of the [Kibana fields mentioned above](#logs-of-each-graphql-query) are not available to the full request logs. +- The names of filters differ. For example, instead of `json.query_analysis.used_fields` you select `json.graphql.used_fields`. diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 52a7f839286..65cf8911e12 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -133,11 +133,9 @@ You can mark that content for translation with: The `~/locale` module exports the following key functions for externalization: -- `__()` (double underscore parenthesis) -- `s__()` (namespaced double underscore parenthesis) -- `__()` Mark content for translation (note the double underscore). -- `s__()` Mark namespaced content for translation -- `n__()` Mark pluralized content for translation +- `__()` Mark content for translation (double underscore parenthesis). +- `s__()` Mark namespaced content for translation (s double underscore parenthesis). +- `n__()` Mark pluralized content for translation (n double underscore parenthesis). ```javascript import { __, s__, n__ } from '~/locale'; @@ -822,11 +820,11 @@ bin/rake gettext:regenerate ``` This command updates the `locale/gitlab.pot` file with the newly externalized strings and removes -any unused strings. Once the changes are on the default branch, [CrowdIn](https://translate.gitlab.com) +any unused strings. Once the changes are on the default branch, [Crowdin](https://translate.gitlab.com) picks them up and presents them for translation. You don't need to check in any changes to the `locale/[language]/gitlab.po` files. They are updated -automatically when [translations from CrowdIn are merged](merging_translations.md). +automatically when [translations from Crowdin are merged](merging_translations.md). If there are merge conflicts in the `gitlab.pot` file, you can delete the file and regenerate it using the same command. diff --git a/doc/development/i18n/index.md b/doc/development/i18n/index.md index c22bb6ff020..0870a0e6750 100644 --- a/doc/development/i18n/index.md +++ b/doc/development/i18n/index.md @@ -32,8 +32,8 @@ See [Externalization for GitLab](externalization.md). ## Translate strings The translation process is managed at [https://translate.gitlab.com](https://translate.gitlab.com) -using [CrowdIn](https://crowdin.com/). -You must create a CrowdIn account before you can submit translations. Once you are signed in, select +using [Crowdin](https://crowdin.com/). +You must create a Crowdin account before you can submit translations. Once you are signed in, select the language you wish to contribute translations to. Voting for translations is also valuable, helping to confirm good translations and flag inaccurate @@ -54,4 +54,4 @@ instructions on becoming a proofreader yourself. Translations are typically included in the next major or minor release. -See [Merging translations from CrowdIn](merging_translations.md). +See [Merging translations from Crowdin](merging_translations.md). diff --git a/doc/development/i18n/merging_translations.md b/doc/development/i18n/merging_translations.md index f89190fc316..43f19f8a9d6 100644 --- a/doc/development/i18n/merging_translations.md +++ b/doc/development/i18n/merging_translations.md @@ -4,27 +4,27 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Merging translations from CrowdIn +# Merging translations from Crowdin -CrowdIn automatically syncs the `gitlab.pot` file with the CrowdIn service, presenting +Crowdin automatically syncs the `gitlab.pot` file with the Crowdin service, presenting newly added externalized strings to the community of translators. -The [GitLab CrowdIn Bot](https://gitlab.com/gitlab-crowdin-bot) also creates merge requests +The [GitLab Crowdin Bot](https://gitlab.com/gitlab-crowdin-bot) also creates merge requests to take newly approved translation submissions and merge them into the `locale/<language>/gitlab.po` files. Check the [merge requests created by `gitlab-crowdin-bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=all&state=opened&author_username=gitlab-crowdin-bot) to see new and merged merge requests. ## Validation -By default CrowdIn commits translations with `[skip ci]` in the commit +By default Crowdin commits translations with `[skip ci]` in the commit message. This avoids an excessive number of pipelines from running. Before merging translations, make sure to trigger a pipeline to validate -translations. Static analysis validates things CrowdIn doesn't do. Create +translations. Static analysis validates things Crowdin doesn't do. Create a new pipeline at [`https://gitlab.com/gitlab-org/gitlab/pipelines/new`](https://gitlab.com/gitlab-org/gitlab/pipelines/new) (requires the Developer role) for the `master-i18n` branch. If there are validation errors, the easiest solution is to disapprove -the offending string in CrowdIn, leaving a comment with what is +the offending string in Crowdin, leaving a comment with what is required to fix the errors. There's an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/23256) that suggests automating this process. Disapproving excludes the @@ -32,18 +32,18 @@ invalid translation. The merge request is then updated within a few minutes. If the translation fails validation due to angle brackets (`<` or `>`), -it should be disapproved in CrowdIn. Our strings must use [variables](externalization.md#html) +it should be disapproved in Crowdin. Our strings must use [variables](externalization.md#html) for HTML instead. -It might be useful to pause the integration on the CrowdIn side for a +It might be useful to pause the integration on the Crowdin side for a moment so translations don't keep coming. You can do this by clicking -**Pause sync** on the [CrowdIn integration settings page](https://translate.gitlab.com/project/gitlab-ee/settings#integration). +**Pause sync** on the [Crowdin integration settings page](https://translate.gitlab.com/project/gitlab-ee/settings#integration). ## Merging translations After all translations are determined to be appropriate and the pipelines pass, you can merge the translations into the default branch. When merging translations, -be sure to select the **Remove source branch** checkbox. This causes CrowdIn +be sure to select the **Remove source branch** checkbox. This causes Crowdin to recreate the `master-i18n` branch from the default branch after merging the new translation. @@ -51,26 +51,26 @@ We are discussing [automating this entire process](https://gitlab.com/gitlab-org ## Recreate the merge request -CrowdIn creates a new merge request as soon as the old one is closed +Crowdin creates a new merge request as soon as the old one is closed or merged. But it does not recreate the `master-i18n` branch every -time. To force CrowdIn to recreate the branch, close any [open merge requests](https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=all&state=opened&author_username=gitlab-crowdin-bot) +time. To force Crowdin to recreate the branch, close any [open merge requests](https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=all&state=opened&author_username=gitlab-crowdin-bot) and delete the [`master-18n`](https://gitlab.com/gitlab-org/gitlab/-/branches/all?utf8=✓&search=master-i18n) branch. This might be needed when the merge request contains failures that have been fixed on the default branch. -## Recreate the GitLab integration in CrowdIn +## Recreate the GitLab integration in Crowdin NOTE: These instructions work only for GitLab Team Members. -If for some reason the GitLab integration in CrowdIn doesn't exist, you can +If for some reason the GitLab integration in Crowdin doesn't exist, you can recreate it with the following steps: 1. Sign in to GitLab as `gitlab-crowdin-bot`. (If you're a GitLab Team Member, find credentials in the GitLab shared [1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams).) -1. Sign in to CrowdIn with the GitLab integration. +1. Sign in to Crowdin with the GitLab integration. 1. Go to **Settings > Integrations > GitLab > Set Up Integration**. 1. Select the `gitlab-org/gitlab` repository. 1. In **Select Branches for Translation**, select `master`. @@ -78,7 +78,7 @@ recreate it with the following steps: ## Manually update the translation levels -There's no automated way to pull the translation levels from CrowdIn, to display +There's no automated way to pull the translation levels from Crowdin, to display this information in the language selection dropdown. Therefore, the translation levels are hard-coded in the `TRANSLATION_LEVELS` constant in [`i18n.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/i18n.rb), and must be regularly updated. diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index 04a9f68abec..61f9d7a25c3 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -6,9 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Translating GitLab -For managing the translation process, we use [CrowdIn](https://crowdin.com). +For managing the translation process, we use [Crowdin](https://crowdin.com). To contribute translations at [`translate.gitlab.com`](https://translate.gitlab.com), -you must create a CrowdIn account. You may create a new account or use any of their supported +you must create a Crowdin account. You may create a new account or use any of their supported sign-in services. ## Language selections @@ -16,12 +16,12 @@ sign-in services. GitLab is being translated into many languages. To select a language to contribute to: 1. Find the language that you want to contribute to, in the - [GitLab CrowdIn project](https://crowdin.com/project/gitlab-ee). + [GitLab Crowdin project](https://crowdin.com/project/gitlab-ee). - If the language you want is available, proceed to the next step. - If the language you want is not available, [open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=✓&state=all&label_name[]=Category%3AInternationalization). - Notify our CrowdIn administrators by including `@gitlab-org/manage/import` in your issue. + Notify our Crowdin administrators by including `@gitlab-org/manage/import` in your issue. - After the issue and any merge requests are complete, restart this procedure. 1. View the list of files and folders. Select `gitlab.pot` to open the translation editor. @@ -30,7 +30,7 @@ GitLab is being translated into many languages. To select a language to contribu The online translation editor is the easiest way to contribute translations. - + - Strings for translation are listed in the left panel. - Translations are entered into the central panel. Multiple translations are required for strings diff --git a/doc/development/image_scaling.md b/doc/development/image_scaling.md index 82ca8cf8e83..e1ffbdb766a 100644 --- a/doc/development/image_scaling.md +++ b/doc/development/image_scaling.md @@ -37,8 +37,8 @@ Furthermore, configuration in Workhorse can lead to the image scaler rejecting a For instance, here are two different URLs that serve the GitLab project avatar both in its original size and scaled down to 64 pixels. Only the second request will trigger the image scaler: -- [`/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png`](https://assets.gitlab-static.net/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png) -- [`/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png?width=64`](https://assets.gitlab-static.net/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png?width=64) +- [`/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png`](https://gitlab.com/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png) +- [`/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png?width=64`](https://gitlab.com/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png?width=64) ## Where do we scale images? diff --git a/doc/development/index.md b/doc/development/index.md index fa49d43d46c..1398104abda 100644 --- a/doc/development/index.md +++ b/doc/development/index.md @@ -139,8 +139,9 @@ In these cases, use the following workflow: and approval from the VP of Development, the DRI for Development Guidelines, @clefelhocz1. -1. After all approvals are complete, assign the merge request to the - Technical Writer for [Development Guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines) +1. After all approvals are complete, review the page's metadata to + [find a Technical Writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) + who can help you merge the changes. for final content review and merge. The Technical Writer may ask for additional approvals as previously suggested before merging the MR. diff --git a/doc/development/integrations/codesandbox.md b/doc/development/integrations/codesandbox.md index caef1cd045b..8fd1f0e331d 100644 --- a/doc/development/integrations/codesandbox.md +++ b/doc/development/integrations/codesandbox.md @@ -18,7 +18,7 @@ Before using CodeSandbox with your local GitLab instance, you must: Follow the GDK [NGINX configuration instructions](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/nginx.md) to enable HTTPS for GDK. 1. Clone the [`codesandbox-client` project](https://github.com/codesandbox/codesandbox-client) locally. If you plan on contributing upstream, you might want to fork and clone first. -1. (Optional) Use correct `python` and `nodejs` versions. Otherwise, `yarn` may fail to +1. Optional. Use correct `python` and `nodejs` versions. Otherwise, `yarn` may fail to install or build some packages. If you're using `asdf` you can run the following commands: ```shell diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 34293845d17..356e731aa87 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -298,7 +298,7 @@ this makes it possible to debug the problem without having to change the log lev #### common `logutil` package -If you are using [go](https://golang.org/) and +If you are using [go](https://go.dev/) and [common](https://gitlab.com/gitlab-org/security-products/analyzers/common), then it is suggested that you use [Logrus](https://github.com/Sirupsen/logrus) and [common's `logutil` package](https://gitlab.com/gitlab-org/security-products/analyzers/common/-/tree/master/logutil) diff --git a/doc/development/interacting_components.md b/doc/development/interacting_components.md index f6fec0cde8c..c137faf464c 100644 --- a/doc/development/interacting_components.md +++ b/doc/development/interacting_components.md @@ -29,5 +29,5 @@ See also [File Storage in GitLab](file_storage.md). ### Forks GitLab supports a great amount of features for [merge requests](../user/project/merge_requests/index.md). One -of them is the ability to create merge requests from and to [forks](../user/project/working_with_projects.md#fork-a-project), +of them is the ability to create merge requests from and to [forks](../user/project/repository/forking_workflow.md#creating-a-fork), which should also be highly considered and tested upon development phase. diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index 0fe9a5362cf..543c5f40f88 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -42,7 +42,7 @@ file, and include the token Base64 encoded in a `secret_token` parameter or in the `Gitlab-Shared-Secret` header. NOTE: -The internal API used by GitLab Pages, and GitLab Kubernetes Agent Server (`kas`) uses JSON Web Token (JWT) +The internal API used by GitLab Pages, and GitLab Agent Server (`kas`) uses JSON Web Token (JWT) authentication, which is different from GitLab Shell. ## Git Authentication @@ -400,13 +400,13 @@ Example response: } ``` -## Kubernetes agent endpoints +## GitLab Agent endpoints > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41045) in GitLab 13.4. > - This feature is not deployed on GitLab.com > - It's not recommended for production use. -The following endpoints are used by the GitLab Kubernetes Agent Server (`kas`) +The following endpoints are used by the GitLab Agent Server (`kas`) for various purposes. These endpoints are all authenticated using JWT. The JWT secret is stored in a file @@ -414,11 +414,11 @@ specified in `config/gitlab.yml`. By default, the location is in the root of the GitLab Rails app in a file called `.gitlab_kas_secret`. WARNING: -The Kubernetes agent is under development and is not recommended for production use. +The GitLab Agent is under development and is not recommended for production use. -### Kubernetes agent information +### GitLab Agent information -Called from GitLab Kubernetes Agent Server (`kas`) to retrieve agent +Called from GitLab Agent Server (`kas`) to retrieve agent information for the given agent token. This returns the Gitaly connection information for the agent's project in order for `kas` to fetch and update the agent's configuration. @@ -434,9 +434,9 @@ curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/agent_info" ``` -### Kubernetes agent project information +### GitLab Agent project information -Called from GitLab Kubernetes Agent Server (`kas`) to retrieve project +Called from GitLab Agent Server (`kas`) to retrieve project information for the given agent token. This returns the Gitaly connection for the requested project. GitLab `kas` uses this to configure the agent to fetch Kubernetes resources from the project repository to @@ -460,9 +460,9 @@ curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/project_info?id=7" ``` -### Kubernetes agent usage metrics +### GitLab Agent usage metrics -Called from GitLab Kubernetes Agent Server (`kas`) to increase the usage +Called from GitLab Agent Server (`kas`) to increase the usage metric counters. | Attribute | Type | Required | Description | @@ -481,9 +481,9 @@ curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Con --data '{"gitops_sync_count":1}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics" ``` -### Kubernetes agent alert metrics +### GitLab Agent alert metrics -Called from GitLab Kubernetes Agent Server (KAS) to save alerts derived from Cilium on Kubernetes +Called from GitLab Agent Server (KAS) to save alerts derived from Cilium on Kubernetes Cluster. | Attribute | Type | Required | Description | @@ -505,7 +505,7 @@ curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" \ ### Create Starboard vulnerability -Called from the GitLab Kubernetes Agent Server (`kas`) to create a security vulnerability +Called from the GitLab Agent Server (`kas`) to create a security vulnerability from a Starboard vulnerability report. This request is idempotent. Multiple requests with the same data create a single vulnerability. diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index 10e6d717a18..629d0027ffe 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -29,7 +29,7 @@ project.feature_available?(:feature_symbol) ## Restricting global features (instance) However, for features such as [Geo](../administration/geo/index.md) and -[Load balancing](../administration/database_load_balancing.md), which cannot be restricted +[Database Load Balancing](../administration/postgresql/database_load_balancing.md), which cannot be restricted to only a subset of projects or namespaces, the check is made directly in the instance license. diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index cbf3c09b28b..69e9f7d16e3 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -160,10 +160,10 @@ query. This in turn makes it much harder for this code to overload a database. ## Use read replicas when possible -In a DB cluster we have many read replicas and one primary. A classic use of scaling the DB is to have read-only actions be performed by the replicas. We use [load balancing](../administration/database_load_balancing.md) to distribute this load. This allows for the replicas to grow as the pressure on the DB grows. +In a DB cluster we have many read replicas and one primary. A classic use of scaling the DB is to have read-only actions be performed by the replicas. We use [load balancing](../administration/postgresql/database_load_balancing.md) to distribute this load. This allows for the replicas to grow as the pressure on the DB grows. By default, queries use read-only replicas, but due to -[primary sticking](../administration/database_load_balancing.md#primary-sticking), GitLab uses the +[primary sticking](../administration/postgresql/database_load_balancing.md#primary-sticking), GitLab uses the primary for some time and reverts to secondaries after they have either caught up or after 30 seconds. Doing this can lead to a considerable amount of unnecessary load on the primary. To prevent switching to the primary [merge request 56849](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56849) introduced the @@ -187,7 +187,7 @@ Internally, our database load balancer classifies the queries based on their mai - Sidekiq background jobs After the above queries are executed, GitLab -[sticks to the primary](../administration/database_load_balancing.md#primary-sticking). +[sticks to the primary](../administration/postgresql/database_load_balancing.md#primary-sticking). To make the inside queries prefer using the replicas, [merge request 59086](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59086) introduced `fallback_to_replicas_for_ambiguous_queries`. This MR is also an example of how we redirected a @@ -205,7 +205,7 @@ Keeping the old behavior requires marking CTEs with the keyword `MATERIALIZED`. When building CTE statements, use the `Gitlab::SQL::CTE` class [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56976) in GitLab 13.11. By default, this `Gitlab::SQL::CTE` class forces materialization through adding the `MATERIALIZED` keyword for PostgreSQL 12 and higher. `Gitlab::SQL::CTE` automatically omits materialization when PostgreSQL 11 is running -(this behavior is implemented using a custom arel node `Gitlab::Database::AsWithMaterialized` under the surface). +(this behavior is implemented using a custom Arel node `Gitlab::Database::AsWithMaterialized` under the surface). WARNING: We plan to drop the support for PostgreSQL 11. Upgrading to GitLab 14.0 requires PostgreSQL 12 or higher. diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index 27a7cd6e85e..27c4edf15f4 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -64,7 +64,7 @@ To illustrate how these problems arise, take a look at this example: In this example, you can imagine that we are updating by one monthly release. But refer to [How long must code be backwards-compatible?](#how-long-must-code-be-backwards-compatible). -| Update step | Postgres DB | Web nodes | API nodes | Sidekiq nodes | Compatibility concerns | +| Update step | PostgreSQL DB | Web nodes | API nodes | Sidekiq nodes | Compatibility concerns | | --- | --- | --- | --- | --- | --- | | Initial state | 🙂 | 🙂 | 🙂 | 🙂 | | | Ran pre-deployment migrations | 🚢 except post-deploy migrations | 🙂 | 🙂 | 🙂 | Rails code in 🙂 is making DB calls to 🚢 | @@ -102,7 +102,7 @@ But the problem isn't just that there are many nodes. The bigger problem is that - "Web app nodes": Handle web requests - "API app nodes": Handle API requests - "Sidekiq app nodes": Handle Sidekiq jobs -- "Postgres database": Handle internal Postgres calls +- "PostgreSQL database": Handle internal PostgreSQL calls - "Redis database": Handle internal Redis calls - "Gitaly nodes": Handle internal Gitaly calls @@ -110,7 +110,7 @@ During an update, there will be [two different versions of GitLab running in dif ## Doesn't the order of update steps matter? -Yes! We have specific instructions for [zero-downtime updates](../update/index.md#upgrading-without-downtime) because it allows us to ignore some permutations of compatibility. This is why we don't worry about Rails code making DB calls to an old Postgres database schema. +Yes! We have specific instructions for [zero-downtime updates](../update/index.md#upgrading-without-downtime) because it allows us to ignore some permutations of compatibility. This is why we don't worry about Rails code making DB calls to an old PostgreSQL database schema. ## I've identified a potential backwards compatibility problem, what can I do about it? diff --git a/doc/development/new_fe_guide/modules/widget_extensions.md b/doc/development/new_fe_guide/modules/widget_extensions.md index d1f6099e908..b833ba7c630 100644 --- a/doc/development/new_fe_guide/modules/widget_extensions.md +++ b/doc/development/new_fe_guide/modules/widget_extensions.md @@ -54,3 +54,26 @@ import issueExtension from '~/vue_merge_request_widget/extensions/issues'; // Register the imported extension registerExtension(issueExtension); ``` + +## Fetching errors + +If `fetchCollapsedData()` or `fetchFullData()` methods throw an error: + +- The loading state of the extension is updated to `LOADING_STATES.collapsedError` and `LOADING_STATES.expandedError` + respectively. +- The extensions header displays an error icon and updates the text to be either: + - The text defined in `$options.i18n.error`. + - "Failed to load" if `$options.i18n.error` is not defined. +- The error is sent to Sentry to log that it occurred. + +To customise the error text, you need to add it to the `i18n` object in your extension: + +```javascript +export default { + //... + i18n: { + //... + error: __('Your error text'), + }, +}; +``` diff --git a/doc/development/performance.md b/doc/development/performance.md index e59f7fb154b..b5294c8359d 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -365,7 +365,7 @@ This patch is available by default for [GCK](https://gitlab.com/gitlab-org/gitlab-compose-kit/-/merge_requests/149) and can additionally be enabled for [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/advanced.md#apply-custom-patches-for-ruby). -This patch provides the following metrics that make it easier to understand efficiency of memory use for a given codepath: +This patch provides the following metrics that make it easier to understand efficiency of memory use for a given code path: - `mem_total_bytes`: the number of bytes consumed both due to new objects being allocated into existing object slots plus additional memory allocated for large objects (that is, `mem_bytes + slot_size * mem_objects`). @@ -384,7 +384,7 @@ and `100M mem_bytes`. You can view the current usage on [GitLab.com](https://log There are two ways of measuring your own code: 1. Review `api_json.log`, `development_json.log`, `sidekiq.log` that includes memory allocation counters. -1. Use `Gitlab::Memory::Instrumentation.with_memory_allocations` for a given codeblock and log it. +1. Use `Gitlab::Memory::Instrumentation.with_memory_allocations` for a given code block and log it. 1. Use [Measuring module](service_measurement.md) ```json diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 71a11d2024c..c7443032d78 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -68,6 +68,7 @@ In addition, there are a few circumstances where we would always run the full RS - when the `pipeline:run-all-rspec` label is set on the merge request - when the merge request is created by an automation (e.g. Gitaly update or MR targeting a stable branch) +- when the merge request is created in a security mirror - when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) ### Jest minimal jobs @@ -83,6 +84,7 @@ In addition, there are a few circumstances where we would always run the full Je - when the `pipeline:run-all-jest` label is set on the merge request - when the merge request is created by an automation (e.g. Gitaly update or MR targeting a stable branch) +- when the merge request is created in a security mirror - when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) - when any frontend "core" file is changed (i.e. `package.json`, `yarn.lock`, `babel.config.js`, `jest.config.*.js`, `config/helpers/**/*.js`) - when any vendored JavaScript file is changed (i.e. `vendor/assets/javascripts/**/*`) @@ -220,6 +222,20 @@ The `* as-if-jh` jobs are run in addition to the regular EE-context jobs. The `j The intent is to ensure that a change doesn't introduce a failure after the `gitlab-org/gitlab` project is synced to the `gitlab-jh/gitlab` project. +## `undercover` RSpec test + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74859) in GitLab 14.6. + +The `rspec:undercoverage` job runs [`undercover`](https://rubygems.org/gems/undercover) +to detect, and fail if any changes introduced in the merge request has zero coverage. + +The `rsepc:undercoverage` job obtains coverage data from the `rspec:coverage` +job. + +In the event of an emergency, or false positive from this job, add the +`pipeline:skip-undercoverage` label to the merge request to allow this job to +fail. + ## PostgreSQL versions testing Our test suite runs against PG12 as GitLab.com runs on PG12 and @@ -820,7 +836,7 @@ We no longer use this optimization for `gitlab-org/gitlab` because the [pack-obj allows Gitaly to serve the full CI/CD fetch traffic now. See [Git fetch caching](#git-fetch-caching). The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable -[defined by GitLab.com shared runners](../ci/runners/runner_cloud/linux_runner_cloud.md#pre-clone-script). +[defined by GitLab.com shared runners](../ci/runners/saas/linux_saas_runner.md#pre-clone-script). --- diff --git a/doc/development/policies.md b/doc/development/policies.md index e5191f00d9a..9a977a49329 100644 --- a/doc/development/policies.md +++ b/doc/development/policies.md @@ -79,7 +79,7 @@ cop](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49771). ## Scores, Order, Performance -To see how the rules get evaluated into a judgment, it is useful in a console to use `policy.debug(:some_ability)`. This prints the rules in the order they are evaluated. +To see how the rules get evaluated into a judgment, open a Rails console and run: `policy.debug(:some_ability)`. This prints the rules in the order they are evaluated. For example, let's say you wanted to debug `IssuePolicy`. You might run the debugger in this way: @@ -222,7 +222,7 @@ delegation would end up with only children whose parents enjoy green vegetables eating it. But a parent may well give their child broccoli, even if they dislike it themselves, because it is good for their child. -The solution it to override the `:eat_broccoli` ability in the child policy: +The solution is to override the `:eat_broccoli` ability in the child policy: ```ruby class ChildPolicy < BasePolicy diff --git a/doc/development/product_analytics/event_dictionary.md b/doc/development/product_analytics/event_dictionary.md deleted file mode 100644 index 3931f0b35ab..00000000000 --- a/doc/development/product_analytics/event_dictionary.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' -remove_date: '2021-12-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after 2021-12-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/product_analytics/index.md b/doc/development/product_analytics/index.md deleted file mode 100644 index 3931f0b35ab..00000000000 --- a/doc/development/product_analytics/index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' -remove_date: '2021-12-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after 2021-12-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/rails_update.md b/doc/development/rails_update.md index f25d68a8900..1a30e606c17 100644 --- a/doc/development/rails_update.md +++ b/doc/development/rails_update.md @@ -71,7 +71,7 @@ To efficiently and quickly find which Rails change caused the spec failure you c gem 'rails', ENV['RAILS_VERSION'], path: ENV['RAILS_FOLDER'] ``` -1. Set the `RAILS_FOLDER` env variable with the folder you cloned Rails into: +1. Set the `RAILS_FOLDER` environment variable with the folder you cloned Rails into: ```shell export RAILS_FOLDER="<GDK_FOLDER>/rails" diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index 5aaf4c72e64..b4deffe162a 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -228,7 +228,7 @@ self.reactive_cache_lease_timeout = 2.minutes - It defaults to 1 minute. ```ruby -self.reactive_cache_lease_timeout = 1.minute +self.reactive_cache_refresh_interval = 1.minute ``` #### `self.reactive_cache_lifetime` diff --git a/doc/development/ruby_upgrade.md b/doc/development/ruby_upgrade.md index f9816986b2d..2102a256645 100644 --- a/doc/development/ruby_upgrade.md +++ b/doc/development/ruby_upgrade.md @@ -146,7 +146,7 @@ When upgrading Ruby, consider updating the following repositories: - [Gitaly](https://gitlab.com/gitlab-org/gitaly) ([example](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3771)) - [GitLab Labkit](https://gitlab.com/gitlab-org/labkit-ruby) ([example](https://gitlab.com/gitlab-org/labkit-ruby/-/merge_requests/79)) - [GitLab Exporter](https://gitlab.com/gitlab-org/gitlab-exporter) ([example](https://gitlab.com/gitlab-org/gitlab-exporter/-/merge_requests/150)) -- [GitLab Experiment](https://gitlab.com/gitlab-org/gitlab-experiment) ([example](https://gitlab.com/gitlab-org/gitlab-experiment/-/merge_requests/128)) +- [GitLab Experiment](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) ([example](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment/-/merge_requests/128)) - [Gollum Lib](https://gitlab.com/gitlab-org/gollum-lib) ([example](https://gitlab.com/gitlab-org/gollum-lib/-/merge_requests/21)) - [GitLab Helm Chart](https://gitlab.com/gitlab-org/charts/gitlab) ([example](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/2162)) - [GitLab Sidekiq fetcher](https://gitlab.com/gitlab-org/sidekiq-reliable-fetch) ([example](https://gitlab.com/gitlab-org/sidekiq-reliable-fetch/-/merge_requests/33)) @@ -190,7 +190,7 @@ via `gdk update`. This pause is a good time to assess the risk of this upgrade for GitLab SaaS. For Ruby upgrades that are high risk, such as major version upgrades, it is recommended to coordinate the changes with the infrastructure team through a [change management request](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/). -Create this issue early to give everyone enough time to schedule and prepare changes. +Create this issue early to give everyone enough time to schedule and prepare changes. ### Make it the default Ruby @@ -205,7 +205,7 @@ in that repository. This change is only necessary when the minor or major versio ([example](https://gitlab.com/gitlab-org/gitlab-compose-kit/-/merge_requests/176).) As mentioned above, if the impact of the Ruby upgrade on SaaS availability is uncertain, it is -prudent to skip this step until you have verified that it runs smootly in production via a staged +prudent to skip this step until you have verified that it runs smoothly in production via a staged rollout. In this case, go to the next step first, and then, after the verification period has passed, promote the new Ruby to be the new default. diff --git a/doc/development/scalability.md b/doc/development/scalability.md index fdae66b7abc..7a3f3c7097d 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -123,8 +123,7 @@ the read replicas. [Omnibus ships with Patroni](../administration/postgresql/rep #### Load-balancing -GitLab EE has [application support for load balancing using read -replicas](../administration/database_load_balancing.md). This load balancer does +GitLab EE has [application support for load balancing using read replicas](../administration/postgresql/database_load_balancing.md). This load balancer does some actions that aren't traditionally available in standard load balancers. For example, the application considers a replica only if its replication lag is low (for example, WAL data behind by less than 100 MB). diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 65fdb815f87..21655d6a8fb 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -188,7 +188,7 @@ and [possessive quantifiers](https://www.regular-expressions.info/possessive.htm #### Go -Go's [`regexp`](https://golang.org/pkg/regexp/) package uses `re2` and isn't vulnerable to backtracking issues. +Go's [`regexp`](https://pkg.go.dev/regexp) package uses `re2` and isn't vulnerable to backtracking issues. ## Further Links @@ -544,7 +544,7 @@ This outputs `1` followed by the content of `/etc/passwd`. ### TLS minimum recommended version -As we have [moved away from supporting TLS 1.0 and 1.1](https://about.gitlab.com/blog/2018/10/15/gitlab-to-deprecate-older-tls/), we should only use TLS 1.2 and above. +As we have [moved away from supporting TLS 1.0 and 1.1](https://about.gitlab.com/blog/2018/10/15/gitlab-to-deprecate-older-tls/), you must use TLS 1.2 and above. #### Ciphers diff --git a/doc/development/service_ping/dictionary.md b/doc/development/service_ping/dictionary.md deleted file mode 100644 index 810c789bc03..00000000000 --- a/doc/development/service_ping/dictionary.md +++ /dev/null @@ -1,4 +0,0 @@ ---- -redirect_to: 'https://metrics.gitlab.com/index.html' -remove_date: '2021-11-10' ---- diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 65a8b4c1cad..c32789740c3 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -26,6 +26,10 @@ To implement a new metric in Service Ping, follow these steps: 1. [Verify your metric](#verify-your-metric) 1. [Set up and test Service Ping locally](#set-up-and-test-service-ping-locally) +NOTE: +When you add or change a Service Metric, you must migrate metrics to [instrumentation classes](metrics_instrumentation.md). +For information about the progress on migrating Service ping metrics, see this [epic](https://gitlab.com/groups/gitlab-org/-/epics/5547). + ## Instrumentation classes We recommend you use [instrumentation classes](metrics_instrumentation.md) in `usage_data.rb` where possible. @@ -795,7 +799,7 @@ To set up Service Ping locally, you must: 1. [Set up local repositories](#set-up-local-repositories). 1. [Test local setup](#test-local-setup). -1. (Optional) [Test Prometheus-based Service Ping](#test-prometheus-based-service-ping). +1. Optional. [Test Prometheus-based Service Ping](#test-prometheus-based-service-ping). ### Set up local repositories diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 6ddbe2f9646..1f751eea4d8 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -8,40 +8,24 @@ info: To determine the technical writer assigned to the Stage/Group associated w > Introduced in GitLab Ultimate 11.2, more statistics. -This guide describes Service Ping's purpose and how it's implemented. - -For more information about Product Intelligence, see: - -- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) -- [Snowplow Guide](../snowplow/index.md) - -More links: - -- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) -- [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/) -- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) - -## What is Service Ping? - -Service Ping is a process in GitLab that collects and sends a weekly payload to GitLab Inc. +Service Ping is a GitLab process that collects and sends a weekly payload to GitLab. The payload provides important high-level data that helps our product, support, -and sales teams understand how GitLab is used. For example, the data helps to: +and sales teams understand how GitLab is used. The data helps to: - Compare counts month over month (or week over week) to get a rough sense for how an instance uses different product features. - Collect other facts that help us classify and understand GitLab installations. -- Calculate our Stage Monthly Active Users (SMAU), which helps to measure the success of our stages +- Calculate our stage monthly active users (SMAU), which helps to measure the success of our stages and features. -Service Ping information is not anonymous. It's linked to the instance's hostname. However, it does +Service Ping information is not anonymous. It's linked to the instance's hostname, but does not contain project names, usernames, or any other specific data. -Sending a Service Ping payload is optional and can be [disabled](#disable-service-ping) on any self-managed instance. -When Service Ping is enabled, GitLab gathers data from the other instances +Sending a Service Ping payload is optional and you can [disable](#disable-service-ping) it on any +self-managed instance. When Service Ping is enabled, GitLab gathers data from the other instances and can show your instance's usage statistics to your users. -### Terminology +## Service Ping terminology We use the following terminology to describe the Service Ping components: @@ -53,12 +37,18 @@ We use the following terminology to describe the Service Ping components: - **MAU**: monthly active users. - **WAU**: weekly active users. -### Why should we enable Service Ping? +### Why enable Service Ping? + +The main purpose of Service Ping is to build a better GitLab. We collect data about how GitLab is used +to understand feature or stage adoption and usage. This data gives an insight into how GitLab adds +value and helps our team understand the reasons why people use GitLab, and with this knowledge we're able to +make better product decisions. + +There are several other benefits to enabling Service Ping: -- The main purpose of Service Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions. - As a benefit of having Service Ping active, GitLab lets you analyze the users' activities over time of your GitLab installation. - As a benefit of having Service Ping active, GitLab provides you with [DevOps Score](../../user/admin_area/analytics/dev_ops_report.md#devops-score), which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring. -- You get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value) +- You get better, more proactive support (assuming that our TAMs and support organization used the data to deliver more value). - You get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? - You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. - Service Ping is enabled by default. To disable it, see [Disable Service Ping](#disable-service-ping). @@ -78,7 +68,7 @@ Because of these limitations we recommend you: > Introduced in GitLab 14.1. -In GitLab versions 14.1 and later, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data through [Service Ping](#what-is-service-ping). Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data. +In GitLab versions 14.1 and later, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data through Service Ping. Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data. #### Features available in 14.1 and later @@ -484,20 +474,34 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta 1. Connect to bastion with agent forwarding: - `$ ssh -A lb-bastion.gprd.gitlab.com`. + ```shell + ssh -A lb-bastion.gprd.gitlab.com + ``` + 1. Create named screen: - `$ screen -S <username>_usage_ping_<date>`. + ```shell + screen -S <username>_usage_ping_<date> + ``` + 1. Connect to console host: + + ```shell + ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal + ``` - `$ ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal`. -1. Run `ServicePing::SubmitService.new.execute` -1. Detach from screen: +1. Run: - `ctrl + a, ctrl + d` -1. Exit from bastion: + ```shell + ServicePing::SubmitService.new.execute + ``` - `$ exit` +1. To detach from screen, press `ctrl + A`, `ctrl + D`. +1. Exit from bastion: + + ```shell + exit + ``` ### Verification (After approx 30 hours) @@ -511,22 +515,49 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta 1. Reconnect to bastion: - `$ ssh -A lb-bastion.gprd.gitlab.com` + ```shell + ssh -A lb-bastion.gprd.gitlab.com + ``` + 1. Find your screen session: - `$ screen -ls` + ```shell + screen -ls + ``` + 1. Attach to your screen session: - `$ screen -x 14226.mwawrzyniak_usage_ping_2021_01_22` -1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload` -1. Check the when the payload was sent: `RawUsageData.last.sent_at` + ```shell + screen -x 14226.mwawrzyniak_usage_ping_2021_01_22 + ``` + +1. Check the last payload in `raw_usage_data` table: + + ```shell + RawUsageData.last.payload + ``` + +1. Check the when the payload was sent: + + ```shell + RawUsageData.last.sent_at + ``` + +### Skip database write operations + +To skip database write operations, DevOps report creation, and storage of usage data payload, pass an optional argument: + +```shell +skip_db_write: +ServicePing::SubmitService.new(skip_db_write: true).execute +``` ## Troubleshooting ### Cannot disable Service Ping using the configuration file The method to disable Service Ping using the GitLab configuration file does not work in -GitLab versions 9.3.0 to 13.12.3. To disable it, you need to use the Admin Area in +GitLab versions 9.3.0 to 13.12.3. To disable it, you must use the Admin Area in the GitLab UI instead. For more information, see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/333269). @@ -603,3 +634,12 @@ To work around this bug, you have two options: 1. Expand **Usage Statistics**. 1. Clear the **Enable Service Ping** checkbox. 1. Select **Save Changes**. + +## Related topics + +- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) +- [Snowplow Guide](../snowplow/index.md) +- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) +- [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/) +- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/) +- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index b3c404de150..808c5064cf3 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -181,6 +181,7 @@ product_group: group::product intelligence value_type: string status: active milestone: 9.1 +instrumentation_class: UuidMetric introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1521 time_frame: none data_source: database @@ -199,22 +200,23 @@ The GitLab codebase provides a dedicated [generator](https://gitlab.com/gitlab-o For uniqueness, the generated files include a timestamp prefix in ISO 8601 format. -The generator takes a list of key paths and 2 options as arguments. It creates metric YAML definitions in the corresponding location: +The generator takes a list of key paths and 3 options as arguments. It creates metric YAML definitions in the corresponding location: - `--ee`, `--no-ee` Indicates if metric is for EE. - `--dir=DIR` Indicates the metric directory. It must be one of: `counts_7d`, `7d`, `counts_28d`, `28d`, `counts_all`, `all`, `settings`, `license`. +- `--class_name=CLASS_NAME` Indicates the instrumentation class. For example `UsersCreatingIssuesMetric`, `UuidMetric` **Single metric example** ```shell -bundle exec rails generate gitlab:usage_metric_definition counts.issues --dir=7d +bundle exec rails generate gitlab:usage_metric_definition counts.issues --dir=7d --class_name=CountIssues create config/metrics/counts_7d/issues.yml ``` **Multiple metrics example** ```shell -bundle exec rails generate gitlab:usage_metric_definition counts.issues counts.users --dir=7d +bundle exec rails generate gitlab:usage_metric_definition counts.issues counts.users --dir=7d --class_name=CountUsersCreatingIssues create config/metrics/counts_7d/issues.yml create config/metrics/counts_7d/users.yml ``` @@ -223,7 +225,7 @@ NOTE: To create a metric definition used in EE, add the `--ee` flag. ```shell -bundle exec rails generate gitlab:usage_metric_definition counts.issues --ee --dir=7d +bundle exec rails generate gitlab:usage_metric_definition counts.issues --ee --dir=7d --class_name=CountUsersCreatingIssues create ee/config/metrics/counts_7d/issues.yml ``` @@ -236,9 +238,9 @@ A YAML metric definition is required for each metric. A dedicated generator is p The generator takes `category` and `event` arguments, as the root key is `redis_hll_counters`, and creates two metric definitions for weekly and monthly time frames: ```shell -bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues i_closed -create config/metrics/counts_7d/i_closed_weekly.yml -create config/metrics/counts_28d/i_closed_monthly.yml +bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues count_users_closing_issues +create config/metrics/counts_7d/count_users_closing_issues_weekly.yml +create config/metrics/counts_28d/count_users_closing_issues_monthly.yml ``` To create a metric definition used in EE, add the `--ee` flag. diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md index 46040146de2..ebfab6341e9 100644 --- a/doc/development/service_ping/metrics_lifecycle.md +++ b/doc/development/service_ping/metrics_lifecycle.md @@ -55,23 +55,24 @@ The correct approach is to add a new metric for GitLab 12.6 release with updated and update existing business analysis artefacts to use `example_metric_without_archived` instead of `example_metric` -## Deprecate a metric +## Remove a metric + +WARNING: +If a metric is not used in Sisense or any other system after 6 months, the +Product Intelligence team marks it as inactive and assigns it to the group owner for review. + +We are working on automating this process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338466) for details. -If a metric is obsolete and you no longer use it, you can mark it as deprecated. +Product Intelligence removes metrics from Service Ping if they are not used in any Sisense dashboard. -For an example of the metric deprecation process take a look at this [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59899) +For an example of the metric removal process, see this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297029). -To deprecate a metric: +To remove a metric: 1. Check the following YAML files and verify the metric is not used in an aggregate: - [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/) - [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/) -1. Create an issue in the [GitLab Data Team - project](https://gitlab.com/gitlab-data/analytics/-/issues). Ask for - confirmation that the metric is not used by other teams, or in any of the SiSense - dashboards. - 1. Verify the metric is not used to calculate the conversational index. The conversational index is a measure that reports back to self-managed instances to inform administrators of the progress of DevOps adoption for the instance. @@ -81,70 +82,6 @@ To deprecate a metric: to view the metrics that are used. The metrics are represented as the keys that are passed as a field argument into the `get_value` method. -1. Document the deprecation in the metric's YAML definition. Set - the `status:` attribute to `deprecated`, for example: - - ```yaml - --- - key_path: analytics_unique_visits.analytics_unique_visits_for_any_target_monthly - description: Visits to any of the pages listed above per month - product_section: dev - product_stage: manage - product_group: group::analytics - product_category: - value_type: number - status: deprecated - time_frame: 28d - data_source: - distribution: - - ce - tier: - - free - ``` - -1. Replace the metric's instrumentation with a fixed value. This avoids wasting - resources to calculate the deprecated metric. In - [`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) - or - [`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb), - replace the code that calculates the metric's value with a fixed value that - indicates it's deprecated: - - ```ruby - module Gitlab - class UsageData - DEPRECATED_VALUE = -1000 - - def analytics_unique_visits_data - results['analytics_unique_visits_for_any_target'] = redis_usage_data { unique_visit_service.unique_visits_for(targets: :analytics) } - results['analytics_unique_visits_for_any_target_monthly'] = DEPRECATED_VALUE - - { analytics_unique_visits: results } - end - # ... - end - end - ``` - -## Remove a metric - -### Removal policy - -WARNING: -A metric that is not used in Sisense or any other system after 6 months is marked by the -Product Intelligence team as inactive and is assigned to the group owner for review. - -We are working on automating this process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338466) for details. - -Metrics can be removed from Service Ping if they: - -- Were previously [deprecated](#deprecate-a-metric). -- Are not used in any Sisense dashboard. - -For an example of the metric removal process take a look at this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297029) - -### To remove a deprecated metric - 1. Verify that removing the metric from the Service Ping payload does not cause errors in [Version App](https://gitlab.com/gitlab-services/version-gitlab-com) when the updated payload is collected and processed. Version App collects @@ -159,9 +96,6 @@ For an example of the metric removal process take a look at this [example issue] Ask for confirmation that the metric is not referred to in any SiSense dashboards and can be safely removed from Service Ping. Use this [example issue](https://gitlab.com/gitlab-data/analytics/-/issues/7539) for guidance. - This step can be skipped if verification done during [deprecation process](#deprecate-a-metric) - reported that metric is not required by any data transformation in Snowflake data warehouse nor it is - used by any of SiSense dashboards. 1. After you verify the metric can be safely removed, update the attributes of the metric's YAML definition: diff --git a/doc/development/session.md b/doc/development/session.md index bee857da323..61a130e3a53 100644 --- a/doc/development/session.md +++ b/doc/development/session.md @@ -51,12 +51,12 @@ Session data can be accessed directly through Redis. This can let you check up o ```ruby # Get a list of sessions -session_ids = Gitlab::Redis::SharedState.with do |redis| - redis.smembers("#{Gitlab::Redis::SharedState::USER_SESSIONS_LOOKUP_NAMESPACE}:#{user.id}") +session_ids = Gitlab::Redis::Sessions.with do |redis| + redis.smembers("#{Gitlab::Redis::Sessions::USER_SESSIONS_LOOKUP_NAMESPACE}:#{user.id}") end # Retrieve a specific session -session_data = Gitlab::Redis::SharedState.with { |redis| redis.get("#{Gitlab::Redis::SharedState::SESSION_NAMESPACE}:#{session_id}") } +session_data = Gitlab::Redis::Sessions.with { |redis| redis.get("#{Gitlab::Redis::Sessions::SESSION_NAMESPACE}:#{session_id}") } Marshal.load(session_data) ``` diff --git a/doc/development/shell_scripting_guide/index.md b/doc/development/shell_scripting_guide/index.md index d3b446d45da..3d58fabad72 100644 --- a/doc/development/shell_scripting_guide/index.md +++ b/doc/development/shell_scripting_guide/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab consists of many various services and sub-projects. The majority of their backend code is written in [Ruby](https://www.ruby-lang.org) and -[Go](https://golang.org). However, some of them use shell scripts for +[Go](https://go.dev/). However, some of them use shell scripts for automation of routine system administration tasks like deployment, installation, etc. It's being done either for historical reasons or as an effort to minimize the dependencies, for instance, for Docker images. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index e28a328888d..2137a7d83e6 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -342,6 +342,7 @@ end > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69372) in GitLab 14.3. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.4. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.6. The deduplication always take into account the latest binary replication pointer, not the first one. This happens because we drop the same job scheduled for the second time and the Write-Ahead Log (WAL) is lost. @@ -353,15 +354,11 @@ This way we are always comparing the latest binary replication pointer, making sure that we read from the replica that is fully caught up. FLAG: -On self-managed GitLab, by default this feature is not available. -To make it available, -ask an administrator to [enable the preserve_latest_wal_locations_for_idempotent_jobs flag](../administration/feature_flags.md). -FLAG: -On self-managed GitLab, by default this feature is not available. -To make it available, -ask an administrator to [enable the `preserve_latest_wal_locations_for_idempotent_jobs` flag](../administration/feature_flags.md). +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to +[disable the feature flag](../administration/feature_flags.md) named `preserve_latest_wal_locations_for_idempotent_jobs`. + This feature flag is related to GitLab development and is not intended to be used by GitLab administrators, though. -On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. +On GitLab.com, this feature is available. ## Limited capacity worker @@ -574,7 +571,7 @@ of reading a stale record is non-zero due to replicas potentially lagging behind When the number of jobs that rely on the database increases, ensuring immediate data consistency can put unsustainable load on the primary database server. We therefore added the ability to use -[database load balancing for Sidekiq workers](../administration/database_load_balancing.md#load-balancing-for-sidekiq). +[Database Load Balancing for Sidekiq workers](../administration/postgresql/database_load_balancing.md). By configuring a worker's `data_consistency` field, we can then allow the scheduler to target read replicas under several strategies outlined below. diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md index fe1de789eae..6da4896c7e7 100644 --- a/doc/development/snowplow/implementation.md +++ b/doc/development/snowplow/implementation.md @@ -13,12 +13,12 @@ This page describes how to: ## Snowplow JavaScript frontend tracking -GitLab provides a `Tracking` interface that wraps the [Snowplow JavaScript tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/) -to track custom events. +GitLab provides a `Tracking` interface that wraps the [Snowplow JavaScript tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/) +to track custom events. For the recommended frontend tracking implementation, see [Usage recommendations](#usage-recommendations). -Tracking implementations must have an `action` and a `category`. You can provide additional +Tracking implementations must have an `action` and a `category`. You can provide additional categories from the [structured event taxonomy](index.md#structured-event-taxonomy) with an `extra` object that accepts key-value pairs. @@ -60,15 +60,15 @@ The following example shows `data-track-*` attributes assigned to a button: | `data-track-action` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field is `activate_form_input` and clicking a button is `click_button`. Replaces `data-track-event`, which was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290962) in GitLab 13.11. | | `data-track-label` | false | The specific element or object to act on. This can be: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | | `data-track-property` | false | Any additional property of the element, or object being acted on. | -| `data-track-value` | false | Describes a numeric value or something directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. | +| `data-track-value` | false | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. | | `data-track-extra` | false | A key-value pair object passed as a valid JSON string. This attribute is added to the `extra` property in our [`gitlab_standard`](schemas.md#gitlab_standard) schema. | | `data-track-context` | false | The `context` as described in our [Structured event taxonomy](index.md#structured-event-taxonomy). | #### Event listeners Event listeners bind at the document level to handle click events in elements with data attributes. -This allows them to be handled when the DOM re-renders or changes. Document-level binding reduces -the likelihood that click events stop propagating up the DOM tree. +This allows them to be handled when the DOM re-renders or changes. Document-level binding reduces +the likelihood that click events stop propagating up the DOM tree. If click events stop propagating, you must implement listeners and [Vue component tracking](#implement-vue-component-tracking) or [raw JavaScript tracking](#implement-raw-javascript-tracking). @@ -102,12 +102,12 @@ track_action: "click_button" }) ### Implement Vue component tracking For custom event tracking, use a Vue `mixin` in components. Vue `mixin` exposes the `Tracking.event` -static method and the `track` method. You can specify tracking options in `data` or `computed`. -These options override any defaults and allow the values to be dynamic from props or based on state. +static method and the `track` method. You can specify tracking options in `data` or `computed`. +These options override any defaults and allow the values to be dynamic from props or based on state. -Several default options are passed when an event is tracked from the component: +Several default options are passed when an event is tracked from the component: -- `category`: If you don't specify, by default `document.body.dataset.page` is used. +- `category`: If you don't specify, by default `document.body.dataset.page` is used. - `label` - `property` - `value` @@ -121,7 +121,7 @@ To implement Vue component tracking: const trackingMixin = Tracking.mixin; ``` -1. Provide categories to track the event from the component. For example, to track all events in a +1. Provide categories to track the event from the component. For example, to track all events in a component with a label, use the `label` category: ```javascript @@ -293,14 +293,14 @@ describe('MyTracking', () => { ### Form tracking -To enable Snowplow automatic [form tracking](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking): +To enable Snowplow automatic [form tracking](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking): -1. Call `Tracking.enableFormTracking` when the DOM is ready. +1. Call `Tracking.enableFormTracking` when the DOM is ready. 1. Provide a `config` object that includes at least one of the following elements: - `forms` determines the forms to track. Identified by the CSS class name. - `fields` determines the fields inside the tracked forms to track. Identified by the field `name`. 1. Optional. Provide a list of contexts as the second argument. The [`gitlab_standard`](schemas.md#gitlab_standard) schema is excluded from these events. - + ```javascript Tracking.enableFormTracking({ forms: { allow: ['sign-in-form', 'password-recovery-form'] }, @@ -339,7 +339,7 @@ Backend tracking provides: - User behavior tracking - Instrumentation to monitor and visualize performance over time in a section or aspect of code. -To add custom event tracking and instrumentation, call the `GitLab::Tracking.event` class method. +To add custom event tracking and instrumentation, call the `GitLab::Tracking.event` class method. For example: ```ruby @@ -361,7 +361,7 @@ Use the following arguments: | `action` | String | | The action being taken. For example, a controller action such as `create`, or an Active Record callback. | | `label` | String | nil | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | | `property` | String | nil | Any additional property of the element, or object being acted on. | -| `value` | Numeric | nil | Describes a numeric value or something directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | +| `value` | Numeric | nil | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | | `context` | Array\[SelfDescribingJSON\] | nil | An array of custom contexts to send with this event. Most events should not have any custom contexts. | | `project` | Project | nil | The project associated with the event. | | `user` | User | nil | The user associated with the event. | @@ -370,7 +370,7 @@ Use the following arguments: ### Unit testing -To test backend Snowplow events, use the `expect_snowplow_event` helper. For more information, see +To test backend Snowplow events, use the `expect_snowplow_event` helper. For more information, see [testing best practices](../testing_guide/best_practices.md#test-snowplow-events). ### Performance @@ -419,17 +419,24 @@ Snowplow Inspector Chrome Extension is a browser extension for testing frontend [Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) is a Docker-based solution for testing backend and frontend in a local development environment. Snowplow Micro -records the same events as the full Snowplow pipeline. To query events, use the Snowplow Micro API. +records the same events as the full Snowplow pipeline. To query events, use the Snowplow Micro API. + +It can be set up automatically using [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit). +See the [how-to docs](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/snowplow_micro.md) for more details. + +Optionally, you can set it up manually, using the following instructions. + +#### Set up Snowplow Micro manually -To install and run Snowplow Micro, complete these steps to modify the +To install and run Snowplow Micro, complete these steps to modify the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit): 1. Ensure [Docker is installed](https://docs.docker.com/get-docker/) and running. -1. To install Snowplow Micro, clone the settings in +1. To install Snowplow Micro, clone the settings in [this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration). -1. Navigate to the directory with the cloned project, +1. Navigate to the directory with the cloned project, and start the appropriate Docker container: ```shell diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index f4f41221699..dbc7a25075f 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -90,7 +90,7 @@ Each click event provides attributes that describe the event. | action | text | true | The action the user takes, or aspect that's being instrumented. The first word must describe the action or aspect. For example, clicks must be `click`, activations must be `activate`, creations must be `create`. Use underscores to describe what was acted on. For example, activating a form field is `activate_form_input`, an interface action like clicking on a dropdown is `click_dropdown`, a behavior like creating a project record from the backend is `create_project`. | | label | text | false | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | | property | text | false | Any additional property of the element, or object being acted on. | -| value | decimal | false | Describes a numeric value or something directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | +| value | decimal | false | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | ### Examples @@ -149,6 +149,19 @@ ORDER BY page_view_start DESC LIMIT 100 ``` +#### Query JSON formatted data + +```sql +SELECT + derived_tstamp, + contexts:data[0]:data:extra:old_format as CURRENT_FORMAT, + contexts:data[0]:data:extra:value as UPDATED_FORMAT +FROM legacy.snowplow_structured_events_all +WHERE event_action in ('wiki_format_updated') +ORDER BY derived_tstamp DESC +LIMIT 100 +``` + ### Web-specific parameters Snowplow JavaScript adds [web-specific parameters](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. diff --git a/doc/development/snowplow/schemas.md b/doc/development/snowplow/schemas.md index 5b9e4f5256e..f66e0566a9c 100644 --- a/doc/development/snowplow/schemas.md +++ b/doc/development/snowplow/schemas.md @@ -18,6 +18,7 @@ The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/g |----------------|---------------------|-----------------------|---------------------------------------------------------------------------------------------| | `project_id` | **{dotted-circle}** | integer | | | `namespace_id` | **{dotted-circle}** | integer | | +| `user_id` | **{dotted-circle}** | integer | User database record ID attribute. This file undergoes a pseudonymization process at the collector level. | | `environment` | **{check-circle}** | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | | `source` | **{check-circle}** | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | | `plan` | **{dotted-circle}** | string (max 32 chars) | Name of the plan for the namespace, such as `free`, `premium`, or `ultimate`. Automatically picked from the `namespace`. | diff --git a/doc/development/sql.md b/doc/development/sql.md index 129280598fe..e2208caf35a 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -249,6 +249,9 @@ In line with our `CodeReuse/ActiveRecord` cop, you should only use forms like use the `ApplicationRecord`-provided `.pluck_primary_key` helper method instead. In the latter, you should add a small helper method to the relevant model. +If you have strong reasons to use `pluck`, it could make sense to limit the number +of records plucked. `MAX_PLUCK` defaults to `1_000` in `ApplicationRecord`. + ## Inherit from ApplicationRecord Most models in the GitLab codebase should inherit from `ApplicationRecord`, diff --git a/doc/development/stage_group_dashboards.md b/doc/development/stage_group_dashboards.md index a887558e473..88e9141574e 100644 --- a/doc/development/stage_group_dashboards.md +++ b/doc/development/stage_group_dashboards.md @@ -60,8 +60,8 @@ component can have 2 indicators: We're working on making this target configurable per endpoint in [this project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525). Learn - how to [customize the request - apdex](application_slis/rails_request_apdex.md), this new apdex + how to + [customize the request Apdex](application_slis/rails_request_apdex.md), this new Apdex measurement is not yet part of the error budget. For Sidekiq job execution, the threshold depends on the [job diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 6a739d9e1a5..0f768a51b66 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -483,6 +483,43 @@ expect(page).to have_css '[data-testid="weight"]', text: 2 expect(page).to have_css '.atwho-view ul', visible: true ``` +##### Interacting with modals + +Use the `within_modal` helper to interact with [GitLab UI modals](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-modal--default). + +```ruby +include Spec::Support::Helpers::ModalHelpers + +within_modal do + expect(page).to have_link _('UI testing docs') + + fill_in _('Search projects'), with: 'gitlab' + + click_button 'Continue' +end +``` + +Furthermore, you can use `accept_gl_confirm` for confirmation modals that only need to be accepted. +This is helpful when migrating [`window.confirm()`](https://developer.mozilla.org/en-US/docs/Web/API/Window/confirm) to [`confirmAction`](https://gitlab.com/gitlab-org/gitlab/-/blob/ee280ed2b763d1278ad38c6e7e8a0aff092f617a/app/assets/javascripts/lib/utils/confirm_via_gl_modal/confirm_via_gl_modal.js#L3). + +```ruby +include Spec::Support::Helpers::ModalHelpers + +accept_gl_confirm do + click_button 'Delete user' +end +``` + +You can also pass the expected confirmation message and button text to `accept_gl_confirm`. + +```ruby +include Spec::Support::Helpers::ModalHelpers + +accept_gl_confirm('Are you sure you want to delete this user?', button_text: 'Delete') do + click_button 'Delete user' +end +``` + ##### Other useful methods After you retrieve an element using a [finder method](#finders), you can invoke a number of @@ -702,6 +739,42 @@ it 'is overdue' do end ``` +#### RSpec helpers + +You can use the `:freeze_time` and `:time_travel_to` RSpec metadata tag helpers to help reduce the amount of +boilerplate code needed to wrap entire specs with the [`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) +methods. + +```ruby +describe 'specs which require time to be frozen', :freeze_time do + it 'freezes time' do + right_now = Time.now + + expect(Time.now).to eq(right_now) + end +end + +describe 'specs which require time to be frozen to a specific date and/or time', time_travel_to: '2020-02-02 10:30:45 -0700' do + it 'freezes time to the specified date and time' do + expect(Time.now).to eq(Time.new(2020, 2, 2, 17, 30, 45, '+00:00')) + end +end +``` + +[Under the hood](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/time_travel.rb), these helpers use the `around(:each)` hook and the block syntax of the +[`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) +methods: + +```ruby +around(:each) do |example| + freeze_time { example.run } +end + +around(:each) do |example| + travel_to(date_or_time) { example.run } +end +``` + ### Feature flags in tests This section was moved to [developing with feature flags](../feature_flags/index.md). @@ -891,7 +964,7 @@ creates and deletes indices between examples to ensure a clean index, so that th for polluting the tests with nonessential data. Most tests for Elasticsearch logic relate to: -- Creating data in Postgres and waiting for it to be indexed in Elasticsearch. +- Creating data in PostgreSQL and waiting for it to be indexed in Elasticsearch. - Searching for that data. - Ensuring that the test gives the expected result. @@ -907,7 +980,7 @@ You do NOT need to add `:clean_gitlab_redis_shared_state` manually. Specs using Elasticsearch require that you: -- Create data in Postgres and then index it into Elasticsearch. +- Create data in PostgreSQL and then index it into Elasticsearch. - Enable Application Settings for Elasticsearch (which is disabled by default). To do so, use: @@ -921,7 +994,7 @@ end Additionally, you can use the `ensure_elasticsearch_index!` method to overcome the asynchronous nature of Elasticsearch. It uses the [Elasticsearch Refresh API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html#refresh-api-desc) to make sure all operations performed on an index since the last refresh are available for search. This method is typically -called after loading data into Postgres to ensure the data is indexed and searchable. +called after loading data into PostgreSQL to ensure the data is indexed and searchable. #### Test Snowplow events diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 9491c89c2a0..543feaa967c 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -16,7 +16,7 @@ In case custom inflection logic is needed, custom inflectors are added in the [q ## Link a test to its test case -Every test should have a corresponding test case as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/). +Every test should have a corresponding test case in the [GitLab project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/-/issues). It's recommended that you reuse the issue created to plan the test as the results issue. If a test case or results issue does not already exist you can create them yourself. Alternatively, you can run the test in a pipeline that has reporting enabled and the test-case reporter will automatically create a new test case and/or results issue and link the results issue to it's corresponding test case. @@ -30,11 +30,11 @@ For example: ```ruby RSpec.describe 'Stage' do describe 'General description of the feature under test' do - it 'test name', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/:test_case_id' do + it 'test name', testcase: 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/:test_case_id' do ... end - it 'another test', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/:another_test_case_id' do + it 'another test', testcase: 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/:another_test_case_id' do ... end end @@ -92,7 +92,7 @@ end There would be two associated test cases, one for each shared example, with the following content: -[Test 1 Test Case](https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/1491): +[Test 1 Test Case](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347774): ````markdown ```markdown @@ -115,12 +115,12 @@ pushes and merges Active and historical test results: -https://gitlab.com/gitlab-org/quality/testcases/-/issues/600 +https://gitlab.com/gitlab-org/quality/testcases/-/issues/2177 ``` ```` -[Test 1 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/600): +[Test 1 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/2177): ````markdown ```markdown @@ -142,7 +142,7 @@ pushes and merges ``` ```` -[Test 2 Test Case](https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/602): +[Test 2 Test Case](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347775): ````markdown ```markdown @@ -165,12 +165,12 @@ user fails to push Active and historical test results: -https://gitlab.com/gitlab-org/quality/testcases/-/issues/602 +https://gitlab.com/gitlab-org/quality/testcases/-/issues/2176 ``` ```` -[Test 2 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/602): +[Test 2 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/2176): ````markdown ```markdown diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 52212410cc6..de34e6a1872 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -67,6 +67,57 @@ If no scope is provided, the feature flag is set instance-wide: Runtime::Feature.enable(:feature_flag_name) ``` +## Working with selectors + +A new feature often replaces a `vue` component or a `haml` file with a new one. +In most cases, the new file or component is accessible only with a feature flag. +This approach becomes problematic when tests must pass both with, and without, +the feature flag enabled. To ensure tests pass in both scenarios: + +1. Create another selector inside the new component or file. +1. Give it the same name as the old one. + +Selectors are connected to a specific frontend file in the [page object](page_objects.md), +and checked for availability inside our `qa:selectors` test. If the mentioned selector +is missing inside that frontend file, the test fails. To ensure selectors are +available when a feature flag is enabled or disabled, add the new selector to the +[page object](page_objects.md), leaving the old selector in place. +The test uses the correct selector and still detects missing selectors. + +If a new feature changes an existing frontend file that already has a selector, +you can add a new selector with the same name. However, only one of the selectors +displays on the page. You should: + +1. Disable the other with the feature flag. +1. Add a comment in the frontend file to delete the old selector from the frontend + file and from the page object file when the feature flag is removed. + +### Example before + +```ruby +# This is the link to the old file +view 'app/views/devise/passwords/edit.html.haml' do + # The new selector should have the same name + element :password_field + ... +end +``` + +### Example after + +```ruby +view 'app/views/devise/passwords/edit.html.haml' do + element :password_field + ... +end + +# Now it can verify the selector is available +view 'app/views/devise/passwords/new_edit_behind_ff.html.haml' do + # The selector has the same name + element :password_field +end +``` + ## Running a scenario with a feature flag enabled It's also possible to run an entire scenario with a feature flag enabled, without having to edit diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index 0819a2f7b54..abba1d51f07 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -4,14 +4,21 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Resource class in GitLab QA +# Resource classes in GitLab QA -Resources are primarily created using Browser UI steps, but can also -be created via the API or the CLI. +Resources are primarily created using Browser UI steps, but can also be created via the API or the CLI. + +A typical resource class is used to create a new resource that can be used in a single test. However, several tests can +end up creating the same kind of resource and use it in ways that mean it could have been +used by more than one test. Creating a new resource each time is not efficient. Therefore, we can also create reusable +resources that are created once and can then be used by many tests. + +In the following section the content focuses on single-use resources, however it also applies to reusable resources. +Information specific to [reusable resources is detailed below](#reusable-resources). ## How to properly implement a resource class? -All resource classes should inherit from `Resource::Base`. +All non-reusable resource classes should inherit from `Resource::Base`. There is only one mandatory method to implement to define a resource class. This is the `#fabricate!` method, which is used to build the resource via the @@ -391,6 +398,96 @@ end In this case, the result is similar to calling `Resource::Shirt.fabricate!`. +## Reusable resources + +Reusable resources are created by the first test that needs a particular kind of resource, and then any test that needs +the same kind of resource can reuse it instead of creating a new one. + +The `ReusableProject` resource is an example of this class: + +```ruby +module QA + module Resource + class ReusableProject < Project # A reusable resource inherits from the resource class that we want to be able to reuse. + prepend Reusable # The Reusable module mixes in some methods that help implement reuse. + + def initialize + super # A ReusableProject is a Project so it should be initialized as one. + + # Some Project attributes aren't valid and need to be overridden. For example, a ReusableProject keeps its name once it's created, + # so we don't add a random string to the name specified. + @add_name_uuid = false + + # It has a default name, and a different name can be specified when a resource is first created. However, the same name must be + # provided any time that instance of the resource is used. + @name = "reusable_project" + + # Several instances of a ReusableProject can exists as long as each is identified via a unique value for `reuse_as`. + @reuse_as = :default_project + end + + # All reusable resource classes must validate that an instance meets the conditions that allow reuse. For example, + # by confirming that the name specified for the instance is valid and doesn't conflict with other instances. + def validate_reuse_preconditions + raise ResourceReuseError unless reused_name_valid? + end + end + end +end +``` + +Consider some examples of how a reusable resource is used: + +```ruby +# This will create a project. +default_project = Resource::ReusableProject.fabricate_via_api! +default_project.name # => "reusable_project" +default_project.reuse_as # => :default_project +``` + +Then in another test we could reuse the project: + +```ruby +# This will fetch the project created above rather than creating a new one. +default_project_again = Resource::ReusableProject.fabricate_via_api! +default_project_again.name # => "reusable_project" +default_project_again.reuse_as # => :default_project +``` + +We can also create another project that we want to change in a way that might not be suitable for tests using the +default project: + +```ruby +project_with_member = Resource::ReusableProject.fabricate_via_api! do |project| + project.name = "project-with-member" + project.reuse_as = :project_with_member +end + +project_with_member.add_member(user) +``` + +Another test can reuse that project: + +```ruby +project_still_has_member = Resource::ReusableProject.fabricate_via_api! do |project| + project.name = "project-with-member" + project.reuse_as = :project_with_member +end + +expect(project_still_has_member).to have_member(user) +``` + +However, if we don't provide the name again an error will be raised: + +```ruby +Resource::ReusableProject.fabricate_via_api! do |project| + project.reuse_as = :project_with_member +end + +# => ResourceReuseError will be raised because it will try to use the default name, "reusable_project", which doesn't +# match the name specified when the project was first fabricated. +``` + ## Where to ask for help? If you need more information, ask for help on `#quality` channel on Slack diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 3749511fef5..cb15dbe023f 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -20,6 +20,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:github` | The test requires a GitHub personal access token. | | `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | | `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | +| `:service_ping_disabled` | The test interacts with the GitLab configuration service ping at the instance level to turn admin setting service ping checkbox on or off. This tag will have the test run only in the `service_ping_disabled` job and must be paired with the `:orchestrated` and `:requires_admin` tags. | | `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. | `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | | `:ldap_no_server` | The test requires a GitLab instance to be configured to use LDAP. To be used with the `:orchestrated` tag. It does not spin up an LDAP server at orchestration time. Instead, it creates the LDAP server at runtime. | @@ -43,6 +44,6 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. | | `:smoke` | The test belongs to the test suite which verifies basic functionality of a GitLab instance.| | `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | -| `:testcase` | The link to the test case issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/). | +| `:testcase` | The link to the test case issue in the [GitLab Project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases). | | `:transient` | The test tests transient bugs. It is excluded by default. | | `:issue`, `:issue_${num}` | Optional links to issues which might be related to the spec. Helps keeping track of related issues and can also be used by tools that create test reports. Currently added automatically to `Allure` test report. Multiple tags can be used by adding optional number postfix like `issue_1`, `issue_2` etc. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 95984a701e7..ef3e0624395 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -294,7 +294,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you need a r export EE_LICENSE=$(cat <path/to/your/gitlab_license>) ``` -1. (Optional) Pull the GitLab image +1. Optional. Pull the GitLab image This step is optional because pulling the Docker image is part of the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb). However, it's easier to monitor the download progress if you pull the image first, and the scenario skips this step after checking that the image is up to date. @@ -496,12 +496,12 @@ Tests that are tagged with `:mobile` can be run against specified mobile devices Running directly against an environment like staging is not recommended because Sauce Labs test logs expose credentials. Therefore, it is best practice and the default to use a tunnel. -Tunnel installation instructions are here [https://docs.saucelabs.com/secure-connections/sauce-connect/installation]. To start the tunnel, after following the installation above, copy the run command in Sauce Labs > Tunnels (must be logged in to Sauce Labs with the credentials found in 1Password) and run in terminal. +For tunnel installation instructions, read [Sauce Connect Proxy Installation](https://docs.saucelabs.com/secure-connections/sauce-connect/installation). To start the tunnel, after following the installation above, copy the run command in Sauce Labs > Tunnels (must be logged in to Sauce Labs with the credentials found in 1Password) and run in terminal. NOTE: It is highly recommended to use `GITLAB_QA_ACCESS_TOKEN` to speed up tests and reduce flakiness. -`QA_REMOTE_MOBILE_DEVICE_NAME` can be any device name listed in [https://saucelabs.com/platform/supported-browsers-devices] under Emulators/simulators and the latest versions of Android or iOS. `QA_BROWSER` must be set to `safari` for iOS devices and `chrome` for Android devices. +`QA_REMOTE_MOBILE_DEVICE_NAME` can be any device name listed in [Supported browsers and devices](https://saucelabs.com/platform/supported-browsers-devices) under Emulators/simulators and the latest versions of Android or iOS. `QA_BROWSER` must be set to `safari` for iOS devices and `chrome` for Android devices. 1. To test against a local instance with a tunnel running, in `gitlab/qa` run: diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 9489020de5d..d2e68ea7715 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -82,26 +82,19 @@ These flaky tests can fail depending on the order they run with other tests. For - <https://gitlab.com/gitlab-org/gitlab/-/issues/327668> -To identify the tests that lead to such failure, we can use `rspec --bisect`, +To identify the tests that lead to such failure, we can use `scripts/rspec_bisect_flaky`, which would give us the minimal test combination to reproduce the failure: -```shell -rspec --bisect ee/spec/services/ee/merge_requests/update_service_spec.rb ee/spec/services/ee/notes/quick_actions_service_spec.rb ee/spec/services/epic_links/create_service_spec.rb ee/spec/services/ee/issuable/bulk_update_service_spec.rb -Bisect started using options: "ee/spec/services/ee/merge_requests/update_service_spec.rb ee/spec/services/ee/notes/quick_actions_service_spec.rb ee/spec/services/epic_links/create_service_spec.rb ee/spec/services/ee/issuable/bulk_update_service_spec.rb" -Running suite to find failures... (2 minutes 18.4 seconds) -Starting bisect with 3 failing examples and 144 non-failing examples. -Checking that failure(s) are order-dependent... failure appears to be order-dependent - -Round 1: bisecting over non-failing examples 1-144 . ignoring examples 1-72 (1 minute 11.33 seconds) -... -Round 7: bisecting over non-failing examples 132-133 . ignoring example 132 (43.78 seconds) -Bisect complete! Reduced necessary non-failing examples from 144 to 1 in 8 minutes 31 seconds. - -The minimal reproduction command is: - rspec ./ee/spec/services/ee/issuable/bulk_update_service_spec.rb[1:2:1:1:1:1,1:2:1:2:1:1,1:2:1:3:1] ./ee/spec/services/epic_links/create_service_spec.rb[1:1:2:2:6:4] -``` +1. First obtain the list of specs that ran before the flaky test. You can search + for the list under `Knapsack node specs:` in the CI job output log. +1. Save the list of specs as a file, and run: + + ```shell + cat knapsack_specs.txt | xargs scripts/rspec_bisect_flaky + ``` -We can reproduce the test failure with the reproduction command above. If we change the order of the tests, the test would pass. +If there is an order-dependency issue, the script above will print the minimal +reproduction. ### Time-sensitive flaky tests diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 3096386d7c3..eb35227a50c 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -395,7 +395,7 @@ it('passes', () => { ``` To modify only the hash, use either the `setWindowLocation` helper, or assign -directly to `window.location.hash`, e.g.: +directly to `window.location.hash`, for example: ```javascript it('passes', () => { @@ -423,9 +423,7 @@ it('passes', () => { ### Waiting in tests Sometimes a test needs to wait for something to happen in the application before it continues. -Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/setTimeout) - -because it makes the reason for waiting unclear. Instead use one of the following approaches. +Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/setTimeout) because it makes the reason for waiting unclear. Instead use one of the following approaches. #### Promises and Ajax calls @@ -633,8 +631,8 @@ The latter is useful when you have `setInterval` in the code. **Remember:** our Non-determinism is the breeding ground for flaky and brittle specs. Such specs end up breaking the CI pipeline, interrupting the work flow of other contributors. -1. Make sure your test subject's collaborators (e.g., Axios, apollo, Lodash helpers) and test environment (e.g., Date) behave consistently across systems and over time. -1. Make sure tests are focused and not doing "extra work" (e.g., needlessly creating the test subject more than once in an individual test) +1. Make sure your test subject's collaborators (for example, Axios, Apollo, Lodash helpers) and test environment (for example, Date) behave consistently across systems and over time. +1. Make sure tests are focused and not doing "extra work" (for example, needlessly creating the test subject more than once in an individual test). ### Faking `Date` for determinism @@ -650,7 +648,7 @@ describe('cool/component', () => { // Default fake `Date` const TODAY = new Date(); - // NOTE: `useFakeDate` cannot be called during test execution (i.e. inside `it`, `beforeEach`, `beforeAll`, etc.). + // NOTE: `useFakeDate` cannot be called during test execution (that is, inside `it`, `beforeEach`, `beforeAll`, etc.). describe("on Ada Lovelace's Birthday", () => { useFakeDate(1815, 11, 10) @@ -670,7 +668,7 @@ Similarly, if you really need to use the real `Date` class, then you can import ```javascript import { useRealDate } from 'helpers/fake_date'; -// NOTE: `useRealDate` cannot be called during test execution (i.e. inside `it`, `beforeEach`, `beforeAll`, etc.). +// NOTE: `useRealDate` cannot be called during test execution (that is, inside `it`, `beforeEach`, `beforeAll`, etc.). describe('with real date', () => { useRealDate(); }); @@ -702,30 +700,33 @@ The more challenging part are mocks, which can be used for functions or even dep ### Manual module mocks Manual mocks are used to mock modules across the entire Jest environment. This is a very powerful testing tool that helps simplify -unit testing by mocking out modules which cannot be easily consumed in our test environment. +unit testing by mocking out modules that cannot be easily consumed in our test environment. -> **WARNING:** Do not use manual mocks if a mock should not be consistently applied in every spec (i.e. it's only needed by a few specs). +> **WARNING:** Do not use manual mocks if a mock should not be consistently applied in every spec (that is, it's only needed by a few specs). > Instead, consider using [`jest.mock(..)`](https://jestjs.io/docs/jest-object#jestmockmodulename-factory-options) > (or a similar mocking function) in the relevant spec file. #### Where should I put manual mocks? Jest supports [manual module mocks](https://jestjs.io/docs/manual-mocks) by placing a mock in a `__mocks__/` directory next to the source module -(e.g. `app/assets/javascripts/ide/__mocks__`). **Don't do this.** We want to keep all of our test-related code in one place (the `spec/` folder). +(for example, `app/assets/javascripts/ide/__mocks__`). **Don't do this.** We want to keep all of our test-related code in one place (the `spec/` folder). If a manual mock is needed for a `node_modules` package, use the `spec/frontend/__mocks__` folder. Here's an example of a [Jest mock for the package `monaco-editor`](https://gitlab.com/gitlab-org/gitlab/-/blob/b7f914cddec9fc5971238cdf12766e79fa1629d7/spec/frontend/__mocks__/monaco-editor/index.js#L1). -If a manual mock is needed for a CE module, place it in `spec/frontend/mocks/ce`. +If a manual mock is needed for a CE module, place the implementation in +`spec/frontend/__helpers__/mocks` and add a line to the `frontend/test_setup` +(or the `frontend/shared_test_setup`) that looks something like: -- Files in `spec/frontend/mocks/ce` mocks the corresponding CE module from `app/assets/javascripts`, mirroring the source module's path. - - Example: `spec/frontend/mocks/ce/lib/utils/axios_utils` mocks the module `~/lib/utils/axios_utils`. -- We don't support mocking EE modules yet. -- If a mock is found for which a source module doesn't exist, the test suite fails. 'Virtual' mocks, or mocks that don't have a 1-to-1 association with a source module, are not supported yet. +```javascript +// "~/lib/utils/axios_utils" is the path to the real module +// "helpers/mocks/axios_utils" is the path to the mocked implementation +jest.mock('~/lib/utils/axios_utils', () => jest.requireActual('helpers/mocks/axios_utils')); +``` #### Manual mock examples -- [`mocks/axios_utils`](https://gitlab.com/gitlab-org/gitlab/-/blob/bd20aeb64c4eed117831556c54b40ff4aee9bfd1/spec/frontend/mocks/ce/lib/utils/axios_utils.js#L1) - +- [`__helpers__/mocks/axios_utils`](https://gitlab.com/gitlab-org/gitlab/-/blob/a50edd12b3b1531389624086b6381a042c8143ef/spec/frontend/__helpers__/mocks/axios_utils.js#L1) - This mock is helpful because we don't want any unmocked requests to pass any tests. Also, we are able to inject some test helpers such as `axios.waitForAll`. - [`__mocks__/mousetrap/index.js`](https://gitlab.com/gitlab-org/gitlab/-/blob/cd4c086d894226445be9d18294a060ba46572435/spec/frontend/__mocks__/mousetrap/index.js#L1) - This mock is helpful because the module itself uses AMD format which webpack understands, but is incompatible with the jest environment. This mock doesn't remove @@ -897,7 +898,7 @@ it.each([ NOTE: Only use template literal block if pretty print is not needed for spec output. For example, empty strings, nested objects etc. -For example, when testing the difference between an empty search string and a non-empty search string, the use of the array block syntax with the pretty print option would be preferred. That way the differences between an empty string e.g. `''` and a non-empty string e.g. `'search string'` would be visible in the spec output. Whereas with a template literal block, the empty string would be shown as a space, which could lead to a confusing developer experience +For example, when testing the difference between an empty search string and a non-empty search string, the use of the array block syntax with the pretty print option would be preferred. That way the differences between an empty string (`''`) and a non-empty string (`'search string'`) would be visible in the spec output. Whereas with a template literal block, the empty string would be shown as a space, which could lead to a confusing developer experience. ```javascript // bad @@ -1190,7 +1191,7 @@ it('renders the component correctly', () => { }) ``` -The above test will create two snapshots, what's important is to decide which of the snapshots provide more value for the codebase safety i.e. if one of these snapshots changes, does that highlight a possible un-wanted break in the codebase? This can help catch unexpected changes if something in an underlying dependency changes without our knowledge. +The above test will create two snapshots. It's important to decide which of the snapshots provide more value for codebase safety. That is, if one of these snapshots changes, does that highlight a possible break in the codebase? This can help catch unexpected changes if something in an underlying dependency changes without our knowledge. ### Pros and Cons diff --git a/doc/development/testing_guide/img/review-app-parent-pipeline.png b/doc/development/testing_guide/img/review-app-parent-pipeline.png Binary files differindex 5686d5f6ebe..dd64d7d1be5 100644 --- a/doc/development/testing_guide/img/review-app-parent-pipeline.png +++ b/doc/development/testing_guide/img/review-app-parent-pipeline.png diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 3190ab6c899..31a807697c5 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -36,6 +36,12 @@ On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in browser performance testing using a [Sitespeed.io Container](../../user/project/merge_requests/browser_performance_testing.md). +## Sample Data for Review Apps + +Upon deployment of a review app, project data is created from the [`sample-gitlab-project`](https://gitlab.com/gitlab-org/sample-data-templates/sample-gitlab-project) template project. This aims to provide projects with prepopulated resources to facilitate manual and exploratory testing. + +The sample projects will be created in the `root` user namespace and can be accessed from the personal projects list for that user. + ## How to ### Redeploy Review App from a clean slate @@ -81,26 +87,22 @@ the GitLab handbook information for the [shared 1Password account](https://about ### Run a Rails console 1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.exec` permission first. -1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), - e.g. `review-qa-raise-e-12chm0`. -1. Find and open the `task-runner` Deployment, e.g. `review-qa-raise-e-12chm0-task-runner`. -1. Click on the Pod in the "Managed pods" section, e.g. `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz`. -1. Click on the `KUBECTL` dropdown, then `Exec` -> `task-runner`. -1. Replace `-c task-runner -- ls` with `-it -- gitlab-rails console` from the +1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`. +1. Find and open the `toolbox` Deployment. For example, `review-qa-raise-e-12chm0-toolbox`. +1. Click on the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz`. +1. Click on the `KUBECTL` dropdown, then `Exec` -> `toolbox`. +1. Replace `-c toolbox -- ls` with `-it -- gitlab-rails console` from the default command or - - Run `kubectl exec --namespace review-qa-raise-e-12chm0 review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz -it -- gitlab-rails console` and - - Replace `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz` + - Run `kubectl exec --namespace review-qa-raise-e-12chm0 review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz -it -- gitlab-rails console` and + - Replace `review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz` with your Pod's name. ### Dig into a Pod's logs 1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.getLogs` permission first. -1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), - e.g. `review-qa-raise-e-12chm0`. -1. Find and open the `migrations` Deployment, e.g. - `review-qa-raise-e-12chm0-migrations.1`. -1. Click on the Pod in the "Managed pods" section, e.g. - `review-qa-raise-e-12chm0-migrations.1-nqwtx`. +1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`. +1. Find and open the `migrations` Deployment. For example, `review-qa-raise-e-12chm0-migrations.1`. +1. Click on the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-migrations.1-nqwtx`. 1. Click on the `Container logs` link. Alternatively, you could use the [Logs Explorer](https://console.cloud.google.com/logs/query;query=?project=gitlab-review-apps) which provides more utility to search logs. An example query for a pod name is as follows: @@ -161,7 +163,7 @@ subgraph "CNG-mirror pipeline" - The `review-build-cng` job automatically starts only if your MR includes [CI or frontend changes](../pipelines.md#changes-patterns). In other cases, the job is manual. - The [`CNG-mirror`](https://gitlab.com/gitlab-org/build/CNG-mirror/pipelines/44364657) pipeline creates the Docker images of - each component (e.g. `gitlab-rails-ee`, `gitlab-shell`, `gitaly` etc.) + each component (for example, `gitlab-rails-ee`, `gitlab-shell`, `gitaly` etc.) based on the commit from the [GitLab pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) and stores them in its [registry](https://gitlab.com/gitlab-org/build/CNG-mirror/container_registry). - We use the [`CNG-mirror`](https://gitlab.com/gitlab-org/build/CNG-mirror) project so that the `CNG`, (Cloud @@ -192,7 +194,7 @@ subgraph "CNG-mirror pipeline" - If the `review-deploy` job keeps failing (and a manual retry didn't help), please post a message in the `#g_qe_engineering_productivity` channel and/or create a `~"Engineering Productivity"` `~"ep::review apps"` `~"type::bug"` issue with a link to your merge request. Note that the deployment failure can - reveal an actual problem introduced in your merge request (i.e. this isn't + reveal an actual problem introduced in your merge request (that is, this isn't necessarily a transient failure)! - If the `review-qa-smoke` job keeps failing (note that we already retry it twice), please check the job's logs: you could discover an actual problem introduced in @@ -231,7 +233,7 @@ that were not removed along with the Kubernetes resources. The cluster is configured via Terraform in the [`engineering-productivity-infrastructure`](https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure) project. Node pool image type must be `Container-Optimized OS (cos)`, not `Container-Optimized OS with Containerd (cos_containerd)`, -due to this [known issue on GitLab Runner Kubernetes executor](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4755) +due to this [known issue on the Kubernetes executor for GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4755) ### Helm @@ -251,7 +253,7 @@ aids in identifying load spikes on the cluster, and if nodes are problematic or ### Database related errors in `review-deploy` or `review-qa-smoke` Occasionally the state of a Review App's database could diverge from the database schema. This could be caused by -changes to migration files or schema, such as a migration being renamed or deleted. This typically manifest in migration errors such as: +changes to migration files or schema, such as a migration being renamed or deleted. This typically manifests in migration errors such as: - migration job failing with a column that already exists - migration job failing with a column that does not exist @@ -283,7 +285,7 @@ If the Docker image does not exist: - Verify the `image.repository` and `image.tag` options in the `helm upgrade --install` command match the repository names used by CNG-mirror pipeline. - Look further in the corresponding downstream CNG-mirror pipeline in `review-build-cng` job. -### Node count is always increasing (i.e. never stabilizing or decreasing) +### Node count is always increasing (never stabilizing or decreasing) **Potential cause:** @@ -370,10 +372,10 @@ effectively preventing all the Review Apps from getting a DNS record assigned, making them unreachable via domain name. This in turn prevented other components of the Review App to properly start -(e.g. `gitlab-runner`). +(for example, `gitlab-runner`). -After some digging, we found that new mounts were failing, when being performed -with transient scopes (e.g. pods) of `systemd-mount`: +After some digging, we found that new mounts fail when performed +with transient scopes (for example, pods) of `systemd-mount`: ```plaintext MountVolume.SetUp failed for volume "dns-gitlab-review-app-external-dns-token-sj5jm" : mount failed: exit status 1 @@ -399,8 +401,8 @@ For the record, the debugging steps to find out this issue were: instances** then click the "SSH" button for the node where the `dns-gitlab-review-app-external-dns` pod runs) 1. In the node: `systemctl --version` => `systemd 232` 1. Gather some more information: - - `mount | grep kube | wc -l` => e.g. 290 - - `systemctl list-units --all | grep -i var-lib-kube | wc -l` => e.g. 142 + - `mount | grep kube | wc -l` (returns a count, for example, 290) + - `systemctl list-units --all | grep -i var-lib-kube | wc -l` (returns a count, for example, 142) 1. Check how many pods are in a bad state: - Get all pods running a given node: `kubectl get pods --field-selector=spec.nodeName=NODE_NAME` - Get all the `Running` pods on a given node: `kubectl get pods --field-selector=spec.nodeName=NODE_NAME | grep Running` diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 29cdfab713e..9ca2d0db93c 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -223,14 +223,14 @@ graph RL Formal definition: <https://en.wikipedia.org/wiki/Integration_testing> These kind of tests ensure that individual parts of the application work well -together, without the overhead of the actual app environment (i.e. the browser). +together, without the overhead of the actual app environment (such as the browser). These tests should assert at the request/response level: status code, headers, body. -They're useful to test permissions, redirections, what view is rendered etc. +They're useful, for example, to test permissions, redirections, API endpoints, what view is rendered, and so forth. | Code path | Tests path | Testing engine | Notes | | --------- | ---------- | -------------- | ----- | -| `app/controllers/` | `spec/requests/`, `spec/controllers` | RSpec | Request specs are preferred over legacy controller specs. | +| `app/controllers/` | `spec/requests/`, `spec/controllers` | RSpec | Request specs are preferred over legacy controller specs. Request specs are encouraged for API endpoints. | | `app/mailers/` | `spec/mailers/` | RSpec | | | `lib/api/` | `spec/requests/api/` | RSpec | | | `app/assets/javascripts/` | `spec/frontend/` | Jest | [More details below](#frontend-integration-tests) | @@ -502,9 +502,7 @@ These tests run against the UI and ensure that basic functionality is working. ### GitLab QA orchestrator -[GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) is a tool that allows to test that all these pieces -integrate well together by building a Docker image for a given version of GitLab -Rails and running end-to-end tests (i.e. using Capybara) against it. +[GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) is a tool that allows you to test that all these pieces integrate well together by building a Docker image for a given version of GitLab Rails and running end-to-end tests (using Capybara) against it. Learn more in the [GitLab QA orchestrator README](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). diff --git a/doc/development/usage_ping/dictionary.md b/doc/development/usage_ping/dictionary.md deleted file mode 100644 index 810c789bc03..00000000000 --- a/doc/development/usage_ping/dictionary.md +++ /dev/null @@ -1,4 +0,0 @@ ---- -redirect_to: 'https://metrics.gitlab.com/index.html' -remove_date: '2021-11-10' ---- diff --git a/doc/development/work_items_widgets.md b/doc/development/work_items_widgets.md new file mode 100644 index 00000000000..92919c10a9f --- /dev/null +++ b/doc/development/work_items_widgets.md @@ -0,0 +1,114 @@ +--- +stage: Plan +group: Project Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- +# Work items widgets + +## Frontend architecture + +Widgets for work items are heavily inspired by [Frontend widgets](fe_guide/widgets.md). +You can expect some differences, because work items are architecturally different from issuables. + +GraphQL (Vue Apollo) constitutes the core of work items widgets' stack. + +### Retrieve widget information for work items + +To display a work item page, the frontend must know which widgets are available +on the work item it is attempting to display. To do so, it needs to fetch the +list of widgets, using a query like this: + +```plaintext +query WorkItem($workItemId: ID!) { + workItem(workItemId: $id) @client { + id + type + widgets { + nodes { + type + } + } + } +} +``` + +### GraphQL queries and mutations + +GraphQL queries and mutations are work item agnostic. Work item queries and mutations +should happen at the widget level, so widgets are standalone reusable components. +The work item query and mutation should support any work item type and be dynamic. +They should allow you to query and mutate any work item attribute by specifying a widget identifier. + +In this query example, the description widget uses the query and mutation to +display and update the description of any work item: + +```plaintext +query { + workItem(input: { + workItemId: "gid://gitlab/AnyWorkItem/2207", + widgetIdentifier: "description", + }) { + id + type + widgets { + nodes { + ... on DescriptionWidget { + contentText + } + } + } + } +} + +``` + +Mutation example: + +```plaintext +mutation { + updateWorkItem(input: { + workItemId: "gid://gitlab/AnyWorkItem/2207", + widgetIdentifier: "description", + value: "the updated description" + }) { + workItem { + id + description + } + } +} + +``` + +### Widget's responsibility and structure + +A widget is responsible for displaying and updating a single attribute, such as +title, description, or labels. Widgets must support any type of work item. +To maximize component reusability, widgets should be field wrappers owning the +work item query and mutation of the attribute it's responsible for. + +A field component is a generic and simple component. It has no knowledge of the +attribute or work item details, such as input field, date selector, or dropdown. + +Widgets must be configurable to support various use cases, depending on work items. +When building widgets, use slots to provide extra context while minimizing +the use of props and injected attributes. + +### Examples + +We have a [dropdown field component](https://gitlab.com/gitlab-org/gitlab/-/blob/eea9ad536fa2d28ee6c09ed7d9207f803142eed7/app/assets/javascripts/vue_shared/components/dropdown/dropdown_widget/dropdown_widget.vue) +for use as reference. + +Any work item widget can wrap the dropdown component. The widget has knowledge of +the attribute it mutates, and owns the mutation for it. Multiple widgets can use +the same field component. For example: + +- Title and description widgets use the input field component. +- Start and end date use the date selector component. +- Labels, milestones, and assignees selectors use the dropdown component. + +Some frontend widgets already use the dropdown component. Use them as a reference +for work items widgets development: + +- `ee/app/assets/javascripts/boards/components/assignee_select.vue` +- `ee/app/assets/javascripts/boards/components/milestone_select.vue` diff --git a/doc/gitlab-basics/index.md b/doc/gitlab-basics/index.md index d2d0c4fad39..5ba5366eafa 100644 --- a/doc/gitlab-basics/index.md +++ b/doc/gitlab-basics/index.md @@ -27,9 +27,9 @@ The following are guides to basic GitLab functionality: projects together. - [Create a branch](create-branch.md), to make changes to files stored in a project's repository. - [Feature branch workflow](feature_branch_workflow.md). -- [Fork a project](../user/project/working_with_projects.md#fork-a-project), to duplicate projects so they can be worked on in parallel. +- [Fork a project](../user/project/repository/forking_workflow.md#creating-a-fork), to duplicate projects so they can be worked on in parallel. - [Add a file](add-file.md), to add new files to a project's repository. -- [Create an issue](../user/project/issues/managing_issues.md#create-a-new-issue), +- [Create an issue](../user/project/issues/managing_issues.md#create-an-issue), to start collaborating within a project. - [Create a merge request](../user/project/merge_requests/creating_merge_requests.md), to request changes made in a branch be merged into a project's repository. diff --git a/doc/install/aws/gitlab_hybrid_on_aws.md b/doc/install/aws/gitlab_hybrid_on_aws.md index 3c035b33be8..dbd23ff2b30 100644 --- a/doc/install/aws/gitlab_hybrid_on_aws.md +++ b/doc/install/aws/gitlab_hybrid_on_aws.md @@ -14,13 +14,13 @@ Amazon provides a managed Kubernetes service offering known as [Amazon Elastic K ## Tested AWS Bill of Materials by reference architecture size -| GitLab Cloud Native Hybrid Ref Arch | GitLab Baseline Perf Test Results Omnibus on Instances | AWS Bill of Materials (BOM) for CNH | AWS Build Performance Testing Results for [CNH](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) | CNH Cost Estimate 3 AZs* | +| GitLab Cloud Native Hybrid Ref Arch | GitLab Baseline Perf Test Results Omnibus on Instances | AWS Bill of Materials (BOM) for CNH | AWS Build Performance Testing Results for [CNH](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) | CNH Cost Estimate 3 AZs* | | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | | [2K Omnibus](../../administration/reference_architectures/2k_users.md) | [2K Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k) | [2K Cloud Native Hybrid on EKS](#2k-cloud-native-hybrid-on-eks) | GPT Test Results | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=544bcf1162beae6b8130ad257d081cdf9d4504e3)<br />(2 AZ Cost Estimate is in BOM Below) | -| [3K](../../administration/reference_architectures/3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [3k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k) | [3K Cloud Native Hybrid on EKS](#3k-cloud-native-hybrid-on-eks) | [3K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt)<br /><br />[3K Elastic Auto Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200_results.txt) | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=f1294fec554e21be999711cddcdab9c5e7f83f14)<br />(2 AZ Cost Estimate is in BOM Below) | -| [5K](../../administration/reference_architectures/5k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [5k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k) | [5K Cloud Native Hybrid on EKS](#5k-cloud-native-hybrid-on-eks) | [5K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt)<br /><br />[5K AutoScale from 25% GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717_results.txt) | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=330ee43c5b14662db5df6e52b34898d181a09e16) | -| [10K](../../administration/reference_architectures/10k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [10k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k) | [10K Cloud Native Hybrid on EKS](#10k-cloud-native-hybrid-on-eks) | [10K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt)<br /><br />[10K Elastic Auto Scale GPT Test Results](hhttps://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) | [10K 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=5ac2e07a22e01c36ee76b5477c5a046cd1bea792) | -| [50K](../../administration/reference_architectures/50k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [50k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k) | [50K Cloud Native Hybrid on EKS](#50k-cloud-native-hybrid-on-eks) | [50K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt)<br /><br />[10K Elastic Auto Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) | [50K 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=b9c9d6ac1d4a7848011d2050cef3120931fb7c22) | +| [3K](../../administration/reference_architectures/3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [3k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k) | [3K Cloud Native Hybrid on EKS](#3k-cloud-native-hybrid-on-eks) | [3K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt)<br /><br />[3K Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200_results.txt) | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=f1294fec554e21be999711cddcdab9c5e7f83f14)<br />(2 AZ Cost Estimate is in BOM Below) | +| [5K](../../administration/reference_architectures/5k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [5k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k) | [5K Cloud Native Hybrid on EKS](#5k-cloud-native-hybrid-on-eks) | [5K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt)<br /><br />[5K AutoScale from 25% GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717_results.txt) | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=330ee43c5b14662db5df6e52b34898d181a09e16) | +| [10K](../../administration/reference_architectures/10k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [10k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k) | [10K Cloud Native Hybrid on EKS](#10k-cloud-native-hybrid-on-eks) | [10K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt)<br /><br />[10K Elastic Auto Scale GPT Test Results](hhttps://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) | [10K 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=5ac2e07a22e01c36ee76b5477c5a046cd1bea792) | +| [50K](../../administration/reference_architectures/50k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [50k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k) | [50K Cloud Native Hybrid on EKS](#50k-cloud-native-hybrid-on-eks) | [50K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt)<br /><br />[10K Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) | [50K 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=b9c9d6ac1d4a7848011d2050cef3120931fb7c22) | \*Cost calculations for actual implementations are a rough guideline with the following considerations: @@ -41,18 +41,20 @@ The developer preview deploys Aurora PostgreSQL, but the release version will de The [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/tree/master) is an effort made by GitLab to create a multi-cloud, multi-GitLab (Omnibus + Cloud Native Hybrid) toolkit to provision GitLab. GET is developed by GitLab developers and is open to community contributions. It is helpful to review the [GitLab Environment Toolkit (GET) Issues](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/issues) to understand if any of them may affect your provisioning plans. -| | [AWS Quick Start for GitLab Cloud Native Hybrid on EKS](https://aws-quickstart.github.io/quickstart-eks-gitlab/) | [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/tree/master) | +| | [AWS Quick Start for GitLab Cloud Native Hybrid on EKS](https://aws-quickstart.github.io/quickstart-eks-gitlab/) | [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit) | | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | +| Licensing | [Open Source (Apache 2.0)](https://github.com/aws-quickstart/quickstart-eks-gitlab/blob/main/LICENSE.txt) | [GitLab Enterprise Edition license](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/blob/main/LICENSE) ([GitLab Premium tier](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/blob/main/README.md)) | +| Overview and Vision | [AWS Quick Start](https://aws.amazon.com/quickstart/) | [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/blob/main/README.md) | | GitLab Reference Architecture Compliant | Yes | Yes | | GitLab Performance Tool (GPT) Tested | Yes | Yes | | Amazon Well Architected Compliant | Yes<br />(via Quick Start program) | Critical portions <br />reviewed by AWS | | Target Cloud Platforms | AWS | AWS, Google, Azure | | IaC Languages | CloudFormation (Quick Starts) | Terraform, Ansible | -| Community Contributions and Participation (EcoSystem) | <u>GitLab QSG</u>: Getting Started<br /><u>For QSG Dependencies (e.g. EKS)</u>: Substantial | Getting Started | -| Compatible with AWS Meta-Automation Services (via CloudFormation) | Service Catalog (Direct Import), Control Tower<br />Quick Starts, SaaS Factory | No | +| Community Contributions and Participation (EcoSystem) | <u>GitLab QSG</u>: Getting Started<br /><u>For QSG Dependencies (for example, EKS)</u>: Substantial | Getting Started | +| Compatible with AWS Meta-Automation Services (via CloudFormation) | - [AWS Service Catalog](https://aws.amazon.com/servicecatalog/) (Direct Import)<br>- [ServiceNow via an AWS Service Catalog Connector](https://docs.aws.amazon.com/servicecatalog/latest/adminguide/integrations-servicenow.html#integrations-servicenow)<br>- [Jira Service Manager via an AWS Service Catalog Connector](https://docs.aws.amazon.com/servicecatalog/latest/adminguide/integrations-jiraservicedesk.html#integrations-jiraservicedesk)<br>- [AWS Control Tower](https://docs.aws.amazon.com/controltower/) ([Integration](https://aws.amazon.com/blogs/infrastructure-and-automation/deploy-aws-quick-start-to-multiple-accounts-using-aws-control-tower/))<br>- Quick Starts<br>- [AWS SaaS Factory](https://aws.amazon.com/partners/programs/saas-factory/) | No | | Results in a Ready-to-Use instance | Yes | Manual Actions or <br />Supplemental IaC Required | | **<u>Configuration Features</u>** | | | -| Can deploy Omnibus GitLab (non-Kubernetes | No | Yes | +| Can deploy Omnibus GitLab (non-Kubernetes) | No | Yes | | Results in a self-healing Gitaly Cluster configuration | Yes | No | | Complete Internal Encryption | 85%, Targeting 100% | Manual | | AWS GovCloud Support | Yes | TBD | @@ -65,7 +67,7 @@ Gitaly Cluster uses a consistency voting system to implement strong consistency ### Streamlined Performance Testing of AWS Quick Start Prepared GitLab Instances -A set of performance testing instructions have been abbreviated for testing a GitLab instance prepared using the AWS Quick Start for GitLab Cloud Native Hybrid on EKS. They assume zero familiarity with GitLab Performance Tool. They can be accessed here: [Performance Testing an Instance Prepared using AWS Quick Start for GitLab Cloud Native Hybrid on EKS](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/Easy-DIY-Perf-Testing.md). +A set of performance testing instructions have been abbreviated for testing a GitLab instance prepared using the AWS Quick Start for GitLab Cloud Native Hybrid on EKS. They assume zero familiarity with GitLab Performance Tool. They can be accessed here: [Performance Testing an Instance Prepared using AWS Quick Start for GitLab Cloud Native Hybrid on EKS](https://gitlab.com/guided-explorations/aws/implementation-patterns/getting-started-gitlab-aws-quick-start/-/wikis/Easy-Performance-Testing-for-AWS-Quick-Start-for-GitLab-CNH). ### AWS GovCloud Support for AWS Quick Start for GitLab CNH on EKS @@ -129,7 +131,7 @@ Some services, such as log aggregation, outbound email are not specified by GitL - TBD **Deploy Now** - Deploy Now links leverage the AWS Quick Start automation and only prepopulate the number of instances and instance types for the Quick Start based on the Bill of Meterials below. You must provide appropriate input for all other parameters by following the guidance in the [Quick Start documentation's Deployment steps](https://aws-quickstart.github.io/quickstart-eks-gitlab/#_deployment_steps) section. + Deploy Now links leverage the AWS Quick Start automation and only pre-populate the number of instances and instance types for the Quick Start based on the Bill of Materials below. You must provide appropriate input for all other parameters by following the guidance in the [Quick Start documentation's Deployment steps](https://aws-quickstart.github.io/quickstart-eks-gitlab/#_deployment_steps) section. - **Deploy Now: AWS Quick Start for 2 AZs** - **Deploy Now: AWS Quick Start for 3 AZs** @@ -174,15 +176,15 @@ If EKS node autoscaling is employed, it is likely that your average loading will **GPT Test Results** -- [3K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt) +- [3K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt) -- [3K AutoScale from 25% GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200_results.txt) +- [3K AutoScale from 25% GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200_results.txt) Elastic Auto Scale GPT Test Results start with an idle scaled cluster and then start the standard GPT test to determine if the EKS Auto Scaler performs well enough to keep up with performance test demands. In general this is substantially harder ramping than the scaling required when the ramping is driven my normal production workloads. **Deploy Now** -Deploy Now links leverage the AWS Quick Start automation and only prepopulate the number of instances and instance types for the Quick Start based on the Bill of Meterials below. You must provide appropriate input for all other parameters by following the guidance in the [Quick Start documentation's Deployment steps](https://aws-quickstart.github.io/quickstart-eks-gitlab/#_deployment_steps) section. +Deploy Now links leverage the AWS Quick Start automation and only pre-populate the number of instances and instance types for the Quick Start based on the Bill of Materials below. You must provide appropriate input for all other parameters by following the guidance in the [Quick Start documentation's Deployment steps](https://aws-quickstart.github.io/quickstart-eks-gitlab/#_deployment_steps) section. - **[Deploy Now: AWS Quick Start for 2 AZs](https://us-east-2.console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/quickcreate?templateUrl=https://aws-quickstart.s3.us-east-1.amazonaws.com/quickstart-eks-gitlab/templates/gitlab-entry-new-vpc.template.yaml&stackName=Gitlab-EKS-3K-Users-2AZs¶m_NumberOfAZs=2¶m_NodeInstanceType=c5.2xlarge¶m_NumberOfNodes=3¶m_MaxNumberOfNodes=3¶m_DBInstanceClass=db.r6g.xlarge¶m_CacheNodes=2¶m_CacheNodeType=cache.m6g.large¶m_GitalyInstanceType=m5.large¶m_NumberOfGitalyReplicas=3¶m_PraefectInstanceType=c5.large¶m_NumberOfPraefectReplicas=3)** - **[Deploy Now: AWS Quick Start for 3 AZs](https://us-east-2.console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/quickcreate?templateUrl=https://aws-quickstart.s3.us-east-1.amazonaws.com/quickstart-eks-gitlab/templates/gitlab-entry-new-vpc.template.yaml&stackName=Gitlab-EKS-3K-Users-3AZs¶m_NumberOfAZs=3¶m_NodeInstanceType=c5.2xlarge¶m_NumberOfNodes=3¶m_MaxNumberOfNodes=3¶m_DBInstanceClass=db.r6g.xlarge¶m_CacheNodes=3¶m_CacheNodeType=cache.m6g.large¶m_GitalyInstanceType=m5.large¶m_NumberOfGitalyReplicas=3¶m_PraefectInstanceType=c5.large¶m_NumberOfPraefectReplicas=3)** @@ -203,7 +205,7 @@ On Demand pricing is used in this table for comparisons, but should not be used | Supporting services such as NGINX, Prometheus, etc | [2 allocations](../../administration/reference_architectures/3k_users.md#cluster-topology) x ([2 vCPU and 7.5 GB](../../administration/reference_architectures/3k_users.md#cluster-topology)) = <br />4 vCPU, 15 GB | | | | **GitLab Ref Arch Raw Total K8s Node Capacity** | 32 vCPU, 56 GB | | | | One Node for Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 16 vCPU, 32GB | | | -| **Grand Total w/ Overheads Full Scale**<br />Minimum hosts = 3 | 48 vCPU, 88 GB | **c5.2xlarge** (8vcpu/16GB) x 5 nodes<br />40 vCPU, 80 GB<br />[Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt) | $1.70/hr | +| **Grand Total w/ Overheads Full Scale**<br />Minimum hosts = 3 | 48 vCPU, 88 GB | **c5.2xlarge** (8vcpu/16GB) x 5 nodes<br />40 vCPU, 80 GB<br />[Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt) | $1.70/hr | | **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 24 vCPU, 48 GB | c5.2xlarge x 4 | $1.36/hr | Other combinations of node type and quantity can be used to meet the Grand Total. Due to the properties of pods, hosts that are overly small may have significant unused capacity. @@ -228,9 +230,9 @@ If EKS node autoscaling is employed, it is likely that your average loading will **GPT Test Results** -- [5K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) +- [5K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) -- [5K AutoScale from 25% GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717_results.txt) +- [5K AutoScale from 25% GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717_results.txt) Elastic Auto Scale GPT Test Results start with an idle scaled cluster and then start the standard GPT test to determine if the EKS Auto Scaler performs well enough to keep up with performance test demands. In general this is substantially harder ramping than the scaling required when the ramping is driven my normal production workloads. @@ -257,7 +259,7 @@ On Demand pricing is used in this table for comparisons, but should not be used | Supporting services such as NGINX, Prometheus, etc | [2 allocations](../../administration/reference_architectures/5k_users.md#cluster-topology) x ([2 vCPU and 7.5 GB](../../administration/reference_architectures/5k_users.md#cluster-topology)) = <br />4 vCPU, 15 GB | | | | **GitLab Ref Arch Raw Total K8s Node Capacity** | 62 vCPU, 96.5 GB | | | | One Node for Quick Start Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 8 vCPU, 16GB | | | -| **Grand Total w/ Overheads Full Scale**<br />Minimum hosts = 3 | 70 vCPU, 112.5 GB | **c5.2xlarge** (8vcpu/16GB) x 9 nodes<br />72 vCPU, 144 GB<br />[Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) | $2.38/hr | +| **Grand Total w/ Overheads Full Scale**<br />Minimum hosts = 3 | 70 vCPU, 112.5 GB | **c5.2xlarge** (8vcpu/16GB) x 9 nodes<br />72 vCPU, 144 GB<br />[Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) | $2.38/hr | | **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 24 vCPU, 48 GB | c5.2xlarge x 7 | $1.85/hr | Other combinations of node type and quantity can be used to meet the Grand Total. Due to the cpu and memory requirements of pods, hosts that are overly small may have significant unused capacity. @@ -282,9 +284,9 @@ If EKS node autoscaling is employed, it is likely that your average loading will **GPT Test Results** -- [10K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt) +- [10K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt) -- [10K Elastic Auto Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) +- [10K Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) Elastic Auto Scale GPT Test Results start with an idle scaled cluster and then start the standard GPT test to determine if the EKS Auto Scaler performs well enough to keep up with performance test demands. In general this is substantially harder ramping than the scaling required when the ramping is driven my normal production workloads. @@ -310,8 +312,8 @@ On Demand pricing is used in this table for comparisons, but should not be used | Supporting services such as NGINX, Prometheus, etc | [2 allocations](../../administration/reference_architectures/10k_users.md#cluster-topology) x ([2 vCPU and 7.5 GB](../../administration/reference_architectures/10k_users.md#cluster-topology))<br />4 vCPU, 15 GB | | | | **GitLab Ref Arch Raw Total K8s Node Capacity** | 128 vCPU, 158 GB | | | | One Node for Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 16 vCPU, 32GB | | | -| **Grand Total w/ Overheads Fully Scaled**<br />Minimum hosts = 3 | 142 vCPU, 190 GB | **c5.4xlarge** (16vcpu/32GB) x 9 nodes<br />144 vCPU, 288GB<br /><br />[Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt) | $6.12/hr | -| **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 40 vCPU, 80 GB | c5.4xlarge x 7<br /><br />[Elastic Auto Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) | $4.76/hr | +| **Grand Total w/ Overheads Fully Scaled**<br />Minimum hosts = 3 | 142 vCPU, 190 GB | **c5.4xlarge** (16vcpu/32GB) x 9 nodes<br />144 vCPU, 288GB<br /><br />[Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt) | $6.12/hr | +| **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 40 vCPU, 80 GB | c5.4xlarge x 7<br /><br />[Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) | $4.76/hr | Other combinations of node type and quantity can be used to meet the Grand Total. Due to the cpu and memory requirements of pods, hosts that are overly small may have significant unused capacity. @@ -335,9 +337,9 @@ If EKS node autoscaling is employed, it is likely that your average loading will **GPT Test Results** -- [50K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt) +- [50K Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt) -- [50K Elastic Auto Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) +- [50K Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) Elastic Auto Scale GPT Test Results start with an idle scaled cluster and then start the standard GPT test to determine if the EKS Auto Scaler performs well enough to keep up with performance test demands. In general this is substantially harder ramping than the scaling required when the ramping is driven my normal production workloads. @@ -363,8 +365,8 @@ On Demand pricing is used in this table for comparisons, but should not be used | Supporting services such as NGINX, Prometheus, etc | [2 allocations](../../administration/reference_architectures/10k_users.md#cluster-topology) x ([2 vCPU and 7.5 GB](../../administration/reference_architectures/10k_users.md#cluster-topology))<br />4 vCPU, 15 GB | | | | **GitLab Ref Arch Raw Total K8s Node Capacity** | 428 vCPU, 533 GB | | | | One Node for Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 16 vCPU, 32GB | | | -| **Grand Total w/ Overheads Fully Scaled**<br />Minimum hosts = 3 | 444 vCPU, 565 GB | **c5.4xlarge** (16vcpu/32GB) x 28 nodes<br />448 vCPU, 896GB<br /><br />[Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt) | $19.04/hr | -| **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 40 vCPU, 80 GB | c5.2xlarge x 10<br /><br />[Elastic Auto Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) | $6.80/hr | +| **Grand Total w/ Overheads Fully Scaled**<br />Minimum hosts = 3 | 444 vCPU, 565 GB | **c5.4xlarge** (16vcpu/32GB) x 28 nodes<br />448 vCPU, 896GB<br /><br />[Full Fixed Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt) | $19.04/hr | +| **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 40 vCPU, 80 GB | c5.2xlarge x 10<br /><br />[Elastic Auto Scale GPT Test Results](https://gitlab.com/guided-explorations/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) | $6.80/hr | Other combinations of node type and quantity can be used to meet the Grand Total. Due to the cpu and memory requirements of pods, hosts that are overly small may have significant unused capacity. diff --git a/doc/install/aws/gitlab_sre_for_aws.md b/doc/install/aws/gitlab_sre_for_aws.md index faec73801ab..2114ed51128 100644 --- a/doc/install/aws/gitlab_sre_for_aws.md +++ b/doc/install/aws/gitlab_sre_for_aws.md @@ -10,7 +10,7 @@ description: Doing SRE for GitLab instances and runners on AWS. ## Gitaly SRE considerations -Gitaly is an embedded service for Git Repository Storage. Gitaly and Gitaly Cluster have been engineered by GitLab to overcome fundamental challenges with horizontal scaling of the open source Git binaries that must be used on the service side of GitLab. Here is indepth technical reading on the topic: +Gitaly is an embedded service for Git Repository Storage. Gitaly and Gitaly Cluster have been engineered by GitLab to overcome fundamental challenges with horizontal scaling of the open source Git binaries that must be used on the service side of GitLab. Here is in-depth technical reading on the topic: ### Why Gitaly was built @@ -66,14 +66,14 @@ All recommendations are for production configurations, including performance tes #### Network I/O recommendations -- Use only instance types [from the list of ones that support ENA advanced networking]( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#instance-type-summary-table) to ensure that cluster replication latency is not due to instance level network I/O bottlenecking. +- Use only instance types [from the list of ones that support ENA advanced networking]( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#instance-type-summary-table) to ensure that cluster replication latency is not due to instance level network I/O bottlenecks. - Choose instances with sizes with more than 10 Gbps - but only if needed and only when having proven a node level network bottleneck with monitoring and/or stress testing. **To accommodate:** - Gitaly nodes do the main work of streaming repositories for push and pull operations (to add development endpoints, and to CI/CD). - Gitaly servers need reasonable low latency between cluster nodes and with Praefect services in order for the cluster to maintain operational and data integrity. -- Gitaly nodes should be selected with network bottlenecking avoidance as a primary consideration. +- Gitaly nodes should be selected with network bottleneck avoidance as a primary consideration. - Gitaly nodes should be monitored for network saturation. - Not all networking issues can be solved through optimizing the node level networking: - Gitaly cluster node replication depends on all networking between nodes. diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index 563673c3260..bec1f3e1142 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -37,12 +37,48 @@ The following are the currently available implementation patterns for GitLab whe The following repository is self-contained in regard to enabling this pattern: [GitLab HA Scaling Runner Vending Machine for AWS EC2 ASG](https://gitlab.com/guided-explorations/aws/gitlab-runner-autoscaling-aws-asg/). The [feature list for this implementation pattern](https://gitlab.com/guided-explorations/aws/gitlab-runner-autoscaling-aws-asg/-/blob/main/FEATURES.md) is good to review to understand the complete value it can deliver. +### Patterns for Using GitLab with AWS + +[The Guided Explorations' subgroup for AWS](https://gitlab.com/guided-explorations/aws) contains a variety of working example projects for: + +- Using GitLab and AWS together. +- Running GitLab infrastructure on AWS. + ## AWS known issues list Known issues are gathered from within GitLab and from customer reported issues. Customers successfully implement GitLab with a variety of "as a Service" components that GitLab has not specifically been designed for, nor has ongoing testing for. While GitLab does take partner technologies very seriously, the highlighting of known issues here is a convenience for implementers and it does not imply that GitLab has targeted compatibility with, nor carries any type of guarantee of running on the partner technology where the issues occur. Please consult individual issues to understand GitLabs stance and plans on any given known issue. See the [GitLab AWS known issues list](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues?label_name%5B%5D=AWS+Known+Issue) for a complete list. +## Official GitLab releases as AMIs + +GitLab produces AMI images during the regular release process. The AMIs can be used for single instance GitLab installation or, by configuring `/etc/gitlab/gitlab.rb`, can be specialized for specific GitLab service roles (for example a Gitaly server). Older releases remain available and can be used to migrate an older GitLab server to AWS. + +Currently the Amazon AMI uses the Amazon prepared Ubuntu AMI (x86 and ARM are available) as its starting point. + +NOTE: +When deploying a GitLab instance using the official AMI, the root password to the instance will be the EC2 **Instance** ID (NOT the AMI ID). This way of setting the root account password is specific to official GitLab published AMIs ONLY. + +Instances running on Community Edition (CE) require a migration to Enterprise Edition (EE) in order to subscribe to the GitLab Premium or Ultimate plan. If you want to pursue a subscription, using the Free-forever plan of Enterprise Edition is the least disruptive method. + +NOTE: +Since any given GitLab upgrade might involve data disk updates or database schema upgrades, simply swapping out the AMI is not sufficent for taking upgrades. + +1. Log in to the AWS Web Console, so that clicking the links in the following step take you directly to the AMI list. +1. Pick the edition you want: + + - [GitLab Enterprise Edition](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=782774275127;search=GitLab%20EE;sort=desc:name): If you want to unlock the enterprise features, a license is needed. + - [GitLab Community Edition](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=782774275127;search=GitLab%20CE;sort=desc:name): The open source version of GitLab. + - [GitLab Premium or Ultimate Marketplace (Prelicensed)](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;source=Marketplace;search=GitLab%20EE;sort=desc:name): 5 user license built into per-minute billing. + +1. AMI IDs are unique per region, so once you've loaded one of the above, select the desired target region in the upper right of the console to see the appropriate AMIs. +1. After the console is loaded, you can add additional search criteria to narrow further. For instance, type `13.` to find only 13.x versions. +1. To launch an EC2 Machine with one of the listed AMIs, check the box at the start of the relevant row, and select the "Launch" button near the top of left of the page. + +NOTE: +If you are trying to restore from an older version of GitLab while moving to AWS, find the +[Enterprise and Community Editions before GitLab 11.10.3](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=855262394183;sort=desc:name). + ## Additional details on implementation patterns GitLab implementation patterns build upon [GitLab Reference Architectures](../../administration/reference_architectures/index.md) in the following ways. diff --git a/doc/install/aws/manual_install_aws.md b/doc/install/aws/manual_install_aws.md index 6fff4225346..6dacd8df3e4 100644 --- a/doc/install/aws/manual_install_aws.md +++ b/doc/install/aws/manual_install_aws.md @@ -28,7 +28,7 @@ The Infrastructure as Code tooling [GitLab Environment Tool (GET)](https://gitla ### Getting started for production-grade Cloud Native Hybrid GitLab -For the Cloud Native Hybrid architectures there are two Infrastructure as Code options which are compared in GitLab Cloud Native Hybrid on AWS EKS implementation pattern in the section [Available Infrastructure as Code for GitLab Cloud Native Hybrid](gitlab_hybrid_on_aws.md#available-infrastructure-as-code-for-gitlab-cloud-native-hybrid). It compares the [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/tree/master) to the AWS Quick Start for GitLab Cloud Native Hybrid on EKS which was codeveloped by GitLab and AWS. GET and the AWS Quick Start are both open source so anyone can build on top of them and contribute improvements to them. +For the Cloud Native Hybrid architectures there are two Infrastructure as Code options which are compared in GitLab Cloud Native Hybrid on AWS EKS implementation pattern in the section [Available Infrastructure as Code for GitLab Cloud Native Hybrid](gitlab_hybrid_on_aws.md#available-infrastructure-as-code-for-gitlab-cloud-native-hybrid). It compares the [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/tree/master) to the AWS Quick Start for GitLab Cloud Native Hybrid on EKS which was co-developed by GitLab and AWS. GET and the AWS Quick Start are both open source so anyone can build on top of them and contribute improvements to them. ## Introduction @@ -391,7 +391,7 @@ persistence and is used to store session data, temporary cache information, and chance to deploy Redis in multiple availability zones. 1. In the settings section: 1. Give the cluster a name (`gitlab-redis`) and a description. - 1. For the version, select the latest of `5.0` series (e.g., `5.0.6`). + 1. For the version, select the latest of the `5.0` series (for example, `5.0.6`). 1. Leave the port as `6379` since this is what we used in our Redis security group above. 1. Select the node type (at least `cache.t3.medium`, but adjust to your needs) and the number of replicas. 1. In the advanced settings section: @@ -533,7 +533,7 @@ gitlab=# \q ```ruby # Disable the built-in Postgres postgresql['enable'] = false - + # Fill in the connection details gitlab_rails['db_adapter'] = "postgresql" gitlab_rails['db_encoding'] = "unicode" @@ -549,7 +549,7 @@ gitlab=# \q ```ruby # Disable the built-in Redis redis['enable'] = false - + # Fill in the connection details gitlab_rails['redis_host'] = "<redis-endpoint>" gitlab_rails['redis_port'] = 6379 @@ -658,12 +658,6 @@ Since we are using the [AWS IAM profile](#create-an-iam-role) we created earlier Remember to run `sudo gitlab-ctl reconfigure` after saving the changes to the `gitlab.rb` file. -NOTE: -One current feature of GitLab that still requires a shared directory (NFS) is -[GitLab Pages](../../user/project/pages/index.md). -There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196) -to eliminate the need for NFS to support GitLab Pages. - --- That concludes the configuration changes for our GitLab instance. Next, we'll create a custom AMI based on this instance to use for our launch configuration and auto scaling group. @@ -812,21 +806,7 @@ After a few minutes, the new version should be up and running. ## Find official GitLab-created AMI IDs on AWS -To find the AMIs generated by GitLab: - -1. Login to AWS Web Console, so that clicking the links below will take you directly to the AMI list. -1. Pick the edition you want: - - - [GitLab Enterprise Edition](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=782774275127;search=GitLab%20EE;sort=desc:name): If you want to unlock the enterprise features, a license is needed. Recommended for this guide. - - [GitLab Community Edition](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=782774275127;search=GitLab%20CE;sort=desc:name): The open source version of GitLab. - - [GitLab Premium or Ultimate Marketplace (Prelicensed)](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;source=Marketplace;search=GitLab%20EE;sort=desc:name): 5 user license built into per-minute billing. -1. AMI IDs are unique per region, so once you've loaded one of the above, select the desired target region in the upper right of the console to see the appropriate AMIs. -1. Once the console is loaded, you can add additional search criteria to narrow further. For instance, `13.` to find only 13.x versions. -1. To launch an EC2 Machine with one of the listed AMIs, check the box at the start of the relevant row, and select the "Launch" button near the top of left of the page. - -NOTE: -If you are trying to restore from an older version of GitLab while moving to AWS, find the -[Enterprise and Community Editions Before 11.10.3](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=855262394183;sort=desc:name). +Read more on how to use [GitLab releases as AMIs](index.md#official-gitlab-releases-as-amis). ## Conclusion @@ -856,7 +836,7 @@ to request additional material: ### Instances are failing health checks -If your instances are failing the load balancer's health checks, verify that they are returning a status `200` from the health check endpoint we configured earlier. Any other status, including redirects (e.g. status `302`) will cause the health check to fail. +If your instances are failing the load balancer's health checks, verify that they are returning a status `200` from the health check endpoint we configured earlier. Any other status, including redirects like status `302`, will cause the health check to fail. You may have to set a password on the `root` user to prevent automatic redirects on the sign-in endpoint before health checks will pass. diff --git a/doc/install/digitaloceandocker.md b/doc/install/digitaloceandocker.md index 845fc3f04e8..c0b9b280d92 100644 --- a/doc/install/digitaloceandocker.md +++ b/doc/install/digitaloceandocker.md @@ -1,8 +1,7 @@ --- -stage: none -group: unassigned +stage: Enablement +group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- # Digital Ocean and Docker Machine test environment **(FREE SELF)** diff --git a/doc/install/docker.md b/doc/install/docker.md index 00e19e2977b..2efe6a3640b 100644 --- a/doc/install/docker.md +++ b/doc/install/docker.md @@ -81,6 +81,7 @@ sudo docker run --detach \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ + --shm-size 256m \ gitlab/gitlab-ee:latest ``` @@ -99,6 +100,7 @@ sudo docker run --detach \ --volume $GITLAB_HOME/config:/etc/gitlab:Z \ --volume $GITLAB_HOME/logs:/var/log/gitlab:Z \ --volume $GITLAB_HOME/data:/var/opt/gitlab:Z \ + --shm-size 256m \ gitlab/gitlab-ee:latest ``` @@ -155,6 +157,7 @@ install, and upgrade your Docker-based GitLab installation: - '$GITLAB_HOME/config:/etc/gitlab' - '$GITLAB_HOME/logs:/var/log/gitlab' - '$GITLAB_HOME/data:/var/opt/gitlab' + shm_size: '256m' ``` 1. Make sure you are in the same directory as `docker-compose.yml` and start @@ -188,6 +191,7 @@ web: - '$GITLAB_HOME/config:/etc/gitlab' - '$GITLAB_HOME/logs:/var/log/gitlab' - '$GITLAB_HOME/data:/var/opt/gitlab' + shm_size: '256m' ``` This is the same as using `--publish 8929:8929 --publish 2224:22`. @@ -199,11 +203,11 @@ configure and deploy your Docker-based GitLab installation in a swarm cluster. In swarm mode you can leverage [Docker secrets](https://docs.docker.com/engine/swarm/secrets/) -and [Docker configs](https://docs.docker.com/engine/swarm/configs/) to efficiently and securely deploy your GitLab instance. +and [Docker configurations](https://docs.docker.com/engine/swarm/configs/) to efficiently and securely deploy your GitLab instance. Secrets can be used to securely pass your initial root password without exposing it as an environment variable. -Configs can help you to keep your GitLab image as generic as possible. +Configurations can help you to keep your GitLab image as generic as possible. -Here's an example that deploys GitLab with four runners as a [stack](https://docs.docker.com/get-started/part5/), using secrets and configs: +Here's an example that deploys GitLab with four runners as a [stack](https://docs.docker.com/get-started/part5/), using secrets and configurations: 1. [Set up a Docker swarm](https://docs.docker.com/engine/swarm/swarm-tutorial/). 1. Create a `docker-compose.yml` file: @@ -221,6 +225,7 @@ Here's an example that deploys GitLab with four runners as a [stack](https://doc - $GITLAB_HOME/data:/var/opt/gitlab - $GITLAB_HOME/logs:/var/log/gitlab - $GITLAB_HOME/config:/etc/gitlab + shm_size: '256m' environment: GITLAB_OMNIBUS_CONFIG: "from_file('/omnibus_config.rb')" configs: @@ -325,6 +330,7 @@ sudo docker run --detach \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ + --shm-size 256m \ gitlab/gitlab-ee:latest ``` @@ -361,6 +367,7 @@ sudo docker run --detach \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ + --shm-size 256m \ gitlab/gitlab-ee:latest ``` @@ -388,6 +395,7 @@ port `2289`: --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ + --shm-size 256m \ gitlab/gitlab-ee:latest ``` @@ -477,6 +485,7 @@ To update GitLab that was [installed using Docker Engine](#install-gitlab-using- --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ + --shm-size 256m \ gitlab/gitlab-ee:latest ``` @@ -643,4 +652,4 @@ purpose. ### Docker containers exhausts space due to the `json-file` -Docker's [default logging driver is `json-file`](https://docs.docker.com/config/containers/logging/configure/#configure-the-default-logging-driver), which performs no log rotation by default. As a result of this lack of rotation, log files stored by the `json-file` driver can consume a significant amount of disk space for containers that generate a lot of output. This can lead to disk space exhaustion. To address this, use [journald](https://docs.docker.com/config/containers/logging/journald/) as the logging driver when available, or [another supported driver](https://docs.docker.com/config/containers/logging/configure/#supported-logging-drivers) with native rotation support. +Docker's [default logging driver is `json-file`](https://docs.docker.com/config/containers/logging/configure/#configure-the-default-logging-driver), which performs no log rotation by default. As a result of this lack of rotation, log files stored by the `json-file` driver can consume a significant amount of disk space for containers that generate a lot of output. This can lead to disk space exhaustion. To address this, use [`journald`](https://docs.docker.com/config/containers/logging/journald/) as the logging driver when available, or [another supported driver](https://docs.docker.com/config/containers/logging/configure/#supported-logging-drivers) with native rotation support. diff --git a/doc/install/index.md b/doc/install/index.md index df2f230fd27..44b234747dc 100644 --- a/doc/install/index.md +++ b/doc/install/index.md @@ -32,6 +32,7 @@ install GitLab: | [Docker](https://docs.gitlab.com/omnibus/docker/) | The GitLab packages, Dockerized. | Use this method if you're familiar with Docker. | | [Source](installation.md) | Install GitLab and all of its components from scratch. | Use this method if none of the previous methods are available for your platform. Useful for unsupported systems like \*BSD.| | [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit#documentation) | The GitLab Environment toolkit provides a set of automation tools to deploy a [reference architecture](../administration/reference_architectures/index.md) on most major cloud providers. | Customers are very welcome to trial and evaluate GET today, however be aware of [key limitations](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit#missing-features-to-be-aware-of) of the current iteration. For production environments further manual setup will be required based on your specific requirements. | +| [GitLab Operator](https://docs.gitlab.com/charts/installation/operator.html) | The GitLab Operator provides an installation and management method for GitLab following the [Kubernetes Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). | Use the GitLab Operator to run GitLab in an [OpenShift](openshift_and_gitlab/index.md) environment. | ## Install GitLab on cloud providers diff --git a/doc/install/installation.md b/doc/install/installation.md index cd00593a99b..f405bc40f43 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -244,13 +244,13 @@ sudo make install GitLab has several daemons written in Go. To install GitLab we need a Go compiler. The instructions below assume you use 64-bit Linux. You can find downloads for other platforms at the [Go download -page](https://golang.org/dl). +page](https://go.dev/dl). ```shell # Remove former Go installation folder sudo rm -rf /usr/local/go -curl --remote-name --progress-bar "https://golang.org/dl/go1.16.10.linux-amd64.tar.gz" +curl --remote-name --progress-bar "https://go.dev/dl/go1.16.10.linux-amd64.tar.gz" echo '414cd18ce1d193769b9e97d2401ad718755ab47816e13b2a1cde203d263b55cf go1.16.10.linux-amd64.tar.gz' | shasum -a256 -c - && \ sudo tar -C /usr/local -xzf go1.16.10.linux-amd64.tar.gz sudo ln -sf /usr/local/go/bin/{go,gofmt} /usr/local/bin/ @@ -1034,7 +1034,7 @@ To use GitLab with HTTPS: 1. Set the `port` option in section 1 to `443`. 1. Set the `https` option in section 1 to `true`. 1. In the `config.yml` of GitLab Shell: - 1. Set `gitlab_url` option to the HTTPS endpoint of GitLab (e.g. `https://git.example.com`). + 1. Set `gitlab_url` option to the HTTPS endpoint of GitLab (for example, `https://git.example.com`). 1. Set the certificates using either the `ca_file` or `ca_path` option. 1. Use the `gitlab-ssl` NGINX example configuration instead of the `gitlab` configuration. 1. Update `YOUR_SERVER_FQDN`. @@ -1119,7 +1119,7 @@ host localhost # Give your setup a name (here: override localhost) hostname 127.0.0.1; # Your server name or IP ``` -You also need to change the corresponding options (e.g. `ssh_user`, `ssh_host`, `admin_uri`) in the `config\gitlab.yml` file. +You also need to change the corresponding options (for example, `ssh_user`, `ssh_host`, `admin_uri`) in the `config\gitlab.yml` file. ### Additional Markup Styles diff --git a/doc/install/openshift_and_gitlab/index.md b/doc/install/openshift_and_gitlab/index.md index f74e9f26362..e102235c4f0 100644 --- a/doc/install/openshift_and_gitlab/index.md +++ b/doc/install/openshift_and_gitlab/index.md @@ -20,7 +20,7 @@ Some components (documented on the GitLab Operator doc) are not supported yet. ## Deploy to and integrate with OpenShift from GitLab -Deploying custom or COTS applications on top of OpenShift from GitLab is supported using [the GitLab Kubernetes Agent](../../user/clusters/agent/index.md). +Deploying custom or COTS applications on top of OpenShift from GitLab is supported using [the GitLab Agent](../../user/clusters/agent/index.md). ## Use OpenShift to run a GitLab Runner Fleet diff --git a/doc/install/pivotal/index.md b/doc/install/pivotal/index.md index ef6eb378346..ee379a3c95f 100644 --- a/doc/install/pivotal/index.md +++ b/doc/install/pivotal/index.md @@ -1,18 +1,9 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../index.md' +remove_date: '2022-03-08' --- -# GitLab Pivotal Tile **(PREMIUM SELF)** +This document was removed. For information about installing GitLab, see [this page](../index.md). -WARNING: -As of September 13, 2017, the GitLab Enterprise Plus for Pivotal Cloud Foundry -tile on Pivotal Network has reached its End of Availability ("EoA") and is no -longer available for download or sale through Pivotal. Current customers with -active subscriptions continue to receive support from GitLab through their -subscription term. Pivotal and GitLab are collaborating on creating a new -Kubernetes-based tile for the Pivotal Container Service. Please contact GitLab -support with any questions regarding GitLab Enterprise Plus for Pivotal Cloud Foundry. - -Original article: <https://docs.pivotal.io/partners/gitlab/index.html>. +<!-- This redirect file can be deleted after <2022-03-08>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/install/relative_url.md b/doc/install/relative_url.md index 569f6e02ea4..43f2414e8f9 100644 --- a/doc/install/relative_url.md +++ b/doc/install/relative_url.md @@ -60,7 +60,7 @@ assumptions are made: Make sure to follow all steps below: -1. (Optional) If you run short on resources, you can temporarily free up some +1. Optional. If you run short on resources, you can temporarily free up some memory by shutting down the GitLab service with the following command: ```shell diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 5dbe9a1154f..037fbd7063d 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -35,7 +35,7 @@ For the installation options, see [the main installation page](index.md). Installation of GitLab on these operating systems is possible, but not supported. Please see the [installation from source guide](installation.md) and the [installation guides](https://about.gitlab.com/install/) for more information. -Please see [OS versions that are no longer supported](../administration/package_information/deprecated_os.md) for Omnibus installs page +Please see [OS versions that are no longer supported](../administration/package_information/supported_os.md#os-versions-that-are-no-longer-supported) for Omnibus installs page for a list of supported and unsupported OS versions as well as the last support GitLab version for that OS. ### Microsoft Windows @@ -57,7 +57,7 @@ Redis version 6.0 or higher is recommended, as this is what ships with ### Storage -The necessary hard drive space largely depends on the size of the repositories you want to store in GitLab but as a *rule of thumb* you should have at least as much free space as all your repositories combined take up. +The necessary hard drive space largely depends on the size of the repositories you want to store in GitLab but as a *guideline* you should have at least as much free space as all your repositories combined take up. The Omnibus GitLab package requires about 2.5 GB of storage space for installation. @@ -302,7 +302,8 @@ The GitLab Runner server requirements depend on: Since the nature of the jobs varies for each use case, you need to experiment by adjusting the job concurrency to get the optimum setting. -For reference, the GitLab.com Runner Cloud [auto-scaling runner for Linux](../ci/runners/build_cloud/linux_build_cloud.md) is configured so that a **single job** runs in a **single instance** with: +For reference, the [SaaS runners on Linux](../ci/runners/build_cloud/linux_build_cloud.md) +are configured so that a **single job** runs in a **single instance** with: - 1 vCPU. - 3.75 GB of RAM. diff --git a/doc/integration/akismet.md b/doc/integration/akismet.md index d5e39a59dff..5541c5914d5 100644 --- a/doc/integration/akismet.md +++ b/doc/integration/akismet.md @@ -20,15 +20,15 @@ NOTE: GitLab submits all issues to Akismet. Akismet configuration is available to users on self-managed GitLab. Akismet is already enabled on -GitLab SaaS (GitLab.com), where it's configuration and management are handled by GitLab Inc. +GitLab SaaS (GitLab.com), where its configuration and management are handled by GitLab Inc. -## Configuration **(FREE SELF)** +## Configure Akismet **(FREE SELF)** To use Akismet: 1. Go to the [Akismet sign-in page](https://akismet.com/account/). 1. Sign in or create a new account. -1. Click **Show** to reveal the API key, and copy the API key's value. +1. Select **Show** to reveal the API key, and copy the API key's value. 1. Sign in to GitLab as an administrator. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Reporting** (`/admin/application_settings/reporting`). @@ -38,7 +38,7 @@ To use Akismet:  -## Training **(FREE SELF)** +## Train the Akismet filter **(FREE SELF)** To better differentiate between spam and ham, you can train the Akismet filter whenever there is a false positive or false negative. diff --git a/doc/integration/auth0.md b/doc/integration/auth0.md index e243e1defe8..8bac95c5f04 100644 --- a/doc/integration/auth0.md +++ b/doc/integration/auth0.md @@ -58,12 +58,14 @@ application. ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "auth0", - "args" => { client_id: 'YOUR_AUTH0_CLIENT_ID', - client_secret: 'YOUR_AUTH0_CLIENT_SECRET', - domain: 'YOUR_AUTH0_DOMAIN', - scope: 'openid profile email' - } + name: "auth0", + # label: "Provider name", # optional label for login button, defaults to "Auth0" + args: { + client_id: "YOUR_AUTH0_CLIENT_ID", + client_secret: "YOUR_AUTH0_CLIENT_SECRET", + domain: "YOUR_AUTH0_DOMAIN", + scope: "openid profile email" + } } ] ``` @@ -72,6 +74,7 @@ application. ```yaml - { name: 'auth0', + # label: 'Provider name', # optional label for login button, defaults to "Auth0" args: { client_id: 'YOUR_AUTH0_CLIENT_ID', client_secret: 'YOUR_AUTH0_CLIENT_SECRET', diff --git a/doc/integration/azure.md b/doc/integration/azure.md index bd16c8c0069..8d69881699b 100644 --- a/doc/integration/azure.md +++ b/doc/integration/azure.md @@ -58,11 +58,12 @@ As you go through the Microsoft procedure, keep the following in mind: ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "azure_oauth2", - "args" => { - "client_id" => "CLIENT ID", - "client_secret" => "CLIENT SECRET", - "tenant_id" => "TENANT ID", + name: "azure_oauth2", + # label: "Provider name", # optional label for login button, defaults to "Azure AD" + args: { + client_id: "CLIENT ID", + client_secret: "CLIENT SECRET", + tenant_id: "TENANT ID", } } ] @@ -72,9 +73,10 @@ As you go through the Microsoft procedure, keep the following in mind: ```yaml - { name: 'azure_oauth2', - args: { client_id: "CLIENT ID", - client_secret: "CLIENT SECRET", - tenant_id: "TENANT ID" } } + # label: 'Provider name', # optional label for login button, defaults to "Azure AD" + args: { client_id: 'CLIENT ID', + client_secret: 'CLIENT SECRET', + tenant_id: 'TENANT ID' } } ``` The `base_azure_url` is optional and can be added for different locales; @@ -167,6 +169,7 @@ Alternatively, add the `User.Read.All` application permission. gitlab_rails['omniauth_providers'] = [ { "name" => "azure_activedirectory_v2", + "label" => "Provider name", # optional label for login button, defaults to "Azure AD v2" "args" => { "client_id" => "CLIENT ID", "client_secret" => "CLIENT SECRET", @@ -180,9 +183,10 @@ Alternatively, add the `User.Read.All` application permission. ```yaml - { name: 'azure_activedirectory_v2', - args: { client_id: "CLIENT ID", - client_secret: "CLIENT SECRET", - tenant_id: "TENANT ID" } } + label: 'Provider name', # optional label for login button, defaults to "Azure AD v2" + args: { client_id: "CLIENT ID", + client_secret: "CLIENT SECRET", + tenant_id: "TENANT ID" } } ``` The `base_azure_url` is optional and can be added for different locales; diff --git a/doc/integration/bitbucket.md b/doc/integration/bitbucket.md index fa3b62ba7af..db7e7d74efe 100644 --- a/doc/integration/bitbucket.md +++ b/doc/integration/bitbucket.md @@ -42,7 +42,7 @@ to the end of the Bitbucket authorization callback URL. - **Name:** This can be anything. Consider something like `<Organization>'s GitLab` or `<Your Name>'s GitLab` or something else descriptive. - - **Application description:** *(Optional)* Fill this in if you wish. + - **Application description:** Optional. Fill this in if you wish. - **Callback URL:** (Required in GitLab versions 8.15 and greater) The URL to your GitLab installation, such as `https://gitlab.example.com/users/auth`. @@ -87,10 +87,11 @@ to the end of the Bitbucket authorization callback URL. ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "bitbucket", - "app_id" => "BITBUCKET_APP_KEY", - "app_secret" => "BITBUCKET_APP_SECRET", - "url" => "https://bitbucket.org/" + name: "bitbucket", + # label: "Provider name", # optional label for login button, defaults to "Bitbucket" + app_id: "BITBUCKET_APP_KEY", + app_secret: "BITBUCKET_APP_SECRET", + url: "https://bitbucket.org/" } ] ``` @@ -102,6 +103,7 @@ to the end of the Bitbucket authorization callback URL. enabled: true providers: - { name: 'bitbucket', + # label: 'Provider name', # optional label for login button, defaults to "Bitbucket" app_id: 'BITBUCKET_APP_KEY', app_secret: 'BITBUCKET_APP_SECRET', url: 'https://bitbucket.org/' } diff --git a/doc/integration/cas.md b/doc/integration/cas.md index 4699f7147aa..9594836164a 100644 --- a/doc/integration/cas.md +++ b/doc/integration/cas.md @@ -37,14 +37,14 @@ configure CAS for back-channel logout. ```ruby gitlab_rails['omniauth_providers'] = [ { - "name"=> "cas3", - "label"=> "cas", - "args"=> { - "url"=> 'CAS_SERVER', - "login_url"=> '/CAS_PATH/login', - "service_validate_url"=> '/CAS_PATH/p3/serviceValidate', - "logout_url"=> '/CAS_PATH/logout' - } + name: "cas3", + label: "Provider name", # optional label for login button, defaults to "Cas3" + args: { + url: "CAS_SERVER", + login_url: "/CAS_PATH/login", + service_validate_url: "/CAS_PATH/p3/serviceValidate", + logout_url: "/CAS_PATH/logout" + } } ] ``` @@ -53,7 +53,7 @@ configure CAS for back-channel logout. ```yaml - { name: 'cas3', - label: 'cas', + label: 'Provider name', # optional label for login button, defaults to "Cas3" args: { url: 'CAS_SERVER', login_url: '/CAS_PATH/login', diff --git a/doc/integration/datadog.md b/doc/integration/datadog.md index 687be5adcf7..89e08d330e8 100644 --- a/doc/integration/datadog.md +++ b/doc/integration/datadog.md @@ -32,20 +32,20 @@ project, group, or instance level: 1. Scroll to **Add an integration**, and select **Datadog**. 1. Select **Active** to enable the integration. 1. Specify the [**Datadog site**](https://docs.datadoghq.com/getting_started/site/) to send data to. -1. (Optional) To override the API URL used to send data directly, provide an **API URL**. +1. Optional. To override the API URL used to send data directly, provide an **API URL**. Used only in advanced scenarios. 1. Provide your Datadog **API key**. -1. (Optional) If you use more than one GitLab instance, provide a unique **Service** name +1. Optional. If you use more than one GitLab instance, provide a unique **Service** name to differentiate between your GitLab instances. -1. (Optional) If you use groups of GitLab instances (such as staging and production +1. Optional. If you use groups of GitLab instances (such as staging and production environments), provide an **Env** name. This value is attached to each span the integration generates. -1. (Optional) Select **Test settings** to test your integration. +1. Optional. Select **Test settings** to test your integration. 1. Select **Save changes**. When the integration sends data, you can view it in the [CI Visibility](https://app.datadoghq.com/ci) section of your Datadog account. -## Related links +## Related topics - [Datadog's CI Visibility](https://docs.datadoghq.com/continuous_integration/) documentation. diff --git a/doc/integration/ding_talk.md b/doc/integration/ding_talk.md index 55d51a5ebff..2bc4f29298a 100644 --- a/doc/integration/ding_talk.md +++ b/doc/integration/ding_talk.md @@ -60,9 +60,10 @@ Sign in to DingTalk Open Platform and create an application on it. DingTalk gene ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "ding_talk", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET" + name: "dingtalk", + # label: "Provider name", # optional label for login button, defaults to "Ding Talk" + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET" } ] ``` @@ -70,7 +71,8 @@ Sign in to DingTalk Open Platform and create an application on it. DingTalk gene For installations from source: ```yaml - - { name: 'ding_talk', + - { name: 'dingtalk', + # label: 'Provider name', # optional label for login button, defaults to "Ding Talk" app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET' } ``` diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index 20f8fdc55f2..8461aca8c8d 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -48,7 +48,7 @@ each node should have: Elasticsearch is *not* included in the Omnibus packages or when you install from source. You must [install it separately](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/install-elasticsearch.html "Elasticsearch 7.x installation documentation") and ensure you select your version. Detailed information on how to install Elasticsearch is out of the scope of this page. -You can install Elasticsearch yourself, or use a cloud hosted offering such as [Elasticsearch Service](https://www.elastic.co/elasticsearch/service)(available on AWS, GCP, or Azure) or the [Amazon Elasticsearch](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) +You can install Elasticsearch yourself, or use a cloud hosted offering such as [Elasticsearch Service](https://www.elastic.co/elasticsearch/service) (available on AWS, GCP, or Azure) or the [Amazon OpenSearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/gsg.html) service. You should install Elasticsearch on a separate server. Running Elasticsearch on the same server as GitLab is not recommended and can cause a degradation in GitLab instance performance. @@ -93,7 +93,7 @@ the indexer itself. This project relies on [International Components for Unicode](https://icu.unicode.org/) (ICU) for text encoding, therefore we must ensure the development packages for your platform are -installed before running `make`. +installed before running `make`. #### Debian / Ubuntu @@ -207,9 +207,9 @@ The following Elasticsearch settings are available: | `Password` | The password of your Elasticsearch instance. | | `Number of Elasticsearch shards` | Elasticsearch indexes are split into multiple shards for performance reasons. In general, you should use at least 5 shards, and indexes with tens of millions of documents need to have more shards ([see below](#guidance-on-choosing-optimal-cluster-configuration)). Changes to this value do not take effect until the index is recreated. You can read more about tradeoffs in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/scalability.html). | | `Number of Elasticsearch replicas` | Each Elasticsearch shard can have a number of replicas. These are a complete copy of the shard, and can provide increased query performance or resilience against hardware failure. Increasing this value increases total disk space required by the index. | -| `Limit namespaces and projects that can be indexed` | Enabling this allows you to select namespaces and projects to index. All other namespaces and projects use database search instead. If you enable this option but do not select any namespaces or projects, none will be indexed. [Read more below](#limit-namespaces-and-projects). -| `Using AWS hosted Elasticsearch with IAM credentials` | Sign your Elasticsearch requests using [AWS IAM authorization](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html), [AWS EC2 Instance Profile Credentials](https://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli), or [AWS ECS Tasks Credentials](https://docs.aws.amazon.com/AmazonECS/latest/userguide/task-iam-roles.html). Please refer to [Identity and Access Management in Amazon Elasticsearch Service](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-ac.html) for details of AWS hosted Elasticsearch domain access policy configuration. | -| `AWS Region` | The AWS region in which your Elasticsearch service is located. | +| `Limit namespaces and projects that can be indexed` | Enabling this allows you to select namespaces and projects to index. All other namespaces and projects use database search instead. If you enable this option but do not select any namespaces or projects, none are indexed. [Read more below](#limit-namespaces-and-projects). +| `Using AWS hosted Elasticsearch with IAM credentials` | Sign your Elasticsearch requests using [AWS IAM authorization](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html), [AWS EC2 Instance Profile Credentials](https://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli), or [AWS ECS Tasks Credentials](https://docs.aws.amazon.com/AmazonECS/latest/userguide/task-iam-roles.html). Please refer to [Identity and Access Management in Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html) for details of AWS hosted OpenSearch domain access policy configuration. | +| `AWS Region` | The AWS region in which your OpenSearch Service is located. | | `AWS Access Key` | The AWS access key. | | `AWS Secret Access Key` | The AWS secret access key. | | `Maximum file size indexed` | See [the explanation in instance limits.](../administration/instance_limits.md#maximum-file-size-indexed). | @@ -233,7 +233,7 @@ If you select `Limit namespaces and projects that can be indexed`, more options You can select namespaces and projects to index exclusively. Note that if the namespace is a group, it includes any subgroups and projects belonging to those subgroups to be indexed as well. -Advanced Search only provides cross-group code/commit search (global) if all name-spaces are indexed. In this particular scenario where only a subset of namespaces are indexed, a global search will not provide a code or commit scope. This is possible only in the scope of an indexed namespace. There is no way to code/commit search in multiple indexed namespaces (when only a subset of namespaces has been indexed). For example if two groups are indexed, there is no way to run a single code search on both. You can only run a code search on the first group and then on the second. +Advanced Search only provides cross-group code/commit search (global) if all name-spaces are indexed. In this particular scenario where only a subset of namespaces are indexed, a global search does not provide a code or commit scope. This is possible only in the scope of an indexed namespace. There is no way to code/commit search in multiple indexed namespaces (when only a subset of namespaces has been indexed). For example if two groups are indexed, there is no way to run a single code search on both. You can only run a code search on the first group and then on the second. You can filter the selection dropdown by writing part of the namespace or project name you're interested in. @@ -279,8 +279,8 @@ To disable the Elasticsearch integration: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Advanced Search**. 1. Uncheck **Elasticsearch indexing** and **Search with Elasticsearch enabled**. -1. Click **Save changes** for the changes to take effect. -1. (Optional) Delete the existing indexes: +1. Select **Save changes**. +1. Optional. Delete the existing indexes: ```shell # Omnibus installations @@ -355,7 +355,7 @@ Setting this value too high can have adverse performance impacts as your cluster may become heavily saturated with searches and writes. Setting this value too low may lead the reindexing process to take a very long time to complete. -The best value for this will depend on your cluster size, whether you're willing +The best value for this depends on your cluster size, whether you're willing to accept some degraded search performance during reindexing, and how important it is for the reindex to finish quickly and resume indexing. @@ -375,7 +375,7 @@ Sometimes, you might want to abandon the unfinished reindex job and resume the i 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Advanced Search**. -1. Expand **Elasticsearch zero-downtime reindexing**. +1. Expand **Advanced Search**. 1. Clear the **Pause Elasticsearch indexing** checkbox. ## Advanced Search migrations @@ -430,23 +430,23 @@ In order to debug issues with the migrations you can check the [`elasticsearch.l ### Retry a halted migration Some migrations are built with a retry limit. If the migration cannot finish within the retry limit, -it will be halted and a notification will be displayed in the Advanced Search integration settings. +it is halted and a notification is displayed in the Advanced Search integration settings. It is recommended to check the [`elasticsearch.log` file](../administration/logs.md#elasticsearchlog) to debug why the migration was halted and make any changes before retrying the migration. Once you believe you've -fixed the cause of the failure, click "Retry migration", and the migration will be scheduled to be retried +fixed the cause of the failure, click "Retry migration", and the migration is scheduled to be retried in the background. If you cannot get the migration to succeed, you may consider the [last resort to recreate the index from scratch](#last-resort-to-recreate-an-index). This may allow you to skip over -the problem because a newly created index will skip all migrations as the index -will be recreated with the correct up-to-date schema. +the problem because a newly created index skips all migrations as the index +is recreated with the correct up-to-date schema. ### All migrations must be finished before doing a major upgrade Before doing a major version GitLab upgrade, you should have completed all migrations that exist up until the latest minor version before that major -version. If you have halted migrations, these will need to be resolved and +version. If you have halted migrations, these need to be resolved and [retried](#retry-a-halted-migration) before proceeding with a major version upgrade. Read more about [upgrading to a new major version](../update/index.md#upgrading-to-a-new-major-version). @@ -468,7 +468,7 @@ The following are some available Rake tasks: | [`sudo gitlab-rake gitlab:elastic:resume_indexing`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Resumes Elasticsearch indexing. | | [`sudo gitlab-rake gitlab:elastic:index_projects`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Iterates over all projects and queues Sidekiq jobs to index them in the background. | | [`sudo gitlab-rake gitlab:elastic:index_projects_status`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Determines the overall status of the indexing. It is done by counting the total number of indexed projects, dividing by a count of the total number of projects, then multiplying by 100. | -| [`sudo gitlab-rake gitlab:elastic:clear_index_status`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Deletes all instances of IndexStatus for all projects. Note that this command will result in a complete wipe of the index, and it should be used with caution. | +| [`sudo gitlab-rake gitlab:elastic:clear_index_status`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Deletes all instances of IndexStatus for all projects. Note that this command results in a complete wipe of the index, and it should be used with caution. | | [`sudo gitlab-rake gitlab:elastic:create_empty_index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Generates empty indexes (the default index and a separate issues index) and assigns an alias for each on the Elasticsearch side only if it doesn't already exist. | | [`sudo gitlab-rake gitlab:elastic:delete_index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Removes the GitLab indexes and aliases (if they exist) on the Elasticsearch instance. | | [`sudo gitlab-rake gitlab:elastic:recreate_index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Wrapper task for `gitlab:elastic:delete_index` and `gitlab:elastic:create_empty_index`. | @@ -501,7 +501,7 @@ I, [2019-03-04T21:27:05.215266 #3384] INFO -- : Indexing GitLab User / test (ID ## Advanced Search index scopes -When performing a search, the GitLab index will use the following scopes: +When performing a search, the GitLab index uses the following scopes: | Scope Name | What it searches | | ---------------- | ---------------------- | @@ -521,25 +521,25 @@ When performing a search, the GitLab index will use the following scopes: For basic guidance on choosing a cluster configuration you may refer to [Elastic Cloud Calculator](https://cloud.elastic.co/pricing). You can find more information below. -- Generally, you will want to use at least a 2-node cluster configuration with one replica, which will allow you to have resilience. If your storage usage is growing quickly, you may want to plan horizontal scaling (adding more nodes) beforehand. -- It's not recommended to use HDD storage with the search cluster, because it will take a hit on performance. It's better to use SSD storage (NVMe or SATA SSD drives for example). +- Generally, you want to use at least a 2-node cluster configuration with one replica, which allows you to have resilience. If your storage usage is growing quickly, you may want to plan horizontal scaling (adding more nodes) beforehand. +- It's not recommended to use HDD storage with the search cluster, because it takes a hit on performance. It's better to use SSD storage (NVMe or SATA SSD drives for example). - You can use the [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) to benchmark search performance with different search cluster sizes and configurations. - `Heap size` should be set to no more than 50% of your physical RAM. Additionally, it shouldn't be set to more than the threshold for zero-based compressed oops. The exact threshold varies, but 26 GB is safe on most systems, but can also be as large as 30 GB on some systems. See [Heap size settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#heap-size-settings) and [Setting JVM options](https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html) for more details. - Number of CPUs (CPU cores) per node usually corresponds to the `Number of Elasticsearch shards` setting described below. -- A good guideline is to ensure you keep the number of shards per node below 20 per GB heap it has configured. A node with a 30GB heap should therefore have a maximum of 600 shards, but the further below this limit you can keep it the better. This will generally help the cluster stay in good health. +- A good guideline is to ensure you keep the number of shards per node below 20 per GB heap it has configured. A node with a 30GB heap should therefore have a maximum of 600 shards, but the further below this limit you can keep it the better. This generally helps the cluster stay in good health. - Number of Elasticsearch shards: - Small shards result in small segments, which increases overhead. Aim to keep the average shard size between at least a few GB and a few tens of GB. - Another consideration is the number of documents. To determine the number of shards to use, sum the numbers in the **Menu > Admin > Dashboard > Statistics** pane (the number of documents to be indexed), divide by 5 million, and add 5. For example: - If you have fewer than about 2,000,000 documents, use the default of 5 shards - 10,000,000 documents: `10000000/5000000 + 5` = 7 shards - 100,000,000 documents: `100000000/5000000 + 5` = 25 shards -- `refresh_interval` is a per index setting. You may want to adjust that from default `1s` to a bigger value if you don't need data in real-time. This will change how soon you will see fresh results. If that's important for you, you should leave it as close as possible to the default value. +- `refresh_interval` is a per index setting. You may want to adjust that from default `1s` to a bigger value if you don't need data in real-time. This changes how soon you see fresh results. If that's important for you, you should leave it as close as possible to the default value. - You might want to raise [`indices.memory.index_buffer_size`](https://www.elastic.co/guide/en/elasticsearch/reference/current/indexing-buffer.html) to 30% or 40% if you have a lot of heavy indexing operations. ### Advanced Search integration settings guidance -- The `Number of Elasticsearch shards` setting usually corresponds with the number of CPUs available in your cluster. For example, if you have a 3-node cluster with 4 cores each, this means you will benefit from having at least 3*4=12 shards in the cluster. It's only possible to change the shards number by using [Split index API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-split-index.html) or by reindexing to a different index with a changed number of shards. -- The `Number of Elasticsearch replicas` setting should most of the time be equal to `1` (each shard will have 1 replica). Using `0` is not recommended, because losing one node will corrupt the index. +- The `Number of Elasticsearch shards` setting usually corresponds with the number of CPUs available in your cluster. For example, if you have a 3-node cluster with 4 cores each, this means you benefit from having at least 3*4=12 shards in the cluster. It's only possible to change the shards number by using [Split index API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-split-index.html) or by reindexing to a different index with a changed number of shards. +- The `Number of Elasticsearch replicas` setting should most of the time be equal to `1` (each shard has 1 replica). Using `0` is not recommended, because losing one node corrupts the index. ### Indexing large instances @@ -548,7 +548,7 @@ This section may be helpful in the event that the other due to large volumes of data being indexed. WARNING: -Indexing a large instance will generate a lot of Sidekiq jobs. +Indexing a large instance generates a lot of Sidekiq jobs. Make sure to prepare for this task by having a [Scalable and Highly Available Setup](../administration/reference_architectures/index.md) or creating [extra Sidekiq processes](../administration/operations/extra_sidekiq_processes.md). @@ -890,11 +890,11 @@ There is also an easy way to check it automatically with `sudo gitlab-rake gitla This exception is seen when your Elasticsearch cluster is configured to reject requests above a certain size (10MiB in this case). This corresponds to the `http.max_content_length` setting in `elasticsearch.yml`. Increase it to a larger size and restart your Elasticsearch cluster. -AWS has [fixed limits](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html) for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of the underlying instance. +AWS has [fixed limits](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/aes-limits.html) for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of the underlying instance. ### My single node Elasticsearch cluster status never goes from `yellow` to `green` even though everything seems to be running properly -**For a single node Elasticsearch cluster the functional cluster health status will be yellow** (never green) because the primary shard is allocated but replicas cannot be as there is no other node to which Elasticsearch can assign a replica. This also applies if you are using the [Amazon Elasticsearch](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html#aes-handling-errors-yellow-cluster-status) service. +**For a single node Elasticsearch cluster the functional cluster health status will be yellow** (never green) because the primary shard is allocated but replicas cannot be as there is no other node to which Elasticsearch can assign a replica. This also applies if you are using the [Amazon OpenSearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/aes-handling-errors.html#aes-handling-errors-yellow-cluster-status) service. WARNING: Setting the number of replicas to `0` is discouraged (this is not allowed in the GitLab Elasticsearch Integration menu). If you are planning to add more Elasticsearch nodes (for a total of more than 1 Elasticsearch) the number of replicas will need to be set to an integer value larger than `0`. Failure to do so will result in lack of redundancy (losing one node will corrupt the index). @@ -981,11 +981,11 @@ however searches will only surface results that can be viewed by the user. Advanced Search will honor all permission checks in the application by filtering out projects that a user does not have access to at search time. -### Access requirements for the self-managed AWS Elasticsearch Service +### Access requirements for the self-managed AWS OpenSearch Service -To use the self-managed AWS Elasticsearch Service with GitLab, configure your instance's domain access policies +To use the self-managed AWS OpenSearch Service with GitLab, configure your instance's domain access policies to contain the actions below. -See [Identity and Access Management in Amazon Elasticsearch Service](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-ac.html) for details. +See [Identity and Access Management in Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html) for details. ```plaintext es:ESHttpDelete diff --git a/doc/integration/facebook.md b/doc/integration/facebook.md index 1a3360aa470..b94fa24d290 100644 --- a/doc/integration/facebook.md +++ b/doc/integration/facebook.md @@ -81,9 +81,10 @@ Facebook. Facebook generates an app ID and secret key for you to use. ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "facebook", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET" + name: "facebook", + # label: "Provider name", # optional label for login button, defaults to "Facebook" + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET" } ] ``` @@ -91,7 +92,9 @@ Facebook. Facebook generates an app ID and secret key for you to use. For installations from source: ```yaml - - { name: 'facebook', app_id: 'YOUR_APP_ID', + - { name: 'facebook', + # label: 'Provider name', # optional label for login button, defaults to "Facebook" + app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET' } ``` diff --git a/doc/integration/github.md b/doc/integration/github.md index 11a9a5ea64d..d8877e069b8 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -50,10 +50,11 @@ Follow these steps to incorporate the GitHub OAuth 2 app in your GitLab server: ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "github", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET", - "args" => { "scope" => "user:email" } + name: "github", + # label: "Provider name", # optional label for login button, defaults to "GitHub" + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET", + args: { scope: "user:email" } } ] ``` @@ -63,11 +64,12 @@ Follow these steps to incorporate the GitHub OAuth 2 app in your GitLab server: ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "github", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET", - "url" => "https://github.example.com/", - "args" => { "scope" => "user:email" } + name: "github", + # label: "Provider name", # optional label for login button, defaults to "GitHub" + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET", + url: "https://github.example.com/", + args: { scope: "user:email" } } ] ``` @@ -85,7 +87,9 @@ Follow these steps to incorporate the GitHub OAuth 2 app in your GitLab server: For GitHub.com: ```yaml - - { name: 'github', app_id: 'YOUR_APP_ID', + - { name: 'github', + # label: 'Provider name', # optional label for login button, defaults to "GitHub" + app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET', args: { scope: 'user:email' } } ``` @@ -94,6 +98,7 @@ Follow these steps to incorporate the GitHub OAuth 2 app in your GitLab server: ```yaml - { name: 'github', + # label: 'Provider name', # optional label for login button, defaults to "GitHub" app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET', url: "https://github.example.com/", @@ -122,12 +127,13 @@ For Omnibus package: ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "github", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET", - "url" => "https://github.example.com/", - "verify_ssl" => false, - "args" => { "scope" => "user:email" } + name: "github", + # label: "Provider name", # optional label for login button, defaults to "GitHub" + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET", + url: "https://github.example.com/", + verify_ssl: false, + args: { scope: "user:email" } } ] ``` @@ -142,6 +148,7 @@ For installation from source: ```yaml - { name: 'github', + # label: 'Provider name', # optional label for login button, defaults to "GitHub" app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET', url: "https://github.example.com/", diff --git a/doc/integration/gitlab.md b/doc/integration/gitlab.md index b69147b3829..2dd357e50a6 100644 --- a/doc/integration/gitlab.md +++ b/doc/integration/gitlab.md @@ -53,10 +53,11 @@ GitLab.com generates an application ID and secret key for you to use. ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "gitlab", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET", - "args" => { "scope" => "api" } + name: "gitlab", + # label: "Provider name", # optional label for login button, defaults to "GitLab.com" + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET", + args: { scope: "api" } } ] ``` @@ -66,10 +67,11 @@ GitLab.com generates an application ID and secret key for you to use. ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "gitlab", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET", - "args" => { "scope" => "api", "client_options" => { "site" => "https://gitlab.example.com/api/v4" } } + name: "gitlab", + label: "Provider name", # optional label for login button, defaults to "GitLab.com" + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET", + args: { scope: "api", client_options: { site: "https://gitlab.example.com/api/v4" } } } ] ``` @@ -78,6 +80,7 @@ GitLab.com generates an application ID and secret key for you to use. ```yaml - { name: 'gitlab', + # label: 'Provider name', # optional label for login button, defaults to "GitLab.com" app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET', args: { scope: 'api' } } @@ -87,6 +90,7 @@ GitLab.com generates an application ID and secret key for you to use. ```yaml - { name: 'gitlab', + label: 'Provider name', # optional label for login button, defaults to "GitLab.com" app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET', args: { scope: 'api', "client_options": { "site": 'https://gitlab.example.com/api/v4' } } diff --git a/doc/integration/gitpod.md b/doc/integration/gitpod.md index 26520df18fa..977e794396e 100644 --- a/doc/integration/gitpod.md +++ b/doc/integration/gitpod.md @@ -5,7 +5,7 @@ group: Editor info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" --- -# Gitpod Integration **(FREE)** +# Gitpod integration **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/228893) in GitLab 13.4. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/258206) in GitLab 13.8 @@ -38,7 +38,7 @@ With the Gitpod integration enabled for your GitLab instance, to enable it for y 1. In the top-right corner, select your avatar. 1. Select **Preferences**. 1. Under **Preferences**, locate the **Integrations** section. -1. Check the **Enable Gitpod integration** checkbox and select the **Save changes** button. +1. Select the **Enable Gitpod integration** checkbox and select **Save changes**. ## Configure a self-managed instance **(FREE SELF)** @@ -50,9 +50,9 @@ For GitLab self-managed instances, a GitLab administrator needs to: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Gitpod** configuration section. - 1. Check the **Enable Gitpod integration** checkbox. + 1. Select the **Enable Gitpod integration** checkbox. 1. Add your Gitpod instance URL (for example, `https://gitpod.example.com`). - 1. Select the **Save changes** button. + 1. Select **Save changes**. Your users can then [enable it for themselves](#enable-gitpod-in-your-user-settings). diff --git a/doc/integration/google.md b/doc/integration/google.md index 5df76ebb3d1..d6a37dbf30f 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -79,10 +79,11 @@ On your GitLab server: ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "google_oauth2", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET", - "args" => { "access_type" => "offline", "approval_prompt" => '' } + name: "google_oauth2", + # label: "Provider name", # optional label for login button, defaults to "Google" + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET", + args: { access_type: "offline", approval_prompt: "" } } ] ``` @@ -91,6 +92,7 @@ On your GitLab server: ```yaml - { name: 'google_oauth2', + # label: 'Provider name', # optional label for login button, defaults to "Google" app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET', args: { access_type: 'offline', approval_prompt: '' } } diff --git a/doc/integration/img/ding_talk_menu.png b/doc/integration/img/ding_talk_menu.png Binary files differindex 2c5a23435fa..fe020cab397 100644 --- a/doc/integration/img/ding_talk_menu.png +++ b/doc/integration/img/ding_talk_menu.png diff --git a/doc/integration/img/omniauth_providers_v_14_6.png b/doc/integration/img/omniauth_providers_v_14_6.png Binary files differnew file mode 100644 index 00000000000..b434e9a210b --- /dev/null +++ b/doc/integration/img/omniauth_providers_v_14_6.png diff --git a/doc/integration/index.md b/doc/integration/index.md index 0b2bf6fde94..61d8547aaf7 100644 --- a/doc/integration/index.md +++ b/doc/integration/index.md @@ -46,7 +46,7 @@ GitLab has integrated with several security partners. For more information, see ## Continuous integration -GitLab can be integrated with the following external service for continuous integration: +GitLab can be integrated with the following external services for continuous integration: - [Jenkins](jenkins.md) CI. - [Datadog](datadog.md), to monitor for CI/CD job failures and performance issues. diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md index 2bbda74533f..822530775e5 100644 --- a/doc/integration/jenkins.md +++ b/doc/integration/jenkins.md @@ -179,6 +179,17 @@ to integrate GitLab and Jenkins. ## Troubleshooting +### Error during GitLab configuration - "Connection failed. Please check your settings" + +If you get this error message while configuring GitLab, the following are possible causes: + +- GitLab is unable to reach your Jenkins instance at the address. If your GitLab instance is self-managed, try pinging the + Jenkins instance at the domain provided on the GitLab instance. +- The Jenkins instance is at a local address and is not included in the + [GitLab installation's allowlist](../security/webhooks.md#allowlist-for-local-requests). +- The credentials for the Jenkins instance do not have sufficient access or are invalid. +- The **Enable authentication for ‘/project’ end-point checkbox** is not selected in your [Jenkin's plugin configuration](#configure-the-jenkins-server). + ### Error in merge requests - "Could not connect to the CI server" This integration relies on Jenkins reporting the build status back to GitLab via diff --git a/doc/integration/jira/connect-app.md b/doc/integration/jira/connect-app.md index 27f482ee2ba..597293ae5ca 100644 --- a/doc/integration/jira/connect-app.md +++ b/doc/integration/jira/connect-app.md @@ -76,7 +76,7 @@ If the app requires additional permissions, [the update must first be manually a ## Install the GitLab.com for Jira Cloud app for self-managed instances **(FREE SELF)** If your GitLab instance is self-managed, you must follow some -extra steps to install the GitLab.com for Jira Cloud app. +extra steps to install the GitLab.com for Jira Cloud app, and your GitLab instance must be accessible by Jira. Each Jira Cloud application must be installed from a single location. Jira fetches a [manifest file](https://developer.atlassian.com/cloud/jira/platform/connect-app-descriptor/) diff --git a/doc/integration/jira/dvcs.md b/doc/integration/jira/dvcs.md index 2b7cc5ff0a6..e69b7675a59 100644 --- a/doc/integration/jira/dvcs.md +++ b/doc/integration/jira/dvcs.md @@ -265,3 +265,22 @@ resynchronize the information: For more information, read [Atlassian's documentation](https://support.atlassian.com/jira-cloud-administration/docs/synchronize-jira-cloud-to-bitbucket/). + +### `Sync Failed` error when refreshing repository data + +If you get a `Sync Failed` error in Jira when [refreshing repository data](#refresh-data-imported-to-jira) for specific projects, check your DVCS connector logs. Look for errors that occur when executing requests to API resources in GitLab. For example: + +```plaintext +Failed to execute request [https://gitlab.com/api/v4/projects/:id/merge_requests?page=1&per_page=100 GET https://gitlab.com/api/v4/projects/:id/merge_requests?page=1&per_page=100 returned a response status of 403 Forbidden] errors: +{"message":"403 Forbidden"} +``` + +If you find a `{"message":"403 Forbidden"}` error, it is possible that this specific project has some [GitLab features disabled](../../user/project/settings/index.md#sharing-and-permissions). +In the example above, the merge requests feature is disabled. + +To resolve the issue, enable the relevant feature: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Visibility, project features, permissions**. +1. Use the toggles to enable the features as needed. diff --git a/doc/integration/jira/img/open_jira_issues_list_v13.2.png b/doc/integration/jira/img/open_jira_issues_list_v13.2.png Binary files differdeleted file mode 100644 index 0cf58433b25..00000000000 --- a/doc/integration/jira/img/open_jira_issues_list_v13.2.png +++ /dev/null diff --git a/doc/integration/jira/img/open_jira_issues_list_v14_6.png b/doc/integration/jira/img/open_jira_issues_list_v14_6.png Binary files differnew file mode 100644 index 00000000000..6f06b7fec9a --- /dev/null +++ b/doc/integration/jira/img/open_jira_issues_list_v14_6.png diff --git a/doc/integration/jira/issues.md b/doc/integration/jira/issues.md index 70e938a24d4..e24862242e1 100644 --- a/doc/integration/jira/issues.md +++ b/doc/integration/jira/issues.md @@ -101,10 +101,10 @@ Consider this example: You can browse, search, and view issues from a selected Jira project directly in GitLab, if your GitLab administrator [has configured it](configure.md). -To do this, in GitLab, go to your project and select **Jira > Issues list**. The issue list +To do this, in GitLab, go to your project and select **Issues > Jira issues**. The issue list sorts by **Created date** by default, with the newest issues listed at the top: - + - To display the most recently updated issues first, select **Last updated**. - You can [search and filter](#search-and-filter-the-issues-list) the issues list. @@ -140,7 +140,7 @@ Enhancements to use these filters through the user interface ## Automatic issue transitions -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/...) in GitLab 13.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55773) in GitLab 13.11. When you configure automatic issue transitions, you can transition a referenced Jira issue to the next available status with a category of **Done**. To configure diff --git a/doc/integration/mattermost/gitlab-mattermost.msc b/doc/integration/mattermost/gitlab-mattermost.msc deleted file mode 100644 index f6d4bf7aa68..00000000000 --- a/doc/integration/mattermost/gitlab-mattermost.msc +++ /dev/null @@ -1,28 +0,0 @@ -msc { - # Use https://mscgen.js.org or mscgen to convert this into PNG - hscale="1.5", - wordwraparcs=on; - - user [ label="User", textbgcolor="blue", textcolor="white" ], - mattermost [ label="Mattermost", textbgcolor="red", textcolor="white"], - gitlab [ label="GitLab", textbgcolor="indigo", textcolor="white"]; - - user=>mattermost [label="GET https://mm.domain.com"]; - mattermost note gitlab [label="Obtain access code", textcolor="green"]; - mattermost=>gitlab [label="GET https://gitlab.domain.com/oauth/authorize", textcolor="indigo"]; - gitlab rbox user [label="GitLab user logs in (if necessary)"]; - gitlab rbox gitlab [label="GitLab verifies client_id matches an OAuth application"]; - gitlab=>user [label="GitLab asks user to authorize Mattermost OAuth app"]; - user=>gitlab [label="User clicks 'Allow'"]; - gitlab rbox gitlab [label="GitLab verifies redirect_uri matches list of valid URLs"]; - gitlab=>user [label="302 Redirect: https://mm.domain.com/signup/gitlab/complete"]; - user=>mattermost [label="GET https://mm.domain.com/signup/gitlab/complete", textcolor="red"]; - mattermost note gitlab [label="Exchange access code for access token", textcolor="green"]; - mattermost=>gitlab [label="POST http://gitlab.domain.com/oauth/token", textcolor="indigo"]; - gitlab=>gitlab [label="Doorkeeper::TokensController#create"]; - gitlab=>mattermost [label="Access token", textcolor="red"]; - mattermost note gitlab [label="Mattermost looks up GitLab user", textcolor="green"]; - mattermost=>gitlab [label="GET https://gitlab.domain.com/api/v4/user", textcolor="indigo"]; - gitlab=>mattermost [label="User details", textcolor="red"]; - mattermost=>user [label="Mattermost/GitLab user ready"]; -} diff --git a/doc/integration/mattermost/img/gitlab-mattermost.png b/doc/integration/mattermost/img/gitlab-mattermost.png Binary files differdeleted file mode 100644 index c3b019988d0..00000000000 --- a/doc/integration/mattermost/img/gitlab-mattermost.png +++ /dev/null diff --git a/doc/integration/mattermost/index.md b/doc/integration/mattermost/index.md index 0489ccd431c..97da971dd75 100644 --- a/doc/integration/mattermost/index.md +++ b/doc/integration/mattermost/index.md @@ -18,8 +18,8 @@ Each release of GitLab Mattermost is compiled and manually tested on an AMD 64 c ## Getting started -GitLab Mattermost expects to run on its own virtual host. In your DNS settings you will need -two entries pointing to the same machine, e.g., `gitlab.example.com` and +GitLab Mattermost expects to run on its own virtual host. In your DNS settings, you need +two entries pointing to the same machine. For example, `gitlab.example.com` and `mattermost.example.com`. GitLab Mattermost is disabled by default. To enable it: @@ -41,7 +41,7 @@ GitLab Mattermost is disabled by default. To enable it: The Omnibus GitLab package attempts to automatically authorize GitLab Mattermost with GitLab if the applications are running on the same server. Automatic authorization requires access to the GitLab database. If the GitLab database is not available -you will need to manually authorize GitLab Mattermost for access to GitLab using the process described in the [Authorize GitLab Mattermost section](#authorize-gitlab-mattermost). +you need to manually authorize GitLab Mattermost for access to GitLab using the process described in the [Authorize GitLab Mattermost section](#authorize-gitlab-mattermost). ## Configuring Mattermost @@ -51,7 +51,7 @@ Mattermost settings and where they can be set is available [in the Mattermost do While using the System Console is recommended, you can also configure Mattermost using one of the following options: 1. Edit the Mattermost configuration directly through `/var/opt/gitlab/mattermost/config.json`. -1. Specify environment variables used to run Mattermost by changing the `mattermost['env']` setting in `gitlab.rb`. Any settings configured in this way will be disabled from the System Console and cannot be changed without restarting Mattermost. +1. Specify environment variables used to run Mattermost by changing the `mattermost['env']` setting in `gitlab.rb`. Any settings configured in this way are disabled from the System Console and cannot be changed without restarting Mattermost. ## Running GitLab Mattermost with HTTPS @@ -71,7 +71,7 @@ mattermost_nginx['redirect_http_to_https'] = true ``` If you haven't named your certificate and key `mattermost.gitlab.example.crt` -and `mattermost.gitlab.example.key` then you'll need to also add the full paths +and `mattermost.gitlab.example.key` then you need to also add the full paths as shown below. ```ruby @@ -85,7 +85,7 @@ Once the configuration is set, run `sudo gitlab-ctl reconfigure` to apply the ch ## Running GitLab Mattermost on its own server -If you want to run GitLab and GitLab Mattermost on two separate servers the GitLab services will still be set up on your GitLab Mattermost server, but they will not accept user requests or +If you want to run GitLab and GitLab Mattermost on two separate servers the GitLab services are still set up on your GitLab Mattermost server, but they do not accept user requests or consume system resources. You can use the following settings and configuration details on the GitLab Mattermost server to effectively disable the GitLab service bundled into the Omnibus package. ```ruby @@ -124,7 +124,7 @@ http://mattermost.example.com/login/gitlab/complete Note that you do not need to select any options under **Scopes**. Choose **Save application**. -Once the application is created you will be provided with an `Application ID` and `Secret`. One other piece of information needed is the URL of GitLab instance. +Once the application is created you are provided with an `Application ID` and `Secret`. One other piece of information needed is the URL of GitLab instance. Return to the server running GitLab Mattermost and edit the `/etc/gitlab/gitlab.rb` configuration file as follows using the values you received above: ```ruby @@ -190,7 +190,7 @@ sudo -i -u gitlab-psql -- /opt/gitlab/embedded/bin/pg_dump -h /var/opt/gitlab/po #### Back up the `data` directory and `config.json` -Mattermost has a `data` directory and `config.json` file that will need to be backed up as well: +Mattermost has a `data` directory and `config.json` file that need to be backed up as well: ```shell sudo tar -zcvf mattermost_data_$(date --rfc-3339=date).gz -C /var/opt/gitlab/mattermost data config.json @@ -339,6 +339,11 @@ Below is a list of Mattermost versions for GitLab 11.10 and later: | 14.2 | 5.37 | | 14.3 | 5.38 | | 14.4 | 5.39 | +| 14.5 | 5.39 | +| 14.6 | 6.1 | + +- GitLab 14.5 remained on Mattermost 5.39 +- GitLab 14.6 updates to Mattermost 6.1 instead of 6.0 NOTE: When upgrading the Mattermost version, it is essential to check the @@ -346,8 +351,8 @@ When upgrading the Mattermost version, it is essential to check the for Mattermost to address any changes or migrations that need to be performed. Starting with GitLab 11.0, GitLab Mattermost can be upgraded through the regular Omnibus GitLab update process. When upgrading previous versions of -GitLab that process can only be used if Mattermost configuration settings have not been changed outside of GitLab (i.e., no changes to Mattermost's `config.json` -file have been made, either directly or via the Mattermost **System Console** which saves back changes to `config.json`.) +GitLab, the update process can only be used if Mattermost configuration settings have not been changed outside of GitLab. That is, no changes to Mattermost's `config.json` +file have been made - either directly or via the Mattermost **System Console**, which saves changes to `config.json`. If you are upgrading to at least GitLab 11.0 or have only configured Mattermost using `gitlab.rb`, you can upgrade GitLab using Omnibus and then run `gitlab-ctl reconfigure` to upgrade GitLab Mattermost to the latest version. @@ -364,6 +369,21 @@ If this is not the case, there are two options: For a complete list of upgrade notices and special considerations for older versions, see the [Mattermost documentation](https://docs.mattermost.com/administration/important-upgrade-notes.html). +## Upgrading GitLab Mattermost to 14.6 + +GitLab 14.6 includes Mattermost 6.1, and also includes the migrations for Mattermost 6.0. For information about upgrading and for ways to reduce the downtime of those migrations, read the [Important Upgrade Notes](https://docs.mattermost.com/administration/important-upgrade-notes.html) for both versions. + +NOTE: +The Mattermost upgrade notes refer to different impacts when used with a PostgreSQL versus a MySQL database. The GitLab Mattermost included with the GitLab Linux packages uses a PostgreSQL database. + +If you need to connect in the database to perform any manual migrations, run the following: + +```console +sudo gitlab-psql -d mattermost_production +``` + +You can then run the necessary queries that are described in the [Important Upgrade Notes](https://docs.mattermost.com/administration/important-upgrade-notes.html). + ## Upgrading GitLab Mattermost from versions prior to 11.0 With version 11.0, GitLab introduced breaking changes which affected Mattermost configuration. @@ -476,7 +496,27 @@ The following image is a sequence diagram for how GitLab works as an OAuth2 provider for Mattermost. You can use this to troubleshoot errors in getting the integration to work: - +```mermaid +sequenceDiagram + User->>Mattermost: GET https://mm.domain.com + Note over Mattermost, GitLab: Obtain access code + Mattermost->>GitLab: GET https://gitlab.domain.com/oauth/authorize + Note over User, GitLab: GitLab user signs in (if necessary) + Note over GitLab: GitLab verifies client_id matches an OAuth application + GitLab->>User: GitLab asks user to authorize Mattermost OAuth app + User->>GitLab: User selects 'Allow' + Note over GitLab: GitLab verifies redirect_uri matches list of valid URLs + GitLab->>User: 302 redirect: https://mm.domain.com/signup/gitlab/complete + User->>Mattermost: GET https://mm.domain.com/signup/gitlab/complete + Note over Mattermost, GitLab: Exchange access code for access token + Mattermost->>GitLab: POST http://gitlab.domain.com/oauth/token + GitLab->>GitLab: Doorkeeper::TokensController#35;create + GitLab->>Mattermost: Access token + Note over Mattermost, GitLab: Mattermost looks up GitLab user + Mattermost->>GitLab: GET https://gitlab.domain.com/api/v4/user + GitLab->>Mattermost: User details + Mattermost->>User: Mattermost/GitLab user ready +``` ## Troubleshooting the Mattermost CLI diff --git a/doc/integration/oauth2_generic.md b/doc/integration/oauth2_generic.md index cdc7e6db61c..3d44da8b4c8 100644 --- a/doc/integration/oauth2_generic.md +++ b/doc/integration/oauth2_generic.md @@ -4,31 +4,39 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Sign into GitLab with (almost) any OAuth2 provider **(FREE SELF)** +# Generic OAuth2 provider **(FREE SELF)** -The `omniauth-oauth2-generic` gem allows Single Sign-On between GitLab and your own OAuth2 provider -(or any OAuth2 provider compatible with this gem) +The `omniauth-oauth2-generic` gem allows single sign-on (SSO) between GitLab +and your OAuth2 provider (or any OAuth2 provider compatible with this gem). -This strategy is designed to allow configuration of the simple OmniAuth SSO process outlined below: +This strategy allows for the configuration of this OmniAuth SSO process: -1. Strategy directs client to your authorization URL (**configurable**), with specified ID and key -1. OAuth provider handles authentication of request, user, and (optionally) authorization to access user's profile -1. OAuth provider directs client back to GitLab where Strategy handles retrieval of access token -1. Strategy requests user information from a **configurable** "user profile" URL (using the access token) -1. Strategy parses user information from the response, using a **configurable** format -1. GitLab finds or creates the returned user and logs them in +1. Strategy directs the client to your authorization URL (**configurable**), with + the specified ID and key. +1. The OAuth2 provider handles authentication of the request, user, and (optionally) + authorization to access user's profile. +1. The OAuth2 provider directs the client back to GitLab where Strategy handles + the retrieval of the access token. +1. Strategy requests user information from a **configurable** "user profile" + URL (using the access token). +1. Strategy parses user information from the response, using a **configurable** + format. +1. GitLab finds or creates the returned user and signs them in. -## Limitations of this Strategy +## Limitations of this strategy -- It can only be used for Single Sign on, and doesn't provide any other access granted by any OAuth provider - (importing projects or users, etc) -- It only supports the Authorization Grant flow (most common for client-server applications, like GitLab) -- It is not able to fetch user information from more than one URL -- It has not been tested with user information formats other than JSON +- It can only be used for single sign-on, and doesn't provide any other access + granted by any OAuth2 provider (like importing projects or users). +- It supports only the Authorization Grant flow (most common for client-server + applications, like GitLab). +- It can't fetch user information from more than one URL. +- It hasn't been tested with user information formats, other than JSON. -## Configuration Instructions +## Configure the OAuth2 provider -1. Register your application in the OAuth2 provider you wish to authenticate with. +To configure the provider: + +1. Register your application in the OAuth2 provider you want to authenticate with. The redirect URI you provide when registering the application should be: @@ -36,13 +44,13 @@ This strategy is designed to allow configuration of the simple OmniAuth SSO proc http://your-gitlab.host.com/users/auth/oauth2_generic/callback ``` -1. You should now be able to get a Client ID and Client Secret. - Where this shows up differs for each provider. - This may also be called Application ID and Secret + You should now be able to get a Client ID and Client Secret. Where this + appears differs for each provider. This may also be called Application ID + and Secret. -1. On your GitLab server, open the configuration file. +1. On your GitLab server, open the appropriate configuration file. - For Omnibus package: + For Omnibus GitLab: ```shell sudo editor /etc/gitlab/gitlab.rb @@ -55,36 +63,37 @@ This strategy is designed to allow configuration of the simple OmniAuth SSO proc sudo -u git -H editor config/gitlab.yml ``` -1. See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings +1. See [Configure initial settings](omniauth.md#configure-initial-settings) for + initial settings. -1. Add the provider-specific configuration for your provider, for example: +1. Add the provider-specific configuration for your provider. For example: ```ruby gitlab_rails['omniauth_providers'] = [ - { 'name' => 'oauth2_generic', - 'label' => '<your_oauth2_label>', - 'app_id' => '<your_app_client_id>', - 'app_secret' => '<your_app_client_secret>', - 'args' => { + { + name: "oauth2_generic", + label: "Provider name", # optional label for login button, defaults to "Oauth2 Generic" + app_id: "<your_app_client_id>", + app_secret: "<your_app_client_secret>", + args: { client_options: { - 'site' => '<your_auth_server_url>', - 'user_info_url' => '/oauth2/v1/userinfo', - 'authorize_url' => '/oauth2/v1/authorize', - 'token_url' => '/oauth2/v1/token' - }, - user_response_structure: { - root_path: [], - id_path: ['sub'], - attributes: { - email: 'email', - name: 'name' - } - }, - authorize_params: { - scope: 'openid profile email' - }, - strategy_class: "OmniAuth::Strategies::OAuth2Generic" - } + site: "<your_auth_server_url>", + user_info_url: "/oauth2/v1/userinfo", + authorize_url: "/oauth2/v1/authorize", + token_url: "/oauth2/v1/token" + }, + user_response_structure: { + root_path: [], + id_path: ["sub"], + attributes: { + email: "email", + name: "name" + } + }, + authorize_params: { + scope: "openid profile email" + }, + strategy_class: "OmniAuth::Strategies::OAuth2Generic" } } ] @@ -92,11 +101,13 @@ This strategy is designed to allow configuration of the simple OmniAuth SSO proc For more information about these settings, see [the gem's README](https://gitlab.com/satorix/omniauth-oauth2-generic#gitlab-config-example). -1. Save the configuration file +1. Save the configuration file. -1. Restart GitLab for the changes to take effect +1. [Restart](../administration/restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. -On the sign in page there should now be a new button below the regular sign in form. -Click the button to begin your provider's authentication process. This directs -the browser to your OAuth2 Provider's authentication page. If everything goes well -the user is returned to your GitLab instance and is signed in. +On the sign-in page there should now be a new button below the regular sign-in +form. Select the button to begin your provider's authentication process. This +directs the browser to your OAuth2 provider's authentication page. If +everything goes well, you are returned to your GitLab instance and are +signed in. diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 5e96a1e7c65..dd51d823109 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -10,6 +10,8 @@ Users can sign in to GitLab by using their credentials from Twitter, GitHub, and [OmniAuth](https://rubygems.org/gems/omniauth/) is the Rack framework that GitLab uses to provide this authentication. + + If you configure OmniAuth, users can continue to sign in using other mechanisms, including standard GitLab authentication or LDAP (if configured). @@ -28,7 +30,7 @@ GitLab supports the following OmniAuth providers. | [Azure v1](azure.md) | `azure_oauth2` | | [Bitbucket Cloud](bitbucket.md) | `bitbucket` | | [CAS](cas.md) | `cas3` | -| [DingTalk](ding_talk.md) | `ding_talk` | +| [DingTalk](ding_talk.md) | `dingtalk` | | [Facebook](facebook.md) | `facebook` | | [Generic OAuth 2.0](oauth2_generic.md) | `oauth2_generic` | | [GitHub](github.md) | `github` | @@ -113,6 +115,12 @@ To change these settings: After configuring these settings, you can configure your chosen [provider](#supported-providers). +### Passwords for users created via OmniAuth + +The [Generated passwords for users created through integrated authentication](../security/passwords_for_integrated_authentication_methods.md) +guide provides an overview about how GitLab generates and sets passwords for +users created with OmniAuth. + ## Enable OmniAuth for an existing user If you're an existing user, after your GitLab account is @@ -129,6 +137,41 @@ provider like Twitter. You can now use your chosen OmniAuth provider to sign in to GitLab. +## Enable or disable sign-in with an OmniAuth provider without disabling import sources + +Administrators can enable or disable sign-in for some OmniAuth providers. + +NOTE: +By default, sign-in is enabled for all the OAuth providers configured in `config/gitlab.yml`. + +To enable or disable an OmniAuth provider: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings**. +1. Expand **Sign-in restrictions**. +1. In the **Enabled OAuth authentication sources** section, select or clear the checkbox for each provider you want to enable or disable. + +## Disable OmniAuth + +In GitLab 11.4 and later, OmniAuth is enabled by default. However, OmniAuth only works +if providers are configured and [enabled](#enable-or-disable-sign-in-with-an-omniauth-provider-without-disabling-import-sources). + +If OmniAuth providers are causing problems even when individually disabled, you +can disable the entire OmniAuth subsystem by modifying the configuration file: + +- **For Omnibus installations** + + ```ruby + gitlab_rails['omniauth_enabled'] = false + ``` + +- **For installations from source** + + ```yaml + omniauth: + enabled: false + ``` + ## Link existing users to OmniAuth users > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36664) in GitLab 13.4. @@ -228,41 +271,6 @@ let us know. We can't officially support every possible authentication mechanism available, but we'd like to at least help those with specific needs. -## Enable or disable sign-in with an OmniAuth provider without disabling import sources - -Administrators can enable or disable sign-in for some OmniAuth providers. - -NOTE: -By default, sign-in is enabled for all the OAuth providers configured in `config/gitlab.yml`. - -To enable or disable an OmniAuth provider: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Settings**. -1. Expand **Sign-in restrictions**. -1. In the **Enabled OAuth authentication sources** section, select or clear the checkbox for each provider you want to enable or disable. - -## Disable OmniAuth - -In GitLab 11.4 and later, OmniAuth is enabled by default. However, OmniAuth only works -if providers are configured and [enabled](#enable-or-disable-sign-in-with-an-omniauth-provider-without-disabling-import-sources). - -If OmniAuth providers are causing problems even when individually disabled, you -can disable the entire OmniAuth subsystem by modifying the configuration file: - -- **For Omnibus installations** - - ```ruby - gitlab_rails['omniauth_enabled'] = false - ``` - -- **For installations from source**: - - ```yaml - omniauth: - enabled: false - ``` - ## Keep OmniAuth user profiles up to date You can enable profile syncing from selected OmniAuth providers. You can sync @@ -344,12 +352,6 @@ one of the OmniAuth users is an administrator. You can also bypass automatic sign-in by browsing to `https://gitlab.example.com/users/sign_in?auto_sign_in=false`. -## Passwords for users created via OmniAuth - -The [Generated passwords for users created through integrated authentication](../security/passwords_for_integrated_authentication_methods.md) -guide provides an overview about how GitLab generates and sets passwords for -users created with OmniAuth. - ## Use a custom OmniAuth provider icon Most supported providers include a built-in icon for the rendered sign-in button. @@ -359,7 +361,7 @@ then override the icon in one of two ways: - **Provide a custom image path**: - 1. *If you are hosting the image outside of your GitLab server domain,* ensure + 1. If you are hosting the image outside of your GitLab server domain, ensure your [content security policies](https://docs.gitlab.com/omnibus/settings/configuration.html#content-security-policy) are configured to allow access to the image file. 1. Depending on your method of installing GitLab, add a custom `icon` parameter diff --git a/doc/integration/openid_connect_provider.md b/doc/integration/openid_connect_provider.md index dd65fb4822a..54d4a5b6bb7 100644 --- a/doc/integration/openid_connect_provider.md +++ b/doc/integration/openid_connect_provider.md @@ -33,6 +33,14 @@ Refer to the [OAuth guide](oauth_provider.md) for basic information on how to se applications in GitLab. To enable OIDC for an application, all you have to do is select the `openid` scope in the application settings. +## Settings discovery + +If your client allows importing OIDC settings from a discovery URL, you can use the following URL to automatically find the correct settings: + +```plaintext +https://gitlab.example.com/.well-known/openid-configuration +``` + ## Shared information The following user information is shared with clients: @@ -51,5 +59,8 @@ The following user information is shared with clients: | `picture` | `string` | URL for the user's GitLab avatar | `groups` | `array` | Paths for the groups the user is a member of, either directly or through an ancestor group. | `groups_direct` | `array` | Paths for the groups the user is a direct member of. +| `https://gitlab.org/claims/groups/owner` | `array` | Names of the groups the user is a direct member of with Owner role +| `https://gitlab.org/claims/groups/maintainer` | `array` | Names of the groups the user is a direct member of with Maintainer role +| `https://gitlab.org/claims/groups/developer` | `array` | Names of the groups the user is a direct member of with Developer role The claims `sub`, `sub_legacy`, `email`, `email_verified` and `groups_direct` are included in the ID token. All other claims are available from the `/oauth/userinfo` endpoint used by OIDC clients. diff --git a/doc/integration/recaptcha.md b/doc/integration/recaptcha.md index 60e2d70ce32..4963dea19a4 100644 --- a/doc/integration/recaptcha.md +++ b/doc/integration/recaptcha.md @@ -12,20 +12,21 @@ to confirm that a real user, not a bot, is attempting to create an account. ## Configuration -To use reCAPTCHA, first you must create a site and private key. +To use reCAPTCHA, first create a site and private key. 1. Go to the [Google reCAPTCHA page](https://www.google.com/recaptcha/admin). -1. Fill out the form necessary to obtain reCAPTCHA v2 keys. -1. Log in to your GitLab server, with administrator credentials. -1. Go to Reporting Applications Settings in the Admin Area (`admin/application_settings/reporting`). -1. Expand the **Spam and Anti-bot Protection** section. -1. Fill all reCAPTCHA fields with keys from previous steps. +1. To get reCAPTCHA v2 keys, fill in the form and select **Submit**. +1. Sign in to your GitLab server as an administrator. +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Reporting** (`admin/application_settings/reporting`). +1. Expand **Spam and Anti-bot Protection**. +1. In the reCAPTCHA fields, enter the keys you obtained in the previous steps. 1. Select the **Enable reCAPTCHA** checkbox. 1. To enable reCAPTCHA for logins via password, select the **Enable reCAPTCHA for login** checkbox. -1. Save the configuration. -1. Change the first line of the `#execute` method in `app/services/spam/spam_verdict_service.rb` - to `return CONDITIONAL_ALLOW` so that the spam check short-circuits and triggers the response to - return `recaptcha_html`. +1. Select **Save changes**. +1. To short-circuit the spam check and trigger the response to return `recaptcha_html`: + 1. Open `app/services/spam/spam_verdict_service.rb`. + 1. Change the first line of the `#execute` method to `return CONDITIONAL_ALLOW`. NOTE: Make sure you are viewing an issuable in a project that is public. If you're working with an issue, the issue is public. diff --git a/doc/integration/salesforce.md b/doc/integration/salesforce.md index 68daced3521..ebd936424d3 100644 --- a/doc/integration/salesforce.md +++ b/doc/integration/salesforce.md @@ -57,9 +57,10 @@ To get the credentials (a pair of Client ID and Client Secret), you must [create ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "salesforce", - "app_id" => "SALESFORCE_CLIENT_ID", - "app_secret" => "SALESFORCE_CLIENT_SECRET" + name: "salesforce", + # label: "Provider name", # optional label for login button, defaults to "Salesforce" + app_id: "SALESFORCE_CLIENT_ID", + app_secret: "SALESFORCE_CLIENT_SECRET" } ] ``` @@ -68,6 +69,7 @@ To get the credentials (a pair of Client ID and Client Secret), you must [create ```yaml - { name: 'salesforce', + # label: 'Provider name', # optional label for login button, defaults to "Salesforce" app_id: 'SALESFORCE_CLIENT_ID', app_secret: 'SALESFORCE_CLIENT_SECRET' } diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 47a35cf21a8..70d6932b9eb 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -98,15 +98,15 @@ as described in the section on [Security](#security). Otherwise, your users are ```ruby gitlab_rails['omniauth_providers'] = [ { - name: 'saml', + name: "saml", + label: "Provider name", # optional label for login button, defaults to "Saml" args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://login.example.com/idp', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' - }, - label: 'Provider name' # optional label for SAML login button, defaults to "Saml" + assertion_consumer_service_url: "https://gitlab.example.com/users/auth/saml/callback", + idp_cert_fingerprint: "43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8", + idp_sso_target_url: "https://login.example.com/idp", + issuer: "https://gitlab.example.com", + name_identifier_format: "urn:oasis:names:tc:SAML:2.0:nameid-format:persistent" + } } ] ``` @@ -118,14 +118,14 @@ as described in the section on [Security](#security). Otherwise, your users are providers: - { name: 'saml', + label: 'Provider name', # optional label for login button, defaults to "Saml" args: { assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', idp_sso_target_url: 'https://login.example.com/idp', issuer: 'https://gitlab.example.com', name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' - }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } } ``` @@ -163,6 +163,74 @@ On the sign in page there should now be a SAML button below the regular sign in Click the icon to begin the authentication process. If everything goes well the user is returned to GitLab and signed in. +### Use multiple SAML identity providers + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14361) in GitLab 14.6. + +You can configure GitLab to use multiple SAML identity providers if: + +- Each provider has a unique name set that matches a name set in `args`. +- The providers' names are: + - Used in OmniAuth configuration for properties based on the provider name. For example, `allowBypassTwoFactor`, `allowSingleSignOn`, and + `syncProfileFromProvider`. + - Used for association to each existing user as an additional identity. +- The `assertion_consumer_service_url` matches the provider name. +- The `strategy_class` is explicitly set because it cannot be inferred from provider name. + +Example multiple providers configuration for Omnibus GitLab: + +```ruby +gitlab_rails['omniauth_providers'] = [ + { + name: 'saml_1', + args: { + name: 'saml_1', # This is mandatory and must match the provider name + strategy_class: 'OmniAuth::Strategies::SAML' + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml_1/callback', # URL must match the name of the provider + ... # Put here all the required arguments similar to a single provider + }, + label: 'Provider 1' # Differentiate the two buttons and providers in the UI + }, + { + name: 'saml_2', + args: { + name: 'saml_2', # This is mandatory and must match the provider name + strategy_class: 'OmniAuth::Strategies::SAML' + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml_2/callback', # URL must match the name of the provider + ... # Put here all the required arguments similar to a single provider + }, + label: 'Provider 2' # Differentiate the two buttons and providers in the UI + } +] +``` + +Example providers configuration for installations from source: + +```yaml +omniauth: + providers: + - { + name: 'saml_1', + args: { + name: 'saml_1', # This is mandatory and must match the provider name + strategy_class: 'OmniAuth::Strategies::SAML', + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml_1/callback', # URL must match the name of the provider + ... # Put here all the required arguments similar to a single provider + }, + label: 'Provider 1' # Differentiate the two buttons and providers in the UI + } + - { + name: 'saml_2', + args: { + name: 'saml_2', # This is mandatory and must match the provider name + strategy_class: 'OmniAuth::Strategies::SAML', + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml_2/callback', # URL must match the name of the provider + ... # Put here all the required arguments similar to a single provider + }, + label: 'Provider 2' # Differentiate the two buttons and providers in the UI + } +``` + ### Notes on configuring your identity provider When configuring a SAML app on the IdP, you need at least: @@ -362,22 +430,21 @@ In addition to the changes in GitLab, make sure that your IdP is returning the ```ruby gitlab_rails['omniauth_providers'] = [ { - name: 'saml', + name: "saml", args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://login.example.com/idp', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', - upstream_two_factor_authn_contexts: - %w( - urn:oasis:names:tc:SAML:2.0:ac:classes:CertificateProtectedTransport - urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorOTPSMS - urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorIGTOKEN - ) - - }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + assertion_consumer_service_url: "https://gitlab.example.com/users/auth/saml/callback", + idp_cert_fingerprint: "43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8", + idp_sso_target_url: "https://login.example.com/idp", + issuer: "https://gitlab.example.com", + name_identifier_format: "urn:oasis:names:tc:SAML:2.0:nameid-format:persistent", + upstream_two_factor_authn_contexts: + %w( + urn:oasis:names:tc:SAML:2.0:ac:classes:CertificateProtectedTransport + urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorOTPSMS + urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorIGTOKEN + ) + }, + label: "Company Login" # optional label for SAML login button, defaults to "Saml" } ] ``` diff --git a/doc/integration/twitter.md b/doc/integration/twitter.md index 50ef04681f0..e1f67df76c3 100644 --- a/doc/integration/twitter.md +++ b/doc/integration/twitter.md @@ -62,9 +62,10 @@ Twitter. Twitter generates a client ID and secret key for you to use. ```ruby gitlab_rails['omniauth_providers'] = [ { - "name" => "twitter", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET" + name: "twitter", + # label: "Provider name", # optional label for login button, defaults to "Twitter" + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET" } ] ``` @@ -73,6 +74,7 @@ Twitter. Twitter generates a client ID and secret key for you to use. ```yaml - { name: 'twitter', + # label: 'Provider name', # optional label for login button, defaults to "Twitter" app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET' } ``` diff --git a/doc/integration/vault.md b/doc/integration/vault.md index 3bca3767785..9e738f8493d 100644 --- a/doc/integration/vault.md +++ b/doc/integration/vault.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/intro/index.md b/doc/intro/index.md deleted file mode 100644 index af50726e30c..00000000000 --- a/doc/intro/index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: '../topics/use_gitlab.md' -remove_date: '2021-12-08' ---- - -This document was moved to [another location](../topics/use_gitlab.md). - -<!-- This redirect file can be deleted after <2021-12-08>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/operations/error_tracking.md b/doc/operations/error_tracking.md index 70e3115a98e..68a0d492c5d 100644 --- a/doc/operations/error_tracking.md +++ b/doc/operations/error_tracking.md @@ -34,7 +34,7 @@ For error tracking to work, you need two pieces: ### Deploying Sentry -You can sign up to the cloud hosted [Sentry](https://sentry.io), deploy your own [on-premise instance](https://github.com/getsentry/onpremise/), or use GitLab to [install Sentry to a Kubernetes cluster](../user/clusters/applications.md#install-sentry-using-gitlab-cicd). To make this easier, we are [considering shipping Sentry with GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5343). +You can sign up to the cloud hosted [Sentry](https://sentry.io), deploy your own [on-premise instance](https://github.com/getsentry/onpremise/), or use GitLab to [install Sentry to a Kubernetes cluster](../user/clusters/applications.md#install-sentry-using-gitlab-cicd). ### Enabling Sentry diff --git a/doc/operations/feature_flags.md b/doc/operations/feature_flags.md index 2ef193b0f5d..49898d2e904 100644 --- a/doc/operations/feature_flags.md +++ b/doc/operations/feature_flags.md @@ -81,7 +81,7 @@ You can apply a feature flag strategy across multiple environments, without defi the strategy multiple times. GitLab Feature Flags use [Unleash](https://docs.getunleash.io/) as the feature flag -engine. In Unleash, there are [strategies](https://docs.getunleash.io/activation_strategy/) +engine. In Unleash, there are [strategies](https://docs.getunleash.io/user_guide/activation_strategy) for granular feature flag controls. GitLab Feature Flags can have multiple strategies, and the supported strategies are: @@ -96,8 +96,7 @@ and selecting **Edit** (**{pencil}**). ### All users -Enables the feature for all users. It uses the [`default`](https://docs.getunleash.io/activation_strategy/#default) -Unleash activation strategy. +Enables the feature for all users. It uses the Standard (`default`) Unleash activation [strategy](https://docs.getunleash.io/user_guide/activation_strategy#standard). ### Percent Rollout @@ -105,8 +104,7 @@ Unleash activation strategy. Enables the feature for a percentage of page views, with configurable consistency of behavior. This consistency is also known as stickiness. It uses the -[`flexibleRollout`](https://docs.getunleash.io/activation_strategy/#flexiblerollout) -Unleash activation strategy. +Gradual Rollout (`flexibleRollout`) Unleash activation [strategy](https://docs.getunleash.io/user_guide/activation_strategy#gradual-rollout). You can configure the consistency to be based on: @@ -134,7 +132,7 @@ Selecting **Random** provides inconsistent application behavior for individual u ### Percent of Users Enables the feature for a percentage of authenticated users. It uses the Unleash activation strategy -[`gradualRolloutUserId`](https://docs.getunleash.io/activation_strategy/#gradualrolloutuserid). +[`gradualRolloutUserId`](https://docs.getunleash.io/user_guide/activation_strategy#gradual-rollout). For example, set a value of 15% to enable the feature for 15% of authenticated users. @@ -156,8 +154,7 @@ ID for the feature to be enabled. See the [Ruby example](#ruby-application-examp > - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/34363) to be defined per environment in GitLab 12.6. Enables the feature for a list of target users. It is implemented -using the Unleash [`userWithId`](https://docs.getunleash.io/activation_strategy/#userwithid) -activation strategy. +using the Unleash UserIDs (`userWithId`) activation [strategy](https://docs.getunleash.io/user_guide/activation_strategy#userids). Enter user IDs as a comma-separated list of values (for example, `user@example.com, user2@example.com`, or `username1,username2,username3`, and so on). Note that @@ -187,8 +184,7 @@ To search for code references of a feature flag: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35930) in GitLab 13.1. Enables the feature for lists of users created [in the Feature Flags UI](#create-a-user-list), or with the [Feature Flag User List API](../api/feature_flag_user_lists.md). -Similar to [User IDs](#user-ids), it uses the Unleash [`userWithId`](https://docs.getunleash.io/activation_strategy/#userwithid) -activation strategy. +Similar to [User IDs](#user-ids), it uses the Unleash UsersIDs (`userWithId`) activation [strategy](https://docs.getunleash.io/user_guide/activation_strategy#userids). It's not possible to *disable* a feature for members of a user list, but you can achieve the same effect by enabling a feature for a user list that doesn't contain the excluded users. diff --git a/doc/operations/incident_management/integrations.md b/doc/operations/incident_management/integrations.md index 92f5a50b1c3..a8b455e05a0 100644 --- a/doc/operations/incident_management/integrations.md +++ b/doc/operations/incident_management/integrations.md @@ -65,11 +65,11 @@ and you can [customize the payload](#customize-the-alert-payload-outside-of-gitl 1. Toggle the **Active** alert setting. The **URL** and **Authorization Key** for the webhook configuration are available in the **View credentials** tab after you save the integration. You must also input the URL and Authorization Key in your external service. - 1. _(Optional)_ To map fields from your monitoring tool's alert to GitLab fields, enter a sample + 1. Optional. To map fields from your monitoring tool's alert to GitLab fields, enter a sample payload and click **Parse payload for custom mapping**. Valid JSON is required. If you update a sample payload, you must also remap the fields. - 1. _(Optional)_ If you provided a valid sample payload, select each value in + 1. Optional. If you provided a valid sample payload, select each value in **Payload alert key** to [map to a **GitLab alert key**](#map-fields-in-custom-alerts). 1. To save your integration, click **Save Integration**. If desired, you can send a test alert from your integration's **Send test alert** tab after the integration is created. @@ -173,7 +173,7 @@ curl --request POST \ The authorization key can be used as the `password`. The `username` is left blank: - username: `<blank>` -- pasword: authorization_key +- password: authorization_key ```shell curl --request POST \ diff --git a/doc/operations/metrics/dashboards/yaml_number_format.md b/doc/operations/metrics/dashboards/yaml_number_format.md index 3b6e10e647e..ce9e359a587 100644 --- a/doc/operations/metrics/dashboards/yaml_number_format.md +++ b/doc/operations/metrics/dashboards/yaml_number_format.md @@ -15,7 +15,7 @@ You can select units to format your charts by adding `format` to your ## Internationalization and localization -Currently, your [internationalization and localization options](https://en.wikipedia.org/wiki/Internationalization_and_localization) for number formatting are dependent on the system you are using i.e. your OS or browser. +Currently, your [internationalization and localization options](https://en.wikipedia.org/wiki/Internationalization_and_localization) for number formatting are dependent on the system you are using (that is, your OS or browser). ## Engineering Notation diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index 399ef40cb40..f05eaa677c1 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -48,20 +48,9 @@ To make sure you can easily run the most recent stable release, we are working hard to keep the update process simple and reliable. If you are unable to follow our monthly release cycle, there are a couple of -cases you need to consider. - -It is considered safe to jump between patch versions and minor versions within -one major version. For example, it is safe to: - -- Upgrade the *minor* version. For example: - - - `13.7.5` -> `13.10.5` - - `12.3.4` -> `12.10.11` - -- Upgrade the *patch* version. For example: - - - `13.0.4` -> `13.0.12` - - `12.10.1` -> `12.10.8` +cases you need to consider. Follow the +[upgrade paths guide](../update/index.md#upgrade-paths) to safely upgrade +between versions. NOTE: Version specific changes in Omnibus GitLab Linux packages can be found in [the Omnibus GitLab documentation](../update/package/index.md#version-specific-changes). diff --git a/doc/push_rules/push_rules.md b/doc/push_rules/push_rules.md index 37aeb335825..425275a0370 100644 --- a/doc/push_rules/push_rules.md +++ b/doc/push_rules/push_rules.md @@ -14,6 +14,10 @@ GitLab already offers [protected branches](../user/project/protected_branches.md cases when you need some specific rules. Some common scenarios: preventing Git tag removal, or enforcing a special format for commit messages. +INFO: +Get access to push rules and more with a +[free 30-day trial of GitLab Ultimate](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-push-rules-docs). + Push rules are [pre-receive Git hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) you can enable in a user-friendly interface. They are defined either: @@ -103,19 +107,19 @@ The following options are available: | Push rule | Description | |---------------------------------|-------------| | Removal of tags with `git push` | Forbid users to remove Git tags with `git push`. Tags can be deleted through the web UI. | -| Check whether the commit author is a GitLab user | Restrict commits to existing GitLab users (checked against their emails). | -| Reject unverified users | GitLab rejects any commit that was not committed by an authenticated user. | +| Check whether the commit author is a GitLab user | Restrict commits to existing GitLab users (checked against their emails). <sup>1</sup> | +| Reject unverified users | GitLab rejects any commit that was not committed by the same user as the user who pushed it, or where the committer's email address is not [confirmed](../security/user_email_confirmation.md). | | Check whether commit is signed through GPG | Reject commit when it is not signed through GPG. Read [signing commits with GPG](../user/project/repository/gpg_signed_commits/index.md). | | Prevent pushing secret files | GitLab rejects any files that are likely to contain secrets. See the [forbidden file names](#prevent-pushing-secrets-to-the-repository). | -| Require expression in commit messages | Only commit messages that match this regular expression are allowed to be pushed. Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | -| Reject expression in commit messages | Only commit messages that do not match this regular expression are allowed to be pushed. Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | -| Restrict by branch name | Only branch names that match this regular expression are allowed to be pushed. Leave empty to allow all branch names. | -| Restrict by commit author's email | Only commit author's email that match this regular expression are allowed to be pushed. Leave empty to allow any email. | -| Prohibited file names | Any committed filenames that match this regular expression and do not already exist in the repository are not allowed to be pushed. Leave empty to allow any filenames. See [common examples](#prohibited-file-names). | +| Require expression in commit messages | Only commit messages that match this regular expression are allowed to be pushed. <sup>2</sup> Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | +| Reject expression in commit messages | Only commit messages that do not match this regular expression are allowed to be pushed. <sup>2</sup> Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | +| Restrict by branch name | Only branch names that match this regular expression are allowed to be pushed. <sup>2</sup> Leave empty to allow all branch names. | +| Restrict by commit author's email | Only commit author's email that match this regular expression are allowed to be pushed. <sup>1</sup> <sup>2</sup> Leave empty to allow any email. | +| Prohibited file names | Any committed filenames that match this regular expression and do not already exist in the repository are not allowed to be pushed. <sup>2</sup> Leave empty to allow any filenames. See [common examples](#prohibited-file-names). | | Maximum file size | Pushes that contain added or updated files that exceed this file size (in MB) are rejected. Set to 0 to allow files of any size. Files tracked by Git LFS are exempted. | -NOTE: -GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in push rules, and you can test them at the [regex101 regex tester](https://regex101.com/). +1. Checks both the commit author and committer. +1. GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in push rules, and you can test them at the [regex101 regex tester](https://regex101.com/). ### Caveat to "Reject unsigned commits" push rule diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index cd541e7827f..676cc529c98 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -13,7 +13,7 @@ all repositories and all attachments. You can only restore a backup to **exactly the same version and type (CE/EE)** of GitLab on which it was created. The best way to migrate your repositories -from one server to another is through backup restore. +from one server to another is through a backup and restore. WARNING: GitLab doesn't back up items that aren't stored in the file system. If you're @@ -70,7 +70,7 @@ Backups do not include: - [Mattermost data](https://docs.mattermost.com/administration/config-settings.html#file-storage) WARNING: -GitLab does not back up any configuration files, SSL certificates, or system +GitLab does not back up any configuration files (`/etc/gitlab`), TLS keys and certificates, or system files. You are highly advised to read about [storing configuration files](#storing-configuration-files). WARNING: @@ -190,8 +190,9 @@ on a Kubernetes cluster, you must follow the [Back up the secrets](https://docs.gitlab.com/charts/backup-restore/backup.html#backup-the-secrets) instructions. -You may also want to back up any TLS keys and certificates, and your -[SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). +You may also want to back up any TLS keys and certificates (`/etc/gitlab/ssl`, `/etc/gitlab/trusted-certs`), and your +[SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079) +to avoid man-in-the-middle attack warnings if you have to perform a full machine restore. If you use Omnibus GitLab, review additional information to [backup your configuration](https://docs.gitlab.com/omnibus/settings/backups.html). @@ -903,7 +904,9 @@ If you fail to restore this encryption key file along with the application data backup, users with two-factor authentication enabled and GitLab Runner loses access to your GitLab server. -You may also want to restore any TLS keys, certificates, or +You may also want to restore your previous `/etc/gitlab/gitlab.rb` (for Omnibus packages) +or `/home/git/gitlab/config/gitlab.yml` (for installations from source) and +any TLS keys, certificates (`/etc/gitlab/ssl`, `/etc/gitlab/trusted-certs`), or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). Starting with GitLab 12.9, if an untarred backup (like the ones made with @@ -1419,6 +1422,13 @@ after which users must reactivate 2FA. DELETE FROM ci_variables; ``` +1. If you know the specific group or project from which you wish to delete variables, you can include a `WHERE` statement to specify that in your `DELETE`: + + ```sql + DELETE FROM ci_group_variables WHERE group_id = <GROUPID>; + DELETE FROM ci_variables WHERE project_id = <PROJECTID>; + ``` + You may need to reconfigure or restart GitLab for the changes to take effect. #### Reset runner registration tokens diff --git a/doc/raketasks/index.md b/doc/raketasks/index.md index 80aa52ed5a4..6227731e807 100644 --- a/doc/raketasks/index.md +++ b/doc/raketasks/index.md @@ -45,6 +45,7 @@ The following Rake tasks are available for use with GitLab: | [SMTP maintenance](../administration/raketasks/smtp.md) | SMTP-related tasks. | | [SPDX license list import](spdx.md) | Import a local copy of the [SPDX license list](https://spdx.org/licenses/) for matching [License Compliance policies](../user/compliance/license_compliance/index.md). | | [Repository storage](../administration/raketasks/storage.md) | List and migrate existing projects and attachments from legacy storage to hashed storage. | +| [Reset user passwords](../security/reset_user_password.md#use-a-rake-task) | Reset user passwords using Rake. | | [Uploads migrate](../administration/raketasks/uploads/migrate.md) | Migrate uploads between local storage and object storage. | | [Uploads sanitize](../administration/raketasks/uploads/sanitize.md) | Remove EXIF data from images uploaded to earlier versions of GitLab. | | [Service Data](../administration/troubleshooting/gitlab_rails_cheat_sheet.md#generate-service-ping) | Generate and troubleshoot [Service Ping](../development/service_ping/index.md). | diff --git a/doc/raketasks/user_management.md b/doc/raketasks/user_management.md index f63c35ab475..09e1b626c0a 100644 --- a/doc/raketasks/user_management.md +++ b/doc/raketasks/user_management.md @@ -176,3 +176,7 @@ cp config/secrets.yml.bak config/secrets.yml sudo /etc/init.d/gitlab start ``` + +## Related topics + +- [Reset a user's password](../security/reset_user_password.md#use-a-rake-task). diff --git a/doc/security/asset_proxy.md b/doc/security/asset_proxy.md index abeb5c401da..6c3bce939df 100644 --- a/doc/security/asset_proxy.md +++ b/doc/security/asset_proxy.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Proxying assets **(FREE SELF)** A possible security concern when managing a public-facing GitLab instance is -the ability to steal a users IP address by referencing images in issues and comments. +the ability to steal a user's IP address by referencing images in issues and comments. For example, adding `` to an issue description causes the image to be loaded from the external diff --git a/doc/security/reset_user_password.md b/doc/security/reset_user_password.md index 8b89200e1a7..a61660f6a2f 100644 --- a/doc/security/reset_user_password.md +++ b/doc/security/reset_user_password.md @@ -5,121 +5,120 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# How to reset user password **(FREE SELF)** +# Reset a user's password **(FREE SELF)** -There are a few ways to reset the password of a user. +You can reset user passwords by using a Rake task, a Rails console, or the +[Users API](../api/users.md#user-modification). -## Rake Task +## Prerequisites + +To reset a user password, you must be an administrator of a self-managed GitLab instance. + +## Use a Rake task > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52347) in GitLab 13.9. -GitLab provides a Rake Task to reset passwords of users using their usernames, -which can be invoked by the following command: +Use the following Rake task to reset a user's password: -```shell -sudo gitlab-rake "gitlab:password:reset" -``` +- **For Omnibus installations** -GitLab asks for a username, a password, and a password confirmation. Upon giving -proper values for them, the password of the specified user is updated. + ```shell + sudo gitlab-rake "gitlab:password:reset" + ``` -The Rake task also takes the username as an argument, as shown in the example -below: +- **For installations from source** -```shell -sudo gitlab-rake "gitlab:password:reset[johndoe]" -``` + ```shell + bundle exec rake "gitlab:password:reset" + ``` -NOTE: -To reset the default admin password, run this Rake task with the username -`root`, which is the default username of that administrator account. +GitLab requests a username, a password, and confirmation of the password. When complete, the user's password is updated. -## Rails console +The Rake task can take a username as an argument. For example, to reset the password for the user with username +`sidneyjones`: -The Rake task is capable of finding users via their usernames. However, if only -user ID or email ID of the user is known, Rails console can be used to find user -using user ID and then change password of the user manually. +- **For Omnibus installations** -1. [Start a Rails console](../administration/operations/rails_console.md) + ```shell + sudo gitlab-rake "gitlab:password:reset[sidneyjones]" + ``` -1. Find the user either by username, user ID or email ID: +- **For installations from source** - ```ruby - user = User.find_by_username 'exampleuser' + ```shell + bundle exec rake "gitlab:password:reset[sidneyjones]" + ``` - #or +## Use a Rails console - user = User.find(123) +If you know the username, user ID, or email address, you can use the Rails console to reset their password: - #or +1. Open a [Rails console](../administration/operations/rails_console.md). +1. Find the user: - user = User.find_by(email: 'user@example.com') - ``` + - By username: -1. Reset the password + ```ruby + user = User.find_by_username 'exampleuser' + ``` - ```ruby - user.password = 'secret_pass' - user.password_confirmation = 'secret_pass' - ``` + - By user ID: -1. When using this method instead of the [Users API](../api/users.md#user-modification), - GitLab sends an email to the user stating that the user changed their - password. If the password was changed by an administrator, execute the - following command to notify the user by email: + ```ruby + user = User.find(123) + ``` - ```ruby - user.send_only_admin_changed_your_password_notification! + - By email address: + + ```ruby + user = User.find_by(email: 'user@example.com') + ``` + +1. Reset the password: + + ```ruby + user.password = 'secret_pass' + user.password_confirmation = 'secret_pass' ``` +1. Optional. Notify the user that an administrator changed their password: + + ```ruby + user.send_only_admin_changed_your_password_notification! + ``` + 1. Save the changes: ```ruby user.save! ``` -1. Exit the console, and then try to sign in with your new password. +1. Exit the console: + + ```ruby + exit + ``` -NOTE: -You can also reset passwords by using the [Users API](../api/users.md#user-modification). +## Reset the root password -## Password reset does not appear to work +To reset the root password, follow the steps listed previously. -If you can't sign on with the new password, it might be because of the [reconfirmation feature](../user/upgrade_email_bypass.md). +- If the root account name hasn't changed, use the username `root`. +- If the root account name has changed and you don't know the new username, + you might be able to use a Rails console with user ID `1`. In almost all + cases, the first user is the default administrator account. -Try fixing this on the rails console. For example, if your new `root` password isn't working: +## Troubleshooting -1. [Start a Rails console](../administration/operations/rails_console.md). +If the new password doesn't work, it might be [an email confirmation issue](../user/upgrade_email_bypass.md). You can +attempt to fix this issue in a Rails console. For example, if a new `root` password isn't working: -1. Find the user and skip reconfirmation, using any of the methods above: +1. Start a [Rails console](../administration/operations/rails_console.md). +1. Find the user and skip reconfirmation: ```ruby user = User.find(1) user.skip_reconfirmation! ``` -1. Try to sign in again. - -## Reset your root password - -The previously described steps can also be used to reset the root password. - -In normal installations where the username of root account hasn't been changed -manually, the Rake task can be used with username `root` to reset the root -password. - -If the username was changed to something else and has been forgotten, one -possible way is to reset the password using Rails console with user ID `1` (in -almost all the cases, the first user is the default administrator account). - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +1. Attempt to sign in again. diff --git a/doc/security/webhooks.md b/doc/security/webhooks.md index 89dd4f8e5fc..47ef90cbe55 100644 --- a/doc/security/webhooks.md +++ b/doc/security/webhooks.md @@ -26,11 +26,11 @@ sent. Webhook requests are made by the GitLab server itself and use a single (optional) secret token per hook for authorization (instead of a user or -repository-specific token). As a result, these may have broader access than -intended to everything running on the server hosting the webhook (which -may include the GitLab server or API itself, e.g., `http://localhost:123`). +repository-specific token). As a result, these requests may have broader access than +intended, including access to everything running on the server hosting the webhook. This +may include the GitLab server or API itself (for example, `http://localhost:123`). Depending on the called webhook, this may also result in network access -to other servers within that webhook server's local network (e.g., +to other servers within that webhook server's local network (for example, `http://192.168.1.12:345`), even if these services are otherwise protected and inaccessible from the outside world. diff --git a/doc/subscriptions/bronze_starter.md b/doc/subscriptions/bronze_starter.md index 78a9e324ada..3a58dd84614 100644 --- a/doc/subscriptions/bronze_starter.md +++ b/doc/subscriptions/bronze_starter.md @@ -28,7 +28,7 @@ the tiers are no longer mentioned in GitLab documentation: - [Creating group memberships via CN](../user/group/index.md#create-group-links-via-cn) - [Group push rules](../user/group/index.md#group-push-rules) - [Managing group memberships via LDAP](../user/group/index.md#manage-group-memberships-via-ldap) - - [Member locking](../user/group/index.md#prevent-members-from-being-added-to-a-group) + - [Member locking](../user/group/index.md#prevent-members-from-being-added-to-projects-in-a-group) - [Overriding user permissions](../user/group/index.md#override-user-permissions) - [User contribution analytics](../user/group/contribution_analytics/index.md) - [Kerberos integration](../integration/kerberos.md) @@ -39,7 +39,7 @@ the tiers are no longer mentioned in GitLab documentation: - Issues: - [Multiple assignees for issues](../user/project/issues/multiple_assignees_for_issues.md) - [Issue weights](../user/project/issues/issue_weight.md) - - [Issue histories](../user/project/issues/issue_data_and_actions.md#issue-history) contain changes to issue description + - Issue histories contain [changes to issue description](../user/discussions/index.md#view-description-change-history) - [Adding an issue to an iteration](../user/project/issues/managing_issues.md#add-an-issue-to-an-iteration) - [Iterations](../user/group/iterations/index.md) - [Kerberos integration](../integration/kerberos.md) diff --git a/doc/subscriptions/gitlab_com/index.md b/doc/subscriptions/gitlab_com/index.md index a26feb6d97e..e174a144cfc 100644 --- a/doc/subscriptions/gitlab_com/index.md +++ b/doc/subscriptions/gitlab_com/index.md @@ -317,39 +317,65 @@ main quota. You can find pricing for additional CI/CD minutes on the - Are only used after the shared quota included in your subscription runs out. - Roll over month to month. -To purchase additional minutes for your group on GitLab SaaS: - -1. From your group, go to **Settings > Usage Quotas**. -1. Select **Buy additional minutes** and GitLab directs you to the Customers Portal. -1. Locate the subscription card that's linked to your group on GitLab SaaS, click **Buy more CI minutes**, and complete the details about the transaction. -1. Once we have processed your payment, the extra CI minutes are synced to your group namespace. -1. To confirm the available CI minutes, go to your group, then **Settings > Usage Quotas**. - - The **Additional minutes** displayed now includes the purchased additional CI minutes, plus any minutes rolled over from last month. - To purchase additional minutes for your personal namespace: 1. In the top-right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Usage Quotas**. 1. Select **Buy additional minutes** and GitLab redirects you to the Customers Portal. -1. Locate the subscription card that's linked to your personal namespace on GitLab SaaS, click **Buy more CI minutes**, and complete the details about the transaction. Once we have processed your payment, the extra CI minutes are synced to your personal namespace. -1. To confirm the available CI minutes for your personal projects, go to the **Usage Quotas** settings again. +1. Locate the subscription card that's linked to your personal namespace on GitLab SaaS, click **Buy more CI minutes**, and complete the details about the transaction. + +After we process your payment, the extra CI minutes are synced to your group +namespace. + +To confirm the available CI minutes for your personal projects, go to the **Usage Quotas** settings again. + +The **Additional minutes** displayed now includes the purchased additional CI +minutes, plus any minutes rolled over from last month. + +Be aware that: + +- Extra CI minutes assigned to one group cannot be transferred to a different + group. +- If you have used more minutes than your default quota, those minutes are + deducted from your Additional Minutes quota immediately after your purchase of + additional minutes. + +### Purchase additional CI minutes on GitLab SaaS + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6574) in GitLab 14.5. + +If you're using GitLab SaaS, you can purchase additional CI minutes so your +pipelines aren't blocked after you have used all your CI minutes from your +main quota. You can find pricing for additional CI/CD minutes on the +[GitLab Pricing page](https://about.gitlab.com/pricing/). Additional minutes: + +- Are only used after the shared quota included in your subscription runs out. +- Roll over month to month. - The **Additional minutes** displayed now includes the purchased additional CI minutes, plus any minutes rolled over from last month. +To purchase additional minutes for your group on GitLab SaaS: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > Usage Quotas**. +1. Select **Buy additional minutes**. +1. Complete the details about the transaction. + +After we process your payment, the extra CI minutes are synced to your group +namespace. + +To confirm the available CI minutes, go to your group, and then select +**Settings > Usage Quotas**. + +The **Additional minutes** displayed now includes the purchased additional CI +minutes, plus any minutes rolled over from last month. Be aware that: -- If you have purchased extra CI minutes before the purchase of a paid plan, - we calculate a pro-rated charge for your paid plan. That means you may - be charged for less than one year because your subscription was previously - created with the extra CI minutes. -- After the extra CI minutes have been assigned to a Group, they can't be transferred - to a different Group by themselves, but they will transfer along with a subscription when - changing the linked namespace for the subscription. -- If you have used more minutes than your default quota, these minutes will - be deducted from your Additional Minutes quota immediately after your purchase of additional - minutes. +- Extra CI minutes assigned to one group cannot be transferred to a different + group. +- If you have used more minutes than your default quota, those minutes are + deducted from your Additional Minutes quota immediately after your purchase of + additional minutes. ## Storage subscription @@ -363,7 +389,9 @@ locked. Projects can only be unlocked by purchasing more storage subscription un ### Purchase more storage -To purchase more storage for either a personal or group namespace: +You can purchase storage for your personal or group namespace. + +#### For your personal namespace 1. Sign in to GitLab SaaS. 1. From either your personal homepage or the group's page, go to **Settings > Usage Quotas**. @@ -383,6 +411,32 @@ To purchase more storage for either a personal or group namespace: The **Purchased storage available** total is incremented by the amount purchased. All locked projects are unlocked and their excess usage is deducted from the additional storage. +#### For your group namespace + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5789) in GitLab 14.6. + +If you're using GitLab SaaS, you can purchase additional storage so your +pipelines aren't blocked after you have used all your storage from your +main quota. You can find pricing for additional storage on the +[GitLab Pricing page](https://about.gitlab.com/pricing/). + +To purchase additional storage for your group on GitLab SaaS: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > Usage Quotas**. +1. Select **Storage** tab. +1. Select **Purchase more storage**. +1. Complete the details. + +After your payment is processed, the extra storage is available for your group +namespace. + +To confirm the available storage, go to your group, and then select +**Settings > Usage Quotas** and select the **Storage** tab. + +The **Purchased storage available** total is incremented by the amount purchased. All locked +projects are unlocked and their excess usage is deducted from the additional storage. + ## Contact Support Learn more about: diff --git a/doc/subscriptions/index.md b/doc/subscriptions/index.md index b6aa2d09770..2cf3b9f7074 100644 --- a/doc/subscriptions/index.md +++ b/doc/subscriptions/index.md @@ -7,6 +7,11 @@ type: index, reference # GitLab subscription **(PREMIUM)** +INFO: +Get advanced search and more with +[a trial of GitLab Ultimate](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=u-subscription-docs). +Free for 30 days. + GitLab offers tiers of features. Your subscription determines which tier you have access to. Subscriptions are valid for 12 months. @@ -149,7 +154,9 @@ To change the namespace linked to a subscription: [linked](#change-the-linked-account) GitLab SaaS account. 1. Navigate to the **Manage Purchases** page. 1. Select **Change linked namespace**. -1. Select the desired group from the **This subscription is for** dropdown. +1. Select the desired group from the **This subscription is for** dropdown. For a group to appear + here, you must have the Owner [role](../user/permissions.md) + for that group. 1. Select **Proceed to checkout**. Subscription charges are calculated based on the total number of users in a group, including its subgroups and nested projects. If the total number of users exceeds the number of seats in your subscription, your account is charged for the additional users. diff --git a/doc/subscriptions/self_managed/index.md b/doc/subscriptions/self_managed/index.md index aee18e3d763..94180da2bbd 100644 --- a/doc/subscriptions/self_managed/index.md +++ b/doc/subscriptions/self_managed/index.md @@ -283,6 +283,30 @@ It also displays the following important statistics: | Maximum users | The highest number of billable users on your system during the term of the loaded license. | | Users over license | Calculated as `Maximum users` - `Users in License` for the current license term. This number incurs a retroactive charge that needs to be paid for at renewal. | +## Export your license usage + +> Introduced in GitLab 14.6. + +If you are an administrator, you can export your license usage into a CSV: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Subscription**. +1. In the top right, select **Export license usage file**. + +This file contains all the information GitLab needs to manually process quarterly reconciliations or renewals. If your instance is firewalled or air-gapped, you can provide GitLab with this information. + +The **License Usage** CSV includes the following details: + +- License key +- Email +- License start date +- License end date +- Company +- Generated at (the timestamp for when the file was exported) +- Table of historical user counts for each day in the period: + - Date the count was recorded + - Active user count + ## Renew your subscription To renew your subscription, diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md index dcab56a0d0f..72106a00191 100644 --- a/doc/system_hooks/system_hooks.md +++ b/doc/system_hooks/system_hooks.md @@ -44,6 +44,10 @@ denied access. System hooks can be used, for example, for logging or changing information in an LDAP server. +In addition to these default events, you can enable triggers for other events, +such as push events, and disable the `repository_update` event +when you create a system hook. + NOTE: We follow the same structure and deprecations as [Webhooks](../user/project/integrations/webhooks.md) for Push and Tag events, but we never display commits. @@ -55,7 +59,7 @@ To create a system hook: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **System Hooks**. 1. Provide the **URL** and **Secret Token**. -1. Select the checkbox next to each **Trigger** you want to enable. +1. Select the checkbox next to each optional **Trigger** you want to enable. 1. Select **Enable SSL verification**, if desired. 1. Click **Add system hook**. diff --git a/doc/topics/autodevops/customize.md b/doc/topics/autodevops/customize.md index 906fea2e6ad..925f657c099 100644 --- a/doc/topics/autodevops/customize.md +++ b/doc/topics/autodevops/customize.md @@ -103,10 +103,14 @@ You can override this behavior by defining specific variables: | Image Path | `$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG` for branch pipelines. `$CI_REGISTRY_IMAGE` for tag pipelines. | `$CI_APPLICATION_REPOSITORY` | | Image Tag | `$CI_COMMIT_SHA` for branch pipelines. `$CI_COMMIT_TAG` for tag pipelines. | `$CI_APPLICATION_TAG` | -These variables also affect Auto Build. If you don't want to build and push an image to +These variables also affect Auto Build and Auto Container Scanning. If you don't want to build and push an image to `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG`, consider including only `Jobs/Deploy.gitlab-ci.yml`, or [disabling the `build` jobs](#disable-jobs). +If you use Auto Container Scanning and set a value for `$CI_APPLICATION_REPOSITORY`, then you should +also update `$CS_DEFAULT_BRANCH_IMAGE`. See [Setting the default branch image](../../user/application_security/container_scanning/index.md#setting-the-default-branch-image) +for more details. + Here is an example setup in your `.gitlab-ci.yml`: ```yaml diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 9340f89c502..585d484d3be 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Auto DevOps **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/38366) in GitLab 11.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/38366) in GitLab 11.0. +> - Support for the GitLab Agent was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299350) in GitLab 14.5. GitLab Auto DevOps is a collection of pre-configured features and integrations that work together to support your software delivery process. diff --git a/doc/topics/autodevops/multiple_clusters_auto_devops.md b/doc/topics/autodevops/multiple_clusters_auto_devops.md index c6df5ac9e02..8156ae7c7ac 100644 --- a/doc/topics/autodevops/multiple_clusters_auto_devops.md +++ b/doc/topics/autodevops/multiple_clusters_auto_devops.md @@ -27,7 +27,7 @@ The following table is an example of how to configure the three different cluste | Cluster name | Cluster environment scope | `KUBE_INGRESS_BASE_DOMAIN` variable value | Variable environment scope | Notes | |--------------|---------------------------|-------------------------------------------|----------------------------|---| | review | `review/*` | `review.example.com` | `review/*` | The review cluster which runs all [Review Apps](../../ci/review_apps/index.md). `*` is a wildcard, used by every environment name starting with `review/`. | -| staging | `staging` | `staging.example.com` | `staging` | (Optional) The staging cluster which runs the deployments of the staging environments. You must [enable it first](customize.md#deploy-policy-for-staging-and-production-environments). | +| staging | `staging` | `staging.example.com` | `staging` | Optional. The staging cluster that runs the deployments of the staging environments. You must [enable it first](customize.md#deploy-policy-for-staging-and-production-environments). | | production | `production` | `example.com` | `production` | The production cluster which runs the production environment deployments. You can use [incremental rollouts](customize.md#incremental-rollout-to-production). | To add a different cluster for each environment: diff --git a/doc/topics/autodevops/stages.md b/doc/topics/autodevops/stages.md index 039c369ce9b..ca004662395 100644 --- a/doc/topics/autodevops/stages.md +++ b/doc/topics/autodevops/stages.md @@ -635,7 +635,7 @@ ciliumNetworkPolicy: #### Enabling Alerts You can also enable alerts. Network policies with alerts are considered only if -[GitLab Kubernetes Agent](../../user/clusters/agent/index.md) +[Agent](../../user/clusters/agent/index.md) has been integrated. You can enable alerts as follows: diff --git a/doc/topics/autodevops/upgrading_postgresql.md b/doc/topics/autodevops/upgrading_postgresql.md index e526ba55465..258195bb89f 100644 --- a/doc/topics/autodevops/upgrading_postgresql.md +++ b/doc/topics/autodevops/upgrading_postgresql.md @@ -177,7 +177,7 @@ NOTE: You can also [scope](../../ci/environments/index.md#scope-environments-with-specs) the `AUTO_DEVOPS_POSTGRES_CHANNEL`, `AUTO_DEVOPS_POSTGRES_DELETE_V1` and -`POSTGRES_VERSION` variables to specific environments, e.g. `staging`. +`POSTGRES_VERSION` variables to specific environments, for example, `staging`. 1. Set `AUTO_DEVOPS_POSTGRES_CHANNEL` to `2`. This opts into using the newer 8.2.1-based PostgreSQL, and removes the older 0.7.1-based diff --git a/doc/topics/cron/index.md b/doc/topics/cron/index.md index f7a22bef07c..de83ec8b51b 100644 --- a/doc/topics/cron/index.md +++ b/doc/topics/cron/index.md @@ -9,7 +9,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w Cron syntax is used to schedule when jobs should run. You may need to use a cron syntax string to -[trigger nightly pipelines](../../ci/triggers/index.md#using-cron-to-trigger-nightly-pipelines), create a [pipeline schedule](../../api/pipeline_schedules.md#create-a-new-pipeline-schedule), or to prevent unintentional releases by setting a [deploy freeze](../../user/project/releases/index.md#prevent-unintentional-releases-by-setting-a-deploy-freeze). @@ -65,5 +64,7 @@ More examples of how to write a cron schedule can be found at ## How GitLab parses cron syntax strings GitLab uses [`fugit`](https://github.com/floraison/fugit) to parse cron syntax -strings on the server and [cron-validate](https://github.com/Airfooox/cron-validate) -to validate cron syntax in the browser. +strings on the server and [cron-validator](https://github.com/TheCloudConnectors/cron-validator) +to validate cron syntax in the browser. GitLab uses +[`cRonstrue`](https://github.com/bradymholt/cRonstrue) to convert cron to human-readable strings +in the browser. diff --git a/doc/topics/git/cherry_picking.md b/doc/topics/git/cherry_picking.md index 64d1914019d..98458133937 100644 --- a/doc/topics/git/cherry_picking.md +++ b/doc/topics/git/cherry_picking.md @@ -74,7 +74,7 @@ into a different branch (`stable`): git cherry-pick <SHA> ``` -## Related links +## Related topics - Cherry-pick commits with [the Commits API](../../api/commits.md#cherry-pick-a-commit) - Git documentation [for cherry-picks](https://git-scm.com/docs/git-cherry-pick) diff --git a/doc/topics/git/lfs/index.md b/doc/topics/git/lfs/index.md index b21237203d7..0fe38e25df5 100644 --- a/doc/topics/git/lfs/index.md +++ b/doc/topics/git/lfs/index.md @@ -161,7 +161,7 @@ To resolve the problem, migrate the affected file (or files) and push back to th git push ``` -1. (Optional) Clean up your `.git` folder: +1. Optional. Clean up your `.git` folder: ```shell git reflog expire --expire-unreachable=now --all diff --git a/doc/topics/plan_and_track.md b/doc/topics/plan_and_track.md index abede62c00b..13cb3f8923b 100644 --- a/doc/topics/plan_and_track.md +++ b/doc/topics/plan_and_track.md @@ -11,6 +11,12 @@ with milestones and track your team's time. Learn how to save time with quick actions, see how GitLab renders Markdown text, and learn how to use Git to interact with GitLab. +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For a thorough demo of Plan features, see +[Multi-team planning with GitLab Ultimate](https://www.youtube.com/watch?v=KmASFwSap7c). +In this video, Gabe describes a use case of a multi-team organization that uses GitLab +with [Scaled Agile Framework (SAFe)](https://about.gitlab.com/solutions/agile-delivery/scaled-agile/). + ## Basic workflow features Planning features everyone needs to use day-to-day. @@ -21,11 +27,14 @@ Planning features everyone needs to use day-to-day. - [To-Do lists](../user/todos.md) - [Using Git](../topics/git/index.md) -## Team Planning +## Team planning Get work done as a team. - [Comments and threads](../user/discussions/index.md) +- [Customer relations (CRM)](../user/crm/index.md) + - [Contacts](../user/crm/index.md#contacts) + - [Organizations](../user/crm/index.md#organizations) - [Issues](../user/project/issues/index.md) - [Iterations](../user/group/iterations/index.md) - [Labels](../user/project/labels.md) @@ -35,9 +44,13 @@ Get work done as a team. - [Time tracking](../user/project/time_tracking.md) - [Wikis](../user/project/wiki/index.md) -## Portfolio Management +## Portfolio management Align your work across teams. - [Epics](../user/group/epics/index.md) + - [Multi-level epics](../user/group/epics/manage_epics.md#multi-level-child-epics) + - [Epic boards](../user/group/epics/epic_boards.md) + - [View health status](../user/project/issues/managing_issues.md#health-status) - [Roadmaps](../user/group/roadmap/index.md) +- [Planning hierarchies](../user/group/planning_hierarchy/index.md) diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md index 42bd41ed74b..8bf14bc2dce 100644 --- a/doc/update/deprecations.md +++ b/doc/update/deprecations.md @@ -45,6 +45,7 @@ The Task Runner pod is used to execute periodic housekeeping tasks within the Gi This will result in the rename of the sub-chart: `gitlab/task-runner` to `gitlab/toolbox`. Resulting pods will be named along the lines of `{{ .Release.Name }}-toolbox`, which will often be `gitlab-toolbox`. They will be locatable with the label `app=toolbox`. Announced: 2021-08-22 +Planned removal: 2021-11-22 ## 14.6 @@ -53,6 +54,7 @@ Announced: 2021-08-22 The [release-cli](https://gitlab.com/gitlab-org/release-cli) will be released as a [generic package](https://gitlab.com/gitlab-org/release-cli/-/packages) starting in GitLab 14.2. We will continue to deploy it as a binary to S3 until GitLab 14.5 and stop distributing it in S3 in GitLab 14.6. Announced: 2021-08-22 +Planned removal: 2021-12-22 ## 14.8 @@ -63,9 +65,19 @@ Distribution support and security updates for openSUSE Leap 15.2 are [ending Dec Starting in 14.5 we are providing packages for openSUSE Leap 15.3, and will stop providing packages for openSUSE Leap 15.2 in the 14.8 milestone. Announced: 2021-11-22 +Planned removal: 2022-02-22 ## 15.0 +### API: `stale` status returned instead of `offline` or `not_connected` + +A breaking change will occur for the Runner [API](https://docs.gitlab.com/ee/api/runners.html#runners-api) endpoints in 15.0. + +Instead of the GitLab Runner API endpoints returning `offline` and `not_connected` for runners that have not contacted the GitLab instance in the past three months, the API endpoints will return the `stale` value, which was introduced in 14.6. + +Announced: 2021-12-22 +Planned removal: 2022-05-22 + ### Audit events for repository push events Audit events for [repository events](https://docs.gitlab.com/ee/administration/audit_events.html#repository-push) are now deprecated and will be removed in GitLab 15.0. @@ -75,6 +87,14 @@ feature flag. Enabling them can cause too many events to be generated which can dramatically slow down GitLab instances. For this reason, they are being removed. Announced: 2021-09-22 +Planned removal: 2022-05-22 + +### CI/CD job name length limit + +In GitLab 15.0 we are going to limit the number of characters in CI/CD job names to 255. Any pipeline with job names that exceed the 255 character limit will stop working after the 15.0 release. + +Announced: 2021-12-22 +Planned removal: 2022-05-22 ### Certificate-based integration with Kubernetes @@ -89,12 +109,14 @@ For a more robust, secure, forthcoming, and reliable integration with Kubernetes [Kubernetes Agent](https://docs.gitlab.com/ee/user/clusters/agent/) to connect Kubernetes clusters with GitLab. Announced: 2021-11-15 +Planned removal: 2022-05-22 ### Converting an instance (shared) runner to a project (specific) runner is deprecated In GitLab 15.0, we will remove the feature that enables you to convert an instance (shared) runner to a project (specific) runner. Users who need to add a runner to only a particular project can register a runner to the project directly. Announced: 2021-11-22 +Planned removal: 2022-05-22 ### Deprecate `Versions` on base `PackageType` @@ -103,12 +125,51 @@ As part of the work to create a [Package Registry GraphQL API](https://gitlab.co In milestone 15.0, we will completely remove `Version` from `PackageType`. Announced: 2021-11-22 +Planned removal: 2022-05-22 + +### Deprecate `pipelines` fields in the Package GraphQL types + +As part of the work to create a [Package Registry GraphQL API](https://gitlab.com/groups/gitlab-org/-/epics/6318), the Package group deprecated the `pipelines` fields in all Package-related GraphQL types. As of GitLab 14.6, the `pipelines` field is deprecated in [`Package`](https://docs.gitlab.com/ee/api/graphql/reference/index.html#package) and [`PackageDetailsType`](https://docs.gitlab.com/ee/api/graphql/reference/index.html#packagedetailstype) due to scalability and performance concerns. + +In milestone 15.0, we will completely remove `pipelines` from `Package` and `PackageDetailsType`. You can follow and contribute to work on a replacement in the epic [GitLab-#7214](https://gitlab.com/groups/gitlab-org/-/epics/7214). + +Announced: 2021-12-22 +Planned removal: 2022-05-22 + +### Deprecate legacy approval status names from License Compliance API + +We deprecated legacy names for approval status of license policy (blacklisted, approved) in the `managed_licenses` API but they are still used in our API queries and responses. They will be removed in 15.0. + +If you are using our License Compliance API you should stop using the `approved` and `blacklisted` query parameters, they are now `allowed` and `denied`. In 15.0 the responses will also stop using `approved` and `blacklisted` so you need to adjust any of your custom tools to use the old and new values so they do not break with the 15.0 release. + +Announced: 2021-12-22 +Planned removal: 2022-05-22 ### Deprecate support for SLES 12 SP2 Long term service and support (LTSS) for SUSE Linux Enterprise Server (SLES) 12 SP2 [ended on March 31, 2021](https://www.suse.com/lifecycle/). The CA certificates on SP2 include the expired DST root certificate, and it's not getting new CA certificate package updates. We have implemented some [workarounds](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/merge_requests/191), but we will not be able to continue to keep the build running properly. Announced: 2021-11-22 +Planned removal: 2022-05-22 + +### Deprecation of Runner status `not_connected` API value + +The GitLab Runner REST and GraphQL [API](https://docs.gitlab.com/ee/api/runners.html#runners-api) endpoints +will return `never_contacted` instead of `not_connected` as the status values in 15.0. + +Runners that have never contacted the GitLab instance will also return `stale` if created more than 3 months ago. + +Announced: 2021-12-22 +Planned removal: 2022-05-22 + +### Deprecation of bundler-audit Dependency Scanning tool + +As of 14.6 bundler-audit is being deprecated from Dependency Scanning. It will continue to be in our CI/CD template while deprecated. We are removing bundler-audit from Dependency Scanning on May 22, 2022 in 15.0. After this removal Ruby scanning functionality will not be affected as it is still being covered by Gemnasium. + +If you have explicitly excluded bundler-audit using DS_EXCLUDED_ANALYZERS you will need to clean up (remove the reference) in 15.0. If you have customized your pipeline's Dependency Scanning configuration, for example to edit the `bundler-audit-dependency_scanning` job, you will want to switch to gemnasium-dependency_scanning before removal in 15.0, to prevent your pipeline from failing. If you have not used the DS_EXCLUDED_ANALYZERS to reference bundler-audit, or customized your template specifically for bundler-audit, you will not need to take action. + +Announced: 2021-12-22 +Planned removal: 2022-05-22 ### GitLab Serverless @@ -117,6 +178,7 @@ Announced: 2021-11-22 We decided to remove the GitLab Serverless features as they never really resonated with our users. Besides, given the continuous development of Kubernetes and Knative, our current implementations do not even work with recent versions. Announced: 2021-09-22 +Planned removal: 2022-05-22 ### Known host required for GitLab Runner SSH executor @@ -125,6 +187,7 @@ In [GitLab 14.3](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/30 In GitLab 15.0 and later, the default value for this configuration option will change from `true` to `false`. This means that strict host key checking will be enforced when using the GitLab Runner SSH executor. Announced: 2021-11-22 +Planned removal: 2022-05-22 ### Legacy database configuration @@ -135,6 +198,16 @@ supported using a single PostgreSQL adapter, whereas the new format is changing This deprecation mainly impacts users compiling GitLab from source because Omnibus will handle this configuration automatically. Announced: 2021-09-22 +Planned removal: 2022-05-22 + +### Must explicitly assign `AuthenticationType` for `[runners.cache.s3]` + +In GitLab 15.0 and later, to access the AWS S3 cache, you must specify the `AuthenticationType` for [`[runners.cache.s3]`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscaches3-section). The `AuthenticationType` must be `IAM` or `credentials`. + +Prior to 14.5, if you did not define the `AuthenticationType`, GitLab Runner chose a type for you. + +Announced: 2021-11-22 +Planned removal: 2022-05-22 ### NFS for Git repository storage deprecated @@ -149,6 +222,7 @@ Gitaly Cluster offers tremendous benefits for our customers such as: We encourage customers currently using NFS for Git repositories to plan their migration by reviewing our documentation on [migrating to Gitaly Cluster](https://docs.gitlab.com/ee/administration/gitaly/index.html#migrate-to-gitaly-cluster). Announced: 2021-06-22 +Planned removal: 2022-05-22 ### OmniAuth Kerberos gem @@ -159,6 +233,7 @@ This gem has not been maintained and has very little usage. We therefore plan to Note that we are not deprecating the Kerberos SPNEGO integration, only the old password-based Kerberos integration. Announced: 2021-09-22 +Planned removal: 2022-05-22 ### Package pipelines in API payload is paginated @@ -167,31 +242,48 @@ A request to the API for `/api/v4/projects/:id/packages` returns a paginated res In milestone 15.0, we will remove the `pipelines` attribute from the API response. Announced: 2021-11-22 +Planned removal: 2022-05-22 ### REST API Runner will not contain `paused` -Runner REST API will not return `paused` as a status in GitLab 15.0. +The GitLab Runner REST and GraphQL API endpoints will not return `paused` or `active` as a status in GitLab 15.0. -Paused runners' status will only relate to runner contact status, such as: -`online`, `offline`, or `not_connected`. Status `paused` will not appear when the runner is -not active. +A runner's status will only relate to runner contact status, such as: +`online`, `offline`, or `not_connected`. Status `paused` or `active` will no longer appear. When checking if a runner is `paused`, API users are advised to check the boolean attribute -`active` to be `false` instead. +`active` to be `false` instead. When checking if a runner is `active`, check if `active` is `true`. + +Announced: 2021-11-22 +Planned removal: 2022-05-22 + +### Removal of `defaultMergeCommitMessageWithDescription` GraphQL API field + +The GraphQL API field `defaultMergeCommitMessageWithDescription` has been deprecated and will be removed in GitLab 15.0. For projects with a commit message template set, it will ignore the template. Announced: 2021-11-22 +Planned removal: 2022-05-22 ### Removal of `promote-db` command from `gitlab-ctl` In GitLab 14.5, we introduced the command `gitlab-ctl promote` to promote any Geo secondary node to a primary during a failover. This command replaces `gitlab-ctl promote-db` which is used to promote database nodes in multi-node Geo secondary sites. `gitlab-ctl promote-db` will continue to function as-is and be available until GitLab 15.0. We recommend that Geo customers begin testing the new `gitlab-ctl promote` command in their staging environments and incorporating the new command in their failover procedures. Announced: 2021-11-22 +Planned removal: 2022-05-22 ### Removal of `promote-to-primary-node` command from `gitlab-ctl` In GitLab 14.5, we introduced the command `gitlab-ctl promote` to promote any Geo secondary node to a primary during a failover. This command replaces `gitlab-ctl promote-to-primary-node` which was only usable for single-node Geo sites. `gitlab-ctl promote-to-primary-node` will continue to function as-is and be available until GitLab 15.0. We recommend that Geo customers begin testing the new `gitlab-ctl promote` command in their staging environments and incorporating the new command in their failover procedures. Announced: 2021-11-22 +Planned removal: 2022-05-22 + +### Remove `type` and `types` keyword in CI/CD configuration + +The `type` and `types` CI/CD keywords will be removed in GitLab 15.0. Pipelines that use these keywords will stop working, so you must switch to `stage` and `stages`, which have the same behavior. + +Announced: 2021-12-22 +Planned removal: 2022-05-22 ### Remove the `:dependency_proxy_for_private_groups` feature flag @@ -200,6 +292,7 @@ We added a feature flag because [GitLab-#11582](https://gitlab.com/gitlab-org/gi In milestone 15.0, we will remove the feature flag entirely. Moving forward, you must authenticate when using the Dependency Proxy. Announced: 2021-11-22 +Planned removal: 2022-05-22 ### Remove the `pipelines` field from the `version` field @@ -211,6 +304,7 @@ In GraphQL, there are two `pipelines` fields that you can use in a [`PackageDeta To mitigate possible performance problems, we will remove the `versions` field's `pipelines` field in milestone 15.0. Although you will no longer be able to get all pipelines for all versions of a package, you can still get the pipelines of a single version through the remaining `pipelines` field for that version. Announced: 2021-11-22 +Planned removal: 2022-05-22 ### Update to the Container Registry group-level API @@ -219,25 +313,22 @@ In milestone 15.0, support for the `tags` and `tags_count` parameters will be re The `GET /groups/:id/registry/repositories` endpoint will remain, but won't return any info about tags. To get the info about tags, you can use the existing `GET /registry/repositories/:id` endpoint, which will continue to support the `tags` and `tag_count` options as it does today. The latter must be called once per image repository. Announced: 2021-11-22 +Planned removal: 2022-05-22 ### Value Stream Analytics filtering calculation change -We are changing how the date filter works in Value Stream Analytics. Instead of filtering by the time that the issue or merge request was created, the date filter will filter by the end event time of the given stage. This will result in completely different figures after this change has rolled out. +We are changing how the date filter works in Value Stream Analytics. Instead of filtering by the time that the issue or merge request was created, the date filter will filter by the end event time of the given stage. This will result in completely different figures after this change has rolled out. If you monitor Value Stream Analytics metrics and rely on the date filter, to avoid losing data, you must save the data prior to this change. Announced: 2021-11-22 +Planned removal: 2022-05-22 -### `AuthenticationType` for `[runners.cache.s3]` must be explicitly assigned +### apiFuzzingCiConfigurationCreate GraphQL mutation -In GitLab 15.0 and later, to access the AWS S3 cache, you must specify the `AuthenticationType` for [`[runners.cache.s3]`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscaches3-section). The `AuthenticationType` must be `IAM` or `credentials`. - -Prior to 14.5, if you did not define the `AuthenticationType`, GitLab Runner chose a type for you. +The API Fuzzing configuration snippet is now being generated client-side and does not require an +API request anymore. We are therefore deprecating the `apiFuzzingCiConfigurationCreate` mutation +which isn't being used in GitLab anymore. -Announced: 2021-11-22 - -### defaultMergeCommitMessageWithDescription GraphQL API field will be removed in GitLab 15.0 - -The GraphQL API field `defaultMergeCommitMessageWithDescription` has been deprecated and will be removed in GitLab 15.0. For projects with a commit message template set, it will ignore the template. - -Announced: 2021-11-22 +Announced: 2021-12-22 +Planned removal: 2022-05-22 diff --git a/doc/update/index.md b/doc/update/index.md index bb7fdaa93c0..98dfee04a41 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -82,17 +82,23 @@ See the guide to [plan your GitLab upgrade](plan_your_upgrade.md). ## Checking for background migrations before upgrading Certain releases may require different migrations to be -finished before you update to the newer version. Additionally check -[batched migrations](#batched-background-migrations) from GitLab 14.0. +finished before you update to the newer version. + +[Batched migrations](#batched-background-migrations) are a migration type available in GitLab 14.0 and later. +Background migrations and batched migrations not the same, so you should check that both are +complete before updating. Decrease the time required to complete these migrations by increasing the number of [Sidekiq workers](../administration/operations/extra_sidekiq_processes.md) that can process jobs in the `background_migration` queue. +### Background migrations + **For Omnibus installations:** ```shell sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.pending' ``` **For installations from source:** @@ -100,6 +106,7 @@ sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaini ```shell cd /home/git/gitlab sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.pending' ``` ### Batched background migrations @@ -212,30 +219,9 @@ sudo -u git -H bundle exec rake gitlab:elastic:list_pending_migrations See [how to retry a halted migration](../integration/elasticsearch.md#retry-a-halted-migration). -## Upgrade paths - -Upgrading across multiple GitLab versions in one go is *only possible by accepting downtime*. -The following examples assume downtime is acceptable while upgrading. -If you don't want any downtime, read how to [upgrade with zero downtime](zero_downtime.md). - -Find where your version sits in the upgrade path below, and upgrade GitLab -accordingly, while also consulting the -[version-specific upgrade instructions](#version-specific-upgrading-instructions): - -`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.15`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.1.Z`](#1410) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) - -The following table, while not exhaustive, shows some examples of the supported -upgrade paths. +## Upgrading without downtime -| Target version | Your version | Supported upgrade path | Note | -| -------------- | ------------ | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -| `14.1.6` | `13.9.2` | `13.9.2` -> `13.12.12` -> `14.0.11` -> `14.1.6` | Two intermediate versions are required: `13.12` and `14.0`, then `14.1`. | -| `13.12.10` | `12.9.2` | `12.9.2` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> `13.8.8` -> `13.12.10` | Four intermediate versions are required: `12.10`, `13.0`, `13.1` and `13.8.8`, then `13.12.10`. | -| `13.2.10` | `11.5.0` | `11.5.0` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> `13.2.10` | Six intermediate versions are required: `11.11`, `12.0`, `12.1`, `12.10`, `13.0` and `13.1`, then `13.2.10`. | -| `12.10.14` | `11.3.4` | `11.3.4` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` | Three intermediate versions are required: `11.11`, `12.0` and `12.1`, then `12.10.14`. | -| `12.9.5` | `10.4.5` | `10.4.5` -> `10.8.7` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.9.5` | Four intermediate versions are required: `10.8`, `11.11`, `12.0` and `12.1`, then `12.9.5`. | -| `12.2.5` | `9.2.6` | `9.2.6` -> `9.5.10` -> `10.8.7` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.2.5` | Five intermediate versions are required: `9.5`, `10.8`, `11.11`, `12.0`, `12.1`, then `12.2.5`. | -| `11.3.4` | `8.13.4` | `8.13.4` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> `11.3.4` | `8.17.7` is the last version in version 8, `9.5.10` is the last version in version 9, `10.8.7` is the last version in version 10. | +Read how to [upgrade without downtime](zero_downtime.md). ## Upgrading to a new major version @@ -247,8 +233,9 @@ cannot guarantee that upgrading between major versions is seamless. It is required to follow the following upgrade steps to ensure a successful *major* version upgrade: 1. Upgrade to the latest minor version of the preceding major version. -1. Upgrade to the first minor version (`X.0.Z`) of the target major version. -1. Proceed with upgrading to a newer release. +1. Upgrade to the next major version (`X.0.Z`). +1. Upgrade to its first minor version (`X.1.Z`). +1. Proceed with upgrading to a newer releases of that major version. Identify a [supported upgrade path](#upgrade-paths). @@ -264,11 +251,34 @@ before proceeding with the major version upgrade. If your GitLab instance has any runners associated with it, it is very important to upgrade GitLab Runner to match the GitLab minor version that was -upgraded to. This is to ensure [compatibility with GitLab versions](https://docs.gitlab.com/runner/#compatibility-with-gitlab-versions). +upgraded to. This is to ensure [compatibility with GitLab versions](https://docs.gitlab.com/runner/#gitlab-runner-versions). -## Upgrading without downtime +## Upgrade paths -Read how to [upgrade without downtime](zero_downtime.md). +Upgrading across multiple GitLab versions in one go is *only possible by accepting downtime*. +The following examples assume downtime is acceptable while upgrading. +If you don't want any downtime, read how to [upgrade with zero downtime](zero_downtime.md). + +Find where your version sits in the upgrade path below, and upgrade GitLab +accordingly, while also consulting the +[version-specific upgrade instructions](#version-specific-upgrading-instructions): + +`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [latest `13.12.Z`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.1.Z`](#1410) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) + +The following table, while not exhaustive, shows some examples of the supported +upgrade paths. +Additional steps between the mentioned versions are possible. We list the minimally necessary steps only. + +| Target version | Your version | Supported upgrade path | Note | +| -------------- | ------------ | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| `14.2.6` | `13.10.2` | `13.10.2` -> `13.12.12` -> `14.0.11` -> `14.1.8` -> `14.2.6` | Three intermediate versions are required: `13.12`, `14.0`, and `14.1`, then `14.2.6`. | +| `14.1.8` | `13.9.2` | `13.9.2` -> `13.12.12` -> `14.0.11` -> `14.1.8` | Two intermediate versions are required: `13.12` and `14.0`, then `14.1.8`. | +| `13.12.10` | `12.9.2` | `12.9.2` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> `13.8.8` -> `13.12.10` | Four intermediate versions are required: `12.10`, `13.0`, `13.1` and `13.8.8`, then `13.12.10`. | +| `13.2.10` | `11.5.0` | `11.5.0` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> `13.2.10` | Six intermediate versions are required: `11.11`, `12.0`, `12.1`, `12.10`, `13.0` and `13.1`, then `13.2.10`. | +| `12.10.14` | `11.3.4` | `11.3.4` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` | Three intermediate versions are required: `11.11`, `12.0` and `12.1`, then `12.10.14`. | +| `12.9.5` | `10.4.5` | `10.4.5` -> `10.8.7` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.9.5` | Four intermediate versions are required: `10.8`, `11.11`, `12.0` and `12.1`, then `12.9.5`. | +| `12.2.5` | `9.2.6` | `9.2.6` -> `9.5.10` -> `10.8.7` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.2.5` | Five intermediate versions are required: `9.5`, `10.8`, `11.11`, `12.0`, and `12.1`, then `12.2.5`. | +| `11.3.4` | `8.13.4` | `8.13.4` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> `11.3.4` | `8.17.7` is the last version in version 8, `9.5.10` is the last version in version 9, `10.8.7` is the last version in version 10. | ## Upgrading between editions @@ -357,8 +367,14 @@ or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [follo ### 14.4.0 -Git 2.33.x and later is required. We recommend you use the -[Git version provided by Gitaly](../install/installation.md#git). +- Git 2.33.x and later is required. We recommend you use the + [Git version provided by Gitaly](../install/installation.md#git). +- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). +- After enabling database load balancing by default in 14.4.0, we found an issue where + [cron jobs would not work if the connection to PostgreSQL was severed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73716), + as Sidekiq would continue using a bad connection. Geo and other features that rely on + cron jobs running regularly do not work until Sidekiq is restarted. We recommend + upgrading to GitLab 14.4.3 and later if this issue affects you. ### 14.3.0 @@ -385,6 +401,8 @@ for how to proceed. sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production ``` +- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). + ### 14.2.0 - [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). @@ -411,15 +429,20 @@ for how to proceed. sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production ``` +- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). + ### 14.1.0 - [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases) but can upgrade to 14.1.Z. -- It is not required for instances already running 14.0.5 (or higher) to stop at 14.1.Z. + + It is not required for instances already running 14.0.5 (or higher) to stop at 14.1.Z. 14.1 is included on the upgrade path for the broadest compatibility with self-managed installations, and ensure 14.0.0-14.0.4 installations do not encounter issues with [batched background migrations](#batched-background-migrations). +- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). + ### 14.0.0 - Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances. @@ -449,6 +472,8 @@ for how to proceed. You should instead follow a [supported upgrade path](#upgrade-paths). - The support of PostgreSQL 11 [has been dropped](../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0. +- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). + #### Upgrading to later 14.Y releases - Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later, @@ -459,48 +484,60 @@ for how to proceed. 1. [Batched background migrations need to finish](#batched-background-migrations) before you update to a later version [and may take longer than usual](#1400). +### 13.12.0 + +See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). + ### 13.11.0 -Git 2.31.x and later is required. We recommend you use the -[Git version provided by Gitaly](../install/installation.md#git). +- Git 2.31.x and later is required. We recommend you use the + [Git version provided by Gitaly](../install/installation.md#git). + +- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). + +### 13.10.0 + +See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). ### 13.9.0 -We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160) -that prevents upgrades to GitLab 13.9.0, 13.9.1, 13.9.2, and 13.9.3 when following the zero-downtime steps. It is necessary -to perform the following additional steps for the zero-downtime upgrade: +- We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160) + that prevents upgrades to GitLab 13.9.0, 13.9.1, 13.9.2, and 13.9.3 when following the zero-downtime steps. It is necessary + to perform the following additional steps for the zero-downtime upgrade: -1. Before running the final `sudo gitlab-rake db:migrate` command on the deploy node, - execute the following queries using the PostgreSQL console (or `sudo gitlab-psql`) - to drop the problematic triggers: + 1. Before running the final `sudo gitlab-rake db:migrate` command on the deploy node, + execute the following queries using the PostgreSQL console (or `sudo gitlab-psql`) + to drop the problematic triggers: - ```sql - drop trigger trigger_e40a6f1858e6 on application_settings; - drop trigger trigger_0d588df444c8 on application_settings; - drop trigger trigger_1572cbc9a15f on application_settings; - drop trigger trigger_22a39c5c25f3 on application_settings; - ``` + ```sql + drop trigger trigger_e40a6f1858e6 on application_settings; + drop trigger trigger_0d588df444c8 on application_settings; + drop trigger trigger_1572cbc9a15f on application_settings; + drop trigger trigger_22a39c5c25f3 on application_settings; + ``` -1. Run the final migrations: + 1. Run the final migrations: - ```shell - sudo gitlab-rake db:migrate - ``` + ```shell + sudo gitlab-rake db:migrate + ``` -If you have already run the final `sudo gitlab-rake db:migrate` command on the deploy node and have -encountered the [column rename issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160), you -see the following error: + If you have already run the final `sudo gitlab-rake db:migrate` command on the deploy node and have + encountered the [column rename issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160), you + see the following error: -```shell --- remove_column(:application_settings, :asset_proxy_whitelist) -rake aborted! -StandardError: An error has occurred, all later migrations canceled: -PG::DependentObjectsStillExist: ERROR: cannot drop column asset_proxy_whitelist of table application_settings because other objects depend on it -DETAIL: trigger trigger_0d588df444c8 on table application_settings depends on column asset_proxy_whitelist of table application_settings -``` + ```shell + -- remove_column(:application_settings, :asset_proxy_whitelist) + rake aborted! + StandardError: An error has occurred, all later migrations canceled: + PG::DependentObjectsStillExist: ERROR: cannot drop column asset_proxy_whitelist of table application_settings because other objects depend on it + DETAIL: trigger trigger_0d588df444c8 on table application_settings depends on column asset_proxy_whitelist of table application_settings + ``` + + To work around this bug, follow the previous steps to complete the update. + More details are available [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160). -To work around this bug, follow the previous steps to complete the update. -More details are available [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160). +- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). ### 13.8.8 @@ -610,6 +647,14 @@ After upgraded to 11.11.8 you can safely upgrade to 12.0.Z. See our [documentation on upgrade paths](../policy/maintenance.md#upgrade-recommendations) for more information. +### Maintenance mode issue in GitLab 13.9 to 14.4 + +When [Maintenance mode](../administration/maintenance_mode/index.md) is enabled, users cannot sign in with SSO, SAML, or LDAP. + +Users who were signed in before Maintenance mode was enabled will continue to be signed in. If the admin who enabled Maintenance mode loses their session, then they will not be able to disable Maintenance mode via the UI. In that case, you can [disable Maintenance mode via the API or Rails console](../administration/maintenance_mode/#disable-maintenance-mode). + +[This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in GitLab 14.5.0, and is expected to be backported to GitLab 14.3 and 14.4. + ## Miscellaneous - [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating diff --git a/doc/update/package/index.md b/doc/update/package/index.md index 27845caed76..d7c18f15e44 100644 --- a/doc/update/package/index.md +++ b/doc/update/package/index.md @@ -159,7 +159,7 @@ install GitLab for the first time or update it. To download and install GitLab: 1. Visit the [official repository](#upgrade-using-the-official-repositories) of your package. -1. Filter the list by searching for the version you want to install (for example 14.1.6). +1. Filter the list by searching for the version you want to install (for example 14.1.8). Multiple packages may exist for a single version, one for each supported distribution and architecture. Next to the filename is a label indicating the distribution, as the filenames may be the same. @@ -188,7 +188,7 @@ For the GitLab Community Edition, replace `gitlab-ee` with ### GitLab 13.7 and later unavailable on Amazon Linux 2 -Amazon Linux 2 is not an [officially supported operating system](../../administration/package_information/deprecated_os.md#supported-operating-systems). +Amazon Linux 2 is not an [officially supported operating system](../../administration/package_information/supported_os.md). However, in past the [official package installation script](https://packages.gitlab.com/gitlab/gitlab-ee/install) installed the `el/6` package repository if run on Amazon Linux. From GitLab 13.7, we no longer provide `el/6` packages so administrators must run the [installation script](https://packages.gitlab.com/gitlab/gitlab-ee/install) diff --git a/doc/update/plan_your_upgrade.md b/doc/update/plan_your_upgrade.md index 320f900dd21..98549cc136a 100644 --- a/doc/update/plan_your_upgrade.md +++ b/doc/update/plan_your_upgrade.md @@ -18,7 +18,7 @@ General notes: to create your plan, share details of your architecture, including: - How is GitLab installed? - What is the operating system of the node? - (check [OS versions that are no longer supported](../administration/package_information/deprecated_os.md) to confirm that later updates are available). + (check [OS versions that are no longer supported](../administration/package_information/supported_os.md#os-versions-that-are-no-longer-supported) to confirm that later updates are available). - Is it a single-node or a multi-node setup? If multi-node, share any architectural details about each node with us. - Are you using [GitLab Geo](../administration/geo/index.md)? If so, share any architectural details about each secondary node. - What else might be unique or interesting in your setup that might be important for us to understand? @@ -71,7 +71,7 @@ comprised of a way to back up the instance and a way to restore it. ### Back up GitLab -Create a backup of GitLab and all its data (database, repos, uploads, builds, +Create a backup of GitLab and all its data (database, repositories, uploads, builds, artifacts, LFS objects, registry, pages). This is vital for making it possible to roll back GitLab to a working state if there's a problem with the upgrade: @@ -112,7 +112,7 @@ to your instance and then upgrade it for any relevant features you're using. - [Determine what upgrade path](index.md#upgrade-paths) to follow. - Account for any [version-specific update instructions](index.md#version-specific-upgrading-instructions). - Account for any [version-specific changes](package/index.md#version-specific-changes). - - Check the [OS compatibility with the target GitLab version](../administration/package_information/deprecated_os.md). + - Check the [OS compatibility with the target GitLab version](../administration/package_information/supported_os.md). - Due to background migrations, plan to pause any further upgrades after upgrading to a new major version. [All migrations must finish running](index.md#checking-for-background-migrations-before-upgrading) @@ -145,7 +145,7 @@ version prior to upgrading the application server. If you're using Geo: -- Review [Geo upgrade documentation](../administration/geo/replication/updating_the_geo_nodes.md). +- Review [Geo upgrade documentation](../administration/geo/replication/updating_the_geo_sites.md). - Read about the [Geo version-specific update instructions](../administration/geo/replication/version_specific_updates.md). - Review Geo-specific steps when [updating the database](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). - Create an upgrade and rollback plan for _each_ Geo node (primary and each secondary). diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index 4343b464ba6..22ffcda9138 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -111,7 +111,7 @@ Download and install Go (for Linux, 64-bit): # Remove former Go installation folder sudo rm -rf /usr/local/go -curl --remote-name --progress-bar "https://golang.org/dl/go1.16.10.linux-amd64.tar.gz" +curl --remote-name --progress-bar "https://go.dev/dl/go1.16.10.linux-amd64.tar.gz" echo '414cd18ce1d193769b9e97d2401ad718755ab47816e13b2a1cde203d263b55cf go1.16.10.linux-amd64.tar.gz' | shasum -a256 -c - && \ sudo tar -C /usr/local -xzf go1.16.10.linux-amd64.tar.gz sudo ln -sf /usr/local/go/bin/{go,gofmt} /usr/local/bin/ diff --git a/doc/update/upgrading_postgresql_using_slony.md b/doc/update/upgrading_postgresql_using_slony.md index 8ccdf8d0077..f2bbc8d7558 100644 --- a/doc/update/upgrading_postgresql_using_slony.md +++ b/doc/update/upgrading_postgresql_using_slony.md @@ -113,7 +113,7 @@ CREATE ROLE slony WITH SUPERUSER LOGIN REPLICATION ENCRYPTED PASSWORD 'password ALTER ROLE slony SET statement_timeout TO 0; ``` -Make sure you replace "password string here" with the actual password for the +Make sure you replace "password string here" with an actual password for the user. A password is required. This user must be created on both the old and new database server using the same password. @@ -230,7 +230,7 @@ Now run the following commands: \i /tmp/migrations.sql ``` -To verify if the structure is in place close the session, start it again, then +To verify if the structure is in place close the session (`\q`), start it again, then run `\d`. If all went well you should see output along the lines of the following: @@ -459,7 +459,7 @@ main Upload this script to the _target_ server and execute it as follows: ```shell -bash path/to/the/script/above.sh +sudo bash path/to/the/script/above.sh ``` This corrects the ownership of sequences and reset the next value for the diff --git a/doc/update/zero_downtime.md b/doc/update/zero_downtime.md index a311731cadd..13658e6071b 100644 --- a/doc/update/zero_downtime.md +++ b/doc/update/zero_downtime.md @@ -29,9 +29,9 @@ If you meet all the requirements above, follow these instructions in order. Ther | [Geo](#geo-deployment) | GitLab EE with Geo enabled | | [Multi-node / HA with Geo](#multi-node--ha-deployment-with-geo) | GitLab CE/EE on multiple nodes | -Each type of deployment will require that you hot reload the `puma` and `sidekiq` processes on all nodes running these +Each type of deployment requires that you hot reload the `puma` and `sidekiq` processes on all nodes running these services after you've upgraded. The reason for this is that those processes each load the GitLab Rails application which reads and loads -the database schema into memory when starting up. Each of these processes will need to be reloaded (or restarted in the case of `sidekiq`) +the database schema into memory when starting up. Each of these processes needs to be reloaded (or restarted in the case of `sidekiq`) to re-read any database changes that have been made by post-deployment migrations. Most of the time you can safely upgrade from a patch release to the next minor @@ -56,7 +56,7 @@ increasing the number of Sidekiq workers that can process jobs in the `background_migration` queue. To see the size of this queue, [Check for background migrations before upgrading](index.md#checking-for-background-migrations-before-upgrading). -As a rule of thumb, any database smaller than 10 GB doesn't take too much time to +As a guideline, any database smaller than 10 GB doesn't take too much time to upgrade; perhaps an hour at most per minor release. Larger databases however may require more time, but this is highly dependent on the size of the database and the migrations that are being performed. @@ -111,26 +111,26 @@ Before following these instructions, note the following **important** informatio 1. Update the GitLab package: - - For GitLab Community Edition: + - For GitLab [Enterprise Edition](https://about.gitlab.com/pricing/): ```shell # Debian/Ubuntu sudo apt-get update - sudo apt-get install gitlab-ce + sudo apt-get install gitlab-ee # Centos/RHEL - sudo yum install gitlab-ce + sudo yum install gitlab-ee ``` - - For GitLab [Enterprise Edition](https://about.gitlab.com/pricing/): + - For GitLab Community Edition: ```shell # Debian/Ubuntu sudo apt-get update - sudo apt-get install gitlab-ee + sudo apt-get install gitlab-ce # Centos/RHEL - sudo yum install gitlab-ee + sudo yum install gitlab-ce ``` 1. To get the regular migrations and latest code in place, run @@ -176,14 +176,14 @@ Upgrades on web (Puma) nodes must be done in a rolling manner, one after another, ensuring at least one node is always up to serve traffic. This is required to ensure zero-downtime. -Puma will enter a blackout period as part of the upgrade, during which they -continue to accept connections but will mark their respective health check +Puma enters a blackout period as part of the upgrade, during which nodes +continue to accept connections but mark their respective health check endpoints to be unhealthy. On seeing this, the load balancer should disconnect them gracefully. -Puma will restart only after completing all the currently processing requests. +Puma restarts only after completing all the currently processing requests. This ensures data and service integrity. Once they have restarted, the health -check end points will be marked healthy. +check end points are marked healthy. The nodes must be updated in the following order to update an HA instance using load balancer to latest GitLab version. @@ -201,14 +201,14 @@ load balancer to latest GitLab version. ```shell # Debian/Ubuntu - sudo apt-get update && sudo apt-get install gitlab-ce + sudo apt-get update && sudo apt-get install gitlab-ee # Centos/RHEL - sudo yum install gitlab-ce + sudo yum install gitlab-ee ``` - If you are an Enterprise Edition user, replace `gitlab-ce` with - `gitlab-ee` in the above command. + If you are a Community Edition user, replace `gitlab-ee` with + `gitlab-ce` in the above command. 1. Get the regular migrations and latest code in place. Before running this step, the deploy node's `/etc/gitlab/gitlab.rb` configuration file must have @@ -254,7 +254,7 @@ the application. Before you update the main application you need to update Praefect. Out of your Praefect nodes, pick one to be your Praefect deploy node. -This is where you will install the new Omnibus package first and run +This is where you install the new Omnibus package first and run database migrations. **Praefect deploy node** @@ -277,13 +277,13 @@ database migrations. ```shell # Debian/Ubuntu - sudo apt-get update && sudo apt-get install gitlab-ce + sudo apt-get update && sudo apt-get install gitlab-ee # Centos/RHEL - sudo yum install gitlab-ce + sudo yum install gitlab-ee ``` - If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command. + If you are a Community Edition user, replace `gitlab-ee` with `gitlab-ce` in the above command. - To apply the Praefect database migrations and restart Praefect, run: @@ -296,10 +296,10 @@ database migrations. - Update the GitLab package: ```shell - sudo apt-get update && sudo apt-get install gitlab-ce + sudo apt-get update && sudo apt-get install gitlab-ee ``` - If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command. + If you are a Community Edition user, replace `gitlab-ee` with `gitlab-ce` in the above command. - Ensure nodes are running the latest code: @@ -330,13 +330,13 @@ node throughout the process. ```shell # Debian/Ubuntu - sudo apt-get update && sudo apt-get install gitlab-ce + sudo apt-get update && sudo apt-get install gitlab-ee # Centos/RHEL - sudo yum install gitlab-ce + sudo yum install gitlab-ee ``` - If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command. + If you are a Community Edition user, replace `gitlab-ee` with `gitlab-ce` in the above command. - Ensure nodes are running the latest code @@ -350,17 +350,17 @@ node throughout the process. ```shell # Debian/Ubuntu - sudo apt-get update && sudo apt-get install gitlab-ce + sudo apt-get update && sudo apt-get install gitlab-ee # Centos/RHEL - sudo yum install gitlab-ce + sudo yum install gitlab-ee ``` - If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command. + If you are a Community Edition user, replace `gitlab-ee` with `gitlab-ce` in the above command. - If you're using PgBouncer: - You'll need to bypass PgBouncer and connect directly to the database master + You need to bypass PgBouncer and connect directly to the database master before running migrations. Rails uses an advisory lock when attempting to run a migration to prevent @@ -391,10 +391,10 @@ node throughout the process. - Update the GitLab package ```shell - sudo apt-get update && sudo apt-get install gitlab-ce + sudo apt-get update && sudo apt-get install gitlab-ee ``` - If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command. + If you are a Community Edition user, replace `gitlab-ee` with `gitlab-ce` in the above command. - Ensure nodes are running the latest code @@ -457,7 +457,7 @@ following command to get address of current Redis primary ``` - If your application node is running a version older than GitLab 12.7.0, you - will have to run the underlying `redis-cli` command (which `get-redis-master` + have to run the underlying `redis-cli` command (which `get-redis-master` command uses) to fetch information about the primary. 1. Get the address of one of the sentinel nodes specified as @@ -653,7 +653,7 @@ setting `gitlab_rails['auto_migrate'] = false` in This section describes the steps required to upgrade a multi-node / HA deployment with Geo. Some steps must be performed on a particular node. This -node will be known as the “deploy node” and is noted through the following +node is known as the “deploy node” and is noted through the following instructions. Updates must be performed in the following order: @@ -737,7 +737,7 @@ sudo touch /etc/gitlab/skip-auto-reconfigure 1. If you're using PgBouncer: - You'll need to bypass PgBouncer and connect directly to the database master + You need to bypass PgBouncer and connect directly to the database master before running migrations. Rails uses an advisory lock when attempting to run a migration to prevent diff --git a/doc/user/admin_area/analytics/dev_ops_report.md b/doc/user/admin_area/analytics/dev_ops_report.md index 62fea3c266a..ede9e342a2e 100644 --- a/doc/user/admin_area/analytics/dev_ops_report.md +++ b/doc/user/admin_area/analytics/dev_ops_report.md @@ -20,11 +20,11 @@ To see DevOps Reports: > [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/20976) from Conversational Development Index in GitLab 12.6. NOTE: -To see the DevOps score, you must activate your GitLab instance's [Service Ping](../settings/usage_statistics.md#service-ping). This is because DevOps Score is a comparative tool, so your score data must be centrally processed by GitLab Inc. first. +To see the DevOps score, you must activate your GitLab instance's [Service Ping](../settings/usage_statistics.md#service-ping). DevOps Score is a comparative tool, so your score data must be centrally processed by GitLab Inc. first. You can use the DevOps score to compare your DevOps status to other organizations. -The DevOps Score tab displays the usage of major GitLab features on your instance over +The DevOps Score tab displays usage of major GitLab features on your instance over the last 30 days, averaged over the number of billable users in that time period. You can also see the Leader usage score, calculated from top-performing instances based on [Service Ping data](../settings/usage_statistics.md#service-ping) that GitLab has collected. @@ -47,29 +47,27 @@ feature is available. > - Multi-select [added](https://gitlab.com/gitlab-org/gitlab/-/issues/333586) in GitLab 14.2. > - Overview table [added](https://gitlab.com/gitlab-org/gitlab/-/issues/335638) in GitLab 14.3. -DevOps Adoption shows you which groups in your organization are using the most essential features of GitLab: - -- Dev - - Approvals - - Code owners - - Issues - - Merge requests -- Sec - - DAST - - Dependency Scanning - - Fuzz Testing - - SAST -- Ops - - Deployments - - Pipelines - - Runners - -To add or remove your groups, in the top right-hand section the page, select **Add or remove groups**. - -DevOps Adoption allows you to: - -- Verify whether you are getting the return on investment that you expected from GitLab. -- Identify specific groups that are lagging in their adoption of GitLab, so you can help them along in their DevOps journey. -- Find the groups that have adopted certain features, and can provide guidance to other groups on how to use those features. +DevOps Adoption shows feature adoption for development, security, and operations. + +| Category | Feature | +| --- | --- | +| Development | Approvals<br>Code owners<br>Issues<br>Merge requests | +| Security | DAST<br>Dependency Scanning<br>Fuzz Testing<br>SAST | +| Operations | Deployments<br>Pipelines<br>Runners | + +You can use Group DevOps Adoption to: + +- Identify specific subgroups that are lagging in their adoption of GitLab features, so you can guide them on +their DevOps journey. +- Find subgroups that have adopted certain features, and provide guidance to other subgroups on +how to use those features. +- Verify if you are getting the return on investment that you expected from GitLab. + +## Add or remove a group + +To add or remove a subgroup from the DevOps Adoption report: + +1. Select **Add or remove groups**. +1. Select the subgroup you want to add or remove and select **Save changes**.  diff --git a/doc/user/admin_area/analytics/usage_trends.md b/doc/user/admin_area/analytics/usage_trends.md index 06995069215..7901d30c3ea 100644 --- a/doc/user/admin_area/analytics/usage_trends.md +++ b/doc/user/admin_area/analytics/usage_trends.md @@ -16,8 +16,11 @@ WARNING: This feature might not be available to you. Check the **version history** note above for details. Usage Trends gives you an overview of how much data your instance contains, and how quickly this volume is changing over time. +Usage Trends data refreshes daily. -To see Usage Trends: +## View Usage Trends + +To view Usage Trends: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Analytics > Usage Trends**. diff --git a/doc/user/admin_area/approving_users.md b/doc/user/admin_area/approving_users.md deleted file mode 100644 index fdf0c7edfc7..00000000000 --- a/doc/user/admin_area/approving_users.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'moderate_users.md' -remove_date: '2021-10-20' ---- - -This document was moved to [another location](moderate_users.md). - -<!-- This redirect file can be deleted after <2021-10-20>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index 4de2397706b..b0f38e8cfb9 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -107,15 +107,16 @@ You can combine the filter options. For example, to list only public projects wi 1. Click the **Public** tab. 1. Enter `score` in the **Filter by name...** input box. -#### Deleted projects **(PREMIUM SELF)** +#### Projects pending deletion **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37014) in GitLab 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37014) in GitLab 13.3. +> - [Tab renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/347468) from **Deleted projects** in GitLab 14.7. -When delayed project deletion is [enabled for a group](../group/index.md#enable-delayed-project-removal), +When delayed project deletion is [enabled for a group](../group/index.md#enable-delayed-project-deletion), projects within that group are not deleted immediately, but only after a delay. To access a list of all projects that are pending deletion: 1. On the top bar, select **Menu > Projects > Explore projects**. -1. Select the **Deleted projects** tab. +1. Select the **Pending deletion** tab (in GitLab 14.7 and later) or the **Deleted projects** tab (GitLab 14.6 and earlier). Listed for each project is: @@ -199,6 +200,7 @@ The following data is included in the export: - Type - Path - Access level ([Project](../permissions.md#project-members-permissions) and [Group](../permissions.md#group-members-permissions)) +- Date of last activity ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345388) in GitLab 14.6). For a list of activities that populate this column, see the [Users API documentation](../../api/users.md#get-user-activities-admin-only).  @@ -257,7 +259,7 @@ To [Create a new group](../group/index.md#create-a-group) click **New group**. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340920) in GitLab 14.4. -You can administer all [topics](../project/working_with_projects.md#explore-topics) in the +You can administer all [topics](../project/working_with_projects.md#explore-topics) in the GitLab instance from the Admin Area's Topics page. To access the Topics page: diff --git a/doc/user/admin_area/moderate_users.md b/doc/user/admin_area/moderate_users.md index a98250dfc56..3f15bd5b4e6 100644 --- a/doc/user/admin_area/moderate_users.md +++ b/doc/user/admin_area/moderate_users.md @@ -53,7 +53,7 @@ To approve or reject a user sign up: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. 1. Select the **Pending approval** tab. -1. (Optional) Select a user. +1. Optional. Select a user. 1. Select the **{settings}** **User administration** dropdown. 1. Select **Approve** or **Reject**. @@ -77,7 +77,7 @@ by removing them in LDAP, or directly from the Admin Area. To do this: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. -1. (Optional) Select a user. +1. Optional. Select a user. 1. Select the **{settings}** **User administration** dropdown. 1. Select **Block**. @@ -101,7 +101,7 @@ A blocked user can be unblocked from the Admin Area. To do this: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. 1. Select on the **Blocked** tab. -1. (Optional) Select a user. +1. Optional. Select a user. 1. Select the **{settings}** **User administration** dropdown. 1. Select **Unblock**. @@ -145,7 +145,7 @@ A user can be deactivated from the Admin Area. To do this: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. -1. (Optional) Select a user. +1. Optional. Select a user. 1. Select the **{settings}** **User administration** dropdown. 1. Select **Deactivate**. @@ -185,7 +185,7 @@ To do this: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. 1. Select the **Deactivated** tab. -1. (Optional) Select a user. +1. Optional. Select a user. 1. Select the **{settings}** **User administration** dropdown. 1. Select **Activate**. @@ -211,7 +211,7 @@ Users can be banned using the Admin Area. To do this: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. -1. (Optional) Select a user. +1. Optional. Select a user. 1. Select the **{settings}** **User administration** dropdown. 1. Select **Ban user**. @@ -224,7 +224,7 @@ A banned user can be unbanned using the Admin Area. To do this: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. 1. Select the **Banned** tab. -1. (Optional) Select a user. +1. Optional. Select a user. 1. Select the **{settings}** **User administration** dropdown. 1. Select **Unban user**. diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md index c511e85f3ce..5868f20d0d8 100644 --- a/doc/user/admin_area/settings/account_and_limit_settings.md +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -9,7 +9,12 @@ type: reference ## Default projects limit -You can change the default maximum number of projects that users can create in their personal namespace: +You can configure the default maximum number of projects new users can create in their +personal namespace. This limit affects only new user accounts created after you change +the setting. This setting is not retroactive for existing users, but you can separately edit +the [project limits for existing users](#projects-limit-for-a-user). + +To configure the maximum number of projects in personal namespaces for new users: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**, then expand **Account and limit**. @@ -18,6 +23,17 @@ You can change the default maximum number of projects that users can create in t If you set **Default projects limit** to 0, users are not allowed to create projects in their users personal namespace. However, projects can still be created in a group. +### Projects limit for a user + +You can edit a specific user, and change the maximum number of projects this user +can create in their personal namespace: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Overview** > **Users**. +1. From the list of users, select a user. +1. Select **Edit**. +1. Increase or decrease the **Projects limit** value. + ## Max attachment size You can change the maximum file size for attachments in comments and replies in GitLab: @@ -59,21 +75,21 @@ If you choose a size larger than the configured value for the web server, you may receive errors. See the [troubleshooting section](#troubleshooting) for more details. -## Personal Access Token prefix +## Personal access token prefix -> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/342327) in GitLab 14.5. Default prefix added. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20968) in GitLab 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342327) in GitLab 14.5, a default prefix. -You can set a global prefix for all generated Personal Access Tokens. +You can specify a prefix for personal access tokens. You might use a prefix +to find tokens more quickly, or for use with automation tools. -A prefix can help you identify PATs visually, as well as with automation tools. +The default prefix is `glpat-` but administrators can change it. -NOTE: -For GitLab.com and self-managed instances, the default prefix is `glpat-`. +[Project access tokens](../../project/settings/project_access_tokens.md) also inherit this prefix. ### Set a prefix -Only a GitLab administrator can set the prefix, which is a global setting applied -to any PAT generated in the system by any user: +To change the default global prefix: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. @@ -81,8 +97,8 @@ to any PAT generated in the system by any user: 1. Fill in the **Personal Access Token prefix** field. 1. Click **Save changes**. -It is also possible to configure the prefix via the [settings API](../../../api/settings.md) -using the `personal_access_token_prefix` field. +You can also configure the prefix by using the +[settings API](../../../api/settings.md). ## Repository size limit **(PREMIUM SELF)** @@ -176,38 +192,46 @@ To set a limit on how long these sessions are valid: 1. Fill in the **Session duration for Git operations when 2FA is enabled (minutes)** field. 1. Click **Save changes**. -## Limit the lifetime of personal access tokens **(ULTIMATE SELF)** +## Limit the lifetime of SSH keys **(ULTIMATE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3649) in GitLab 12.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1007) in GitLab 14.6 [with a flag](../../../administration/feature_flags.md) named `ff_limit_ssh_key_lifetime`. Disabled by default. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/346753) in GitLab 14.6. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, +ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) named `ff_limit_ssh_key_lifetime`. +On GitLab.com, this feature is not available. Users can optionally specify a lifetime for -[personal access tokens](../../profile/personal_access_tokens.md). +[SSH keys](../../../ssh/index.md). This lifetime is not a requirement, and can be set to any arbitrary number of days. -Personal access tokens are the only tokens needed for programmatic access to GitLab. +SSH keys are user credentials to access GitLab. However, organizations with security requirements may want to enforce more protection by -requiring the regular rotation of these tokens. +requiring the regular rotation of these keys. ### Set a lifetime Only a GitLab administrator can set a lifetime. Leaving it empty means there are no restrictions. -To set a lifetime on how long personal access tokens are valid: +To set a lifetime on how long SSH keys are valid: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Account and limit** section. -1. Fill in the **Maximum allowable lifetime for personal access tokens (days)** field. +1. Fill in the **Maximum allowable lifetime for SSH keys (days)** field. 1. Click **Save changes**. -Once a lifetime for personal access tokens is set, GitLab: +Once a lifetime for SSH keys is set, GitLab: -- Applies the lifetime for new personal access tokens, and require users to set an expiration date - and a date no later than the allowed lifetime. -- After three hours, revoke old tokens with no expiration date or with a lifetime longer than the - allowed lifetime. Three hours is given to allow administrators to change the allowed lifetime, - or remove it, before revocation takes place. +- Requires users to set an expiration date that is no later than the allowed lifetime on new + SSH keys. +- Applies the lifetime restriction to existing SSH keys. Keys with no expiry or a lifetime + greater than the maximum immediately become invalid. + +NOTE: +When a user's SSH key becomes invalid they can delete and re-add the same key again. ## Allow expired SSH keys to be used **(ULTIMATE SELF)** @@ -225,6 +249,39 @@ To allow the use of expired SSH keys: Disabling SSH key expiration immediately enables all expired SSH keys. +## Limit the lifetime of personal access tokens **(ULTIMATE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3649) in GitLab 12.6. + +Users can optionally specify a lifetime for +[personal access tokens](../../profile/personal_access_tokens.md). +This lifetime is not a requirement, and can be set to any arbitrary number of days. + +Personal access tokens are the only tokens needed for programmatic access to GitLab. +However, organizations with security requirements may want to enforce more protection by +requiring the regular rotation of these tokens. + +### Set a lifetime + +Only a GitLab administrator can set a lifetime. Leaving it empty means +there are no restrictions. + +To set a lifetime on how long personal access tokens are valid: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Account and limit** section. +1. Fill in the **Maximum allowable lifetime for personal access tokens (days)** field. +1. Click **Save changes**. + +Once a lifetime for personal access tokens is set, GitLab: + +- Applies the lifetime for new personal access tokens, and require users to set an expiration date + and a date no later than the allowed lifetime. +- After three hours, revoke old tokens with no expiration date or with a lifetime longer than the + allowed lifetime. Three hours is given to allow administrators to change the allowed lifetime, + or remove it, before revocation takes place. + ## Allow expired Personal Access Tokens to be used **(ULTIMATE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214723) in GitLab 13.1. diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index 565e905d732..c6ebce03b06 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -232,11 +232,10 @@ To enable or disable the banner: ## Required pipeline configuration **(PREMIUM SELF)** -WARNING: -This feature is being re-evaluated in favor of a different -[compliance solution](https://gitlab.com/groups/gitlab-org/-/epics/3156). -We recommend that users who haven't yet implemented this feature wait for -the new solution. +NOTE: +An alternative [compliance solution](../../project/settings/index.md#compliance-pipeline-configuration) +is available. We recommend this alternative solution because it provides greater flexibility, +allowing required pipelines to be assigned to specific compliance framework labels. You can set a [CI/CD template](../../../ci/examples/index.md#cicd-templates) as a required pipeline configuration for all projects on a GitLab instance. You can diff --git a/doc/user/admin_area/settings/files_api_rate_limits.md b/doc/user/admin_area/settings/files_api_rate_limits.md index 4f0f50dbcd2..675561ce9cf 100644 --- a/doc/user/admin_area/settings/files_api_rate_limits.md +++ b/doc/user/admin_area/settings/files_api_rate_limits.md @@ -7,13 +7,8 @@ type: reference # Files API rate limits **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68561) in GitLab 14.3. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it -available, ask an administrator to [enable the `files_api_throttling` flag](../../../administration/feature_flags.md). -On GitLab.com, this feature is available but can be configured by GitLab.com -administrators only. The feature is not ready for production use. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68561) in GitLab 14.3. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75918) in GitLab 14.6. [Feature flag files_api_throttling](https://gitlab.com/gitlab-org/gitlab/-/issues/338903) removed. The [Repository files API](../../../api/repository_files.md) enables you to fetch, create, update, and delete files in your repository. To improve the security @@ -29,10 +24,9 @@ the general user and IP rate limits for requests to the and IP rate limits already in place, and increase or decrease the rate limits for the Files API. No other new features are provided by this override. -Prerequisites: +Prerequisite: - You must have the Administrator role for your instance. -- The `files_api_throttling` feature flag must be enabled. To override the general user and IP rate limits for requests to the Repository files API: diff --git a/doc/user/admin_area/settings/gitaly_timeouts.md b/doc/user/admin_area/settings/gitaly_timeouts.md index 5da43935052..fac23cb48af 100644 --- a/doc/user/admin_area/settings/gitaly_timeouts.md +++ b/doc/user/admin_area/settings/gitaly_timeouts.md @@ -22,6 +22,6 @@ The following timeouts are available. | Timeout | Default | Description | |:--------|:-----------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Default | 55 seconds | Timeout for most Gitaly calls (not enforced for `git` `fetch` and `push` operations, or Sidekiq jobs). For example, checking if a repository exists on disk. Makes sure that Gitaly calls made within a web request cannot exceed the entire request timeout. It should be shorter than the worker timeout that can be configured for [Puma](../../../install/requirements.md#puma-settings). If a Gitaly call timeout exceeds the worker timeout, the remaining time from the worker timeout is used to avoid having to terminate the worker. | +| Default | 55 seconds | Timeout for most Gitaly calls (not enforced for `git` `fetch` and `push` operations, or Sidekiq jobs). For example, checking if a repository exists on disk. Makes sure that Gitaly calls made within a web request cannot exceed the entire request timeout. It should be shorter than the [worker timeout](../../../administration/operations/puma.md#worker-timeout) that can be configured for [Puma](../../../install/requirements.md#puma-settings). If a Gitaly call timeout exceeds the worker timeout, the remaining time from the worker timeout is used to avoid having to terminate the worker. | | Fast | 10 seconds | Timeout for fast Gitaly operations used within requests, sometimes multiple times. For example, checking if a repository exists on disk. If fast operations exceed this threshold, there may be a problem with a storage shard. Failing fast can help maintain the stability of the GitLab instance. | | Medium | 30 seconds | Timeout for Gitaly operations that should be fast (possibly within requests) but preferably not used multiple times within a request. For example, loading blobs. Timeout that should be set between Default and Fast. | diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index fd44d6445cf..1087f50b215 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -97,8 +97,8 @@ delete a project. To allow only users with the Administrator role to delete proj > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/255449) in GitLab 14.2 for groups created after August 12, 2021. -Projects in a group (but not a personal namespace) can be deleted after a delayed period. -You can [configure it in group settings](../../group/index.md#enable-delayed-project-removal). +[Delayed project deletion](../../project/settings/index.md#delayed-project-deletion) allows projects in a group (not a personal namespace) +to be deleted after a period of delay. To enable delayed project deletion by default in new groups: @@ -110,14 +110,13 @@ To enable delayed project deletion by default in new groups: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32935) in GitLab 12.6. By default, a project marked for deletion is permanently removed with immediate effect. +See [delayed project deletion](../../project/settings/index.md#delayed-project-deletion) to learn more. By default, a group marked for deletion is permanently removed after seven days. WARNING: The default behavior of [Delayed Project deletion](https://gitlab.com/gitlab-org/gitlab/-/issues/32935) in GitLab 12.6 was changed to [Immediate deletion](https://gitlab.com/gitlab-org/gitlab/-/issues/220382) in GitLab 13.2. -Projects in a group (but not a personal namespace) can be deleted after a delayed period, by -[configuring in Group Settings](../../group/index.md#enable-delayed-project-removal). The default period is seven days, and can be changed. Setting this period to `0` enables immediate removal of projects or groups. diff --git a/doc/user/analytics/ci_cd_analytics.md b/doc/user/analytics/ci_cd_analytics.md index e949f968c2b..f083f886924 100644 --- a/doc/user/analytics/ci_cd_analytics.md +++ b/doc/user/analytics/ci_cd_analytics.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/38318) to CI/CD Analytics in GitLab 12.8. GitLab tracks the history of your pipeline successes and failures, as well as how long each pipeline -ran. To view this information, go to **Analytics > CI/CD Analytics**. +ran. To view this information for a project, go to **Analytics > CI/CD Analytics**. View successful pipelines: diff --git a/doc/user/analytics/img/product_analytics_commits_per_mr_v14_4.png b/doc/user/analytics/img/product_analytics_commits_per_mr_v14_4.png Binary files differindex 649416c02c0..2bfde7beead 100644 --- a/doc/user/analytics/img/product_analytics_commits_per_mr_v14_4.png +++ b/doc/user/analytics/img/product_analytics_commits_per_mr_v14_4.png diff --git a/doc/user/analytics/img/productivity_analytics_time_to_merge_v14_4.png b/doc/user/analytics/img/productivity_analytics_time_to_merge_v14_4.png Binary files differindex b5d0dd4d2ea..0b30aff2c7a 100644 --- a/doc/user/analytics/img/productivity_analytics_time_to_merge_v14_4.png +++ b/doc/user/analytics/img/productivity_analytics_time_to_merge_v14_4.png diff --git a/doc/user/analytics/img/productivity_analytics_trendline_v14_4.png b/doc/user/analytics/img/productivity_analytics_trendline_v14_4.png Binary files differindex da5d3aec957..e0b3c54dee2 100644 --- a/doc/user/analytics/img/productivity_analytics_trendline_v14_4.png +++ b/doc/user/analytics/img/productivity_analytics_trendline_v14_4.png diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md index 0cc21e3f390..6a157dbb5ae 100644 --- a/doc/user/analytics/index.md +++ b/doc/user/analytics/index.md @@ -100,29 +100,29 @@ We use the following terms to describe GitLab analytics: - All incidents are related to a [production environment](../../ci/environments/index.md#deployment-tier-of-environments). - Incidents and deployments have a strictly one-to-one relationship (meaning any incident is related to only one production deployment, and any production deployment is related to no more than one incident). -- **Lead time:** The duration of your value stream, from start to finish. Different to -[Lead time for changes](#lead-time-for-changes). Often displayed in combination with "cycle time," -which is shorter. GitLab measures lead time from issue creation to issue close. GitLab displays lead +- **Lead time:** The duration of your value stream, from start to finish. Different to +[Lead time for changes](#lead-time-for-changes). Often displayed in combination with "cycle time," +which is shorter. GitLab measures lead time from issue creation to issue close. GitLab displays lead time in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md). -- **Mean Time to Change (MTTC):** The average duration between idea and delivery. GitLab measures +- **Mean Time to Change (MTTC):** The average duration between idea and delivery. GitLab measures MTTC from issue creation to the issue's latest related merge request's deployment to production. -- **Mean Time to Detect (MTTD):** The average duration that a bug goes undetected in production. +- **Mean Time to Detect (MTTD):** The average duration that a bug goes undetected in production. GitLab measures MTTD from deployment of bug to issue creation. -- **Mean Time To Merge (MTTM):** The average lifespan of a merge request. GitLab measures MTTM from -merge request creation to merge request merge (and closed/un-merged merge requests are excluded). +- **Mean Time To Merge (MTTM):** The average lifespan of a merge request. GitLab measures MTTM from +merge request creation to merge request merge (and closed/un-merged merge requests are excluded). For more information, see [Merge Request Analytics](merge_request_analytics.md). -- **Mean Time to Recover/Repair/Resolution/Resolve/Restore (MTTR):** The average duration that a bug +- **Mean Time to Recover/Repair/Resolution/Resolve/Restore (MTTR):** The average duration that a bug is not fixed in production. GitLab measures MTTR from deployment of bug to deployment of fix. -- **Throughput:** The number of issues closed or merge requests merged (not closed) in a period of +- **Throughput:** The number of issues closed or merge requests merged (not closed) in a period of time. Often measured per sprint. GitLab displays merge request throughput in [Merge Request Analytics](merge_request_analytics.md). -- **Value Stream:** The entire work process that is followed to deliver value to customers. For example, -the [DevOps lifecycle](https://about.gitlab.com/stages-devops-lifecycle/) is a value stream that starts +- **Value Stream:** The entire work process that is followed to deliver value to customers. For example, +the [DevOps lifecycle](https://about.gitlab.com/stages-devops-lifecycle/) is a value stream that starts with "plan" and ends with "monitor". GitLab helps you track your value stream using [Value Stream Analytics](value_stream_analytics.md). -- **Velocity:** The total issue burden completed in some period of time. The burden is usually measured -in points or weight, often per sprint. For example, your velocity may be "30 points per sprint". GitLab +- **Velocity:** The total issue burden completed in some period of time. The burden is usually measured +in points or weight, often per sprint. For example, your velocity may be "30 points per sprint". GitLab measures velocity as the total points or weight of issues closed in a given period of time. ## Lead time for changes -"Lead Time for Changes" differs from "Lead Time" because it "focuses on measuring only the time to +"Lead Time for Changes" differs from "Lead Time" because it "focuses on measuring only the time to deliver a feature once it has been developed", as described in ([Measuring DevOps Performance](https://devops.com/measuring-devops-performance/)). diff --git a/doc/user/analytics/productivity_analytics.md b/doc/user/analytics/productivity_analytics.md index da55a0f093c..e1ba2f5565e 100644 --- a/doc/user/analytics/productivity_analytics.md +++ b/doc/user/analytics/productivity_analytics.md @@ -34,14 +34,14 @@ Metrics and visualizations of **merged** merge requests are available on a proje ### Time to merge -The **Time to merge** histogram shows the number of merge requests and the number +The **Time to merge** histogram shows the number of merge requests and the number of days it took to merge after creation. Select a column to filter subsequent charts.  ### Trendline -The **Trendline** scatterplot shows all merge requests on a certain date, +The **Trendline** scatterplot shows all merge requests on a certain date, and the days it took to complete the action and a 30 day rolling median. Select the dropdown to view: - Time from first commit to first comment. @@ -55,15 +55,15 @@ and the days it took to complete the action and a 30 day rolling median. Select ### Commits and merge request size -Under the **Trendline** scatterplot, the left-side histogram shows -the time taken (in hours) between commits and comments until the merge +Under the **Trendline** scatterplot, the left-side histogram shows +the time taken (in hours) between commits and comments until the merge request is merged. Select the dropdown to view: - Time from first commit to first comment. - Time from first comment until last commit. - Time from last commit to merge. -The right-side histogram shows the size or complexity of a merge request. +The right-side histogram shows the size or complexity of a merge request. Select the dropdown to view: - Number of commits per merge request. @@ -74,7 +74,7 @@ Select the dropdown to view: ### Merge request list -The **List** table shows a list of merge requests with their respective time duration metrics. +The **List** table shows a list of merge requests with their respective time duration metrics. Sort metrics by: @@ -83,7 +83,7 @@ Sort metrics by: - Time from last commit to merge. Filter metrics by: - + - Number of commits per merge request. - Number of lines of code per commit. - Number of files touched. diff --git a/doc/user/analytics/value_stream_analytics.md b/doc/user/analytics/value_stream_analytics.md index 9c1a8893f95..cb6b2e49f60 100644 --- a/doc/user/analytics/value_stream_analytics.md +++ b/doc/user/analytics/value_stream_analytics.md @@ -102,6 +102,20 @@ The **Time** metrics near the top of the page are measured as follows: - **Lead time**: Median time from issue created to issue closed. - **Cycle time**: Median time from first commit to issue closed. (You can associate a commit with an issue by [crosslinking in the commit message](../project/issues/crosslinking_issues.md#from-commit-messages).) +- **Lead Time for Changes**: median duration between merge request merge and deployment to a production environment for all MRs deployed in the given time period. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340150) in GitLab 14.5 (Ultimate only). + +## Deployment metrics (**PREMIUM**) + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337256) in GitLab 11.3. + +Value Stream Analytics exposes two deployment related metrics near the top of the page: + +- **Deploys:** The number of successful deployments in the date range. +- **Deployment Frequency:** The average number of successful deployments. + +The deployment metrics calculation uses the same method as the +[group-level Value Stream Analytics](../group/value_stream_analytics/index.md#how-metrics-are-measured). +Both of them are based on the [DORA API](../../api/dora/metrics.md#devops-research-and-assessment-dora-key-metrics-api). ## How the stages are measured diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index 5cef0040ac3..a0f14ea59a1 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -12,6 +12,10 @@ parameters to unexpected values in an effort to cause unexpected behavior and er backend. This helps you discover bugs and potential security issues that other QA processes may miss. +INFO: +Try fuzz testing in GitLab Ultimate. +[It's free for 30 days](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=u-api-fuzzing-docs). + We recommend that you use fuzz testing in addition to [GitLab Secure](../index.md)'s other security scanners and your own test processes. If you're using [GitLab CI/CD](../../../ci/index.md), you can run fuzz tests as part your CI/CD workflow. @@ -1181,7 +1185,7 @@ A bug exists in versions of the API Fuzzing analyzer prior to v1.6.196 that can The version information can be found in the job details for the `apifuzzer_fuzz` job. -If the issue is occuring with versions v1.6.196 or greater, please contact Support and provide the following information: +If the issue is occurring with versions v1.6.196 or greater, please contact Support and provide the following information: 1. Reference this troubleshooting section and ask for the issue to be escalated to the Dynamic Analysis Team. 1. The full console output of the job. @@ -1289,6 +1293,25 @@ The API Fuzzing template supports launching a docker container containing an API TODO --> +## Get support or request an improvement + +To get support for your particular problem please use the [getting help channels](https://about.gitlab.com/get-help/). + +The [GitLab issue tracker on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues) is the right place for bugs and feature proposals about API Security and API Fuzzing. +Please use `~"Category:API Security"` [label](../../../development/contributing/issue_workflow.md#labels) when opening a new issue regarding API fuzzing to ensure it is quickly reviewed by the right people. Please refer to our [review response SLO](../../../development/code_review.md#review-response-slo) to understand when you should receive a response. + +[Search the issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) for similar entries before submitting your own, there's a good chance somebody else had the same issue or feature proposal. Show your support with an award emoji and or join the discussion. + +When experiencing a behavior not working as expected, consider providing contextual information: + +- GitLab version if using a self-managed instance. +- `.gitlab-ci.yml` job definition. +- Full job console output. +- Scanner log file available as a job artifact named `gl-api-security-scanner.log`. + +WARNING: +**Sanitize data attached to a support issue**. Please remove sensitive information, including: credentials, passwords, tokens, keys, and secrets. + ## Glossary - Assert: Assertions are detection modules used by checks to trigger a fault. Many assertions have diff --git a/doc/user/application_security/cluster_image_scanning/index.md b/doc/user/application_security/cluster_image_scanning/index.md index 790b428bac9..c3a2c179590 100644 --- a/doc/user/application_security/cluster_image_scanning/index.md +++ b/doc/user/application_security/cluster_image_scanning/index.md @@ -29,7 +29,7 @@ To integrate GitLab with security scanners other than those listed here, see You can use cluster image scanning through the following methods: - [The cluster image scanning analyzer](#use-the-cluster-image-scanning-analyzer) -- [The GitLab Kubernetes agent](#cluster-image-scanning-with-the-gitlab-kubernetes-agent) +- [The GitLab Agent](#cluster-image-scanning-with-the-gitlab-agent) ## Use the cluster image scanning analyzer @@ -153,7 +153,7 @@ The included template: fetches vulnerabilities found by [Starboard Operator](https://aquasecurity.github.io/starboard/v0.10.3/operator/). GitLab saves the results as a -[Cluster Image Scanning report artifact](../../../ci/yaml/index.md#artifactsreportscluster_image_scanning) +[Cluster Image Scanning report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportscluster_image_scanning) that you can download and analyze later. When downloading, you always receive the most recent artifact. @@ -177,6 +177,7 @@ You can [configure](#customize-the-cluster-image-scanning-settings) analyzers by | `CIS_RESOURCE_NAMESPACE` | `""` | Namespace of the Kubernetes resource you want to filter vulnerabilities for. For example, `production`. | | `CIS_RESOURCE_KIND` | `""` | Kind of the Kubernetes resource you want to filter vulnerabilities for. For example, `deployment`. | | `CIS_CLUSTER_IDENTIFIER` | `""` | ID of the Kubernetes cluster integrated with GitLab. This is used to map vulnerabilities to the cluster so they can be filtered in the Vulnerability Report page. | +| `CIS_CLUSTER_AGENT_IDENTIFIER` | `""` | ID of the Kubernetes cluster agent integrated with GitLab. This maps vulnerabilities to the agent so they can be filtered in the Vulnerability Report page. | #### Override the cluster image scanning template @@ -274,26 +275,22 @@ Here's an example cluster image scanning report: } ``` -## Cluster image scanning with the GitLab Kubernetes Agent +## Cluster image scanning with the GitLab Agent -You can use the [GitLab Kubernetes Agent](../../clusters/agent/index.md) to +You can use the [GitLab Agent](../../clusters/agent/index.md) to scan images from within your Kubernetes cluster and record the vulnerabilities in GitLab. ### Prerequisites - [Starboard Operator](https://aquasecurity.github.io/starboard/v0.10.3/operator/installation/kubectl/) installed and configured in your cluster. -- [GitLab Kubernetes Agent](../../clusters/agent/install/index.md) +- [GitLab Agent](../../clusters/agent/install/index.md) set up in GitLab, installed in your cluster, and configured using a configuration repository. ### Configuration -The GitLab Kubernetes agent begins to run cluster image scanning once the `cluster_image_scanning` -directive is added to your Kubernetes Agent configuration repository. - -See the [Kubernetes agent configuration repository](../../clusters/agent/repository.md#scan-your-container-images-for-vulnerabilities) -reference to learn more about the cluster image scanning configuration options for the -GitLab Kubernetes agent. +The Agent runs the cluster image scanning once the `cluster_image_scanning` +directive is added to your [Agent's configuration repository](../../clusters/agent/repository.md#scan-your-container-images-for-vulnerabilities). ## Security Dashboard diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index a913d5fba92..cdcd334dba6 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -46,14 +46,14 @@ You can configure the following security controls: - Select **Manage scans** to manage the saved DAST scans, site profiles, and scanner profiles. For more details, read [DAST on-demand scans](../dast/index.md#on-demand-scans). - [Dependency Scanning](../dependency_scanning/index.md) - - Select **Configure via Merge Request** to create a merge request with the changes required to + - Select **Configure with a merge request** to create a merge request with the changes required to enable Dependency Scanning. For more details, see [Enable Dependency Scanning via an automatic merge request](../dependency_scanning/index.md#enable-dependency-scanning-via-an-automatic-merge-request). - [Container Scanning](../container_scanning/index.md) - Can be configured with `.gitlab-ci.yml`. For more details, read [Container Scanning](../../../user/application_security/container_scanning/index.md#configuration). - [Cluster Image Scanning](../cluster_image_scanning/index.md) - Can be configured with `.gitlab-ci.yml`. For more details, read [Cluster Image Scanning](../../../user/application_security/cluster_image_scanning/#configuration). - [Secret Detection](../secret_detection/index.md) - - Select **Configure via Merge Request** to create a merge request with the changes required to + - Select **Configure with a merge request** to create a merge request with the changes required to enable Secret Detection. For more details, read [Enable Secret Detection via an automatic merge request](../secret_detection/index.md#enable-secret-detection-via-an-automatic-merge-request). - [API Fuzzing](../api_fuzzing/index.md) - Select **Enable API Fuzzing** to use API Fuzzing for the current project. For more details, read [API Fuzzing](../../../user/application_security/api_fuzzing/index.md#enable-web-api-fuzzing). diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index da2816ab6ed..bea9284873c 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -9,11 +9,27 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3672) in GitLab 10.4. +INFO: +Try out Container Scanning in GitLab Ultimate. +[It's free for 30 days](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=u-container-scanning-docs). + Your application's Docker image may itself be based on Docker images that contain known -vulnerabilities. By including an extra job in your pipeline that scans for those vulnerabilities and -displays them in a merge request, you can use GitLab to audit your Docker-based apps. +vulnerabilities. By including an extra Container Scanning job in your pipeline that scans for those +vulnerabilities and displays them in a merge request, you can use GitLab to audit your Docker-based +apps. + +Container Scanning is often considered part of Software Composition Analysis (SCA). SCA can contain +aspects of inspecting the items your code uses. These items typically include application and system +dependencies that are almost always imported from external sources, rather than sourced from items +you wrote yourself. + +GitLab offers both Container Scanning and [Dependency Scanning](../dependency_scanning/) +to ensure coverage for all of these dependency types. To cover as much of your risk area as +possible, we encourage you to use all of our security scanners. -GitLab provides integration with open-source tools for vulnerability static analysis in containers: +## Overview + +GitLab integrates with open-source tools for vulnerability static analysis in containers: - [Trivy](https://github.com/aquasecurity/trivy) - [Grype](https://github.com/anchore/grype) @@ -43,19 +59,9 @@ To enable container scanning in your pipeline, you need the following: - An image matching the [supported distributions](#supported-distributions). - [Build and push](../../packages/container_registry/index.md#build-and-push-by-using-gitlab-cicd) the Docker image to your project's container registry. -- The name of the Docker image to scan, in the `DOCKER_IMAGE` [configuration variable](#available-cicd-variables). - If you're using a third-party container registry, you might need to provide authentication credentials through the `DOCKER_USER` and `DOCKER_PASSWORD` [configuration variables](#available-cicd-variables). - For example, if you are connecting to AWS ECR, you might use the following: - -```yaml -export AWS_ECR_PASSWORD=$(aws ecr get-login-password --region region) - -include: - - template: Security/Container-Scanning.gitlab-ci.yml - DOCKER_USER: AWS - DOCKER_PASSWORD: "$AWS_ECR_PASSWORD" -``` + For more details on how to use these variables, see [authenticate to a remote registry](#authenticate-to-a-remote-registry). ## Configuration @@ -75,31 +81,29 @@ The included template: (see [requirements](#requirements)) and scans it for possible vulnerabilities. GitLab saves the results as a -[Container Scanning report artifact](../../../ci/yaml/index.md#artifactsreportscontainer_scanning) +[Container Scanning report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportscontainer_scanning) that you can download and analyze later. When downloading, you always receive the most-recent -artifact. +artifact. If [dependency scan is enabled](#dependency-list), +a [Dependency Scanning report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsdependency_scanning) +is also created. The following is a sample `.gitlab-ci.yml` that builds your Docker image, pushes it to the container registry, and scans the image: ```yaml -build: - image: docker:latest - stage: build - services: - - docker:dind - variables: - IMAGE: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA - script: - - docker info - - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - - docker build -t $IMAGE . - - docker push $IMAGE - include: + - template: Jobs/Build.gitlab-ci.yml - template: Security/Container-Scanning.gitlab-ci.yml + +container_scanning: + variables: + CS_DEFAULT_BRANCH_IMAGE: $CI_REGISTRY_IMAGE/$CI_DEFAULT_BRANCH:$CI_COMMIT_SHA ``` +Setting `CS_DEFAULT_BRANCH_IMAGE` avoids duplicate vulnerability findings when an image name differs across branches. +The value of `CS_DEFAULT_BRANCH_IMAGE` indicates the name of the scanned image as it appears on the default branch. +For more details on how this deduplication is achieved, see [Setting the default branch image](#setting-the-default-branch-image). + ### Customizing the container scanning settings There may be cases where you want to customize how GitLab scans your containers. For example, you @@ -120,6 +124,92 @@ variables: SECURE_LOG_LEVEL: 'debug' ``` +#### Scan an image in a remote registry + +To scan images located in a registry other than the project's, use the following `.gitlab-ci.yml`: + +```yaml +include: + - template: Security/Container-Scanning.gitlab-ci.yml + +container_scanning: + variables: + DOCKER_IMAGE: example.com/user/image:tag +``` + +##### Authenticate to a remote registry + +Scanning an image in a private registry requires authentication. Provide the username in the `DOCKER_USER` +variable, and the password in the `DOCKER_PASSWORD` configuration variable. + +For example, to scan an image from AWS Elastic Container Registry: + +```yaml +container_scanning: + before_script: + - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" --output "awscliv2.zip" + - unzip awscliv2.zip + - ./aws/install + - aws --version + - export AWS_ECR_PASSWORD=$(aws ecr get-login-password --region region) + +include: + - template: Security/Container-Scanning.gitlab-ci.yml + DOCKER_IMAGE: <aws_account_id>.dkr.ecr.<region>.amazonaws.com/<image>:<tag> + DOCKER_USER: AWS + DOCKER_PASSWORD: "$AWS_ECR_PASSWORD" +``` + +#### Dependency list + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345434) in GitLab 14.6. + +The `CS_DISABLE_DEPENDENCY_LIST` CI/CD variable controls whether the scan creates a +[Dependency List](../dependency_list/) +report. The variable's default setting of `false` causes the scan to create the report. To disable +the report, set the variable to `true`: + +For example: + +```yaml +include: + - template: Security/Container-Scanning.gitlab-ci.yml + +container_scanning: + variables: + CS_DISABLE_DEPENDENCY_LIST: "true" +``` + +#### Report language-specific findings + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7277) in GitLab 14.6. + +The `CS_DISABLE_LANGUAGE_VULNERABILITY_SCAN` CI/CD variable controls whether the scan reports +findings related to programming languages. The languages supported depend on the +[scanner used](#change-scanners): + +- [Trivy](https://aquasecurity.github.io/trivy/latest/vulnerability/detection/language/). +- [Grype](https://github.com/anchore/grype#features). + +By default, the report only includes packages managed by the Operating System (OS) package manager +(for example, `yum`, `apt`, `apk`, `tdnf`). To report security findings in non-OS packages, set +`CS_DISABLE_LANGUAGE_VULNERABILITY_SCAN` to `"false"`: + +```yaml +include: + - template: Security/Container-Scanning.gitlab-ci.yml + +container_scanning: + variables: + CS_DISABLE_LANGUAGE_VULNERABILITY_SCAN: "false" +``` + +When you enable this feature, you may see [duplicate findings](../terminology/#duplicate-finding) +in the [Vulnerability Report](../vulnerability_report/) +if [Dependency Scanning](../dependency_scanning/) +is enabled for your project. This happens because GitLab can't automatically deduplicate the +findings reported by the two different analyzers. + #### Available CI/CD variables You can [configure](#customizing-the-container-scanning-settings) analyzers by using the following CI/CD variables: @@ -130,6 +220,9 @@ You can [configure](#customizing-the-container-scanning-settings) analyzers by u | `CI_APPLICATION_REPOSITORY` | `$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG` | Docker repository URL for the image to be scanned. | All | | `CI_APPLICATION_TAG` | `$CI_COMMIT_SHA` | Docker repository tag for the image to be scanned. | All | | `CS_ANALYZER_IMAGE` | `registry.gitlab.com/security-products/container-scanning:4` | Docker image of the analyzer. | All | +| `CS_DEFAULT_BRANCH_IMAGE` | `""` | The name of the `DOCKER_IMAGE` on the default branch. See [Setting the default branch image](#setting-the-default-branch-image) for more details. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338877) in GitLab 14.5. | All | +| `CS_DISABLE_DEPENDENCY_LIST` | `"false"` | Disable Dependency Scanning for packages installed in the scanned image. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345434) in GitLab 14.6. | All | +| `CS_DISABLE_LANGUAGE_VULNERABILITY_SCAN` | `"true"` | Disable scanning for language-specific packages installed in the scanned image. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345434) in GitLab 14.6. | All | | `CS_DOCKER_INSECURE` | `"false"` | Allow access to secure Docker registries using HTTPS without validating the certificates. | All | | `CS_REGISTRY_INSECURE` | `"false"` | Allow access to insecure registries (HTTP only). Should only be set to `true` when testing the image locally. Works with all scanners, but the registry must listen on port `80/tcp` for Trivy to work. | All | | `CS_SEVERITY_THRESHOLD` | `UNKNOWN` | Severity level threshold. The scanner outputs vulnerabilities with severity level higher than or equal to this threshold. Supported levels are Unknown, Low, Medium, High, and Critical. | Trivy | @@ -225,6 +318,51 @@ Prior to the GitLab 14.0 release, any variable defined under the scope `containe considered for scanners other than Clair. In GitLab 14.0 and later, all variables can be defined either as a global variable or under `container_scanning`. +### Setting the default branch image + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338877) in GitLab 14.5. + +By default, container scanning assumes that the image naming convention stores any branch-specific +identifiers in the image tag rather than the image name. When the image name differs between the +default branch and the non-default branch, previously-detected vulnerabilities show up as newly +detected in merge requests. + +When the same image has different names on the default branch and a non-default branch, you can use +the `CS_DEFAULT_BRANCH_IMAGE` variable to indicate what that image's name is on the default branch. +GitLab then correctly determines if a vulnerability already exists when running scans on non-default +branches. + +As an example, suppose the following: + +- Non-default branches publish images with the naming convention + `$CI_REGISTRY_IMAGE/$CI_COMMIT_BRANCH:$CI_COMMIT_SHA`. +- The default branch publishes images with the naming convention + `$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA`. + +In this example, you can use the following CI/CD configuration to ensure that vulnerabilities aren't +duplicated: + +```yaml +include: + - template: Security/Container-Scanning.gitlab-ci.yml + +container_scanning: + variables: + CS_DEFAULT_BRANCH_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA + before_script: + - export DOCKER_IMAGE="$CI_REGISTRY_IMAGE/$CI_COMMIT_BRANCH:$CI_COMMIT_SHA" + - | + if [ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]; then + export DOCKER_IMAGE="$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" + fi +``` + +`CS_DEFAULT_BRANCH_IMAGE` should remain the same for a given `DOCKER_IMAGE`. If it changes, then a +duplicate set of vulnerabilities are created, which must be manually dismissed. + +When using [Auto DevOps](../../../topics/autodevops/index.md), `CS_DEFAULT_BRANCH_IMAGE` is +automatically set to `$CI_REGISTRY_IMAGE/$CI_DEFAULT_BRANCH:$CI_APPLICATION_TAG`. + ### Using a custom SSL CA certificate authority You can use the `ADDITIONAL_CA_CERT_BUNDLE` CI/CD variable to configure a custom SSL CA certificate authority, which is used to verify the peer when fetching Docker images from a registry which uses HTTPS. The `ADDITIONAL_CA_CERT_BUNDLE` value should contain the [text representation of the X.509 PEM public-key certificate](https://tools.ietf.org/html/rfc7468#section-5.1). For example, to configure this value in the `.gitlab-ci.yml` file, use the following: @@ -496,7 +634,7 @@ Here's an example container scanning report: ```json-doc { - "version": "3.0.0", + "version": "14.0.0", "vulnerabilities": [ { "id": "df52bc8ce9a2ae56bbcb0c4ecda62123fbd6f69b", @@ -518,7 +656,8 @@ Here's an example container scanning report: "version": "1.4.8" }, "operating_system": "debian:9.4", - "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e" + "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e", + "default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest" }, "identifiers": [ { diff --git a/doc/user/application_security/coverage_fuzzing/index.md b/doc/user/application_security/coverage_fuzzing/index.md index 9bb13d26d90..b35c2ed79cf 100644 --- a/doc/user/application_security/coverage_fuzzing/index.md +++ b/doc/user/application_security/coverage_fuzzing/index.md @@ -213,7 +213,7 @@ is a good way to balance the needs of letting a developer's per-commit pipeline and also giving the fuzzer a large amount of time to fully explore and test the app. Long-running fuzzing jobs are usually necessary for the coverage guided fuzzer to find deeper bugs -in your latest codebase. THe following is an example of what `.gitlab-ci.yml` looks like in this +in your latest codebase. The following is an example of what `.gitlab-ci.yml` looks like in this workflow (for the full example, see the [repository](https://gitlab.com/gitlab-org/security-products/demos/coverage-fuzzing/go-fuzzing-example/-/tree/continuous_fuzzing)): ```yaml diff --git a/doc/user/application_security/dast/browser_based.md b/doc/user/application_security/dast/browser_based.md index 10ca3430b48..5d1e57553f4 100644 --- a/doc/user/application_security/dast/browser_based.md +++ b/doc/user/application_security/dast/browser_based.md @@ -61,14 +61,15 @@ The browser-based crawler can be configured using CI/CD variables. | `DAST_BROWSER_NUMBER_OF_BROWSERS` | number | `3` | The maximum number of concurrent browser instances to use. For shared runners on GitLab.com, we recommended a maximum of three. Private runners with more resources may benefit from a higher number, but are likely to produce little benefit after five to seven instances. | | `DAST_BROWSER_COOKIES` | dictionary | `abtesting_group:3,region:locked` | A cookie name and value to be added to every request. | | `DAST_BROWSER_LOG` | List of strings | `brows:debug,auth:debug` | A list of modules and their intended log level. | -| `DAST_BROWSER_NAVIGATION_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `15s` | The maximum amount of time to wait for a browser to navigate from one page to another. | -| `DAST_BROWSER_ACTION_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to complete an action. | -| `DAST_BROWSER_STABILITY_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis. | -| `DAST_BROWSER_NAVIGATION_STABILITY_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after a navigation completes. | -| `DAST_BROWSER_ACTION_STABILITY_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `800ms` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after completing an action. | -| `DAST_BROWSER_SEARCH_ELEMENT_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `3s` | The maximum amount of time to allow the browser to search for new elements or navigations. | -| `DAST_BROWSER_EXTRACT_ELEMENT_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `5s` | The maximum amount of time to allow the browser to extract newly found elements or navigations. | -| `DAST_BROWSER_ELEMENT_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `600ms` | The maximum amount of time to wait for an element before determining it is ready for analysis. | +| `DAST_BROWSER_NAVIGATION_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `15s` | The maximum amount of time to wait for a browser to navigate from one page to another. | +| `DAST_BROWSER_ACTION_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to complete an action. | +| `DAST_BROWSER_STABILITY_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis. | +| `DAST_BROWSER_NAVIGATION_STABILITY_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after a navigation completes. | +| `DAST_BROWSER_ACTION_STABILITY_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `800ms` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after completing an action. | +| `DAST_BROWSER_SEARCH_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `3s` | The maximum amount of time to allow the browser to search for new elements or navigations. | +| `DAST_BROWSER_EXTRACT_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `5s` | The maximum amount of time to allow the browser to extract newly found elements or navigations. | +| `DAST_BROWSER_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `600ms` | The maximum amount of time to wait for an element before determining it is ready for analysis. | +| `DAST_BROWSER_PAGE_READY_SELECTOR` | selector | `css:#page-is-ready` | Selector that when detected as visible on the page, indicates to the analyzer that the page has finished loading and the scan can continue. Note: When this selector is set, but the element is not found, the scanner waits for the period defined in `DAST_BROWSER_STABILITY_TIMEOUT` before continuing the scan. This can significantly increase scanning time if the element is not present on multiple pages within the site. | The [DAST variables](index.md#available-cicd-variables) `SECURE_ANALYZERS_PREFIX`, `DAST_FULL_SCAN_ENABLED`, `DAST_AUTO_UPDATE_ADDONS`, `DAST_EXCLUDE_RULES`, `DAST_REQUEST_HEADERS`, `DAST_HTML_REPORT`, `DAST_MARKDOWN_REPORT`, `DAST_XML_REPORT`, `DAST_AUTH_URL`, `DAST_USERNAME`, `DAST_PASSWORD`, `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, `DAST_FIRST_SUBMIT_FIELD`, `DAST_SUBMIT_FIELD`, `DAST_EXCLUDE_URLS`, `DAST_AUTH_VERIFICATION_URL`, `DAST_BROWSER_AUTH_VERIFICATION_SELECTOR`, `DAST_BROWSER_AUTH_VERIFICATION_LOGIN_FORM`, `DAST_BROWSER_AUTH_REPORT`, @@ -99,7 +100,7 @@ You can manage the trade-off between coverage and scan time with the following m Due to poor network conditions or heavy application load, the default timeouts may not be applicable to your application. -Browser-based scans offer the ability to adjust various timeouts to ensure it continues smoothly as it transitions from one page to the next. These values are configured using a [Duration string](https://golang.org/pkg/time/#ParseDuration), which allow you to configure durations with a prefix: `m` for minutes, `s` for seconds, and `ms` for milliseconds. +Browser-based scans offer the ability to adjust various timeouts to ensure it continues smoothly as it transitions from one page to the next. These values are configured using a [Duration string](https://pkg.go.dev/time#ParseDuration), which allow you to configure durations with a prefix: `m` for minutes, `s` for seconds, and `ms` for milliseconds. Navigations, or the act of loading a new page, usually require the most amount of time because they are loading multiple new resources such as JavaScript or CSS files. Depending on the size of these resources, or the speed at which they are returned, the default `DAST_BROWSER_NAVIGATION_TIMEOUT` may not be sufficient. diff --git a/doc/user/application_security/dast/checks/1004.1.md b/doc/user/application_security/dast/checks/1004.1.md index cbbcea1d34d..dfbc600b05b 100644 --- a/doc/user/application_security/dast/checks/1004.1.md +++ b/doc/user/application_security/dast/checks/1004.1.md @@ -36,6 +36,6 @@ Set-Cookie: {cookie_name}=<random secure value>; HttpOnly ## Links -- [owasp](https://owasp.org/www-community/HttpOnly) -- [cwe](https://cwe.mitre.org/data/definitions/1004.html) +- [OWASP](https://owasp.org/www-community/HttpOnly) +- [CWE](https://cwe.mitre.org/data/definitions/1004.html) - [Mozilla MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies) diff --git a/doc/user/application_security/dast/checks/16.1.md b/doc/user/application_security/dast/checks/16.1.md index bb030d2f9c4..157b2b96ed4 100644 --- a/doc/user/application_security/dast/checks/16.1.md +++ b/doc/user/application_security/dast/checks/16.1.md @@ -29,5 +29,5 @@ header to disable user agents from mis-interpreting resources. ## Links -- [cwe](https://cwe.mitre.org/data/definitions/16.html) +- [CWE](https://cwe.mitre.org/data/definitions/16.html) - [Mozilla Blog on MIME Confusion attacks](https://blog.mozilla.org/security/2016/08/26/mitigating-mime-confusion-attacks-in-firefox/) diff --git a/doc/user/application_security/dast/checks/16.3.md b/doc/user/application_security/dast/checks/16.3.md index e4dcf3ece4b..6f80a2a32c6 100644 --- a/doc/user/application_security/dast/checks/16.3.md +++ b/doc/user/application_security/dast/checks/16.3.md @@ -31,5 +31,5 @@ information from the `X-Powered-By` header. ## Links -- [cwe](https://cwe.mitre.org/data/definitions/16.html) -- [PHP expose_php](https://www.php.net/manual/en/ini.core.php#ini.expose-php) +- [CWE](https://cwe.mitre.org/data/definitions/16.html) +- [PHP `expose_php`](https://www.php.net/manual/en/ini.core.php#ini.expose-php) diff --git a/doc/user/application_security/dast/checks/16.4.md b/doc/user/application_security/dast/checks/16.4.md index c0161c910b0..1f72a80cb29 100644 --- a/doc/user/application_security/dast/checks/16.4.md +++ b/doc/user/application_security/dast/checks/16.4.md @@ -25,4 +25,4 @@ Consult your proxy/load balancer documentation or provider on how to disable rev ## Links -- [cwe](https://cwe.mitre.org/data/definitions/16.html) +- [CWE](https://cwe.mitre.org/data/definitions/16.html) diff --git a/doc/user/application_security/dast/checks/16.5.md b/doc/user/application_security/dast/checks/16.5.md index 8a6f3cd8b6a..28bb9f7ee4b 100644 --- a/doc/user/application_security/dast/checks/16.5.md +++ b/doc/user/application_security/dast/checks/16.5.md @@ -4,7 +4,7 @@ group: Dynamic Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# AspNet Header(s) exposes version information +# AspNet header exposes version information ## Description @@ -26,5 +26,5 @@ section of the `Web.config` file. ## Links -- [cwe](https://cwe.mitre.org/data/definitions/16.html) +- [CWE](https://cwe.mitre.org/data/definitions/16.html) - [IIS Remove Unwanted Headers](https://techcommunity.microsoft.com/t5/iis-support-blog/remove-unwanted-http-response-headers/ba-p/369710) diff --git a/doc/user/application_security/dast/checks/16.6.md b/doc/user/application_security/dast/checks/16.6.md new file mode 100644 index 00000000000..ddd3a10c5f8 --- /dev/null +++ b/doc/user/application_security/dast/checks/16.6.md @@ -0,0 +1,37 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# AspNetMvc header exposes version information + +## Description + +The target website returns AspNet header(s) along with version information of this website. By +exposing these values attackers may attempt to identify if the target software is vulnerable to known +vulnerabilities. Or catalog known sites running particular versions to exploit in the future when a +vulnerability is identified in the particular version. + +## Remediation + +To remove the `X-AspNetMvc-Version` information set `MvcHandler.DisableMvcResponseHeader = true;` in the +`Global.asax.cs` file in the `Application_Start()` method. + +```cs +protected void Application_Start() +{ + MvcHandler.DisableMvcResponseHeader = true; +} +``` + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 16.6 | true | 16 | Passive | Low | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/16.html) +- [IIS Remove Unwanted Headers](https://techcommunity.microsoft.com/t5/iis-support-blog/remove-unwanted-http-response-headers/ba-p/369710) diff --git a/doc/user/application_security/dast/checks/614.1.md b/doc/user/application_security/dast/checks/614.1.md index 74ac73935f1..46f7f61b0c7 100644 --- a/doc/user/application_security/dast/checks/614.1.md +++ b/doc/user/application_security/dast/checks/614.1.md @@ -36,5 +36,5 @@ Set-Cookie: {cookie_name}=<random secure value>; Secure ## Links - [owasp](https://owasp.org/www-community/controls/SecureCookieAttribute) -- [cwe](https://cwe.mitre.org/data/definitions/614.html) +- [CWE](https://cwe.mitre.org/data/definitions/614.html) - [Mozilla MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies) diff --git a/doc/user/application_security/dast/checks/693.1.md b/doc/user/application_security/dast/checks/693.1.md index 07cb368b39a..d3f4c72c676 100644 --- a/doc/user/application_security/dast/checks/693.1.md +++ b/doc/user/application_security/dast/checks/693.1.md @@ -30,7 +30,7 @@ misinterpreted. ## Links -- [owasp](https://owasp.org/www-project-secure-headers/#x-content-type-options) -- [cwe](https://cwe.mitre.org/data/definitions/693.html) +- [OWASP](https://owasp.org/www-project-secure-headers/#x-content-type-options) +- [CWE](https://cwe.mitre.org/data/definitions/693.html) - [Mozilla Blog on MIME Confusion attacks](https://blog.mozilla.org/security/2016/08/26/mitigating-mime-confusion-attacks-in-firefox/) - [Mozilla MDN on X-Content-Type-Options](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options) diff --git a/doc/user/application_security/dast/checks/index.md b/doc/user/application_security/dast/checks/index.md index f1a68387eb1..a3b89e09751 100644 --- a/doc/user/application_security/dast/checks/index.md +++ b/doc/user/application_security/dast/checks/index.md @@ -15,6 +15,7 @@ The [DAST browser-based crawler](../browser_based.md) provides a number of vulne | [16.2](16.2.md) | Server header exposes version information | Low | Passive | | [16.3](16.3.md) | X-Powered-By header exposes version information | Low | Passive | | [16.4](16.4.md) | X-Backend-Server header exposes server information | Info | Passive | -| [16.5](16.5.md) | AspNet Header(s) exposes version information | Low | Passive | +| [16.5](16.5.md) | AspNet header exposes version information | Low | Passive | +| [16.6](16.6.md) | AspNetMvc header exposes version information | Low | Passive | | [614.1](614.1.md) | Sensitive cookie without `Secure` attribute | Low | Passive | | [693.1](693.1.md) | Missing X-Content-Type-Options: nosniff | Low | Passive | diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 0d8b55a92a9..4de7a566769 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -16,6 +16,10 @@ Dynamic Application Security Testing (DAST) examines applications for vulnerabilities like these in deployed environments. DAST uses the open source tool [OWASP Zed Attack Proxy](https://www.zaproxy.org/) for analysis. +INFO: +Want to try out security scanning? +[Try GitLab Ultimate free for 30 days](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=u-dast-docs). + After DAST creates its report, GitLab evaluates it for discovered vulnerabilities between the source and target branches. Relevant findings are noted in the merge request. @@ -254,7 +258,7 @@ The included template creates a `dast` job in your CI/CD pipeline and scans your project's running application for possible vulnerabilities. The results are saved as a -[DAST report artifact](../../../ci/yaml/index.md#artifactsreportsdast) +[DAST report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsdast) that you can later download and analyze. Due to implementation limitations, we always take the latest DAST artifact available. Behind the scenes, the [GitLab DAST Docker image](https://gitlab.com/security-products/dast) @@ -956,9 +960,34 @@ An on-demand scan can be run in active or passive mode: ### View on-demand DAST scans -To view running and completed on-demand DAST scans for a project, go to +To view running completed and scheduled on-demand DAST scans for a project, go to **Security & Compliance > On-demand Scans** in the left sidebar. +- To view both running and completed scans, select **All**. +- To view running scans only, select **Running**. +- To view finished scans, select **Finished**. A finished scan is a scan that either succeeded, + failed, or was canceled. +- To view scheduled scans, select **Scheduled**. It shows on-demand scans that have a schedule + set up. Those are _not_ included in the **All** tab. + +#### Cancel an on-demand scan + +To cancel a pending or running on-demand scan, select **Cancel** (**{cancel}**) in the +on-demand scans list. + +#### Retry an on-demand scan + +To retry a scan that failed or succeeded with warnings, select **Retry** (**{retry}**) in the +on-demand scans list. + +#### View an on-demand scan's results + +To view a finished scan's results, select **View results** in the on-demand scans list. + +#### Edit an on-demand scan + +To edit an on-demand scan's settings, select **Edit** (**{pencil}**) in the **Scheduled** tab. + ### Run an on-demand DAST scan Prerequisites: @@ -1023,7 +1052,7 @@ The on-demand DAST scan runs, and the project's dashboard shows the results. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.3. [Deployed behind the `dast_on_demand_scans_scheduler` flag](../../../administration/feature_flags.md), disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. -> - [Feature flag dast_on_demand_scans_scheduler removed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.5. +> - [Feature flag `dast_on_demand_scans_scheduler` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.5. To schedule a scan: @@ -1344,27 +1373,6 @@ The DAST tool always emits a JSON report file called `gl-dast-report.json` and sample reports can be found in the [DAST repository](https://gitlab.com/gitlab-org/security-products/dast/-/tree/master/test/end-to-end/expect). -### Other formats - -Reports can also be generated in Markdown, HTML, and XML. These can be published as artifacts using the following configuration: - -```yaml -include: - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_HTML_REPORT: report.html - DAST_MARKDOWN_REPORT: report.md - DAST_XML_REPORT: report.xml - artifacts: - paths: - - $DAST_HTML_REPORT - - $DAST_MARKDOWN_REPORT - - $DAST_XML_REPORT - - gl-dast-report.json -``` - ## Optimizing DAST By default, DAST downloads all artifacts defined by previous jobs in the pipeline. If diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index f3ab25ccffa..0db5fb2d868 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -1138,7 +1138,7 @@ A bug exists in versions of the DAST API analyzer prior to v1.6.196 that can cau The version information can be found in the job details for the `dast_api` job. -If the issue is occuring with versions v1.6.196 or greater, please contact Support and provide the following information: +If the issue is occurring with versions v1.6.196 or greater, please contact Support and provide the following information: 1. Reference this troubleshooting section and ask for the issue to be escalated to the Dynamic Analysis Team. 1. The full console output of the job. @@ -1209,6 +1209,25 @@ deploy-test-target: - environment_url.txt ``` +## Get support or request an improvement + +To get support for your particular problem please use the [getting help channels](https://about.gitlab.com/get-help/). + +The [GitLab issue tracker on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues) is the right place for bugs and feature proposals about API Security and DAST API. +Please use `~"Category:API Security"` [label](../../../development/contributing/issue_workflow.md#labels) when opening a new issue regarding DAST API to ensure it is quickly reviewed by the right people. Please refer to our [review response SLO](../../../development/code_review.md#review-response-slo) to understand when you should receive a response. + +[Search the issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) for similar entries before submitting your own, there's a good chance somebody else had the same issue or feature proposal. Show your support with an award emoji and or join the discussion. + +When experiencing a behavior not working as expected, consider providing contextual information: + +- GitLab version if using a self-managed instance. +- `.gitlab-ci.yml` job definition. +- Full job console output. +- Scanner log file available as a job artifact named `gl-api-security-scanner.log`. + +WARNING: +**Sanitize data attached to a support issue**. Please remove sensitive information, including: credentials, passwords, tokens, keys, and secrets. + ## Glossary - Assert: Assertions are detection modules used by checks to trigger a vulnerability. Many assertions have diff --git a/doc/user/application_security/dependency_list/index.md b/doc/user/application_security/dependency_list/index.md index b0d8af2606f..baafdcda6e0 100644 --- a/doc/user/application_security/dependency_list/index.md +++ b/doc/user/application_security/dependency_list/index.md @@ -7,7 +7,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Dependency list **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10075) in GitLab 12.0. +> - Application dependencies [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10075) in GitLab 12.0. +> - System dependencies [introduced](https://gitlab.com/groups/gitlab-org/-/epics/6698) in GitLab 14.6. Use the dependency list to review your project's dependencies and key details about those dependencies, including their known vulnerabilities. It is a collection of dependencies in your project, including existing and new findings. @@ -22,8 +23,9 @@ The dependency list only shows the results of the last successful pipeline to ru To view your project's dependencies, ensure you meet the following requirements: -- The [Dependency Scanning](../dependency_scanning/index.md) CI job must be - configured for your project. +- The [Dependency Scanning](../dependency_scanning/index.md) + or [Container Scanning](../container_scanning/index.md) + CI job must be configured for your project. - Your project uses at least one of the [languages and package managers](../dependency_scanning/index.md#supported-languages-and-package-managers) supported by Gemnasium. @@ -38,7 +40,7 @@ GitLab displays dependencies with the following information: |-----------|-------------| | Component | The dependency's name and version. | | Packager | The packager used to install the dependency. | -| Location | A link to the packager-specific lock file in your project that declared the dependency. It also shows the [dependency path](#dependency-paths) to a top-level dependency, if any, and if supported. | +| Location | For system dependencies, this lists the image that was scanned. For application dependencies, this shows a link to the packager-specific lock file in your project that declared the dependency. It also shows the [dependency path](#dependency-paths) to a top-level dependency, if any, and if supported. | | License | Links to dependency's software licenses. | Displayed dependencies are initially sorted by the severity of their known vulnerabilities, if any. They @@ -63,6 +65,7 @@ Dependency paths are supported for the following package managers: - [NuGet](https://www.nuget.org/) - [Yarn 1.x](https://classic.yarnpkg.com/lang/en/) +- [sbt](https://www.scala-sbt.org) ## Licenses @@ -82,4 +85,4 @@ You can download your project's list of dependencies and their details in JSON f ### Using the API -You can download your project's list of dependencies [using the API](../../../api/dependencies.md#list-project-dependencies). Note this only provides the dependencies identified by the gemnasium family of analyzers and [not any other of the GitLab dependency analyzers](../dependency_scanning/analyzers.md). +You can download your project's list of dependencies [using the API](../../../api/dependencies.md#list-project-dependencies). Note this only provides the dependencies identified by the Gemnasium family of analyzers and [not any other of the GitLab dependency analyzers](../dependency_scanning/analyzers.md). diff --git a/doc/user/application_security/dependency_scanning/analyzers.md b/doc/user/application_security/dependency_scanning/analyzers.md index 8559d5af02e..1b502b306bb 100644 --- a/doc/user/application_security/dependency_scanning/analyzers.md +++ b/doc/user/application_security/dependency_scanning/analyzers.md @@ -32,6 +32,9 @@ to launch dedicated containers for each analysis. Dependency Scanning is pre-configured with a set of **default images** that are maintained by GitLab, but users can also integrate their own **custom images**. +WARNING: +The `bundler-audit` analyzer is deprecated and will be removed in GitLab 15.0 since it duplicates the functionality of the `gemnasium` analyzer. For more information, read the [deprecation announcement](../../../update/deprecations.md#deprecation-of-bundler-audit-dependency-scanning-tool). + ## Official default analyzers Any custom change to the official analyzers can be achieved by using a @@ -118,12 +121,12 @@ The following table lists the data available for each official analyzer. | File | ✓ | ⚠ | ✓ | | Start line | 𐄂 | 𐄂 | 𐄂 | | End line | 𐄂 | 𐄂 | 𐄂 | -| External ID (e.g., CVE) | ✓ | ✓ | ⚠ | +| External ID (for example, CVE) | ✓ | ✓ | ⚠ | | URLs | ✓ | ✓ | ✓ | | Internal doc/explanation | ✓ | 𐄂 | 𐄂 | | Solution | ✓ | ✓ | 𐄂 | | Confidence | 𐄂 | 𐄂 | 𐄂 | -| Affected item (e.g. class or package) | ✓ | ✓ | ✓ | +| Affected item (for example, class or package) | ✓ | ✓ | ✓ | | Source code extract | 𐄂 | 𐄂 | 𐄂 | | Internal ID | ✓ | 𐄂 | 𐄂 | | Date | ✓ | 𐄂 | 𐄂 | @@ -134,4 +137,4 @@ The following table lists the data available for each official analyzer. - 𐄂 => we don't have that data, or it would need to develop specific or inefficient/unreliable logic to obtain it. The values provided by these tools are heterogeneous, so they are sometimes -normalized into common values (e.g., `severity`, `confidence`, etc). +normalized into common values (for example, `severity`, `confidence`, etc). diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index 3c6db8c3ee9..192d8449297 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -7,10 +7,43 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Dependency Scanning **(ULTIMATE)** +INFO: +Try out Dependency Scanning in GitLab Ultimate. +[It's free for 30 days](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=u-dependency-scanning-docs). + The Dependency Scanning feature can automatically find security vulnerabilities in your -dependencies while you're developing and testing your applications. For example, dependency scanning -lets you know if your application uses an external (open source) library that is known to be -vulnerable. You can then take action to protect your application. +software dependencies while you're developing and testing your applications. For example, +dependency scanning lets you know if your application uses an external (open source) +library that is known to be vulnerable. You can then take action to protect your application. + +Dependency Scanning is often considered part of Software Composition Analysis (SCA). SCA can contain +aspects of inspecting the items your code uses. These items typically include application and system +dependencies that are almost always imported from external sources, rather than sourced from items +you wrote yourself. + +GitLab offers both Dependency Scanning and Container Scanning +to ensure coverage for all of these dependency types. To cover as much of your risk area as +possible, we encourage you to use all of our security scanners: + +- Dependency Scanning analyzes your project and tells you which software dependencies, + including upstream dependencies, have been included in your project, and what known + risks the dependencies contain. Dependency Scanning modifies its behavior based + on the language and package manager of the project. It typically looks for a lock file + then performs a build to fetch upstream dependency information. In the case of + containers, Dependency Scanning uses the compatible manifest and reports only these + declared software dependencies (and those installed as a sub-dependency). + Dependency Scanning can not detect software dependencies that are pre-bundled + into the container's base image. To identify pre-bundled dependencies, enable + [Container Scanning](../container_scanning/) language scanning using the + [`CS_DISABLE_LANGUAGE_VULNERABILITY_SCAN` variable](../container_scanning/#report-language-specific-findings). +- [Container Scanning](../container_scanning/) analyzes your containers and tells + you about known risks in the operating system's (OS) packages. You can configure it + to also report on software and language dependencies, if you enable it and use + the [`CS_DISABLE_LANGUAGE_VULNERABILITY_SCAN` variable](../container_scanning/#report-language-specific-findings). + Turning this variable on can result in some duplicate findings, as we do not yet + de-duplicate results between Container Scanning and Dependency Scanning. For more details, + efforts to de-duplicate these findings can be tracked in + [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/348655). ## Overview @@ -142,7 +175,7 @@ table.supported-languages ul { <tr> <td>Go</td> <td>N/A</td> - <td><a href="https://golang.org/">Go</a></td> + <td><a href="https://go.dev/">Go</a></td> <td><code>go.sum</code></td> <td><a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">Gemnasium</a></td> <td>Y</td> @@ -279,10 +312,10 @@ table.supported-languages ul { GitLab analyzers obtain dependency information using one of the following two methods: -1. [Parsing lockfiles directly.](#obtaining-dependendency-information-by-parsing-lockfiles) -1. [Running a package manager or build tool to generate a dependency information file which is then parsed.](#obtaining-dependendency-information-by-running-a-package-manager-to-generate-a-parsable-file) +1. [Parsing lockfiles directly.](#obtaining-dependency-information-by-parsing-lockfiles) +1. [Running a package manager or build tool to generate a dependency information file which is then parsed.](#obtaining-dependency-information-by-running-a-package-manager-to-generate-a-parsable-file) -#### Obtaining dependendency information by parsing lockfiles +#### Obtaining dependency information by parsing lockfiles The following package managers use lockfiles that GitLab analyzers are capable of parsing directly: @@ -296,7 +329,7 @@ The following package managers use lockfiles that GitLab analyzers are capable o | npm | v1, v2 | [6.x](https://gitlab.com/gitlab-org/security-products/tests/js-npm/-/blob/master/package-lock.json#L4), [7.x](https://gitlab.com/gitlab-org/security-products/tests/js-npm/-/blob/lockfile-v2-FREEZE/package-lock.json#L4) | | yarn | v1 | [1.x](https://gitlab.com/gitlab-org/security-products/tests/js-yarn/-/blob/master/yarn.lock) | -#### Obtaining dependendency information by running a package manager to generate a parsable file +#### Obtaining dependency information by running a package manager to generate a parsable file To support the following package managers, the GitLab analyzers proceed in two steps: @@ -309,7 +342,7 @@ To support the following package managers, the GitLab analyzers proceed in two s | sbt | [1.3.8](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L4) | [1.0.4](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L263), [1.1.4](https://gitlab.com/gitlab-org/security-products/tests/scala-sbt-multiproject/-/blob/main/project/build.properties#L1), [1.1.6](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L272), [1.2.8](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L281), [1.3.12](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L290), [1.4.6](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L299) | | Maven | [3.6.3](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L3) | [3.6.3](https://gitlab.com/gitlab-org/security-products/tests/java-maven/-/blob/master/pom.xml#L3) | | Gradle | [6.7.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L5) | [5.6.4](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/master/gradle/wrapper/gradle-wrapper.properties#L3), [6.5](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-14/gradle/wrapper/gradle-wrapper.properties#L3), [6.7-rc-1](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-15/gradle/wrapper/gradle-wrapper.properties#L3), [6.9](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-14-gradle-6-9/gradle/wrapper/gradle-wrapper.properties#L3), [7.0-rc-2](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-16/gradle/wrapper/gradle-wrapper.properties#L3) | -| setuptools | [50.3.2](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/Dockerfile#L27) | | +| setuptools | [50.3.2](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/Dockerfile#L27) | [57.5.0](https://gitlab.com/gitlab-org/security-products/tests/python-setuptools/-/blob/main/setup.py) | | pip | [20.2.4](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/Dockerfile#L26) | [20.x](https://gitlab.com/gitlab-org/security-products/tests/python-pip/-/blob/master/requirements.txt) | | Pipenv | [2018.11.26](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/v2.18.4/requirements.txt#L13) | [2018.11.26](https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/blob/pipfile-lock-FREEZE/Pipfile.lock#L6)<sup><b><a href="#exported-dependency-information-notes-2">2</a></b></sup>, [2018.11.26](https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/blob/master/Pipfile) | @@ -370,7 +403,7 @@ We only execute one build in the directory where a build file has been detected, Please note, we support the following types of Java project structures: - [multi-project sbt builds](https://www.scala-sbt.org/1.x/docs/Multi-Project.html) -- [multi-project gradle builds](https://docs.gradle.org/current/userguide/intro_multi_project_builds.html) +- [multi-project Gradle builds](https://docs.gradle.org/current/userguide/intro_multi_project_builds.html) - [multi-module maven projects](https://maven.apache.org/pom.html#Aggregation) #### JavaScript @@ -425,7 +458,7 @@ include: The included template creates dependency scanning jobs in your CI/CD pipeline and scans your project's source code for possible vulnerabilities. The results are saved as a -[dependency scanning report artifact](../../../ci/yaml/index.md#artifactsreportsdependency_scanning) +[dependency scanning report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsdependency_scanning) that you can later download and analyze. Due to implementation limitations, we always take the latest dependency scanning artifact available. @@ -440,7 +473,7 @@ from the Security Configuration page. 1. In the project where you want to enable Dependency Scanning, navigate to **Security & Compliance > Configuration**. -1. In the **Dependency Scanning** row, select **Configure via Merge Request**. +1. In the **Dependency Scanning** row, select **Configure with a merge request**. This automatically creates a merge request with the changes necessary to enable Dependency Scanning that you can review and merge to complete the configuration. @@ -506,7 +539,7 @@ The following variables allow configuration of global dependency scanning settin | `ADDITIONAL_CA_CERT_BUNDLE` | Bundle of CA certs to trust. The bundle of certificates provided here is also used by other tools during the scanning process, such as `git`, `yarn`, or `npm`. See [Using a custom SSL CA certificate authority](#using-a-custom-ssl-ca-certificate-authority) for more details. | | `DS_EXCLUDED_ANALYZERS` | Specify the analyzers (by name) to exclude from Dependency Scanning. For more information, see [Dependency Scanning Analyzers](analyzers.md). | | `DS_DEFAULT_ANALYZERS` | ([**DEPRECATED - use `DS_EXCLUDED_ANALYZERS` instead**](https://gitlab.com/gitlab-org/gitlab/-/issues/287691)) Override the names of the official default images. For more information, see [Dependency Scanning Analyzers](analyzers.md). | -| `DS_EXCLUDED_PATHS` | Exclude vulnerabilities from output based on the paths. A comma-separated list of patterns. Patterns can be globs, or file or folder paths (for example, `doc,spec`). Parent directories also match patterns. Default: `"spec, test, tests, tmp"`. | +| `DS_EXCLUDED_PATHS` | Exclude files and directories from the scan based on the paths. A comma-separated list of patterns. Patterns can be globs, or file or folder paths (for example, `doc,spec`). Parent directories also match patterns. Default: `"spec, test, tests, tmp"`. | | `SECURE_ANALYZERS_PREFIX` | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). | | `SECURE_LOG_LEVEL` | Set the minimum logging level. Messages of this logging level or higher are output. From highest to lowest severity, the logging levels are: `fatal`, `error`, `warn`, `info`, `debug`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10880) in GitLab 13.1. Default: `info`. | @@ -596,7 +629,7 @@ The dependency scanning tool emits a JSON report file. For more information, see Here's an example dependency scanning report: -```json-doc +```json { "version": "2.0", "vulnerabilities": [ @@ -709,7 +742,7 @@ Please check the [Release Process documentation](https://gitlab.com/gitlab-org/s ## Contributing to the vulnerability database -You can search the [gemnasium-db](https://gitlab.com/gitlab-org/security-products/gemnasium-db) project +You can search the [`gemnasium-db`](https://gitlab.com/gitlab-org/security-products/gemnasium-db) project to find a vulnerability in the Gemnasium database. You can also [submit new vulnerabilities](https://gitlab.com/gitlab-org/security-products/gemnasium-db/blob/master/CONTRIBUTING.md). @@ -781,7 +814,7 @@ Support for custom certificate authorities was introduced in the following versi Add the following configuration to your `.gitlab-ci.yml` file. You must change the value of `SECURE_ANALYZERS_PREFIX` to refer to your local Docker container registry. You must also change the value of `GEMNASIUM_DB_REMOTE_URL` to the location of your offline Git copy of the -[gemnasium-db advisory database](https://gitlab.com/gitlab-org/security-products/gemnasium-db/): +[`gemnasium-db` advisory database](https://gitlab.com/gitlab-org/security-products/gemnasium-db/): ```yaml include: @@ -1033,3 +1066,19 @@ scan occurs. Because the cache is downloaded before the analyzer run occurs, the file in the `CI_BUILDS_DIR` directory triggers the dependency scanning job. We recommend committing the lock files, which prevents this warning. + +### I no longer get the latest Docker image after setting `DS_MAJOR_VERSION` or `DS_ANALYZER_IMAGE` + +If you have manually set `DS_MAJOR_VERSION` or `DS_ANALYZER_IMAGE` for specific reasons, +and now must update your configuration to again get the latest patched versions of our +analyzers, edit your `gitlab-ci.yml` file and either: + +- Set your `DS_MAJOR_VERSION` to match the latest version as seen in + [our current Dependency Scanning template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml#L18). +- If you hardcoded the `DS_ANALYZER_IMAGE` variable directly, change it to match the latest + line as found in our [current Dependency Scanning template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml). + The line number will vary depending on which scanning job you edited. + + For example, currently the `gemnasium-maven-dependency_scanning` job pulls the latest + `gemnasium-maven` Docker image because `DS_ANALYZER_IMAGE` is set to + `"$SECURE_ANALYZERS_PREFIX/gemnasium-maven:$DS_MAJOR_VERSION"`. diff --git a/doc/user/application_security/iac_scanning/index.md b/doc/user/application_security/iac_scanning/index.md index a58a00a869b..c17ebc68b4d 100644 --- a/doc/user/application_security/iac_scanning/index.md +++ b/doc/user/application_security/iac_scanning/index.md @@ -32,14 +32,14 @@ GitLab IaC scanning supports a variety of IaC configuration files. Our IaC secur | Configuration File Type | Scan tool | Introduced in GitLab Version | |------------------------------------------|----------------------------------|-------------------------------| -| Ansible | [kics](https://kics.io/) | 14.5 | -| AWS CloudFormation | [kics](https://kics.io/) | 14.5 | -| Kubernetes | [kics](https://kics.io/) | 14.5 | -| Terraform | [kics](https://kics.io/) | 14.5 | +| Ansible | [KICS](https://kics.io/) | 14.5 | +| AWS CloudFormation | [KICS](https://kics.io/) | 14.5 | +| Kubernetes | [KICS](https://kics.io/) | 14.5 | +| Terraform | [KICS](https://kics.io/) | 14.5 | ### Making IaC analyzers available to all GitLab tiers -All open source (OSS) analyzers are availibile with the GitLab Free tier. Future propietary analyzers may be restricted to higher tiers. +All open source (OSS) analyzers are available with the GitLab Free tier. Future proprietary analyzers may be restricted to higher tiers. #### Summary of features per tier @@ -74,7 +74,7 @@ The included template creates IaC scanning jobs in your CI/CD pipeline and scans your project's configuration files for possible vulnerabilities. The results are saved as a -[SAST report artifact](../../../ci/yaml/index.md#artifactsreportssast) +[SAST report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportssast) that you can download and analyze. ### Enable IaC Scanning via an automatic merge request @@ -84,7 +84,7 @@ from the Security Configuration page: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > Configuration**. -1. In the **Infrastructure as Code (IaC) Scanning** row, select **Configure via Merge Request**. +1. In the **Infrastructure as Code (IaC) Scanning** row, select **Configure with a merge request**. This automatically creates a merge request with the changes necessary to enable IaC Scanning that you can review and merge to complete the configuration. diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index d5e801ced9c..5500f5a10c4 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -16,6 +16,10 @@ GitLab can check your application for security vulnerabilities including: Statistics and details on vulnerabilities are included in the merge request. Providing actionable information _before_ changes are merged enables you to be proactive. +INFO: +Want to try out security scanning? +[Try GitLab Ultimate free for 30 days](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=u-application-security-docs). + GitLab also provides high-level statistics of vulnerabilities across projects and groups: - The [Security Dashboard](security_dashboard/index.md) provides a @@ -42,7 +46,7 @@ GitLab uses the following tools to scan and report known vulnerabilities found i | [Secret Detection](secret_detection/index.md) | Analyze Git history for leaked secrets. | | [Security Dashboard](security_dashboard/index.md) | View vulnerabilities in all your projects and groups. | | [Static Application Security Testing (SAST)](sast/index.md) | Analyze source code for known vulnerabilities. | -| [Infrastructure as Code (IaC) Scanning](iac_scanning/index.md) | Analyze your IaC coniguration files for known vulnerabilities. | +| [Infrastructure as Code (IaC) Scanning](iac_scanning/index.md) | Analyze your IaC configuration files for known vulnerabilities. | | [Coverage fuzzing](coverage_fuzzing/index.md) | Find unknown bugs and vulnerabilities with coverage-guided fuzzing. | | [Cluster Image Scanning](cluster_image_scanning/index.md) | Scan Kubernetes clusters for known vulnerabilities. | diff --git a/doc/user/application_security/policies/img/security_policy_project_v14_3.png b/doc/user/application_security/policies/img/security_policy_project_v14_3.png Binary files differdeleted file mode 100644 index 5e3aefaeb81..00000000000 --- a/doc/user/application_security/policies/img/security_policy_project_v14_3.png +++ /dev/null diff --git a/doc/user/application_security/policies/img/security_policy_project_v14_6.png b/doc/user/application_security/policies/img/security_policy_project_v14_6.png Binary files differnew file mode 100644 index 00000000000..feea1b1b01c --- /dev/null +++ b/doc/user/application_security/policies/img/security_policy_project_v14_6.png diff --git a/doc/user/application_security/policies/index.md b/doc/user/application_security/policies/index.md index 4d8be411dc5..e6dbd96537f 100644 --- a/doc/user/application_security/policies/index.md +++ b/doc/user/application_security/policies/index.md @@ -159,8 +159,8 @@ at the bottom of the editor. You can use policy alerts to track your policy's impact. Alerts are only available if you've [installed](../../clusters/agent/repository.md) -and [configured](../../clusters/agent/install/index.md#create-an-agent-record-in-gitlab) -a Kubernetes Agent for this project. +and [configured](../../clusters/agent/install/index.md#register-an-agent-with-gitlab) +an agent for this project. There are two ways to create policy alerts: @@ -228,7 +228,13 @@ must create an association between that project and the project you want to appl project you would like to link from the dropdown menu. 1. Select **Save**. -  +  + +### Unlink Security Policy projects + +Project owners can unlink Security Policy projects from development projects. To do this, follow +the steps described in [Security Policy project selection](#security-policy-project-selection), +but select the trash can icon in the modal. ### Scan Execution Policy editor @@ -237,9 +243,9 @@ Only project Owners have the [permissions](../../permissions.md#project-members- to select Security Policy Project. Once your policy is complete, save it by selecting **Create merge request** -at the bottom of the editor. You will be redirected to the merge request on the project's +at the bottom of the editor. You are redirected to the merge request on the project's configured security policy project. If one does not link to your project, a security -policy project will be automatically created. Existing policies can also be +policy project is automatically created. Existing policies can also be removed from the editor interface by selecting **Delete policy** at the bottom of the editor. @@ -287,7 +293,7 @@ This rule enforces the defined actions and schedules a scan on the provided date | `type` | `string` | `schedule` | The rule's type. | | `branches` | `array` of `string` | `*` or the branch's name | The branch the given policy applies to (supports wildcard). | | `cadence` | `string` | CRON expression (for example, `0 0 * * *`) | A whitespace-separated string containing five fields that represents the scheduled time. | -| `clusters` | `object` | | The cluster where the given policy will enforce running selected scans (only for `container_scanning`/`cluster_image_scanning` scans). The key of the object is the name of the Kubernetes cluster configured for your project in GitLab. In the optionally provided value of the object, you can precisely select Kubernetes resources that will be scanned. | +| `clusters` | `object` | | The cluster where the given policy enforces running selected scans (only for `container_scanning`/`cluster_image_scanning` scans). The key of the object is the name of the Kubernetes cluster configured for your project in GitLab. In the optionally provided value of the object, you can precisely select Kubernetes resources that are scanned. | #### `cluster` schema @@ -295,10 +301,10 @@ Use this schema to define `clusters` objects in the [`schedule` rule type](#sche | Field | Type | Possible values | Description | |--------------|---------------------|--------------------------|-------------| -| `containers` | `array` of `string` | | The container name that will be scanned (only the first value is currently supported). | -| `resources` | `array` of `string` | | The resource name that will be scanned (only the first value is currently supported). | -| `namespaces` | `array` of `string` | | The namespace that will be scanned (only the first value is currently supported). | -| `kinds` | `array` of `string` | `deployment`/`daemonset` | The resource kind that should be scanned (only the first value is currently supported). | +| `containers` | `array` of `string` | | The container name to be scanned (only the first value is currently supported). | +| `resources` | `array` of `string` | | The resource name to be scanned (only the first value is currently supported). | +| `namespaces` | `array` of `string` | | The namespace to be scanned (only the first value is currently supported). | +| `kinds` | `array` of `string` | `deployment`/`daemonset` | The resource kind to be scanned (only the first value is currently supported). | ### `scan` action type @@ -307,9 +313,10 @@ rule in the defined policy are met. | Field | Type | Possible values | Description | |-------|------|-----------------|-------------| -| `scan` | `string` | `dast`, `secret_detection` | The action's type. | +| `scan` | `string` | `dast`, `secret_detection`, `sast`, `container_scanning`, `cluster_image_scanning` | The action's type. | | `site_profile` | `string` | Name of the selected [DAST site profile](../dast/index.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | | `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/index.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| +| `variables` | `object` | | Set of variables applied and enforced for the selected scan. The object's key is the variable name with a value provided as a string. | Note the following: @@ -327,9 +334,10 @@ Note the following: - A secret detection scan runs in `normal` mode when executed as part of a pipeline, and in [`historic`](../secret_detection/index.md#full-history-secret-detection) mode when executed as part of a scheduled scan. -- A container scanning and cluster image scanning scans configured for the `pipeline` rule type will ignore the cluster defined in the `clusters` object. - They will use predefined CI/CD variables defined for your project. Cluster selection with the `clusters` object is supported for the `schedule` rule type. +- A container scanning and cluster image scanning scans configured for the `pipeline` rule type ignores the cluster defined in the `clusters` object. + They use predefined CI/CD variables defined for your project. Cluster selection with the `clusters` object is supported for the `schedule` rule type. Cluster with name provided in `clusters` object must be created and configured for the project. To be able to successfully perform the `container_scanning`/`cluster_image_scanning` scans for the cluster you must follow instructions for the [Cluster Image Scanning feature](../cluster_image_scanning/index.md#prerequisites). +- The SAST scan uses the default template and runs in a [child pipeline](../../../ci/pipelines/parent_child_pipelines.md). ### Example security policies project @@ -357,7 +365,7 @@ scan_execution_policy: - type: schedule branches: - main - cadence: */10 * * * * + cadence: "*/10 * * * *" actions: - scan: dast scanner_profile: Scanner Profile C @@ -372,13 +380,16 @@ scan_execution_policy: - main actions: - scan: secret_detection + - scan: sast + variables: + SAST_EXCLUDED_ANALYZERS: brakeman - scan: container_scanning - name: Enforce Cluster Image Scanning on production-cluster every 24h description: This policy enforces Cluster Image Scanning scan to run every 24 hours enabled: true rules: - type: schedule - cadence: '15 3 * * *' + cadence: "15 3 * * * clusters: production-cluster: containers: @@ -399,7 +410,8 @@ In this example: `release/v1.2.1`), DAST scans run with `Scanner Profile A` and `Site Profile B`. - DAST and secret detection scans run every 10 minutes. The DAST scan runs with `Scanner Profile C` and `Site Profile D`. -- Secret detection and container scanning scans run for every pipeline executed on the `main` branch. +- Secret detection, container scanning, and SAST scans run for every pipeline executed on the `main` + branch. The SAST scan runs with the `SAST_EXCLUDED_ANALYZER` variable set to `"brakeman"`. - Cluster Image Scanning scan runs every 24h. The scan runs on the `production-cluster` cluster and fetches vulnerabilities from the container with the name `database` configured for deployment with the name `production-application` in the `production-namespace` namespace. diff --git a/doc/user/application_security/sast/analyzers.md b/doc/user/application_security/sast/analyzers.md index 06c57e68121..8c7e03f69fd 100644 --- a/doc/user/application_security/sast/analyzers.md +++ b/doc/user/application_security/sast/analyzers.md @@ -59,7 +59,7 @@ support the following features: ## Official default analyzers Any custom change to the official analyzers can be achieved by using a -[CI/CD variable in your `.gitlab-ci.yml`](index.md#customizing-the-sast-settings). +[CI/CD variable in your `.gitlab-ci.yml`](index.md#available-cicd-variables). ### Using a custom Docker mirror diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index af8585c6a18..fd05ecad8f2 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -144,7 +144,7 @@ as shown in the following table: | Capability | In Free | In Ultimate | |:---------------------------------------------------------------------------------------|:--------------------|:-------------------| | [Configure SAST Scanners](#configuration) | **{check-circle}** | **{check-circle}** | -| [Customize SAST Settings](#customizing-the-sast-settings) | **{check-circle}** | **{check-circle}** | +| [Customize SAST Settings](#available-cicd-variables) | **{check-circle}** | **{check-circle}** | | View [JSON Report](#reports-json-format) | **{check-circle}** | **{check-circle}** | | Presentation of JSON Report in Merge Request | **{dotted-circle}** | **{check-circle}** | | [Address vulnerabilities](../../application_security/vulnerabilities/index.md) | **{dotted-circle}** | **{check-circle}** | @@ -184,7 +184,7 @@ The included template creates SAST jobs in your CI/CD pipeline and scans your project's source code for possible vulnerabilities. The results are saved as a -[SAST report artifact](../../../ci/yaml/index.md#artifactsreportssast) +[SAST report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportssast) that you can later download and analyze. Due to implementation limitations, we always take the latest SAST artifact available. @@ -242,25 +242,6 @@ The configuration tool works best with no existing `.gitlab-ci.yml` file, or wit configuration file. If you have a complex GitLab configuration file it may not be parsed successfully, and an error may occur. -### Customizing the SAST settings - -The SAST settings can be changed through [CI/CD variables](#available-cicd-variables) -by using the -[`variables`](../../../ci/yaml/index.md#variables) parameter in `.gitlab-ci.yml`. -In the following example, we include the SAST template and at the same time we -set the `SAST_GOSEC_LEVEL` variable to `2`: - -```yaml -include: - - template: Security/SAST.gitlab-ci.yml - -variables: - SAST_GOSEC_LEVEL: 2 -``` - -Because the template is [evaluated before](../../../ci/yaml/index.md#include) -the pipeline configuration, the last mention of the variable takes precedence. - ### Overriding SAST jobs WARNING: @@ -305,16 +286,16 @@ spotbugs-sast: ### Customize rulesets **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235382) in GitLab 13.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235382) in GitLab 13.5. +> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/339614) support for +> passthrough chains. Expanded to include additional passthrough types of `file`, `git`, and `url` in GitLab 14.6. You can customize the default scanning rules provided by our SAST analyzers. +Ruleset customization supports two capabilities that can be used +simultaneously: -Ruleset customization supports two capabilities: - -1. Disabling predefined rules (available for all analyzers). -1. Modifying the default behavior of a given analyzer (only available for `nodejs-scan` and `gosec`). - -These capabilities can be used simultaneously. +- [Disabling predefined rules](index.md#disable-predefined-analyzer-rules). Available for all analyzers. +- Modifying the default behavior of a given analyzer by [synthesizing and passing a custom configuration](index.md#synthesize-a-custom-configuration). Available for only `nodejs-scan`, `gosec`, and `semgrep`. To customize the default scanning rules, create a file containing custom rules. These rules are passed through to the analyzer's underlying scanner tools. @@ -323,81 +304,303 @@ To create a custom ruleset: 1. Create a `.gitlab` directory at the root of your project, if one doesn't already exist. 1. Create a custom ruleset file named `sast-ruleset.toml` in the `.gitlab` directory. -1. In the `sast-ruleset.toml` file, do one of the following: - - - Disable predefined rules belonging to SAST analyzers. In this example, the three disabled rules - belong to `eslint` and `sobelow` by matching the corresponding identifiers' `type` and `value`: - - ```toml - [eslint] - [[eslint.ruleset]] - disable = true - [eslint.ruleset.identifier] - type = "eslint_rule_id" - value = "security/detect-object-injection" - - [[eslint.ruleset]] - disable = true - [eslint.ruleset.identifier] - type = "cwe" - value = "185" - - [sobelow] - [[sobelow.ruleset]] - disable = true - [sobelow.ruleset.identifier] - type = "sobelow_rule_id" - value = "sql_injection" - ``` - - - Define a custom analyzer configuration. In this example, customized rules are defined for the - `nodejs-scan` scanner: - - ```toml - [nodejs-scan] - description = 'custom ruleset for nodejs-scan' - - [[nodejs-scan.passthrough]] - type = "raw" - value = ''' - - nodejs-extensions: - - .js - - template-extensions: - - .new - - .hbs - - '' - - ignore-filenames: - - skip.js - - ignore-paths: - - __MACOSX - - skip_dir - - node_modules - - ignore-extensions: - - .hbs - - ignore-rules: - - regex_injection_dos - - pug_jade_template - - express_xss - - ''' - ``` - - - Provide the name of the file containing a custom analyzer configuration. In this example, - customized rules for the `gosec` scanner are contained in the file `gosec-config.json`: - - ```toml - [gosec] - description = 'custom ruleset for gosec' - - [[gosec.passthrough]] - type = "file" - value = "gosec-config.json" - ``` + +#### Disable predefined analyzer rules + +To disable analyzer rules: + +1. Set the `disabled` flag to `true` in the context of a `ruleset` section + +1. In one or more `ruleset.identifier` sub sections, list the rules that you want disabled. Every `ruleset.identifier` section has: + +- a `type` field, to name the predefined rule identifier that the targeted analyzer uses. + +- a `value` field, to name the rule to be disabled. + +##### Example: Disable predefined rules of SAST analyzers + +In the following example, the disabled rules are assigned to `eslint` +and `sobelow` by matching the `type` and `value` of identifiers: + +```toml +[eslint] + [[eslint.ruleset]] + disable = true + [eslint.ruleset.identifier] + type = "eslint_rule_id" + value = "security/detect-object-injection" + + [[eslint.ruleset]] + disable = true + [eslint.ruleset.identifier] + type = "cwe" + value = "185" + +[sobelow] + [[sobelow.ruleset]] + disable = true + [sobelow.ruleset.identifier] + type = "sobelow_rule_id" + value = "sql_injection" +``` + +#### Synthesize a custom configuration + +To create a custom configuration, you can use passthrough chains. + +A passthrough is a single step in a passthrough chain. The passthrough is evaluated +in a sequence to incrementally build a configuration. The configuration is then +passed to the target analyzer. + +A configuration section for an analyzer has the following +parameters: + +| Parameter | Explanation | +| ------------- | ------ | +| `description` | Description about the analyzer configuration section. | +| `targetdir` | The `targetdir` parameter defines the directory where the final configuration is located. If `targetdir` is empty, the analyzer uses a random directory. The maximum size of `targetdir` is 100MB. | +| `validate` | If set to `true`, the target files for passthroughs (`raw`, `file` and `url`) are validated. The validation works for `yaml`, `xml`, `json` and `toml` files. The proper validator is identified based on the extension of the target file. By default, `validate` is set to `false`. | +| `interpolate` | If set to `true`, environment variable interpolation is enabled so that the configuration uses secrets/tokens. We advise using this feature with caution to not leak any secrets. By default, `interpolate` is set to `false`. | +| `timeout` | The total `timeout` for the evaluation of a passthrough chain is set to 60 seconds. If `timeout` is not set, the default timeout is 60 seconds. The timeout cannot exceed 300 seconds. | + +A configuration section can include one or more passthrough sections. The maximum number of passthrough sections is 20. +There are several types of passthroughs: + +| Type | Description | +| ------ | ------ | +| `file` | Use a file that is already available in the Git repository. | +| `raw` | Provide the configuration inline. | +| `git` | Pull the configuration from a remote Git repository. | +| `url` | Fetch the analyzer configuration through HTTP. | + +If multiple passthrough sections are defined in a passthrough chain, their +position in the chain defines the order in which they are evaluated. + +- Passthroughs listed later in the chain sequence have a higher precedence. +- Passthroughs with a higher precedence overwrite (default) and append data + yielded by previous passthroughs. This is useful for cases where you need to + use or modify an existing configuration. + +Configure a passthrough these parameters: + +| Parameter | Explanation | +| ------------ | ----------- | +| `type` | One of `file`, `raw`, `git` or `url`. | +| `target` | The target file that contains the data written by the passthrough evaluation. If no value is provided, a random target file is generated. | +| `mode` | `overwrite`: if `target` exists, overwrites the file; `append`: append to file instead. The default is `overwrite`. | +| `ref` | This option only applies to the `git` passthrough type and contains the name of the branch or the SHA to be used. | +| `subdir` | This option only applies to the `git` passthrough type and can be used to only consider a certain subdirectory of the source Git repository. | +| `value` | For the `file` `url` and `git` types, `value` defines the source location of the file/Git repository; for the `raw` type, `value` carries the raw content to be passed through. | +| `validator` | Can be used to explicitly invoke validators (`xml`, `yaml`, `json`, `toml`) on the target files after the application of a passthrough. Per default, no validator is set. | + +The amount of data generated by a single passthrough is limited to 1MB. + +#### Passthrough configuration examples + +##### Raw passthrough for nodejs-scan + +Define a custom analyzer configuration. In this example, customized rules are +defined for the `nodejs-scan` scanner: + +```toml +[nodejs-scan] + description = 'custom ruleset for nodejs-scan' + + [[nodejs-scan.passthrough]] + type = "raw" + value = ''' +- nodejs-extensions: + - .js + + template-extensions: + - .new + - .hbs + - '' + + ignore-filenames: +- skip.js + + ignore-paths: + - __MACOSX + - skip_dir + - node_modules + + ignore-extensions: + - .hbs + + ignore-rules: + - regex_injection_dos + - pug_jade_template + - express_xss + +''' +``` + +##### File passthrough for gosec + +Provide the name of the file containing a custom analyzer configuration. In +this example, customized rules for the `gosec` scanner are contained in the +file `gosec-config.json`: + +```toml +[gosec] + description = 'custom ruleset for gosec' + + [[gosec.passthrough]] + type = "file" + value = "gosec-config.json" +``` + +##### Passthrough chain for semgrep + +In the below example, we generate a custom configuration under the `/sgrules` +target directory with a total `timeout` of 60 seconds. + +Several passthrouh types generate a configuration for the target analyzer: + +- Two `git` passthrough sections pull the head of branch + `refs/remotes/origin/test` from the `myrules` Git repository, and revision + `97f7686` from the `sast-rules` Git repostory. From the `sast-rules` Git + repository, only data from the `go` subdirectory is considered. + - The `sast-rules` entry has a higher precedence because it appears later in + the configuration. + - If there is a filename collision between files in both repositories, files + from the `sast` repository overwrite files from the `myrules` repository, + as `sast-rules` has higher precedence. +- The `raw` entry creates a file named `insecure.yml` under `/sgrules`. The + full path is `/sgrules/insecure.yml`. +- The `url` entry fetches a configuration made available through a URL and + stores it in the `/sgrules/gosec.yml` file. + +Afterwards, semgrep is invoked with the final configuration located under +`/sgrules`. + +```toml +[semgrep] + description = 'semgrep custom rules configuration' + targetdir = "/sgrules" + timeout = 60 + + [[semgrep.passthrough]] + type = "git" + value = "https://gitlab.com/user/myrules.git" + ref = "refs/remotes/origin/test" + + [[semgrep.passthrough]] + type = "git" + value = "https://gitlab.com/gitlab-org/secure/gsoc-sast-vulnerability-rules/playground/sast-rules.git" + ref = "97f7686db058e2141c0806a477c1e04835c4f395" + subdir = "go" + + [[semgrep.passthrough]] + type = "raw" + target = "insecure.yml" + value = """ +rules: +- id: "insecure" + patterns: + - pattern: "func insecure() {...}" + message: | + Insecure function insecure detected + metadata: + cwe: "CWE-200: Exposure of Sensitive Information to an Unauthorized Actor" + severity: "ERROR" + languages: + - "go" + """ + + [[semgrep.passthrough]] + type = "url" + value = "https://semgrep.dev/c/p/gosec" + target = "gosec.yml" +``` + +##### Interpolation + +The code snippet below shows an example configuration that uses an environment +variable `$GITURL` to access a private repositories with a Git URL. The variable contains +a username and token in the `value` field (for example `https://user:token@url`). +It does not explicitly store credentials in the configuration file. To reduce the risk of leaking secrets through created paths and files, use this feature with caution. + +```toml +[semgrep] + description = 'semgrep custom rules configuration' + targetdir = "/sgrules" + interpolate = true + + [[semgrep.passthrough]] + type = "git" + value = "$GITURL" + ref = "refs/remotes/origin/main" +``` + +##### Configure the append mode for passthroughs + +To append data to previous passthroughs, use the `append` mode for the +passthrough types `file`, `url`, and `raw`. + +Passthroughs in `override` mode overwrite files +created when preceding passthroughs in the chain find a naming +collision. If `mode` is set to `append`, a passthrough appends data to the +files created by its predecessors instead of overwriting. + +In the below semgrep configuration,`/sgrules/insecure.yml` assembles two passthroughs. The rules are: + +- `insecure` +- `secret` + +These rules add a search pattern to the analyzer and extends semgrep capabilities. + +For passthrough chains we recommend that you enable validation. To enable validation, +you can either: + +- set `validate` to `true` + +- set a passthrough `validator` to `xml`, `json`, `yaml`, or `toml`. + +```toml +[semgrep] + description = 'semgrep custom rules configuration' + targetdir = "/sgrules" + validate = true + + [[semgrep.passthrough]] + type = "raw" + target = "insecure.yml" + value = """ +rules: +- id: "insecure" + patterns: + - pattern: "func insecure() {...}" + message: | + Insecure function insecure detected + metadata: + cwe: "... + severity: "ERROR" + languages: + - "go" +""" + + [[semgrep.passthrough]] + type = "raw" + mode = "append" + target = "insecure.yml" + value = """ +- id: "secret" + patterns: + - pattern-either: + - pattern: "$MASK = \"...\"" + - metavariable-regex: + metavariable: "$MASK" + regex: "(password|pass|passwd|pwd|secret|token)" + message: | + Use of Hard-coded Password + cwe: "..." + severity: "ERROR" + languages: + - "go" +""" +``` ### False Positive Detection **(ULTIMATE)** @@ -483,7 +686,20 @@ can use `MAVEN_REPO_PATH`. See ### Available CI/CD variables -SAST can be [configured](#customizing-the-sast-settings) using CI/CD variables. +SAST can be configured using the [`variables`](../../../ci/yaml/index.md#variables) parameter in +`.gitlab-ci.yml`. + +The following example includes the SAST template to override the `SAST_GOSEC_LEVEL` +variable to `2`. The template is [evaluated before](../../../ci/yaml/index.md#include) the pipeline +configuration, so the last mention of the variable takes precedence. + +```yaml +include: + - template: Security/SAST.gitlab-ci.yml + +variables: + SAST_GOSEC_LEVEL: 2 +``` #### Logging level diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index 140f660d729..b5e54e35e58 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -34,33 +34,8 @@ GitLab displays identified secrets visibly in a few places: ## Supported secrets Secret Detection detects a variety of common secrets by default. You can also customize the secret detection patterns using [custom rulesets](#custom-rulesets). - -The [default ruleset provided by TruffleHog and Gitleaks](https://gitlab.com/gitlab-org/security-products/analyzers/secrets/-/blob/master/gitleaks.toml) includes the following key types: - -- Cloud services: - - Amazon Web Services (AWS) - - Google Cloud Platform (GCP) - - Heroku API -- Encryption keys: - - PKCS8 - - RSA - - SSH - - PGP - - DSA - - EC -- Social media platforms: - - Facebook API - - Twitter API -- Cloud SaaS vendors: - - GitHub API - - Shopify API - - Slack Token - - Slack Webhook - - Stripe API - - Twilio API - - Generic API key strings starting with `api-` -- Password in URL -- U.S. Social Security Number +The [default ruleset](https://gitlab.com/gitlab-org/security-products/analyzers/secrets/-/blob/master/gitleaks.toml) includes **90+ secret detection patterns**. +You can contribute "well-identifiable" secrets by follow the steps detailed in the [community contributions guidelines](https://gitlab.com/gitlab-org/gitlab/-/issues/345453). WARNING: Gitleaks does not support scanning binary files. @@ -134,7 +109,7 @@ The included template creates Secret Detection jobs in your CI/CD pipeline and s your project's source code for secrets. The results are saved as a -[Secret Detection report artifact](../../../ci/yaml/index.md#artifactsreportssecret_detection) +[Secret Detection report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportssecret_detection) that you can later download and analyze. Due to implementation limitations, we always take the latest Secret Detection artifact available. @@ -148,10 +123,10 @@ from the Security Configuration page. 1. In the project where you want to enable Secret Detection, go to **Security & Compliance > Configuration**. -1. In the **Secret Detection** row, select **Configure via Merge Request**. +1. In the **Secret Detection** row, select **Configure with a merge request**. This automatically creates a merge request with the changes necessary to enable Secret Detection -that you can review and merge to complete the configuration. +that you can review and merge to complete the configuration. NOTE: The configuration tool works best with no existing `.gitlab-ci.yml` file, or with a minimal @@ -210,10 +185,12 @@ Secret Detection can be customized by defining available CI/CD variables: ### Custom rulesets **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211387) in GitLab 13.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211387) in GitLab 13.5. +> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/339614) support for +> passthrough chains. Expanded to include additional passthrough types of `file`, `git`, and `url` in GitLab 14.6. You can customize the default secret detection rules provided with GitLab. -Customization allows replace the default secret detection rules with rules that you define. +Customization allows replacing the default secret detection rules with rules that you define. To create a custom ruleset: @@ -251,6 +228,9 @@ To create a custom ruleset: value = "config/gitleaks.toml" ``` +Passthroughs can also be chained to build more complex configurations. +For more details, see [SAST Customize ruleset section](../sast/index.md#customize-rulesets). + ### Logging level To control the verbosity of logs set the `SECURE_LOG_LEVEL` CI/CD variable. Messages of this logging level or higher are output. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10880) in GitLab 13.1. diff --git a/doc/user/application_security/security_dashboard/img/pipeline_security_dashboard_v14_4.png b/doc/user/application_security/security_dashboard/img/pipeline_security_dashboard_v14_4.png Binary files differindex ac123d2b528..ad9122ee23c 100644 --- a/doc/user/application_security/security_dashboard/img/pipeline_security_dashboard_v14_4.png +++ b/doc/user/application_security/security_dashboard/img/pipeline_security_dashboard_v14_4.png diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index 87875ec15ba..5afbe1ca54e 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -7,6 +7,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Security Dashboards and Security Center **(ULTIMATE)** +INFO: +Want to try out security scanning? +[Try GitLab Ultimate free for 30 days](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=u-security-dashboard-docs). + GitLab provides a comprehensive set of features for viewing and managing vulnerabilities: - Security dashboards: An overview of the security status in your personal [Security Center](#security-center), [groups](#group-security-dashboard), and diff --git a/doc/user/application_security/vulnerability_report/img/operational_vulnerability_tab_v14_6.png b/doc/user/application_security/vulnerability_report/img/operational_vulnerability_tab_v14_6.png Binary files differnew file mode 100644 index 00000000000..52a298584dd --- /dev/null +++ b/doc/user/application_security/vulnerability_report/img/operational_vulnerability_tab_v14_6.png diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index d13647937a2..3773bb59c5a 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -7,8 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Vulnerability Report **(ULTIMATE)** -The Vulnerability Report provides information about vulnerabilities from scans of the branch most -recently merged into the default branch. It is available for groups, projects, and the Security Center. +The Vulnerability Report provides information about vulnerabilities from scans of the default branch. It is available for projects, groups, and the Security Center. At all levels, the Vulnerability Report contains: @@ -215,3 +214,12 @@ You can dismiss a vulnerability for the entire project: 1. Optional. Add a reason for the dismissal and select **Save comment**. To undo this action, select a different status from the same menu. + +## Operational vulnerabilities + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6345) in GitLab 14.6. + +The **Operational vulnerabilities** tab lists vulnerabilities found by the `cluster_image_scanner`. +This tab appears on the project, group, and Security Center vulnerability reports. + + diff --git a/doc/user/asciidoc.md b/doc/user/asciidoc.md index cd166666ad6..da75c008ed1 100644 --- a/doc/user/asciidoc.md +++ b/doc/user/asciidoc.md @@ -492,10 +492,13 @@ image::screenshot.png[block image,800,450] Press image:reload.svg[reload,16,opts=interactive] to reload the page. video::movie.mp4[width=640,start=60,end=140,options=autoplay] +``` -video::aHjpOzsQ9YI[youtube] +GitLab does not support embedding YouTube and Vimeo videos in AsciiDoc content. +Use a standard AsciiDoc link: -video::300817511[vimeo] +```plaintext +https://www.youtube.com/watch?v=BlaZ65-b7y0[Link text for the video] ``` ### Breaks diff --git a/doc/user/award_emojis.md b/doc/user/award_emojis.md index 88651688779..e4fcdcd4653 100644 --- a/doc/user/award_emojis.md +++ b/doc/user/award_emojis.md @@ -10,7 +10,7 @@ When you're collaborating online, you get fewer opportunities for high-fives and thumbs-ups. Emoji can be awarded to [issues](project/issues/index.md), [merge requests](project/merge_requests/index.md), [snippets](snippets.md), and anywhere you can have a thread. - + Award emoji make it much easier to give and receive feedback without a long comment thread. @@ -34,11 +34,9 @@ downvotes. Award emoji can also be applied to individual comments when you want to celebrate an accomplishment or agree with an opinion. -To: +To add an award emoji: -- Add an award emoji, click the smile in the top right of the comment and pick an emoji from the dropdown. -- Remove an award emoji, click the emoji again. +1. In the top right of the comment, select the smile (**{slight-smile}**). +1. Select an emoji from the dropdown list. - - - +To remove an award emoji, select the emoji again. diff --git a/doc/user/clusters/agent/ci_cd_tunnel.md b/doc/user/clusters/agent/ci_cd_tunnel.md index 0dfdb37dc1f..93768164df2 100644 --- a/doc/user/clusters/agent/ci_cd_tunnel.md +++ b/doc/user/clusters/agent/ci_cd_tunnel.md @@ -19,11 +19,11 @@ Only CI/CD jobs set in the configuration project can access one of the configure ## Prerequisites -- A running [`kas` instance](install/index.md#set-up-the-kubernetes-agent-server). -- A [configuration repository](install/index.md#define-a-configuration-repository) with an Agent config file +- A running [`kas` instance](install/index.md#set-up-the-agent-server). +- A [configuration repository](install/index.md#define-a-configuration-repository) with an agent config file installed (`.gitlab/agents/<agent-name>/config.yaml`). -- An [Agent record](install/index.md#create-an-agent-record-in-gitlab). -- The Agent [installed in the cluster](install/index.md#install-the-agent-into-the-cluster). +- A [registered agent](install/index.md#register-an-agent-with-gitlab). +- The agent [installed in the cluster](install/index.md#install-the-agent-into-the-cluster). ## Use the CI/CD Tunnel to run Kubernetes commands from GitLab CI/CD diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index 80b9f3f17f5..c950a4f0dc0 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -4,19 +4,24 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Kubernetes Agent **(FREE)** +# GitLab Agent for Kubernetes **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223061) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223061) in GitLab 13.4. > - Support for `grpcs` [introduced](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/7) in GitLab 13.6. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300960) in GitLab 13.10, KAS became available on GitLab.com under `wss://kas.gitlab.com` through an Early Adopter Program. -> - Introduced in GitLab 13.11, the GitLab Kubernetes Agent became available to every project on GitLab.com. -> - The GitLab Kubernetes Agent was [moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) to GitLab Free in 14.5. +> - Introduced in GitLab 13.11, the GitLab Agent became available to every project on GitLab.com. +> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. +> - [Renamed](https://gitlab.com/groups/gitlab-org/-/epics/7167) from "GitLab Kubernetes Agent" to "GitLab Agent for Kubernetes" in GitLab 14.6. -The [GitLab Kubernetes Agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) ("Agent", for short) +The [GitLab Agent for Kubernetes](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) ("Agent", for short) is an active in-cluster component for connecting Kubernetes clusters to GitLab safely to support cloud-native deployment, management, and monitoring. The Agent is installed into the cluster through code, providing you with a fast, safe, stable, and scalable solution. +INFO: +Get Network Security Alerts in GitLab by upgrading to Ultimate. +[Try a free 30-day trial now](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-cluster-agent-docs). + With GitOps, you can manage containerized clusters and applications from a Git repository that: - Is the single source of truth of your system. @@ -34,7 +39,7 @@ the all-in-one DevOps platform for your product and your team. ## Agent's features -By using the GitLab Kubernetes Agent, you can: +By using the Agent, you can: - Connect GitLab with a Kubernetes cluster behind a firewall or a Network Address Translation (NAT). @@ -49,7 +54,7 @@ from GitLab CI/CD jobs while keeping the cluster's APIs safe and unexposed to the internet. - [Deploy the GitLab Runner in a Kubernetes cluster](https://docs.gitlab.com/runner/install/kubernetes-agent.html). -See the [GitLab Kubernetes Agent roadmap](https://gitlab.com/groups/gitlab-org/-/epics/3329) to track its development. +See the [Agent roadmap](https://gitlab.com/groups/gitlab-org/-/epics/3329) to track its development. To contribute to the Agent, see the [Agent's development documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/doc). @@ -64,7 +69,7 @@ sequenceDiagram participant D as Developer participant A as Application code repository participant M as Manifest repository - participant K as Kubernetes Agent + participant K as GitLab Agent participant C as Agent configuration repository loop Regularly K-->>C: Grab the configuration @@ -81,7 +86,7 @@ For more details, refer to our [architecture documentation](https://gitlab.com/g ## Install the Agent in your cluster -See how to [install the GitLab Kubernetes Agent in your cluster](install/index.md). +See how to [install the Agent in your cluster](install/index.md). ## GitOps deployments **(PREMIUM)** @@ -129,7 +134,28 @@ with the following differences: - When you define a configuration repository, you must do so with [Cilium settings](repository.md#surface-network-security-alerts-from-cluster-to-gitlab). - You do not need to specify the `gitops` configuration section. -## Remove the GitLab Kubernetes Agent +## Remove an agent + +1. Get the `<cluster-agent-id>` and the `<cluster-agent-token-id>` from a query in the interactive GraphQL explorer. +For GitLab.com, go to <https://gitlab.com/-/graphql-explorer> to open GraphQL Explorer. +For self-managed GitLab instances, go to `https://gitlab.example.com/-/graphql-explorer`, replacing `gitlab.example.com` with your own instance's URL. + + ```graphql + query{ + project(fullPath: "<full-path-to-agent-configuration-project>") { + clusterAgent(name: "<agent-name>") { + id + tokens { + edges { + node { + id + } + } + } + } + } + } + ``` 1. Remove an Agent record with GraphQL by deleting the `clusterAgent` and the `clusterAgentToken`. @@ -158,7 +184,7 @@ with the following differences: } ``` -1. Delete the GitLab Kubernetes Agent in your cluster: +1. Delete the Agent in your cluster: ```shell kubectl delete -n gitlab-kubernetes-agent -f ./resources.yml @@ -166,14 +192,14 @@ with the following differences: ## Troubleshooting -If you face any issues while using GitLab Kubernetes Agent, you can read the -service logs with the following command +If you face any issues while using the Agent, read the +service logs with the following command: ```shell kubectl logs -f -l=app=gitlab-kubernetes-agent -n gitlab-kubernetes-agent ``` -GitLab administrators can additionally view the [Kubernetes Agent Server logs](../../../administration/clusters/kas.md#troubleshooting). +GitLab administrators can additionally view the [GitLab Agent Server logs](../../../administration/clusters/kas.md#troubleshooting). ### Agent logs diff --git a/doc/user/clusters/agent/install/index.md b/doc/user/clusters/agent/install/index.md index fad9d4f08f1..cf8467a8d28 100644 --- a/doc/user/clusters/agent/install/index.md +++ b/doc/user/clusters/agent/install/index.md @@ -4,11 +4,11 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install the GitLab Kubernetes Agent **(FREE)** +# Install the GitLab Agent **(FREE)** -> [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) to GitLab Free in 14.5. +> [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. -To get started with the GitLab Kubernetes Agent, install it in your cluster. +To get started with the Agent, install it in your cluster. Pre-requisites: @@ -17,24 +17,24 @@ Pre-requisites: ## Installation steps -To install the [GitLab Kubernetes Agent](../index.md) in your cluster: +To install the [Agent](../index.md) in your cluster: -1. [Set up the Kubernetes Agent Server](#set-up-the-kubernetes-agent-server) for your GitLab instance. +1. [Set up the Agent Server](#set-up-the-agent-server) for your GitLab instance. 1. [Define a configuration repository](#define-a-configuration-repository). -1. [Create an Agent record in GitLab](#create-an-agent-record-in-gitlab). -1. [Install the Agent into the cluster](#install-the-agent-into-the-cluster). -1. [Generate and copy a Secret token used to connect to the Agent](#create-the-kubernetes-secret). +1. [Register an agent with GitLab](#register-an-agent-with-gitlab). +1. [Install the agent into the cluster](#install-the-agent-into-the-cluster). +1. [Generate and copy a Secret token used to connect to the agent](#create-the-kubernetes-secret). 1. [Create manifest files](#create-manifest-files). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch a GitLab 14.2 [walking-through video](https://www.youtube.com/watch?v=XuBpKtsgGkE) with this process. -### Set up the Kubernetes Agent Server +### Set up the Agent Server -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.10, the GitLab Kubernetes Agent Server (KAS) became available on GitLab.com under `wss://kas.gitlab.com`. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.10, the GitLab Agent Server (KAS) became available on GitLab.com under `wss://kas.gitlab.com`. To use the KAS: -- If you are a self-managed user, follow the instructions to [install the Kubernetes Agent Server](../../../../administration/clusters/kas.md). +- If you are a self-managed user, follow the instructions to [install the Agent Server](../../../../administration/clusters/kas.md). - If you are a GitLab.com user, when you [set up the configuration repository](#define-a-configuration-repository) for your agent, use `wss://kas.gitlab.com` as the `--kas-address`. ### Define a configuration repository @@ -42,7 +42,7 @@ To use the KAS: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in GitLab 13.7, the Agent manifest configuration can be added to multiple directories (or subdirectories) of its repository. > - Group authorization was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. -To configure an Agent, you need: +To create an agent, you need: 1. A GitLab repository to hold the configuration file. 1. Install the Agent in a cluster. @@ -58,27 +58,24 @@ In your repository, add the Agent configuration file under: Make sure that `<agent-name>` conforms to the [Agent's naming format](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/identity_and_auth.md#agent-identity-and-name). -Your `config.yaml` file specifies all configurations of the Agent, such as: +WARNING: +The agent is only recognized if you use `.yaml` extension for the `config.yaml` file. The extension `.yml` is **not** recognized. -- The manifest projects to synchronize. -- The groups that can access this Agent via the [CI/CD Tunnel](../ci_cd_tunnel.md). -- The address of the `hubble-relay` for the Network Security policy integrations. +You **don't have to add any content** to this file when you create it. The fact that the file exists +tells GitLab that this is an agent configuration file. It doesn't do anything so far, but, later on, you can use this +file to [configure the agent](../repository.md) by setting up parameters such as: -As an example, a minimal Agent configuration that sets up only the manifest -synchronizations is: +- Groups and projects that can access the agent via the [CI/CD Tunnel](../ci_cd_tunnel.md). +- [Manifest projects to synchronize](../repository.md#synchronize-manifest-projects). +- The address of the `hubble-relay` for the [Network Security policy integrations](../../../project/clusters/protect/index.md). -```yaml -gitops: - manifest_projects: - # The `id` is the path to the Git repository holding your manifest files - - id: "path/to/your-manifest-project-1" - paths: - - glob: '/**/*.{yaml,yml,json}' -``` +To see all the settings available, read the [Agent configuration repository documentation](../repository.md). -All the options for the [Kubernetes Agent configuration repository](../repository.md) are documented separately. +### Access your cluster from GitLab CI/CD -### Create an Agent record in GitLab +Use the [CI/CD Tunnel](../ci_cd_tunnel.md#example-for-a-kubectl-command-using-the-cicd-tunnel) to access your cluster from GitLab CI/CD. + +### Register an agent with GitLab > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5786) in GitLab 14.1, you can create a new Agent record directly from the GitLab UI. @@ -91,7 +88,7 @@ In GitLab: 1. Ensure that [GitLab CI/CD is enabled in your project](../../../../ci/enable_or_disable_ci.md#enable-cicd-in-a-project). 1. From your project's sidebar, select **Infrastructure > Kubernetes clusters**. 1. Select **Actions**. -1. From the **Select an Agent** dropdown, select the Agent you want to connect and select **Register Agent** to access the installation form. +1. From the **Select an agent** dropdown, select the agent you want to connect and select **Register an agent** to access the installation form. 1. The form reveals your registration token. Securely store this secret token as you cannot view it again. 1. Copy the command under **Recommended installation method**. @@ -100,7 +97,7 @@ In your computer: 1. Open your local terminal and connect to your cluster. 1. Run the command you copied from the installation form. -### Install the Agent into the cluster +### Install the agent into the cluster To install the in-cluster component of the Agent, first you need to define a namespace. To create a new namespace, for example, `gitlab-kubernetes-agent`, run: @@ -113,7 +110,7 @@ To perform a one-liner installation, run the command below. Make sure to replace - `your-agent-token` with the token received from the previous step (identified as `secret` in the JSON output). - `gitlab-kubernetes-agent` with the namespace you defined in the previous step. -- `wss://kas.gitlab.example.com` with the configured access of the Kubernetes Agent Server (KAS). For GitLab.com users, the KAS is available under `wss://kas.gitlab.com`. +- `wss://kas.gitlab.example.com` with the configured access of the Agent Server (KAS). For GitLab.com users, the KAS is available under `wss://kas.gitlab.com`. - `--agent-version=vX.Y.Z` with the latest released patch version matching your GitLab installation's major and minor versions. For example, for GitLab v13.9.0, use `--agent-version=v13.9.1`. You can find your GitLab version under the "Help/Help" menu. ```shell @@ -151,7 +148,7 @@ Kubernetes resources required for the Agent to be installed. You can modify this example [`resources.yml` file](#example-resourcesyml-file) in the following ways: - Replace `namespace: gitlab-kubernetes-agent` with `namespace: <YOUR-DESIRED-NAMESPACE>`. -- You can configure `kas-address` (Kubernetes Agent Server) in several ways. +- You can configure `kas-address` (Agent Server) in several ways. The agent can use the WebSockets or gRPC protocols to connect to the Agent Server. Select the option appropriate for your cluster configuration and GitLab architecture: - The `wss` scheme (an encrypted WebSockets connection) is specified by default @@ -334,7 +331,7 @@ data: ## Example projects -The following example projects can help you get started with the Kubernetes Agent. +The following example projects can help you get started with the Agent. - [Configuration repository](https://gitlab.com/gitlab-org/configure/examples/kubernetes-agent) - This basic GitOps example deploys NGINX: [Manifest repository](https://gitlab.com/gitlab-org/configure/examples/gitops-project) @@ -342,18 +339,44 @@ The following example projects can help you get started with the Kubernetes Agen ## View installed Agents Users with at least the [Developer](../../../permissions.md) can access the user interface -for the GitLab Kubernetes Agent at **Infrastructure > Kubernetes clusters**, under the +for the Agent at **Infrastructure > Kubernetes clusters**, under the **Agent** tab. This page lists all registered agents for the current project, and the configuration directory for each agent: - + -Additional management interfaces are planned for the GitLab Kubernetes Agent. +Additional management interfaces are planned for the GitLab Agent. [Provide more feedback in the related epic](https://gitlab.com/groups/gitlab-org/-/epics/4739). +## View Agent activity information + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/277323) in GitLab 14.6. + +Users with at least the [Developer](../../../permissions.md) can view the Agent's activity events. +The activity logs help you to identify problems and get the information you need for troubleshooting. +You can see events from a week before the current date. +To access an agent's activity: + +1. Go to your agent's configuration repository. +1. From the sidebar, select **Infrastructure > Kubernetes clusters**. +1. Select the **Agent** tab. +1. Select the agent you want to see the activity. + +You can see the following events on the activity list: + +- Agent registration: + - When a new token is **created**. +- Connection events: + - When an agent is successfully **connected** to a cluster. + +Note that the connection status is logged when you connect an agent for the first time +or after more than an hour of inactivity. + + + ## Upgrades and version compatibility -The GitLab Kubernetes Agent is comprised of two major components: `agentk` and `kas`. +The Agent is comprised of two major components: `agentk` and `kas`. As we provide `kas` installers built into the various GitLab installation methods, the required `kas` version corresponds to the GitLab `major.minor` (X.Y) versions. At the same time, `agentk` and `kas` can differ by 1 minor version in either direction. For example, diff --git a/doc/user/clusters/agent/repository.md b/doc/user/clusters/agent/repository.md index 6ceb2766cc9..c8ab037118e 100644 --- a/doc/user/clusters/agent/repository.md +++ b/doc/user/clusters/agent/repository.md @@ -4,20 +4,20 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Kubernetes Agent configuration repository **(FREE)** +# Agent configuration repository **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.7. -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.11, the Kubernetes Agent became available on GitLab.com. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in GitLab 13.7. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.11, the GitLab Agent became available on GitLab.com. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) the `ci_access` attribute in GitLab 14.3. +> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332227) in GitLab 14.0, the `resource_inclusions` and `resource_exclusions` attributes were removed and `reconcile_timeout`, `dry_run_strategy`, `prune`, `prune_timeout`, `prune_propagation_policy`, and `inventory_policy` attributes were added. -> - The `ci_access` attribute was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. -> - The GitLab Kubernetes Agent was [moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) to GitLab Free in 14.5. WARNING: This feature might not be available to you. Check the **version history** note above for details. -The [GitLab Kubernetes Agent integration](index.md) supports hosting your configuration for -multiple GitLab Kubernetes Agents in a single repository. These agents can be running -in the same cluster or in multiple clusters, and potentially with more than one Agent per cluster. +The [GitLab Agent](index.md) supports hosting your configuration for +multiple agents in a single repository. These agents can be running +in the same cluster or in multiple clusters, and potentially with more than one agent per cluster. The Agent bootstraps with the GitLab installation URL and an authentication token, and you provide the rest of the configuration in your repository, following @@ -128,7 +128,7 @@ operations. If such functionality is needed, you may use multiple agents reading manifests from the same repository. Ensure not to specify "overlapping" globs to avoid synchronizing the same files more than once. -This is detected by the GitLab Kubernetes Agent and leads to an error. +This is detected by the Agent and leads to an error. INCORRECT - both globs match `*.yaml` files in the root directory: @@ -299,7 +299,7 @@ cilium: hubble_relay_address: "hubble-relay.gitlab-managed-apps.svc.cluster.local:80" ``` -## Scan your container images for vulnerabilities +## Scan your container images for vulnerabilities **(ULTIMATE)** You can use [cluster image scanning](../../application_security/cluster_image_scanning/index.md) to scan container images in your cluster for security vulnerabilities. @@ -385,7 +385,7 @@ In this example, the following resources are scanned: ## Debugging -To debug the cluster-side component (`agentk`) of the GitLab Kubernetes Agent, set the log +To debug the cluster-side component (`agentk`) of the Agent, set the log level according to the available options: - `off` diff --git a/doc/user/clusters/cost_management.md b/doc/user/clusters/cost_management.md index 3ad994ecd7e..14850ca85b3 100644 --- a/doc/user/clusters/cost_management.md +++ b/doc/user/clusters/cost_management.md @@ -22,8 +22,7 @@ insights within GitLab: ## Configure cluster cost management -To get started with cluster cost management, you need [Maintainer](../permissions.md) -permissions in a project or group. +To get started with cluster cost management, you need the [Maintainer role](../permissions.md) in a project or group. 1. Clone the [`kubecost-cost-model`](https://gitlab.com/gitlab-examples/kubecost-cost-model/) example repository, which contains minor modifications to the upstream Kubecost diff --git a/doc/user/clusters/environments.md b/doc/user/clusters/environments.md index 72bc7b398dc..71eb08ee640 100644 --- a/doc/user/clusters/environments.md +++ b/doc/user/clusters/environments.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Cluster Environments (DEPRECATED) **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13392) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3 for group-level clusters. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14809) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.4 for instance-level clusters. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13392) in GitLab 12.3 for group-level clusters. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14809) in GitLab 12.4 for instance-level clusters. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. WARNING: diff --git a/doc/user/clusters/img/gitlab_agent_activity_events_v14_6.png b/doc/user/clusters/img/gitlab_agent_activity_events_v14_6.png Binary files differnew file mode 100644 index 00000000000..90b41ee849c --- /dev/null +++ b/doc/user/clusters/img/gitlab_agent_activity_events_v14_6.png diff --git a/doc/user/clusters/integrations.md b/doc/user/clusters/integrations.md index d04cf64d93a..ee7452537fd 100644 --- a/doc/user/clusters/integrations.md +++ b/doc/user/clusters/integrations.md @@ -32,9 +32,9 @@ to automate this step. Prometheus and Elastic Stack cluster integrations can only be enabled for clusters [connected through cluster certificates](../project/clusters/add_existing_cluster.md). -To enable Prometheus for your cluster connected through the [GitLab Kubernetes Agent](agent/index.md), you can [integrate it manually](../project/integrations/prometheus.md#manual-configuration-of-prometheus). +To enable Prometheus for your cluster connected through the [GitLab Agent](agent/index.md), you can [integrate it manually](../project/integrations/prometheus.md#manual-configuration-of-prometheus). -There is no option to enable Elastic Stack for your cluster if it is connected with the GitLab Kubernetes Agent. +There is no option to enable Elastic Stack for your cluster if it is connected with the GitLab Agent. Follow this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/300230) for updates. ## Prometheus cluster integration @@ -44,7 +44,7 @@ Follow this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/300230) for up WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. However, you can **still use** Prometheus for Kubernetes clusters connected to GitLab through the -[GitLab Kubernetes Agent](agent/index.md) by [enabling Prometheus manually](../project/integrations/prometheus.md#manual-configuration-of-prometheus). +[Agent](agent/index.md) by [enabling Prometheus manually](../project/integrations/prometheus.md#manual-configuration-of-prometheus). You can integrate your Kubernetes cluster with [Prometheus](https://prometheus.io/) for monitoring key metrics of your diff --git a/doc/user/clusters/management_project.md b/doc/user/clusters/management_project.md index 1332310b850..e28f9275b50 100644 --- a/doc/user/clusters/management_project.md +++ b/doc/user/clusters/management_project.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: The cluster management project was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. -To manage cluster applications, use the [GitLab Kubernetes Agent](agent/index.md) +To manage cluster applications, use the [GitLab Agent](agent/index.md) with the [Cluster Management Project Template](management_project_template.md). A project can be designated as the management project for a cluster. diff --git a/doc/user/clusters/management_project_template.md b/doc/user/clusters/management_project_template.md index c663246cdd8..2d7e89ef765 100644 --- a/doc/user/clusters/management_project_template.md +++ b/doc/user/clusters/management_project_template.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25318) in GitLab 12.10 with Helmfile support via Helm v2. > - Helm v2 support was [dropped](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63577) in GitLab 14.0. Use Helm v3 instead. -> - [Migrated](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/24) to the GitLab Kubernetes Agent in GitLab 14.5. +> - [Migrated](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/24) to the GitLab Agent in GitLab 14.5. Use a repository to install, manage, and deploy clusters applications through code. @@ -31,10 +31,10 @@ you can manage cluster applications with [Helm v3](https://helm.sh/). - An `applications` directory with a `helmfile.yaml` configured for each application available in the template. -## Use the Kubernetes Agent with the Cluster Management Project Template +## Use the Agent with the Cluster Management Project Template To use a new project created from the Cluster Management Project Template -with a cluster connected to GitLab through the [GitLab Kubernetes Agent](agent/index.md), +with a cluster connected to GitLab through the [GitLab Agent](agent/index.md), you have two options: - [Use one single project](#single-project) to configure the Agent and manage cluster applications. diff --git a/doc/user/clusters/migrating_from_gma_to_project_template.md b/doc/user/clusters/migrating_from_gma_to_project_template.md index 2da058fb5bc..20c4408d57e 100644 --- a/doc/user/clusters/migrating_from_gma_to_project_template.md +++ b/doc/user/clusters/migrating_from_gma_to_project_template.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Migrate from GitLab Managed Apps to Cluster Management Projects +# Migrate from GitLab Managed Apps to Cluster Management Projects **(FREE)** The [GitLab Managed Apps](applications.md) were deprecated in GitLab 14.0 in favor of [Cluster Management Projects](management_project.md). diff --git a/doc/user/compliance/compliance_dashboard/index.md b/doc/user/compliance/compliance_dashboard/index.md deleted file mode 100644 index e7f6f908860..00000000000 --- a/doc/user/compliance/compliance_dashboard/index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: '../compliance_report/index.md' -remove_date: '2021-10-23' ---- - -This file was moved to [another location](../compliance_report/index.md). - -<!-- This redirect file can be deleted after <2021-10-23>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/compliance/license_compliance/img/policies_maintainer_edit_v14_3.png b/doc/user/compliance/license_compliance/img/policies_maintainer_edit_v14_3.png Binary files differindex 85b2a52e04b..256c66bf7d8 100644 --- a/doc/user/compliance/license_compliance/img/policies_maintainer_edit_v14_3.png +++ b/doc/user/compliance/license_compliance/img/policies_maintainer_edit_v14_3.png diff --git a/doc/user/compliance/license_compliance/index.md b/doc/user/compliance/license_compliance/index.md index 319c1ca6278..f89165e7e2d 100644 --- a/doc/user/compliance/license_compliance/index.md +++ b/doc/user/compliance/license_compliance/index.md @@ -14,6 +14,10 @@ project's dependencies for their licenses. You can then decide whether to allow each license. For example, if your application uses an external (open source) library whose license is incompatible with yours, then you can deny the use of that license. +INFO: +Try License Compliance scanning to search project dependencies in GitLab Ultimate. +[It's free for 30 days](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=u-compliance-docs). + You can take advantage of License Compliance by either: - [Including the job](#configuration) @@ -22,8 +26,9 @@ You can take advantage of License Compliance by either: [Auto License Compliance](../../../topics/autodevops/stages.md#auto-license-compliance), provided by [Auto DevOps](../../../topics/autodevops/index.md). -The [License Finder](https://github.com/pivotal/LicenseFinder) scan tool runs as part of the CI/CD -pipeline, and detects the licenses in use. GitLab checks the License Compliance report, compares the +To detect the licenses in use, License Compliance uses the [License Finder](https://github.com/pivotal/LicenseFinder) scan tool that runs as part of the CI/CD pipeline. +For the job to activate, License Finder needs to find a compatible package definition in the project directory. For details, see the [Activation on License Finder documentation](https://github.com/pivotal/LicenseFinder#activation). +GitLab checks the License Compliance report, compares the licenses between the source and target branches, and shows the information right on the merge request. Denied licenses are indicated by a `x` red icon next to them as well as new licenses that need a decision from you. In addition, you can [manually allow or deny](#policies) licenses in your @@ -47,6 +52,7 @@ When GitLab detects a **Denied** license, you can view it in the [license list](  You can view and modify existing policies from the [policies](#policies) tab. +  ## License expressions @@ -126,7 +132,7 @@ the `license_management` job, so you must migrate to the `license_scanning` job `License-Scanning.gitlab-ci.yml` template. The results are saved as a -[License Compliance report artifact](../../../ci/yaml/index.md#artifactsreportslicense_scanning) +[License Compliance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportslicense_scanning) that you can later download and analyze. Due to implementation limitations, we always take the latest License Compliance artifact available. Behind the scenes, the [GitLab License Compliance Docker image](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder) @@ -172,7 +178,7 @@ For that, a `SETUP_CMD` CI/CD variable can be passed to the container, with the required commands to run before the license detection. If present, this variable overrides the setup step necessary to install all the packages -of your application (e.g.: for a project with a `Gemfile`, the setup step could be +of your application (for example: for a project with a `Gemfile`, the setup step could be `bundle install`). For example: @@ -190,8 +196,8 @@ directory of your project. ### Working with Monorepos -Depending on your language, you may need to specify the path to the individual -projects of a monorepo using the `LICENSE_FINDER_CLI_OPTS` variable. Passing in +Depending on your language, you may need to specify the path to the individual +projects of a monorepo using the `LICENSE_FINDER_CLI_OPTS` variable. Passing in the project paths can significantly speed up builds over using the `--recursive` license_finder option. @@ -540,24 +546,24 @@ configured to use this as the default `CA_CERT_PATH`. ### Configuring Go projects To configure [Go modules](https://github.com/golang/go/wiki/Modules) -based projects, specify [CI/CD variables](https://golang.org/pkg/cmd/go/#hdr-Environment_variables) +based projects, specify [CI/CD variables](https://pkg.go.dev/cmd/go#hdr-Environment_variables) in the `license_scanning` job's [variables](#available-cicd-variables) section in `.gitlab-ci.yml`. -If a project has [vendored](https://golang.org/pkg/cmd/go/#hdr-Vendor_Directories) its modules, +If a project has [vendored](https://pkg.go.dev/cmd/go#hdr-Vendor_Directories) its modules, then the combination of the `vendor` directory and `mod.sum` file are used to detect the software licenses associated with the Go module dependencies. #### Using private Go registries -You can use the [`GOPRIVATE`](https://golang.org/pkg/cmd/go/#hdr-Environment_variables) -and [`GOPROXY`](https://golang.org/pkg/cmd/go/#hdr-Environment_variables) +You can use the [`GOPRIVATE`](https://pkg.go.dev/cmd/go#hdr-Environment_variables) +and [`GOPROXY`](https://pkg.go.dev/cmd/go#hdr-Environment_variables) environment variables to control where modules are sourced from. Alternatively, you can use -[`go mod vendor`](https://golang.org/ref/mod#tmp_28) to vendor a project's modules. +[`go mod vendor`](https://go.dev/ref/mod#tmp_28) to vendor a project's modules. #### Custom root certificates for Go -You can specify the [`-insecure`](https://golang.org/pkg/cmd/go/internal/get/) flag by exporting the -[`GOFLAGS`](https://golang.org/cmd/go/#hdr-Environment_variables) +You can specify the [`-insecure`](https://pkg.go.dev/cmd/go/internal/get) flag by exporting the +[`GOFLAGS`](https://pkg.go.dev/cmd/go#hdr-Environment_variables) environment variable. For example: ```yaml diff --git a/doc/user/crm/crm_contacts_v14_6.png b/doc/user/crm/crm_contacts_v14_6.png Binary files differnew file mode 100644 index 00000000000..37a615f3926 --- /dev/null +++ b/doc/user/crm/crm_contacts_v14_6.png diff --git a/doc/user/crm/crm_organizations_v14_6.png b/doc/user/crm/crm_organizations_v14_6.png Binary files differnew file mode 100644 index 00000000000..2bde3823cc7 --- /dev/null +++ b/doc/user/crm/crm_organizations_v14_6.png diff --git a/doc/user/crm/index.md b/doc/user/crm/index.md new file mode 100644 index 00000000000..d68ce0a4f7a --- /dev/null +++ b/doc/user/crm/index.md @@ -0,0 +1,141 @@ +--- +stage: Plan +group: Product Planning +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Customer relations management (CRM) **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2256) in GitLab 14.6 [with a flag](../../administration/feature_flags.md) named `customer_relations`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `customer_relations`. +On GitLab.com, this feature is not available. +You should not use this feature for production environments. + +With customer relations management (CRM) you can create a record of contacts +(individuals) and organizations (companies) and relate them to issues. + +You can use contacts and organizations to tie work to customers for billing and reporting purposes. +To read more about what is planned for the future, see [issue 2256](https://gitlab.com/gitlab-org/gitlab/-/issues/2256). + +## Contacts + +### View contacts linked to a group + +To view a group's contacts: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Customer relations > Contacts**. + + + +### Create a contact + +To create a contact: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Customer relations > Contacts**. +1. Select **New contact**. +1. Complete all required fields. +1. Select **Create new contact**. + +You can also [create](../../api/graphql/reference/index.md#mutationcustomerrelationscontactcreate) +contacts using the GraphQL API. + +### Edit a contact + +To edit an existing contact: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Customer relations > Contacts**. +1. Next to the contact you wish to edit, select **Edit** (**{pencil}**). +1. Edit the required fields. +1. Select **Save changes**. + +You can also [edit](../../api/graphql/reference/index.md#mutationcustomerrelationscontactupdate) +contacts using the GraphQL API. + +## Organizations + +### View organizations + +To view a group's organizations: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Customer relations > Organizations**. + + + +### Create an organization + +To create an organization: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Customer relations > Organizations**. +1. Select **New organization**. +1. Complete all required fields. +1. Select **Create new organization**. + +You can also [create](../../api/graphql/reference/index.md#mutationcustomerrelationsorganizationcreate) +organizations using the GraphQL API. + +### Edit an organization + +You can only [edit](../../api/graphql/reference/index.md#mutationcustomerrelationsorganizationupdate) +organizations using the GraphQL API. + +## Issues + +### View issues linked to a contact + +To view a contact's issues: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Customer relations > Contacts**. +1. Next to the contact whose issues you wish to view, select **View issues** (**{issues}**). + +### View issues linked to an organization + +To view an organization's issues: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Customer relations > Organizations**. +1. Next to the organization whose issues you wish to view, select **View issues** (**{issues}**). + +### View contacts linked to an issue + +You can view contacts associated with an issue in the right sidebar. + +To view a contact's details, hover over the contact's name. + + + +You can also view issue contacts using the +[GraphQL](../../api/graphql/reference/index.md#mutationcustomerrelationsorganizationcreate) +API. + +### Add or remove issue contacts + +Prerequisites: + +- You must have at least the [Developer role](../permissions.md#project-members-permissions) for a group. + +### Add contacts to an issue + +To add contacts to an issue use the `/add_contacts` +[quick action](../project/quick_actions.md). + +You can also add, remove, or replace issue contacts using the +[GraphQL](../../api/graphql/reference/index.md#mutationissuesetcrmcontacts) +API. + +### Remove contacts from an issue + +To remove contacts from an issue use the `/remove_contacts` +[quick action](../project/quick_actions.md). + +You can also add, remove, or replace issue contacts using the +[GraphQL](../../api/graphql/reference/index.md#mutationissuesetcrmcontacts) +API. diff --git a/doc/user/crm/issue_crm_contacts_v14_6.png b/doc/user/crm/issue_crm_contacts_v14_6.png Binary files differnew file mode 100644 index 00000000000..9c18b1a8cdb --- /dev/null +++ b/doc/user/crm/issue_crm_contacts_v14_6.png diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 4d6cff96169..7831fdff249 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -34,6 +34,20 @@ You can create comments in places like: Each object can have as many as 5,000 comments. +## Mentions + +You can mention a user or a group present in your GitLab instance with `@username` or +`@groupname`. All mentioned users are notified with to-do items and emails. +Users can change this setting for themselves in the +[notification settings](../profile/notifications.md). + +You can quickly see which comments involve you, because +mentions for yourself (the user currently signed in) are highlighted +in a different color. + +Avoid mentioning `@all` in issues and merge requests, because it sends an email notification +to all the members of that project's group. This might be interpreted as spam. + ## Add a comment to a merge request diff You can add comments to a merge request diff. These comments @@ -90,8 +104,10 @@ An icon is displayed on the image and a comment field is displayed. If you have ["reply by email"](../../administration/reply_by_email.md) configured, you can reply to comments by sending an email. -- When you reply to a standard comment, another standard comment is created. +- When you reply to a standard comment, it creates another standard comment. - When you reply to a threaded comment, it creates a reply in the thread. +- When you [send an email to an issue email address](../project/issues/managing_issues.md#copy-issue-email-address), + it creates a standard comment. You can use [Markdown](../markdown.md) and [quick actions](../project/quick_actions.md) in your email replies. @@ -145,7 +161,7 @@ For issues and merge requests with many comments, you can filter the page to sho 1. Open a merge request's **Discussion** tab, or epic or issue's **Overview** tab. 1. On the right side of the page, select from the filter: - - **Show all activity**: Display all user comments and system notes + - **Show all activity**: Display all user comments and system notes. (issue updates, mentions from other issues, changes to the description, and so on). - **Show comments only**: Display only user comments. - **Show history only**: Display only activity notes. @@ -155,6 +171,27 @@ For issues and merge requests with many comments, you can filter the page to sho GitLab saves your preference, so it persists when you visit the same page again from any device you're logged into. +## View description change history **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10103) in GitLab 12.6. + +You can see changes to the description listed in the history. + +To compare the changes, select **Compare with previous version**. + +## Change activity sort order + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14588) in GitLab 12.10. + +You can reverse the default order and interact with the activity feed sorted by most recent items +at the top. Your preference is saved in local storage and automatically applies to every issue, +merge request, or epic you view. + +To change the activity sort order: + +1. Select the **Oldest first** (or **Newest first**) dropdown list. +1. Select either oldest or newest items to be shown first. + ## Assign an issue to the commenting user > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/191455) in GitLab 13.1. diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index a3aecff6f73..8d858a282dd 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -85,7 +85,7 @@ are included when cloning. Top-level groups created after August 12, 2021 have delayed project deletion enabled by default. Projects are permanently deleted after a seven-day delay. -You can disable this by changing the [group setting](../group/index.md#enable-delayed-project-removal). +You can disable this by changing the [group setting](../group/index.md#enable-delayed-project-deletion). ## Alternative SSH port @@ -141,7 +141,24 @@ the related documentation. | [Scheduled Job Archival](../../user/admin_area/settings/continuous_integration.md#archive-jobs) | 3 months | Never | | Max test cases per [unit test report](../../ci/unit_test_reports.md) | `500_000` | Unlimited | | [Max registered runners](../../administration/instance_limits.md#number-of-registered-runners-per-scope) | Free tier: `50` per-group / `50` per-project <br/> All paid tiers: `1_000` per-group / `1_000` per-project | `1_000` per-group / `1_000` per-project | -| [Limit dotenv variables](../../administration/instance_limits.md#limit-dotenv-variables) | Free tier: `50` / Premium tier: `100` / Ultimate tier: `150` | Unlimited | +| [Limit dotenv variables](../../administration/instance_limits.md#limit-dotenv-variables) | Free tier: `50` / Premium tier: `100` / Ultimate tier: `150` | 150 | + +## Package registry limits + +The [maximum file size](../../administration/instance_limits.md#file-size-limits) +for a package uploaded to the [GitLab Package Registry](../../user/packages/package_registry/index.md) +varies by format: + +| Package type | GitLab.com | +|--------------|------------| +| Conan | 5 GB | +| Generic | 5 GB | +| Helm | 5 MB | +| Maven | 5 GB | +| npm: | 5 GB | +| NuGet | 5 GB | +| PyPI | 5 GB | +| Terraform | 1 GB | ## Account and limit settings @@ -200,11 +217,11 @@ The following limits apply for [Webhooks](../project/integrations/webhooks.md): | [Number of webhooks](../../administration/instance_limits.md#number-of-webhooks) | `100` per project, `50` per group | `100` per project, `50` per group | | Maximum payload size | 25 MB | 25 MB | -## Shared Runner Cloud runners +## Runner SaaS -GitLab has shared runners on GitLab.com that you can use to run your CI jobs. +Runner SaaS is the hosted, secure, and managed build environment you can use to run CI/CD jobs for your GitLab.com hosted project. -For more information, see [GitLab Runner Cloud runners](../../ci/runners/index.md). +For more information, see [Runner SaaS](../../ci/runners/index.md). ## Sidekiq diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index 9c95b2b21a4..b2b16321488 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -12,7 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. To connect clusters to GitLab, -use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). +use the [GitLab Agent](../../clusters/agent/index.md). Similar to [project-level](../../project/clusters/index.md) and [instance-level](../../instance/clusters/index.md) Kubernetes clusters, diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md index 9378b3922b5..2214a18475a 100644 --- a/doc/user/group/custom_project_templates.md +++ b/doc/user/group/custom_project_templates.md @@ -9,49 +9,49 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6861) in GitLab 11.6. -[Group owners](../permissions.md#group-members-permissions) can set a subgroup to -be the source of project templates that are selectable when a new project is created -in the group. These templates can be selected when you go to **New project > Create from template** -in the group and select the **Group** tab. +When you create a project, you can [choose from a list of templates](../project/working_with_projects.md#create-a-project). +These templates, for things like GitLab Pages or Ruby, populate the new project with a copy of the files contained in the +template. This information is identical to the information used by [GitLab project import/export](../project/settings/import_export.md) +and can help you start a new project more quickly. -Every project in the subgroup, but not nested subgroups, can be selected by members -of the group when a new project is created. +You can [customize the list](../project/working_with_projects.md#create-a-project) of available templates, so +that all projects in your group have the same list. To do this, you populate a subgroup with the projects you want to +use as templates. -- Public projects can be selected by any signed-in user as a template for a new project, - if all enabled [project features](../project/settings/index.md#sharing-and-permissions) - except for **GitLab Pages** and **Security & Compliance** are set to **Everyone With Access**. - The same applies to internal projects. -- Private projects can be selected only by users who are members of the projects. +You can also configure [custom templates for the instance](../admin_area/custom_project_templates.md). -Repository and database information that is copied over to each new project is identical to the -data exported with the [GitLab Project Import/Export](../project/settings/import_export.md). +## Set up group-level project templates -To set custom project templates at the instance level, see [Custom instance-level project templates](../admin_area/custom_project_templates.md). +Prerequisite: -## Set up group-level project templates +- You must have the [Owner role for the group](../permissions.md#group-members-permissions). To set up custom project templates in a group, add the subgroup that contains the project templates to the group settings: 1. In the group, create a [subgroup](subgroups/index.md). 1. [Add projects to the new subgroup](index.md#add-projects-to-a-group) as your templates. -1. In the left menu for the group, go to **Settings > General**. +1. In the left menu for the group, select **Settings > General**. 1. Expand **Custom project templates** and select the subgroup. -If all enabled [project features](../project/settings/index.md#sharing-and-permissions) -(except for GitLab Pages) are set to **Everyone With Access**, then every project -template in the subgroup is available to every member of the group. +The next time a group member creates a project, they can select any of the projects in the subgroup. -Any projects added to the subgroup later can be selected the next time a group member -creates a new project. +Projects in nested subgroups are not included in the template list. + +## Which projects are available as templates + +- Public and internal projects can be selected by any signed-in user as a template for a new project, + if all [project features](../project/settings/index.md#sharing-and-permissions) + except for **GitLab Pages** and **Security & Compliance** are set to **Everyone With Access**. +- Private projects can be selected only by users who are members of the projects. -### Example structure +## Example structure -Here's a sample group/project structure for project templates, for a hypothetical _Acme Co_: +Here's a sample group and project structure for project templates, for `myorganization`: ```plaintext # GitLab instance and group -gitlab.com/acmeco/ +gitlab.com/myorganization/ # Subgroups internal tools @@ -61,7 +61,7 @@ gitlab.com/acmeco/ # Project templates client-site-django client-site-gatsby - client-site-hTML + client-site-html # Other projects client-site-a diff --git a/doc/user/group/devops_adoption/index.md b/doc/user/group/devops_adoption/index.md index 36ccfc1031f..4151745189d 100644 --- a/doc/user/group/devops_adoption/index.md +++ b/doc/user/group/devops_adoption/index.md @@ -16,94 +16,81 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Overview table [added](https://gitlab.com/gitlab-org/gitlab/-/issues/335638) in GitLab 14.3. > - Adoption over time chart [added](https://gitlab.com/gitlab-org/gitlab/-/issues/337561) in GitLab 14.4. -Prerequisites: +DevOps Adoption shows you how groups in your organization adopt and use the most essential features of GitLab. -- You must have at least the [Reporter role](../../permissions.md) for the group. +You can use Group DevOps Adoption to: -To access Group DevOps Adoption, go to your group and select **Analytics > DevOps Adoption**. +- Identify specific subgroups that are lagging in their adoption of GitLab features, so you can guide them on +their DevOps journey. +- Find subgroups that have adopted certain features, and provide guidance to other subgroups on +how to use those features. +- Verify if you are getting the return on investment that you expected from GitLab. -Group DevOps Adoption shows you how individual groups and subgroups within your organization use the following features: + -- Dev - - Approvals - - Code owners - - Issues - - Merge requests -- Sec - - DAST - - Dependency Scanning - - Fuzz Testing - - SAST -- Ops - - Deployments - - Pipelines - - Runners +## View DevOps Adoption -When managing groups in the UI, you can add or remove your subgroups with the **Add or remove subgroups** -button, in the top right hand section of your Groups pages. +Prerequisite: -With DevOps Adoption you can: +- You must have at least the [Reporter role](../../permissions.md) for the group. -- Verify whether you are getting the return on investment that you expected from GitLab. -- Identify specific subgroups that are lagging in their adoption of GitLab, so you can help them along in their DevOps journey. -- Find the subgroups that have adopted certain features, and can provide guidance to other subgroups on how to use those features. +To view DevOps Adoption: - +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Analytics > DevOps adoption** + +## DevOps Adoption categories + +DevOps Adoption shows feature adoption for development, security, and operations. + +| Category | Feature | +| --- | --- | +| Development | Approvals<br>Code owners<br>Issues<br>Merge requests | +| Security | DAST<br>Dependency Scanning<br>Fuzz Testing<br>SAST | +| Operations | Deployments<br>Pipelines<br>Runners | + +## Feature adoption -Feature adoption is based on usage in the previous calendar month. Data is updated on the first day -of each month. If the monthly update fails, it tries again daily until successful. +DevOps Adoption shows feature adoption data for groups and subgroups for the previous calendar month. -## Enable data processing +A feature shows as **adopted** when a group has used the feature in a project during the time period. +This includes projects in any subgroups of the group. For example, if an issue was created in a project in a group, the group has adopted issues in that time. -Group DevOps Adoption relies on data that has been gathered by a weekly data processing task. -This task is disabled by default. +### Exceptions to feature adoption data -To begin using Group DevOps Adoption, access the feature for the first time. GitLab automatically -enables the data processing for that group. The group data doesn't appear immediately, because -GitLab requires around a minute to process it. +When GitLab measures DevOps Adoption, some common DevOps information is not included: -## What is displayed +- Dormant projects. It doesn't matter how many projects in the group use a feature. Even if you have many dormant projects, it doesn't lower the adoption. +- New GitLab features. Adoption is the total number of features adopted, not the percent of features. -DevOps Adoption displays feature adoption data for the given group -and any added subgroups for the current calendar month. -Each group appears as a separate row in the table. -For each row, a feature is considered "adopted" if it has been used in a project in the given group -during the time period (including projects in any subgroups of the given group). +## When DevOps Adoption data is gathered -## Adoption over time +A weekly task processes data for DevOps Adoption. This task is disabled until you access +DevOps Adoption for a group for the first time. -The **Adoption over time** chart in the **Overview** tab displays DevOps Adoption over time. The chart displays the total number of adopted features from the previous twelve months, -from when you enabled DevOps Adoption for the group. +The data processing task updates the data on the first day of each month. If the monthly update +fails, the task tries daily until it succeeds. -The tooltip displays information about the features tracked for individual months. +DevOps Adoption data may take up to a minute to appear while GitLab processes the group's data. -## When is a feature considered adopted +## View feature adoption over time -A feature is considered "adopted" if it has been used anywhere in the group in the specified time. -For example, if an issue was created in one project in a group, the group is considered to have -"adopted" issues in that time. +The **Adoption over time** chart shows the total number of adopted features from the previous +twelve months. The chart only shows data from when you enabled DevOps Adoption for the group. -## No penalties for common adoption patterns +To view feature adoption over time: -DevOps Adoption is designed not to penalize for any circumstances or practices that are common in DevOps. -Following this guideline, GitLab doesn't penalize for: +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Analytics > DevOps adoption**. +1. Select the **Overview** tab. -1. Having dormant projects. It's common for groups to have a mix of active and dormant projects, - so we should not consider adoption to be low if there are relatively many dormant projects. - This means we should not measure adoption by how many projects in the group have used a feature, - only by whether a feature was used anywhere in the group. -1. GitLab adding new features over time. It's common for group feature usage to be consistent - over time, so we should not consider adoption to have decreased if GitLab adds features. - This means we should not measure adoption by percentages, only total counts. +Tooltips display information about the features tracked for individual months. -## Add a subgroup +## Add or remove a subgroup -DevOps Adoption can also display data for subgroups within the given group, -to show you differences in adoption across the group. -To add subgroups to your Group DevOps Adoption report: +To add or remove a subgroup from the DevOps Adoption report: 1. Select **Add or remove subgroups**. -1. Select the subgroups you want to add and select **Save changes**. +1. Select the subgroup you want to add or remove and select **Save changes**. -The subgroup data might not appear immediately, because GitLab requires around a minute to collect -the data. +It may take up to a minute for subgroup data to appear while GitLab collects the data. diff --git a/doc/user/group/epics/epic_boards.md b/doc/user/group/epics/epic_boards.md index d184030718a..1bc1e4d703b 100644 --- a/doc/user/group/epics/epic_boards.md +++ b/doc/user/group/epics/epic_boards.md @@ -9,6 +9,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5067) in GitLab 13.10. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/290039) in GitLab 14.1. +INFO: +Try epic boards and more with a +[free 30-day trial of GitLab Ultimate](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-epics-boards-docs). + Epic boards build on the existing [epic tracking functionality](index.md) and [labels](../../project/labels.md). Your epics appear as cards in vertical lists, organized by their assigned labels. diff --git a/doc/user/group/epics/index.md b/doc/user/group/epics/index.md index 65e0f31324b..3889398e2f8 100644 --- a/doc/user/group/epics/index.md +++ b/doc/user/group/epics/index.md @@ -1,5 +1,4 @@ --- -type: reference, howto stage: Plan group: Product Planning info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments @@ -7,8 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Epics **(PREMIUM)** -> - Introduced in GitLab 10.2. -> - Single-level epics were [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) from GitLab Ultimate to GitLab Premium in 12.8. +> Single-level epics were [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) from GitLab Ultimate to GitLab Premium in 12.8. + +INFO: +Check out [multi-level child epics](manage_epics.md#multi-level-child-epics) with a +[free 30-day trial of GitLab Ultimate](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-epics-docs). When [issues](../../project/issues/index.md) share a theme across projects and milestones, you can manage them by using epics. @@ -37,9 +39,9 @@ graph TD Child_epic --> Issue2 ``` -## Roadmap in epics **(ULTIMATE)** +Also, read more about possible [planning hierarchies](../planning_hierarchy/index.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7327) in GitLab 11.10. +## Roadmap in epics **(ULTIMATE)** If your epic contains one or more [child epics](manage_epics.md#multi-level-child-epics) that have a start or due date, a visual diff --git a/doc/user/group/epics/manage_epics.md b/doc/user/group/epics/manage_epics.md index 72fa9e1e310..f5b1a2a6ee6 100644 --- a/doc/user/group/epics/manage_epics.md +++ b/doc/user/group/epics/manage_epics.md @@ -332,7 +332,7 @@ Only issues from projects that are in groups can be promoted. When you attempt t issue, a warning is displayed. Promoting a confidential issue to an epic makes all information related to the issue public as epics are public to group members. -When the quick action is executed: +When an issue is promoted to an epic: - An epic is created in the same group as the project of the issue. - Subscribers of the issue are notified that the epic was created. diff --git a/doc/user/group/index.md b/doc/user/group/index.md index f0e08301a1b..db6ed02f405 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -321,7 +321,7 @@ To share a group after enabling this feature: 1. Go to your group's page. 1. On the left sidebar, go to **Group information > Members**, and then select **Invite a group**. 1. Select a group, and select a **Max role**. -1. (Optional) Select an **Access expiration date**. +1. Optional. Select an **Access expiration date**. 1. Select **Invite**. ## Manage group memberships via LDAP **(PREMIUM SELF)** @@ -508,7 +508,7 @@ To prevent a project from being shared with other groups: This setting applies to all subgroups unless overridden by a group owner. Groups already added to a project lose access when the setting is enabled. -## Prevent members from being added to a group **(PREMIUM)** +## Prevent members from being added to projects in a group **(PREMIUM)** As a group owner, you can prevent any new project membership for all projects in a group, allowing tighter control over project membership. @@ -516,7 +516,11 @@ projects in a group, allowing tighter control over project membership. For example, if you want to lock the group for an [Audit Event](../../administration/audit_events.md), you can guarantee that project membership cannot be modified during the audit. -To prevent members from being added to a group: +You can still invite groups or to add members to groups, implicitly giving members access to projects in the **locked** group. + +The setting does not cascade. Projects in subgroups observe the subgroup configuration, ignoring the parent group. + +To prevent members from being added to projects in a group: 1. Go to the group's **Settings > General** page. 1. Expand the **Permissions, LFS, 2FA** section. @@ -557,12 +561,15 @@ You should consider these security implications before configuring IP address re - **Administrators and group owners**: Users with these permission levels can always access the group settings, regardless of IP restriction, but they cannot access projects belonging to the group when accessing from a disallowed IP address. -- **GitLab API and runner activities**: Only the [Groups](../../api/groups.md) - and [Projects](../../api/projects.md) APIs are protected by IP address restrictions. +- **GitLab API and runner activities**: Only the [group](../../api/groups.md) (including all + [group resources](../../api/api_resources.md#group-resources)) APIs and [project](../../api/api_resources.md#project-resources) + (including all [project resources](../../api/api_resources.md#project-resources)) APIs are protected by IP address restrictions. When you register a runner, it is not bound by the IP restrictions. When the runner requests a new job or an update to a job's state, it is also not bound by the IP restrictions. But when the running CI/CD job sends Git requests from a restricted IP address, the IP restriction prevents code from being cloned. +- **User dashboard activity**: Users may still see some events from the IP restricted groups and projects + on their dashboard. Activity may include push, merge, issue, or comment events. To restrict group access by IP address: @@ -660,18 +667,15 @@ To disable group mentions: 1. Select **Disable group mentions**. 1. Select **Save changes**. -## Enable delayed project removal **(PREMIUM)** +## Enable delayed project deletion **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220382) in GitLab 13.2. > - [Inheritance and enforcement added](https://gitlab.com/gitlab-org/gitlab/-/issues/321724) in GitLab 13.11. > - [Instance setting to enable by default added](https://gitlab.com/gitlab-org/gitlab/-/issues/255449) in GitLab 14.2. -Projects can be configured to be deleted either: - -- Immediately. -- After a delayed interval. During this interval period, the projects are in a read-only state - and can be restored. The default interval period is seven days but - [is configurable](../admin_area/settings/visibility_and_access_controls.md#default-deletion-delay). +[Delayed project deletion](../project/settings/index.md#delayed-project-deletion) can be enabled for groups. When enabled, projects in +the group are deleted after a period of delay. During this period, projects are in a read-only state and can be restored. The default +period is seven days but [is configurable at the instance level](../admin_area/settings/visibility_and_access_controls.md#default-deletion-delay). On self-managed GitLab, projects are deleted immediately by default. In GitLab 14.2 and later, an administrator can @@ -685,12 +689,12 @@ To enable delayed deletion of projects in a group: 1. Go to the group's **Settings > General** page. 1. Expand the **Permissions, LFS, 2FA** section. -1. Check **Enable delayed project removal**. +1. Check **Enable delayed project deletion**. 1. Optional. To prevent subgroups from changing this setting, select **Enforce for all subgroups**. 1. Select **Save changes**. NOTE: -In GitLab 13.11 and above the group setting for delayed project removal is inherited by subgroups. As discussed in [Cascading settings](../../development/cascading_settings.md) inheritance can be overridden, unless enforced by an ancestor. +In GitLab 13.11 and above the group setting for delayed project deletion is inherited by subgroups. As discussed in [Cascading settings](../../development/cascading_settings.md) inheritance can be overridden, unless enforced by an ancestor. ## Prevent project forking outside group **(PREMIUM)** @@ -799,13 +803,5 @@ the following checks when creating or updating namespaces or groups: - Namespaces must not have parents. - Group parents must be groups and not namespaces. -You can disable the validation if GitLab shows the following errors: - -- `A user namespace cannot have a parent`. -- `A group cannot have a user namespace as its parent`. - -To disable the validation, -[disable the `validate_namespace_parent_type` flag](../../administration/feature_flags.md). - -In the unlikely event that you had to disable this feature flag to prevent errors, +In the unlikely event that you see these errors in your GitLab installation, [contact Support](https://about.gitlab.com/support/) so that we can improve this validation. diff --git a/doc/user/group/iterations/index.md b/doc/user/group/iterations/index.md index 7a1cbe86d58..8a79effba15 100644 --- a/doc/user/group/iterations/index.md +++ b/doc/user/group/iterations/index.md @@ -7,14 +7,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Iterations **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214713) in GitLab 13.1. -> - Deployed behind a feature flag, disabled by default. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/221047) in GitLab 13.2. -> - Enabled on GitLab.com. -> - Can be enabled or disabled per-group. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-iterations). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214713) in GitLab 13.1 [with a flag](../../../administration/feature_flags.md) named `group_iterations`. Disabled by default. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/221047) in GitLab 13.2. > - Moved to GitLab Premium in 13.9. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/221047) in GitLab 14.6. [Feature flag `group_iterations`](https://gitlab.com/gitlab-org/gitlab/-/issues/221047) removed. Iterations are a way to track issues over a period of time. This allows teams to track velocity and volatility metrics. Iterations can be used with [milestones](../../project/milestones/index.md) @@ -169,31 +165,7 @@ To group issues by label: 1. Select the **Filter by label** dropdown. 1. Select the labels you want to group by in the labels dropdown. You can also search for labels by typing in the search input. -1. Select or tap outside of the label dropdown. The page is now grouped by the selected labels. - -## Enable or disable iterations **(PREMIUM SELF)** - -GitLab Iterations feature is deployed with a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can disable it for your instance. `:group_iterations` can be enabled or disabled per-group. - -To enable it: - -```ruby -# Instance-wide -Feature.enable(:group_iterations) -# or by group -Feature.enable(:group_iterations, Group.find(<group ID>)) -``` - -To disable it: - -```ruby -# Instance-wide -Feature.disable(:group_iterations) -# or by group -Feature.disable(:group_iterations, Group.find(<group ID>)) -``` +1. Select any area outside the label dropdown list. The page is now grouped by the selected labels. ### Enable or disable iteration cadences **(PREMIUM SELF)** diff --git a/doc/user/group/planning_hierarchy/img/epic-view-ancestors-in-sidebar_v14_6.png b/doc/user/group/planning_hierarchy/img/epic-view-ancestors-in-sidebar_v14_6.png Binary files differnew file mode 100644 index 00000000000..373b861239b --- /dev/null +++ b/doc/user/group/planning_hierarchy/img/epic-view-ancestors-in-sidebar_v14_6.png diff --git a/doc/user/group/planning_hierarchy/img/hierarchy_with_multi_level_epics.png b/doc/user/group/planning_hierarchy/img/hierarchy_with_multi_level_epics.png Binary files differnew file mode 100644 index 00000000000..d264ebf10d7 --- /dev/null +++ b/doc/user/group/planning_hierarchy/img/hierarchy_with_multi_level_epics.png diff --git a/doc/user/group/planning_hierarchy/img/issue-view-parent-epic-in-sidebar_v14_6.png b/doc/user/group/planning_hierarchy/img/issue-view-parent-epic-in-sidebar_v14_6.png Binary files differnew file mode 100644 index 00000000000..95a5777674a --- /dev/null +++ b/doc/user/group/planning_hierarchy/img/issue-view-parent-epic-in-sidebar_v14_6.png diff --git a/doc/user/group/planning_hierarchy/index.md b/doc/user/group/planning_hierarchy/index.md new file mode 100644 index 00000000000..5887328abe4 --- /dev/null +++ b/doc/user/group/planning_hierarchy/index.md @@ -0,0 +1,67 @@ +--- +type: reference +stage: Plan +group: Product Planning +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Planning hierarchies **(PREMIUM)** + +Planning hierarchies are an integral part of breaking down your work in GitLab. +To understand how you can use epics and issues together in hierarchies, remember the following: + +- [Epics](../epics/index.md) exist in groups. +- [Issues](../../project/issues/index.md) exist in projects. + +GitLab is not opinionated on how you structure your work and the hierarchy you can build with multi-level +epics. For example, you can use the hierarchy as a folder of issues for bigger initiatives. + +To learn about hierarchies in general, common frameworks, and using GitLab for +portfolio management, see +[How to use GitLab for Agile portfolio planning and project management](https://about.gitlab.com/blog/2020/11/11/gitlab-for-agile-portfolio-planning-project-management/). + +## Hierarchies with epics + +With epics, you can achieve the following hierarchy: + +```mermaid +graph TD + Group_epic --> Project1_Issue1 + Group_epic --> Project1_Issue2 + Group_epic --> Project2_Issue1 +``` + +### Hierarchies with multi-level epics **(ULTIMATE)** + +With the addition of [multi-level epics](../epics/manage_epics.md#multi-level-child-epics) and up to +seven levels of nested epics, you can achieve the following hierarchy: + +<!-- +Image below was generated with the following Mermaid code. +Attached as an image because a rendered diagram doesn't look clear on the docs page. + +```mermaid +classDiagram + direction TD + class Epic + class Issue + + Epic *-- "0..7" Epic +Epic "1"*-- "0..*" Issue +``` + + --> + + + +## View ancestry of an epic + +In an epic, you can view the ancestors as parents in the right sidebar under **Ancestors**. + + + +## View ancestry of an issue + +In an issue, you can view the parented epic above the issue in the right sidebar under **Epic**. + + diff --git a/doc/user/group/roadmap/index.md b/doc/user/group/roadmap/index.md index 206f3172170..7d489bc5b2d 100644 --- a/doc/user/group/roadmap/index.md +++ b/doc/user/group/roadmap/index.md @@ -19,7 +19,7 @@ Epics and milestones in a group containing a start date or due date can be visua of a timeline (that is, a Gantt chart). The Roadmap page shows the epics and milestones in a group, one of its subgroups, or a project in one of the groups. -On the epic bars, you can see the each epic's title, progress, and completed weight percentage. +On the epic bars, you can see each epic's title, progress, and completed weight percentage. When you hover over an epic bar, a popover appears with the epic's title, start date, due date, and weight completed. diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index 402007b85b2..7443be250bb 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -14,6 +14,10 @@ This page describes SAML for groups. For instance-wide SAML on self-managed GitL SAML on GitLab.com allows users to sign in through their SAML identity provider. If the user is not already a member, the sign-in process automatically adds the user to the appropriate group. +INFO: +Use your own SAML authentication to log in to [GitLab.com](http://gitlab.com/). +[Try GitLab Ultimate free for 30 days](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-saml-sso-docs). + User synchronization of SAML SSO groups is supported through [SCIM](scim_setup.md). SCIM supports adding and removing users from the GitLab group automatically. For example, if you remove a user from the SCIM app, SCIM removes that same user from the GitLab group. @@ -29,14 +33,16 @@ If required, you can find [a glossary of common terms](../../../integration/saml Alternatively GitLab provides [metadata XML configuration](#metadata-configuration). See [specific identity provider documentation](#providers) for more details. 1. Configure the SAML response to include a NameID that uniquely identifies each user. -1. Configure [required assertions](#assertions) at minimum containing - the user's email address. +1. Configure the required [user attributes](#user-attributes), ensuring you include the user's email address. 1. While the default is enabled for most SAML providers, please ensure the app is set to have service provider initiated calls in order to link existing GitLab accounts. 1. Once the identity provider is set up, move on to [configuring GitLab](#configuring-gitlab).  +If your account is the only owner in the group after SAML is set up, you can't unlink the account. To [unlink the account](#unlinking-accounts), +set up another user as a group owner. + ### NameID GitLab.com uses the SAML NameID to identify users. The NameID element: @@ -60,15 +66,16 @@ Once users have signed into GitLab using the SSO SAML setup, changing the `NameI We recommend setting the NameID format to `Persistent` unless using a field (such as email) that requires a different format. Most NameID formats can be used, except `Transient` due to the temporary nature of this format. -### Assertions +### User attributes + +To create users with the correct information for improved [user access and management](#user-access-and-management), +the user's details must be passed to GitLab as attributes in the SAML assertion. At a minimum, the user's email address +must be specified as an attribute named `email` or `mail`. -For users to be created with the right information with the improved [user access and management](#user-access-and-management), -the user details need to be passed to GitLab as SAML assertions. +GitLab.com supports the following attributes: -At a minimum, the user's email address *must* be specified as an assertion named `email` or `mail`. -See [the assertions list](../../../integration/saml.md#assertions) for other available claims. -In addition to the attributes in the linked assertions list, GitLab.com supports `username` -and `nickname` attributes. +- `username` or `nickname`. We recommend you configure only one of these. +- The [attributes also available](../../../integration/saml.md#assertions) to self-managed GitLab instances. ### Metadata configuration @@ -104,7 +111,7 @@ The certificate [fingerprint algorithm](../../../integration/saml.md#notes-on-co > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/211962) in GitLab 13.8 with allowing group owners to not go through SSO. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/9152) in GitLab 13.11 with enforcing open SSO session to use Git if this setting is switched on. -With this option enabled, users (except owners) must go through your group's GitLab single sign-on URL if they wish to access group resources through the UI. Users can't be manually added as members. +With this option enabled, users (except users with the Owner role) must access GitLab using your group GitLab single sign-on URL to access group resources. Users added manually as members can't access group resources. SSO enforcement does not affect sign in or access to any resources outside of the group. Users can view which groups and projects they are a member of without SSO sign in. @@ -119,7 +126,7 @@ SSO has the following effects when enabled: - For groups, users can't share a project in the group outside the top-level group, even if the project is forked. - For Git activity over SSH and HTTPS, users must have at least one active session signed-in through SSO before they can push to or - pull from a GitLab repository. + pull from a GitLab repository. - Credentials that are not tied to regular users (for example, access tokens and deploy keys) do not have the SSO check enforced. - Users must be signed-in through SSO before they can pull images using the [Dependency Proxy](../../packages/dependency_proxy/index.md). <!-- Add bullet for API activity when https://gitlab.com/gitlab-org/gitlab/-/issues/9152 is complete --> @@ -342,6 +349,11 @@ Ensure your SAML identity provider sends an attribute statement named `Groups` o </saml:AttributeStatement> ``` +Other attribute names such as `http://schemas.microsoft.com/ws/2008/06/identity/claims/groups` +are not accepted as a source of groups. +See the [SAML troubleshooting page](../../../administration/troubleshooting/group_saml_scim.md) +for examples on configuring the required attribute name in the SAML identity provider's settings. + NOTE: The value for `Groups` or `groups` in the SAML response can be either the group name or the group ID. To inspect the SAML response, you can use one of these [SAML debugging tools](#saml-debugging-tools). @@ -362,7 +374,7 @@ To link the SAML groups from the `saml:AttributeStatement` example above: If a user is a member of multiple SAML groups mapped to the same GitLab group, the user gets the highest access level from the groups. For example, if one group is linked as `Guest` and another `Maintainer`, a user in both groups gets `Maintainer` -access. +access. Users granted: @@ -374,7 +386,10 @@ Users granted: ### Automatic member removal After a group sync, users who are not members of a mapped SAML group are removed from -the GitLab group. +the GitLab group. Even if SSO authentication is successful, if an existing user is not a member of any of the configured groups: + +- They get an "unauthorized" message if they try to view the group. +- All of their permissions to subgroups and projects are also removed. For example, in the following diagram: @@ -475,6 +490,24 @@ Specific attention should be paid to: - The presence of a `X509Certificate`, which we require to verify the response signature. - The `SubjectConfirmation` and `Conditions`, which can cause errors if misconfigured. +#### Generate a SAML Response + +SAML Responses can be used to preview the attribute names and values sent in the assertions list while attempting to sign in using an IdP. + +To generate a SAML Response: + +1. Install either: + - [SAML Chrome Panel](https://chrome.google.com/webstore/detail/saml-chrome-panel/paijfdbeoenhembfhkhllainmocckace) for Chrome. + - [SAML-tracer](https://addons.mozilla.org/en-US/firefox/addon/saml-tracer/) for Firefox. +1. Open a new browser tab. +1. Open the SAML tracer console: + - Chrome: Right click on the page, select **Inspect**, then click on the SAML tab in the opened developer console. + - Firefox: Select the SAML-tracer icon located on the browser toolbar. +1. Go to the GitLab single sign-on URL for the group in the same browser tab with the SAML tracer open. +1. Select **Authorize** or attempt to log in. A SAML response is displayed in the tracer console that resembles this + [example SAML response](#example-saml-response). +1. Within the SAML tracer, select the **Export** icon to save the response in JSON format. + ### Verifying configuration For convenience, we've included some [example resources](../../../administration/troubleshooting/group_saml_scim.md) used by our Support Team. While they may help you verify the SAML app configuration, they are not guaranteed to reflect the current state of third-party products. diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index dd4558b4a3e..2651bcb9e12 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -115,12 +115,7 @@ configuration. Otherwise, the Okta SCIM app may not work properly. 1. Sign in to Okta. 1. Ensure you are in the Admin section by selecting the **Admin** button located in the top right. The admin button is not visible from the admin page. - - NOTE: - If you're using the Developer Console, select **Developer Console** in the top - bar and then select **Classic UI**. Otherwise, you may not see the buttons described in the following steps: - -1. In the **Application** tab, select **Add Application**. +1. In the **Application** tab, select **Browse App Catalog**. 1. Search for **GitLab**, find and select on the 'GitLab' application. 1. On the GitLab application overview page, select **Add**. 1. Under **Application Visibility** select both checkboxes. Currently the GitLab application does not support SAML authentication so the icon should not be shown to users. @@ -170,14 +165,11 @@ During provisioning: - Duplicate usernames are also handled, by adding suffix `1` upon user creation. For example, due to already existing `test_user` username, `test_user1` is used. -As long as [Group SAML](index.md) has been configured, existing GitLab.com users can link to their accounts in one of the following ways: - -- By updating their *primary* email address in their GitLab.com user account to match their identity provider's user profile email address. -- By following these steps: +If [Group SAML](index.md) has been configured and you have an existing GitLab.com account, you can link your SCIM and SAML identities: - 1. Sign in to GitLab.com if needed. - 1. In the identity provider's dashboard select the GitLab app or visit the **GitLab single sign-on URL**. - 1. Select the **Authorize**. +1. Update the [primary email](../../profile/index.md#change-your-primary-email) address in your GitLab.com user account to match the + user profile email address in your identity provider. +1. [Link your SAML identity](index.md#linking-saml-to-your-existing-gitlabcom-account). We recommend users do this prior to turning on sync, because while synchronization is active, there may be provisioning errors for existing users. @@ -303,3 +295,12 @@ As a workaround, try an alternate mapping: 1. Follow the Azure mapping instructions from above. 1. Delete the `name.formatted` target attribute entry. 1. Change the `displayName` source attribute to have `name.formatted` target attribute. + +#### Failed to match an entry in the source and target systems Group 'Group-Name' + +Group provisioning in Azure can fail with the `Failed to match an entry in the source and target systems Group 'Group-Name'` error message, +and the error response can include a HTML result of the GitLab URL `https://gitlab.com/users/sign_in`. + +This error is harmless and occurs because Group provisioning was turned on but GitLab SCIM integration does not support it nor require it. To +remove the error, follow the instructions in the Azure configuration guide to disable the option +[`Synchronize Azure Active Directory Groups to AppName`](#azure-configuration-steps). diff --git a/doc/user/group/settings/import_export.md b/doc/user/group/settings/import_export.md index 3692a2636ab..5745c499c5c 100644 --- a/doc/user/group/settings/import_export.md +++ b/doc/user/group/settings/import_export.md @@ -9,19 +9,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2888) in GitLab 13.0 as an experimental feature. May change in future releases. -Existing groups running on any GitLab instance or GitLab.com can be exported with all their related data and moved to a -new GitLab instance. +You can export groups, with all their related data, from one GitLab instance to another. +You can also [export projects](../../project/settings/import_export.md). -The **GitLab import/export** button is displayed if the group import option in enabled. +## Enable export for a group -See also: +Prerequisite: -- [Group Import/Export API](../../../api/group_import_export.md) -- [Project Import/Export](../../project/settings/import_export.md) -- [Project Import/Export API](../../../api/project_import_export.md) +- You must have the [Owner role](../../permissions.md) for the group. -Users with the [Owner role](../../permissions.md) for a group can enable -import and export for that group: +To enable import and export for a group: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. @@ -70,8 +67,11 @@ WARNING: This feature will be [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6 and replaced by [GitLab Migration](../import/). -Users with the [Owner role](../../permissions.md) for a group can export the -contents of that group: +Prerequisites: + +- You must have the [Owner role](../../permissions.md) for the group. + +To export the contents of a group: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Settings > General**. @@ -88,6 +88,8 @@ NOTE: The maximum import file size can be set by the Administrator, default is `0` (unlimited). As an administrator, you can modify the maximum import file size. To do so, use the `max_import_size` option in the [Application settings API](../../../api/settings.md#change-application-settings) or the [Admin UI](../../admin_area/settings/account_and_limit_settings.md). Default [modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50MB to 0 in GitLab 13.8. +You can also use the [group import/export API](../../../api/group_import_export.md). + ### Between CE and EE You can export groups from the [Community Edition to the Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/) and vice versa. @@ -96,6 +98,10 @@ The Enterprise Edition retains some group data that isn't part of the Community ## Importing the group +WARNING: +This feature will be [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) +in GitLab 14.8 and replaced by [GitLab Migration](../import/). + 1. Create a new group: - On the top bar, select **New** (**{plus}**) and then **New group**. - On an existing group's page, select the **New subgroup** button. diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index a0a13c71d95..b91e258b04a 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -86,8 +86,8 @@ the date filter behavior to filter the end event time of the currently selected The change makes it possible to get a much better picture about the completed items within the stage and helps uncover long-running items. -For example, an issue was created a year ago and the current stage was finished in the current month. -If you were to look at the metrics for the last three months, this issue would not be included in the calculation of +For example, an issue was created a year ago and the current stage was finished in the current month. +If you were to look at the metrics for the last three months, this issue would not be included in the calculation of the stage metrics. With the new date filter, this item would be included. DISCLAIMER: @@ -100,7 +100,7 @@ sole discretion of GitLab Inc. ## How metrics are measured -> DORA API-based deployment metrics for group-level Value Stream Analytics were +> DORA API-based deployment metrics for group-level Value Stream Analytics were > [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/337256) from GitLab Ultimate to GitLab Premium in 14.3. The "Time" metrics near the top of the page are measured as follows: diff --git a/doc/user/img/award_emoji_comment_awarded.png b/doc/user/img/award_emoji_comment_awarded.png Binary files differdeleted file mode 100644 index 111793ebf8a..00000000000 --- a/doc/user/img/award_emoji_comment_awarded.png +++ /dev/null diff --git a/doc/user/img/award_emoji_comment_picker.png b/doc/user/img/award_emoji_comment_picker.png Binary files differdeleted file mode 100644 index 07f90c898ed..00000000000 --- a/doc/user/img/award_emoji_comment_picker.png +++ /dev/null diff --git a/doc/user/img/award_emoji_select.png b/doc/user/img/award_emoji_select.png Binary files differdeleted file mode 100644 index 269282b94b0..00000000000 --- a/doc/user/img/award_emoji_select.png +++ /dev/null diff --git a/doc/user/img/award_emoji_select_v14_6.png b/doc/user/img/award_emoji_select_v14_6.png Binary files differnew file mode 100644 index 00000000000..c8185a1b4cb --- /dev/null +++ b/doc/user/img/award_emoji_select_v14_6.png diff --git a/doc/user/infrastructure/clusters/connect/index.md b/doc/user/infrastructure/clusters/connect/index.md index 21387998a17..9e57622875d 100644 --- a/doc/user/infrastructure/clusters/connect/index.md +++ b/doc/user/infrastructure/clusters/connect/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w The [certificate-based Kubernetes integration with GitLab](../index.md) was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) -in GitLab 14.5. To connect your clusters, use the [GitLab Kubernetes Agent](../../../clusters/agent/index.md). +in GitLab 14.5. To connect your clusters, use the [GitLab Agent](../../../clusters/agent/index.md). <!-- TBA: (We need to resolve https://gitlab.com/gitlab-org/gitlab/-/issues/343660 before adding this line) If you don't have a cluster yet, create one and connect it to GitLab through the Agent. diff --git a/doc/user/infrastructure/clusters/deploy/inventory_object.md b/doc/user/infrastructure/clusters/deploy/inventory_object.md index d5840641aab..47063dcae96 100644 --- a/doc/user/infrastructure/clusters/deploy/inventory_object.md +++ b/doc/user/infrastructure/clusters/deploy/inventory_object.md @@ -9,10 +9,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332227) in GitLab 14.0. An inventory object is a `ConfigMap` object for keeping track of the set of objects applied to a cluster. -When you remove objects from a manifest repository, GitLab Kubernetes Agent uses a corresponding inventory object to +When you remove objects from a manifest repository, the Agent uses a corresponding inventory object to prune (delete) objects from the cluster. -The GitLab Kubernetes Agent creates an inventory object for each manifest project specified in the +The Agent creates an inventory object for each manifest project specified in the `gitops.manifest_projects` configuration section. The inventory object has to be stored somewhere in the cluster. The default behavior is: @@ -20,10 +20,10 @@ The default behavior is: explicitly, the inventory object is stored in the `default` namespace. - The `name` is generated from the numeric project ID of the manifest project and the numeric agent ID. - This way the GitLab Kubernetes Agent constructs the name and local where the inventory object is + This way the Agent constructs the name and local where the inventory object is stored in the cluster. -The GitLab Kubernetes Agent cannot locate the existing inventory object if you: +The Agent cannot locate the existing inventory object if you: - Change `gitops.manifest_projects[].default_namespace` parameter. - Move manifests into another project. @@ -57,13 +57,13 @@ metadata: ## Using GitOps with pre-existing Kubernetes objects -The GitLab Kubernetes Agent treats manifest files in the manifest repository as the source of truth. When it applies +The Agent treats manifest files in the manifest repository as the source of truth. When it applies objects from the files to the cluster, it tracks them in an inventory object. If an object already exists, -GitLab Kubernetes Agent behaves differently based on the `gitops.manifest_projects[].inventory_policy` configuration. +The Agent behaves differently based on the `gitops.manifest_projects[].inventory_policy` configuration. Check the table below with the available options and when to use them. `inventory_policy` value | Description | ------------------------ | ------------------------------------------------------------------------------------------- | `must_match` | This is the default policy. A live object must have the `config.k8s.io/owning-inventory` annotation set to the same value as the `cli-utils.sigs.k8s.io/inventory-id` label on the corresponding inventory object to be updated. Object is not updated and an error is reported if the values don't match or the object doesn't have the annotation. | `adopt_if_no_inventory` | This mode allows to "adopt" an object if it doesn't have the `config.k8s.io/owning-inventory` annotation. Use this mode if you want to start managing existing objects using the GitOps feature. Once all objects have been "adopted", we recommend you to put the setting back into the default `must_match` mode to avoid any unexpected adoptions. | -`adopt_all` | This mode allows to "adopt" an object even if it has the `config.k8s.io/owning-inventory` annotation set to a different value. This mode can be useful if you want to migrate a set of objects from one agent to another one or from some other tool to the GitLab Kubernetes Agent. Once all objects have been "adopted", we recommend you to put the setting back into the default `must_match` mode to avoid any unexpected adoptions. | +`adopt_all` | This mode allows to "adopt" an object even if it has the `config.k8s.io/owning-inventory` annotation set to a different value. This mode can be useful if you want to migrate a set of objects from one agent to another one or from some other tool to the Agent. Once all objects have been "adopted", we recommend you to put the setting back into the default `must_match` mode to avoid any unexpected adoptions. | diff --git a/doc/user/infrastructure/clusters/index.md b/doc/user/infrastructure/clusters/index.md index 06a77912876..144a8cd2d31 100644 --- a/doc/user/infrastructure/clusters/index.md +++ b/doc/user/infrastructure/clusters/index.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Kubernetes clusters **(FREE)** -To connect clusters to GitLab, use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). +To connect clusters to GitLab, use the [GitLab Agent](../../clusters/agent/index.md). ## Certificate-based Kubernetes integration (DEPRECATED) @@ -24,7 +24,7 @@ It had the following issues: - Users were constantly reporting issues with features based on this model. For this reason, we started to build features based on a new model, the -[GitLab Kubernetes Agent](../../clusters/agent/index.md). +[GitLab Agent](../../clusters/agent/index.md). Maintaining both methods in parallel caused a lot of confusion and significantly increased the complexity to use, develop, maintain, and document them. For this reason, we decided to deprecate them to focus on the @@ -38,7 +38,7 @@ Follow this [epic](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) for updates. You can find technical information about why we moved away from cluster certificates into -the Kubernetes Agent model on the [Agent's blueprint documentation](../../../architecture/blueprints/gitlab_to_kubernetes_communication/index.md). +the GitLab Agent model on the [Agent's blueprint documentation](../../../architecture/blueprints/gitlab_to_kubernetes_communication/index.md). ## Deprecated features @@ -52,13 +52,12 @@ the Kubernetes Agent model on the [Agent's blueprint documentation](../../../arc - [Cluster integrations](../../clusters/integrations.md) - [Cluster cost management](../../clusters/cost_management.md) - [Cluster environments](../../clusters/environments.md) -- [Canary Deployments](../../project/canary_deployments.md) +- [Advanced traffic control with Canary Ingress](../../project/canary_deployments.md#advanced-traffic-control-with-canary-ingress-deprecated) - [Serverless](../../project/clusters/serverless/index.md) - [Deploy Boards](../../project/deploy_boards.md) - [Pod logs](../../project/clusters/kubernetes_pod_logs.md) - [Clusters health](manage/clusters_health.md) - [Crossplane integration](../../clusters/crossplane.md) -- [Auto Deploy](../../../topics/autodevops/stages.md#auto-deploy) - [Web terminals](../../../administration/integration/terminal.md) ### Cluster levels diff --git a/doc/user/infrastructure/iac/index.md b/doc/user/infrastructure/iac/index.md index 89df9c1d18f..15a680e2193 100644 --- a/doc/user/infrastructure/iac/index.md +++ b/doc/user/infrastructure/iac/index.md @@ -96,10 +96,17 @@ owned by GitLab, where everyone can contribute. The [documentation of the provider](https://registry.terraform.io/providers/gitlabhq/gitlab/latest/docs) is available as part of the official Terraform provider documentations. -## Create a new cluster through IaC +## Create a new cluster through IaC (DEPRECATED) Learn how to [create a new cluster on Google Kubernetes Engine (GKE)](../clusters/connect/new_gke_cluster.md). +NOTE: +The linked tutorial connects the cluster to GitLab through cluster certificates, +and this method was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) +in GitLab 14.5. You can still create a cluster through IaC and then connect it to GitLab +through the [Agent](../../clusters/agent/index.md), the default and fully supported +method to connect clusters to GitLab. + ## Troubleshooting ### `gitlab_group_share_group` resources not detected when subgroup state is refreshed diff --git a/doc/user/infrastructure/iac/mr_integration.md b/doc/user/infrastructure/iac/mr_integration.md index e92b2d919ae..ab59f8ad64b 100644 --- a/doc/user/infrastructure/iac/mr_integration.md +++ b/doc/user/infrastructure/iac/mr_integration.md @@ -10,7 +10,7 @@ Collaborating around Infrastructure as Code (IaC) changes requires both code cha ## Output Terraform Plan information into a merge request -Using the [GitLab Terraform Report artifact](../../../ci/yaml/index.md#artifactsreportsterraform), +Using the [GitLab Terraform Report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsterraform), you can expose details from `terraform plan` runs directly into a merge request widget, enabling you to see statistics about the resources that Terraform creates, modifies, or destroys. @@ -62,7 +62,7 @@ To manually configure a GitLab Terraform Report artifact: 1. Define a `script` that runs `terraform plan` and `terraform show`. These commands pipe the output and convert the relevant bits into a store variable `PLAN_JSON`. This JSON is used to create a - [GitLab Terraform Report artifact](../../../ci/yaml/index.md#artifactsreportsterraform). + [GitLab Terraform Report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsterraform). The Terraform report obtains a Terraform `tfplan.json` file. The collected Terraform plan report is uploaded to GitLab as an artifact, and is shown in merge requests. diff --git a/doc/user/infrastructure/iac/terraform_state.md b/doc/user/infrastructure/iac/terraform_state.md index 84d1edbe2f7..a45ef02622f 100644 --- a/doc/user/infrastructure/iac/terraform_state.md +++ b/doc/user/infrastructure/iac/terraform_state.md @@ -17,7 +17,7 @@ to securely store the state files in local storage (the default) or WARNING: Using local storage (the default) on clustered deployments of GitLab will result in a split state across nodes, making subsequent executions of Terraform inconsistent. -You are highly advised to use a remote storage in that case. +You are highly advised to use a remote storage resource in that case. The GitLab managed Terraform state backend can store your Terraform state easily and securely, and spares you from setting up additional remote resources like @@ -28,7 +28,7 @@ Amazon S3 or Google Cloud Storage. Its features include: - Locking and unlocking state. - Remote Terraform plan and apply execution. -A GitLab **administrator** must [setup the Terraform state storage configuration](../../../administration/terraform_state.md) +A GitLab **administrator** must [set up the Terraform state storage configuration](../../../administration/terraform_state.md) before using this feature. ## Permissions for using Terraform @@ -89,7 +89,7 @@ local machine, this is a simple way to get started: ``` If you already have a GitLab-managed Terraform state, you can use the `terraform init` command -with the prepopulated parameters values: +with the pre-populated parameters values: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Infrastructure > Terraform**. @@ -300,7 +300,7 @@ any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, -rerun this command to reinitialize your working directory. If you forget, other +re-run this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. ``` diff --git a/doc/user/infrastructure/index.md b/doc/user/infrastructure/index.md index 3bb518596cc..d8e75928675 100644 --- a/doc/user/infrastructure/index.md +++ b/doc/user/infrastructure/index.md @@ -30,11 +30,11 @@ Learn more about how GitLab can help you run [Infrastructure as Code](iac/index. ## Integrated Kubernetes management The GitLab integration with Kubernetes helps you to install, configure, manage, deploy, and troubleshoot -cluster applications. With the GitLab Kubernetes Agent, you can connect clusters behind a firewall, +cluster applications. With the GitLab Agent, you can connect clusters behind a firewall, have real-time access to API endpoints, perform pull-based or push-based deployments for production and non-production environments, and much more. -Learn more about the [GitLab Kubernetes Agent](../clusters/agent/index.md). +Learn more about the [GitLab Agent](../clusters/agent/index.md). ## Runbooks in GitLab diff --git a/doc/user/infrastructure/mr_integration.md b/doc/user/infrastructure/mr_integration.md deleted file mode 100644 index 81e8f7cbd33..00000000000 --- a/doc/user/infrastructure/mr_integration.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'iac/mr_integration.md' -remove_date: '2021-11-26' ---- - -This document was moved to [another location](iac/mr_integration.md). - -<!-- This redirect file can be deleted after <2021-11-26>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/infrastructure/terraform_state.md b/doc/user/infrastructure/terraform_state.md deleted file mode 100644 index e71291d502e..00000000000 --- a/doc/user/infrastructure/terraform_state.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'iac/terraform_state.md' -remove_date: '2021-11-26' ---- - -This document was moved to [another location](iac/terraform_state.md). - -<!-- This redirect file can be deleted after <2021-11-26>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/instance/clusters/index.md b/doc/user/instance/clusters/index.md index 4bbd82d01a8..a184f00f6f6 100644 --- a/doc/user/instance/clusters/index.md +++ b/doc/user/instance/clusters/index.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. To connect clusters to GitLab, -use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). +use the [GitLab Agent](../../clusters/agent/index.md). Similar to [project-level](../../project/clusters/index.md) and [group-level](../../group/clusters/index.md) Kubernetes clusters, diff --git a/doc/user/markdown.md b/doc/user/markdown.md index d20e9c7a30e..1b3cd5d4478 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -11,7 +11,7 @@ GitLab automatically renders Markdown content. For example, when you add a comme you type the text in the Markdown language. When you save the issue, the text is rendered with a set of styles. These styles are described on this page. -For example, in Markdown, an ordered list looks like this: +For example, in Markdown, an unordered list looks like this: ```markdown - Cat @@ -249,7 +249,7 @@ the content. This data can be used by static site generators like [Jekyll](https When you view a Markdown file rendered by GitLab, front matter is displayed as-is, in a box at the top of the document. The HTML content displays after the front matter. To view an example, you can toggle between the source and rendered version of a -[GitLab documentation file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/README.md). +[GitLab documentation file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/index.md). In GitLab, front matter is used only in Markdown files and wiki pages, not the other places where Markdown formatting is supported. It must be at the very top of the document @@ -516,30 +516,30 @@ version to reference other projects from the same namespace. GitLab Flavored Markdown recognizes the following: -| references | input | cross-project reference | shortcut inside same namespace | -| :------------------------------ | :------------------------- | :-------------------------------------- | :----------------------------- | -| specific user | `@user_name` | | | -| specific group | `@group_name` | | | -| entire team | `@all` | | | -| project | `namespace/project>` | | | -| issue | ``#123`` | `namespace/project#123` | `project#123` | -| merge request | `!123` | `namespace/project!123` | `project!123` | -| snippet | `$123` | `namespace/project$123` | `project$123` | -| epic **(ULTIMATE)** | `&123` | `group1/subgroup&123` | | -| vulnerability **(ULTIMATE)** (1)| `[vulnerability:123]` | `[vulnerability:namespace/project/123]` | `[vulnerability:project/123]` | -| feature flag | `[feature_flag:123]` | `[feature_flag:namespace/project/123]` | `[feature_flag:project/123]` | -| label by ID | `~123` | `namespace/project~123` | `project~123` | -| one-word label by name | `~bug` | `namespace/project~bug` | `project~bug` | -| multi-word label by name | `~"feature request"` | `namespace/project~"feature request"` | `project~"feature request"` | -| scoped label by name | `~"priority::high"` | `namespace/project~"priority::high"` | `project~"priority::high"` | -| project milestone by ID | `%123` | `namespace/project%123` | `project%123` | -| one-word milestone by name | `%v1.23` | `namespace/project%v1.23` | `project%v1.23` | -| multi-word milestone by name | `%"release candidate"` | `namespace/project%"release candidate"` | `project%"release candidate"` | -| specific commit | `9ba12248` | `namespace/project@9ba12248` | `project@9ba12248` | -| commit range comparison | `9ba12248...b19a04f5` | `namespace/project@9ba12248...b19a04f5` | `project@9ba12248...b19a04f5` | -| repository file references | `[README](doc/README.md)` | | | -| repository file line references | `[README](doc/README.md#L13)` | | | -| [alert](../operations/incident_management/alerts.md) | `^alert#123` | `namespace/project^alert#123` | `project^alert#123` | +| references | input | cross-project reference | shortcut inside same namespace | +| :--------------------------------------------------- | :---------------------------- | :----------------------------------------- | :------------------------------- | +| specific user | `@user_name` | | | +| specific group | `@group_name` | | | +| entire team | `@all` | | | +| project | `namespace/project>` | | | +| issue | ``#123`` | `namespace/project#123` | `project#123` | +| merge request | `!123` | `namespace/project!123` | `project!123` | +| snippet | `$123` | `namespace/project$123` | `project$123` | +| [epic](group/epics/index.md) | `&123` | `group1/subgroup&123` | | +| vulnerability **(ULTIMATE)** <sup>1</sup> | `[vulnerability:123]` | `[vulnerability:namespace/project/123]` | `[vulnerability:project/123]` | +| feature flag | `[feature_flag:123]` | `[feature_flag:namespace/project/123]` | `[feature_flag:project/123]` | +| label by ID | `~123` | `namespace/project~123` | `project~123` | +| one-word label by name | `~bug` | `namespace/project~bug` | `project~bug` | +| multi-word label by name | `~"feature request"` | `namespace/project~"feature request"` | `project~"feature request"` | +| scoped label by name | `~"priority::high"` | `namespace/project~"priority::high"` | `project~"priority::high"` | +| project milestone by ID | `%123` | `namespace/project%123` | `project%123` | +| one-word milestone by name | `%v1.23` | `namespace/project%v1.23` | `project%v1.23` | +| multi-word milestone by name | `%"release candidate"` | `namespace/project%"release candidate"` | `project%"release candidate"` | +| specific commit | `9ba12248` | `namespace/project@9ba12248` | `project@9ba12248` | +| commit range comparison | `9ba12248...b19a04f5` | `namespace/project@9ba12248...b19a04f5` | `project@9ba12248...b19a04f5` | +| repository file references | `[README](doc/README.md)` | | | +| repository file line references | `[README](doc/README.md#L13)` | | | +| [alert](../operations/incident_management/alerts.md) | `^alert#123` | `namespace/project^alert#123` | `project^alert#123` | 1. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222483) in GitLab 13.7. @@ -554,6 +554,16 @@ In addition to this, links to some objects are also recognized and formatted. So - The issues designs tab: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234/designs"`, which are rendered as `#1234 (designs)`. - Links to individual designs: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234/designs/layout.png"`, which are rendered as `#1234[layout.png]`. +### Show the issue, merge request, or epic title in the reference + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15694) in GitLab 14.6. + +To include the title in the rendered link of an issue, merge request, or epic, add a plus (`+`) +at the end of the reference. For example, a reference like `#123+` is rendered as +`The issue title (#123)`. + +URL references like `https://gitlab.com/gitlab-org/gitlab/-/issues/1234+` are also expanded. + ### Embedding metrics in GitLab Flavored Markdown Metric charts can be embedded in GitLab Flavored Markdown. Read @@ -988,6 +998,8 @@ Here's a sample audio clip: ### Inline HTML +> Allowing `rel="license"` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20857) in GitLab 14.6. + To see the second example of Markdown rendered in HTML, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#inline-html). @@ -996,6 +1008,7 @@ You can also use raw HTML in your Markdown, and it usually works pretty well. See the documentation for HTML::Pipeline's [SanitizationFilter](https://github.com/jch/html-pipeline/blob/v2.12.3/lib/html/pipeline/sanitization_filter.rb#L42) class for the list of allowed HTML tags and attributes. In addition to the default `SanitizationFilter` allowlist, GitLab allows `span`, `abbr`, `details` and `summary` elements. +`rel="license"` is allowed on links to support the [Rel-License microformat](https://microformats.org/wiki/rel-license) and license attribution. ```html <dl> diff --git a/doc/user/operations_dashboard/index.md b/doc/user/operations_dashboard/index.md index 29ca3a48e70..47c41e85345 100644 --- a/doc/user/operations_dashboard/index.md +++ b/doc/user/operations_dashboard/index.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Operations Dashboard **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5781) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/9218) to [GitLab Premium](https://about.gitlab.com/pricing/) in 11.10. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5781) in GitLab 11.5. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/9218) from GitLab Ultimate to GitLab Premium in 11.10. The Operations Dashboard provides a summary of each project's operational health, including pipeline and alert status. diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index 7861258e23f..23bd140d4b7 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -9,6 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15886) in GitLab 13.2. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. > - Support for Composer 2.0 [added](https://gitlab.com/gitlab-org/gitlab/-/issues/259840) in GitLab 13.10. +> - Deploy token support [added](https://gitlab.com/gitlab-org/gitlab/-/issues/240897) in GitLab 14.6. WARNING: The Composer package registry for GitLab is under development and isn't ready for production use due to @@ -88,13 +89,12 @@ Prerequisites: - A valid `composer.json` file. - The Packages feature is enabled in a GitLab repository. - The project ID, which is on the project's home page. -- A [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api`. +- One of the following token types: + - A [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api`. + - A [deploy token](../../project/deploy_tokens/index.md) + with the scope set to `write_package_registry`. - NOTE: - [Deploy tokens](../../project/deploy_tokens/index.md) are - [not yet supported](https://gitlab.com/gitlab-org/gitlab/-/issues/240897) for use with Composer. - -To publish the package: +To publish the package with a personal access token: - Send a `POST` request to the [Packages API](../../../api/packages.md). @@ -109,6 +109,21 @@ To publish the package: - `<tag>` is the Git tag name of the version you want to publish. To publish a branch, use `branch=<branch>` instead of `tag=<tag>`. +To publish the package with a deploy token: + +- Send a `POST` request to the [Packages API](../../../api/packages.md). + + For example, you can use `curl`: + + ```shell + curl --data tag=<tag> --header "Deploy-Token: <deploy-token>" "https://gitlab.example.com/api/v4/projects/<project_id>/packages/composer" + ``` + + - `<deploy-token>` is your deploy token + - `<project_id>` is your project ID. + - `<tag>` is the Git tag name of the version you want to publish. + To publish a branch, use `branch=<branch>` instead of `tag=<tag>`. + You can view the published package by going to **Packages & Registries > Package Registry** and selecting the **Composer** tab. @@ -159,11 +174,11 @@ Prerequisites: - A package in the Package Registry. - The group ID, which is on the group's home page. -- A [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to, at minimum, `read_api`. - - NOTE: - [Deploy tokens](../../project/deploy_tokens/index.md) are - [not yet supported](https://gitlab.com/gitlab-org/gitlab/-/issues/240897) for use with Composer. +- One of the following token types: + - A [personal access token](../../../user/profile/personal_access_tokens.md) + with the scope set to, at minimum, `api`. + - A [deploy token](../../project/deploy_tokens/index.md) + with the scope set to `read_package_registry`, `write_package_registry`, or both. To install a package: @@ -213,6 +228,8 @@ To install a package: 1. Create an `auth.json` file with your GitLab credentials: + Using a personal access token: + ```shell composer config gitlab-token.<DOMAIN-NAME> <personal_access_token> ``` @@ -229,6 +246,26 @@ To install a package: } ``` + Using a deploy token: + + ```shell + composer config gitlab-token.<DOMAIN-NAME> <deploy_token_username> <deploy_token> + ``` + + Result in the `auth.json` file: + + ```json + { + ... + "gitlab-token": { + "<DOMAIN-NAME>": { + "username": "<deploy_token_username>", + "token": "<deploy_token>", + ... + } + } + ``` + You can unset this with the command: ```shell @@ -236,7 +273,8 @@ To install a package: ``` - `<DOMAIN-NAME>` is the GitLab instance URL `gitlab.com` or `gitlab.example.com`. - - `<personal_access_token>` with the scope set to `read_api`. + - `<personal_access_token>` with the scope set to `api`, or `<deploy_token>` with the scope set + to `read_package_registry` and/or `write_package_registry`. 1. If you are on a GitLab self-managed instance, add `gitlab-domains` to `composer.json`. @@ -298,10 +336,19 @@ To install a package: WARNING: Never commit the `auth.json` file to your repository. To install packages from a CI/CD job, -consider using the [`composer config`](https://getcomposer.org/doc/articles/handling-private-packages.md#satis) tool with your personal access token +consider using the [`composer config`](https://getcomposer.org/doc/articles/handling-private-packages.md#satis) tool with your access token stored in a [GitLab CI/CD variable](../../../ci/variables/index.md) or in [HashiCorp Vault](../../../ci/secrets/index.md). +### Working with Deploy Tokens + +Although Composer packages are accessed at the group level, a group or project deploy token can be +used to access them: + +- A group deploy token has access to all packages published to projects in that group or its + subgroups. +- A project deploy token only has access to packages published to that particular project. + ## Supported CLI commands The GitLab Composer repository supports the following Composer CLI commands: diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index 0f32f68d250..731ba04a9f7 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -103,6 +103,29 @@ A package with the recipe `Hello/0.1@mycompany/beta` is created. For more details about creating and managing Conan packages, see the [Conan documentation](https://docs.conan.io/en/latest/creating_packages.html). +#### Package without a username and a channel + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345055) in GitLab 14.6. + +Even though they are [recommended](https://docs.conan.io/en/latest/reference/conanfile/attributes.html#user-channel) +to distinguish your package from a similarly named existing package, +the username and channel are not mandatory fields for a Conan package. + +You can create a package without a username and channel by removing them from +the `create` command: + +```shell +conan create . +``` + +The username _and_ the channel must be blank. If only one of these fields is +blank, the request is rejected. + +NOTE: +Empty usernames and channels can only be used if you use a [project remote](#add-a-remote-for-your-project). +If you use an [instance remote](#add-a-remote-for-your-instance), the username +and the channel must be set. + ## Add the Package Registry as a Conan remote To run `conan` commands, you must add the Package Registry as a Conan remote for diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index c9cdc8643f4..9497dd1625b 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -370,10 +370,17 @@ WARNING: Deleting images is a destructive action and can't be undone. To restore a deleted image, you must rebuild and re-upload it. -NOTE: -Administrators should review how to -[garbage collect](../../../administration/packages/container_registry.md#container-registry-garbage-collection) -the deleted images. +On self-managed instances, deleting an image doesn't free up storage space - it only marks the image +as eligible for deletion. To actually delete images and recover storage space, in case they're +unreferenced, administrators must run [garbage collection](../../../administration/packages/container_registry.md#container-registry-garbage-collection). + +On GitLab.com, the latest version of the Container Registry includes an automatic online garbage +collector. For more information, see [this blog post](https://about.gitlab.com/blog/2021/10/25/gitlab-com-container-registry-update/). +This is an instance-wide feature, rolling out gradually to a subset of the user base, so some new image repositories created +from GitLab 14.5 onwards are served by this new version of the Container Registry. In this new +version of the Container Registry, layers that aren't referenced by any image manifest, and image +manifests that have no tags and aren't referenced by another manifest (such as multi-architecture +images), are automatically scheduled for deletion after 24 hours if left unreferenced. ### Delete images from within GitLab diff --git a/doc/user/packages/dependency_proxy/index.md b/doc/user/packages/dependency_proxy/index.md index fbd1cb84580..8b34634318c 100644 --- a/doc/user/packages/dependency_proxy/index.md +++ b/doc/user/packages/dependency_proxy/index.md @@ -89,6 +89,10 @@ You can authenticate using: - A [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `read_registry` and `write_registry`. - A [group deploy token](../../../user/project/deploy_tokens/index.md#group-deploy-token) with the scope set to `read_registry` and `write_registry`. +Users accessing the Dependency Proxy with a personal access token or username and password require +at least [Guest membership](../../permissions.md#group-members-permissions) +to the group they pull images from. + #### SAML SSO When [SSO enforcement](../../group/saml_sso/index.md#sso-enforcement) @@ -200,7 +204,7 @@ on the GitLab server. The next time you pull the same image, GitLab gets the lat information about the image from Docker Hub, but serves the existing blobs from the GitLab server. -## Clear the Dependency Proxy cache +## Reduce storage usage Blobs are kept forever on the GitLab server, and there is no hard limit on how much data can be stored. @@ -215,6 +219,16 @@ If you clear the cache, the next time a pipeline runs it must pull an image or t ### Cleanup policies +#### Enable cleanup policies from within GitLab + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340777) in GitLab 14.6 + +You can enable an automatic time-to-live (TTL) policy for the Dependency Proxy from the user +interface. To do this, navigate to your group's **Settings > Packages & Registries > Dependency Proxy** +and enable the setting to automatically clear items from the cache after 90 days. + +#### Enable cleanup policies with GraphQL + > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294187) in GitLab 14.4. The cleanup policy is a scheduled job you can use to clear cached images that are no longer used, @@ -245,8 +259,7 @@ mutation { ``` See the [Getting started with GraphQL](../../../api/graphql/getting_started.md) -guide to learn how to make GraphQL queries. Support for enabling and configuring cleanup policies in -the UI is tracked in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/340777). +guide to learn how to make GraphQL queries. When the policy is initially enabled, the default TTL setting is 90 days. Once enabled, stale dependency proxy files are queued for deletion each day. Deletion may not occur right away due to diff --git a/doc/user/packages/go_proxy/index.md b/doc/user/packages/go_proxy/index.md index 1cf3132489a..29455fdbb35 100644 --- a/doc/user/packages/go_proxy/index.md +++ b/doc/user/packages/go_proxy/index.md @@ -144,8 +144,8 @@ If you're unfamiliar with managing dependencies in Go, or Go in general, review the following documentation: - [Dependency Management in Go](../../../development/go_guide/dependencies.md) -- [Go Modules Reference](https://golang.org/ref/mod) -- [Documentation (`golang.org`)](https://golang.org/doc/) +- [Go Modules Reference](https://go.dev/ref/mod) +- [Documentation (`golang.org`)](https://go.dev/doc/) - [Learn (`go.dev/learn`)](https://go.dev/learn/) ### Set environment variables diff --git a/doc/user/packages/helm_repository/index.md b/doc/user/packages/helm_repository/index.md index c88fba83dc7..488345965f9 100644 --- a/doc/user/packages/helm_repository/index.md +++ b/doc/user/packages/helm_repository/index.md @@ -36,15 +36,11 @@ To authenticate to the Helm repository, you need either: ## Publish a package -WARNING: -The `helm-push` command is broken in Helm 3.7. For more information, see the [open issue](https://github.com/chartmuseum/helm-push/issues/109) -in the Chart Museum project. - NOTE: You can publish Helm charts with duplicate names or versions. If duplicates exist, GitLab always returns the chart with the latest version. -Once built, a chart can be uploaded to the desired channel with `curl` or `helm-push`: +Once built, a chart can be uploaded to the desired channel with `curl` or `helm cm-push`: - With `curl`: @@ -61,11 +57,11 @@ Once built, a chart can be uploaded to the desired channel with `curl` or `helm- [URL-encoded](../../../api/index.md#namespaced-path-encoding) path of the project (like `group%2Fproject`). - `<channel>`: the name of the channel (like `stable`). -- With the [`helm-push`](https://github.com/chartmuseum/helm-push/#readme) plugin: +- With the [`helm cm-push`](https://github.com/chartmuseum/helm-push/#readme) plugin: ```shell helm repo add --username <username> --password <access_token> project-1 https://gitlab.example.com/api/v4/projects/<project_id>/packages/helm/<channel> - helm push mychart-0.1.0.tgz project-1 + helm cm-push mychart-0.1.0.tgz project-1 ``` - `<username>`: the GitLab username or the deploy token username. @@ -135,12 +131,6 @@ To fix the error, use the correct version syntax and upload the chart again. ### `helm push` results in an error -The `helm push` plugin is not yet supported in Helm 3.7. If you try to push a chart using -`helm push`, it produces the following error: - -```plaintext -Error: this feature has been marked as experimental and is not enabled by default. Please set HELM_EXPERIMENTAL_OCI=1 in your environment to use this feature -``` - -To continue to use the plugin, you can push an image using [curl](#use-cicd-to-publish-a-helm-package) -or downgrade your version of Helm. +Helm 3.7 introduced a breaking change for the `helm-push` plugin. You can update the +[Chart Museum plugin](https://github.com/chartmuseum/helm-push/#readme) +to use `helm cm-push`. diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index 03209da7ac8..1086de1fa92 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -17,6 +17,10 @@ Only [scoped](https://docs.npmjs.com/misc/scope/) packages are supported. For documentation of the specific API endpoints that the npm package manager client uses, see the [npm API documentation](../../../api/packages/npm.md). +WARNING: +Never hardcode GitLab tokens (or any tokens) directly in `.npmrc` files or any other files that can +be committed to a repository. + ## Build an npm package This section covers how to install npm or Yarn and build a package for your @@ -430,14 +434,16 @@ You can route package requests to organizations and users outside of GitLab. To do this, add lines to your `.npmrc` file. Replace `my-org` with the namespace or group that owns your project's repository, and use your organization's URL. The name is case-sensitive and must match the name of your group or namespace exactly. +Use environment variables to set up your tokens: `export MY_TOKEN="<your token>"`. + ```shell @foo:registry=https://gitlab.example.com/api/v4/packages/npm/ -//gitlab.example.com/api/v4/packages/npm/:_authToken= "<your_token>" -//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken= "<your_token>" +//gitlab.example.com/api/v4/packages/npm/:_authToken=${MY_TOKEN} +//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=${MY_TOKEN} @my-other-org:registry=https://gitlab.example.com/api/v4/packages/npm/ -//gitlab.example.com/api/v4/packages/npm/:_authToken= "<your_token>" -//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken= "<your_token>" +//gitlab.example.com/api/v4/packages/npm/:_authToken=${MY_TOKEN} +//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=${MY_TOKEN} ``` ### npm metadata diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index 98cccd72425..37b6404d487 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -107,6 +107,7 @@ You can now add a new source to NuGet with: - [NuGet CLI](#add-a-source-with-the-nuget-cli) - [Visual Studio](#add-a-source-with-visual-studio) - [.NET CLI](#add-a-source-with-the-net-cli) +- [Configuration file](#add-a-source-with-a-configuration-file) ### Add a source with the NuGet CLI @@ -215,6 +216,51 @@ If you get a warning, ensure that the **Location**, **Username**, and A project-level endpoint is required to publish NuGet packages to the Package Registry. A project-level endpoint is also required to install NuGet packages from a project. +To use the [project-level](#use-the-gitlab-endpoint-for-nuget-packages) +NuGet endpoint, add the Package Registry as a source with `nuget`: + +```shell +dotnet nuget add source "https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/nuget/index.json" --name <source_name> --username <gitlab_username or deploy_token_username> --password <gitlab_personal_access_token or deploy_token> +``` + +- `<source_name>` is the desired source name. +- `--store-password-in-clear-text` might be necessary depending on your operating system. + +For example: + +```shell +dotnet nuget add source "https://gitlab.example.com/api/v4/projects/10/packages/nuget/index.json" --name gitlab --username carol --password 12345678asdf +``` + +#### Group-level endpoint + +To install a NuGet package from a group, use a group-level endpoint. + +To use the [group-level](#use-the-gitlab-endpoint-for-nuget-packages) +NuGet endpoint, add the Package Registry as a source with `nuget`: + +```shell +dotnet nuget add source "https://gitlab.example.com/api/v4/groups/<your_group_id>/-/packages/nuget/index.json" --name <source_name> --username <gitlab_username or deploy_token_username> --password <gitlab_personal_access_token or deploy_token> +``` + +- `<source_name>` is the desired source name. +- `--store-password-in-clear-text` might be necessary depending on your operating system. + +For example: + +```shell +dotnet nuget add source "https://gitlab.example.com/api/v4/groups/23/-/packages/nuget/index.json" --name gitlab --username carol --password 12345678asdf +``` + +### Add a source with a configuration file + +#### Project-level endpoint + +A project-level endpoint is required to: + +- Publish NuGet packages to the Package Registry. +- Install NuGet packages from a project. + To use the [project-level](#use-the-gitlab-endpoint-for-nuget-packages) Package Registry as a source for .NET: 1. In the root of your project, create a file named `nuget.config`. @@ -229,13 +275,20 @@ To use the [project-level](#use-the-gitlab-endpoint-for-nuget-packages) Package </packageSources> <packageSourceCredentials> <gitlab> - <add key="Username" value="<gitlab_username or deploy_token_username>" /> - <add key="ClearTextPassword" value="<gitlab_personal_access_token or deploy_token>" /> + <add key="Username" value="%GITLAB_PACKAGE_REGISTRY_USERNAME%" /> + <add key="ClearTextPassword" value="%GITLAB_PACKAGE_REGISTRY_PASSWORD%" /> </gitlab> </packageSourceCredentials> </configuration> ``` +1. Configure the necessary environment variables: + + ```shell + export GITLAB_PACKAGE_REGISTRY_USERNAME=<gitlab_username or deploy_token_username> + export GITLAB_PACKAGE_REGISTRY_PASSWORD=<gitlab_personal_access_token or deploy_token> + ``` + #### Group-level endpoint To install a package from a group, use a group-level endpoint. @@ -254,13 +307,20 @@ To use the [group-level](#use-the-gitlab-endpoint-for-nuget-packages) Package Re </packageSources> <packageSourceCredentials> <gitlab> - <add key="Username" value="<gitlab_username or deploy_token_username>" /> - <add key="ClearTextPassword" value="<gitlab_personal_access_token or deploy_token>" /> + <add key="Username" value="%GITLAB_PACKAGE_REGISTRY_USERNAME%" /> + <add key="ClearTextPassword" value="%GITLAB_PACKAGE_REGISTRY_PASSWORD%" /> </gitlab> </packageSourceCredentials> </configuration> ``` +1. Configure the necessary environment variables: + + ```shell + export GITLAB_PACKAGE_REGISTRY_USERNAME=<gitlab_username or deploy_token_username> + export GITLAB_PACKAGE_REGISTRY_PASSWORD=<gitlab_personal_access_token or deploy_token> + ``` + ## Publish a NuGet package Prerequisite: diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index 3204ac07d6a..28e1571b4f8 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -147,7 +147,7 @@ The Package Registry supports the following formats: | [Go](../go_proxy/index.md) | 13.1+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3043) | | [Ruby gems](../rubygems_registry/index.md) | 13.10+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3200) | -[Status](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga): +[Status](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga): - Alpha: behind a feature flag and not officially supported. - Beta: several known issues that may prevent expected use. diff --git a/doc/user/packages/terraform_module_registry/index.md b/doc/user/packages/terraform_module_registry/index.md index 7f101adccad..b8dc071fc30 100644 --- a/doc/user/packages/terraform_module_registry/index.md +++ b/doc/user/packages/terraform_module_registry/index.md @@ -29,15 +29,15 @@ Prerequisites: - You need to [authenticate with the API](../../../api/index.md#authentication). If authenticating with a deploy token, it must be configured with the `write_package_registry` scope. ```plaintext -PUT /projects/:id/packages/terraform/modules/:module_name/:module_system/:module_version/file +PUT /projects/:id/packages/terraform/modules/:module-name/:module-system/:module-version/file ``` | Attribute | Type | Required | Description | | -------------------| --------------- | ---------| -------------------------------------------------------------------------------------------------------------------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../../api/index.md#namespaced-path-encoding). | -| `module_name` | string | yes | The package name. It can contain only lowercase letters (`a-z`), uppercase letter (`A-Z`), numbers (`0-9`), or hyphens (`-`) and cannot exceed 64 characters. -| `module_system` | string | yes | The package system. It can contain only lowercase letters (`a-z`) and numbers (`0-9`), and cannot exceed 64 characters. -| `module_version` | string | yes | The package version. It must be valid according to the [Semantic Versioning Specification](https://semver.org/). +| `module-name` | string | yes | The package name. It can contain only lowercase letters (`a-z`), uppercase letter (`A-Z`), numbers (`0-9`), or hyphens (`-`) and cannot exceed 64 characters. +| `module-system` | string | yes | The package system. It can contain only lowercase letters (`a-z`) and numbers (`0-9`), and cannot exceed 64 characters. +| `module-version` | string | yes | The package version. It must be valid according to the [Semantic Versioning Specification](https://semver.org/). Provide the file content in the request body. @@ -97,7 +97,7 @@ You can then reference your Terraform Module from a downstream Terraform project ```plaintext module "<module>" { - source = "gitlab.com/<namespace>/<module_name>/<module_system>" + source = "gitlab.com/<namespace>/<module-name>/<module-system>" } ``` diff --git a/doc/user/permissions.md b/doc/user/permissions.md index eb79d5099eb..4336c58b56c 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -103,6 +103,7 @@ The following table lists project permissions available for each role: | [Issues](project/issues/index.md):<br>View related issues | ✓ | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Set weight | ✓ (*16*) | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>View [confidential issues](project/issues/confidential_issues.md) | (*2*) | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Close / reopen | | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Lock threads | | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Manage related issues | | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Manage tracker | | ✓ | ✓ | ✓ | ✓ | @@ -119,7 +120,7 @@ The following table lists project permissions available for each role: | [Merge requests](project/merge_requests/index.md):<br>Apply code change suggestions | | | ✓ | ✓ | ✓ | | [Merge requests](project/merge_requests/index.md):<br>Approve (*9*) | | | ✓ | ✓ | ✓ | | [Merge requests](project/merge_requests/index.md):<br>Assign | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Create | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Create (*18*) | | | ✓ | ✓ | ✓ | | [Merge requests](project/merge_requests/index.md):<br>Add labels | | | ✓ | ✓ | ✓ | | [Merge requests](project/merge_requests/index.md):<br>Lock threads | | | ✓ | ✓ | ✓ | | [Merge requests](project/merge_requests/index.md):<br>Manage or accept | | | ✓ | ✓ | ✓ | @@ -233,6 +234,7 @@ The following table lists project permissions available for each role: 1. Guest users can only set metadata (for example, labels, assignees, or milestones) when creating an issue. They cannot change the metadata on existing issues. 1. In GitLab 14.5 or later, Guests are not allowed to [create incidents](../operations/incident_management/incidents.md#incident-creation). +1. In projects that accept contributions from external members, users can create, edit, and close their own merge requests. ## Project features permissions diff --git a/doc/user/profile/account/delete_account.md b/doc/user/profile/account/delete_account.md index c555f5ca8cc..96415279de4 100644 --- a/doc/user/profile/account/delete_account.md +++ b/doc/user/profile/account/delete_account.md @@ -5,7 +5,7 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Deleting a User account +# Deleting a User account **(FREE)** Users can be deleted from a GitLab instance, either by: @@ -24,7 +24,7 @@ As a user, to delete your own account: 1. On the left sidebar, select **Account**. 1. Select **Delete account**. -## As an administrator +## As an administrator **(FREE SELF)** As an administrator, to delete a user account: @@ -42,11 +42,12 @@ Using the **Delete user and contributions** option may result in removing more data than intended. Please see [associated records](#associated-records) below for additional details. -## Associated Records +### Associated records -> - Introduced for issues in [GitLab 9.0](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7393). -> - Introduced for merge requests, award emoji, notes, and abuse reports in [GitLab 9.1](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10467). -> - Hard deletion from abuse reports and spam logs was introduced in [GitLab 9.1](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10273), and from the API in [GitLab 9.3](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/11853). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7393) for issues in GitLab 9.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10467) for merge requests, award emoji, notes, and abuse reports in GitLab 9.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10273) hard deletion from abuse reports and spam logs in GitLab 9.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/11853) hard deletion from the API in GitLab 9.3. There are two options for deleting users: diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index e4e7e7b9c1a..343f8e328ba 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -20,8 +20,7 @@ password secret. NOTE: When you enable 2FA, don't forget to back up your [recovery codes](#recovery-codes)! -In addition to time-based one time passwords (TOTP), GitLab supports U2F -(universal 2nd factor) and WebAuthn (experimental) devices as the second factor +In addition to time-based one time passwords (TOTP), GitLab supports WebAuthn devices as the second factor of authentication. After being enabled, in addition to supplying your username and password to sign in, you're prompted to activate your U2F / WebAuthn device (usually by pressing a button on it) which performs secure authentication on @@ -80,11 +79,11 @@ in a safe place. ### One-time password via FortiAuthenticator -> - Introduced in [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/212312) +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/212312) in GitLab 13.5. > - It's deployed behind a feature flag, disabled by default. > - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-fortiauthenticator-integration). -You can use FortiAuthenticator as an OTP provider in GitLab. Users must exist in +You can use FortiAuthenticator as a one-time password (OTP) provider in GitLab. Users must exist in both FortiAuthenticator and GitLab with the exact same username, and users must have FortiToken configured in FortiAuthenticator. @@ -154,7 +153,7 @@ Feature.enable(:forti_authenticator, User.find(<user ID>)) ### One-time password via FortiToken Cloud -> - Introduced in [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/212313). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/212313) in GitLab 13.7. > - It's deployed behind a feature flag, disabled by default. > - It's disabled on GitLab.com. > - It's not recommended for production use. @@ -163,7 +162,7 @@ Feature.enable(:forti_authenticator, User.find(<user ID>)) WARNING: This feature might not be available to you. Check the **version history** note above for details. -You can use FortiToken Cloud as an OTP provider in GitLab. Users must exist in +You can use FortiToken Cloud as a one-time password (OTP) provider in GitLab. Users must exist in both FortiToken Cloud and GitLab with the exact same username, and users must have FortiToken configured in FortiToken Cloud. @@ -269,11 +268,11 @@ Click on **Register U2F Device** to complete the process. ### WebAuthn device -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22506) in GitLab 13.4. -> - It's [deployed behind a feature flag](../../feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-webauthn). **(FREE SELF)** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22506) in GitLab 13.4 [with a flag](../../../administration/feature_flags.md) named `webauthn`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/232671) in GitLab 14.6. + +FLAG: +On self-managed GitLab, by default this feature is available. To disable the feature, ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) named `webauthn`. If you disable the WebAuthn feature flag after WebAuthn devices have been registered, these devices are not usable until you re-enable this feature. On GitLab.com, this feature is available. The WebAuthn workflow is [supported by](https://caniuse.com/#search=webauthn) the following desktop browsers: @@ -350,7 +349,7 @@ request, and you're automatically signed in. ### Sign in by using a WebAuthn device In supported browsers you should be automatically prompted to activate your WebAuthn device -(e.g. by touching/pressing its button) after entering your credentials. +(for example, by touching or pressing its button) after entering your credentials. A message displays, indicating that your device responded to the authentication request and you're automatically signed in. @@ -465,13 +464,20 @@ If you regenerate 2FA recovery codes, save them. You can't use any previously cr ### Have 2FA disabled on your account -If you cannot use a saved recovery code or generate new recovery codes then please submit a [support ticket](https://support.gitlab.com/hc/en-us/requests/new) requesting that a GitLab global administrator disables two-factor authentication for your account. Please note that only the actual owner of the account can make this request and that disabling this setting will temporarily leave your account in a less secure state. You should therefore sign in and re-enable two-factor authentication as soon as possible. +If you can't use a saved recovery code or generate new recovery codes, submit a [support ticket](https://support.gitlab.com/hc/en-us/requests/new) to +request a GitLab global administrator disable two-factor authentication for your account. Note that: + +- Only the owner of the account can make this request. +- This service is only available for accounts that have a GitLab.com subscription. For more information, see our + [blog post](https://about.gitlab.com/blog/2020/08/04/gitlab-support-no-longer-processing-mfa-resets-for-free-users/). +- Disabling this setting temporarily leaves your account in a less secure state. You should sign in and re-enable two-factor authentication + as soon as possible. ## Note to GitLab administrators - You need to take special care to that 2FA keeps working after [restoring a GitLab backup](../../../raketasks/backup_restore.md). -- To ensure 2FA authorizes correctly with TOTP server, you may want to ensure +- To ensure 2FA authorizes correctly with time-based one time passwords (TOTP) server, you may want to ensure your GitLab server's time is synchronized via a service like NTP. Otherwise, you may have cases where authorization always fails because of time differences. - The GitLab U2F implementation does _not_ work when the GitLab instance is accessed from @@ -488,25 +494,6 @@ If you cannot use a saved recovery code or generate new recovery codes then plea - To enforce 2FA at the system or group levels see [Enforce Two-factor Authentication](../../../security/two_factor_authentication.md). -## Enable or disable WebAuthn **(FREE SELF)** - -Support for WebAuthn is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:webauthn) -``` - -To disable it: - -```ruby -Feature.disable(:webauthn) -``` - ## Troubleshooting If you are receiving an `invalid pin code` error, this may indicate that there is a time sync issue between the authentication application and the GitLab instance itself. diff --git a/doc/user/profile/active_sessions.md b/doc/user/profile/active_sessions.md index 1797307a00c..b30ee002758 100644 --- a/doc/user/profile/active_sessions.md +++ b/doc/user/profile/active_sessions.md @@ -5,14 +5,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Active sessions +# Active sessions **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17867) in GitLab 10.8. GitLab lists all devices that have logged into your account. You can review the sessions, and revoke any you don't recognize. -## Listing all active sessions +## List all active sessions To list all active sessions: @@ -29,7 +29,7 @@ To list all active sessions: GitLab allows users to have up to 100 active sessions at once. If the number of active sessions exceeds 100, the oldest ones are deleted. -## Revoking a session +## Revoke a session To revoke an active session: @@ -40,7 +40,7 @@ To revoke an active session: NOTE: When any session is revoked all **Remember me** tokens for all -devices are revoked. See ['Why do I keep getting signed out?'](index.md#why-do-i-keep-getting-signed-out) +devices are revoked. See [Why do I keep getting signed out?](index.md#why-do-i-keep-getting-signed-out) for more information about the **Remember me** feature. <!-- ## Troubleshooting diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index d9f10b58c3f..90cb6502bbd 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -100,17 +100,20 @@ When visiting the public page of a user, you can only see the projects which you If the [public level is restricted](../admin_area/settings/visibility_and_access_controls.md#restrict-visibility-levels), user profiles are only visible to signed-in users. -## User profile README +## Add details to your profile with a README > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232157) in GitLab 14.5. -You can add a README section to your profile that can include more information and [formatting](../markdown.md) than -your profile's bio. +If you want to add more information to your profile page, you can create a README file. When you populate the README file with information, it's included on your profile page. To add a README to your profile: -1. Create a new public project with the same name as your GitLab username. +1. Create a new public project with the same project path as your GitLab username. 1. Create a README file inside this project. The file can be any valid [README or index file](../project/repository/index.md#readme-and-index-files). +1. Populate the README file with [Markdown](../markdown.md). + +To use an existing project, [update the path](../project/settings/index.md#renaming-a-repository) of the project to match +your username. ## Add external accounts to your user profile page @@ -257,6 +260,29 @@ To change your commit email: 1. In the **Commit email** dropdown list, select an email address. 1. Select **Update profile settings**. +## Change your primary email + +Your primary email: + +- Is the default email address for your login, commit email, and notification email. +- Must be already [linked to your user profile](#add-emails-to-your-user-profile). + +To change your primary email: + +1. In the top-right corner, select your avatar. +1. Select **Edit profile**. +1. In the **Email** field, enter your new email address. +1. Select **Update profile settings**. + +## Set your public email + +You can select one of your [configured email addresses](#add-emails-to-your-user-profile) to be displayed on your public profile: + +1. In the top-right corner, select your avatar. +1. Select **Edit profile**. +1. In the **Public email** field, select one of the available email addresses. +1. Select **Update profile settings**. + ### Use an automatically-generated private commit email GitLab provides an automatically-generated private commit email address, diff --git a/doc/user/profile/notifications.md b/doc/user/profile/notifications.md index 9faa4b78f8c..acbaf62579f 100644 --- a/doc/user/profile/notifications.md +++ b/doc/user/profile/notifications.md @@ -67,7 +67,7 @@ For each project and group you can select one of the following levels: | Global | Your global settings apply. | | Watch | Receive notifications for any activity. | | Participate | Receive notifications for threads you have participated in. | -| On mention | Receive notifications when you are [mentioned](../project/issues/issue_data_and_actions.md#mentions) in a comment. | +| On mention | Receive notifications when you are [mentioned](../discussions/index.md#mentions) in a comment. | | Disabled | Receive no notifications. | | Custom | Receive notifications for selected events. | @@ -203,7 +203,7 @@ In issues, merge requests, and epics, for most events, the notification is sent - Participants: - The author and assignee. - Authors of comments. - - Anyone [mentioned](../project/issues/issue_data_and_actions.md#mentions) by username in the title + - Anyone [mentioned](../discussions/index.md#mentions) by username in the title or description. - Anyone mentioned by username in a comment if their notification level is "Participating" or higher. - Watchers: users with notification level "Watch". @@ -287,7 +287,7 @@ The participants are: - Authors of the design (can be multiple people if different authors have uploaded different versions of the design). - Authors of comments on the design. -- Anyone that is [mentioned](../project/issues/issue_data_and_actions.md#mentions) in a comment on the design. +- Anyone that is [mentioned](../discussions/index.md#mentions) in a comment on the design. ## Opt out of all GitLab emails @@ -316,19 +316,19 @@ a merge request or an issue. The following table lists all GitLab-specific email headers: -| Header | Description | -|------------------------------------|-------------------------------------------------------------------------| -| `List-Id` | The path of the project in an RFC 2919 mailing list identifier. You can use it for email organization with filters. | -| `X-GitLab-(Resource)-ID` | The ID of the resource the notification is for. The resource, for example, can be `Issue`, `MergeRequest`, `Commit`, or another such resource. | -| `X-GitLab-Discussion-ID` | The ID of the thread the comment belongs to, in notification emails for comments. | -| `X-GitLab-Group-Id` **(PREMIUM)** | The group's ID. Only present on notification emails for epics. | -| `X-GitLab-Group-Path` **(PREMIUM)** | The group's path. Only present on notification emails for epics. | -| [`X-GitLab-NotificationReason`](#x-gitlab-notificationreason) | The reason for the notification. This can be `mentioned`, `assigned`, or `own_activity`. | -| `X-GitLab-Pipeline-Id` | The ID of the pipeline the notification is for, in notification emails for pipelines. | -| `X-GitLab-Project-Id` | The project's ID. | -| `X-GitLab-Project-Path` | The project's path. | -| `X-GitLab-Project` | The name of the project the notification belongs to. | -| `X-GitLab-Reply-Key` | A unique token to support reply by email. | +| Header | Description | +|---------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------| +| `List-Id` | The path of the project in an RFC 2919 mailing list identifier. You can use it for email organization with filters. | +| `X-GitLab-(Resource)-ID` | The ID of the resource the notification is for. The resource, for example, can be `Issue`, `MergeRequest`, `Commit`, or another such resource. | +| `X-GitLab-Discussion-ID` | The ID of the thread the comment belongs to, in notification emails for comments. | +| `X-GitLab-Group-Id` | The group's ID. Only present on notification emails for [epics](../group/epics/index.md). | +| `X-GitLab-Group-Path` | The group's path. Only present on notification emails for [epics](../group/epics/index.md) | +| [`X-GitLab-NotificationReason`](#x-gitlab-notificationreason) | The reason for the notification. This can be `mentioned`, `assigned`, or `own_activity`. | +| `X-GitLab-Pipeline-Id` | The ID of the pipeline the notification is for, in notification emails for pipelines. | +| `X-GitLab-Project-Id` | The project's ID. | +| `X-GitLab-Project-Path` | The project's path. | +| `X-GitLab-Project` | The name of the project the notification belongs to. | +| `X-GitLab-Reply-Key` | A unique token to support reply by email. | ### X-GitLab-NotificationReason diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index 197ba4647b2..ea66f3e508f 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -7,10 +7,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Personal access tokens **(FREE)** -> - Introduced in GitLab 12.6: [Notifications for expiring tokens](https://gitlab.com/gitlab-org/gitlab/-/issues/3649). -> - Introduced in GitLab Ultimate 12.6: [Token lifetime limits](https://gitlab.com/gitlab-org/gitlab/-/issues/3649). -> - Introduced in GitLab 13.3: [Additional notifications for expiring tokens](https://gitlab.com/gitlab-org/gitlab/-/issues/214721). -> - Introduced in GitLab 14.1: [Prefill token name and scopes](https://gitlab.com/gitlab-org/gitlab/-/issues/334664). +> - Notifications for expiring tokens [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3649) in GitLab 12.6. +> - Token lifetime limits [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3649) in GitLab 12.6. +> - Additional notifications for expiring tokens [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214721) in GitLab 13.3. +> - Prefill for token name and scopes [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334664) in GitLab 14.1. Personal access tokens can be an alternative to [OAuth2](../../api/oauth2.md) and used to: diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index e079e6dcbee..63be88f90d6 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: concepts, howto --- -# Profile preferences +# Profile preferences **(FREE)** A user's profile preferences page allows the user to customize various aspects of GitLab to their liking. @@ -120,7 +120,7 @@ You can include the following options for your default dashboard view: - Your [To-Do List](../todos.md) - Assigned Issues - Assigned Merge Requests -- Operations Dashboard **(PREMIUM)** +- [Operations Dashboard](../operations_dashboard/index.md) ### Group overview content @@ -130,7 +130,7 @@ displayed on a group's home page. You can choose between 2 options: - Details (default) -- [Security dashboard](../application_security/security_dashboard/index.md) **(ULTIMATE)** +- [Security dashboard](../application_security/security_dashboard/index.md) ### Project overview content diff --git a/doc/user/profile/unknown_sign_in_notification.md b/doc/user/profile/unknown_sign_in_notification.md index 7aa1ae89c9f..be86db3daf5 100644 --- a/doc/user/profile/unknown_sign_in_notification.md +++ b/doc/user/profile/unknown_sign_in_notification.md @@ -1,11 +1,10 @@ --- -type: concepts, howto stage: Manage group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Email notification for unknown sign-ins +# Email notification for unknown sign-ins **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27211) in GitLab 13.0. diff --git a/doc/user/project/clusters/add_eks_clusters.md b/doc/user/project/clusters/add_eks_clusters.md index e03e5b10236..a5473629d4f 100644 --- a/doc/user/project/clusters/add_eks_clusters.md +++ b/doc/user/project/clusters/add_eks_clusters.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. WARNING: -This feature was deprecated in GitLab 14.5. Use [Infrastrucure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac) +This feature was deprecated in GitLab 14.5. Use [Infrastructure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac-deprecated) to create new clusters. Through GitLab, you can create new clusters and add existing clusters hosted on Amazon Elastic @@ -19,11 +19,11 @@ Kubernetes Service (EKS). ## Connect an existing EKS cluster If you already have an EKS cluster and want to connect it to GitLab, -use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). +use the [GitLab Agent](../../clusters/agent/index.md). ## Create a new EKS cluster -To create a new cluster from GitLab, use [Infrastructure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac). +To create a new cluster from GitLab, use [Infrastructure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac-deprecated). ### How to create a new cluster on EKS through cluster certificates (DEPRECATED) diff --git a/doc/user/project/clusters/add_existing_cluster.md b/doc/user/project/clusters/add_existing_cluster.md index fcf2583d3ab..acc5ed4cb30 100644 --- a/doc/user/project/clusters/add_existing_cluster.md +++ b/doc/user/project/clusters/add_existing_cluster.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. -To connect your cluster to GitLab, use the [GitLab Kubernetes Agent](../../clusters/agent/index.md) +To connect your cluster to GitLab, use the [GitLab Agent](../../clusters/agent/index.md) instead. If you have an existing Kubernetes cluster, you can add it to a project, group, diff --git a/doc/user/project/clusters/add_gke_clusters.md b/doc/user/project/clusters/add_gke_clusters.md index 30be319f2df..99edddeb3e9 100644 --- a/doc/user/project/clusters/add_gke_clusters.md +++ b/doc/user/project/clusters/add_gke_clusters.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. -Use [Infrastrucure as Code](../../infrastructure/clusters/connect/new_gke_cluster.md) +Use [Infrastructure as Code](../../infrastructure/clusters/connect/new_gke_cluster.md) to create a cluster hosted on Google Kubernetes Engine (GKE). Through GitLab, you can create new and connect existing clusters @@ -19,7 +19,7 @@ hosted on Google Kubernetes Engine (GKE). ## Connect an existing GKE cluster If you already have a GKE cluster and want to connect it to GitLab, -use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). +use the [GitLab Agent](../../clusters/agent/index.md). ## Create a new GKE cluster from GitLab diff --git a/doc/user/project/clusters/add_remove_clusters.md b/doc/user/project/clusters/add_remove_clusters.md index 49708e3b6aa..a0fca517f2e 100644 --- a/doc/user/project/clusters/add_remove_clusters.md +++ b/doc/user/project/clusters/add_remove_clusters.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/327908) in GitLab 14.0. -To create a new cluster use [Infrastructure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac). +To create a new cluster use [Infrastructure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac-deprecated). NOTE: Every new Google Cloud Platform (GCP) account receives @@ -29,7 +29,7 @@ in a few clicks. > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/327908) in GitLab 14.0. -As of GitLab 14.0, use [Infrastructure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac) +As of GitLab 14.0, use [Infrastructure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac-deprecated) to **safely create new clusters from GitLab**. Creating clusters from GitLab using cluster certificates is still available on the @@ -49,7 +49,7 @@ supports connecting existing clusters using the certificate-based connection met ## Add existing cluster -As of GitLab 14.0, use the [GitLab Kubernetes Agent](../../clusters/agent/index.md) +As of GitLab 14.0, use the [GitLab Agent](../../clusters/agent/index.md) to connect your cluster to GitLab. Alternatively, you can [add an existing cluster](add_existing_cluster.md) @@ -57,7 +57,7 @@ through the certificate-based method, but we don't recommend using this method f ## Configure your cluster -As of GitLab 14.0, use the [GitLab Kubernetes Agent](../../clusters/agent/index.md) +As of GitLab 14.0, use the [GitLab Agent](../../clusters/agent/index.md) to configure your cluster. ## Disable a cluster diff --git a/doc/user/project/clusters/cluster_access.md b/doc/user/project/clusters/cluster_access.md index 510aad821cf..4e00ae0dd07 100644 --- a/doc/user/project/clusters/cluster_access.md +++ b/doc/user/project/clusters/cluster_access.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. -To connect your cluster to GitLab, use the [GitLab Kubernetes Agent](../../clusters/agent/index.md) +To connect your cluster to GitLab, use the [GitLab Agent](../../clusters/agent/index.md) instead. When creating a cluster in GitLab, you are asked if you would like to create either: diff --git a/doc/user/project/clusters/deploy_to_cluster.md b/doc/user/project/clusters/deploy_to_cluster.md index c3a71ec8585..e89ce12b35e 100644 --- a/doc/user/project/clusters/deploy_to_cluster.md +++ b/doc/user/project/clusters/deploy_to_cluster.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. -To connect your cluster to GitLab, use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). +To connect your cluster to GitLab, use the [GitLab Agent](../../clusters/agent/index.md). To deploy with the Agent, use the [CI/CD Tunnel](../../clusters/agent/ci_cd_tunnel.md). A Kubernetes cluster can be the destination for a deployment job. If diff --git a/doc/user/project/clusters/gitlab_managed_clusters.md b/doc/user/project/clusters/gitlab_managed_clusters.md index ad378be2d9a..686b3026b2c 100644 --- a/doc/user/project/clusters/gitlab_managed_clusters.md +++ b/doc/user/project/clusters/gitlab_managed_clusters.md @@ -12,7 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. -To connect your cluster to GitLab, use the [GitLab Kubernetes Agent](../../../user/clusters/agent/index.md). +To connect your cluster to GitLab, use the [GitLab Agent](../../../user/clusters/agent/index.md). To manage applications, use the [Cluster Project Management Template](../../../user/clusters/management_project_template.md). You can choose to allow GitLab to manage your cluster for you. If your cluster diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index c16c6446acd..dc37060593c 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -12,7 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. To connect clusters to GitLab, use the -[GitLab Kubernetes Agent](../../clusters/agent/index.md). +[GitLab Agent](../../clusters/agent/index.md). [Project-level](../../infrastructure/clusters/connect/index.md#cluster-levels-deprecated) Kubernetes clusters allow you to connect a Kubernetes cluster to a project in GitLab. diff --git a/doc/user/project/clusters/multiple_kubernetes_clusters.md b/doc/user/project/clusters/multiple_kubernetes_clusters.md index 540907bf915..12527853b40 100644 --- a/doc/user/project/clusters/multiple_kubernetes_clusters.md +++ b/doc/user/project/clusters/multiple_kubernetes_clusters.md @@ -13,7 +13,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: Using multiple Kubernetes clusters for a single project **with cluster certificates** was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. -To connect clusters to GitLab, use the [GitLab Kubernetes Agent](../../../user/clusters/agent/index.md). +To connect clusters to GitLab, use the [GitLab Agent](../../../user/clusters/agent/index.md). You can associate more than one Kubernetes cluster to your project. That way you can have different clusters for different environments, diff --git a/doc/user/project/clusters/protect/container_host_security/index.md b/doc/user/project/clusters/protect/container_host_security/index.md index c005ce64bb5..98ba4a1f84d 100644 --- a/doc/user/project/clusters/protect/container_host_security/index.md +++ b/doc/user/project/clusters/protect/container_host_security/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w NOTE: In GitLab 14.5, using a certificate to connect GitLab to a Kubernetes cluster is [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). You can continue using Container Host Security, even though it relies on this certificate-based -method. The work to allow all aspects of Container Host Security to function through the [GitLab Kubernetes Agent](../../../../clusters/agent/index.md) +method. The work to allow all aspects of Container Host Security to function through the [GitLab Agent](../../../../clusters/agent/index.md) instead of the certificate-based method can be tracked [in this GitLab issue](https://gitlab.com/gitlab-org/gitlab/-/issues/299350). Container Host Security in GitLab provides Intrusion Detection and Prevention capabilities that can diff --git a/doc/user/project/clusters/protect/container_network_security/index.md b/doc/user/project/clusters/protect/container_network_security/index.md index eb15675da19..06dc6b24620 100644 --- a/doc/user/project/clusters/protect/container_network_security/index.md +++ b/doc/user/project/clusters/protect/container_network_security/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w NOTE: In GitLab 14.5, using a certificate to connect GitLab to a Kubernetes cluster is [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). You can continue using Container Network Security, even though it relies on this certificate-based -method. The work to allow all aspects of Container Network Security to function through the [GitLab Kubernetes Agent](../../../../clusters/agent/index.md) +method. The work to allow all aspects of Container Network Security to function through the [GitLab Agent](../../../../clusters/agent/index.md) instead of the certificate-based method can be tracked [in this GitLab issue](https://gitlab.com/gitlab-org/gitlab/-/issues/299350) and [this GitLab Epic](https://gitlab.com/groups/gitlab-org/-/epics/7057). Container Network Security in GitLab provides basic firewall functionality by leveraging Cilium diff --git a/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md b/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md index 283e6c0b81c..340c9397e9c 100644 --- a/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md +++ b/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md @@ -192,11 +192,10 @@ violations but don't block any traffic. To set Cilium to Blocking mode, you must lines to the `applications/cilium/values.yaml` file in your cluster management project: ```yaml -config: - policyAuditMode: false +policyEnforcementMode: "always" monitor: - eventTypes: ["drop"] + eventTypes: ["drop", "policy-verdict"] ``` ### Traffic is not being allowed as expected diff --git a/doc/user/project/clusters/serverless/aws.md b/doc/user/project/clusters/serverless/aws.md index 06fa18d80c9..ccf90a3d3dd 100644 --- a/doc/user/project/clusters/serverless/aws.md +++ b/doc/user/project/clusters/serverless/aws.md @@ -290,7 +290,7 @@ The example code is available: - As a [clonable repository](https://gitlab.com/gitlab-org/serverless/examples/serverless-framework-js). - In a version with [tests and secret variables](https://gitlab.com/gitlab-org/project-templates/serverless-framework/). -You can also use a [template](../../working_with_projects.md#project-templates) +You can also use a [template](../../working_with_projects.md#create-a-project) (based on the version with tests and secret variables) from within the GitLab UI (see the `Serverless Framework/JS` template). diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md index f6598f8846b..265a60c6f2c 100644 --- a/doc/user/project/clusters/serverless/index.md +++ b/doc/user/project/clusters/serverless/index.md @@ -539,7 +539,7 @@ server that has Python 3 installed, and may not work on other operating systems or with other versions of Python. 1. Install Certbot by running the - [`certbot-auto` wrapper script](https://certbot.eff.org/docs/install.html#certbot-auto). + [`certbot-auto` wrapper script](https://eff-certbot.readthedocs.io/install.html#certbot-auto). On the command line of your server, run the following commands: ```shell diff --git a/doc/user/project/code_owners.md b/doc/user/project/code_owners.md index c138dc64d19..a95e4d2bc26 100644 --- a/doc/user/project/code_owners.md +++ b/doc/user/project/code_owners.md @@ -11,6 +11,10 @@ type: reference > - Code Owners for merge request approvals was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4418) in GitLab Premium 11.9. > - Moved to GitLab Premium in 13.9. +INFO: +Get access to Code Owners and more with a +[free 30-day trial of GitLab Ultimate](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-code-owners-docs). + Code Owners define who owns specific files or directories in a repository. - The users you define as Code Owners are displayed in the UI when you browse directories. @@ -173,12 +177,16 @@ entries under **Database**. The entries defined under the sections **Documentati ### Make a Code Owners section optional -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232995) in GitLab Premium 13.8 behind a feature flag, enabled by default. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53227) in GitLab 13.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232995) in GitLab Premium 13.8. -You can make a section optional, so that approval from the Code Owners in that section is optional. +You can designate optional sections in your Code Owners file. Prepend the +section name with the caret `^` character to treat the entire section as optional. +Optional sections enable you to designate responsible parties for various parts +of your codebase, but not require approval from them. This approach provides +a more relaxed policy for parts of your project that are frequently updated, +but don't require stringent reviews. -Put a caret `^` character before the Code Owners section name. For example: +In this example, the `[Go]` section is optional: ```plaintext [Documentation] @@ -200,8 +208,12 @@ If a section is duplicated in the file, and one of them is marked as optional an Optional sections in the `CODEOWNERS` file are treated as optional only when changes are submitted by using merge requests. If a change is submitted directly to the protected branch, approval from Code Owners is still required, even if the -section is marked as optional. [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/297638) -to allow direct pushes to the protected branch for sections marked as optional. +section is marked as optional. + +### Allowed to Push + +The Code Owner approval and protected branch features do not apply to users who +are **Allowed to push**. ## Example `CODEOWNERS` file diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index 5b19a54bd91..66e5931fa4c 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -71,7 +71,7 @@ To create the `.gitlab/issue_templates` directory: 1. Select **New directory**. 1. Name your directory `issue_templates` and commit to your default branch. -To check if this has worked correctly, [create a new issue](issues/managing_issues.md#create-a-new-issue) +To check if this has worked correctly, [create a new issue](issues/managing_issues.md#create-an-issue) and see if you can choose a description template. ## Create a merge request template diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index e1a81ae1bba..72bf0841687 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -52,9 +52,12 @@ self-managed GitLab instance. - If you're importing to a self-managed GitLab instance, you can alternatively use the [GitHub Rake task](../../../administration/raketasks/github_import.md) to import projects without the constraints of a [Sidekiq](../../../development/sidekiq_style_guide.md) worker. -- If you're importing from GitHub Enterprise to your self-managed GitLab instance, you must first enable - [GitHub integration](../../../integration/github.md). +- If you're importing from GitHub Enterprise to your self-managed GitLab instance: + - You must first enable [GitHub integration](../../../integration/github.md). - To import projects from GitHub Enterprise to GitLab.com, use the [Import API](../../../api/import.md). + - If GitLab is behind a HTTP/HTTPS proxy you must populate the [allowlist for local requests](../../../security/webhooks.md#allowlist-for-local-requests) + with `github.com` and `api.github.com` to solve the hostname. For more information, read the issue + [Importing a GitHub project requires DNS resolution even when behind a proxy](https://gitlab.com/gitlab-org/gitlab/-/issues/37941) - If you're importing from GitHub.com to your self-managed GitLab instance, setting up GitHub integration is not required. You can use the [Import API](../../../api/import.md). diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index 6e02a9bf5ab..9d7ed593d41 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -76,7 +76,7 @@ a self-managed instance from an old server to a new server. The backups produced don't depend on the operating system running GitLab. You can therefore use the restore method to switch between different operating system distributions or versions, as long -as the same GitLab version [is available for installation](../../../administration/package_information/deprecated_os.md). +as the same GitLab version [is available for installation](../../../administration/package_information/supported_os.md). To instead merge two self-managed GitLab instances together, use the instructions in [Migrate from self-managed GitLab to GitLab.com](#migrate-from-self-managed-gitlab-to-gitlabcom). diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 78cd2f8fb79..07e8ea1dc06 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -146,7 +146,7 @@ There are numerous [APIs](../../api/index.md) to use with your projects: - [Markdown](../../api/markdown.md) - [Merge Requests](../../api/merge_requests.md) - [Milestones](../../api/milestones.md) -- [Services](../../api/services.md) +- [Services](../../api/integrations.md) - [Snippets](../../api/project_snippets.md) - [Templates](../../api/project_templates.md) - [Traffic](../../api/project_statistics.md) diff --git a/doc/user/project/insights/index.md b/doc/user/project/insights/index.md index 436df07d673..957290c5f20 100644 --- a/doc/user/project/insights/index.md +++ b/doc/user/project/insights/index.md @@ -149,7 +149,7 @@ Supported values are: | Name | Example | | ----- | ------- | | `bar` |  | -| `bar` (time series, i.e. when `group_by` is used) |  | +| `bar` (time series, that is when `group_by` is used) |  | | `line` |  | | `stacked-bar` |  | diff --git a/doc/user/project/integrations/asana.md b/doc/user/project/integrations/asana.md index 963fca34827..b4d7790df1d 100644 --- a/doc/user/project/integrations/asana.md +++ b/doc/user/project/integrations/asana.md @@ -37,7 +37,7 @@ Complete these steps in GitLab: 1. Select **Asana**. 1. Ensure that the **Active** toggle is enabled. 1. Paste the token you generated in Asana. -1. (Optional) To restrict this setting to specific branches, list them in the **Restrict to branch** +1. Optional. To restrict this setting to specific branches, list them in the **Restrict to branch** field, separated with commas. 1. Select **Save changes** or optionally select **Test settings**. diff --git a/doc/user/project/integrations/bamboo.md b/doc/user/project/integrations/bamboo.md index 58cfd8c3a2f..38de8d9f1af 100644 --- a/doc/user/project/integrations/bamboo.md +++ b/doc/user/project/integrations/bamboo.md @@ -4,59 +4,66 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Atlassian Bamboo Service **(FREE)** +# Atlassian Bamboo integration **(FREE)** -GitLab provides integration with Atlassian Bamboo for continuous integration. -When configured, pushes to a project trigger a build in Bamboo automatically. -Merge requests also display CI/CD status showing whether the build is pending, -failed, or completed successfully. It also provides a link to the Bamboo build -page for more information. +You can automatically trigger builds in Atlassian Bamboo when you push changes +to your project in GitLab. -Bamboo doesn't quite provide the same features as a traditional build system when -it comes to accepting webhooks and commit data. There are a few things that -need to be configured in a Bamboo build plan before GitLab can integrate. +When this integration is configured, merge requests also display the following information: -## Setup +- A CI/CD status that shows if the build is pending, failed, or has completed successfully. +- A link to the Bamboo build page for more information. -### Complete these steps in Bamboo +Bamboo doesn't provide the same features as a traditional build system when +accepting webhooks and commit data. You must configure a Bamboo +build plan before you configure the integration in GitLab. -1. Navigate to a Bamboo build plan and choose **Configure plan** from the **Actions** - dropdown. +## Configure Bamboo + +1. In Bamboo, go to a build plan and choose **Actions > Configure plan**. 1. Select the **Triggers** tab. -1. Click **Add trigger**. -1. Enter a description such as **GitLab trigger**. -1. Choose **Repository triggers the build when changes are committed**. +1. Select **Add trigger**. +1. Enter a description like `GitLab trigger`. +1. Select **Repository triggers the build when changes are committed**. 1. Select the checkbox for one or more repositories. -1. Enter the GitLab IP address in the **Trigger IP addresses** box. This is a - list of IP addresses that are allowed to trigger Bamboo builds. +1. Enter the GitLab IP address in **Trigger IP addresses**. These IP addresses + are allowed to trigger Bamboo builds. 1. Save the trigger. -1. In the left pane, select a build stage. If you have multiple build stages - you want to select the last stage that contains the Git checkout task. +1. In the left pane, select a build stage. If you have multiple build stages, + select the last stage that contains the Git checkout task. 1. Select the **Miscellaneous** tab. -1. Under **Pattern Match Labeling** put `${bamboo.repository.revision.number}` - in the **Labels** box. -1. Save - -Bamboo is now ready to accept triggers from GitLab. Next, set up the Bamboo -service in GitLab. - -### Complete these steps in GitLab - -1. Navigate to the project you want to configure to trigger builds. -1. Navigate to the [Integrations page](overview.md#accessing-integrations) -1. Click **Atlassian Bamboo**. -1. Ensure that the **Active** toggle is enabled. -1. Enter the base URL of your Bamboo server. `https://bamboo.example.com` -1. Enter the build key from your Bamboo build plan. Build keys are typically made - up from the Project Key and Plan Key that are set on project/plan creation and - separated with a dash (`-`), for example **PROJ-PLAN**. This is a short, all - uppercase identifier that is unique. When viewing a plan in Bamboo, the - build key is also shown in the browser URL, for example `https://bamboo.example.com/browse/PROJ-PLAN`. -1. If necessary, enter username and password for a Bamboo user that has +1. Under **Pattern Match Labeling** enter `${bamboo.repository.revision.number}` + in **Labels**. +1. Select **Save**. + +Bamboo is ready to accept triggers from GitLab. Next, set up the Bamboo +integration in GitLab. + +## Configure GitLab + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. +1. Select **Atlassian Bamboo**. +1. Ensure the **Active** checkbox is selected. +1. Enter the base URL of your Bamboo server. For example, `https://bamboo.example.com`. +1. Enter the [build key](#identify-the-bamboo-build-plan-build-key) from your Bamboo + build plan. +1. If necessary, enter a username and password for a Bamboo user that has access to trigger the build plan. Leave these fields blank if you do not require authentication. -1. Save or optionally click **Test Settings**. **Test Settings** - actually triggers a build in Bamboo. +1. Optional. To test the configuration and trigger a build in Bamboo, + select **Test Settings**. +1. Select **Save changes**. + +### Identify the Bamboo build plan build key + +A build key is a unique identifier typically made up from the project key and +plan key. +Build keys are short, all uppercase, and separated with a dash (`-`), +for example `PROJ-PLAN`. + +The build key is included in the browser URL when you view a plan in +Bamboo. For example, `https://bamboo.example.com/browse/PROJ-PLAN`. ## Troubleshooting diff --git a/doc/user/project/integrations/github.md b/doc/user/project/integrations/github.md index 47a81594ca9..9f1ea3796c6 100644 --- a/doc/user/project/integrations/github.md +++ b/doc/user/project/integrations/github.md @@ -6,17 +6,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitHub project integration **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3836) in GitLab Premium 10.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3836) in GitLab 10.6. -GitLab provides an integration for updating the pipeline statuses on GitHub. -This is especially useful if using GitLab for CI/CD only. - -This project integration is separate from the [instance wide GitHub integration](../import/github.md#mirror-a-repository-and-share-pipeline-status) -and is automatically configured on [GitHub import](../../../integration/github.md). +You can update GitHub with pipeline status updates from GitLab. +This integration can help you if you use GitLab for CI/CD.  -## Configuration +This project integration is separate from the [instance-wide GitHub integration](../import/github.md#mirror-a-repository-and-share-pipeline-status) +and is automatically configured when you import a [GitHub project](../../../integration/github.md). + +## Configure the integration This integration requires a [GitHub API token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) with `repo:status` access granted. @@ -26,31 +26,39 @@ Complete these steps on GitHub: 1. Go to your **Personal access tokens** page at <https://github.com/settings/tokens>. 1. Select **Generate new token**. 1. Under **Note**, enter a name for the new token. -1. Ensure that `repo:status` is checked and select **Generate token**. +1. Ensure `repo:status` is selected and select **Generate token**. 1. Copy the generated token to use in GitLab. Complete these steps in GitLab: -1. Go to the project you want to configure. -1. Go to the [Integrations page](overview.md#accessing-integrations) +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **GitHub**. -1. Ensure that the **Active** toggle is enabled. -1. Paste the token you generated on GitHub. -1. Enter the path to your project on GitHub, such as `https://github.com/username/repository`. -1. (Optional) To disable static status check names, clear the **Enable static status check names** checkbox. -1. Select **Save changes** or optionally select **Test settings**. +1. Ensure the **Active** checkbox is selected. +1. In **Token**, paste the token you generated on GitHub. +1. In **Repository URL**, enter the path to your project on GitHub, such as `https://github.com/username/repository`. +1. Optional. To disable [static status check names](#static-or-dynamic-status-check-names), clear the **Enable static status check names** checkbox. +1. Optional. Select **Test settings**. +1. Select **Save changes**. After configuring the integration, see [Pipelines for external pull requests](../../../ci/ci_cd_for_external_repos/#pipelines-for-external-pull-requests) to configure pipelines to run for open pull requests. ### Static or dynamic status check names -> - Introduced in GitLab 11.5: using static status check names as opt-in option. -> - [In GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/9931), static status check names is default behavior for new projects. +> - Introduced in GitLab 11.5 with static status check names as an opt-in option. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/9931) in GitLab 12.4 to make static status check names the default behavior for new projects. + +A status check name can be static or dynamic: + +- **Static**: The hostname of your + GitLab instance is appended to the status check name. -This makes it possible to mark these status checks as **Required** on GitHub. +- **Dynamic**: The branch name is appended + to the status check name. -When **Enable static status check names** is checked on the integration page, your -GitLab instance host name is appended to a status check name. +The **Enable static status check names** option enables you to configure +required status checks in GitHub, which need a consistent (static) name to work correctly. -When unchecked, it uses dynamic status check names and appends the branch name. +If you [disable this option](#configure-the-integration), +GitLab uses dynamic status check names instead. diff --git a/doc/user/project/integrations/hangouts_chat.md b/doc/user/project/integrations/hangouts_chat.md index bcaedbc4b10..7a96bb74e3f 100644 --- a/doc/user/project/integrations/hangouts_chat.md +++ b/doc/user/project/integrations/hangouts_chat.md @@ -32,7 +32,7 @@ Select a room and create a webhook: 1. Enter the room where you want to receive notifications from GitLab. 1. Open the room dropdown menu on the top-left and select **Manage webhooks**. 1. Enter the name for your webhook, for example "GitLab integration". -1. (Optional) Add an avatar for your bot. +1. Optional. Add an avatar for your bot. 1. Select **Save**. 1. Copy the webhook URL. @@ -46,7 +46,7 @@ Enable the Google Chat integration in GitLab: 1. Scroll down to the end of the page where you find a **Webhook** field. 1. Enter the webhook URL you copied from Google Chat. 1. Select the events you want to be notified about in your Google Chat room. -1. (Optional) Select **Test settings** to verify the connection. +1. Optional. Select **Test settings** to verify the connection. 1. Select **Save changes**. To test the integration, make a change based on the events you selected and diff --git a/doc/user/project/integrations/img/webhook_testing.png b/doc/user/project/integrations/img/webhook_testing.png Binary files differindex 27836556acc..88ce05668f9 100644 --- a/doc/user/project/integrations/img/webhook_testing.png +++ b/doc/user/project/integrations/img/webhook_testing.png diff --git a/doc/user/project/integrations/irker.md b/doc/user/project/integrations/irker.md index b96605ff5c9..279b139bacd 100644 --- a/doc/user/project/integrations/irker.md +++ b/doc/user/project/integrations/irker.md @@ -10,7 +10,7 @@ GitLab provides a way to push update messages to an irker server. When configured, pushes to a project trigger the service to send data directly to the irker server. -See also the [irker integration API documentation](../../../api/services.md). +See also the [irker integration API documentation](../../../api/integrations.md). For more information, see the [irker project homepage](https://gitlab.com/esr/irker). diff --git a/doc/user/project/integrations/mattermost.md b/doc/user/project/integrations/mattermost.md index 119f219499c..f3f8d900e12 100644 --- a/doc/user/project/integrations/mattermost.md +++ b/doc/user/project/integrations/mattermost.md @@ -50,12 +50,12 @@ Then fill in the integration configuration: - **Webhook**: The incoming webhook URL on Mattermost, similar to `http://mattermost.example/hooks/5xo…`. -- **Username**: (Optional) The username shown in messages sent to Mattermost. +- **Username**: Optional. The username shown in messages sent to Mattermost. To change the bot's username, provide a value. - **Notify only broken pipelines**: If you enable the **Pipeline** event, and you want notifications about failed pipelines only. - **Branches for which notifications are to be sent**: The branches to send notifications for. -- **Labels to be notified**: (Optional) Labels required for the issue or merge request +- **Labels to be notified**: Optional. Labels required for the issue or merge request to trigger a notification. Leave blank to notify for all issues and merge requests. - **Labels to be notified behavior**: When you use the **Labels to be notified** filter, messages are sent when an issue or merge request contains _any_ of the labels specified diff --git a/doc/user/project/integrations/pivotal_tracker.md b/doc/user/project/integrations/pivotal_tracker.md index 93a3490e4b6..8b17f4afaa8 100644 --- a/doc/user/project/integrations/pivotal_tracker.md +++ b/doc/user/project/integrations/pivotal_tracker.md @@ -42,6 +42,6 @@ Complete these steps in GitLab: 1. Select **Pivotal Tracker**. 1. Ensure that the **Active** toggle is enabled. 1. Paste the token you generated in Pivotal Tracker. -1. (Optional) To restrict this setting to specific branches, list them in the **Restrict to branch** +1. Optional. To restrict this setting to specific branches, list them in the **Restrict to branch** field, separated with commas. 1. Select **Save changes** or optionally select **Test settings**. diff --git a/doc/user/project/integrations/slack.md b/doc/user/project/integrations/slack.md index d399c7f2901..87f38c3482b 100644 --- a/doc/user/project/integrations/slack.md +++ b/doc/user/project/integrations/slack.md @@ -31,7 +31,7 @@ to control GitLab from Slack. Slash commands are configured separately. [Triggers for Slack notifications](#triggers-for-slack-notifications). By default, messages are sent to the channel you configured during [Slack configuration](#configure-slack). -1. (Optional) To send messages to a different channel, multiple channels, or as +1. Optional. To send messages to a different channel, multiple channels, or as a direct message: - *To send messages to channels,* enter the Slack channel names, separated by commas. @@ -42,7 +42,7 @@ to control GitLab from Slack. Slash commands are configured separately. 1. In **Webhook**, enter the webhook URL you copied in the [Slack configuration](#configure-slack) step. -1. (Optional) In **Username**, enter the username of the Slack bot that sends +1. Optional. In **Username**, enter the username of the Slack bot that sends the notifications. 1. Select the **Notify only broken pipelines** checkbox to notify only on failures. 1. In the **Branches for which notifications are to be sent** dropdown, select which types of branches diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index 4c35f007fc7..47a2d215024 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -219,27 +219,6 @@ This ordering also affects [issue lists](issues/sorting_issue_lists.md). Changing the order in an issue board changes the ordering in an issue list, and vice versa. -### GraphQL-based issue boards - -<!-- This anchor is linked from #blocked-issues as well. --> - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285074) in GitLab 13.9. -> - [Deployed behind a feature flag](../feature_flags.md), enabled by default. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/248908) in GitLab 14.1 -> - [Feature flag `graphql_board_lists`](https://gitlab.com/gitlab-org/gitlab/-/issues/248908) removed in GitLab 14.3 - -There can be -[risks when disabling released features](../../administration/feature_flags.md#risks-when-disabling-released-features). -Refer to this feature's version history for more details. - -Using GraphQL-based boards gives you these -additional features: - -- [Edit more issue attributes](#edit-an-issue) -- [View blocked issues](#blocked-issues) - -Learn more about the known issues in [epic 5596](https://gitlab.com/groups/gitlab-org/-/epics/5596). - ## GitLab Enterprise features for issue boards GitLab issue boards are available on the GitLab Free tier, but some @@ -334,19 +313,13 @@ As in other list types, click the trash icon to remove a list. ### Iteration lists **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250479) in GitLab 13.11. -> - Enabled on GitLab.com and is ready for production use. -> - Enabled with `iteration_board_lists` flag for self-managed GitLab and is ready for production use. -> GitLab administrators can opt to [disable the feature flag](#enable-or-disable-iteration-lists-in-boards). - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, ask an -administrator to [disable the `iteration_board_lists` flag](../../administration/feature_flags.md). -On GitLab.com, this feature is available. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250479) in GitLab 13.11 [with a flag](../../administration/feature_flags.md) named `iteration_board_lists`. Enabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75404) in GitLab 14.6. Feature flag `iteration_board_lists` removed. You're also able to create lists of an iteration. -These are lists that filter issues by the assigned -iteration. To add an iteration list: +These lists filter issues by the assigned iteration. + +To add an iteration list: 1. Select **Create list**. 1. Select **Iteration**. @@ -434,8 +407,6 @@ status. When you hover over the blocked icon (**{issue-block}**), a detailed information popover is displayed. -This feature is only supported when using the [GraphQL-based boards](#graphql-based-issue-boards). The feature is enabled by default regardless when you use group issue boards in epic swimlanes mode. -  ## Actions you can take on an issue board @@ -457,25 +428,25 @@ If you're not able to do some of the things above, make sure you have the right ### Edit an issue +> Editing title, iteration, and confidentiality [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/248908) in GitLab 14.1. + You can edit an issue without leaving the board view. To open the right sidebar, select an issue card (not its title). You can edit the following issue attributes in the right sidebar: - Assignees -- [Epic](../group/epics/index.md) -- Milestone -- Time tracking value (view only) +- Confidentiality - Due date +- [Epic](../group/epics/index.md) +- [Iteration](../group/iterations/index.md) - Labels -- [Weight](issues/issue_weight.md) +- Milestone - Notifications setting - -When you use [GraphQL-based boards](#graphql-based-issue-boards), you can also edit the following issue attributes: - - Title -- [Iteration](../group/iterations/index.md) -- Confidentiality +- [Weight](issues/issue_weight.md) + +Additionally, you can also see the time tracking value. ### Create a new list @@ -620,13 +591,12 @@ and the target list. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18954) in GitLab 12.4. > - [Placed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61955) behind a [feature flag](../feature_flags.md), disabled by default in GitLab 14.0. -> - Disabled on GitLab.com. -> - Not recommended for production use. -> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-multi-selecting-issue-cards). **(FREE SELF)** -This in-development feature might not be available for your use. There can be -[risks when enabling features still in development](../../administration/feature_flags.md#risks-when-enabling-features-still-in-development). -Refer to this feature's version history for more details. +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an +administrator to [enable the feature flag](../../administration/feature_flags.md) named `board_multi_select`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. You can select multiple issue cards, then drag the group to another position within the list, or to another list. This makes it faster to reorder many issues at once. @@ -668,41 +638,3 @@ A few things to remember: - For performance and visibility reasons, each list shows the first 20 issues by default. If you have more than 20 issues, start scrolling down and the next 20 appear. - -### Enable or disable iteration lists in boards **(PREMIUM SELF)** - -The iteration list is under development but ready for production use. It is -deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can disable it. - -To enable it: - -```ruby -Feature.enable(:iteration_board_lists) -``` - -To disable it: - -```ruby -Feature.disable(:iteration_board_lists) -``` - -### Enable or disable multi-selecting issue cards **(FREE SELF)** - -Multi-selecting issue cards is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:board_multi_select) -``` - -To disable it: - -```ruby -Feature.disable(:board_multi_select) -``` diff --git a/doc/user/project/issues/confidential_issues.md b/doc/user/project/issues/confidential_issues.md index b8a01f7ccd6..e8c58f2feb9 100644 --- a/doc/user/project/issues/confidential_issues.md +++ b/doc/user/project/issues/confidential_issues.md @@ -42,6 +42,11 @@ system note in the issue's comments.  +When an issue is made confidential, only users with at least the [Reporter role](../../permissions.md) +for the project have access to the issue. +Users with Guest or [Minimal](../../permissions.md#users-with-minimal-access) roles can't access +the issue even if they were actively participating before the change. + ## Indications of a confidential issue There are a few things that visually separate a confidential issue from a @@ -88,7 +93,7 @@ sees in the project's search results respectively. |:---------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| |  |  | -## Related links +## Related topics - [Merge requests for confidential issues](../merge_requests/confidential.md) - [Make an epic confidential](../../group/epics/manage_epics.md#make-an-epic-confidential) diff --git a/doc/user/project/issues/csv_import.md b/doc/user/project/issues/csv_import.md index 02a4f6a4384..69dc0cbafd8 100644 --- a/doc/user/project/issues/csv_import.md +++ b/doc/user/project/issues/csv_import.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Issues can be imported to a project by uploading a CSV file with the columns `title` and `description`. Other columns are **not** imported. If you want to -retain columns such as labels and milestones, consider the [Move Issue feature](managing_issues.md#moving-issues). +retain columns such as labels and milestones, consider the [Move Issue feature](managing_issues.md#move-an-issue). The user uploading the CSV file is set as the author of the imported issues. @@ -48,7 +48,7 @@ When importing issues from a CSV file, it must be formatted in a certain way: - **double-quote character:** The double-quote (`"`) character is used to quote fields, enabling the use of the column separator within a field (see the third line in the sample CSV data below). To insert a double-quote (`"`) within a quoted - field, use two double-quote characters in succession, i.e. `""`. + field, use two double-quote characters in succession (`""`). - **data rows:** After the header row, succeeding rows must follow the same column order. The issue title is required while the description is optional. diff --git a/doc/user/project/issues/design_management.md b/doc/user/project/issues/design_management.md index 37c00bf0efa..ecf35fc4dcf 100644 --- a/doc/user/project/issues/design_management.md +++ b/doc/user/project/issues/design_management.md @@ -271,4 +271,4 @@ This is rendered as: User activity events on designs (creation, deletion, and updates) are tracked by GitLab and displayed on the [user profile](../../profile/index.md#access-your-user-profile), [group](../../group/index.md#view-group-activity), -and [project](../working_with_projects.md#project-activity) activity pages. +and [project](../working_with_projects.md#view-project-activity) activity pages. diff --git a/doc/user/project/issues/img/button_close_issue_v13_6.png b/doc/user/project/issues/img/button_close_issue_v13_6.png Binary files differdeleted file mode 100644 index 6c7f8da496b..00000000000 --- a/doc/user/project/issues/img/button_close_issue_v13_6.png +++ /dev/null diff --git a/doc/user/project/issues/img/comment-or-discussion.png b/doc/user/project/issues/img/comment-or-discussion.png Binary files differdeleted file mode 100644 index a29014c984c..00000000000 --- a/doc/user/project/issues/img/comment-or-discussion.png +++ /dev/null diff --git a/doc/user/project/issues/img/create_mr_from_issue.png b/doc/user/project/issues/img/create_mr_from_issue.png Binary files differdeleted file mode 100644 index d05a678cd17..00000000000 --- a/doc/user/project/issues/img/create_mr_from_issue.png +++ /dev/null diff --git a/doc/user/project/issues/img/delete_issue_v13_11.png b/doc/user/project/issues/img/delete_issue_v13_11.png Binary files differdeleted file mode 100644 index d9905012eab..00000000000 --- a/doc/user/project/issues/img/delete_issue_v13_11.png +++ /dev/null diff --git a/doc/user/project/issues/img/disable_issue_auto_close.png b/doc/user/project/issues/img/disable_issue_auto_close.png Binary files differdeleted file mode 100644 index 5894d39622a..00000000000 --- a/doc/user/project/issues/img/disable_issue_auto_close.png +++ /dev/null diff --git a/doc/user/project/issues/img/issue_activity_sort_order_v12_10.png b/doc/user/project/issues/img/issue_activity_sort_order_v12_10.png Binary files differdeleted file mode 100644 index 3e19ee1ac66..00000000000 --- a/doc/user/project/issues/img/issue_activity_sort_order_v12_10.png +++ /dev/null diff --git a/doc/user/project/issues/img/issue_type_change_v13_12.png b/doc/user/project/issues/img/issue_type_change_v13_12.png Binary files differdeleted file mode 100644 index 55aa607b878..00000000000 --- a/doc/user/project/issues/img/issue_type_change_v13_12.png +++ /dev/null diff --git a/doc/user/project/issues/img/issues_main_view_numbered.png b/doc/user/project/issues/img/issues_main_view_numbered.png Binary files differdeleted file mode 100644 index 92b9df44972..00000000000 --- a/doc/user/project/issues/img/issues_main_view_numbered.png +++ /dev/null diff --git a/doc/user/project/issues/img/merge_request_closes_issue_v13_11.png b/doc/user/project/issues/img/merge_request_closes_issue_v13_11.png Binary files differdeleted file mode 100644 index 26b42239c12..00000000000 --- a/doc/user/project/issues/img/merge_request_closes_issue_v13_11.png +++ /dev/null diff --git a/doc/user/project/issues/img/new_issue_from_email.png b/doc/user/project/issues/img/new_issue_from_email.png Binary files differdeleted file mode 100644 index 6da899ea37c..00000000000 --- a/doc/user/project/issues/img/new_issue_from_email.png +++ /dev/null diff --git a/doc/user/project/issues/img/new_issue_from_issue_board.png b/doc/user/project/issues/img/new_issue_from_issue_board.png Binary files differdeleted file mode 100644 index 30a1ffb9011..00000000000 --- a/doc/user/project/issues/img/new_issue_from_issue_board.png +++ /dev/null diff --git a/doc/user/project/issues/img/new_issue_from_open_issue_v13_6.png b/doc/user/project/issues/img/new_issue_from_open_issue_v13_6.png Binary files differdeleted file mode 100644 index 902aa40614a..00000000000 --- a/doc/user/project/issues/img/new_issue_from_open_issue_v13_6.png +++ /dev/null diff --git a/doc/user/project/issues/img/new_issue_from_projects_dashboard.png b/doc/user/project/issues/img/new_issue_from_projects_dashboard.png Binary files differdeleted file mode 100644 index 474ca2b45c0..00000000000 --- a/doc/user/project/issues/img/new_issue_from_projects_dashboard.png +++ /dev/null diff --git a/doc/user/project/issues/img/new_issue_from_tracker_list.png b/doc/user/project/issues/img/new_issue_from_tracker_list.png Binary files differdeleted file mode 100644 index 66793cb44fa..00000000000 --- a/doc/user/project/issues/img/new_issue_from_tracker_list.png +++ /dev/null diff --git a/doc/user/project/issues/img/new_issue_v13_1.png b/doc/user/project/issues/img/new_issue_v13_1.png Binary files differdeleted file mode 100644 index a66846c234e..00000000000 --- a/doc/user/project/issues/img/new_issue_v13_1.png +++ /dev/null diff --git a/doc/user/project/issues/img/select_project_from_group_level_issue_tracker_v13_11.png b/doc/user/project/issues/img/select_project_from_group_level_issue_tracker_v13_11.png Binary files differdeleted file mode 100644 index 4612ae254d4..00000000000 --- a/doc/user/project/issues/img/select_project_from_group_level_issue_tracker_v13_11.png +++ /dev/null diff --git a/doc/user/project/issues/img/show-all-activity.png b/doc/user/project/issues/img/show-all-activity.png Binary files differdeleted file mode 100644 index 55c6f5ab5db..00000000000 --- a/doc/user/project/issues/img/show-all-activity.png +++ /dev/null diff --git a/doc/user/project/issues/img/sidebar_move_issue.png b/doc/user/project/issues/img/sidebar_move_issue.png Binary files differdeleted file mode 100644 index 031284a24b2..00000000000 --- a/doc/user/project/issues/img/sidebar_move_issue.png +++ /dev/null diff --git a/doc/user/project/issues/img/similar_issues.png b/doc/user/project/issues/img/similar_issues.png Binary files differdeleted file mode 100644 index c5b7902b2ac..00000000000 --- a/doc/user/project/issues/img/similar_issues.png +++ /dev/null diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index 64838b261ce..bd0cf92e320 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -29,24 +29,25 @@ To learn how the GitLab Strategic Marketing department uses GitLab issues with [ ## Related topics -- [Create issues](managing_issues.md#create-a-new-issue) +- [Create issues](managing_issues.md#create-an-issue) - [Create an issue from a template](../../project/description_templates.md#use-the-templates) -- [Move issues](managing_issues.md#moving-issues) -- [Close issues](managing_issues.md#closing-issues) -- [Delete issues](managing_issues.md#deleting-issues) +- [Edit issues](managing_issues.md#edit-an-issue) +- [Move issues](managing_issues.md#move-an-issue) +- [Close issues](managing_issues.md#close-an-issue) +- [Delete issues](managing_issues.md#delete-an-issue) - [Promote issues](managing_issues.md#promote-an-issue-to-an-epic) - [Set a due date](due_dates.md) - [Import issues](csv_import.md) - [Export issues](csv_export.md) - [Upload designs to issues](design_management.md) - [Linked issues](related_issues.md) +- [Similar issues](managing_issues.md#similar-issues) +- [Health status](managing_issues.md#health-status) - [Cross-link issues](crosslinking_issues.md) -- [Bulk edit issues](../issues/managing_issues.md) - [Sort issue lists](sorting_issue_lists.md) - [Search for issues](../../search/index.md#filter-issue-and-merge-request-lists) - [Epics](../../group/epics/index.md) - [Issue boards](../issue_board.md) - [Issues API](../../../api/issues.md) - [Configure an external issue tracker](../../../integration/external-issue-tracker.md) -- [Parts of an issue](issue_data_and_actions.md) - [Tasks](../../tasks.md) diff --git a/doc/user/project/issues/issue_data_and_actions.md b/doc/user/project/issues/issue_data_and_actions.md index 6bb60e5e31a..66ca29c094e 100644 --- a/doc/user/project/issues/issue_data_and_actions.md +++ b/doc/user/project/issues/issue_data_and_actions.md @@ -1,315 +1,9 @@ --- -stage: Plan -group: Project Management -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2022-02-24' --- -# Issue Data and Actions **(FREE)** +This file was moved to [another location](index.md). -Please read through the [GitLab Issue Documentation](index.md) for an overview on GitLab Issues. - -## Parts of an Issue - -The image below illustrates what an issue may look like. Certain parts -look slightly different or are absent, depending on the GitLab version -and the user's permissions. - -You can find all of an issue's information on one page. - - - -The numbers in the image correspond to the following features: - -- **1.** [Issue actions](#issue-actions) -- **2.** [To Do](#to-do) -- **3.** [Assignee](#assignee) - - **3.1.** [Multiple Assignees](#multiple-assignees) -- **4.** [Epic](#epic) -- **5.** [Milestone](#milestone) -- **6.** [Time tracking](#time-tracking) -- **7.** [Due date](#due-date) -- **8.** [Labels](#labels) -- **9.** [Weight](#weight) -- **10.** [Confidentiality](#confidentiality) -- **11.** [Lock issue](#lock-issue) -- **12.** [Participants](#participants) -- **13.** [Notifications](#notifications) -- **14.** [Reference](#reference) -- [Issue email](#email) -- **15.** [Edit](#edit) -- **16.** [Description](#description) -- **17.** [Mentions](#mentions) -- **18.** [Linked Issues](#linked-issues) -- **19.** [Related Merge Requests](#related-merge-requests) -- **20.** [Award emoji](#award-emoji) -- **21.** [Show all activity](#show-all-activity) -- **22.** [Create Merge Request](#create-merge-request) -- **23.** [Issue history](#issue-history) - - [Activity sort order](#activity-sort-order) -- **24.** [Comments](#comments) -- **25.** [Submit comment, start a thread, or comment and close](#submit-comment-start-a-thread-or-comment-and-close) -- **26.** [Zoom meetings](#zoom-meetings) - -Many of the elements of the issue screen refresh automatically, such as the title and -description, when they are changed by another user. Comments and system notes also -update automatically in response to various actions and content updates. - -### Issue actions - -In an open issue, you can close it by selecting the **Close issue** button. -The issue is marked as closed but is not deleted. - -To reopen a closed issue, select the **Reopen issue** button. -A reopened issue is no different from any other open issue. - -To access additional actions, select the vertical ellipsis -(**{ellipsis_v}**) button: - -- To create a new issue in the same project, select **New issue** in the dropdown menu. - -- If you are not the issue author, you can [submit an abuse report](../../report_abuse.md). - Select **Report abuse** in the dropdown menu. - -### To Do - -You can add issues to and remove issues from your [GitLab To-Do List](../../todos.md). - -The button to do this has a different label depending on whether the issue is already on your To-Do -List or not. If the issue is: - -- Already on your To-Do List: The button is labeled **Mark as done**. Click the button to remove the issue from your To-Do List. -- Not on your To-Do List: The button is labeled **Add a to do**. Click the button to add the issue to your To-Do List. - -### Assignee - -An issue can be assigned to: - -- Yourself. -- Another person. -- [Many people](#multiple-assignees). **(PREMIUM)** - -The assignees can be changed as often as needed. The idea is that the assignees are -responsible for that issue until it's reassigned to someone else to take it from there. -When assigned to someone, it appears in their assigned issues list. - -NOTE: -If a user is not member of that project, it can only be -assigned to them if they created the issue themselves. - -#### Multiple Assignees **(PREMIUM)** - -Often, multiple people work on the same issue together. This can be difficult -to track in large teams where there is shared ownership of an issue. - -To help with this, you can use GitLab to -[assign multiple people](multiple_assignees_for_issues.md) to an issue. - -### Epic **(PREMIUM)** - -You can assign issues to an [Epic](../../group/epics/index.md), which allows better -management of groups of related issues. - -### Milestone - -Select a [milestone](../milestones/index.md) to attribute that issue to. - -### Time tracking - -Use [GitLab Quick Actions](../quick_actions.md) to [track estimates and time -spent on issues](../time_tracking.md). You can add a [time estimate](../time_tracking.md#estimates) -for resolving the issue, and also add [the time spent](../time_tracking.md#time-spent) -to resolve the issue. - -### Due date - -When you work on a tight schedule, it's important to have a way to set a deadline for -implementations and for solving problems. This can be done in the [due date](due_dates.md) -element. Due dates can be changed as many times as needed. - -### Labels - -Categorize issues by giving them [labels](../labels.md). They help to organize workflows, -and they enable you to work with the [issue board](../issue_board.md). - -Group Labels, which allow you to use the same labels for all projects in the same -group, can also be given to issues. They work exactly the same, but are immediately -available to all projects in the group. - -If a label doesn't exist yet, you can create one by clicking **Edit** -followed by **Create new label** in the dropdown menu. - -### Weight **(PREMIUM)** - -[Assign a weight](issue_weight.md) to an issue. -Larger values are used to indicate more effort is required to complete the issue. Only -positive values or zero are allowed. - -### Confidentiality - -You can [set an issue to be confidential](confidential_issues.md). Unauthorized users -cannot access the issue, and it is not listed in the project's issue boards nor list for them. - -### Lock issue - -You can [lock the issue](../../discussions/index.md#prevent-comments-by-locking-an-issue) -to prevent further comments from being added. - -### Participants - -All the users involved in that issue. Either they participated in the [thread](../../discussions/index.md), -or were mentioned in the description or threads. - -### Notifications - -Select the toggle to enable or disable [notifications](../../profile/notifications.md#notifications-on-issues-merge-requests-and-epics) -for the issue. Notifications are automatically enabled after you participate in the issue in any way. - -### Reference - -- A quick "copy" button for that issue's reference, which looks like - `foo/bar#xxx`, where `foo` is the `username` or `groupname`, `bar` is the - `project-name`, and `xxx` is the issue number. - -### Email - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18816) in GitLab 13.8. - -Guest users can see a button in the right sidebar to copy the email address for the issue. -Sending an email to this address creates a comment containing the email body. - -### Edit - -Clicking this icon opens the issue for editing. All the fields which -were shown when the issue was created are displayed for editing. -This icon is only displayed if the user has permission to edit the issue. - -### Description - -The plain text title and description of the issue fill the top center of the issue page. -The description fully supports [GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown), -allowing many formatting options. - -[In GitLab 12.6](https://gitlab.com/gitlab-org/gitlab/-/issues/10103) and later, changes to an -issue's description are listed in the [issue history](#issue-history). **(PREMIUM)** - -### Mentions - -You can mention a user or a group present in your GitLab instance with `@username` or -`@groupname`. All mentioned users are notified via to-do items and emails, -unless they have disabled all [notifications](#notifications) in their user settings. -This is controlled in the [notification settings](../../profile/notifications.md). - -Mentions for yourself (the user currently signed in) are highlighted -in a different color, which allows you to quickly see which comments involve you. - -Avoid mentioning `@all` in issues and merge requests, as it sends an email notification -to all the members of that project's group. This might be interpreted as spam. - -### Linked Issues - -Issues that were mentioned as [linked issues](related_issues.md) are listed here. -You can also click the `+` to add more linked issues. - -### Related Merge Requests - -Merge requests that were mentioned in that issue's description or in the issue thread -are listed as [related merge requests](crosslinking_issues.md#from-merge-requests) here. -Also, if the current issue was mentioned as related in another merge request, that -merge request is also listed here. - -### Award emoji - -You can award emojis to issues. You can select the "thumbs up" and "thumbs down", -or the gray "smiley-face" to choose from the list of available -[GitLab Flavored Markdown Emoji](../../markdown.md#emojis). - -NOTE: -Posting "+1" as a comment in a thread spams all subscribed participants of that issue, -clutters the threads, and is not recommended. Awarding an emoji is a way -to let them know your reaction without notifying them. - -### Show all activity - -You can filter what is displayed in the issue history by clicking on **Show all activity** -and selecting either: - -- **Show comments only**, which only shows threads and hides updates to the issue. -- **Show history only**, which hides threads and only shows updates. - -Also: - -- You can mention a user or a group present in your GitLab instance with - `@username` or `@groupname` and they are notified via to-do items - and emails, unless they have disabled all [notifications](#notifications) - in their user settings. -- Mentions for yourself (the current logged-in user) are highlighted - in a different color, which allows you to quickly see which comments involve you. - - - -### Create Merge Request - -Create a new branch and [**Draft** merge request](../merge_requests/drafts.md) -in one action. The branch is named `issuenumber-title` by default, but you can -choose any name, and GitLab verifies that it is not already in use. The merge request -inherits the milestone and labels of the issue, and is set to automatically -close the issue when it is merged. - - - -Optionally, you can choose to create a [new branch](../repository/web_editor.md#create-a-new-branch-from-an-issue) -only, named after that issue. - -### Issue history - -All comments and updates to the issue are tracked and listed here, but this can be -filtered, as shown above. - -#### Activity sort order - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14588) in GitLab 12.10. - -You can reverse the default order and interact with the activity feed sorted by most recent items -at the top. Your preference is saved via local storage and automatically applied to every issue -you view. - -To change the activity sort order, click the **Oldest first** dropdown menu and select either oldest -or newest items to be shown first. - - - -### Comments - -Collaborate in the issue by posting comments in its thread. This text field also fully -supports [GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown). - -### Submit comment, start a thread, or comment and close - -After you write a comment, you can: - -- Click **Comment** to publish your comment. -- Choose **Start thread** from the dropdown list and start a new [thread](../../discussions/index.md#create-a-thread-without-replying-to-a-comment) - in that issue's main thread to discuss specific points. This invites other participants - to reply directly to your thread, keeping related comments grouped together. - - - -You can also close the issue from here, so you don't need to scroll to the top of the issue page. - -### Zoom meetings - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31103) in GitLab 12.3. - -You can attach and remove Zoom meetings to issues using the `/zoom` and `/remove_zoom` [quick actions](../quick_actions.md) as part of -[GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown). - -Attaching a [Zoom](https://zoom.us) call an issue -results in a **Join Zoom meeting** button at the top of the issue, just under the header. - -Read more how to [add or remove a zoom meeting](associate_zoom_meeting.md). - -### Publish an issue **(ULTIMATE)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/30906) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.1. - -If a status page application is associated with the project, you can use the `/publish` [quick action](../quick_actions.md) to publish the issue. Refer to [GitLab Status Page](../../../operations/incident_management/status_page.md) for more information. +<!-- This redirect file can be deleted after <2022-02-24>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index 6b3d5d6563a..1a23902514a 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -4,293 +4,411 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Managing issues **(FREE)** +# Manage issues **(FREE)** [GitLab Issues](index.md) are the fundamental medium for collaborating on ideas and planning work in GitLab. -Key actions for issues include: +## Create an issue -- [Creating issues](#create-a-new-issue) -- [Moving issues](#moving-issues) -- [Closing issues](#closing-issues) -- [Deleting issues](#deleting-issues) -- [Promoting issues](#promote-an-issue-to-an-epic) **(PREMIUM)** +When you create an issue, you are prompted to enter the fields of the issue. +If you know the values you want to assign to an issue, you can use +[quick actions](../quick_actions.md) to enter them. -## Create a new issue +You can create an issue in many ways in GitLab: -When you create a new issue, you are prompted to fill in the [data and fields of the issue](issue_data_and_actions.md), -as illustrated below. If you know the values you want to assign to an issue, you can use the -[Quick actions](../quick_actions.md) feature to input values. +- [From a project](#from-a-project) +- [From a group](#from-a-group) +- [From another issue](#from-another-issue) +- [From an issue board](#from-an-issue-board) +- [By sending an email](#by-sending-an-email) +- Using a URL with prefilled fields +- [Using Service Desk](#using-service-desk) -While creating an issue, you can associate it to an existing epic from current group by -selecting it using **Epic** dropdown. +### From a project -### Accessing the New Issue form +Prerequisites: -There are many ways to get to the New Issue form from a project's page: +- You must have at least the [Guest role](../../permissions.md) for the project. -- Navigate to your **Project's Dashboard** > **Issues** > **New Issue**: +To create an issue: -  +1. On the top bar, select **Menu > Projects** and find your project. +1. Either: -- From an **open issue** in your project, click the vertical ellipsis (**{ellipsis_v}**) button - to open a dropdown menu, and then click **New Issue** to create a new issue in the same project: + - On the left sidebar, select **Issues**, and then, in the top right corner, select **New issue**. + - On the top bar, select the plus sign (**{plus-square}**) and then, under **This project**, + select **New issue**. -  +1. Complete the [fields](#fields-in-the-new-issue-form). +1. Select **Create issue**. -- From your **Project's Dashboard**, click the plus sign (**+**) to open a dropdown - menu with a few options. Select **New Issue** to create an issue in that project: +The newly created issue opens. -  +### From a group -- From an **issue board**, create a new issue by clicking on the plus sign (**+**) at the top of a list. - It opens a new issue for that project, pre-labeled with its respective list. +Issues belong to projects, but when you're in a group, you can access and create issues that belong +to the projects in the group. -  +Prerequisites: -### Elements of the New Issue form +- You must have at least the [Guest role](../../permissions.md) for the project in the group. -> Ability to add the new issue to an epic [was introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13847) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.1. +To create an issue from a group: - +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Issues**. +1. In the top right corner, select **Select project to create issue**. +1. Select the project you'd like to create an issue for. The button now reflects the selected + project. +1. Select **New issue in `<project name>`**. +1. Complete the [fields](#fields-in-the-new-issue-form). +1. Select **Create issue**. -When you're creating a new issue, these are the fields you can fill in: +The newly created issue opens. -- Title -- Description -- Checkbox to make the issue [confidential](confidential_issues.md) -- Assignee -- Weight -- [Epic](../../group/epics/index.md) -- Due date -- Milestone -- Labels +The project you selected most recently becomes the default for your next visit. +This can save you a lot of time and clicks, if you mostly create issues for the same project. -### New issue from the group-level issue tracker +### From another issue -To visit the issue tracker for all projects in your group: +> New issue becoming linked to the issue of origin [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68226) in GitLab 14.3. -1. Go to the group dashboard. -1. On the left sidebar, select **Issues**. -1. In the top-right, select the **Select project to create issue** button. -1. Select the project you'd like to create an issue for. The button now reflects the selected - project. -1. Select the button to create an issue in the selected project. +You can create a new issue from an existing one. The two issues can then be marked as related. - +Prerequisites: -The project you selected most recently becomes the default for your next visit. -This should save you a lot of time and clicks, if you mostly create issues for the same project. +- You must have at least the [Guest role](../../permissions.md) for the project. -### New issue via Service Desk +To create an issue from another issue: -Enable [Service Desk](../service_desk.md) for your project and offer email support. -Now, when your customer sends a new email, a new issue can be created in -the appropriate project and followed up from there. +1. In an existing issue, select the vertical ellipsis (**{ellipsis_v}**). +1. Select **New issue**. +1. Complete the [fields](#fields-in-the-new-issue-form). + The new issue's description is prefilled with `Related to #123`, where `123` is the ID of the + issue of origin. If you keep this mention in the description, the two issues become + [linked](related_issues.md). +1. Select **Create issue**. -### New issue via email +The newly created issue opens. -A link to **Email a new issue to this project** is displayed at the bottom of a project's -**Issues List** page. The link is shown only if your GitLab instance has [incoming email](../../../administration/incoming_email.md) -configured and there is at least one issue in the issue list. +### From an issue board - +You can create a new issue from an [issue board](../issue_board.md). -When you click this link, an email address is generated and displayed, which should be used -by **you only**, to create issues in this project. You can save this address as a -contact in your email client for quick access. +Prerequisites: -WARNING: -This is a private email address, generated just for you. **Keep it to yourself**, -as anyone who knows it can create issues or merge requests as if they -were you. If the address is compromised, or you want to regenerate it, -click **Email a new issue to this project**, followed by **reset it**. +- You must have at least the [Guest role](../../permissions.md) for the project. + +To create an issue from a project issue board: + +1. On the top bar, select **Menu > Projects** and find your project. +1. Select **Issues > Boards**. +1. At the top of a board list, select **New issue** (**{plus-square}**). +1. Enter the issue's title. +1. Select **Create issue**. + +To create an issue from a group issue board: + +1. On the top bar, select **Menu > Groups** and find your group. +1. Select **Issues > Boards**. +1. At the top of a board list, select **New issue** (**{plus-square}**). +1. Enter the issue's title. +1. Under **Projects**, select the project in the group that the issue should belong to. +1. Select **Create issue**. -Sending an email to this address creates a new issue associated with your account for -this project, where: +The issue is created and shows up in the board list. It shares the list's characteristic, so, for +example, if the list is scoped to a label `Frontend`, the new issue also has this label. -- The email subject becomes the issue title. -- The email body becomes the issue description. -- [Markdown](../../markdown.md) and [quick actions](../quick_actions.md) are supported. +### By sending an email -NOTE: -In GitLab 11.7, we updated the format of the generated email address. However the -older format is still supported, allowing existing aliases or contacts to continue working. +> Generated email address format changed in GitLab 11.7. +> The older format is still supported, so existing aliases and contacts still work. -### New issue via URL with prefilled fields +You can send an email to create an issue in a project on the project's +**Issues List** page. + +Prerequisites: + +- Your GitLab instance must have [incoming email](../../../administration/incoming_email.md) + configured. +- There must be at least one issue in the issue list. +- You must have at least the [Guest role](../../permissions.md) for the project. + +To email an issue to a project: + +1. On the top bar, select **Menu > Projects** and find your project. +1. Select **Issues**. +1. At the bottom of the page, select **Email a new issue to this project**. +1. To copy the email address, select **Copy** (**{copy-to-clipboard}**). +1. From your email client, send an email to this address. + The subject is used as the title of the new issue, and the email body becomes the description. + You can use [Markdown](../../markdown.md) and [quick actions](../quick_actions.md). + +A new issue is created, with your user as the author. +You can save this address as a contact in your email client to use it again. + +WARNING: +The email address you see is a private email address, generated just for you. +**Keep it to yourself**, because anyone who knows it can create issues or merge requests as if they +were you. + +To regenerate the email address: + +1. On the issues list, select **Email a new issue to this project**. +1. Select **reset this token**. + +### Using a URL with prefilled values To link directly to the new issue page with prefilled fields, use query string parameters in a URL. You can embed a URL in an external -HTML page to create issues with certain -fields prefilled. +HTML page to create issues with certain fields prefilled. + +| Field | URL parameter | Notes | +| -------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------- | +| Title | `issue[title]` | Must be [URL-encoded](../../../api/index.md#namespaced-path-encoding). | +| Issue type | `issue[issue_type]` | Either `incident` or `issue`. | +| Description template | `issuable_template` | Cannot be used at the same time as `issue[description]`. Must be [URL-encoded](../../../api/index.md#namespaced-path-encoding). | +| Description | `issue[description]` | Cannot be used at the same time as `issuable_template`. Must be [URL-encoded](../../../api/index.md#namespaced-path-encoding). | +| Confidential | `issue[confidential]` | If `true`, the issue is marked as confidential. | + +Adapt these examples to form your new issue URL with prefilled fields. +To create an issue in the GitLab project: + +- With a prefilled title and description: + + ```plaintext + https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue[title]=Whoa%2C%20we%27re%20half-way%20there&issue[description]=Whoa%2C%20livin%27%20in%20a%20URL + ``` -| Field | URL Parameter Name | Notes | -|----------------------|-----------------------|-------------------------------------------------------| -| title | `issue[title]` | | -| description | `issue[description]` | Cannot be used at the same time as `issuable_template`. | -| description template | `issuable_template` | Cannot be used at the same time as `issue[description]`. | -| issue type | `issue[issue_type]` | Either `incident` or `issue`. | -| confidential | `issue[confidential]` | Parameter value must be `true` to set to confidential. | +- With a prefilled title and description template: -Follow these examples to form your new issue URL with prefilled fields. + ```plaintext + https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue[title]=Validate%20new%20concept&issuable_template=Feature%20Proposal%20-%20basic + ``` -- For a new issue in the GitLab Community Edition project with a pre-filled title - and a pre-filled description, the URL would be `https://gitlab.com/gitlab-org/gitlab-foss/-/issues/new?issue[title]=Validate%20new%20concept&issue[description]=Research%20idea` -- For a new issue in the GitLab Community Edition project with a pre-filled title - and a pre-filled description template, the URL would be `https://gitlab.com/gitlab-org/gitlab-foss/-/issues/new?issue[title]=Validate%20new%20concept&issuable_template=Research%20proposal` -- For a new issue in the GitLab Community Edition project with a pre-filled title, - a pre-filled description, and the confidential flag set, the URL would be `https://gitlab.com/gitlab-org/gitlab-foss/-/issues/new?issue[title]=Validate%20new%20concept&issue[description]=Research%20idea&issue[confidential]=true` +- With a prefilled title, description, and marked as confidential: -## Bulk edit issues at the project level + ```plaintext + https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue[title]=Validate%20new%20concept&issue[description]=Research%20idea&issue[confidential]=true + ``` -> - Assigning epic ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210470) in GitLab 13.2. +### Using Service Desk + +To offer email support, enable [Service Desk](../service_desk.md) for your project. + +Now, when your customer sends a new email, a new issue can be created in +the appropriate project and followed up from there. + +### Fields in the new issue form + +> Adding the new issue to an epic [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13847) in GitLab 13.1. + +When you're creating a new issue, you can complete the following fields: + +- Title +- Type: either issue (default) or incident +- [Description template](../description_templates.md): overwrites anything in the Description text box +- Description: you can use [Markdown](../../markdown.md) and [quick actions](../quick_actions.md) +- Checkbox to make the issue [confidential](confidential_issues.md) +- [Assignees](#assignee) +- [Weight](issue_weight.md) +- [Epic](../../group/epics/index.md) +- [Due date](due_dates.md) +- [Milestone](../milestones/index.md) +- [Labels](../labels.md) + +## Edit an issue + +You can edit an issue's title and description. + +Prerequisites: + +- You must have at least the [Reporter role](../../permissions.md) for the project. + +To edit an issue: + +1. To the right of the title, select **Edit title and description** (**{pencil}**). +1. Edit the available fields. +1. Select **Save changes**. + +### Bulk edit issues from a project + +> - Assigning epic [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210470) in GitLab 13.2. > - Editing health status [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218395) in GitLab 13.2. > - Editing iteration [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196806) in GitLab 13.9. -Users with permission level of [Reporter or higher](../../permissions.md) can manage issues. +You can edit multiple issues at a time when you're in a project. + +Prerequisites: + +- You must have at least the [Reporter role](../../permissions.md) for the project. + +To edit multiple issues at the same time: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Issues**. +1. Select **Edit issues**. A sidebar on the right of your screen appears. +1. Select the checkboxes next to each issue you want to edit. +1. From the sidebar, edit the available fields. +1. Select **Update all**. When bulk editing issues in a project, you can edit the following attributes: -- Status (open/closed) -- Assignee +- Status (open or closed) +- [Assignees](#assignee) - [Epic](../../group/epics/index.md) - [Milestone](../milestones/index.md) - [Labels](../labels.md) - [Health status](#health-status) -- Notification subscription +- [Notification](../../profile/notifications.md) subscription - [Iteration](../../group/iterations/index.md) -To update multiple project issues at the same time: - -1. In a project, go to **Issues > List**. -1. Click **Edit issues**. A sidebar on the right-hand side of your screen appears with editable fields. -1. Select the checkboxes next to each issue you want to edit. -1. Select the appropriate fields and their values from the sidebar. -1. Click **Update all**. - -## Bulk edit issues at the group level +### Bulk edit issues from a group > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7249) in GitLab 12.1. -> - Assigning epic ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210470) in GitLab 13.2. +> - Assigning epic [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210470) in GitLab 13.2. > - Editing health status [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218395) in GitLab 13.2. > - Editing iteration [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196806) in GitLab 13.9. -Users with permission level of [Reporter or higher](../../permissions.md) can manage issues. +You can edit multiple issues across multiple projects when you're in a group. + +Prerequisites: + +- You must have at least the [Reporter role](../../permissions.md) for a group. + +To edit multiple issues at the same time: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Issues**. +1. Select **Edit issues**. A sidebar on the right of your screen appears. +1. Select the checkboxes next to each issue you want to edit. +1. From the sidebar, edit the available fields. +1. Select **Update all**. When bulk editing issues in a group, you can edit the following attributes: - [Epic](../../group/epics/index.md) - [Milestone](../milestones/index.md) +- [Iteration](../../group/iterations/index.md) - [Labels](../labels.md) - [Health status](#health-status) -- [Iteration](../../group/iterations/index.md) -To update multiple project issues at the same time: +## Move an issue -1. In a group, go to **Issues > List**. -1. Click **Edit issues**. A sidebar on the right-hand side of your screen appears with editable fields. -1. Select the checkboxes next to each issue you want to edit. -1. Select the appropriate fields and their values from the sidebar. -1. Click **Update all**. - -## Moving issues - -Moving an issue copies it to the target project, and closes it in the originating project. +When you move an issue, it's closed and copied to the target project. The original issue is not deleted. A system note, which indicates where it came from and went to, is added to both issues. -The "Move issue" button is at the bottom of the right-sidebar when viewing the issue. +Prerequisites: - +- You must have at least the [Reporter role](../../permissions.md) for the project. -### Moving issues in bulk **(FREE SELF)** +To move an issue: -If you have advanced technical skills you can also bulk move all the issues from -one project to another in the rails console. The below script moves all issues -that are not in status **closed** from one project to another. +1. Go to the issue. +1. On the right sidebar, select **Move issue**. +1. Search for a project to move the issue to. +1. Select **Move**. -To access rails console run `sudo gitlab-rails console` on the GitLab server and run the below -script. Please be sure to change `project`, `admin_user`, and `target_project` to your values. -We do also recommend [creating a backup](../../../raketasks/backup_restore.md) before -attempting any changes in the console. +### Bulk move issues **(FREE SELF)** -```ruby -project = Project.find_by_full_path('full path of the project where issues are moved from') -issues = project.issues -admin_user = User.find_by_username('username of admin user') # make sure user has permissions to move the issues -target_project = Project.find_by_full_path('full path of target project where issues moved to') +You can move all open issues from one project to another. -issues.each do |issue| - if issue.state != "closed" && issue.moved_to.nil? - Issues::MoveService.new(project, admin_user).execute(issue, target_project) - else - puts "issue with id: #{issue.id} and title: #{issue.title} was not moved" - end -end; nil -``` +Prerequisites: + +- You must have at least the [Reporter role](../../permissions.md) for the project. + +To do it: + +1. Optional (but recommended). [Create a backup](../../../raketasks/backup_restore.md) before + attempting any changes in the console. +1. Open the [Rails console](../../../administration/operations/rails_console.md). +1. Run the following script. Make sure to change `project`, `admin_user`, and `target_project` to + your values. + + ```ruby + project = Project.find_by_full_path('full path of the project where issues are moved from') + issues = project.issues + admin_user = User.find_by_username('username of admin user') # make sure user has permissions to move the issues + target_project = Project.find_by_full_path('full path of target project where issues moved to') + + issues.each do |issue| + if issue.state != "closed" && issue.moved_to.nil? + Issues::MoveService.new(project, admin_user).execute(issue, target_project) + else + puts "issue with id: #{issue.id} and title: #{issue.title} was not moved" + end + end; nil + ``` + +1. To exit the Rails console, enter `quit`. -## Closing issues +## Close an issue -When you decide that an issue is resolved, or no longer needed, you can close the issue -using the close button: +When you decide that an issue is resolved or no longer needed, you can close it. +The issue is marked as closed but is not deleted. - +Prerequisites: -You can also close an issue from the [issue boards](../issue_board.md) by dragging an issue card -from its list and dropping it into the **Closed** list. +- You must have at least the [Reporter role](../../permissions.md) for the project. - +To close an issue, you can do the following: + +- At the top of the issue, select **Close issue**. +- In an [issue board](../issue_board.md), drag an issue card from its list into the **Closed** list. + +  + +### Reopen a closed issue + +Prerequisites: + +- You must have at least the [Reporter role](../../permissions.md) for the project. + +To reopen a closed issue, at the top of the issue, select **Reopen issue**. +A reopened issue is no different from any other open issue. ### Closing issues automatically -When a commit or merge request resolves issues, the issues -can be closed automatically when the commit reaches the project's default branch. +You can close issues automatically by using certain words in the commit message or MR description. -If a commit message or merge request description contains text matching a [defined pattern](#default-closing-pattern), -all issues referenced in the matched text are closed. This happens when the commit -is pushed to a project's [**default** branch](../repository/branches/default.md), -or when a commit or merge request is merged into it. +If a commit message or merge request description contains text matching the [defined pattern](#default-closing-pattern), +all issues referenced in the matched text are closed when either: -For example, if `Closes #4, #6, Related to #5` is included in a Merge Request -description, issues `#4` and `#6` are closed automatically when the MR is merged, but not `#5`. -Using `Related to` flags `#5` as a [related issue](related_issues.md), -but is not closed automatically. +- The commit is pushed to a project's [**default** branch](../repository/branches/default.md). +- The commit or merge request is merged into the default branch. - +For example, if you include `Closes #4, #6, Related to #5` in a merge request +description: -If the issue is in a different repository than the MR, add the full URL for the issue(s): +- Issues `#4` and `#6` are closed automatically when the MR is merged. +- Issue `#5` is marked as a [related issue](related_issues.md), but it's not closed automatically. -```markdown -Closes #4, #6, and https://gitlab.com/<username>/<projectname>/issues/<xxx> -``` +Alternatively, when you [create a merge request from an issue](../merge_requests/getting_started.md#merge-requests-to-close-issues), +it inherits the issue's milestone and labels. For performance reasons, automatic issue closing is disabled for the very first push from an existing repository. #### Default closing pattern -When not specified, this default issue closing pattern is used: +To automatically close an issue, use the following keywords followed by the issue reference. -```shell -\b((?:[Cc]los(?:e[sd]?|ing)|\b[Ff]ix(?:e[sd]|ing)?|\b[Rr]esolv(?:e[sd]?|ing)|\b[Ii]mplement(?:s|ed|ing)?)(:?) +(?:(?:issues? +)?%{issue_ref}(?:(?: *,? +and +| *,? *)?)|([A-Z][A-Z0-9_]+-\d+))+) -``` - -This translates to the following keywords: +Available keywords: - Close, Closes, Closed, Closing, close, closes, closed, closing - Fix, Fixes, Fixed, Fixing, fix, fixes, fixed, fixing - Resolve, Resolves, Resolved, Resolving, resolve, resolves, resolved, resolving - Implement, Implements, Implemented, Implementing, implement, implements, implemented, implementing -Note that `%{issue_ref}` is a complex regular expression defined inside the GitLab -source code that can match references to: +Available issue reference formats: - A local issue (`#123`). - A cross-project issue (`group/project#123`). -- A link to an issue (`https://gitlab.example.com/group/project/issues/123`). +- The full URL of an issue (`https://gitlab.example.com/group/project/issues/123`). -For example the following commit message: +For example: ```plaintext Awesome commit message @@ -300,54 +418,93 @@ This commit is also related to #17 and fixes #18, #19 and https://gitlab.example.com/group/otherproject/issues/23. ``` -closes `#18`, `#19`, `#20`, and `#21` in the project this commit is pushed to, +The previous commit message closes `#18`, `#19`, `#20`, and `#21` in the project this commit is pushed to, as well as `#22` and `#23` in `group/otherproject`. `#17` is not closed as it does -not match the pattern. It works with multi-line commit messages as well as one-liners -when used from the command line with `git commit -m`. +not match the pattern. + +You can use the closing patterns in multi-line commit messages or one-liners +done from the command line with `git commit -m`. -#### Disabling automatic issue closing +The default issue closing pattern regex: + +```shell +\b((?:[Cc]los(?:e[sd]?|ing)|\b[Ff]ix(?:e[sd]|ing)?|\b[Rr]esolv(?:e[sd]?|ing)|\b[Ii]mplement(?:s|ed|ing)?)(:?) +(?:(?:issues? +)?%{issue_ref}(?:(?: *,? +and +| *,? *)?)|([A-Z][A-Z0-9_]+-\d+))+) +``` + +#### Disable automatic issue closing > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19754) in GitLab 12.7. -The automatic issue closing feature can be disabled on a per-project basis -in the [project's repository settings](../settings/index.md). Referenced -issues are still displayed, but are not closed automatically. +You can disable the automatic issue closing feature on a per-project basis +in the [project's settings](../settings/index.md). + +Prerequisites: - +- You must have at least the [Maintainer role](../../permissions.md) for the project. -The automatic issue closing is also disabled in a project if the project has the issue tracker +To disable automatic issue closing: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Default branch**. +1. Select **Auto-close referenced issues on default branch**. +1. Select **Save changes**. + +Referenced issues are still displayed, but are not closed automatically. + +The automatic issue closing is disabled by default in a project if the project has the issue tracker disabled. If you want to enable automatic issue closing, make sure to [enable GitLab Issues](../settings/index.md#sharing-and-permissions). -This only applies to issues affected by new merge requests or commits. Already -closed issues remain as-is. +Changing this setting applies only to new merge requests or commits. Already +closed issues remain as they are. If issue tracking is enabled, disabling automatic issue closing only applies to merge requests -attempting to automatically close issues within the same project. +attempting to automatically close issues in the same project. Merge requests in other projects can still close another project's issues. -#### Customizing the issue closing pattern **(FREE SELF)** +#### Customize the issue closing pattern **(FREE SELF)** + +Prerequisites: -In order to change the default issue closing pattern, GitLab administrators must edit the +- You must have the [administrator access level](../../../administration/index.md) for your GitLab instance. + +To change the default issue closing pattern, edit the [`gitlab.rb` or `gitlab.yml` file](../../../administration/issue_closing_pattern.md) of your installation. ## Change the issue type -Users with the [Developer role](../../permissions.md) -can change an issue's type. To do this, edit the issue and select an issue type from the -**Issue type** selector menu: +Prerequisites: + +- You must be the issue author or have at least the [Reporter role](../../permissions.md) for the project. + +To change issue type: + +1. To the right of the title, select **Edit title and description** (**{pencil}**). +1. Edit the issue and select an issue type from the **Issue type** dropdown list: -- [Issue](index.md) -- [Incident](../../../operations/incident_management/index.md) + - Issue + - [Incident](../../../operations/incident_management/index.md) - +1. Select **Save changes**. -## Deleting issues +## Delete an issue -Users with the [Owner role](../../permissions.md) can delete an issue by -editing it and selecting **Delete issue**. +> Deleting from the vertical ellipsis menu [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299933) in GitLab 14.6. - +Prerequisites: + +- You must have the [Owner role](../../permissions.md) for a project. + +To delete an issue: + +1. In an issue, select the vertical ellipsis (**{ellipsis_v}**). +1. Select **Delete issue**. + +Alternatively: + +1. In an issue, select **Edit title and description** (**{pencil}**). +1. Select **Delete issue**. ## Promote an issue to an epic **(PREMIUM)** @@ -355,16 +512,16 @@ editing it and selecting **Delete issue**. > - Moved to GitLab Premium in 12.8. > - Promoting issues to epics via the UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233974) in GitLab Premium 13.6. -You can promote an issue to an epic in the immediate parent group. +You can promote an issue to an [epic](../../group/epics/index.md) in the immediate parent group. To promote an issue to an epic: -1. In an issue, select the vertical ellipsis (**{ellipsis_v}**) button. +1. In an issue, select the vertical ellipsis (**{ellipsis_v}**). 1. Select **Promote to epic**. Alternatively, you can use the `/promote` [quick action](../quick_actions.md#issues-merge-requests-and-epics). -Read more about promoting an issue to an epic on the [Manage epics page](../../group/epics/manage_epics.md#promote-an-issue-to-an-epic). +Read more about [promoting an issues to epics](../../group/epics/manage_epics.md#promote-an-issue-to-an-epic). ## Add an issue to an iteration **(PREMIUM)** @@ -373,12 +530,41 @@ Read more about promoting an issue to an epic on the [Manage epics page](../../g To add an issue to an [iteration](../../group/iterations/index.md): -1. In an issue sidebar, click **Edit** next to **Iteration**. A dropdown appears. -1. Click an iteration you'd like to associate this issue with. +1. Go to the issue. +1. On the right sidebar, in the **Iteration** section, select **Edit**. +1. From the dropdown list, select the iteration to associate this issue with. +1. Select any area outside the dropdown list. + +Alternatively, you can use the `/iteration` [quick action](../quick_actions.md#issues-merge-requests-and-epics). + +## Copy issue reference + +To refer to an issue elsewhere in GitLab, you can use its full URL or a short reference, which looks like +`namespace/project-name#123`, where `namespace` is either a group or a username. + +To copy the issue reference to your clipboard: -You can also use the `/iteration` -[quick action](../quick_actions.md#issues-merge-requests-and-epics) -in a comment or description field. +1. Go to the issue. +1. On the right sidebar, next to **Reference**, select **Copy Reference** (**{copy-to-clipboard}**). + +You can now paste the reference into another description or comment. + +Read more about issue references in [GitLab-Flavored Markdown](../../markdown.md#gitlab-specific-references). + +## Copy issue email address + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18816) in GitLab 13.8. + +You can create a comment in an issue by sending an email. +Sending an email to this address creates a comment that contains the email body. + +To learn more about creating comments by sending an email and the necessary configuration, see +[Reply to a comment by sending email](../../discussions/index.md#reply-to-a-comment-by-sending-email). + +To copy the issue's email address: + +1. Go to the issue. +1. On the right sidebar, next to **Issue email**, select **Copy Reference** (**{copy-to-clipboard}**). ## Real-time sidebar @@ -393,35 +579,84 @@ On GitLab.com, this feature is available. Assignees in the sidebar are updated in real time. +## Assignee + +An issue can be assigned to one or [more users](multiple_assignees_for_issues.md). + +The assignees can be changed as often as needed. The idea is that the assignees are +people responsible for an issue. +When an issue is assigned to someone, it appears in their assigned issues list. + +If a user is not a member of a project, an issue can only be assigned to them if they create it +themselves or another project member assigns them. + +To change the assignee on an issue: + +1. Go to your issue. +1. On the right sidebar, in the **Assignee** section, select **Edit**. +1. From the dropdown list, select the user to add as an assignee. +1. Select any area outside the dropdown list. + ## Similar issues -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22866) in GitLab 11.6. +To prevent duplication of issues on the same topic, GitLab searches for similar issues +when you create a new issue. -To prevent duplication of issues for the same topic, GitLab searches for similar issues -when new issues are being created. +Prerequisites: -As you type in the title field of the **New Issue** page, GitLab searches titles and descriptions -across all issues to in the current project. Only issues you have access to are returned. -Up to five similar issues, sorted by most recently updated, are displayed below the title box. -[GraphQL](../../../api/graphql/index.md) must be enabled to use this feature. +- [GraphQL](../../../api/graphql/index.md) must be enabled. - +As you type in the title text box of the **New issue** page, GitLab searches titles and descriptions +across all issues in the current project. Only issues you have access to are returned. +Up to five similar issues, sorted by most recently updated, are displayed below the title text box. ## Health status **(ULTIMATE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36427) in GitLab Ultimate 12.10. -> - Health status of closed issues [can't be edited](https://gitlab.com/gitlab-org/gitlab/-/issues/220867) in GitLab Ultimate 13.4 and later. -> - Issue health status visible in issue lists [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45141) in GitLab Ultimate 13.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36427) in GitLab 12.10. +> - Health status of closed issues [can't be edited](https://gitlab.com/gitlab-org/gitlab/-/issues/220867) in GitLab 13.4 and later. +> - Issue health status visible in issue lists [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45141) in GitLab 13.6. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/213567) in GitLab 13.7. To help you track issue statuses, you can assign a status to each issue. -This marks issues as progressing as planned or needs attention to keep on schedule: +This status marks issues as progressing as planned or needing attention to keep on schedule. + +Prerequisites: + +- You must have at least the [Reporter role](../../permissions.md) for the project. + +To edit health status of an issue: + +1. Go to the issue. +1. On the right sidebar, in the **Health status** section, select **Edit**. +1. From the dropdown list, select the status to add to this issue: -- On track (green) -- Needs attention (amber) -- At risk (red) + - On track (green) + - Needs attention (amber) + - At risk (red) + +You can then see the issue's status in the issues list and the epic tree. After an issue is closed, its health status can't be edited and the **Edit** button becomes disabled until the issue is reopened. -You can then see issue statuses in the issues list and the epic tree. +## Publish an issue **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/30906) in GitLab 13.1. + +If a status page application is associated with the project, you can use the `/publish` +[quick action](../quick_actions.md) to publish the issue. + +For more information, see [GitLab Status Page](../../../operations/incident_management/status_page.md). + +## Issue-related quick actions + +You can also use [quick actions](../quick_actions.md#issues-merge-requests-and-epics) to manage issues. + +Some actions don't have corresponding UI buttons yet. +You can do the following **only by using quick actions**: + +- [Add or remove a Zoom meeting](associate_zoom_meeting.md) (`/zoom` and `/remove_zoom`). +- [Publish an issue](#publish-an-issue) (`/publish`). +- Clone an issue to the same or another project (`/clone`). +- Close an issue and mark as a duplicate of another issue (`/duplicate`). +- Copy labels and milestone from another merge request in the project (`/copy_metadata`). diff --git a/doc/user/project/issues/multiple_assignees_for_issues.md b/doc/user/project/issues/multiple_assignees_for_issues.md index 189777d40e7..98e940b6b51 100644 --- a/doc/user/project/issues/multiple_assignees_for_issues.md +++ b/doc/user/project/issues/multiple_assignees_for_issues.md @@ -12,8 +12,7 @@ In large teams, where there is shared ownership of an issue, it can be difficult to track who is working on it, who already completed their contributions, who didn't even start yet. -In [GitLab Enterprise Edition](https://about.gitlab.com/pricing/), -you can also select multiple assignees to an issue, making it easier to +You can also select multiple [assignees](managing_issues.md#assignee) for an issue, making it easier to track, and making clearer who is accountable for it.  diff --git a/doc/user/project/issues/related_issues.md b/doc/user/project/issues/related_issues.md index e4972181a5a..8a2a104c54d 100644 --- a/doc/user/project/issues/related_issues.md +++ b/doc/user/project/issues/related_issues.md @@ -26,7 +26,7 @@ To manage linked issues through our API, visit the [issue links API documentatio 1. Link one issue to another by selecting the add linked issue button (**{plus}**) in the **Linked issues** section of an issue. -1. Select the relationship the between the two issues. Either: +1. Select the relationship between the two issues. Either: - **relates to** - **blocks** **(PREMIUM)** - **is blocked by** **(PREMIUM)** diff --git a/doc/user/project/issues/sorting_issue_lists.md b/doc/user/project/issues/sorting_issue_lists.md index ebfc723280f..0340f15c25c 100644 --- a/doc/user/project/issues/sorting_issue_lists.md +++ b/doc/user/project/issues/sorting_issue_lists.md @@ -37,7 +37,7 @@ creation date. Issues created most recently are first. ## Sorting by due date When you sort by **Due date**, the issue list changes to sort ascending by the issue -[due date](issue_data_and_actions.md#due-date). Issues with the earliest due date are first, +[due date](due_dates.md). Issues with the earliest due date are first, and issues without a due date are last. ## Sorting by label priority diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index e9cbe012110..8874512f9c3 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -16,7 +16,7 @@ Labels are a key part of [issue boards](issue_board.md). With labels you can: - Categorize epics, issues, and merge requests using colors and descriptive titles like `bug`, `feature request`, or `docs`. - Dynamically filter and manage epics, issues, and merge requests. -- [Search lists of issues, merge requests, and epics](../search/index.md#issues-and-merge-requests), +- [Search lists of issues, merge requests, and epics](../search/index.md#search-issues-and-merge-requests), as well as [issue boards](../search/index.md#issue-boards). ## Project labels and group labels @@ -42,8 +42,8 @@ To assign or unassign a label: You can search repeatedly to add more labels. The selected labels are marked with a checkmark. 1. Click the labels you want to assign or unassign. -1. To apply your changes to labels, click **X** next to **Assign labels** or anywhere outside the - label section. +1. To apply your changes to labels, select **X** next to **Assign labels** or select any area + outside the label section. Alternatively, to unassign a label, click the **X** on the label you want to unassign. @@ -72,9 +72,9 @@ To create a new project label: 1. Select the **New label** button. 1. In the **Title** field, enter a short, descriptive name for the label. You can also use this field to create [scoped, mutually exclusive labels](#scoped-labels). -1. (Optional) In the **Description** field, you can enter additional +1. Optional. In the **Description** field, you can enter additional information about how and when to use this label. -1. (Optional) Select a background color for the label by selecting one of the +1. Optional. Select a background color for the label by selecting one of the available colors, or by entering a hex color value in the **Background color** field. 1. Select **Create label**. @@ -86,7 +86,7 @@ label section of the right sidebar of an issue or a merge request: 1. Click **Create project label**. - Fill in the name field. Note that you can't specify a description if creating a label this way. You can add a description later by editing the label (see below). - - (Optional) Select a color by clicking on the available colors, or input a hex + - Optional. Select a color by clicking on the available colors, or input a hex color value for a specific color. 1. Click **Create**. @@ -189,7 +189,7 @@ For example, filtering by the `platform::*` label returns issues that have `plat `platform::Android`, or `platform::Linux` labels. NOTE: -This is not available on the [issues or merge requests dashboard pages](../search/index.md#issues-and-merge-requests). +This is not available on the [issues or merge requests dashboard pages](../search/index.md#search-issues-and-merge-requests). ### Workflows with scoped labels diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index adf0a115c6e..283576fb4e9 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -52,7 +52,8 @@ Prerequisite: To add groups to a project: -1. Go to your project and select **Project information > Members**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Project information > Members**. 1. On the **Invite group** tab, under **Select a group to invite**, choose a group. 1. Select the highest max [role](../../permissions.md) for users in the group. 1. Optional. Choose an expiration date. On that date, the user can no longer access the project. @@ -75,7 +76,8 @@ Prerequisite: To import users: -1. Go to your project and select **Project information > Members**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Project information > Members**. 1. On the **Invite member** tab, at the bottom of the panel, select **Import**. 1. Select the project. You can view only the projects for which you're a maintainer. 1. Select **Import project members**. @@ -115,7 +117,8 @@ Prerequisites: To remove a member from a project: -1. Go to your project and select **Project information > Members**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Project information > Members**. 1. Next to the project member you want to remove, select **Remove member** **{remove}**. 1. Optional. In the confirmation box, select the **Also unassign this user from related issues and merge requests** checkbox. @@ -136,7 +139,8 @@ You can filter and sort members in a project. ### Display inherited members -1. Go to your project and select **Project information > Members**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Project information > Members**. 1. In the **Filter members** box, select `Membership` `=` `Inherited`. 1. Press Enter. @@ -144,7 +148,8 @@ You can filter and sort members in a project. ### Display direct members -1. Go to your project and select **Project information > Members**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Project information > Members**. 1. In the **Filter members** box, select `Membership` `=` `Direct`. 1. Press Enter. @@ -166,7 +171,7 @@ You can sort members by **Account**, **Access granted**, **Max role**, or **Last GitLab users can request to become a member of a project. -1. Go to the project you'd like to be a member of. +1. On the top bar, select **Menu > Projects** and find the project you want to be a member of. 1. By the project name, select **Request Access**.  @@ -189,8 +194,9 @@ Prerequisite: - You must be the project owner. -1. Go to the project and select **Settings > General**. -1. Expand the **Visibility, project features, permissions** section. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Visibility, project features, permissions**. 1. Under **Project visibility**, select **Users can request access**. 1. Select **Save changes**. @@ -213,7 +219,8 @@ This feature might not be available to you. Check the **version history** note a In GitLab 13.11, you can optionally replace the form to add a member with a modal window. To add a member after enabling this feature: -1. Go to your project and select **Project information > Members**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Project information > Members**. 1. Select **Invite members**. 1. Enter an email address and select a role. 1. Optional. Select an **Access expiration date**. diff --git a/doc/user/project/members/share_project_with_groups.md b/doc/user/project/members/share_project_with_groups.md index abca140411a..4c96c4d9f56 100644 --- a/doc/user/project/members/share_project_with_groups.md +++ b/doc/user/project/members/share_project_with_groups.md @@ -4,7 +4,7 @@ group: Workspace info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Share projects with other groups +# Share projects with other groups **(FREE)** You can share projects with other [groups](../../group/index.md). This makes it possible to add a group of users to a project with a single action. @@ -30,18 +30,20 @@ To share 'Project Acme' with the 'Engineering' group: 1. For 'Project Acme' use the left navigation menu to go to **Project information > Members**. 1. Select the **Invite group** tab. 1. Add the 'Engineering' group with the maximum access level of your choice. -1. Optionally, select an expiring date. -1. Click **Invite**. -1. After sharing 'Project Acme' with 'Engineering': - - The group is listed in the **Groups** tab. - - The project is listed on the group dashboard. +1. Optional. Select an expiration date. +1. Select **Invite**. + +After sharing 'Project Acme' with 'Engineering': + +- The group is listed in the **Groups** tab. +- The project is listed on the group dashboard. -Note that you can only share a project with: +You can share a project only with: -- groups for which you have an explicitly defined membership -- groups that contain a nested subgroup or project for which you have an explicitly defined role +- Groups for which you have an explicitly defined membership. +- Groups that contain a nested subgroup or project for which you have an explicitly defined role. -Administrators are able to share projects with any group in the system. +Administrators can share projects with any group in the system. ### Share a project modal window @@ -61,7 +63,7 @@ To share a project after enabling this feature: 1. Go to your project's page. 1. On the left sidebar, go to **Project information > Members**, and then select **Invite a group**. 1. Select a group, and select a **Max role**. -1. (Optional) Select an **Access expiration date**. +1. Optional. Select an **Access expiration date**. 1. Select **Invite**. ### Enable or disable modal window **(FREE SELF)** diff --git a/doc/user/project/merge_requests/approvals/index.md b/doc/user/project/merge_requests/approvals/index.md index d873f715557..dddd3925dbb 100644 --- a/doc/user/project/merge_requests/approvals/index.md +++ b/doc/user/project/merge_requests/approvals/index.md @@ -105,7 +105,7 @@ Without the approvals, the work cannot merge. Required approvals enable multiple - [Require approval from a security team](../../../application_security/index.md#security-approvals-in-merge-requests) before merging code that could introduce a vulnerability. **(ULTIMATE)** -## Related links +## Related topics - [Merge request approvals API](../../../../api/merge_request_approvals.md) - [Instance-level approval rules](../../../admin_area/merge_requests_approvals.md) for self-managed installations diff --git a/doc/user/project/merge_requests/approvals/rules.md b/doc/user/project/merge_requests/approvals/rules.md index 1249aa826fa..f4393b2b76d 100644 --- a/doc/user/project/merge_requests/approvals/rules.md +++ b/doc/user/project/merge_requests/approvals/rules.md @@ -63,7 +63,7 @@ To edit a merge request approval rule: 1. Go to your project and select **Settings > General**. 1. Expand **Merge request (MR) approvals**, and then select **Edit**. -1. (Optional) Change the **Rule name**. +1. Optional. Change the **Rule name**. 1. Set the number of required approvals in **Approvals required**. The minimum value is `0`. 1. Add or remove eligible approvers, as needed: - *To add users or groups as approvers,* search for users or groups that are diff --git a/doc/user/project/merge_requests/approvals/settings.md b/doc/user/project/merge_requests/approvals/settings.md index 56e93741c1a..a6ca9423df0 100644 --- a/doc/user/project/merge_requests/approvals/settings.md +++ b/doc/user/project/merge_requests/approvals/settings.md @@ -157,7 +157,7 @@ You can also enforce merge request approval settings: If the settings are inherited by a group or project, they cannot be changed in the group or project that inherited them. -## Related links +## Related topics - [Instance-level merge request approval settings](../../../admin_area/merge_requests_approvals.md) - [Compliance report](../../../compliance/compliance_report/index.md) diff --git a/doc/user/project/merge_requests/browser_performance_testing.md b/doc/user/project/merge_requests/browser_performance_testing.md index eff3a5bd99e..e59456e5b34 100644 --- a/doc/user/project/merge_requests/browser_performance_testing.md +++ b/doc/user/project/merge_requests/browser_performance_testing.md @@ -40,7 +40,7 @@ Consider the following workflow: ## How browser performance testing works First, define a job in your `.gitlab-ci.yml` file that generates the -[Browser Performance report artifact](../../../ci/yaml/index.md#artifactsreportsbrowser_performance). +[Browser Performance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsbrowser_performance). GitLab then checks this report, compares key performance metrics for each page between the source and target branches, and shows the information in the merge request. @@ -89,7 +89,7 @@ The above example: GitLab 12.3 or earlier, you must [add the configuration manually](#gitlab-versions-132-and-earlier). The template uses the [GitLab plugin for sitespeed.io](https://gitlab.com/gitlab-org/gl-performance), -and it saves the full HTML sitespeed.io report as a [Browser Performance report artifact](../../../ci/yaml/index.md#artifactsreportsbrowser_performance) +and it saves the full HTML sitespeed.io report as a [Browser Performance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsbrowser_performance) that you can later download and analyze. This implementation always takes the latest Browser Performance artifact available. If [GitLab Pages](../pages/index.md) is enabled, you can view the report directly in your browser. diff --git a/doc/user/project/merge_requests/cherry_pick_changes.md b/doc/user/project/merge_requests/cherry_pick_changes.md index 4a2319774ac..15ba6e9de98 100644 --- a/doc/user/project/merge_requests/cherry_pick_changes.md +++ b/doc/user/project/merge_requests/cherry_pick_changes.md @@ -76,10 +76,10 @@ merge request is from a fork: 1. Click on the **Options** dropdown and select **Cherry-pick** to show the cherry-pick modal. 1. In **Pick into project** and **Pick into branch**, select the destination project and branch:  -1. (Optional) Select **Start a new merge request** if you're ready to create a merge request. +1. Optional. Select **Start a new merge request** if you're ready to create a merge request. 1. Click **Cherry-pick**. -## Related links +## Related topics - The [Commits API](../../../api/commits.md) enables you to add custom messages to changes you cherry-pick through the API. diff --git a/doc/user/project/merge_requests/code_quality.md b/doc/user/project/merge_requests/code_quality.md index 9bfbbd8fc6f..b791bce5749 100644 --- a/doc/user/project/merge_requests/code_quality.md +++ b/doc/user/project/merge_requests/code_quality.md @@ -87,7 +87,7 @@ include: The above example creates a `code_quality` job in your CI/CD pipeline which scans your source code for code quality issues. The report is saved as a -[Code Quality report artifact](../../../ci/yaml/index.md#artifactsreportscodequality) +[Code Quality report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportscodequality) that you can later download and analyze. It's also possible to override the URL to the Code Quality image by @@ -343,7 +343,7 @@ It's possible to have a custom tool provide Code Quality reports in GitLab. To do this: 1. Define a job in your `.gitlab-ci.yml` file that generates the - [Code Quality report artifact](../../../ci/yaml/index.md#artifactsreportscodequality). + [Code Quality report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportscodequality). 1. Configure your tool to generate the Code Quality report artifact as a JSON file that implements a subset of the [Code Climate spec](https://github.com/codeclimate/platform/blob/master/spec/analyzers/SPEC.md#data-types). diff --git a/doc/user/project/merge_requests/commit_templates.md b/doc/user/project/merge_requests/commit_templates.md index b615c86288c..bffb66755e0 100644 --- a/doc/user/project/merge_requests/commit_templates.md +++ b/doc/user/project/merge_requests/commit_templates.md @@ -7,15 +7,42 @@ type: reference, howto # Commit message templates **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5. +> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) squash commit templates in GitLab 14.6. -## Merge commit message template +GitLab uses commit templates to create default messages for specific types of +commits. These templates encourage commit messages to follow a particular format, +or contain specific information. Users can override these templates when merging +a merge request. -As a project maintainer, you're able to configure merge commit message template. It will be used during merge to -create commit message. Template uses similar syntax to +Commit templates use syntax similar to the syntax for [review suggestions](reviews/suggestions.md#configure-the-commit-message-for-applied-suggestions). -Default merge commit message can be recreated using following template: +## Configure commit templates + +Change the commit templates for your project if the default templates don't +contain the information you need. + +Prerequisite: + +- You must have at least the Maintainer role for a project. + +To do this: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General** and expand **Merge requests**. +1. Depending on the type of template you want to create, scroll to either + [**Merge commit message template**](#default-template-for-merge-commits) or + [**Squash commit message template**](#default-template-for-squash-commits). +1. For your desired commit type, enter your default message. You can use both static + text and [variables](#supported-variables-in-commit-templates). Each template + is limited to a maximum of 500 characters, though after replacing the templates + with data, the final message may be longer. +1. Select **Save changes**. + +## Default template for merge commits + +The default template for merge commit messages is: ```plaintext Merge branch '%{source_branch}' into '%{target_branch}' @@ -27,25 +54,35 @@ Merge branch '%{source_branch}' into '%{target_branch}' See merge request %{reference} ``` -This commit message can be customized to follow any guidelines you might have. -To do so, expand the **Merge requests** tab within your project's **General** -settings and change the **Merge commit message template** text: +## Default template for squash commits + +If you have configured your project to [squash commits on merge](squash_and_merge.md), +GitLab creates a squash commit message with this template: + +```plaintext +%{title} +``` + +## Supported variables in commit templates + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5. +> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/346805) `first_commit` and `first_multiline_commit` variables in GitLab 14.6. - +Commit message templates support these variables: -You can use static text and following variables: +| Variable | Description | Output example | +|----------|-------------|----------------| +| `%{source_branch}` | The name of the branch being merged. | `my-feature-branch` | +| `%{target_branch}` | The name of the branch that the changes are applied to. | `main` | +| `%{title}` | Title of the merge request. | `Fix tests and translations` | +| `%{issues}` | String with phrase `Closes <issue numbers>`. Contains all issues mentioned in the merge request description that match [issue closing patterns](../issues/managing_issues.md#closing-issues-automatically). Empty if no issues are mentioned. | `Closes #465, #190 and #400` | +| `%{description}` | Description of the merge request. | `Merge request description.`<br>`Can be multiline.` | +| `%{reference}` | Reference to the merge request. | `group-name/project-name!72359` | +| `%{first_commit}` | Full message of the first commit in merge request diff. | `Update README.md` | +| `%{first_multiline_commit}` | Full message of the first commit that's not a merge commit and has more than one line in message body. Merge Request title if all commits aren't multiline. | `Update README.md`<br><br>`Improved project description in readme file.` | -| Variable | Description | Output example | -|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------| -| `%{source_branch}` | The name of the branch that is being merged. | `my-feature-branch` | -| `%{target_branch}` | The name of the branch that the changes are applied to. | `master` | -| `%{title}` | Title of the merge request. | Fix stuff | -| `%{issues}` | String with phrase "Closes <issue numbers>" with all issues mentioned in the MR description matching [issue closing patterns](../issues/managing_issues.md#closing-issues-automatically). It will be empty when no issues were mentioned. | `Closes #465, #190 and #400` | -| `%{description}` | Description of the merge request. | Merge request description.<br>Can be multiline. | -| `%{reference}` | Reference to the merge request. | group-name/project-name!72359 | +Empty variables that are the only word in a line are removed, along with all newline characters preceding it. -NOTE: -Empty variables that are the only word in a line will be removed along with all newline characters preceding it. +## Related topics -Merge commit template field has a limit of 500 characters. This limit only applies to the template -itself. +- [Squash and merge](squash_and_merge.md). diff --git a/doc/user/project/merge_requests/confidential.md b/doc/user/project/merge_requests/confidential.md index ff2e6acf123..10c63421876 100644 --- a/doc/user/project/merge_requests/confidential.md +++ b/doc/user/project/merge_requests/confidential.md @@ -70,7 +70,7 @@ Open a merge request - You are satisfied the problem is resolved in your private fork. - You are ready to make the confidential commits public. -## Related links +## Related topics - [Confidential issues](../issues/confidential_issues.md) - [Make an epic confidential](../../group/epics/manage_epics.md#make-an-epic-confidential) diff --git a/doc/user/project/merge_requests/creating_merge_requests.md b/doc/user/project/merge_requests/creating_merge_requests.md index 918f9830edc..220049d9a88 100644 --- a/doc/user/project/merge_requests/creating_merge_requests.md +++ b/doc/user/project/merge_requests/creating_merge_requests.md @@ -21,6 +21,10 @@ You can create a merge request from the list of merge requests. 1. Select a source and target branch and then **Compare branches and continue**. 1. Fill out the fields and select **Create merge request**. +NOTE: +Merge requests are designed around a one-to-one (1:1) branch relationship. Only one open merge request may +be associated with a given target branch at a time. + ## From an issue You can [create a merge request from an issue](../repository/web_editor.md#create-a-new-branch-from-an-issue). @@ -155,6 +159,8 @@ branch already exists, the patches are applied on top of it. ## Set the default target project +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/58093) in GitLab 13.11. + Merge requests have a source and a target project that are the same, unless forking is involved. Creating a fork of the project can cause either of these scenarios when you create a new merge request: diff --git a/doc/user/project/merge_requests/getting_started.md b/doc/user/project/merge_requests/getting_started.md index 006e6d4a8aa..323b7505190 100644 --- a/doc/user/project/merge_requests/getting_started.md +++ b/doc/user/project/merge_requests/getting_started.md @@ -88,7 +88,7 @@ Choose an assignee to designate someone as the person responsible for the first [review of the merge request](reviews/index.md). Open the drop down box to search for the user you wish to assign, and the merge request is added to their -[assigned merge request list](../../search/index.md#issues-and-merge-requests). +[assigned merge request list](../../search/index.md#search-issues-and-merge-requests). #### Multiple assignees **(PREMIUM)** @@ -136,10 +136,18 @@ To learn more, read [Review a merge request](reviews/index.md). ### Merge requests to close issues -If the merge request is being created to resolve an issue, you can -add a note in the description which sets it to -[automatically close the issue](../issues/managing_issues.md#closing-issues-automatically) -when merged. +To create a merge request to close an issue when it's merged, you can either: + +- [Add a note in the MR description](../issues/managing_issues.md#closing-issues-automatically). +- In the issue, select **Create a merge request**. Then, you can either: + + - Create a new branch and [a draft merge request](../merge_requests/drafts.md) + in one action. The branch is named `issuenumber-title` by default, but you can + choose any name, and GitLab verifies that it's not already in use. The merge request + inherits the milestone and labels of the issue, and is set to automatically + close the issue when it is merged. + - Create a [new branch](../repository/web_editor.md#create-a-new-branch-from-an-issue) + only, with its name starting with the issue number. If the issue is [confidential](../issues/confidential_issues.md), you may want to use a different workflow for diff --git a/doc/user/project/merge_requests/img/merge_commit_message_template_v14_5.png b/doc/user/project/merge_requests/img/merge_commit_message_template_v14_5.png Binary files differdeleted file mode 100644 index f18ca640d38..00000000000 --- a/doc/user/project/merge_requests/img/merge_commit_message_template_v14_5.png +++ /dev/null diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 54b97eb5732..8222d696853 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -74,7 +74,7 @@ change and whether you need access to a development environment: If you decide to permanently stop work on a merge request, GitLab recommends you close the merge request rather than -[delete it](#delete-a-merge-request). Users with +[delete it](#delete-a-merge-request). The author and assignees of a merge request, and users with Developer, Maintainer, or Owner [roles](../../permissions.md) in a project can close merge requests in the project: diff --git a/doc/user/project/merge_requests/load_performance_testing.md b/doc/user/project/merge_requests/load_performance_testing.md index 1d892a3c2e1..7b157aa94d8 100644 --- a/doc/user/project/merge_requests/load_performance_testing.md +++ b/doc/user/project/merge_requests/load_performance_testing.md @@ -28,7 +28,7 @@ GET calls to a popular API endpoint in your application to see how it performs. ## How Load Performance Testing works First, define a job in your `.gitlab-ci.yml` file that generates the -[Load Performance report artifact](../../../ci/yaml/index.md#artifactsreportsload_performance). +[Load Performance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsload_performance). GitLab checks this report, compares key load performance metrics between the source and target branches, and then shows the information in a merge request widget: @@ -140,7 +140,7 @@ For example, you can override the duration of the test with a CLI option: GitLab only displays the key performance metrics in the MR widget if k6's results are saved via [summary export](https://k6.io/docs/results-visualization/json#summary-export) -as a [Load Performance report artifact](../../../ci/yaml/index.md#artifactsreportsload_performance). +as a [Load Performance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsload_performance). The latest Load Performance artifact available is always used, using the summary values from the test. @@ -161,7 +161,7 @@ such as: ``http.get(`${__ENV.ENVIRONMENT_URL}`)``. For example: 1. In the `review` job: - 1. Capture the dynamic URL and save it into a `.env` file, e.g. `echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env`. + 1. Capture the dynamic URL and save it into a `.env` file, for example, `echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env`. 1. Set the `.env` file to be a [job artifact](../../../ci/pipelines/job_artifacts.md#job-artifacts). 1. In the `load_performance` job: 1. Set it to depend on the review job, so it inherits the environment file. diff --git a/doc/user/project/merge_requests/reviews/index.md b/doc/user/project/merge_requests/reviews/index.md index 597dcb3dfb9..c34a8116625 100644 --- a/doc/user/project/merge_requests/reviews/index.md +++ b/doc/user/project/merge_requests/reviews/index.md @@ -43,7 +43,7 @@ To start your review: - **Add to review**: Keep this comment private and add to the current review. These review comments are marked **Pending** and are visible only to you. - **Add comment now**: Submits the specific comment as a regular comment instead of as part of the review. -1. (Optional) You can use [quick actions](../../quick_actions.md) inside review comments. +1. Optional. You can use [quick actions](../../quick_actions.md) inside review comments. The comment shows the actions to perform after publication, but does not perform them until you submit your review. 1. When your review is complete, you can [submit the review](#submit-a-review). Your comments @@ -72,7 +72,7 @@ When you submit your review, GitLab: ### Resolve or unresolve thread with a comment -Review comments can also resolve or unresolve [resolvable threads](../../../discussions/index.md#resolve-a-thread)). +Review comments can also resolve or unresolve [resolvable threads](../../../discussions/index.md#resolve-a-thread). When replying to a comment, a checkbox is displayed to resolve or unresolve the thread after publication. diff --git a/doc/user/project/merge_requests/reviews/suggestions.md b/doc/user/project/merge_requests/reviews/suggestions.md index 7bfb8e52279..c25b9e15974 100644 --- a/doc/user/project/merge_requests/reviews/suggestions.md +++ b/doc/user/project/merge_requests/reviews/suggestions.md @@ -144,6 +144,6 @@ to your branch to address your reviewers' requests. WARNING: Suggestions applied from multiple authors creates a commit authored by the user applying the suggestions. -## Related links +## Related topics - [Suggestions API](../../../../api/suggestions.md) diff --git a/doc/user/project/merge_requests/squash_and_merge.md b/doc/user/project/merge_requests/squash_and_merge.md index c3fc2fa871f..3fe82fb8ef3 100644 --- a/doc/user/project/merge_requests/squash_and_merge.md +++ b/doc/user/project/merge_requests/squash_and_merge.md @@ -29,11 +29,9 @@ The squashed commit in this example is followed by a merge commit, because the m **Project Settings > General > Merge requests > Merge method > Fast-forward merge**. The squashed commit's default commit message is taken from the merge request title. +You can [edit the default message for squash commits](commit_templates.md). -NOTE: -This only takes effect if there are at least 2 commits. As there is nothing to squash, the commit message does not change if there is only 1 commit. - -It can be customized before merging a merge request. +It can also be customized before merging a merge request.  @@ -126,6 +124,10 @@ NOTE: If your project is set to **Do not allow** Squash and Merge, the users still have the option to squash commits locally through the command line and force-push to their remote branch before merging. +## Related topics + +- [Commit message templates](commit_templates.md). + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/project/merge_requests/status_checks.md b/doc/user/project/merge_requests/status_checks.md index 08b82462187..f5ebfb3668c 100644 --- a/doc/user/project/merge_requests/status_checks.md +++ b/doc/user/project/merge_requests/status_checks.md @@ -181,6 +181,6 @@ You should: - Check the [GitLab status page](https://status.gitlab.com/) if the problem persists, to see if there is a wider outage. -## Related links +## Related topics - [External status checks API](../../../api/status_checks.md) diff --git a/doc/user/project/merge_requests/test_coverage_visualization.md b/doc/user/project/merge_requests/test_coverage_visualization.md index b36510c2df8..1f84964c619 100644 --- a/doc/user/project/merge_requests/test_coverage_visualization.md +++ b/doc/user/project/merge_requests/test_coverage_visualization.md @@ -29,7 +29,7 @@ between pipeline completion and the visualization loading on the page. For the coverage analysis to work, you have to provide a properly formatted [Cobertura XML](https://cobertura.github.io/cobertura/) report to -[`artifacts:reports:cobertura`](../../../ci/yaml/index.md#artifactsreportscobertura). +[`artifacts:reports:cobertura`](../../../ci/yaml/artifacts_reports.md#artifactsreportscobertura). This format was originally developed for Java, but most coverage analysis frameworks for other languages have plugins to add support for it, like: @@ -52,10 +52,15 @@ from any job in any stage in the pipeline. The coverage displays for each line: Hovering over the coverage bar provides further information, such as the number of times the line was checked by tests. -NOTE: +### Limits + A limit of 100 `<source>` nodes for Cobertura format XML files applies. If your Cobertura report exceeds 100 nodes, there can be mismatches or no matches in the merge request diff view. +A single Cobertura XML file can be no more than 10MiB. For large projects, split the Cobertura XML into +smaller files. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328772) for more details. +When submitting many files, it can take a few minutes for coverage to show on a merge request. + ### Artifact expiration By default, the [pipeline artifact](../../../ci/pipelines/pipeline_artifacts.md#storage) used diff --git a/doc/user/project/merge_requests/versions.md b/doc/user/project/merge_requests/versions.md index 3922ee4d770..796ffc7866c 100644 --- a/doc/user/project/merge_requests/versions.md +++ b/doc/user/project/merge_requests/versions.md @@ -2,10 +2,9 @@ stage: Create group: Code Review info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, concepts --- -# Merge requests versions +# Merge requests versions **(FREE)** Every time you push to a branch that is tied to a merge request, a new version of merge request diff is created. When you visit a merge request that contains diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index 3f7d5b6aac1..fb61e123faf 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -119,7 +119,7 @@ Every issue and merge request can be assigned a milestone. The milestones are vi ### Filtering in list pages -From the project and group issue/merge request list pages, you can [filter](../../search/index.md#issues-and-merge-requests) by both group and project milestones. +From the project and group issue/merge request list pages, you can [filter](../../search/index.md#search-issues-and-merge-requests) by both group and project milestones. ### Filtering in issue boards diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md index 410fdab15e7..165131d50a4 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md @@ -36,12 +36,15 @@ for the most popular hosting services: - [Bluehost](https://www.bluehost.com/help/article/dns-management-add-edit-or-delete-dns-entries) - [Cloudflare](https://support.cloudflare.com/hc/en-us/articles/201720164-Creating-a-Cloudflare-account-and-adding-a-website) - [cPanel](https://documentation.cpanel.net/display/84Docs/Edit+DNS+Zone) +- [DigitalOcean](https://docs.digitalocean.com/products/networking/dns/how-to/manage-records/) - [DreamHost](https://help.dreamhost.com/hc/en-us/articles/215414867-How-do-I-add-custom-DNS-records-) +- [Gandi](https://docs.gandi.net/en/domain_names/faq/dns_records.html) - [Go Daddy](https://www.godaddy.com/help/add-an-a-record-19238) - [Hostgator](https://www.hostgator.com/help/article/changing-dns-records) - [Inmotion hosting](https://www.bluehost.com/help/article/dns-management-add-edit-or-delete-dns-entries) - [Media Temple](https://mediatemple.net/community/products/dv/204403794/how-can-i-change-the-dns-records-for-my-domain) - [Microsoft](https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-2000-server/bb727018(v=technet.10)) +- [Namecheap](https://www.namecheap.com/support/knowledgebase/subcategory/2237/host-records-setup/) <!-- vale gitlab.Spelling = YES --> diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md index ee4320d5ea1..ec66dce41f9 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md @@ -25,7 +25,8 @@ and steps below. (`*.gitlab.io`, for GitLab.com). - A custom domain name `example.com` or subdomain `subdomain.example.com`. - Access to your domain's server control panel to set up DNS records: - - A DNS A or CNAME record pointing your domain to GitLab Pages server. + - A DNS record (`A`, `ALIAS`, or `CNAME`) pointing your domain to the GitLab Pages server. If + there are multiple DNS records on that name, you must use an `ALIAS` record. - A DNS `TXT` record to verify your domain's ownership. - Set either `external_http` or `external_https` in `/etc/gitlab/gitlab.rb` to the IP and port of your [Pages Daemon](../../../../administration/pages/index.md#overview). @@ -109,15 +110,15 @@ as it most likely doesn't work if you set an [`MX` record](dns_concepts.md#mx-re Subdomains (`subdomain.example.com`) require: -- A DNS [CNAME record](dns_concepts.md#cname-record) pointing your subdomain to the Pages server. +- A DNS [`ALIAS` or `CNAME` record](dns_concepts.md#cname-record) pointing your subdomain to the Pages server. - A DNS [TXT record](dns_concepts.md#txt-record) to verify your domain's ownership. -| From | DNS Record | To | -| ------------------------------------------------------- | ---------- | --------------------- | -| `subdomain.example.com` | CNAME | `namespace.gitlab.io` | -| `_gitlab-pages-verification-code.subdomain.example.com` | `TXT` | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | +| From | DNS Record | To | +|:--------------------------------------------------------|:----------------|:----------------------| +| `subdomain.example.com` | `ALIAS`/`CNAME` | `namespace.gitlab.io` | +| `_gitlab-pages-verification-code.subdomain.example.com` | `TXT` | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | -Note that, whether it's a user or a project website, the `CNAME` +Note that, whether it's a user or a project website, the DNS record should point to your Pages domain (`namespace.gitlab.io`), without any `/project-name`. @@ -130,8 +131,8 @@ domain to the same website, for instance, `example.com` and `www.example.com`. They require: -- A DNS A record for the domain. -- A DNS CNAME record for the subdomain. +- A DNS `A` record for the domain. +- A DNS `ALIAS`/`CNAME` record for the subdomain. - A DNS `TXT` record for each. | From | DNS Record | To | @@ -147,7 +148,7 @@ If you're using Cloudflare, check > **Notes**: > -> - **Do not** use a CNAME record if you want to point your +> - **Do not** use a `CNAME` record if you want to point your `domain.com` to your GitLab Pages site. Use an `A` record instead. > - **Do not** add any special chars after the default Pages domain. For example, don't point `subdomain.domain.com` to @@ -231,7 +232,7 @@ If you use Cloudflare, you can redirect `www` to `domain.com` without adding both `www.domain.com` and `domain.com` to GitLab. To do so, you can use Cloudflare's page rules associated to a -CNAME record to redirect `www.domain.com` to `domain.com`. You +`CNAME` record to redirect `www.domain.com` to `domain.com`. You can use the following setup: 1. In Cloudflare, create a DNS `A` record pointing `domain.com` to `35.185.44.232`. diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md index ee1004a3416..75fc407ce3f 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md @@ -67,7 +67,7 @@ associated Pages domain. GitLab also renews it automatically. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30146) in GitLab 13.0. -If you get an error **Something went wrong while obtaining the Let's Encrypt certificate**, first, make sure that your pages site is set to "Everyone" in your project's **Settings > General > Visbility**. This allows the Let's Encrypt Servers reach your pages site. Once this is confirmed, you can try obtaining the certificate again by following these steps: +If you get an error **Something went wrong while obtaining the Let's Encrypt certificate**, first, make sure that your pages site is set to "Everyone" in your project's **Settings > General > Visibility**. This allows the Let's Encrypt Servers reach your pages site. Once this is confirmed, you can try obtaining the certificate again by following these steps: 1. Go to your project's **Settings > Pages**. 1. Click **Edit** on your domain. diff --git a/doc/user/project/pages/getting_started/pages_forked_sample_project.md b/doc/user/project/pages/getting_started/pages_forked_sample_project.md index 386ed566225..b43af2f0efe 100644 --- a/doc/user/project/pages/getting_started/pages_forked_sample_project.md +++ b/doc/user/project/pages/getting_started/pages_forked_sample_project.md @@ -18,9 +18,9 @@ configured to generate a Pages site. To fork a sample project and create a Pages website: 1. View the sample projects by navigating to the [GitLab Pages examples](https://gitlab.com/pages) group. -1. Click the name of the project you want to [fork](../../../../user/project/working_with_projects.md#fork-a-project). -1. In the top right, click the **Fork** button, and then choose a namespace to fork to. -1. Go to your project's **CI/CD > Pipelines** and click **Run pipeline**. +1. Select the name of the project you want to [fork](../../repository/forking_workflow.md#creating-a-fork). +1. In the top right, select **Fork** and then choose a namespace to fork to. +1. For your project, on the left sidebar, select **CI/CD > Pipelines** and then **Run pipeline**. GitLab CI/CD builds and deploys your site. The site can take approximately 30 minutes to deploy. @@ -31,13 +31,13 @@ that immediately publishes your changes to the Pages site. You can take some **optional** further steps: -- _Remove the fork relationship._ If you want to contribute to the project you forked from, +- Remove the fork relationship. If you want to contribute to the project you forked from, you can keep this relationship. Otherwise, go to your project's **Settings > General**, expand **Advanced settings**, and scroll down to **Remove fork relationship**:  -- _Change the URL to match your namespace._ If your Pages site is hosted on GitLab.com, +- Change the URL to match your namespace. If your Pages site is hosted on GitLab.com, you can rename it to `<namespace>.gitlab.io`, where `<namespace>` is your GitLab namespace (the one you chose when you forked the project). diff --git a/doc/user/project/pages/getting_started/pages_from_scratch.md b/doc/user/project/pages/getting_started/pages_from_scratch.md index eb5f3a1bbf0..e0c10e27ec3 100644 --- a/doc/user/project/pages/getting_started/pages_from_scratch.md +++ b/doc/user/project/pages/getting_started/pages_from_scratch.md @@ -27,7 +27,7 @@ To create a GitLab Pages website: ## Prerequisites -You must have a [blank project](../../working_with_projects.md#blank-projects) in GitLab. +You must have a [blank project](../../working_with_projects.md#create-a-blank-project) in GitLab. ## Create the project files diff --git a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md index 978e35b3a9f..1f60aafe71b 100644 --- a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md +++ b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md @@ -1,165 +1,9 @@ --- -stage: Release -group: Release -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -description: "How to secure GitLab Pages websites with Let's Encrypt (manual process, deprecated)." +redirect_to: 'custom_domains_ssl_tls_certification/lets_encrypt_integration.md' +remove_date: '2022-03-14' --- -# Let's Encrypt for GitLab Pages (manual process, deprecated) **(FREE)** +This file was moved to [another location](custom_domains_ssl_tls_certification/lets_encrypt_integration.md). -WARNING: -This method is still valid but was **deprecated** in favor of the -[Let's Encrypt integration](custom_domains_ssl_tls_certification/lets_encrypt_integration.md) -introduced in GitLab 12.1. - -If you have a GitLab Pages website served under your own domain, -you might want to secure it with a SSL/TLS certificate. - -[Let's Encrypt](https://letsencrypt.org) is a free, automated, and -open source Certificate Authority. - -## Requirements - -To follow along with this tutorial, we assume you already have: - -- [Created a project](index.md#getting-started) in GitLab - containing your website's source code. -- Acquired a domain (`example.com`) and added a [DNS entry](custom_domains_ssl_tls_certification/index.md#set-up-pages-with-a-custom-domain) - pointing it to your Pages website. -- [Added your domain to your Pages project](custom_domains_ssl_tls_certification/index.md#steps) - and verified your ownership. -- Cloned your project into your computer. -- Your website up and running, served under HTTP protocol at `http://example.com`. - -## Obtaining a Let's Encrypt certificate - -Once you have the requirements addressed, follow the instructions -below to learn how to obtain the certificate. - -Note that these instructions were tested on macOS Mojave. For other operating systems the steps -might be slightly different. Follow the -[CertBot instructions](https://certbot.eff.org/) according to your OS. - -1. On your computer, open a terminal and navigate to your repository's - root directory: - - ```shell - cd path/to/dir - ``` - -1. Install CertBot (the tool Let's Encrypt uses to issue certificates): - - ```shell - brew install certbot - ``` - -1. Request a certificate for your domain (`example.com`) and - provide an email account (`your@email.com`) to receive notifications: - - ```shell - sudo certbot certonly -a manual -d example.com --email your@email.com - ``` - - Alternatively, you can register without adding an email account, - but you aren't notified about the certificate expiration's date: - - ```shell - sudo certbot certonly -a manual -d example.com --register-unsafely-without-email - ``` - - NOTE: - Read through CertBot's documentation on their - [command line options](https://certbot.eff.org/docs/using.html#certbot-command-line-options). - -1. You're prompted with a message to agree with their terms. - Press `A` to agree and `Y` to let they log your IP. - - CertBot then prompts you with the following message: - - ```shell - Create a file containing just this data: - - Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP.HUGNKk82jlsmOOfphlt8Jy69iuglsn095nxOMH9j3Yb - - And make it available on your web server at this URL: - - http://example.com/.well-known/acme-challenge/Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP - - Press Enter to Continue - ``` - -1. **Do not press Enter yet.** Let's Encrypt needs to verify your - domain ownership before issuing the certificate. To do so, create 3 - consecutive directories under your website's root: - `/.well-known/acme-challenge/Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP/` - and add to the last folder an `index.html` file containing the content - referred on the previous prompt message: - - ```shell - Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP.HUGNKk82jlsmOOfphlt8Jy69iuglsn095nxOMH9j3Yb - ``` - - Note that this file needs to be accessed under - `http://example.com/.well-known/acme-challenge/Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP` - to allow Let's Encrypt to verify the ownership of your domain, - therefore, it needs to be part of the website content under the - repository's [`public`](index.md#how-it-works) folder. - -1. Add, commit, and push the file into your repository in GitLab. Once the pipeline - passes, press **Enter** on your terminal to continue issuing your - certificate. CertBot then prompts you with the following message: - - ```shell - Waiting for verification... - Cleaning up challenges - - IMPORTANT NOTES: - - Congratulations! Your certificate and chain have been saved at: - /etc/letsencrypt/live/example.com/fullchain.pem - Your key file has been saved at: - /etc/letsencrypt/live/example.com/privkey.pem - Your cert will expire on 2019-03-12. To obtain a new or tweaked - version of this certificate in the future, simply run certbot - again. To non-interactively renew *all* of your certificates, run - "certbot renew" - - If you like Certbot, please consider supporting our work by: - - Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate - Donating to EFF: https://eff.org/donate-le - ``` - -## Add your certificate to GitLab Pages - -Now that your certificate has been issued, let's add it to your Pages site: - -1. Back at GitLab, navigate to your project's **Settings > Pages**, - find your domain and click **Details** and **Edit** to add your certificate. -1. From your terminal, copy and paste the certificate into the first field - **Certificate (PEM)**: - - ```shell - sudo cat /etc/letsencrypt/live/example.com/fullchain.pem | pbcopy - ``` - -1. Copy and paste the private key into the second field **Key (PEM)**: - - ```shell - sudo cat /etc/letsencrypt/live/example.com/privkey.pem | pbcopy - ``` - -1. Click **Save changes** to apply them to your website. -1. Wait a few minutes for the configuration changes to take effect. -1. Visit your website at `https://example.com`. - -To force `https` connections on your site, navigate to your -project's **Settings > Pages** and check **Force HTTPS (requires -valid certificates)**. - -## Renewal - -Let's Encrypt certificates expire every 90 days and you must -renew them periodically. To renew all your certificates at once, run: - -```shell -sudo certbot renew -``` +<!-- This redirect file can be deleted after <2022-03-14>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/project/pages/redirects.md b/doc/user/project/pages/redirects.md index 3deea92f56e..beafbc86cb5 100644 --- a/doc/user/project/pages/redirects.md +++ b/doc/user/project/pages/redirects.md @@ -92,7 +92,7 @@ you can explicitly set your own. The following HTTP codes are supported: > - [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/458) in GitLab 14.3. > - Enabled on GitLab.com. -> - Enabled by default in self-managed GitLab behind the [`FF_ENABLE_REDIRECTS` feature flag](#feature-flag-for-redirects). +> - Enabled on self-managed in [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/618). To create a redirect, add a rule that includes a `from` path, a `to` path, and an [HTTP status code](#http-status-codes): @@ -261,36 +261,7 @@ However, there are some minor differences: literal `:placeholder`). - GitLab redirects to `/new/`. -## Features behind feature flags - -Some Pages features are behind feature flags. - -### Feature flag for redirects - -FLAG: -Redirects in GitLab Pages is under development, and is deployed behind a feature flag -that is **enabled by default**. - -To disable redirects, for [Omnibus installations](../../../administration/pages/index.md), define the -`FF_ENABLE_REDIRECTS` environment variable in the -[global settings](../../../administration/pages/index.md#global-settings). -Add the following line to `/etc/gitlab/gitlab.rb` and -[reconfigure the instance](../../../administration/restart_gitlab.md#omnibus-gitlab-reconfigure). - -```ruby -gitlab_pages['env']['FF_ENABLE_REDIRECTS'] = 'false' -``` - -For [source installations](../../../administration/pages/source.md), define the -`FF_ENABLE_REDIRECTS` environment variable, then -[restart GitLab](../../../administration/restart_gitlab.md#installations-from-source): - -```shell -export FF_ENABLE_REDIRECTS="false" -/path/to/pages/bin/gitlab-pages -config gitlab-pages.conf -``` - -### Feature flag for rewrites +## Feature flag for rewrites FLAG: Rewrites in GitLab Pages is under development, and is deployed behind a feature flag diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index a4d7b94506b..75a5a2a6a4f 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -35,7 +35,7 @@ If you manually enter a parameter, it must be enclosed in double quotation marks - ASCII letters - Numbers (0-9) -- Underscore (`_`), hyphen (`-`), question mark (`?`), dot (`.`), or ampersand (`&`) +- Underscore (`_`), hyphen (`-`), question mark (`?`), dot (`.`), ampersand (`&`) or at (`@`) Parameters are case-sensitive. Autocomplete handles this, and the insertion of quotation marks, automatically. @@ -45,14 +45,15 @@ of quotation marks, automatically. The following quick actions are applicable to descriptions, discussions, and threads. Some quick actions might not be available to all subscription tiers. +<!-- Keep this table sorted alphabetically --> + | Command | Issue | Merge request | Epic | Action | |:--------------------------------------|:-----------------------|:-----------------------|:-----------------------|:-------| +| `/add_contacts email1 email2` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add one or more [CRM contacts](../crm/index.md) ([introduced in GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73413)). | | `/approve` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Approve the merge request. | -| `/assign @user` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Assign one user. | -| `/assign @user1 @user2` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Assign multiple users. | +| `/assign @user1 @user2` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Assign one or more users. | | `/assign me` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Assign yourself. | -| `/assign_reviewer @user` or `/reviewer @user` or `/request_review @user` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Assign one user as a reviewer. | -| `/assign_reviewer @user1 @user2` or `/reviewer @user1 @user2` or `/request_review @user1 @user2` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Assign multiple users as reviewers. | +| `/assign_reviewer @user1 @user2` or `/reviewer @user1 @user2` or `/request_review @user1 @user2` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Assign one or more users as reviewers. | | `/assign_reviewer me` or `/reviewer me` or `/request_review me` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Assign yourself as a reviewer. | | `/award :emoji:` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Toggle emoji award. | | `/child_epic <epic>` | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Add child epic to `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic`, or a URL to an epic ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab/-/issues/7330)). | @@ -81,11 +82,12 @@ threads. Some quick actions might not be available to all subscription tiers. | `/promote_to_incident` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Promote issue to incident ([introduced in GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/issues/296787)). | | `/publish` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Publish issue to an associated [Status Page](../../operations/incident_management/status_page.md) ([Introduced in GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/30906)) | | `/reassign @user1 @user2` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Replace current assignees with those specified. | -| `/rebase` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Rebase source branch. This schedules a background task that attempts to rebase the changes in the source branch on the latest commit of the target branch. If `/rebase` is used, `/merge` is ignored to avoid a race condition where the source branch is merged or deleted before it is rebased. If there are merge conflicts, GitLab displays a message that a rebase cannot be scheduled. Rebase failures are displayed with the merge request status. | | `/reassign_reviewer @user1 @user2` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Replace current reviewers with those specified. | +| `/rebase` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Rebase source branch. This schedules a background task that attempts to rebase the changes in the source branch on the latest commit of the target branch. If `/rebase` is used, `/merge` is ignored to avoid a race condition where the source branch is merged or deleted before it is rebased. If there are merge conflicts, GitLab displays a message that a rebase cannot be scheduled. Rebase failures are displayed with the merge request status. | | `/relabel ~label1 ~label2` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Replace current labels with those specified. | | `/relate #issue1 #issue2` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Mark issues as related. | | `/remove_child_epic <epic>` | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Remove child epic from `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic`, or a URL to an epic ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab/-/issues/7330)). | +| `/remove_contacts email1 email2` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Remove one or more [CRM contacts](../crm/index.md) ([introduced in GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73413)). | | `/remove_due_date` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Remove due date. | | `/remove_epic` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Remove from epic. | | `/remove_estimate` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Remove time estimate. | diff --git a/doc/user/project/releases/img/feature_count_v14_6.png b/doc/user/project/releases/img/feature_count_v14_6.png Binary files differnew file mode 100644 index 00000000000..0b1a0552631 --- /dev/null +++ b/doc/user/project/releases/img/feature_count_v14_6.png diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index abedae5d10b..239e6c9cea8 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -1,5 +1,4 @@ --- -type: reference, howto stage: Release group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments @@ -9,26 +8,33 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41766) in GitLab 11.7. -To introduce a checkpoint in your source code history, you can assign a -[Git tag](https://git-scm.com/book/en/v2/Git-Basics-Tagging) at the moment of release. -However, in most cases, your users need more than just the raw source code. -They need compiled objects or other assets output by your CI/CD system. +In GitLab, a release enables you to create a snapshot of your project for your users, including +installation packages and release notes. You can create a GitLab release on any branch. Creating a +release also creates a [Git tag](https://git-scm.com/book/en/v2/Git-Basics-Tagging) to mark the +release point in the source code. -A GitLab Release can be: +WARNING: +Deleting a Git tag associated with a release also deletes the release. + +A release can include: - A snapshot of the source code of your repository. - [Generic packages](../../packages/generic_packages/index.md) created from job artifacts. - Other metadata associated with a released version of your code. +- Release notes. -You can create a GitLab release on any branch. When you create a release: +When you [create a release](#create-a-release): - GitLab automatically archives source code and associates it with the release. - GitLab automatically creates a JSON file that lists everything in the release, so you can compare and audit releases. This file is called [release evidence](#release-evidence). -- You can add release notes and a message for the tag associated with the release. -After you create a release, you can [associate milestones with it](#associate-milestones-with-a-release), -and attach [release assets](#release-assets), like runbooks or packages. +When you create a release, or after, you can: + +- Add release notes. +- Add a message for the Git tag associated with the release. +- [Associate milestones with it](#associate-milestones-with-a-release). +- Attach [release assets](#release-assets), like runbooks or packages. ## View releases @@ -46,49 +52,84 @@ To view a list of releases: - On private projects, this number is visible to users with Reporter [permissions](../../permissions.md#project-members-permissions) or higher. -### Sort Releases +### Sort releases -On the top right of the **Releases** page, you can use the sorting button to order releases by -**Released date** or **Created date**. You can sort releases in ascending or descending order. +To sort releases by **Released date** or **Created date**, select from the sort order dropdown. To +switch between ascending or descending order, select **Sort order**. - + ## Create a release > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32812) in GitLab 12.9. Releases can be created directly in the GitLab UI. -You can create a release in the user interface, or by using the -[Releases API](../../../api/releases/index.md#create-a-release). -We recommend using the API to create releases as one of the last steps in your -CI/CD pipeline. +You can create a release: -Only users with at least the Developer role can create releases. -Read more about [Release permissions](#release-permissions). +- [Using a job in your CI/CD pipeline](#create-a-release-by-using-a-cicd-job). +- [In the Releases page](#create-a-release-in-the-releases-page). +- [In the Tags page](#create-a-release-in-the-tags-page). +- Using the [Releases API](../../../api/releases/index.md#create-a-release). + +We recommend creating a release as one of the last steps in your CI/CD pipeline. + +Prerequisites: + +- You must have at least the Developer role for a project. For more information, read +[Release permissions](#release-permissions). + +### Create a release in the Releases page -To create a new release through the GitLab UI: +To create a release in the Releases page: +1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Deployments > Releases** and select **New release**. -1. Open the [**Tag name**](#tag-name) dropdown. Select an existing tag or type - in a new tag name. Selecting an existing tag that is already associated with - a release will result in a validation error. -1. If creating a new tag, open the **Create from** dropdown. Select a - branch, tag, or commit SHA to use when creating the new tag. -1. Optionally, fill out any additional information about the release, such as its - [title](#title), [milestones](#associate-milestones-with-a-release), - [release notes](#release-notes-description), or [assets links](#links). -1. Click **Create release**. - -## Create a release by using a CI/CD job +1. From the [**Tag name**](#tag-name) dropdown, either: + - Select an existing Git tag. Selecting an existing tag that is already associated with a release + results in a validation error. + - Enter a new Git tag name. + 1. From the **Create from** dropdown, select a branch or commit SHA to use when creating the + new tag. +1. Optional. Enter additional information about the release, including: + - [Title](#title). + - [Milestones](#associate-milestones-with-a-release). + - [Release notes](#release-notes-description). + - [Asset links](#links). +1. Select **Create release**. + +### Create a release in the Tags page + +To create a release in the Tags page, add release notes to either an existing or a new Git tag. + +To add release notes to a new Git tag: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Repository > Tags**. +1. Select **New tag**. +1. Optional. Enter a tag message in the **Message** text box. +1. In the **Release notes** text box, enter the release's description. + You can use Markdown and drag and drop files to this text box. +1. Select **Create tag**. + +To edit release notes of an existing Git tag: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Repository > Tags**. +1. Select **Edit release notes** (**{pencil}**). +1. In the **Release notes** text box, enter the release's description. + You can use Markdown and drag and drop files to this text box. +1. Select **Save changes**. + +### Create a release by using a CI/CD job > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19298) in GitLab 12.7. You can create a release directly as part of the GitLab CI/CD pipeline -by using [the `release` keyword](../../../ci/yaml/index.md#release) in the job definition. +by using the [`release` keyword](../../../ci/yaml/index.md#release) in the job definition. -The release is created only if the job processes without error. If the Rails API returns an error -during release creation, the release job fails. +The release is created only if the job processes without error. If the API returns an error during +release creation, the release job fails. -### CI/CD example of the `release` keyword +#### CI/CD example of the `release` keyword To create a release when you push a Git tag, or when you add a Git tag in the UI by going to **Repository > Tags**: @@ -100,7 +141,7 @@ release_job: rules: - if: $CI_COMMIT_TAG # Run this job when a tag is created manually script: - - echo 'running release_job' + - echo "running release_job" release: name: 'Release $CI_COMMIT_TAG' description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION must be defined @@ -149,7 +190,7 @@ release_job: when: never # Do not run this job when a tag is created manually - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch script: - - echo 'running release_job for $TAG' + - echo "running release_job for $TAG" release: name: 'Release $TAG' description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION and the $TAG @@ -207,7 +248,7 @@ which requires the text representation of the certificate. ### `release-cli` command line -The entries under the `release` node are transformed into a `bash` command line and sent +The entries under the `release` node are transformed into Bash commands and sent to the Docker container, which contains the [release-cli](https://gitlab.com/gitlab-org/release-cli). You can also call the `release-cli` directly from a `script` entry. @@ -224,14 +265,14 @@ A pipeline can have multiple `release` jobs, for example: ```yaml ios-release: script: - - echo 'iOS release job' + - echo "iOS release job" release: tag_name: v1.0.0-ios description: 'iOS release v1.0.0' android-release: script: - - echo 'Android release job' + - echo "Android release job" release: tag_name: v1.0.0-android description: 'Android release v1.0.0' @@ -255,7 +296,8 @@ release tag. When the `released_at` date and time has passed, the badge is autom ## Edit a release -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26016) in GitLab 12.6. Asset link editing was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9427) in GitLab 12.10. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26016) in GitLab 12.6. +> - Asset link editing [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9427) in GitLab 12.10. Only users with at least the Developer role can edit releases. Read more about [Release permissions](#release-permissions). @@ -271,29 +313,6 @@ You can edit the release title, notes, associated milestones, and asset links. To change the release date use the [Releases API](../../../api/releases/index.md#update-a-release). -## Add release notes to Git tags - -If you have an existing Git tag, you can add release notes to it. - -You can do this in the user interface, or by using the [Releases API](../../../api/releases/index.md). -We recommend using the API to add release notes as one of the last steps in your CI/CD release pipeline. - -In the interface, to add release notes to a new Git tag: - -1. Navigate to your project's **Repository > Tags**. -1. Click **New tag**. -1. In the **Release notes** field, enter the release's description. - You can use Markdown and drag and drop files to this field. -1. Click **Create tag**. - -In the interface, to add release notes to an existing Git tag: - -1. Navigate to your project's **Repository > Tags**. -1. Click **Edit release notes** (the pencil icon). -1. In the **Release notes** field, enter the release's description. - You can use Markdown in this field, and drag and drop files to it. -1. Click **Save changes**. - ## Associate milestones with a release > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29020) in GitLab 12.5. @@ -325,6 +344,11 @@ Here is an example of milestones with no releases, one release, and two releases  +NOTE: +A subgroup's project releases cannot be associated with a supergroup's milestone. To learn +more, read issue #328054, +[Releases cannot be associated with a supergroup milestone](https://gitlab.com/gitlab-org/gitlab/-/issues/328054). + ## Get notified when a release is created > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26001) in GitLab 12.4. @@ -576,6 +600,29 @@ links to a release is not recommended, because artifacts are ephemeral and are used to pass data in the same pipeline. This means there's a risk that they could either expire or someone might manually delete them. +#### Number of new and total features **(FREE SAAS)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235618) in GitLab 13.5. + +On [GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/releases), you can view the number of new and total features in the project. + + + +The totals are displayed on [shields](https://shields.io/) and are generated per release by +[a Rake task in the `www-gitlab-com` repo](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/lib/tasks/update_gitlab_project_releases_page.rake). + +| Item | Formula | +| ------ | ------ | +| `New features` | Total count of release posts across all tiers for a single release in the project. | +| `Total features` | Total count of release posts in reverse order for all releases in the project. | + +The counts are also shown by license tier. + +| Item | Formula | +| ------ | ------ | +| `New features` | Total count of release posts across a single tier for a single release in the project. | +| `Total features` | Total count of release posts across a single tier in reverse order for all releases in the project. | + ## Release evidence > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26019) in GitLab 12.6. @@ -776,7 +823,7 @@ You can copy the example project to your own group or instance for testing. More ### Getting `403 Forbidden` or `Something went wrong while creating a new release` errors when creating, updating or deleting releases and their assets -If the release is associted with a [protected tag](../protected_tags.md), +If the release is associated with a [protected tag](../protected_tags.md), the UI/API request might result in an authorization failure. Make sure that the user or a service/bot account is allowed to [create the protected tag](../protected_tags.md#configuring-protected-tags) too. diff --git a/doc/user/project/releases/release_cli.md b/doc/user/project/releases/release_cli.md index c23ceaa1aba..b55f0b0a734 100644 --- a/doc/user/project/releases/release_cli.md +++ b/doc/user/project/releases/release_cli.md @@ -1,11 +1,10 @@ --- -type: reference, howto stage: Release group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install the `release-cli` for the Shell executor +# Install the `release-cli` for the Shell executor **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/release-cli/-/issues/21) in GitLab 13.8. > - [Changed](https://gitlab.com/gitlab-org/release-cli/-/merge_requests/108) in GitLab 14.2, the `release-cli` binaries are also [available in the Package Registry](https://gitlab.com/jaime/release-cli/-/packages). diff --git a/doc/user/project/repository/branches/default.md b/doc/user/project/repository/branches/default.md index bc6bc799670..65b7c4a9881 100644 --- a/doc/user/project/repository/branches/default.md +++ b/doc/user/project/repository/branches/default.md @@ -41,7 +41,7 @@ To update the default branch name for an individual [project](../../index.md): 1. Sign in to GitLab with at least the [Maintainer](../../../permissions.md) role. 1. In the left navigation menu, go to **Settings > Repository**. 1. Expand **Default branch**, and select a new default branch. -1. (Optional) Select the **Auto-close referenced issues on default branch** checkbox to close +1. Optional. Select the **Auto-close referenced issues on default branch** checkbox to close issues when a merge request [uses a closing pattern](../../issues/managing_issues.md#closing-issues-automatically). 1. Select **Save changes**. @@ -134,7 +134,7 @@ renames a Git repository's (`example`) default branch. [change the default branch for this project](#change-the-default-branch-name-for-a-project). Select `main` as your new default branch. 1. Protect your new `main` branch as described in the [protected branches documentation](../../protected_branches.md). -1. (Optional) If you want to delete the old default branch: +1. Optional. If you want to delete the old default branch: 1. Verify that nothing is pointing to it. 1. Delete the branch on the remote: diff --git a/doc/user/project/repository/forking_workflow.md b/doc/user/project/repository/forking_workflow.md index bf4ef21e31b..1ab21286a8e 100644 --- a/doc/user/project/repository/forking_workflow.md +++ b/doc/user/project/repository/forking_workflow.md @@ -26,17 +26,17 @@ To fork an existing project in GitLab: 1. Select the project to fork to: - - *(Recommended method)* Below **Select a namespace to fork the project**, identify + - Recommended method. Below **Select a namespace to fork the project**, identify the project you want to fork to, and click **Select**. Only namespaces you have Developer and higher [permissions](../../permissions.md) for are shown.  - - *(Experimental method)* If your GitLab administrator has + - Experimental method. If your GitLab administrator has [enabled the experimental fork project form](#enable-or-disable-the-fork-project-form), read [Create a fork with the fork project form](#create-a-fork-with-the-fork-project-form). - Only namespaces you have Developer and higher - [permissions](../../permissions.md) for are shown. + Only namespaces you have at least the Developer + [role](../../permissions.md) for are shown. NOTE: The project path must be unique in the namespace. @@ -101,7 +101,7 @@ To use it, follow the instructions at [Creating a fork](#creating-a-fork) and pr - The project name. - The project URL. - The project slug. -- *(Optional)* The project description. +- Optional. The project description. - The visibility level for your fork. ### Enable or disable the fork project form **(FREE SELF)** diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md index 6aa32b1b816..27767f8d325 100644 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ b/doc/user/project/repository/gpg_signed_commits/index.md @@ -40,7 +40,7 @@ For a commit to be verified by GitLab: account. - One of the emails in the GPG key must match a **verified** email address used by the committer in GitLab. This address will be part of the public key. - If you want to keep this address private, use the automatically generated + If you want to keep this address private, use the automatically generated [private commit email address](../../../profile/index.md#use-an-automatically-generated-private-commit-email) GitLab provides in your profile. - The committer's email address must match the verified email address from the @@ -54,17 +54,18 @@ started: 1. [Install GPG](https://www.gnupg.org/download/index.html) for your operating system. If your operating system has `gpg2` installed, replace `gpg` with `gpg2` in the following commands. -1. Generate the private/public key pair with the following command, which will - spawn a series of questions: +1. Generate the private/public key pair with the command appropriate for your version + of `gpg`. This command spawns a series of questions: ```shell + # Use this command for the default version of gpg, including + # Gpg4win on Windows, and most macOS versions: + gpg --gen-key + + # Use this command for versions of GPG later than 2.1.17: gpg --full-gen-key ``` - NOTE: - In some cases like Gpg4win on Windows and other macOS versions, the command - here may be `gpg --gen-key`. - 1. The first question is which algorithm can be used. Select the kind you want or press <kbd>Enter</kbd> to choose the default (RSA and RSA): @@ -200,7 +201,7 @@ key to use. Replace `30F2B65B9246B6CA` with your GPG key ID. -1. (Optional) If Git is using `gpg` and you get errors like `secret key not available` +1. Optional. If Git is using `gpg` and you get errors like `secret key not available` or `gpg: signing failed: secret key not available`, run the following command to change to `gpg2`: diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index bb1a55f6b2b..54e9470892c 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -248,9 +248,19 @@ When you [rename a user](../../profile/index.md#change-your-username), - The redirects are available as long as the original path is not claimed by another group, user, or project. -## Related links +## Related topics -- [GitLab Workflow VS Code extension](vscode.md) +- [GitLab Workflow VS Code extension](vscode.md). +- To lock files and prevent change conflicts, use [file locking](../file_lock.md). +- [Repository API](../../../api/repositories.md). +- [Find files](file_finder.md) in a repository. +- [Branches](branches/index.md). +- [File templates](web_editor.md#template-dropdowns). +- [Create a directory](web_editor.md#create-a-directory). +- [Start a merge request](web_editor.md#tips). +- [Find file history](git_history.md). +- [Identify changes by line (Git blame)](git_blame.md). +- [Use Jupyter notebooks with GitLab](jupyter_notebooks/index.md). ## Troubleshooting @@ -287,16 +297,3 @@ The same approach should also allow misidentified file types to be fixed. ``` `*.txt` files have an entry in the heuristics file. This example prevents parsing of these files. - -## Related topics - -- To lock files and prevent change conflicts, use [file locking](../file_lock.md). -- [Repository API](../../../api/repositories.md). -- [Find files](file_finder.md) in a repository. -- [Branches](branches/index.md). -- [File templates](web_editor.md#template-dropdowns). -- [Create a directory](web_editor.md#create-a-directory). -- [Start a merge request](web_editor.md#tips). -- [Find file history](git_history.md). -- [Identify changes by line (Git blame)](git_blame.md). -- [Use Jupyter notebooks with GitLab](jupyter_notebooks/index.md). diff --git a/doc/user/project/repository/jupyter_notebooks/img/jupyter_notebook_diff_v14_5.png b/doc/user/project/repository/jupyter_notebooks/img/jupyter_notebook_diff_v14_5.png Binary files differindex bc63322cd65..3dc940c23de 100644 --- a/doc/user/project/repository/jupyter_notebooks/img/jupyter_notebook_diff_v14_5.png +++ b/doc/user/project/repository/jupyter_notebooks/img/jupyter_notebook_diff_v14_5.png diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index 8074f311e5f..ba105a970a7 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -175,10 +175,10 @@ request, you can create a new branch upfront.  1. Enter a new **Branch name**. -1. (Optional) Change the **Create from** field to choose which branch, tag, or +1. Optional. Change the **Create from** field to choose which branch, tag, or commit SHA this new branch originates from. This field autocompletes if you start typing an existing branch or tag. -1. Click **Create branch** to return to the file browser on this new branch. +1. To return to the file browser on this new branch, select **Create branch**.  @@ -201,10 +201,10 @@ SHA: 1. Give the tag a name such as `v1.0.0`. 1. Choose the branch or SHA from which you want to create this new tag. -1. (Optional) Add a message and release notes. The release notes section supports +1. Optional. Add a message and release notes. The release notes section supports Markdown format. -1. (Optional) Upload an attachment. -1. Click **Create tag**, and GitLab redirects you to the tag list page. +1. Optional. Upload an attachment. +1. Select **Create tag**. GitLab redirects you to the tag list page.  diff --git a/doc/user/project/repository/x509_signed_commits/index.md b/doc/user/project/repository/x509_signed_commits/index.md index 2db7ae308bd..f9d9af3e672 100644 --- a/doc/user/project/repository/x509_signed_commits/index.md +++ b/doc/user/project/repository/x509_signed_commits/index.md @@ -290,7 +290,7 @@ To investigate why a commit shows as `Unverified`: 1. The verification status is stored in the database. To display the database record: ```ruby - pp X509CommitSignature.by_commit_sha(commit.sha);nil + pp CommitSignatures::X509CommitSignature.by_commit_sha(commit.sha);nil ``` If all the previous checks returned the correct values: diff --git a/doc/user/project/requirements/index.md b/doc/user/project/requirements/index.md index 6ee23c91680..294e493dfe9 100644 --- a/doc/user/project/requirements/index.md +++ b/doc/user/project/requirements/index.md @@ -17,6 +17,10 @@ In 14.4, Requirements was moved under **Issues**. With requirements, you can set criteria to check your products against. They can be based on users, stakeholders, system, software, or anything else you find important to capture. +INFO: +Meet your compliance needs with requirements in GitLab. +[Try Ultimate free for 30 days](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=u-project-requirements-docs). + A requirement is an artifact in GitLab which describes the specific behavior of your product. Requirements are long-lived and don't disappear unless manually cleared. @@ -134,7 +138,7 @@ You can also sort the requirements list by: > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.1. > - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/215514) ability to specify individual requirements and their statuses in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.2. -GitLab supports [requirements test reports](../../../ci/yaml/index.md#artifactsreportsrequirements) now. +GitLab supports [requirements test reports](../../../ci/yaml/artifacts_reports.md#artifactsreportsrequirements) now. You can add a job to your CI pipeline that, when triggered, marks all existing requirements as Satisfied (you may manually satisfy a requirement in the edit form [edit a requirement](#edit-a-requirement)). diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md index 4f8e068ca2d..072d685bde4 100644 --- a/doc/user/project/service_desk.md +++ b/doc/user/project/service_desk.md @@ -274,7 +274,7 @@ When configured, the custom suffix creates a new Service Desk email address, con For example, suppose the `mygroup/myproject` project Service Desk settings has the following configured: -- Project name suffix is set to `support`. +- Email address suffix is set to `support`. - Service Desk email address is configured to `contact+%{key}@example.com`. The Service Desk email address for this project is: `contact+mygroup-myproject-support@example.com`. @@ -289,6 +289,8 @@ In these issues, you can also see our friendly neighborhood [Support Bot](#suppo ### As an end user (issue creator) +> Support for additional email headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346600) in GitLab 14.6. +> In earlier versions, the Service Desk email address had to be in the "To" field. To create a Service Desk issue, an end user does not need to know anything about the GitLab instance. They just send an email to the address they are given, and receive an email back confirming receipt: @@ -304,6 +306,9 @@ are sent as emails: Any responses they send via email are displayed in the issue itself. +For information about headers used for treating email, see +[the incoming email documentation](../../administration/incoming_email.md#accepted-headers). + ### As a responder to the issue For responders to the issue, everything works just like other GitLab issues. diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 74879dae2d6..da0336d01ff 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -44,8 +44,9 @@ Note the following: a maintainer or administrator role in the group where the exported project lives. - Project members with the [Owner role](../../permissions.md) are imported as Maintainers. - Imported users can be mapped by their public email on self-managed instances, if an administrative user (not an owner) does the import. - Additionally, the user must be an existing member of the namespace, or the user can be added as a -member of the project for contributions to be mapped. + The public email is not set by default. Users must [set it in their profiles](../../profile/index.md#set-your-public-email) + for mapping to work correctly. Additionally, the user must be an existing member of the namespace, + or the user can be added as a member of the project for contributions to be mapped. Otherwise, a supplementary comment is left to mention that the original author and the MRs, notes, or issues are owned by the importer. - For project migration imports performed over GitLab.com Groups, preserving author information is diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index 5c4d4649240..c6cbd45a6ab 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -7,18 +7,18 @@ type: reference, index, howto # Project settings **(FREE)** -NOTE: -Only project maintainers and administrators have the [permissions](../../permissions.md#project-members-permissions) -to access project settings. - The **Settings** page in GitLab provides a centralized home for your [project](../index.md) configuration options. To access it, go to your project's homepage -and, in the left navigation menu, clicking **Settings**. To reduce complexity, settings are -grouped by topic into sections. To display all settings in a section, click **Expand**. +and, in the left navigation menu, select **Settings**. To reduce complexity, settings are +grouped by topic into sections. To display all settings in a section, select **Expand**. In GitLab versions [13.10 and later](https://gitlab.com/groups/gitlab-org/-/epics/4842), GitLab displays a search box to help you find the settings you want to view. +NOTE: +Only users who have the [Maintainer role](../../permissions.md) for the project and administrators can +access project settings. + ## General settings Under a project's general settings, you can find everything concerning the @@ -315,7 +315,7 @@ Set up your project's merge request settings: - Enable [require an associated issue from Jira](../../../integration/jira/issues.md#require-associated-jira-issue-for-merge-requests-to-be-merged). - Enable [`delete source branch after merge` option by default](../merge_requests/getting_started.md#deleting-the-source-branch). - Configure [suggested changes commit messages](../merge_requests/reviews/suggestions.md#configure-the-commit-message-for-applied-suggestions). -- Configure [merge commit message template](../merge_requests/commit_templates.md). +- Configure [merge and squash commit message templates](../merge_requests/commit_templates.md). - Configure [the default target project](../merge_requests/creating_merge_requests.md#set-the-default-target-project) for merge requests coming from forks. ### Service Desk @@ -449,17 +449,22 @@ To delete a project: This action deletes a project including all associated resources (issues, merge requests, and so on). -In [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/issues/220382) and later, on Premium or higher tiers, -group Owners can [configure](../../group/index.md#enable-delayed-project-removal) projects in a group -to be deleted after a delayed period. -When enabled, actual deletion happens after number of days -specified in [instance settings](../../admin_area/settings/visibility_and_access_controls.md#default-deletion-delay). - WARNING: -The default behavior of [delayed project deletion](https://gitlab.com/gitlab-org/gitlab/-/issues/32935) in GitLab 12.6 was changed to -[Immediate deletion](https://gitlab.com/gitlab-org/gitlab/-/issues/220382) in GitLab 13.2. +The default deletion behavior for projects was changed to [delayed project deletion](https://gitlab.com/gitlab-org/gitlab/-/issues/32935) +in GitLab 12.6, and then to [immediate deletion](https://gitlab.com/gitlab-org/gitlab/-/issues/220382) in GitLab 13.2. + +#### Delayed project deletion **(PREMIUM)** + +Projects in a group (not a personal namespace) can be deleted after a delay period. Multiple settings can affect whether +delayed project deletion is enabled for a particular project: + +- Self-managed instance [settings](../../admin_area/settings/visibility_and_access_controls.md#default-delayed-project-deletion). + You can enable delayed project deletion as the default setting for new groups, and configure the number of days for the + delay. For GitLab.com, see the [GitLab.com settings](../../gitlab_com/index.md#delayed-project-deletion). +- Group [settings](../../group/index.md#enable-delayed-project-deletion) to enabled delayed project deletion for all + projects in the group. -#### Delete a project immediately **(PREMIUM)** +##### Delete a project immediately > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/191367) in GitLab 14.1. diff --git a/doc/user/project/settings/project_access_tokens.md b/doc/user/project/settings/project_access_tokens.md index 85e412e7a41..44ece6cb172 100644 --- a/doc/user/project/settings/project_access_tokens.md +++ b/doc/user/project/settings/project_access_tokens.md @@ -12,111 +12,91 @@ type: reference, howto > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235765) in GitLab 13.5. > - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/342327) in GitLab 14.5. Default prefix added. -Project access tokens are similar to [personal access tokens](../../profile/personal_access_tokens.md) -except they are attached to a project rather than a user. They can be used to: +You can use a project access token to authenticate: -- Authenticate with the [GitLab API](../../../api/index.md#personalproject-access-tokens). -- Authenticate with Git using HTTP Basic Authentication. If you are asked for a username when - authenticating, you can use any non-empty value because only the token is needed. +- With the [GitLab API](../../../api/index.md#personalproject-access-tokens). +- With Git, when using HTTP Basic Authentication. -Project access tokens: +After you configure a project access token, you don't need a password when you authenticate. +Instead, you can enter any non-blank value. -- Expire on the date you define, at midnight UTC. -- Are supported for self-managed instances on Free tier and above. Free self-managed instances - should: - - Review their security and compliance policies with regards to +Project access tokens are similar to [personal access tokens](../../profile/personal_access_tokens.md), +except they are associated with a project rather than a user. + +You can use project access tokens: + +- On GitLab SaaS if you have the Premium license tier or higher. Personal access tokens are not available with a [trial license](https://about.gitlab.com/free-trial/). +- On self-managed instances of GitLab, with any license tier. If you have the Free tier: + - Review your security and compliance policies around [user self-enrollment](../../admin_area/settings/sign_up_restrictions.md#disable-new-sign-ups). - Consider [disabling project access tokens](#enable-or-disable-project-access-token-creation) to lower potential abuse. -- Are also supported on GitLab SaaS Premium and above (excluding [trial licenses](https://about.gitlab.com/free-trial/).) -For examples of how you can use a project access token to authenticate with the API, see the -[relevant section from our API Docs](../../../api/index.md#personalproject-access-tokens). +Project access tokens inherit the [default prefix setting](../../admin_area/settings/account_and_limit_settings.md#personal-access-token-prefix) +configured for personal access tokens. -NOTE: -For GitLab.com and self-managed instances, the default prefix is `glpat-`. +## Create a project access token -## Creating a project access token +To create a project access token: -1. Log in to GitLab. -1. Navigate to the project you would like to create an access token for. -1. In the **Settings** menu choose **Access Tokens**. -1. Choose a name and optional expiry date for the token. -1. Choose a role for the token. -1. Choose the [desired scopes](#limiting-scopes-of-a-project-access-token). -1. Click the **Create project access token** button. -1. Save the project access token somewhere safe. Once you leave or refresh - the page, you don't have access to it again. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Access Tokens**. +1. Enter a name. +1. Optional. Enter an expiry date for the token. The token will expire on that date at midnight UTC. +1. Select a role for the token. +1. Select the [desired scopes](#scopes-for-a-project-access-token). +1. Select **Create project access token**. -## Project bot users +A project access token is displayed. Save the project access token somewhere safe. After you leave or refresh the page, you can't view it again. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210181) in GitLab 13.0. -> - [Excluded from license seat use](https://gitlab.com/gitlab-org/gitlab/-/issues/223695) in GitLab 13.5. +## Revoke a project access token -Project bot users are [GitLab-created service accounts](../../../subscriptions/self_managed/index.md#billable-users) and do not count as licensed seats. +To revoke a project access token: -For each project access token created, a bot user is created and added to the project with -the [specified level permissions](../../permissions.md#project-members-permissions). - -For the bot: - -- The name is set to the name of the token. -- The username is set to `project_{project_id}_bot` for the first access token, such as `project_123_bot`. -- The email is set to `project{project_id}_bot@example.com`, for example `project123_bot@example.com`. -- For additional access tokens in the same project, the username is set to `project_{project_id}_bot{bot_count}`, for example `project_123_bot1`. -- For additional acess tokens in the same project, the email is set to `project{project_id}_bot{bot_count}@example.com`, for example `project123_bot1@example.com` - -API calls made with a project access token are associated with the corresponding bot user. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Access Tokens**. +1. Next to the project access token to revoke, select **Revoke**. -These bot users are included in a project's **Project information > Members** list but cannot be modified. Also, a bot -user cannot be added to any other project. +## Scopes for a project access token -When the project access token is [revoked](#revoking-a-project-access-token), the bot user is deleted -and all records are moved to a system-wide user with the username "Ghost User". For more -information, see [Associated Records](../../profile/account/delete_account.md#associated-records). +The scope determines the actions you can perform when you authenticate with a project access token. -## Revoking a project access token - -At any time, you can revoke any project access token by clicking the -respective **Revoke** button in **Settings > Access Tokens**. - -## Limiting scopes of a project access token - -Project access tokens can be created with one or more scopes that allow various -actions that a given token can perform. The available scopes are depicted in -the following table. - -| Scope | Description | -| ------------------ | ----------- | -| `api` | Grants complete read/write access to the scoped project API, including the [Package Registry](../../packages/package_registry/index.md). | -| `read_api` | Grants read access to the scoped project API, including the [Package Registry](../../packages/package_registry/index.md). | -| `read_registry` | Allows read-access (pull) to [container registry](../../packages/container_registry/index.md) images if a project is private and authorization is required. | -| `write_registry` | Allows write-access (push) to [container registry](../../packages/container_registry/index.md). | -| `read_repository` | Allows read-only access (pull) to the repository. | -| `write_repository` | Allows read-write access (pull, push) to the repository. | +| Scope | Description | +|:-------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `api` | Grants complete read and write access to the scoped project API, including the [Package Registry](../../packages/package_registry/index.md). | +| `read_api` | Grants read access to the scoped project API, including the [Package Registry](../../packages/package_registry/index.md). | +| `read_registry` | Allows read access (pull) to the [Container Registry](../../packages/container_registry/index.md) images if a project is private and authorization is required. | +| `write_registry` | Allows write access (push) to the [Container Registry](../../packages/container_registry/index.md). | +| `read_repository` | Allows read access (pull) to the repository. | +| `write_repository` | Allows read and write access (pull and push) to the repository. | ## Enable or disable project access token creation > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/287707) in GitLab 13.11. -You may enable or disable project access token creation for all projects in a group in **Group > Settings > General > Permissions, LFS, 2FA > Allow project access token creation**. -Even when creation is disabled, you can still use and revoke existing project access tokens. -This setting is available only on top-level groups. +To enable or disable project access token creation for all projects in a top-level group: -## Group access token workaround **(FREE SELF)** +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > General**. +1. Expand **Permissions, LFS, 2FA**. +1. Under **Permissions**, turn on or off **Allow project access token creation**. -NOTE: -This section describes a workaround and is subject to change. +Even when creation is disabled, you can still use and revoke existing project access tokens. + +## Group access tokens **(FREE SELF)** -Group access tokens let you use a single token to: +With group access tokens, you can use a single token to: -- Perform actions at the group level. +- Perform actions for groups. - Manage the projects within the group. -- In [GitLab 14.2](https://gitlab.com/gitlab-org/gitlab/-/issues/330718) and later, authenticate - with Git over HTTPS. +- In [GitLab 14.2](https://gitlab.com/gitlab-org/gitlab/-/issues/330718) and later, authenticate with Git over HTTPS. + +NOTE: +You cannot use the UI to create a group access token. [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/214045) +to add this functionality. This section describes a workaround. -We don't support group access tokens in the GitLab UI, though GitLab self-managed -administrators can create them using the [Rails console](../../../administration/operations/rails_console.md). +If you are an administrator of a self-managed GitLab instance, you can create a group access token in the +[Rails console](../../../administration/operations/rails_console.md). <div class="video-fallback"> For a demo of the group access token workaround, see <a href="https://www.youtube.com/watch?v=W2fg1P1xmU0">Demo: Group Level Access Tokens</a>. @@ -127,37 +107,83 @@ administrators can create them using the [Rails console](../../../administration ### Create a group access token -To create a group access token, run the following in a Rails console: +To create a group access token: -```ruby -admin = User.find(1) # group admin -group = Group.find(109) # the group you want to create a token for -bot = Users::CreateService.new(admin, { name: 'group_token', username: "group_#{group.id}_bot", email: "group_#{group.id}_bot@example.com", user_type: :project_bot }).execute # create the group bot user -# for further group access tokens, the username should be group_#{group.id}_bot#{bot_count}, e.g. group_109_bot2, and their email should be group_109_bot2@example.com -bot.confirm # confirm the bot -group.add_user(bot, :maintainer) # add the bot to the group at the desired access level -token = bot.personal_access_tokens.create(scopes:[:api, :write_repository], name: 'group_token') # give it a PAT -gtoken = token.token # get the token value -``` +1. Run the following commands in a [Rails console](../../../administration/operations/rails_console.md): + + ```ruby + # Set the GitLab administration user to use. If user ID 1 is not available or is not an adinistrator, use 'admin = User.admins.first' instead to select an admininistrator. + admin = User.find(1) + + # Set the group group you want to create a token for. For example, group with ID 109. + group = Group.find(109) + + # Create the group bot user. For further group access tokens, the username should be group_#{group.id}_bot#{bot_count}. For example, group_109_bot2 and email address group_109_bot2@example.com. + bot = Users::CreateService.new(admin, { name: 'group_token', username: "group_#{group.id}_bot", email: "group_#{group.id}_bot@example.com", user_type: :project_bot }).execute + + # Confirm the group bot. + bot.confirm + + # Add the bot to the group with the required role. + group.add_user(bot, :maintainer) -Test if the generated group access token works: + # Give the bot a personal access token. + token = bot.personal_access_tokens.create(scopes:[:api, :write_repository], name: 'group_token') -1. Pass the group access token in the `PRIVATE-TOKEN` header to GitLab REST APIs. For example: + # Get the token value. + gtoken = token.token + ``` - - [Create an epic](../../../api/epics.md#new-epic) on the group. - - [Create a project pipeline](../../../api/pipelines.md#create-a-new-pipeline) - in one of the group's projects. - - [Create an issue](../../../api/issues.md#new-issue) in one of the group's projects. +1. Test if the generated group access token works: -1. Use the group token to [clone a group's project](../../../gitlab-basics/start-using-git.md#clone-with-https) - using HTTPS. + 1. Use the group access token in the `PRIVATE-TOKEN` header with GitLab REST APIs. For example: + + - [Create an epic](../../../api/epics.md#new-epic) in the group. + - [Create a project pipeline](../../../api/pipelines.md#create-a-new-pipeline) in one of the group's projects. + - [Create an issue](../../../api/issues.md#new-issue) in one of the group's projects. + + 1. Use the group token to [clone a group's project](../../../gitlab-basics/start-using-git.md#clone-with-https) + using HTTPS. ### Revoke a group access token -To revoke a group access token, run the following in a Rails console: +To revoke a group access token, run the following command in a [Rails console](../../../administration/operations/rails_console.md): ```ruby bot = User.find_by(username: 'group_109_bot') # the owner of the token you want to revoke token = bot.personal_access_tokens.last # the token you want to revoke token.revoke! ``` + +## Project bot users + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210181) in GitLab 13.0. +> - [Excluded from license seat use](https://gitlab.com/gitlab-org/gitlab/-/issues/223695) in GitLab 13.5. + +Project bot users are [GitLab-created service accounts](../../../subscriptions/self_managed/index.md#billable-users). +Each time you create a project access token, a bot user is created and added to the project. +These bot users do not count as licensed seats. + +The bot users have [permissions](../../permissions.md#project-members-permissions) that correspond with the +selected role and [scope](#scopes-for-a-project-access-token) of the project access token. + +- The name is set to the name of the token. +- The username is set to `project_{project_id}_bot` for the first access token. For example, `project_123_bot`. +- The email is set to `project{project_id}_bot@example.com`. For example, `project123_bot@example.com`. +- For additional access tokens in the same project, the username is set to `project_{project_id}_bot{bot_count}`. For + example, `project_123_bot1`. +- For additional access tokens in the same project, the email is set to `project{project_id}_bot{bot_count}@example.com`. + For example, `project123_bot1@example.com`. + +API calls made with a project access token are associated with the corresponding bot user. + +Bot users: + +- Are included in a project's member list but cannot be modified. +- Cannot be added to any other project. + +When the project access token is [revoked](#revoke-a-project-access-token): + +- The bot user is deleted. +- All records are moved to a system-wide user with the username `Ghost User`. For more information, see + [associated records](../../profile/account/delete_account.md#associated-records). diff --git a/doc/user/project/static_site_editor/index.md b/doc/user/project/static_site_editor/index.md index 431250a817d..50b1a3a929d 100644 --- a/doc/user/project/static_site_editor/index.md +++ b/doc/user/project/static_site_editor/index.md @@ -72,11 +72,11 @@ First, set up the project. Once done, you can use the Static Site Editor to 1. To get started, create a new project from the [Static Site Editor - Middleman](https://gitlab.com/gitlab-org/project-templates/static-site-editor-middleman) template. You can either [fork it](../repository/forking_workflow.md#creating-a-fork) - or [create a new project from a template](../working_with_projects.md#built-in-templates). + or [create a new project from a template](../working_with_projects.md#create-a-project-from-a-built-in-template). 1. Edit the [`data/config.yml`](#static-site-generator-configuration) configuration file to replace `<username>` and `<project-name>` with the proper values for your project's path. -1. (Optional) Edit the [`.gitlab/static-site-editor.yml`](#static-site-editor-configuration-file) file +1. Optional. Edit the [`.gitlab/static-site-editor.yml`](#static-site-editor-configuration-file) file to customize the behavior of the Static Site Editor. 1. When you submit your changes, GitLab triggers a CI/CD pipeline to deploy your project with GitLab Pages. 1. When the pipeline finishes, from your project's left-side menu, go to **Settings > Pages** to find the URL of your new website. @@ -96,15 +96,15 @@ After setting up your project, you can start editing content directly from the S To edit a file: 1. Visit the page you want to edit. -1. Click the **Edit this page** button. +1. Select **Edit this page**. 1. The file is opened in the Static Site Editor in **WYSIWYG** mode. If you wish to edit the raw Markdown instead, you can toggle the **Markdown** mode in the bottom-right corner. 1. When you're done, click **Submit changes...**. -1. (Optional) Adjust the default title and description of the merge request, to submit +1. Optional. Adjust the default title and description of the merge request, to submit with your changes. Alternatively, select a [merge request template](../../../user/project/description_templates.md#create-a-merge-request-template) from the dropdown menu and edit it accordingly. -1. Click **Submit changes**. +1. Select **Submit changes**. 1. A new merge request is automatically created and you can assign a colleague for review. ### Text @@ -123,11 +123,11 @@ The Static Site Editors supports Markdown files (`.md`, `.md.erb`) for editing t You can upload image files via the WYSIWYG editor directly to the repository to default upload directory `source/images`. To do so: -1. Click the image icon (**{doc-image}**). -1. Choose the **Upload file** tab. -1. Click **Choose file** to select a file from your computer. -1. Optional: add a description to the image for SEO and accessibility ([ALT text](https://moz.com/learn/seo/alt-text)). -1. Click **Insert image**. +1. Select the image icon (**{doc-image}**). +1. Select the **Upload file** tab. +1. To select a file from your computer, select **Choose file**. +1. Optional. Add a description to the image for SEO and accessibility ([ALT text](https://moz.com/learn/seo/alt-text)). +1. Select **Insert image**. The selected file can be any supported image file (`.png`, `.jpg`, `.jpeg`, `.gif`). The editor renders thumbnail previews so you can verify the correct image is included and there aren't any references to @@ -137,11 +137,11 @@ missing images. You can also link to an image if you'd like: -1. Click the image icon (**{doc-image}**). -1. Choose the **Link to an image** tab. +1. Select the image icon (**{doc-image}**). +1. Select the **Link to an image** tab. 1. Add the link to the image into the **Image URL** field (use the full path; relative paths are not supported yet). -1. Optional: add a description to the image for SEO and accessibility ([ALT text](https://moz.com/learn/seo/alt-text)). -1. Click **Insert image**. +1. Optional. Add a description to the image for SEO and accessibility ([ALT text](https://moz.com/learn/seo/alt-text)). +1. Select **Insert image**. The link can reference images already hosted in your project, an asset hosted externally on a content delivery network, or any other external URL. The editor renders thumbnail previews diff --git a/doc/user/project/time_tracking.md b/doc/user/project/time_tracking.md index 8902bdc21c4..b41ea30bfef 100644 --- a/doc/user/project/time_tracking.md +++ b/doc/user/project/time_tracking.md @@ -123,7 +123,7 @@ To do so: With this option enabled, `75h` is displayed instead of `1w 4d 3h`. -## Related links +## Related topics - [Time tracking solutions page](https://about.gitlab.com/solutions/time-tracking/) - Time tracking GraphQL references: diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index d75a180492e..40c9ae8bd59 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -456,7 +456,7 @@ when: - <kbd>Control</kbd> + <kbd>S</kbd> (or <kbd>Command</kbd> + <kbd>S</kbd> on Mac) is pressed while editing a file. -- Anything outside the file editor is clicked after editing a file. +- You select any area outside the file editor after editing a file. - A file or folder is created, deleted, or renamed. ### Limitations diff --git a/doc/user/project/wiki/img/content_editor_v14.0.png b/doc/user/project/wiki/img/content_editor_v14.0.png Binary files differdeleted file mode 100644 index b44a633073d..00000000000 --- a/doc/user/project/wiki/img/content_editor_v14.0.png +++ /dev/null diff --git a/doc/user/project/wiki/img/content_editor_v14.6.png b/doc/user/project/wiki/img/content_editor_v14.6.png Binary files differnew file mode 100644 index 00000000000..55fca0ace1e --- /dev/null +++ b/doc/user/project/wiki/img/content_editor_v14.6.png diff --git a/doc/user/project/wiki/img/use_new_editor_button_v14.0.png b/doc/user/project/wiki/img/use_new_editor_button_v14.0.png Binary files differdeleted file mode 100644 index d9a5cf83302..00000000000 --- a/doc/user/project/wiki/img/use_new_editor_button_v14.0.png +++ /dev/null diff --git a/doc/user/project/wiki/img/use_new_editor_button_v14.6.png b/doc/user/project/wiki/img/use_new_editor_button_v14.6.png Binary files differnew file mode 100644 index 00000000000..078fed8a1e9 --- /dev/null +++ b/doc/user/project/wiki/img/use_new_editor_button_v14.6.png diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index 5d2a0530f68..9f1670d3f4c 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -87,7 +87,7 @@ Users with the [Developer role](../../permissions.md) can create new wiki pages: [special characters](#special-characters-in-page-titles) for subdirectories and formatting, and have [length restrictions](#length-restrictions-for-file-and-directory-names). 1. Add content to your wiki page. -1. (Optional) Attach a file, and GitLab stores it according to your installed version of GitLab: +1. Optional. Attach a file, and GitLab stores it according to your installed version of GitLab: - *Files added in [GitLab 11.3 and later](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33475):* Files are stored in the wiki's Git repository. - *Files added GitLab 11.2 and earlier:* Files are stored in GitLab itself. To add @@ -236,7 +236,7 @@ GitLab tracks wiki creation, deletion, and update events. These events are displ - [User profile](../../profile/index.md#access-your-user-profile). - Activity pages, depending on the type of wiki: - [Group activity](../../group/index.md#view-group-activity). - - [Project activity](../working_with_projects.md#project-activity). + - [Project activity](../working_with_projects.md#view-project-activity). Commits to wikis are not counted in [repository analytics](../../analytics/repository_analytics.md). @@ -290,7 +290,7 @@ To add a link to an external wiki from a project's left sidebar: 1. On the left sidebar, select **Settings > Integrations**. 1. Select **External wiki**. 1. Add the URL to your external wiki. -1. (Optional) Select **Test settings** to verify the connection. +1. Optional. To verify the connection, select **Test settings**. 1. Select **Save changes**. You can now see the **External wiki** option from your project's @@ -339,27 +339,24 @@ experience in the Wiki. To opt in for the new editor: 1. Create a new wiki page, or edit an existing one. 1. Ensure the wiki page uses the Markdown format. Other formats are not yet supported. -1. Below the **Format** select box, select **Use the new editor**: +1. Above the content field, select **Edit rich text**: -  +  ### Use the Content Editor 1. [Create](#create-a-new-wiki-page) a new wiki page, or [edit](#edit-a-wiki-page) an existing one. 1. Select **Markdown** as your format. -1. Below the **Format** select box, select **Use new editor**. +1. Above **Content**, select **Edit rich text**. 1. Customize your page's content using the various formatting options available in the content editor. 1. Select **Create page** for a new page, or **Save changes** for an existing page: -  +  ### Switch back to the old editor 1. *If you're editing the page in the content editor,* scroll to **Content**. -1. Select **Switch me back to the classic editor**. -1. Select **Switch to classic editor** in the confirmation popup to confirm. - -When you switch back to the old editor, any unsaved changes are lost. +1. Select **Edit source**. ### GitLab Flavored Markdown support diff --git a/doc/user/project/working_with_projects.md b/doc/user/project/working_with_projects.md index 3c82e544e26..b5b3f2d2085 100644 --- a/doc/user/project/working_with_projects.md +++ b/doc/user/project/working_with_projects.md @@ -4,35 +4,59 @@ group: Workspace info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" --- -# Working with projects **(FREE)** +# Manage projects **(FREE)** Most work in GitLab is done in a [project](../../user/project/index.md). Files and code are saved in projects, and most features are in the scope of projects. -## Explore projects +## View projects -You can explore other popular projects available on GitLab. To explore projects: +To explore projects: 1. On the top bar, select **Menu > Projects**. 1. Select **Explore projects**. -GitLab displays a list of projects, sorted by last updated date. To view -projects with the most [stars](#star-a-project), click **Most stars**. To view -projects with the largest number of comments in the past month, click **Trending**. +The **Projects** page shows a list of projects, sorted by last updated date. + +- To view projects with the most [stars](#star-a-project), select **Most stars**. +- To view projects with the largest number of comments in the past month, select **Trending**. NOTE: -By default, `/explore` is visible to unauthenticated users. However, if the +The **Explore projects** tab is visible to unauthenticated users unless the [**Public** visibility level](../admin_area/settings/visibility_and_access_controls.md#restrict-visibility-levels) -is restricted, `/explore` is visible only to signed-in users. +is restricted. Then the tab is visible only to signed-in users. + +### Who can view the **Projects** page + +When you select a project, the project landing page shows the project contents. + +For public projects, and members of internal and private projects +with [permissions to view the project's code](../permissions.md#project-members-permissions), +the project landing page shows: + +- A [`README` or index file](repository/index.md#readme-and-index-files). +- A list of directories in the project's repository. + +For users without permission to view the project's code, the landing page shows: + +- The wiki homepage. +- The list of issues in the project. + +### Access a project page with the project ID + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53671) in GitLab 11.8. + +To access a project from the GitLab UI using the project ID, +visit the `/projects/:id` URL in your browser or other tool accessing the project. ## Explore topics -You can explore popular project topics available on GitLab. To explore project topics: +To explore project topics: 1. On the top bar, select **Menu > Projects**. 1. Select **Explore topics**. -GitLab displays a list of topics sorted by the number of associated projects. +The **Projects** page shows list of topics sorted by the number of associated projects. To view projects associated with a topic, select a topic from the list. You can assign topics to a project on the [Project Settings page](settings/index.md#topics). @@ -44,290 +68,280 @@ If you're an instance administrator, you can administer all project topics from To create a project in GitLab: -1. In your dashboard, select **New project** or use the **New...** **{plus-square}** icon -on the top bar. The **New project** page opens. +1. On the top bar, select **Menu > Project**. +1. Select **Create new project**. 1. On the **New project** page, choose if you want to: - - Create a [blank project](#blank-projects). - - Create a project using one of the available [project templates](#project-templates). - - [Import a project](../../user/project/import/index.md) from a different repository, - if enabled on your GitLab instance. Contact your GitLab administrator if this is unavailable. - - Run [CI/CD pipelines for external repositories](../../ci/ci_cd_for_external_repos/index.md). **(PREMIUM)** + - Create a [blank project](#create-a-blank-project). + - Create a project from a: + - [built-in template](#create-a-project-from-a-built-in-template). + - [custom template](#create-a-project-from-a-custom-template). + - [HIPAA audit protocol template](#create-a-project-from-the-hipaa-audit-protocol-template). + - [Import a project](../../user/project/import/index.md) + from a different repository. Contact your GitLab administrator if this option is not available. + - [Connect an external repository to GitLab CI/CD](../../ci/ci_cd_for_external_repos/index.md). NOTE: For a list of words that can't be used as project names see -[Reserved project and group names](../../user/reserved_names.md). - -### Blank projects - -To create a new blank project on the **New project** page: - -1. Click **Create blank project** -1. Provide the following information: - - The name of your project in the **Project name** field. You can't use - special characters, but you can use spaces, hyphens, underscores, or even - emoji. When adding the name, the **Project slug** auto populates. - The slug is what the GitLab instance uses as the URL path to the project. - If you want a different slug, input the project name first, - then change the slug after. - - The path to your project in the **Project slug** field. This is the URL - path for your project that the GitLab instance uses. If the - **Project name** is blank, it auto populates when you fill in - the **Project slug**. - - The **Project description (optional)** field enables you to enter a - description for your project's dashboard, which helps others - understand what your project is about. Though it's not required, it's a good - idea to fill this in. - - Changing the **Visibility Level** modifies the project's - [viewing and access rights](../../public_access/public_access.md) for users. - - Selecting the **Initialize repository with a README** option creates a - README file so that the Git repository is initialized, has a default branch, and - can be cloned. -1. Click **Create project**. - -### Project templates - -Project templates can pre-populate a new project with the necessary files to get you -started quickly. - -There are two main types of project templates: - -- [Built-in templates](#built-in-templates), sourced from the following groups: - - [`project-templates`](https://gitlab.com/gitlab-org/project-templates) - - [`pages`](https://gitlab.com/pages) -- [Custom project templates](#custom-project-templates), for custom templates - configured by GitLab administrators and users. - -#### Built-in templates - -Built-in templates are project templates that are: - -- Developed and maintained in the [`project-templates`](https://gitlab.com/gitlab-org/project-templates) - and [`pages`](https://gitlab.com/pages) groups. -- Released with GitLab. -- Anyone can contribute a built-in template by following [these steps](https://about.gitlab.com/community/contribute/project-templates/). - -To use a built-in template on the **New project** page: - -1. Click **Create from template** +[reserved project and group names](../../user/reserved_names.md). + +## Create a blank project + +To create a blank project: + +1. On the top bar, select **Menu > Project**. +1. Select **Create new project**. +1. Select **Create blank project**. +1. Enter the project details: + - In the **Project name** field, enter the name of your project. You can use spaces, hyphens, + underscores, and emoji. You cannot use special characters. After you enter the name, + the **Project slug** populates. + - In the **Project slug** field, enter the path to your project. The GitLab instance uses the + slug as the URL path to the project. To change the slug, first enter the project name, + then change the slug. + - In the **Project description (optional)** field, enter the description of your project's dashboard. + - To modify the project's [viewing and access rights](../../public_access/public_access.md) for + users, change the **Visibility Level**. + - To create README file so that the Git repository is initialized, has a default branch, and + can be cloned, select **Initialize repository with a README**. + - To analyze the source code in the project for known security vulnerabilities, + select **Enable Static Application Security Testing (SAST)**. +1. Select **Create project**. + +## Create a project from a built-in template + +A built-in project template populates a new project with files to get you started. +Built-in templates are sourced from the following groups: + +- [`project-templates`](https://gitlab.com/gitlab-org/project-templates) +- [`pages`](https://gitlab.com/pages) + +Anyone can contribute a built-in template by following [these steps](https://about.gitlab.com/community/contribute/project-templates/). + +To create a project from a built-in template: + +1. On the top bar, select **Menu > Project**. +1. Select **Create new project**. +1. Select **Create from template**. 1. Select the **Built-in** tab. -1. From the list of available built-in templates, click the: - - **Preview** button to look at the template source itself. - - **Use template** button to start creating the project. -1. Finish creating the project by filling out the project's details. The process is - the same as creating a [blank project](#blank-projects). +1. From the list of templates: + - To view a preview of the template, select **Preview**. + - To use a template for the project, select **Use template**. +1. Enter the project details: + - In the **Project name** field, enter the name of your project. You can use spaces, hyphens, + underscores, and emoji. You cannot use special characters. After you enter the name, + the **Project slug** populates. + - In the **Project slug** field, enter the path to your project. The GitLab instance uses the + slug as the URL path to the project. To change the slug, first enter the project name, + then change the slug. + - In the **Project description (optional)** field, enter the description of your project's dashboard. + - To modify the project's [viewing and access rights](../../public_access/public_access.md) for users, + change the **Visibility Level**. +1. Select **Create project**. + +## Create a project from a custom template **(PREMIUM)** -##### Enterprise templates **(ULTIMATE)** - -GitLab is developing Enterprise templates to help you streamline audit management with selected regulatory standards. These templates automatically import issues that correspond to each regulatory requirement. - -To create a new project with an Enterprise template, on the **New project** page: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6860) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.2. -1. Click **Create from template** +Custom project templates are available at: + +- The [instance-level](../../user/admin_area/custom_project_templates.md) +- The [group-level](../../user/group/custom_project_templates.md) + +1. On the top bar, select **Menu > Project**. +1. Select **Create new project**. +1. Select **Create from template**. +1. Select the **Instance** or **Group** tab. +1. From the list of templates: + - To view a preview of the template, select **Preview**. + - To use a template for the project, select **Use template**. +1. Enter the project details: + - In the **Project name** field, enter the name of your project. You can use spaces, hyphens, + underscores, and emoji. You cannot use special characters. After you enter the name, + the **Project slug** populates. + - In the **Project slug** field, enter the path to your project. The GitLab instance uses the + slug as the URL path to the project. To change the slug, first enter the project name, + then change the slug. + - The description of your project's dashboard in the **Project description (optional)** field. + - To modify the project's [viewing and access rights](../../public_access/public_access.md) for users, + change the **Visibility Level**. +1. Select **Create project**. + +## Create a project from the HIPAA Audit Protocol template **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13756) in GitLab 12.10 + +The HIPAA Audit Protocol template contains issues for audit inquiries in the +HIPAA Audit Protocol published by the U.S Department of Health and Human Services. + +To create a project from the HIPAA Audit Protocol template: + +1. On the top bar, select **Menu > Project**. +1. Select **Create new project**. +1. Select **Create from template**. 1. Select the **Built-in** tab. -1. From the list of available built-in Enterprise templates, click the: - - **Preview** button to look at the template source itself. - - **Use template** button to start creating the project. -1. Finish creating the project by filling out the project's details. The process is the same as creating a [blank project](#blank-projects). +1. Locate the **HIPAA Audit Protocol** template: + - To view a preview of the template, select **Preview**. + - To use the template for the project, select **Use template**. +1. Enter the project details: + - In the **Project name** field, enter the name of your project. You can use spaces, hyphens, + underscores, and emoji. You cannot use special characters. After you enter the name, + the **Project slug** populates. + - In the **Project slug** field, enter the path to your project. The GitLab instance uses the + slug as the URL path to the project. To change the slug, first enter the project name, + then change the slug. + - In the **Project description (optional)** field, enter the description of your project's dashboard. + - To modify the project's [viewing and access rights](../../public_access/public_access.md) for users, + change the **Visibility Level**. +1. Select **Create project**. + +## Push to create a new project -Available Enterprise templates include: +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/26388) in GitLab 10.5. -- HIPAA Audit Protocol template ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13756) in GitLab 12.10) +Use `git push` to push a local project repository to GitLab. After you push a repository, +GitLab creates your project in your chosen namespace. -NOTE: -You can improve the existing built-in templates or contribute new ones in the -[`project-templates`](https://gitlab.com/gitlab-org/project-templates) and -[`pages`](https://gitlab.com/pages) groups by following [these steps](https://gitlab.com/gitlab-org/project-templates/contributing). +You cannot use `git push` to create projects with project paths that: -##### Custom project templates **(PREMIUM)** +- Have previously been used. +- Have been [renamed](settings/index.md#renaming-a-repository). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6860) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.2. +Previously used project paths have a redirect. The redirect causes push attempts to redirect requests +to the renamed project location, instead of creating a new project. To create a new project for a previously +used or renamed project, use the [UI](#create-a-project) or the [Projects API](../../api/projects.md#create-project). -Creating new projects based on custom project templates is a convenient option for -quickly starting projects. +Prerequisites: -Custom projects are available at the [instance-level](../../user/admin_area/custom_project_templates.md) -from the **Instance** tab, or at the [group-level](../../user/group/custom_project_templates.md) -from the **Group** tab, on the **Create from template** page. +- To push with SSH, you must have [an SSH key](../../ssh/index.md) that is +[added to your GitLab account](../../ssh/index.md#add-an-ssh-key-to-your-gitlab-account). +- You must have permission to add new projects to a namespace. To check if you have permission: -To use a custom project template on the **New project** page: + 1. On the top bar, select **Menu > Project**. + 1. Select **Groups**. + 1. Select a group. + 1. Confirm that **New project** is visible in the upper right + corner. Contact your GitLab + administrator if you require permission. -1. Click **Create from template** -1. Select the **Instance** tab or the **Group** tab. -1. From the list of available custom templates, click the: - - **Preview** button to look at the template source itself. - - **Use template** button to start creating the project. -1. Finish creating the project by filling out the project's details. The process is - the same as creating a [blank project](#blank-projects). +To push your repository and create a project: -## Push to create a new project +1. Push with SSH or HTTPS: + - To push with SSH: -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/26388) in GitLab 10.5. + ```shell + git push --set-upstream git@gitlab.example.com:namespace/myproject.git master + ``` -When you create a new repository locally, you don't have to sign in to the GitLab -interface to create a project and -[clone its repository](../../gitlab-basics/start-using-git.md#clone-a-repository). -You can directly push your new repository to GitLab, which creates your new project -without leaving your terminal. - -To push a new project: - -1. Identify the [namespace](../group/index.md#namespaces) you want to add the new - project to, as you need this information in a future step. To determine if you have - permission to create new projects in a namespace, view the group's page in a - web browser and confirm the page displays a **New project** button. - - NOTE: - As project creation permissions can have many factors, contact your - GitLab administrator if you're unsure. - -1. If you want to push using SSH, ensure you have [created a SSH key](../../ssh/index.md) and - [added it to your GitLab account](../../ssh/index.md#add-an-ssh-key-to-your-gitlab-account). -1. Push with one of the following methods. Replace `gitlab.example.com` with the - domain name of the machine that hosts your Git repository, `namespace` with the name of - your namespace, and `myproject` with the name of your new project: - - To push with SSH: `git push --set-upstream git@gitlab.example.com:namespace/myproject.git master` - - To push with HTTPS: `git push --set-upstream https://gitlab.example.com/namespace/myproject.git master` - Optional: to export existing repository tags, append the `--tags` flag to your `git push` command. -1. When the push completes, GitLab displays a message: - - ```plaintext - remote: The private project namespace/myproject was created. - ``` + - To push with HTTPS: -1. (Optional) To configure the remote, alter the command - `git remote add origin https://gitlab.example.com/namespace/myproject.git` - to match your namespace and project names. + ```shell + git push --set-upstream https://gitlab.example.com/namespace/myproject.git master + ``` -You can view your new project at `https://gitlab.example.com/namespace/myproject`. -Your project's visibility is set to **Private** by default, but you can change it -in your [project's settings](../../public_access/public_access.md#change-project-visibility)). + - For `gitlab.example.com`, use the domain name of the machine that hosts your Git repository. + - For `namespace`, use the name of your [namespace](../group/index.md#namespaces). + - For `myproject`, use the name of your project. + - Optional. To export existing repository tags, append the `--tags` flag to your `git push` command. +1. Optional. To configure the remote: -This feature does not work for project paths that have previously been in use and -[renamed](settings/index.md#renaming-a-repository). A redirect exists over the previous project path -that causes push attempts to redirect requests to the renamed project location, instead of creating -a new project. To create a new project, use the [Web UI](#create-a-project) or the -[Projects API](../../api/projects.md#create-project). + ```shell + git remote add origin https://gitlab.example.com/namespace/myproject.git + ``` -## Fork a project +When the push completes, GitLab displays the message: -A fork is a copy of an original repository that you put in another namespace -where you can experiment and apply changes that you can later decide whether or -not to share, without affecting the original project. +```shell +remote: The private project namespace/myproject was created. +``` -It takes just a few steps to [fork a project in GitLab](repository/forking_workflow.md#creating-a-fork). +To view your new project, go to `https://gitlab.example.com/namespace/myproject`. +Your project's visibility is set to **Private** by default. To change project visibility, adjust your +[project's settings](../../public_access/public_access.md#change-project-visibility). ## Star a project -You can star a project to make it easier to find projects you frequently use. -The number of stars a project has can indicate its popularity. +You can add a star to projects you use frequently to make them easier to find. -To star a project: +To add a star to a project: -1. Go to the home page of the project you want to star. -1. In the upper right corner of the page, click **Star**. +1. On the top bar, select **Menu > Project**. +1. Select **Your projects** or **Explore projects**. +1. Select a project. +1. In the upper right corner of the page, select **Star**. -To view your starred projects: +## View starred projects -1. On the top bar, select **Menu > Projects**. -1. Select **Starred Projects**. +1. On the top bar, select **Menu > Project**. +1. Select **Starred projects**. 1. GitLab displays information about your starred projects, including: - - Project description, including name, description, and icon - - Number of times this project has been starred - - Number of times this project has been forked - - Number of open merge requests - - Number of open issues + - Project description, including name, description, and icon. + - Number of times this project has been starred. + - Number of times this project has been forked. + - Number of open merge requests. + - Number of open issues. ## Delete a project -To delete a project, first navigate to the home page for that project. +After you delete a project, projects in personal namespaces are deleted immediately. To delay deletion of projects in a group +you can [enable delayed project removal](../group/index.md#enable-delayed-project-deletion). -1. Navigate to **Settings > General**. +To delete a project: + +1. On the top bar, select **Menu > Project**. +1. Select **Your projects** or **Explore projects**. +1. Select a project. +1. Select **Settings > General**. 1. Expand the **Advanced** section. 1. Scroll down to the **Delete project** section. -1. Click **Delete project** -1. Confirm this action by typing in the expected text. +1. Select **Delete project** +1. Confirm this action by completing the field. -Projects in personal namespaces are deleted immediately on request. For information on delayed deletion of projects in a group, please see [Enable delayed project removal](../group/index.md#enable-delayed-project-removal). +## View project activity -## Project settings +To view the activity of a project: -Set the project's visibility level and the access levels to its various pages -and perform actions like archiving, renaming or transferring a project. +1. On the top bar, select **Menu > Project**. +1. Select **Your projects** or **Explore projects**. +1. Select a project. +1. On the left sidebar, select **Project information > Activity**. +1. Select a tab to view the type of project activity. -Read through the documentation on [project settings](settings/index.md). +## Leave a project -## Project activity +If you leave a project you are no longer a project +member and cannot contribute. -To view the activity of a project: +To leave a project: -1. On the left sidebar, select **Project information > Activity**. -1. Select a tab to view **All** the activity, or to filter it by any of these criteria: - - **Push events** - - **Merge events** - - **Issue events** - - **Comments** - - **Team** - - **Wiki** - -### Leave a project - -**Leave project** only displays on the project's dashboard -when a project is part of a group (under a -[group namespace](../group/index.md#namespaces)). -If you choose to leave a project you are no longer a project -member, and cannot contribute. - -## Use your project as a Go package - -Any project can be used as a Go package. GitLab responds correctly to `go get` -and `godoc.org` discovery requests, including the -[`go-import`](https://golang.org/cmd/go/#hdr-Remote_import_paths) and -[`go-source`](https://github.com/golang/gddo/wiki/Source-Code-Links) meta tags. - -Private projects, including projects in subgroups, can be used as a Go package, -but may require configuration to work correctly. GitLab responds correctly -to `go get` discovery requests for projects that *are not* in subgroups, -regardless of authentication or authorization. -[Authentication](#authenticate-go-requests) is required to use a private project -in a subgroup as a Go package. Otherwise, GitLab truncates the path for -private projects in subgroups to the first two segments, causing `go get` to -fail. - -GitLab implements its own Go proxy. This feature must be enabled by an -administrator and requires additional configuration. See [GitLab Go -Proxy](../packages/go_proxy/index.md). - -### Disable Go module features for private projects - -In Go 1.12 and later, Go queries module proxies and checksum databases in the -process of [fetching a -module](../../development/go_guide/dependencies.md#fetching). This can be -selectively disabled with `GOPRIVATE` (disable both), -[`GONOPROXY`](../../development/go_guide/dependencies.md#proxies) (disable proxy -queries), and [`GONOSUMDB`](../../development/go_guide/dependencies.md#fetching) -(disable checksum queries). - -`GOPRIVATE`, `GONOPROXY`, and `GONOSUMDB` are comma-separated lists of Go -modules and Go module prefixes. For example, -`GOPRIVATE=gitlab.example.com/my/private/project` disables queries for that -one project, but `GOPRIVATE=gitlab.example.com` disables queries for *all* -projects on GitLab.com. Go does not query module proxies if the module name or a -prefix of it appears in `GOPRIVATE` or `GONOPROXY`. Go does not query checksum -databases if the module name or a prefix of it appears in `GONOPRIVATE` or -`GONOSUMDB`. - -### Authenticate Go requests - -To authenticate requests to private projects made by Go, use a [`.netrc` -file](https://everything.curl.dev/usingcurl/netrc) and a [personal access -token](../profile/personal_access_tokens.md) in the password field. **This only -works if your GitLab instance can be accessed with HTTPS.** The `go` command -does not transmit credentials over insecure connections. This authenticates -all HTTPS requests made directly by Go, but does not authenticate requests made -through Git. +1. On the top bar, select **Menu > Project**. +1. Select **Your projects** or **Explore projects**. +1. Select a project. +1. Select **Leave project**. The **Leave project** option only displays +on the project dashboard when a project is part of a group under a +[group namespace](../group/index.md#namespaces). -For example: +## Use a project as a Go package + +Prerequisites: + +- Contact your administrator to enable the [GitLab Go Proxy](../packages/go_proxy/index.md). +- To use a private project in a subgroup as a Go package, you must [authenticate Go requests](#authenticate-go-requests-to-private-projects). Go requests that are not authenticated cause +`go get` to fail. You don't need to authenticate Go requests for projects that are not in subgroups. + +To use a project as a Go package, use the `go get` and `godoc.org` discovery requests. You can use the meta tags: + +- [`go-import`](https://pkg.go.dev/cmd/go#hdr-Remote_import_paths) +- [`go-source`](https://github.com/golang/gddo/wiki/Source-Code-Links) + +### Authenticate Go requests to private projects + +Prerequisites: + +- Your GitLab instance must be accessible with HTTPS. +- You must have a [personal access token](../profile/personal_access_tokens.md). + +To authenticate Go requests, create a [`.netrc`](https://everything.curl.dev/usingcurl/netrc) file with the following information: ```plaintext machine gitlab.example.com @@ -335,95 +349,110 @@ login <gitlab_user_name> password <personal_access_token> ``` -NOTE: On Windows, Go reads `~/_netrc` instead of `~/.netrc`. -### Authenticate Git fetches +The `go` command does not transmit credentials over insecure connections. It authenticates +HTTPS requests made by Go, but does not authenticate requests made +through Git. -If a module cannot be fetched from a proxy, Go falls back to using Git (for -GitLab projects). Git uses `.netrc` to authenticate requests. You can also -configure Git to either: +### Authenticate Git requests -- Embed specific credentials in the request URL. -- Use SSH instead of HTTPS, as Go always uses HTTPS to fetch Git repositories. +If Go cannot fetch a module from a proxy, it uses Git. Git uses a `.netrc` file to authenticate requests, but you can +configure other authentication methods. -```shell -# Embed credentials in any request to GitLab.com: -git config --global url."https://${user}:${personal_access_token}@gitlab.example.com".insteadOf "https://gitlab.example.com" +Configure Git to either: -# Use SSH instead of HTTPS: -git config --global url."git@gitlab.example.com".insteadOf "https://gitlab.example.com" -``` +- Embed credentials in the request URL: -### Fetch Go modules from Geo secondary sites + ```shell + git config --global url."https://${user}:${personal_access_token}@gitlab.example.com".insteadOf "https://gitlab.example.com" + ``` + +- Use SSH instead of HTTPS: + + ```shell + git config --global url."git@gitlab.example.com".insteadOf "https://gitlab.example.com" + ``` + +### Disable Go module fetching for private projects + +To [fetch modules or packages](../../development/go_guide/dependencies.md#fetching), Go uses +the [environment variables](../../development/go_guide/dependencies.md#proxies): -As Go modules are stored in Git repositories, you can use the [Geo](../../administration/geo/index.md) -feature that allows Git repositories to be accessed on the secondary Geo servers. +- `GOPRIVATE` +- `GONOPROXY` +- `GONOSUMDB` -In the following examples, the primary's site domain name is `gitlab.example.com`, -and the secondary's is `gitlab-secondary.example.com`. +To disable fetching: -`go get` will initially generate some HTTP traffic to the primary, but when the module -download commences, the `insteadOf` configuration sends the traffic to the secondary. +1. Disable `GOPRIVATE`: + - To disable queries for one project, disable `GOPRIVATE=gitlab.example.com/my/private/project`. + - To disable queries for all projects on GitLab.com, disable `GOPRIVATE=gitlab.example.com`. +1. Disable proxy queries in `GONOPROXY`. +1. Disable checksum queries in `GONOSUMDB`. -#### Use SSH to access the Geo secondary +- If the module name or its prefix is in `GOPRIVATE` or `GONOPROXY`, Go does not query module +proxies. +- If the module name or its prefix is in `GONOPRIVATE` or `GONOSUMDB`, Go does not query +Checksum databases. -To fetch Go modules from the secondary using SSH: +### Fetch Go modules from Geo secondary sites + +Use [Geo](../../administration/geo/index.md) to access Git repositories that contain Go modules +on secondary Geo servers. + +You can use SSH or HTTP to access the Geo secondary server. + +#### Use SSH to access the Geo secondary server + +To access the Geo secondary server with SSH: 1. Reconfigure Git on the client to send traffic for the primary to the secondary: - ```plaintext + ```shell git config --global url."git@gitlab-secondary.example.com".insteadOf "https://gitlab.example.com" git config --global url."git@gitlab-secondary.example.com".insteadOf "http://gitlab.example.com" ``` -1. Ensure the client is set up for SSH access to GitLab repositories. This can be tested on the primary, - and GitLab will replicate the public key to the secondary. + - For `gitlab.example.com`, use the primary site domain name. + - For `gitlab-secondary.example.com`, use the secondary site domain name. + +1. Ensure the client is set up for SSH access to GitLab repositories. You can test this on the primary, + and GitLab replicates the public key to the secondary. + +The `go get` request generates HTTP traffic to the primary Geo server. When the module +download starts, the `insteadOf` configuration sends the traffic to the secondary Geo server. #### Use HTTP to access the Geo secondary -Using HTTP to fetch Go modules does not work with CI/CD job tokens, only with -persistent access tokens that are replicated to the secondary. +You must use persistent access tokens that replicate to the secondary server. You cannot use +CI/CD job tokens to fetch Go modules with HTTP. -To fetch Go modules from the secondary using HTTP: +To access the Geo secondary server with HTTP: -1. Put in place a Git `insteadOf` redirect on the client: +1. Add a Git `insteadOf` redirect on the client: - ```plaintext + ```shell git config --global url."https://gitlab-secondary.example.com".insteadOf "https://gitlab.example.com" ``` + - For `gitlab.example.com`, use the primary site domain name. + - For `gitlab-secondary.example.com`, use the secondary site domain name. + 1. Generate a [personal access token](../profile/personal_access_tokens.md) and - provide those credentials in the client's `~/.netrc` file: + add the credentials in the client's `~/.netrc` file: - ```plaintext + ```shell machine gitlab.example.com login USERNAME password TOKEN machine gitlab-secondary.example.com login USERNAME password TOKEN ``` -## Access project page with project ID +The `go get` request generates HTTP traffic to the primary Geo server. When the module +download starts, the `insteadOf` configuration sends the traffic to the secondary Geo server. -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53671) in GitLab 11.8. +## Related topics -To quickly access a project from the GitLab UI using the project ID, -visit the `/projects/:id` URL in your browser or other tool accessing the project. - -## Project's landing page - -The project's landing page shows different information depending on -the project's visibility settings and user permissions. - -For public projects, and to members of internal and private projects -with [permissions to view the project's code](../permissions.md#project-members-permissions): - -- The content of a - [`README` or an index file](repository/index.md#readme-and-index-files) - is displayed (if any), followed by the list of directories in the - project's repository. -- If the project doesn't contain either of these files, the - visitor sees the list of files and directories of the repository. - -For users without permissions to view the project's code, GitLab displays: - -- The wiki homepage, if any. -- The list of issues in the project. +- [Import a project](../../user/project/import/index.md). +- [Connect an external repository to GitLab CI/CD](../../ci/ci_cd_for_external_repos/index.md). +- [Fork a project](repository/forking_workflow.md#creating-a-fork). +- [Adjust project visibility and access levels](settings/index.md#sharing-and-permissions). diff --git a/doc/user/report_abuse.md b/doc/user/report_abuse.md index 810e1427e11..c0fb29b435a 100644 --- a/doc/user/report_abuse.md +++ b/doc/user/report_abuse.md @@ -16,54 +16,51 @@ A GitLab administrator [can then choose](admin_area/review_abuse_reports.md) to: You can report a user through their: -- [Profile](#reporting-abuse-through-a-users-profile) -- [Comments](#reporting-abuse-through-a-users-comment) -- [Issues and Merge requests](#reporting-abuse-through-a-users-issue-or-merge-request) +- [Profile](#report-abuse-from-the-users-profile-page) +- [Comments](#report-abuse-from-a-users-comment) +- [Issues](#report-abuse-from-an-issue) +- [Merge requests](#report-abuse-from-a-merge-request) - [Snippets](snippets.md#mark-snippet-as-spam) -## Reporting abuse through a user's profile +## Report abuse from the user's profile page To report abuse from a user's profile page: -1. Click on the exclamation point report abuse button at the top right of the - user's profile. +1. Anywhere in GitLab, select the name of the user. +1. In the top right corner of the user's profile, select **Report abuse** (**{information-o}**). 1. Complete an abuse report. -1. Click the **Send report** button. +1. Select **Send report**. -## Reporting abuse through a user's comment +## Report abuse from a user's comment To report abuse from a user's comment: -1. Click on the vertical ellipsis (⋮) more actions button to open the dropdown. -1. Select **Report as abuse**. +1. In the comment, in the top right corner, select **More actions** (**{ellipsis_v}**). +1. Select **Report abuse to admin**. 1. Complete an abuse report. -1. Click the **Send report** button. +1. Select **Send report**. NOTE: A URL to the reported user's comment is pre-filled in the abuse report's **Message** field. -## Reporting abuse through a user's issue or merge request +## Report abuse from an issue -The **Report abuse** button is displayed at the top right of the issue or merge request: - -- When **Report abuse** is selected from the menu that appears when the - **Close issue** or **Close merge request** button is clicked, for users that - have permission to close the issue or merge request. -- When viewing the issue or merge request, for users that don't have permission - to close the issue or merge request. +1. On the issue, in the top right corner, select the vertical ellipsis (**{ellipsis_v}**). +1. Select **Report abuse**. +1. Submit an abuse report. +1. Select **Send report**. -With the **Report abuse** button displayed, to submit an abuse report: +## Report abuse from a merge request -1. Click the **Report abuse** button. +1. On the merge request, in the top right corner, either: + - Select **Report abuse**. This option is displayed if you do not have permission to close the merge request. + - Next to **Mark as draft**, select the down arrow (**{chevron-down}**) and then select **Report abuse**. + This option is displayed if you have permission to close the merge request. 1. Submit an abuse report. -1. Click the **Send report** button. - -NOTE: -A URL to the reported user's issue or merge request is pre-filled -in the abuse report's **Message** field. +1. Select **Send report**. -## Managing abuse reports +## Related topics -Administrators are able to view and resolve abuse reports. -For more information, see [abuse reports administration documentation](admin_area/review_abuse_reports.md). +- Administrators can view and resolve abuse reports. + For more information, see [abuse reports administration documentation](admin_area/review_abuse_reports.md). diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index f68951badff..b9c45bce43a 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -14,6 +14,10 @@ This is the user documentation. To configure the Advanced Search, visit the [administrator documentation](../../integration/elasticsearch.md). Advanced Search is enabled in GitLab.com. +INFO: +Get advanced search and more with a +[free 30-day trial of GitLab Ultimate](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-advanced-search-docs). + GitLab Advanced Search expands on the Basic Search with an additional set of features for faster, more advanced searches across the entire GitLab instance when searching in: @@ -48,13 +52,13 @@ The Advanced Search can be useful in various scenarios: ## Use the Advanced Search syntax -Elasticsearch has only data for the default branch. That means that if you go +Elasticsearch has data for the default branch only. That means that if you go to the repository tree and switch the branch from the default to something else, -then the "Code" tab in the search result page will be served by the basic +then the **Code** tab in the search result page is served by the basic search even if Elasticsearch is enabled. The Advanced Search syntax supports fuzzy or exact search queries with prefixes, -boolean operators, and much more. Use the search as before and GitLab will show +boolean operators, and much more. Use the search as before and GitLab shows you matching code from each project you have access to.  @@ -62,8 +66,8 @@ you matching code from each project you have access to. Full details can be found in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/5.3/query-dsl-simple-query-string-query.html#_simple_query_string_syntax), but here's a quick guide: -- Searches look for all the words in a query, in any order - e.g.: searching - issues for [`display bug`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=display+bug&group_id=9970&project_id=278964) and [`bug display`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+Display&group_id=9970&project_id=278964) will return the same results. +- Searches look for all the words in a query, in any order - for example: searching + issues for [`display bug`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=display+bug&group_id=9970&project_id=278964) and [`bug display`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+Display&group_id=9970&project_id=278964) return the same results. - To find the exact phrase (stemming still applies), use double quotes: [`"display bug"`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=%22display+bug%22&group_id=9970&project_id=278964) - To find bugs not mentioning display, use `-`: [`bug -display`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+-display&group_id=9970&project_id=278964) - To find a bug in display or banner, use `|`: [`bug display | banner`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+display+%7C+banner&group_id=9970&project_id=278964) @@ -81,7 +85,7 @@ Advanced Search also supports the use of filters. The available filters are: - `blob`: Filters by Git `object ID`. Exact match only. To use them, add them to your keyword in the format `<filter_name>:<value>` without -any spaces between the colon (`:`) and the value. When no keyword is provided, an asterisk (`*`) will be used as the keyword. +any spaces between the colon (`:`) and the value. When no keyword is provided, an asterisk (`*`) is used as the keyword. Examples: diff --git a/doc/user/search/img/dashboard_links_v13_11.png b/doc/user/search/img/dashboard_links_v13_11.png Binary files differdeleted file mode 100644 index 5f2e61eee1e..00000000000 --- a/doc/user/search/img/dashboard_links_v13_11.png +++ /dev/null diff --git a/doc/user/search/img/dashboard_links_v14_6.png b/doc/user/search/img/dashboard_links_v14_6.png Binary files differnew file mode 100644 index 00000000000..52ae39d9d1a --- /dev/null +++ b/doc/user/search/img/dashboard_links_v14_6.png diff --git a/doc/user/search/img/filter_approved_by_merge_requests_v13_0.png b/doc/user/search/img/filter_approved_by_merge_requests_v13_0.png Binary files differdeleted file mode 100644 index 5020da90297..00000000000 --- a/doc/user/search/img/filter_approved_by_merge_requests_v13_0.png +++ /dev/null diff --git a/doc/user/search/img/filter_approved_by_merge_requests_v14_6.png b/doc/user/search/img/filter_approved_by_merge_requests_v14_6.png Binary files differnew file mode 100644 index 00000000000..8d47fdc2652 --- /dev/null +++ b/doc/user/search/img/filter_approved_by_merge_requests_v14_6.png diff --git a/doc/user/search/img/filter_approver_merge_requests.png b/doc/user/search/img/filter_approver_merge_requests.png Binary files differdeleted file mode 100644 index 4c28ee17f47..00000000000 --- a/doc/user/search/img/filter_approver_merge_requests.png +++ /dev/null diff --git a/doc/user/search/img/filter_approver_merge_requests_v14_6.png b/doc/user/search/img/filter_approver_merge_requests_v14_6.png Binary files differnew file mode 100644 index 00000000000..58950031378 --- /dev/null +++ b/doc/user/search/img/filter_approver_merge_requests_v14_6.png diff --git a/doc/user/search/img/filtering_merge_requests_by_date_v13_6.png b/doc/user/search/img/filtering_merge_requests_by_date_v13_6.png Binary files differdeleted file mode 100644 index 8f59534ef1c..00000000000 --- a/doc/user/search/img/filtering_merge_requests_by_date_v13_6.png +++ /dev/null diff --git a/doc/user/search/img/filtering_merge_requests_by_date_v14_6.png b/doc/user/search/img/filtering_merge_requests_by_date_v14_6.png Binary files differnew file mode 100644 index 00000000000..398820f7864 --- /dev/null +++ b/doc/user/search/img/filtering_merge_requests_by_date_v14_6.png diff --git a/doc/user/search/img/filtering_merge_requests_by_environment_v13_6.png b/doc/user/search/img/filtering_merge_requests_by_environment_v13_6.png Binary files differdeleted file mode 100644 index e73a0995d73..00000000000 --- a/doc/user/search/img/filtering_merge_requests_by_environment_v13_6.png +++ /dev/null diff --git a/doc/user/search/img/filtering_merge_requests_by_environment_v14_6.png b/doc/user/search/img/filtering_merge_requests_by_environment_v14_6.png Binary files differnew file mode 100644 index 00000000000..c35f2c8a58b --- /dev/null +++ b/doc/user/search/img/filtering_merge_requests_by_environment_v14_6.png diff --git a/doc/user/search/img/issue_search_by_term.png b/doc/user/search/img/issue_search_by_term.png Binary files differdeleted file mode 100644 index 64450c6a891..00000000000 --- a/doc/user/search/img/issue_search_by_term.png +++ /dev/null diff --git a/doc/user/search/img/issue_search_filter_v12_7.png b/doc/user/search/img/issue_search_filter_v12_7.png Binary files differdeleted file mode 100644 index 102a2e0859c..00000000000 --- a/doc/user/search/img/issue_search_filter_v12_7.png +++ /dev/null diff --git a/doc/user/search/img/issues_assigned_to_you.png b/doc/user/search/img/issues_assigned_to_you.png Binary files differdeleted file mode 100644 index 55986eedcba..00000000000 --- a/doc/user/search/img/issues_assigned_to_you.png +++ /dev/null diff --git a/doc/user/search/img/issues_filter_none_any.png b/doc/user/search/img/issues_filter_none_any.png Binary files differdeleted file mode 100644 index 9682fc55315..00000000000 --- a/doc/user/search/img/issues_filter_none_any.png +++ /dev/null diff --git a/doc/user/search/img/issues_mrs_shortcut_v14_4.png b/doc/user/search/img/issues_mrs_shortcut_v14_4.png Binary files differdeleted file mode 100644 index 2610e611446..00000000000 --- a/doc/user/search/img/issues_mrs_shortcut_v14_4.png +++ /dev/null diff --git a/doc/user/search/img/issues_mrs_shortcut_v14_6.png b/doc/user/search/img/issues_mrs_shortcut_v14_6.png Binary files differnew file mode 100644 index 00000000000..52753eb8fc7 --- /dev/null +++ b/doc/user/search/img/issues_mrs_shortcut_v14_6.png diff --git a/doc/user/search/img/project_search.png b/doc/user/search/img/project_search.png Binary files differdeleted file mode 100644 index b2525b2c771..00000000000 --- a/doc/user/search/img/project_search.png +++ /dev/null diff --git a/doc/user/search/index.md b/doc/user/search/index.md index 325e5386417..aa4950cfa33 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -5,26 +5,20 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: index, reference, howto --- -# Search through GitLab **(FREE)** +# Searching in GitLab **(FREE)** -## Issues and merge requests +## Search issues and merge requests -To search through issues and merge requests in multiple projects, use the **Issues** or **Merge Requests** links -in the top-right part of your screen. These instructions are valid for both. +To search through issues and merge requests in multiple projects, on the top bar, select the **Issues** or **Merge Requests** links. -The numbers in the right-hand side of the top menu indicate how many issues, merge requests, -and to-do items are assigned to you: +The numbers indicate how many issues, merge requests, and to-do items are assigned to you: - + - **{issues}** **Issues**: The open issues assigned to you. - **{merge-request-open}** **Merge requests**: The [merge requests](../project/merge_requests/index.md) assigned to you. - **{todo-done}** **To-do items**: The [to-do items](../todos.md) assigned to you. -When you click **Issues**, GitLab shows the opened issues assigned to you: - - - You can search through **Open**, **Closed**, or **All** issues. You can also filter the results using the search and filter field, as described in @@ -35,7 +29,7 @@ You can also filter the results using the search and filter field, as described GitLab shows shortcuts to issues and merge requests created by you or assigned to you in the search field in the upper right corner: - + ### Filter issue and merge request lists @@ -63,25 +57,13 @@ groups: - `=`: Is - `!=`: Is not ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18059) in GitLab 12.7) 1. Enter the text to [filter the attribute by](#filters-autocomplete). + You can filter some attributes by **None** or **Any**. 1. Repeat this process to filter by multiple attributes. Multiple attributes are joined by a logical - `AND`. For example, filtering by an Author and Milestone `!=` 12.6 filters for the issues where the - author matches your selection, and the milestone is not 12.6: - -  + `AND`. GitLab displays the results on-screen, but you can also [retrieve them as an RSS feed](#retrieve-search-results-as-feed). -### Filtering by **None** / **Any** - -Some filter fields like milestone and assignee, allow you to filter by **None** or **Any**. - - - -Selecting **None** returns results that have an empty value for that field. For example: no milestone, no assignee. - -Selecting **Any** does the opposite. It returns results that have a non-empty value for that field. - ### Searching for specific terms You can filter issues and merge requests by specific terms included in titles or descriptions. @@ -95,8 +77,6 @@ You can filter issues and merge requests by specific terms included in titles or issues for `included in titles` is same as `included titles` - Search is limited to 4096 characters and 64 terms per query. - - ### Retrieve search results as feed > Feeds for merge requests were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66336) in GitLab 14.3. @@ -128,7 +108,7 @@ You can filter the **Issues** list to individual instances by their ID. For exam To filter merge requests by an individual approver, you can type (or select from the dropdown) **Approver** and select the user. - + ### Filtering merge requests by "approved by" **(PREMIUM)** @@ -138,7 +118,7 @@ the dropdown) **Approver** and select the user. To filter merge requests already approved by a specific individual, you can type (or select from the dropdown) **Approved-By** and select the user. - + ### Filtering merge requests by reviewer **(FREE)** @@ -161,13 +141,15 @@ you can type (or select from the dropdown) the following: When filtering by an environment, a dropdown presents all environments that you can choose from: - + -When filtering by a deploy date, you must enter the date manually. Deploy dates +When filtering by `Deployed-before` or `Deployed-after`, the date refers to when +the deployment to an environment (triggered by the merge commit) completed successfully. +You must enter the deploy date manually. Deploy dates use the format `YYYY-MM-DD`, and must be quoted if you wish to specify both a date and time (`"YYYY-MM-DD HH:MM"`): - + ## Filters autocomplete @@ -245,20 +227,9 @@ and **Labels**, select multiple issues to add to a list of your choice:  -## Shortcut - -To view issues and merge requests created or assigned to you in a project: - -1. Go to your project. -1. In the top navigation bar, click the search box to display a list of issues and - merge requests. -1. Select your desired issue or merge request: - -  - -### Autocomplete suggestions +## Autocomplete suggestions -You can also type in this search bar to see autocomplete suggestions for: +In the search bar, you can view autocomplete suggestions for: - Projects and groups - Various help pages (try and type **API help**) diff --git a/doc/user/snippets.md b/doc/user/snippets.md index e2cb9937f76..dc3ab35067e 100644 --- a/doc/user/snippets.md +++ b/doc/user/snippets.md @@ -52,7 +52,7 @@ You can create snippets in multiple ways, depending on whether you want to creat Filenames with appropriate extensions display [syntax highlighting](#filenames). Failure to add a filename can cause a known [copy-pasting bug](https://gitlab.com/gitlab-org/gitlab/-/issues/22870). If you don't provide a filename, GitLab [creates a name for you](#filenames). -1. (Optional) Add [multiple files](#add-or-remove-multiple-files) to your snippet. +1. Optional. Add [multiple files](#add-or-remove-multiple-files) to your snippet. 1. Select a visibility level, and select **Create snippet**. After you create a snippet, you can still [add more files to it](#add-or-remove-multiple-files). diff --git a/doc/user/tasks.md b/doc/user/tasks.md index e5e5510407e..5fde64e3578 100644 --- a/doc/user/tasks.md +++ b/doc/user/tasks.md @@ -15,15 +15,18 @@ For the latest updates, check the [Tasks Roadmap](https://gitlab.com/groups/gitl FLAG: On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `work_items`. +On GitLab.com, this feature is not available. The feature is not ready for production use. Use tasks to track steps needed for the [issue](project/issues/index.md) to be closed. When planning an issue, you need a way to capture and break down technical -requirements or steps necessary to complete it. An issue with tasks is better defined, +requirements or steps necessary to complete it. An issue with related tasks is better defined, and so you can provide a more accurate issue weight and completion criteria. -To see the roadmap for migrating issues and [epics](group/epics/index.md) +Tasks are a type of work item, a step towards [default issue types](https://gitlab.com/gitlab-org/gitlab/-/issues/323404) +in GitLab. +For the roadmap of migrating issues and [epics](group/epics/index.md) to work items and adding custom work item types, visit [epic 6033](https://gitlab.com/groups/gitlab-org/-/epics/6033) or [Plan direction page](https://about.gitlab.com/direction/plan/). @@ -31,4 +34,4 @@ to work items and adding custom work item types, visit ## View a task The only way to view a task is to open it with a deep link, -for example: `/<group_name>/<project_name>/-/work_item/1` +for example: `/<group_name>/<project_name>/-/work_item/1`. diff --git a/doc/user/todos.md b/doc/user/todos.md index 9a985664ed1..2486db813ff 100644 --- a/doc/user/todos.md +++ b/doc/user/todos.md @@ -29,7 +29,7 @@ Many to-do items are created automatically. A to-do item is added to your To-Do List when: - An issue or merge request is assigned to you. -- You're [mentioned](project/issues/issue_data_and_actions.md#mentions) in the description or +- You're [mentioned](discussions/index.md#mentions) in the description or comment of an issue, merge request, or epic. - You are mentioned in a comment on a commit or design. - The CI/CD pipeline for your merge request fails. diff --git a/doc/user/workspace/index.md b/doc/user/workspace/index.md index 8ca7f7defb2..cf35f082880 100644 --- a/doc/user/workspace/index.md +++ b/doc/user/workspace/index.md @@ -6,6 +6,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Workspace +DISCLAIMER: +This page contains information related to upcoming products, features, and functionality. +It is important to note that the information presented is for informational purposes only. +Please do not rely on this information for purchasing or planning purposes. +As with all projects, the items mentioned on this page are subject to change or delay. +The development, release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + Workspace will be above the [top-level namespaces](../group/index.md#namespaces) for you to manage everything you do as a GitLab administrator, including: |
