diff options
Diffstat (limited to 'doc')
552 files changed, 12346 insertions, 5378 deletions
diff --git a/doc/.vale/gitlab/InclusionAbleism.yml b/doc/.vale/gitlab/InclusionAbleism.yml new file mode 100644 index 00000000000..29b26547130 --- /dev/null +++ b/doc/.vale/gitlab/InclusionAbleism.yml @@ -0,0 +1,14 @@ +--- +# Suggestion: gitlab.InclusionAbleism +# +# Suggests alternatives for words that foster ableism. +# +# For a list of all options, see https://errata-ai.gitbook.io/vale/getting-started/styles +extends: substitution +message: 'Use inclusive language. Consider "%s" instead of "%s".' +link: https://docs.gitlab.com/ee/development/documentation/styleguide.html#inclusive-language +level: suggestion +ignorecase: true +swap: + sanity (?:check|test): check for completeness + dummy: placeholder, sample, fake diff --git a/doc/.vale/gitlab/InclusionCultural.yml b/doc/.vale/gitlab/InclusionCultural.yml new file mode 100644 index 00000000000..5bfad84f100 --- /dev/null +++ b/doc/.vale/gitlab/InclusionCultural.yml @@ -0,0 +1,16 @@ +--- +# Warning: gitlab.InclusionCultural +# +# Suggests alternatives for words that are culturally inappropriate. +# +# For a list of all options, see https://errata-ai.gitbook.io/vale/getting-started/styles +extends: substitution +message: 'Use inclusive language. Consider "%s" instead of "%s".' +link: https://docs.gitlab.com/ee/development/documentation/styleguide.html#inclusive-language +level: warning +ignorecase: true +swap: + blacklist(?:ed|ing|s)?: denylist + whitelist(?:ed|ing|s)?: allowlist + master: primary, main + slave: secondary diff --git a/doc/.vale/gitlab/InclusionGender.yml b/doc/.vale/gitlab/InclusionGender.yml new file mode 100644 index 00000000000..389d76d7a24 --- /dev/null +++ b/doc/.vale/gitlab/InclusionGender.yml @@ -0,0 +1,18 @@ +--- +# Suggestion: gitlab.InclusionGender +# +# Suggests alternatives for words that are gender-specific. +# +# For a list of all options, see https://errata-ai.gitbook.io/vale/getting-started/styles +extends: substitution +message: 'Use inclusive language. Consider "%s" instead of "%s".' +link: https://docs.gitlab.com/ee/development/documentation/styleguide.html#inclusive-language +level: suggestion +ignorecase: true +swap: + mankind: humanity, people + manpower: GitLab team members + he: they + his: their + she: they + hers: their diff --git a/doc/.vale/gitlab/SubstitutionWarning.yml b/doc/.vale/gitlab/SubstitutionWarning.yml index 5324a48e38c..394bd5df08d 100644 --- a/doc/.vale/gitlab/SubstitutionWarning.yml +++ b/doc/.vale/gitlab/SubstitutionWarning.yml @@ -11,7 +11,6 @@ link: https://about.gitlab.com/handbook/communication/#top-misused-terms level: warning ignorecase: true swap: - admin: administrator blacklist(ed|ing)?: denylist code base: codebase config: configuration @@ -20,4 +19,3 @@ swap: filesystem: file system info: information repo: repository - whitelist(ed|ing)?: allowlist diff --git a/doc/.vale/gitlab/spelling-exceptions.txt b/doc/.vale/gitlab/spelling-exceptions.txt index bf816afdfab..eec1d03d1a5 100644 --- a/doc/.vale/gitlab/spelling-exceptions.txt +++ b/doc/.vale/gitlab/spelling-exceptions.txt @@ -298,6 +298,7 @@ OmniAuth onboarding OpenID OpenShift +Opsgenie Packagist parallelization parallelizations @@ -382,6 +383,7 @@ reusability reverified reverifies reverify +RHEL rollout rollouts rsync diff --git a/doc/README.md b/doc/README.md index efae2cdd3ff..03ecbef56ea 100644 --- a/doc/README.md +++ b/doc/README.md @@ -92,7 +92,7 @@ The following documentation relates to the DevOps **Manage** stage: | Manage topics | Description | |:--------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Authentication and<br/>Authorization](administration/auth/README.md) **(CORE ONLY)** | Supported authentication and authorization providers. | -| [GitLab Value Stream Analytics](user/project/cycle_analytics.md) | Measure the time it takes to go from an [idea to production](https://about.gitlab.com/blog/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab) for each project you have. | +| [GitLab Value Stream Analytics](user/analytics/value_stream_analytics.md) | Measure the time it takes to go from an [idea to production](https://about.gitlab.com/blog/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab) for each project you have. | | [Instance-level Analytics](user/admin_area/analytics/index.md) | Discover statistics on how many GitLab features you use and user activity. | <div align="right"> @@ -121,7 +121,7 @@ The following documentation relates to the DevOps **Plan** stage: | [Milestones](user/project/milestones/index.md) | Set milestones for delivery of issues and merge requests, with optional due date. | | [Project Issue Board](user/project/issue_board.md) | Display issues on a Scrum or Kanban board. | | [Quick Actions](user/project/quick_actions.md) | Shortcuts for common actions on issues or merge requests, replacing the need to click buttons or use dropdowns in GitLab's UI. | -| [Related Issues](user/project/issues/related_issues.md) **(STARTER)** | Create a relationship between issues. | +| [Related Issues](user/project/issues/related_issues.md) | Create a relationship between issues. | | [Requirements Management](user/project/requirements/index.md) **(ULTIMATE)** | Check your products against a set of criteria. | | [Roadmap](user/group/roadmap/index.md) **(ULTIMATE)** | Visualize epic timelines. | | [Service Desk](user/project/service_desk.md) | A simple way to allow people to create issues in your GitLab instance without needing their own user account. | @@ -295,7 +295,7 @@ The following documentation relates to the DevOps **Secure** stage: | [Dependency Scanning](user/application_security/dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | | [Dynamic Application Security Testing (DAST)](user/application_security/dast/index.md) **(ULTIMATE)** | Analyze running web applications for known vulnerabilities. | | [Group Security Dashboard](user/application_security/security_dashboard/index.md#group-security-dashboard) **(ULTIMATE)** | View vulnerabilities in all the projects in a group and its subgroups. | -| [Instance Security Dashboard](user/application_security/security_dashboard/index.md#instance-security-dashboard) **(ULTIMATE)** | View vulnerabilities in all the projects you're interested in. | +| [Instance Security Center](user/application_security/security_dashboard/index.md#instance-security-center) **(ULTIMATE)** | View vulnerabilities in all the projects you're interested in. | | [License Compliance](user/compliance/license_compliance/index.md) **(ULTIMATE)** | Search your project's dependencies for their licenses. | | [Pipeline Security](user/application_security/security_dashboard/index.md#pipeline-security) **(ULTIMATE)** | View the security reports for your project's pipelines. | | [Project Security Dashboard](user/application_security/security_dashboard/index.md#project-security-dashboard) **(ULTIMATE)** | View the latest security reports for your project. | diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 099346b2b0b..ac972e2e33e 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan](https://about.gitlab.com/pricing/). GitLab system administrators can also take advantage of the logs located on the -filesystem. See [the logs system documentation](logs.md) for more details. +file system. See [the logs system documentation](logs.md) for more details. ## Overview @@ -108,7 +108,7 @@ Server-wide audit logging introduces the ability to observe user actions across the entire instance of your GitLab server, making it easy to understand who changed what and when for audit purposes. -To view the server-wide admin log, visit **Admin Area > Monitoring > Audit Log**. +To view the server-wide administrator log, visit **Admin Area > Monitoring > Audit Log**. In addition to the group and project events, the following user actions are also recorded: @@ -126,6 +126,7 @@ recorded: - User was added ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8) - User was blocked via Admin Area ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8) - User was blocked via API ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25872) in GitLab 12.9) +- Failed second-factor authentication attempt ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16826) in GitLab 13.5) It's possible to filter particular actions by choosing an audit data type from the filter dropdown box. You can further filter by specific group, project, or user @@ -151,7 +152,7 @@ on adding these events into GitLab: The current architecture of audit events is not prepared to receive a very high amount of records. It may make the user interface for your project or audit logs very busy, and the disk space consumed by the -`audit_events` PostgreSQL table will increase considerably. It's disabled by default +`audit_events` PostgreSQL table may increase considerably. It's disabled by default to prevent performance degradations on GitLab instances with very high Git write traffic. In an upcoming release, Audit Logs for Git push events will be enabled @@ -172,6 +173,7 @@ the steps bellow. ```ruby Feature.enable(:repository_push_audit_event) + ``` ## Export to CSV **(PREMIUM ONLY)** @@ -183,6 +185,7 @@ the steps bellow. CAUTION: **Warning:** This feature might not be available to you. Check the **version history** note above for details. +If available, you can enable it with a [feature flag](#enable-or-disable-audit-log-export-to-csv). Export to CSV allows customers to export the current filter view of your audit log as a CSV file, diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index 926a4abab7d..cf82454cfd2 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -18,7 +18,7 @@ providers: - [Azure](../../integration/azure.md) - [Bitbucket Cloud](../../integration/bitbucket.md) - [CAS](../../integration/cas.md) -- [Crowd](../../integration/crowd.md) +- [Crowd](crowd.md) - [Facebook](../../integration/facebook.md) - [GitHub](../../integration/github.md) - [GitLab.com](../../integration/gitlab.md) diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 8862776ee1b..dc46c0756db 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -11,7 +11,7 @@ Geo replicates your database, your Git repositories, and few other assets. We will support and replicate more data in the future, that will enable you to failover with minimal effort, in a disaster situation. -See [Geo current limitations](../index.md#current-limitations) for more information. +See [Geo limitations](../index.md#limitations) for more information. CAUTION: **Warning:** Disaster recovery for multi-secondary configurations is in **Alpha**. @@ -84,8 +84,8 @@ must disable the **primary** node. single recommendation. You may need to: - Reconfigure the load balancers. - - Change DNS records (for example, point the primary DNS record to the **secondary** - node in order to stop usage of the **primary** node). + - Change DNS records (for example, point the primary DNS record to the + **secondary** node to stop usage of the **primary** node). - Stop the virtual servers. - Block traffic through a firewall. - Revoke object storage permissions from the **primary** node. @@ -129,6 +129,9 @@ Note the following when promoting a secondary: ``` 1. Promote the **secondary** node to the **primary** node. + +DANGER: **Danger:** +In GitLab 13.2 and later versions, promoting a secondary node to a primary while the secondary is paused fails. We are [investigating the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/225173). Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. To promote the secondary node to primary along with preflight checks: @@ -159,6 +162,9 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. +DANGER: **Danger:** +In GitLab 13.2 and later versions, promoting a secondary node to a primary while the secondary is paused fails. We are [investigating the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/225173). Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. + 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -201,6 +207,9 @@ an external PostgreSQL database, as it can only perform changes on a **secondary node with GitLab and the database on the same machine. As a result, a manual process is required: +DANGER: **Danger:** +In GitLab 13.2 and later versions, promoting a secondary node to a primary while the secondary is paused fails. We are [investigating the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/225173). Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. + 1. Promote the replica database associated with the **secondary** site. This will set the database to read-write: - Amazon RDS - [Promoting a Read Replica to Be a Standalone DB Instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 9b9c386652c..1238c4d8e2a 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -27,7 +27,7 @@ have a high degree of confidence in being able to perform them accurately. ## Not all data is automatically replicated -If you are using any GitLab features that Geo [doesn't support](../index.md#current-limitations), +If you are using any GitLab features that Geo [doesn't support](../index.md#limitations), you must make separate provisions to ensure that the **secondary** node has an up-to-date copy of any data associated with that feature. This may extend the required scheduled maintenance period significantly. diff --git a/doc/administration/geo/disaster_recovery/promotion_runbook.md b/doc/administration/geo/disaster_recovery/promotion_runbook.md index fb2353513df..7eb6ef01aee 100644 --- a/doc/administration/geo/disaster_recovery/promotion_runbook.md +++ b/doc/administration/geo/disaster_recovery/promotion_runbook.md @@ -1,269 +1,5 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -type: howto +redirect_to: runbooks/planned_failover_single_node.md --- -CAUTION: **Caution:** -This runbook is in **alpha**. For complete, production-ready documentation, see the -[disaster recovery documentation](index.md). - -# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)** - -## Geo planned failover runbook 1 - -| Component | Configuration | -| ----------- | --------------- | -| PostgreSQL | Omnibus-managed | -| Geo site | Single-node | -| Secondaries | One | - -This runbook will guide you through a planned failover of a single-node Geo site -with one secondary. The following general architecture is assumed: - -```mermaid -graph TD - subgraph main[Geo deployment] - subgraph Primary[Primary site] - Node_1[(GitLab node)] - end - subgraph Secondary1[Secondary site] - Node_2[(GitLab node)] - end - end -``` - -This guide will result in the following: - -1. An offline primary. -1. A promoted secondary that is now the new primary. - -What is not covered: - -1. Re-adding the old **primary** as a secondary. -1. Adding a new secondary. - -### Preparation - -NOTE: **Note:** -Before following any of those steps, make sure you have `root` access to the -**secondary** to promote it, since there isn't provided an automated way to -promote a Geo replica and perform a failover. - -On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to -review its status. Replicated objects (shown in green) should be close to 100%, -and there should be no failures (shown in red). If a large proportion of -objects aren't yet replicated (shown in gray), consider giving the node more -time to complete. - - - -If any objects are failing to replicate, this should be investigated before -scheduling the maintenance window. After a planned failover, anything that -failed to replicate will be **lost**. - -You can use the -[Geo status API](../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) -to review failed objects and the reasons for failure. -A common cause of replication failures is the data being missing on the -**primary** node - you can resolve these failures by restoring the data from backup, -or removing references to the missing data. - -The maintenance window won't end until Geo replication and verification is -completely finished. To keep the window as short as possible, you should -ensure these processes are close to 100% as possible during active use. - -If the **secondary** node is still replicating data from the **primary** node, -follow these steps to avoid unnecessary data loss: - -1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) - is implemented, updates must be prevented from happening manually to the - **primary**. Note that your **secondary** node still needs read-only - access to the **primary** node during the maintenance window: - - 1. At the scheduled time, using your cloud provider or your node's firewall, block - all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and - the **secondary** node's IP. - - For instance, you can run the following commands on the **primary** node: - - ```shell - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT --destination-port 22 -j REJECT - - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 80 -j REJECT - - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 443 -j REJECT - ``` - - From this point, users will be unable to view their data or make changes on the - **primary** node. They will also be unable to log in to the **secondary** node. - However, existing sessions will work for the remainder of the maintenance period, and - public data will be accessible throughout. - - 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via - another IP. The server should refuse connection. - - 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an - existing Git repository with an SSH remote URL. The server should refuse - connection. - - 1. On the **primary** node, disable non-Geo periodic background jobs by navigating - to **Admin Area > Monitoring > Background Jobs > Cron**, clicking `Disable All`, - and then clicking `Enable` for the `geo_sidekiq_cron_config_worker` cron job. - This job will re-enable several other cron jobs that are essential for planned - failover to complete successfully. - -1. Finish replicating and verifying all data: - - CAUTION: **Caution:** - Not all data is automatically replicated. Read more about - [what is excluded](planned_failover.md#not-all-data-is-automatically-replicated). - - 1. If you are manually replicating any - [data not managed by Geo](../replication/datatypes.md#limitations-on-replicationverification), - trigger the final replication process now. - 1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** - and wait for all queues except those with `geo` in the name to drop to 0. - These queues contain work that has been submitted by your users; failing over - before it is completed will cause the work to be lost. - 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the - following conditions to be true of the **secondary** node you are failing over to: - - All replication meters to each 100% replicated, 0% failures. - - All verification meters reach 100% verified, 0% failures. - - Database replication lag is 0ms. - - The Geo log cursor is up to date (0 events behind). - - 1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** - and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. - 1. On the **secondary** node, use [these instructions](../../raketasks/check.md) - to verify the integrity of CI artifacts, LFS objects, and uploads in file - storage. - - At this point, your **secondary** node will contain an up-to-date copy of everything the - **primary** node has, meaning nothing will be lost when you fail over. - -1. In this final step, you need to permanently disable the **primary** node. - - CAUTION: **Caution:** - When the **primary** node goes offline, there may be data saved on the **primary** node - that has not been replicated to the **secondary** node. This data should be treated - as lost if you proceed. - - TIP: **Tip:** - If you plan to [update the **primary** domain DNS record](index.md#step-4-optional-updating-the-primary-domain-dns-record), - you may wish to lower the TTL now to speed up propagation. - - When performing a failover, we want to avoid a split-brain situation where - writes can occur in two different GitLab instances. So to prepare for the - failover, you must disable the **primary** node: - - - If you have SSH access to the **primary** node, stop and disable GitLab: - - ```shell - sudo gitlab-ctl stop - ``` - - Prevent GitLab from starting up again if the server unexpectedly reboots: - - ```shell - sudo systemctl disable gitlab-runsvdir - ``` - - NOTE: **Note:** - (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being - started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). - It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. - - NOTE: **Note:** - (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu - or any other distribution based on the Upstart init system, you can prevent GitLab - from starting if the machine reboots as `root` with - `initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`. - - - If you do not have SSH access to the **primary** node, take the machine offline and - prevent it from rebooting. Since there are many ways you may prefer to accomplish - this, we will avoid a single recommendation. You may need to: - - - Reconfigure the load balancers. - - Change DNS records (for example, point the **primary** DNS record to the **secondary** - node in order to stop usage of the **primary** node). - - Stop the virtual servers. - - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. - - Physically disconnect a machine. - -### Promoting the **secondary** node - -Note the following when promoting a secondary: - -- A new **secondary** should not be added at this time. If you want to add a new - **secondary**, do this after you have completed the entire process of promoting - the **secondary** to the **primary**. -- If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` - error during this process, read - [the troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). - -To promote the secondary node: - -1. SSH in to your **secondary** node and login as root: - - ```shell - sudo -i - ``` - -1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by - removing any lines that enabled the `geo_secondary_role`: - - ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. - geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. - roles ['geo_secondary_role'] - ``` - -1. Run the following command to list out all preflight checks and automatically - check if replication and verification are complete before scheduling a planned - failover to ensure the process will go smoothly: - - ```shell - gitlab-ctl promotion-preflight-checks - ``` - -1. Promote the **secondary**: - - ```shell - gitlab-ctl promote-to-primary-node - ``` - - If you have already run the [preflight checks](planned_failover.md#preflight-checks) - or don't want to run them, you can skip them: - - ```shell - gitlab-ctl promote-to-primary-node --skip-preflight-check - ``` - - You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: - - ```shell - sudo gitlab-ctl promote-to-primary-node --force - ``` - -1. Verify you can connect to the newly promoted **primary** node using the URL used - previously for the **secondary** node. - - If successful, the **secondary** node has now been promoted to the **primary** node. - -### Next steps - -To regain geographic redundancy as quickly as possible, you should -[add a new **secondary** node](../setup/index.md). To -do that, you can re-add the old **primary** as a new secondary and bring it back -online. +This document was moved to [another location](runbooks/planned_failover_single_node.md). diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md new file mode 100644 index 00000000000..1e3bac0b354 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -0,0 +1,274 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + +CAUTION: **Caution:** +This runbook is in **alpha**. For complete, production-ready documentation, see the +[disaster recovery documentation](../index.md). + +# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)** + +## Geo planned failover for a multi-node configuration + +| Component | Configuration | +|-------------|-----------------| +| PostgreSQL | Omnibus-managed | +| Geo site | Multi-node | +| Secondaries | One | + +This runbook will guide you through a planned failover of a multi-node Geo site +with one secondary. The following [2000 user reference architecture](../../../../administration/reference_architectures/2k_users.md) is assumed: + +```mermaid +graph TD + subgraph main[Geo deployment] + subgraph Primary[Primary site, multi-node] + Node_1[Rails node 1] + Node_2[Rails node 2] + Node_3[PostgreSQL node] + Node_4[Gitaly node] + Node_5[Redis node] + Node_6[Monitoring node] + end + subgraph Secondary[Secondary site, multi-node] + Node_7[Rails node 1] + Node_8[Rails node 2] + Node_9[PostgreSQL node] + Node_10[Gitaly node] + Node_11[Redis node] + Node_12[Monitoring node] + end + end +``` + +The load balancer node and optional NFS server are omitted for clarity. + +This guide will result in the following: + +1. An offline primary. +1. A promoted secondary that is now the new primary. + +What is not covered: + +1. Re-adding the old **primary** as a secondary. +1. Adding a new secondary. + +### Preparation + +NOTE: **Note:** +Before following any of those steps, make sure you have `root` access to the +**secondary** to promote it, since there isn't provided an automated way to +promote a Geo replica and perform a failover. + +On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to +review its status. Replicated objects (shown in green) should be close to 100%, +and there should be no failures (shown in red). If a large proportion of +objects aren't yet replicated (shown in gray), consider giving the node more +time to complete. + + + +If any objects are failing to replicate, this should be investigated before +scheduling the maintenance window. After a planned failover, anything that +failed to replicate will be **lost**. + +You can use the +[Geo status API](../../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) +to review failed objects and the reasons for failure. +A common cause of replication failures is the data being missing on the +**primary** node - you can resolve these failures by restoring the data from backup, +or removing references to the missing data. + +The maintenance window won't end until Geo replication and verification is +completely finished. To keep the window as short as possible, you should +ensure these processes are close to 100% as possible during active use. + +If the **secondary** node is still replicating data from the **primary** node, +follow these steps to avoid unnecessary data loss: + +1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) + is implemented, updates must be prevented from happening manually to the + **primary**. Note that your **secondary** node still needs read-only + access to the **primary** node during the maintenance window: + + 1. At the scheduled time, using your cloud provider or your node's firewall, block + all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and + the **secondary** node's IP. + + For instance, you can run the following commands on the **primary** node: + + ```shell + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT --destination-port 22 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 80 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 443 -j REJECT + ``` + + From this point, users will be unable to view their data or make changes on the + **primary** node. They will also be unable to log in to the **secondary** node. + However, existing sessions will work for the remainder of the maintenance period, and + public data will be accessible throughout. + + 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + another IP. The server should refuse connection. + + 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + existing Git repository with an SSH remote URL. The server should refuse + connection. + + 1. On the **primary** node, disable non-Geo periodic background jobs by navigating + to **Admin Area > Monitoring > Background Jobs > Cron**, clicking `Disable All`, + and then clicking `Enable` for the `geo_sidekiq_cron_config_worker` cron job. + This job will re-enable several other cron jobs that are essential for planned + failover to complete successfully. + +1. Finish replicating and verifying all data: + + CAUTION: **Caution:** + Not all data is automatically replicated. Read more about + [what is excluded](../planned_failover.md#not-all-data-is-automatically-replicated). + + 1. If you are manually replicating any + [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), + trigger the final replication process now. + 1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all queues except those with `geo` in the name to drop to 0. + These queues contain work that has been submitted by your users; failing over + before it is completed will cause the work to be lost. + 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the + following conditions to be true of the **secondary** node you are failing over to: + - All replication meters to each 100% replicated, 0% failures. + - All verification meters reach 100% verified, 0% failures. + - Database replication lag is 0ms. + - The Geo log cursor is up to date (0 events behind). + + 1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. + 1. On the **secondary** node, use [these instructions](../../../raketasks/check.md) + to verify the integrity of CI artifacts, LFS objects, and uploads in file + storage. + + At this point, your **secondary** node will contain an up-to-date copy of everything the + **primary** node has, meaning nothing will be lost when you fail over. + +1. In this final step, you need to permanently disable the **primary** node. + + CAUTION: **Caution:** + When the **primary** node goes offline, there may be data saved on the **primary** node + that has not been replicated to the **secondary** node. This data should be treated + as lost if you proceed. + + TIP: **Tip:** + If you plan to [update the **primary** domain DNS record](../index.md#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. + + When performing a failover, we want to avoid a split-brain situation where + writes can occur in two different GitLab instances. So to prepare for the + failover, you must disable the **primary** node: + + - If you have SSH access to the **primary** node, stop and disable GitLab: + + ```shell + sudo gitlab-ctl stop + ``` + + Prevent GitLab from starting up again if the server unexpectedly reboots: + + ```shell + sudo systemctl disable gitlab-runsvdir + ``` + + NOTE: **Note:** + (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being + started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). + It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. + + NOTE: **Note:** + (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu + or any other distribution based on the Upstart init system, you can prevent GitLab + from starting if the machine reboots as `root` with + `initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`. + + - If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting. Since there are many ways you may prefer to accomplish + this, we will avoid a single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (for example, point the **primary** DNS record to the + **secondary** node to stop using the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. + +### Promoting the **secondary** node + +NOTE: **Note:** +A new **secondary** should not be added at this time. If you want to add a new +**secondary**, do this after you have completed the entire process of promoting +the **secondary** to the **primary**. + +CAUTION: **Caution:** +If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` error during this process, read +[the troubleshooting advice](../../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). + +The `gitlab-ctl promote-to-primary-node` command cannot be used yet in +conjunction with multiple servers, as it can only +perform changes on a **secondary** with only a single machine. Instead, you must +do this manually. + +DANGER: **Danger:** +In GitLab 13.2 and later versions, promoting a secondary node to a primary while the secondary is paused fails. We are [investigating the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/225173). Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. + +1. SSH in to the PostgreSQL node in the **secondary** and trigger PostgreSQL to + promote to read-write: + + ```shell + sudo gitlab-pg-ctl promote + ``` + + In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). + +1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to + reflect its new status as **primary** by removing any lines that enabled the + `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + + After making these changes [Reconfigure GitLab](../../../restart_gitlab.md#omnibus-gitlab-reconfigure) each + machine so the changes take effect. + +1. Promote the **secondary** to **primary**. SSH into a single Rails node + server and execute: + + ```shell + sudo gitlab-rake geo:set_secondary_as_primary + ``` + +1. Verify you can connect to the newly promoted **primary** using the URL used + previously for the **secondary**. + +1. Success! The **secondary** has now been promoted to **primary**. + +### Next steps + +To regain geographic redundancy as quickly as possible, you should +[add a new **secondary** node](../../setup/index.md). To +do that, you can re-add the old **primary** as a new secondary and bring it back +online. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md new file mode 100644 index 00000000000..5e847030077 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -0,0 +1,269 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + +CAUTION: **Caution:** +This runbook is in **alpha**. For complete, production-ready documentation, see the +[disaster recovery documentation](../index.md). + +# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)** + +## Geo planned failover for a single-node configuration + +| Component | Configuration | +|-------------|-----------------| +| PostgreSQL | Omnibus-managed | +| Geo site | Single-node | +| Secondaries | One | + +This runbook will guide you through a planned failover of a single-node Geo site +with one secondary. The following general architecture is assumed: + +```mermaid +graph TD + subgraph main[Geo deployment] + subgraph Primary[Primary site] + Node_1[(GitLab node)] + end + subgraph Secondary1[Secondary site] + Node_2[(GitLab node)] + end + end +``` + +This guide will result in the following: + +1. An offline primary. +1. A promoted secondary that is now the new primary. + +What is not covered: + +1. Re-adding the old **primary** as a secondary. +1. Adding a new secondary. + +### Preparation + +NOTE: **Note:** +Before following any of those steps, make sure you have `root` access to the +**secondary** to promote it, since there isn't provided an automated way to +promote a Geo replica and perform a failover. + +On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to +review its status. Replicated objects (shown in green) should be close to 100%, +and there should be no failures (shown in red). If a large proportion of +objects aren't yet replicated (shown in gray), consider giving the node more +time to complete. + + + +If any objects are failing to replicate, this should be investigated before +scheduling the maintenance window. After a planned failover, anything that +failed to replicate will be **lost**. + +You can use the +[Geo status API](../../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) +to review failed objects and the reasons for failure. +A common cause of replication failures is the data being missing on the +**primary** node - you can resolve these failures by restoring the data from backup, +or removing references to the missing data. + +The maintenance window won't end until Geo replication and verification is +completely finished. To keep the window as short as possible, you should +ensure these processes are close to 100% as possible during active use. + +If the **secondary** node is still replicating data from the **primary** node, +follow these steps to avoid unnecessary data loss: + +1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) + is implemented, updates must be prevented from happening manually to the + **primary**. Note that your **secondary** node still needs read-only + access to the **primary** node during the maintenance window: + + 1. At the scheduled time, using your cloud provider or your node's firewall, block + all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and + the **secondary** node's IP. + + For instance, you can run the following commands on the **primary** node: + + ```shell + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT --destination-port 22 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 80 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 443 -j REJECT + ``` + + From this point, users will be unable to view their data or make changes on the + **primary** node. They will also be unable to log in to the **secondary** node. + However, existing sessions will work for the remainder of the maintenance period, and + public data will be accessible throughout. + + 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + another IP. The server should refuse connection. + + 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + existing Git repository with an SSH remote URL. The server should refuse + connection. + + 1. On the **primary** node, disable non-Geo periodic background jobs by navigating + to **Admin Area > Monitoring > Background Jobs > Cron**, clicking `Disable All`, + and then clicking `Enable` for the `geo_sidekiq_cron_config_worker` cron job. + This job will re-enable several other cron jobs that are essential for planned + failover to complete successfully. + +1. Finish replicating and verifying all data: + + CAUTION: **Caution:** + Not all data is automatically replicated. Read more about + [what is excluded](../planned_failover.md#not-all-data-is-automatically-replicated). + + 1. If you are manually replicating any + [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), + trigger the final replication process now. + 1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all queues except those with `geo` in the name to drop to 0. + These queues contain work that has been submitted by your users; failing over + before it is completed will cause the work to be lost. + 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the + following conditions to be true of the **secondary** node you are failing over to: + - All replication meters to each 100% replicated, 0% failures. + - All verification meters reach 100% verified, 0% failures. + - Database replication lag is 0ms. + - The Geo log cursor is up to date (0 events behind). + + 1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. + 1. On the **secondary** node, use [these instructions](../../../raketasks/check.md) + to verify the integrity of CI artifacts, LFS objects, and uploads in file + storage. + + At this point, your **secondary** node will contain an up-to-date copy of everything the + **primary** node has, meaning nothing will be lost when you fail over. + +1. In this final step, you need to permanently disable the **primary** node. + + CAUTION: **Caution:** + When the **primary** node goes offline, there may be data saved on the **primary** node + that has not been replicated to the **secondary** node. This data should be treated + as lost if you proceed. + + TIP: **Tip:** + If you plan to [update the **primary** domain DNS record](../index.md#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. + + When performing a failover, we want to avoid a split-brain situation where + writes can occur in two different GitLab instances. So to prepare for the + failover, you must disable the **primary** node: + + - If you have SSH access to the **primary** node, stop and disable GitLab: + + ```shell + sudo gitlab-ctl stop + ``` + + Prevent GitLab from starting up again if the server unexpectedly reboots: + + ```shell + sudo systemctl disable gitlab-runsvdir + ``` + + NOTE: **Note:** + (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being + started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). + It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. + + NOTE: **Note:** + (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu + or any other distribution based on the Upstart init system, you can prevent GitLab + from starting if the machine reboots as `root` with + `initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`. + + - If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting. Since there are many ways you may prefer to accomplish + this, we will avoid a single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (for example, point the **primary** DNS record to the + **secondary** node to stop using the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. + +### Promoting the **secondary** node + +Note the following when promoting a secondary: + +- A new **secondary** should not be added at this time. If you want to add a new + **secondary**, do this after you have completed the entire process of promoting + the **secondary** to the **primary**. +- If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` + error during this process, read + [the troubleshooting advice](../../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). + +To promote the secondary node: + +1. SSH in to your **secondary** node and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by + removing any lines that enabled the `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + +1. Run the following command to list out all preflight checks and automatically + check if replication and verification are complete before scheduling a planned + failover to ensure the process will go smoothly: + + ```shell + gitlab-ctl promotion-preflight-checks + ``` + +1. Promote the **secondary**: + + ```shell + gitlab-ctl promote-to-primary-node + ``` + + If you have already run the [preflight checks](../planned_failover.md#preflight-checks) + or don't want to run them, you can skip them: + + ```shell + gitlab-ctl promote-to-primary-node --skip-preflight-check + ``` + + You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: + + ```shell + sudo gitlab-ctl promote-to-primary-node --force + ``` + +1. Verify you can connect to the newly promoted **primary** node using the URL used + previously for the **secondary** node. + + If successful, the **secondary** node has now been promoted to the **primary** node. + +### Next steps + +To regain geographic redundancy as quickly as possible, you should +[add a new **secondary** node](../../setup/index.md). To +do that, you can re-add the old **primary** as a new secondary and bring it back +online. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 6fdf213ac78..47d19c1e12c 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -39,7 +39,7 @@ Implementing Geo provides the following benefits: In addition, it: -- Can be used for cloning and fetching projects, in addition to reading any data available in the GitLab web interface (see [current limitations](#current-limitations)). +- Can be used for cloning and fetching projects, in addition to reading any data available in the GitLab web interface (see [limitations](#limitations)). - Overcomes slow connections between distant offices, saving time by improving speed for distributed teams. - Helps reducing the loading time for automated tasks, custom integrations, and internal workflows. - Can quickly fail over to a **secondary** node in a [disaster recovery](disaster_recovery/index.md) scenario. @@ -69,7 +69,7 @@ Keep in mind that: - Replicate repositories, LFS Objects, and Attachments (HTTPS + JWT). - Since GitLab Premium 10.0, the **primary** node no longer talks to **secondary** nodes to notify for changes (API). - Pushing directly to a **secondary** node (for both HTTP and SSH, including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. -- There are [limitations](#current-limitations) in the current implementation. +- There are [limitations](#limitations) when using Geo. ### Architecture @@ -195,6 +195,9 @@ For information on how to update your Geo nodes to the latest GitLab version, se > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +DANGER: **Danger:** +In GitLab 13.2 and later versions, promoting a secondary node to a primary while the secondary is paused fails. We are [investigating the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/225173). Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. + In some circumstances, like during [upgrades](replication/updating_the_geo_nodes.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. Pausing and resuming replication is done via a command line tool from the secondary node. @@ -247,7 +250,7 @@ For more information on removing a Geo node, see [Removing **secondary** Geo nod To find out how to disable Geo, see [Disabling Geo](replication/disable_geo.md). -## Current limitations +## Limitations CAUTION: **Caution:** This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. @@ -261,6 +264,7 @@ This list of limitations only reflects the latest version of GitLab. If you are - Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. - [External merge request diffs](../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. - GitLab Runners cannot register with a **secondary** node. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). +- Geo **secondary** nodes can not be configured to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536). ### Limitations on replication/verification diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 166a724f9c1..52a2b1521a9 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -47,6 +47,8 @@ verification methods: | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | | Blobs | Package registry _(filesystem)_ | Geo with API | _Not implemented_ | | Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Versioned Terraform State _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo nodes. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance @@ -160,39 +162,33 @@ replicating data from those features will cause the data to be **lost**. If you wish to use those features on a **secondary** node, or to execute a failover successfully, you must replicate their data using some other means. -| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Notes | -|:------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------|:----------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------| -| Application data in PostgreSQL | **Yes** (10.2) | **Yes** (10.2) | | -| Project repository | **Yes** (10.2) | **Yes** (10.7) | | -| Project wiki repository | **Yes** (10.2) | **Yes** (10.7) | | -| Project designs repository | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | | -| Uploads | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Verified only on transfer, or manually (*1*) | -| LFS objects | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Verified only on transfer, or manually (*1*). Unavailable for new LFS objects in 11.11.x and 12.0.x (*2*). | -| CI job artifacts (other than traces) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only manually (*1*) | -| Archived traces | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only on transfer, or manually (*1*) | -| Personal snippets | **Yes** (10.2) | **Yes** (10.2) | | -| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | | -| Project snippets | **Yes** (10.2) | **Yes** (10.2) | | -| Object pools for forked project deduplication | **Yes** | No | | -| [Server-side Git hooks](../../server_hooks.md) | No | No | | -| [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | | -| [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | | -| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | | -| [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Maven Repository](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Conan Repository](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | -| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | -| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | -| [External merge request diffs](../../merge_request_diffs.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/33817) | No | | -| [Terraform State](../../terraform_state.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3112)(*3*) | No | | -| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3111)(*3*) | No | | -| Content in object storage | **Yes** (12.4) | No | | - -- (*1*): The integrity can be verified manually using - [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing - the output between them. -- (*2*): GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new - LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). -- (*3*): If you are using Object Storage, the replication can be performed by the - Object Storage provider if supported. Please see [Geo with Object Storage](object_storage.md) +| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (please see [Geo with Object Storage](object_storage.md)) | Notes | +|:---------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +| [Project repository](../../..//user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | +| [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | +| [Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. | +| [LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | +| [Personal snippets](../../../user/snippets.md#personal-snippets) | **Yes** (10.2) | **Yes** (10.2) | No | | +| [Project snippets](../../../user/snippets.md#project-snippets) | **Yes** (10.2) | **Yes** (10.2) | No | | +| [CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta) . | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | +| [Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | +| [Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | +| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | | +| [Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | +| [Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | +| [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Maven Repository](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Conan Repository](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default | +| [External merge request diffs](../../merge_request_diffs.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/33817) | No | Via Object Storage provider if supported. Native Geo support (Beta). | | +| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | +| [Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | +| [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | +| [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | +| [CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | +| [Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | +| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index aed8e5fc3bc..14a11d9c1e3 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -29,7 +29,7 @@ anymore on these nodes. You can follow our docs to [remove your secondary Geo no If the current node that you want to keep using is a secondary node, you need to first promote it to primary. You can use our steps on [how to promote a secondary node](../disaster_recovery/#step-3-promoting-a-secondary-node) -in order to do that. +to do that. ## Remove the primary node from the UI diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index 3892d73b465..f7f391b360e 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -67,3 +67,7 @@ That's totally fine. We use HTTP(s) to fetch repository changes from the **prima ## Is this possible to set up a Docker Registry for a **secondary** node that mirrors the one on the **primary** node? Yes. See [Docker Registry for a **secondary** node](docker_registry.md). + +## Can I login to a secondary node? + +Yes, but secondary nodes receive all authentication data (like user accounts and logins) from the primary instance. This means you will be re-directed to the primary for authentication and routed back afterwards. diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 8247b8c6336..efd070635cb 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -114,6 +114,22 @@ The following are GitLab upgrade validation tests we performed. The following are PostgreSQL upgrade validation tests we performed. +### September 2020 + +[Verify PostgreSQL 12 upgrade for Geo installations](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5454): + +- Description: With PostgreSQL 12 available as an opt-in version in GitLab 13.3, we tested upgrading + existing Geo installations from PostgreSQL 11 to 12. We also re-tested fresh installations of GitLab + with Geo after fixes were made to support PostgreSQL 12. These tests were done using a + [nightly build](https://packages.gitlab.com/gitlab/nightly-builds/packages/ubuntu/bionic/gitlab-ee_13.3.6+rnightly.169516.d5209202-0_amd64.deb) + of GitLab 13.4. +- Outcome: Tests were successful for Geo deployments with a single database node on the primary and secondary. + We encountered known issues with repmgr and Patroni managed PostgreSQL clusters on the Geo primary. Using + PostgreSQL 12 with a database cluster on the primary is not recommended until the issues are resolved. +- Known issues for PostgreSQL clusters: + - [Ensure Patroni detects PostgreSQL update](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5423) + - [Allow configuring permanent replication slots in patroni](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5628) + ### August 2020 [Verify Geo installation with PostgreSQL 12](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5453): diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index cba41c375a3..9828c52ee7d 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -133,7 +133,7 @@ Configure the following services, again using the non-Geo multi-node documentation: - [Configuring Redis for GitLab](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application) for multiple nodes. -- [Gitaly](../../high_availability/gitaly.md), which will store data that is +- [Gitaly](../../gitaly/index.md), which will store data that is synchronized from the **primary** node. NOTE: **Note:** diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index f6d6f39fb19..0b6ff867f11 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -386,6 +386,15 @@ This happens when you have added IP addresses without a subnet mask in `postgres To fix this, add the subnet mask in `/etc/gitlab/gitlab.rb` under `postgresql['md5_auth_cidr_addresses']` to respect the CIDR format (i.e. `1.2.3.4/32`). +### Message: `Found data in the gitlabhq_production database!` when running `gitlab-ctl replicate-geo-database` + +This happens if data is detected in the `projects` table. When one or more projects are detected, the operation +is aborted to prevent accidental data loss. To bypass this message, pass the `--force` option to the command. + +In GitLab 13.4, a seed project is added when GitLab is first installed. This makes it necessary to pass `--force` even +on a new Geo secondary node. There is an [issue to account for seed projects](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5618) +when checking the database. + ### Very large repositories never successfully synchronize on the **secondary** node GitLab places a timeout on all repository clones, including project imports @@ -483,8 +492,8 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl start geo-postgresql ``` - Reconfigure in order to recreate the folders and make sure permissions and ownership - are correctly + Reconfigure to recreate the folders and make sure permissions and ownership + are correct: ```shell gitlab-ctl reconfigure diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index b78aeb06ebf..1af2b8d0b88 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -21,14 +21,17 @@ Updating Geo nodes involves performing: NOTE: **Note:** These general update steps are not intended for [high-availability deployments](https://docs.gitlab.com/omnibus/update/README.html#multi-node--ha-deployment), and will cause downtime. If you want to avoid downtime, consider using [zero downtime updates](https://docs.gitlab.com/omnibus/update/README.html#zero-downtime-updates). +DANGER: **Danger:** +In GitLab 13.2 and later versions, promoting a secondary node to a primary while the secondary is paused fails. We are [investigating the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/225173). Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. + To update the Geo nodes when a new GitLab version is released, update **primary** and all **secondary** nodes: 1. **Optional:** [Pause replication on each **secondary** node.](../index.md#pausing-and-resuming-replication) 1. Log into the **primary** node. -1. [Update GitLab on the **primary** node using Omnibus](https://docs.gitlab.com/omnibus/update/README.html). +1. [Update GitLab on the **primary** node using Omnibus's Geo-specific steps](https://docs.gitlab.com/omnibus/update/README.html#geo-deployment). 1. Log into each **secondary** node. -1. [Update GitLab on each **secondary** node using Omnibus](https://docs.gitlab.com/omnibus/update/README.html). +1. [Update GitLab on each **secondary** node using Omnibus's Geo-specific steps](https://docs.gitlab.com/omnibus/update/README.html#geo-deployment). 1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication) 1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 1ae246e3e61..71facb808ab 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -447,8 +447,8 @@ Omnibus is the following: > **IMPORTANT**: With GitLab 9.0, the PostgreSQL version is updated to 9.6 and manual steps are -required in order to update the **secondary** nodes and keep the Streaming -Replication working. Downtime is required, so plan ahead. +required to update the **secondary** nodes and keep the Streaming Replication +working. Downtime is required, so plan ahead. The following steps apply only if you update from a 8.17 GitLab version to 9.0+. For previous versions, update to GitLab 8.17 first before attempting to diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index aefa8a0e399..09b9c71aeb7 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -17,9 +17,10 @@ NOTE: **Note:** The stages of the setup process must be completed in the documented order. Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). -This document describes the minimal steps you have to take in order to -replicate your **primary** GitLab database to a **secondary** node's database. You may -have to change some values according to your database setup, how big it is, etc. +This document describes the minimal steps you have to take to replicate your +**primary** GitLab database to a **secondary** node's database. You may have to +change some values, based on attributes including your database's setup and +size. You are encouraged to first read through all the steps before executing them in your testing/production environment. @@ -433,6 +434,11 @@ data before running `pg_basebackup`. NOTE: **Note:** Replication slot names must only contain lowercase letters, numbers, and the underscore character. + NOTE: **Note:** + In GitLab 13.4, a seed project is added when GitLab is first installed. This makes it necessary to pass `--force` even + on a new Geo secondary node. There is an [issue to account for seed projects](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5618) + when checking the database. + When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` user in the first step. diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index e6b137bac29..750e6aab687 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -90,7 +90,7 @@ When running Gitaly on its own server, note the following regarding GitLab versi leveraged for redundancy on block-level Git data, but only has to be mounted on the Gitaly servers. - From GitLab 11.8 to 12.2, it is possible to use Elasticsearch in a Gitaly setup that doesn't use - NFS. In order to use Elasticsearch in these versions, the + NFS. To use Elasticsearch in these versions, the [repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer) must be enabled in your GitLab configuration. - [Since GitLab 12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/6481), the new indexer is @@ -382,10 +382,10 @@ if previously enabled manually. Gitaly makes the following assumptions: - Your `gitaly1.internal` Gitaly server can be reached at `gitaly1.internal:8075` from your Gitaly - clients, and that Gitaly server can read and write to `/mnt/gitlab/default` and + clients, and that Gitaly server can read, write, and set permissions on `/mnt/gitlab/default` and `/mnt/gitlab/storage1`. - Your `gitaly2.internal` Gitaly server can be reached at `gitaly2.internal:8075` from your Gitaly - clients, and that Gitaly server can read and write to `/mnt/gitlab/storage2`. + clients, and that Gitaly server can read, write, and set permissions on `/mnt/gitlab/storage2`. - Your `gitaly1.internal` and `gitaly2.internal` Gitaly servers can reach each other. You can't define Gitaly servers with some as a local Gitaly server @@ -424,17 +424,17 @@ server (with `gitaly_address`) unless you setup with special storages: default: gitaly_address: tcp://gitaly1.internal:8075 - path: /some/dummy/path + path: /some/local/path storage1: gitaly_address: tcp://gitaly1.internal:8075 - path: /some/dummy/path + path: /some/local/path storage2: gitaly_address: tcp://gitaly2.internal:8075 - path: /some/dummy/path + path: /some/local/path ``` NOTE: **Note:** - `/some/dummy/path` should be set to a local folder that exists, however no data will be stored in + `/some/local/path` should be set to a local folder that exists, however no data will be stored in this folder. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. @@ -482,6 +482,14 @@ git_data_dirs({ 'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075', 'path' => '/mnt/gitlab/git-data' }, 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, }) + +# Make Gitaly accept connections on all network interfaces +gitaly['listen_addr'] = "0.0.0.0:8075" + +# Or for TLS +gitaly['tls_listen_addr'] = "0.0.0.0:9999" +gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" +gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" ``` `path` can only be included for storage shards on the local Gitaly server. @@ -532,20 +540,12 @@ corresponding to each Gitaly server must be installed on that Gitaly server. Additionally, the certificate (or its certificate authority) must be installed on all: -- Gitaly servers, including the Gitaly server using the certificate. +- Gitaly servers. - Gitaly clients that communicate with it. -The process is documented in the -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) -and repeated below. - Note the following: -- The certificate must specify the address you use to access the Gitaly server. If you are: - - Addressing the Gitaly server by a hostname, you can either use the Common Name field for this, - or add it as a Subject Alternative Name. - - Addressing the Gitaly server by its IP address, you must add it as a Subject Alternative Name to - the certificate. [gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). +- The certificate must specify the address you use to access the Gitaly server. You must add the hostname or IP address as a Subject Alternative Name to the certificate. - You can configure Gitaly servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to gradually transition from unencrypted to encrypted traffic if necessary. @@ -631,17 +631,17 @@ To configure Gitaly with TLS: storages: default: gitaly_address: tls://gitaly1.internal:9999 - path: /some/dummy/path + path: /some/local/path storage1: gitaly_address: tls://gitaly1.internal:9999 - path: /some/dummy/path + path: /some/local/path storage2: gitaly_address: tls://gitaly2.internal:9999 - path: /some/dummy/path + path: /some/local/path ``` NOTE: **Note:** - `/some/dummy/path` should be set to a local folder that exists, however no data will be stored + `/some/local/path` should be set to a local folder that exists, however no data will be stored in this folder. This will no longer be necessary after [Gitaly issue #1282](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. @@ -1021,6 +1021,9 @@ The second facet presents the only real solution. For this, we developed ## Troubleshooting Gitaly +Check [Gitaly timeouts](../../user/admin_area/settings/gitaly_timeouts.md) when troubleshooting +Gitaly. + ### Checking versions when using standalone Gitaly servers When using standalone Gitaly servers, you must make sure they are the same version @@ -1242,13 +1245,6 @@ unset http_proxy unset https_proxy ``` -### Gitaly not listening on new address after reconfiguring - -When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` -values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. - -When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2521) is resolved. - ### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly server If this error occurs even though file permissions are correct, it's likely that diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 876904a2093..45c077cded1 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -547,14 +547,14 @@ To configure Praefect with TLS: storages: default: gitaly_address: tls://praefect1.internal:3305 - path: /some/dummy/path + path: /some/local/path storage1: gitaly_address: tls://praefect2.internal:3305 - path: /some/dummy/path + path: /some/local/path ``` NOTE: **Note:** - `/some/dummy/path` should be set to a local folder that exists, however no + `/some/local/path` should be set to a local folder that exists, however no data will be stored in this folder. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. @@ -993,6 +993,8 @@ information, see the [strong consistency epic](https://gitlab.com/groups/gitlab- To enable strong consistency: +- In GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable + strong consistency. - In GitLab 13.4 and later, the strong consistency voting strategy has been improved. Instead of requiring all nodes to agree, only the primary and half of the secondaries need to agree. This strategy is enabled by default. To diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 0c211c220d7..53001b946d8 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -138,8 +138,8 @@ Most of the time we use `git cat-file --batch` processes for that. For better performance, Gitaly can re-use these `git cat-file` processes across RPC calls. Previously used processes are kept around in a ["Git cat-file cache"](https://about.gitlab.com/blog/2019/07/08/git-performance-on-nfs/#enter-cat-file-cache). -In order to control how much system resources this uses, we have a maximum number -of cat-file processes that can go into the cache. +To control how much system resources this uses, we have a maximum number of +cat-file processes that can go into the cache. The default limit is 100 `cat-file`s, which constitute a pair of `git cat-file --batch` and `git cat-file --batch-check` processes. If diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 4110f8b7646..2882b05f415 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -28,6 +28,9 @@ the `pushes_since_gc` value is 200 a `git gc` will be run. `git add`. - `git repack` ([man page](https://mirrors.edge.kernel.org/pub/software/scm/git/docs/git-repack.html)) re-organize existing packs into a single, more efficient pack. +Housekeeping will also [remove unreferenced LFS files](../raketasks/cleanup.md#remove-unreferenced-lfs-files) +from your project on the same schedule as the `git gc` operation, freeing up storage space for your project. + You can find this option under your project's **Settings > General > Advanced**.  diff --git a/doc/administration/img/export_audit_log_v13_4.png b/doc/administration/img/export_audit_log_v13_4.png Binary files differindex 1b404b5742c..e4ba330b8a9 100644 --- a/doc/administration/img/export_audit_log_v13_4.png +++ b/doc/administration/img/export_audit_log_v13_4.png diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index c0c03044225..f8c1a550b67 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -90,7 +90,7 @@ Be careful when choosing the domain used for receiving incoming email. For the sake of example, suppose your top-level company domain is `hooli.com`. All employees in your company have an email address at that domain via Google Apps, and your company's private Slack instance requires a valid `@hooli.com` -email address in order to sign up. +email address to sign up. If you also host a public-facing GitLab instance at `hooli.com` and set your incoming email domain to `hooli.com`, an attacker could abuse the "Create new diff --git a/doc/administration/index.md b/doc/administration/index.md index a6448fcf64f..076658ead0e 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -52,8 +52,10 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [GitLab Pages configuration](pages/index.md): Enable and configure GitLab Pages. - [GitLab Pages configuration for GitLab source installations](pages/source.md): Enable and configure GitLab Pages on [source installations](../install/installation.md#installation-from-source). - [Uploads administration](uploads.md): Configure GitLab uploads storage. -- [Environment variables](environment_variables.md): Supported environment variables that can be used to override their defaults values in order to configure GitLab. -- [Plugins](plugins.md): With custom plugins, GitLab administrators can introduce custom integrations without modifying GitLab's source code. +- [Environment variables](environment_variables.md): Supported environment + variables that can be used to override their default values to configure + GitLab. +- [Plugins](file_hooks.md): With custom plugins, GitLab administrators can introduce custom integrations without modifying GitLab's source code. - [Enforcing Terms of Service](../user/admin_area/settings/terms.md) - [Third party offers](../user/admin_area/settings/third_party_offers.md) - [Compliance](compliance.md): A collection of features from across the application that you may configure to help ensure that your GitLab instance and DevOps workflow meet compliance standards. @@ -113,7 +115,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Kerberos authentication](../integration/kerberos.md) **(STARTER ONLY)** - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). - [Email users](../tools/email.md): Email GitLab users from within GitLab. **(STARTER ONLY)** -- [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. +- [User Cohorts](../user/admin_area/analytics/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. - [Audit logs and events](audit_events.md): View the changes made within the GitLab server for: - Groups and projects. **(STARTER)** - Instances. **(PREMIUM ONLY)** diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index abd98002934..6729338e0c7 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -480,7 +480,7 @@ indexed](#maximum-file-size-indexed)). - For self-managed installations it is unlimited by default This limit can be configured for self-managed installations when [enabling -Elasticsearch](../integration/elasticsearch.md#enabling-elasticsearch). +Elasticsearch](../integration/elasticsearch.md#enabling-advanced-search). NOTE: **Note:** Set the limit to `0` to disable it. @@ -552,6 +552,9 @@ Plan.default.actual_limits.update!(maven_max_file_size: 100.megabytes) # For PyPI Packages Plan.default.actual_limits.update!(pypi_max_file_size: 100.megabytes) + +# For Debian Packages +Plan.default.actual_limits.update!(debian_max_file_size: 100.megabytes) ``` Set the limit to `0` to allow any file size. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 2a79923b793..fd658116289 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -99,8 +99,13 @@ artifacts, you can use an object storage like AWS S3 instead. This configuration relies on valid AWS credentials to be configured already. Use an object storage option like AWS S3 to store job artifacts. +If you configure GitLab to store artifacts on object storage, you may also want to +[eliminate local disk usage for job logs](job_logs.md#prevent-local-disk-usage). +In both cases, job logs are archived and moved to object storage when the job completes. + DANGER: **Danger:** -If you configure GitLab to store CI logs and artifacts on object storage, you must also enable [incremental logging](job_logs.md#new-incremental-logging-architecture). Otherwise, job logs will disappear or not be saved. +In a multi-server setup you must use one of the options to +[eliminate local disk usage for job logs](job_logs.md#prevent-local-disk-usage), or job logs could be lost. [Read more about using object storage with GitLab](object_storage.md). @@ -117,9 +122,9 @@ For source installations the following settings are nested under `artifacts:` an |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where Artifacts will be stored| | -| `direct_upload` | Set to true to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | -| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | -| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | +| `direct_upload` | Set to `true` to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | +| `background_upload` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | +| `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | #### Connection settings @@ -203,9 +208,9 @@ _The artifacts are stored by default in enabled: true object_store: enabled: true - remote_directory: "artifacts" # The bucket name + remote_directory: "artifacts" # The bucket name connection: - provider: AWS # Only AWS supported at the moment + provider: AWS # Only AWS supported at the moment aws_access_key_id: AWS_ACCESS_KEY_ID aws_secret_access_key: AWS_SECRET_ACCESS_KEY region: eu-central-1 @@ -316,9 +321,9 @@ _The uploads are stored by default in **In Omnibus installations:** -In order to migrate back to local storage: +To migrate back to local storage: -1. Set both `direct_upload` and `background_upload` to false in `gitlab.rb`, under the artifacts object storage settings. +1. Set both `direct_upload` and `background_upload` to `false` in `gitlab.rb`, under the artifacts object storage settings. 1. [Reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Run `gitlab-rake gitlab:artifacts:migrate_to_local`. 1. Disable object_storage for artifacts in `gitlab.rb`: @@ -419,10 +424,10 @@ generated by [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). that are located in the artifacts archive itself. The metadata file is in a binary format, with additional Gzip compression. -GitLab does not extract the artifacts archive in order to save space, memory -and disk I/O. It instead inspects the metadata file which contains all the -relevant information. This is especially important when there is a lot of -artifacts, or an archive is a very large file. +GitLab doesn't extract the artifacts archive to save space, memory, and disk +I/O. It instead inspects the metadata file which contains all the relevant +information. This is especially important when there is a lot of artifacts, or +an archive is a very large file. When clicking on a specific file, [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) extracts it from the archive and the download begins. This implementation saves space, diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index c34035e3c0c..c89ffb8da8b 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -65,6 +65,15 @@ job logs are automatically migrated to it along with the other job artifacts. See "Phase 4: uploading" in [Data flow](#data-flow) to learn about the process. +## Prevent local disk usage + +If you want to avoid any local disk usage for job logs, +you can do so using one of the following options: + +- Enable the [beta incremental logging](#new-incremental-logging-architecture) feature. +- Set the [job logs location](#changing-the-job-logs-local-location) + to an NFS drive. + ## How to remove job logs There isn't a way to automatically expire old job logs, but it's safe to remove diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index beecd9e4783..428a4b97a38 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -72,10 +72,9 @@ Then run `sudo gitlab-ctl reconfigure` for the changes to take effect. missing images for user email addresses that are not found on the Libravatar service. -In order to use a set other than `identicon`, replace the `&d=identicon` -portion of the URL with another supported set. -For example, you can use the `retro` set, in which case the URL would look like: -`plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` +To use a set other than `identicon`, replace the `&d=identicon` portion of the +URL with another supported set. For example, you can use the `retro` set, in +which case the URL would look like: `plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` ## Usage examples for Microsoft Office 365 @@ -84,8 +83,8 @@ Note that this service requires a login, so this use case is most useful in a corporate installation where all users have access to Office 365. ```ruby -gitlab_rails['gravatar_plain_url'] = 'http://outlook.office365.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' -gitlab_rails['gravatar_ssl_url'] = 'https://outlook.office365.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' +gitlab_rails['gravatar_plain_url'] = 'http://outlook.office.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' +gitlab_rails['gravatar_ssl_url'] = 'https://outlook.office.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' ``` <!-- ## Troubleshooting diff --git a/doc/administration/logs.md b/doc/administration/logs.md index fcd6264dafd..88736170d23 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -100,9 +100,9 @@ The ActionCable connection or channel class is used as the `controller`. ```json { - "method":{}, - "path":{}, - "format":{}, + "method":null, + "path":null, + "format":null, "controller":"IssuesChannel", "action":"subscribe", "status":200, diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 3f4cd6e2751..3c093d44780 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -17,12 +17,11 @@ By default, merge request diffs are stored in the database, in a table named `merge_request_diff_files`. Larger installations may find this table grows too large, in which case, switching to external storage is recommended. -## Using external storage - Merge request diffs can be stored on disk, or in object storage. In general, it -is better to store the diffs in the database than on disk. +is better to store the diffs in the database than on disk. A compromise is available +that only [stores outdated diffs](#alternative-in-database-storage) outside of database. -To enable external storage of merge request diffs, follow the instructions below. +## Using external storage **In Omnibus installations:** @@ -68,16 +67,40 @@ To enable external storage of merge request diffs, follow the instructions below ## Using object storage -CAUTION: **WARNING:** - Currently migrating to object storage is **non-reversible** +CAUTION: **Warning:** +Currently migrating to object storage is **non-reversible** Instead of storing the external diffs on disk, we recommended the use of an object store like AWS S3 instead. This configuration relies on valid AWS credentials to be configured already. +**In Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['external_diffs_enabled'] = true + ``` + +1. Set [object storage settings](#object-storage-settings). +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**In installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + external_diffs: + enabled: true + ``` + +1. Set [object storage settings](#object-storage-settings). +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. + [Read more about using object storage with GitLab](object_storage.md). -## Object Storage Settings +### Object Storage Settings NOTE: **Note:** In GitLab 13.2 and later, we recommend using the @@ -92,12 +115,12 @@ then `object_store:`. On Omnibus installations, they are prefixed by |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where external diffs will be stored| | -| `direct_upload` | Set to true to enable direct upload of external diffs without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | -| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | -| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | +| `direct_upload` | Set to `true` to enable direct upload of external diffs without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | +| `background_upload` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | +| `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | -### S3 compatible connection settings +#### S3 compatible connection settings See [the available connection settings for different providers](object_storage.md#connection-settings). diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index efe31997b25..562e0298871 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -9,17 +9,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7, behind a disabled feature flag (`self_monitoring_project`). > - The feature flag was removed and the Self Monitoring Project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) in GitLab 12.8. -GitLab has been adding the ability for administrators to see insights into the health of -their GitLab instance. In order to surface this experience in a native way, similar to how -you would interact with an application deployed via GitLab, a base project called -"GitLab self monitoring" with -[internal visibility](../../../public_access/public_access.md#internal-projects) will be -added under a group called "GitLab Instance Administrators" specifically created for -visualizing and configuring the monitoring of your GitLab instance. - -All administrators at the time of creation of the project and group will be added -as maintainers of the group and project, and as an admin, you'll be able to add new -members to the group in order to give them maintainer access to the project. +GitLab has been adding the ability for administrators to see insights into the +health of their GitLab instance. To surface this experience in a native way +(similar to how you would interact with an application deployed using GitLab), +a base project called "GitLab self monitoring" with +[internal visibility](../../../public_access/public_access.md#internal-projects) +will be added under a group called "GitLab Instance Administrators" +specifically created for visualizing and configuring the monitoring of your +GitLab instance. + +All administrators at the time of creation of the project and group will be +added as maintainers of the group and project, and as an admin, you'll be able +to add new members to the group to give them maintainer access to the project. This project is used to self monitor your GitLab instance. The metrics dashboard of the project shows some basic resource usage charts, such as CPU and memory usage @@ -74,7 +75,8 @@ you should ## Taking action on Prometheus alerts **(ULTIMATE)** You can [add a webhook](../../../operations/metrics/alerts.md#external-prometheus-instances) -to the Prometheus configuration in order for GitLab to receive notifications of any alerts. +to the Prometheus configuration for GitLab to receive notifications of any +alerts. Once the webhook is setup, you can [take action on incoming alerts](../../../operations/metrics/alerts.md#trigger-actions-from-alerts). diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 6f22327e499..e3cccad5e0b 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -11,7 +11,7 @@ GitLab comes with its own application performance measuring system as of GitLab Community and Enterprise editions. Apart from this introduction, you are advised to read through the following -documents in order to understand and properly configure GitLab Performance Monitoring: +documents to understand and properly configure GitLab Performance Monitoring: - [GitLab Configuration](gitlab_configuration.md) - [Prometheus documentation](../prometheus/index.md) diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index ae31a3db023..06b67a19483 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -184,12 +184,12 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_package_files_synced` | Gauge | 13.3 | Number of syncable package files synced on secondary | `url` | | `geo_package_files_failed` | Gauge | 13.3 | Number of syncable package files failed to sync on secondary | `url` | | `geo_package_files_registry` | Gauge | 13.3 | Number of package files in the registry | `url` | -| `geo_terraform_states` | Gauge | 13.3 | Number of terraform states on primary | `url` | -| `geo_terraform_states_checksummed` | Gauge | 13.3 | Number of terraform states checksummed on primary | `url` | -| `geo_terraform_states_checksum_failed` | Gauge | 13.3 | Number of terraform states failed to calculate the checksum on primary | `url` | -| `geo_terraform_states_synced` | Gauge | 13.3 | Number of syncable terraform states synced on secondary | `url` | -| `geo_terraform_states_failed` | Gauge | 13.3 | Number of syncable terraform states failed to sync on secondary | `url` | -| `geo_terraform_states_registry` | Gauge | 13.3 | Number of terraform states in the registry | `url` | +| `geo_terraform_state_versions` | Gauge | 13.5 | Number of terraform state versions on primary | `url` | +| `geo_terraform_state_versions_checksummed` | Gauge | 13.5 | Number of terraform state versions checksummed on primary | `url` | +| `geo_terraform_state_versions_checksum_failed` | Gauge | 13.5 | Number of terraform state versions failed to calculate the checksum on primary | `url` | +| `geo_terraform_state_versions_synced` | Gauge | 13.5 | Number of syncable terraform state versions synced on secondary | `url` | +| `geo_terraform_state_versions_failed` | Gauge | 13.5 | Number of syncable terraform state versions failed to sync on secondary | `url` | +| `geo_terraform_state_versions_registry` | Gauge | 13.5 | Number of terraform state versions in the registry | `url` | | `global_search_bulk_cron_queue_size` | Gauge | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | | | `global_search_awaiting_indexing_queue_size` | Gauge | 13.2 | Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused | | | `geo_merge_request_diffs` | Gauge | 13.4 | Number of merge request diffs on primary | `url` | @@ -204,6 +204,9 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_snippet_repositories_synced` | Gauge | 13.4 | Number of syncable snippets synced on secondary | `url` | | `geo_snippet_repositories_failed` | Gauge | 13.4 | Number of syncable snippets failed on secondary | `url` | | `geo_snippet_repositories_registry` | Gauge | 13.4 | Number of syncable snippets in the registry | `url` | +| `limited_capacity_worker_running_jobs` | Gauge | 13.5 | Number of running jobs | `worker` | +| `limited_capacity_worker_max_running_jobs` | Gauge | 13.5 | Maximum number of running jobs | `worker` | +| `limited_capacity_worker_remaining_work_count` | Gauge | 13.5 | Number of jobs waiting to be enqueued | `worker` | ## Database load balancing metrics **(PREMIUM ONLY)** diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 7d93e9797be..fcfa253a1e3 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -60,9 +60,9 @@ it's not recommended to change the port Prometheus listens on, as this might affect or conflict with other services running on the GitLab server. Proceed at your own risk. -In order to access Prometheus from outside the GitLab server you will need to -set a FQDN or IP in `prometheus['listen_address']`. -To change the address/port that Prometheus listens on: +To access Prometheus from outside the GitLab server, set an FQDN or IP in +`prometheus['listen_address']`. To change the address/port that Prometheus +listens on: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line: diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index b54f05ad536..8fd89b77cc3 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -296,6 +296,25 @@ Having multiple NFS mounts will require manually making sure the data directorie are empty before attempting a restore. Read more about the [restore prerequisites](../raketasks/backup_restore.md). +## Testing NFS + +Once you've set up the NFS server and client, you can verify NFS is configured correctly +by testing the following commands: + +```shell +sudo mkdir /gitlab-nfs/test-dir +sudo chown git /gitlab-nfs/test-dir +sudo chgrp gitlab-www /gitlab-nfs/test-dir +sudo chgrp root /gitlab-nfs/test-dir +sudo chmod 2755 /gitlab-nfs/test-dir +sudo -u git mkdir /gitlab-nfs/test-dir/test2 +sudo -u git chmod 2755 /gitlab-nfs/test-dir/test2 +sudo ls -lah /gitlab-nfs/test-dir/test2 +sudo -u git rm -r /gitlab-nfs/test-dir +``` + +Any `Operation not permitted` errors means you should investigate your NFS server export options. + ## NFS in a Firewalled Environment If the traffic between your NFS server and NFS client(s) is subject to port filtering diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 39365ffe404..f1352446136 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -369,7 +369,7 @@ Here are the valid connection parameters for Rackspace Cloud, provided by | `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` | | `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` | | `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | -| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for temporary URLs. Read more [here](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl) | `ABC123DEF456ABC123DEF456ABC123DE` | +| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for [temporary URLs](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | NOTE: **Note:** Regardless of whether the container has public access enabled or disabled, Fog will @@ -503,13 +503,13 @@ supported by consolidated configuration form, refer to the following guides: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| | [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| -| [Job artifacts](job_artifacts.md#using-object-storage) and [incremental logging](job_logs.md#new-incremental-logging-architecture) | Yes | +| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](uploads.md#using-object-storage) | Yes | | [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | No | | [Merge request diffs](merge_request_diffs.md#using-object-storage) | Yes | | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | -| [Packages](packages/index.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Packages](packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | | [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | @@ -520,12 +520,13 @@ supported by consolidated configuration form, refer to the following guides: If you're working to [scale out](reference_architectures/index.md) your GitLab implementation, or add fault tolerance and redundancy, you may be looking at removing dependencies on block or network filesystems. -See the following guides and +See the following additional guides and [note that Pages requires disk storage](#gitlab-pages-requires-nfs): 1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk. 1. Configure [database lookup of SSH keys](operations/fast_ssh_key_lookup.md) to eliminate the need for a shared `authorized_keys` file. +1. [Prevent local disk usage for job logs](job_logs.md#prevent-local-disk-usage). ## Warnings, limitations, and known issues @@ -569,7 +570,8 @@ The dependency on disk storage also prevents Pages being deployed using the ### Incremental logging is required for CI to use object storage If you configure GitLab to use object storage for CI logs and artifacts, -[you must also enable incremental logging](job_artifacts.md#using-object-storage). +you can avoid [local disk usage for job logs](job_logs.md#data-flow) by enabling +[beta incremental logging](job_logs.md#new-incremental-logging-architecture). ### Proxy Download diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 74af5c8149b..29bf83ce2e0 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -10,15 +10,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Container Registry manifest `v1` support was added in GitLab 8.9 to support > Docker versions earlier than 1.10. -NOTE: **Note:** -This document is the administrator's guide. To learn how to use GitLab Container -Registry, see the [user documentation](../../user/packages/container_registry/index.md). +With the GitLab Container Registry, every project can have its +own space to store Docker images. -With the Container Registry integrated into GitLab, every project can have its -own space to store its Docker images. +Read more about the Docker Registry in [the Docker documentation](https://docs.docker.com/registry/introduction/). -You can read more about the Docker Registry at -<https://docs.docker.com/registry/introduction/>. +This document is the administrator's guide. To learn how to use the GitLab Container +Registry, see the [user documentation](../../user/packages/container_registry/index.md). ## Enable the Container Registry @@ -37,7 +35,6 @@ Otherwise, the Container Registry is not enabled. To enable it: - You can configure it for your [GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain), or - You can configure it for [a different domain](#configure-container-registry-under-its-own-domain). -NOTE: **Note:** The Container Registry works under HTTPS by default. You can use HTTP but it's not recommended and is out of the scope of this document. Read the [insecure Registry documentation](https://docs.docker.com/registry/insecure/) @@ -47,12 +44,12 @@ if you want to implement this. If you have installed GitLab from source: -1. You will have to [install Registry](https://docs.docker.com/registry/deploying/) by yourself. -1. After the installation is complete, you will have to configure the Registry's - settings in `gitlab.yml` in order to enable it. -1. Use the sample NGINX configuration file that is found under +1. You must [install Registry](https://docs.docker.com/registry/deploying/) by yourself. +1. After the installation is complete, to enable it, you must configure the Registry's + settings in `gitlab.yml`. +1. Use the sample NGINX configuration file from under [`lib/support/nginx/registry-ssl`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/support/nginx/registry-ssl) and edit it to match the - `host`, `port` and TLS certs paths. + `host`, `port`, and TLS certificate paths. The contents of `gitlab.yml` are: @@ -67,19 +64,18 @@ registry: issuer: gitlab-issuer ``` -where: +Where: | Parameter | Description | | --------- | ----------- | | `enabled` | `true` or `false`. Enables the Registry in GitLab. By default this is `false`. | -| `host` | The host URL under which the Registry will run and the users will be able to use. | -| `port` | The port under which the external Registry domain will listen on. | -| `api_url` | The internal API URL under which the Registry is exposed to. It defaults to `http://localhost:5000`. | +| `host` | The host URL under which the Registry runs and users can use. | +| `port` | The port the external Registry domain listens on. | +| `api_url` | The internal API URL under which the Registry is exposed. It defaults to `http://localhost:5000`. | | `key` | The private key location that is a pair of Registry's `rootcertbundle`. Read the [token auth configuration documentation](https://docs.docker.com/registry/configuration/#token). | | `path` | This should be the same directory like specified in Registry's `rootdirectory`. Read the [storage configuration documentation](https://docs.docker.com/registry/configuration/#storage). This path needs to be readable by the GitLab user, the web-server user and the Registry user. Read more in [#configure-storage-for-the-container-registry](#configure-storage-for-the-container-registry). | | `issuer` | This should be the same value as configured in Registry's `issuer`. Read the [token auth configuration documentation](https://docs.docker.com/registry/configuration/#token). | -NOTE: **Note:** A Registry init file is not shipped with GitLab if you install it from source. Hence, [restarting GitLab](../restart_gitlab.md#installations-from-source) will not restart the Registry should you modify its settings. Read the upstream documentation on how to achieve that. @@ -98,21 +94,20 @@ auth: ``` CAUTION: **Caution:** -If `auth` is not set up, users will be able to pull Docker images without authentication. +If `auth` is not set up, users can pull Docker images without authentication. ## Container Registry domain configuration There are two ways you can configure the Registry's external domain. Either: -- [Use the existing GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain) where in that case - the Registry will have to listen on a port and reuse GitLab's TLS certificate, +- [Use the existing GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain). + The Registry listens on a port and reuse GitLab's TLS certificate. - [Use a completely separate domain](#configure-container-registry-under-its-own-domain) with a new TLS certificate for that domain. -Since the container Registry requires a TLS certificate, in the end it all boils -down to how easy or pricey it is to get a new one. +Because the Container Registry requires a TLS certificate, cost may be a factor. -Please take this into consideration before configuring the Container Registry +Take this into consideration before configuring the Container Registry for the first time. ### Configure Container Registry under an existing GitLab domain @@ -126,9 +121,8 @@ Registry is exposed to the outside world is `5050`, here is what you need to set in `gitlab.rb` or `gitlab.yml` if you are using Omnibus GitLab or installed GitLab from source respectively. -NOTE: **Note:** -Be careful to choose a port different than the one that Registry listens to (`5000` by default), -otherwise you will run into conflicts. +Ensure you choose a port different than the one that Registry listens to (`5000` by default), +otherwise there will be conflicts. **Omnibus GitLab installations** @@ -139,7 +133,7 @@ otherwise you will run into conflicts. registry_external_url 'https://gitlab.example.com:5050' ``` - Note how the `registry_external_url` is listening on HTTPS under the + The `registry_external_url` is listening on HTTPS under the existing GitLab URL, but on a different port. If your TLS certificate is not in `/etc/gitlab/ssl/gitlab.example.com.crt` @@ -160,7 +154,6 @@ otherwise you will run into conflicts. openssl s_client -showcerts -servername gitlab.example.com -connect gitlab.example.com:5050 > cacert.pem ``` -NOTE: **Note:** If your certificate provider provides the CA Bundle certificates, append them to the TLS certificate file. **Installations from source** @@ -188,11 +181,10 @@ docker login gitlab.example.com:5050 ### Configure Container Registry under its own domain If the Registry is configured to use its own domain, you will need a TLS -certificate for that specific domain (e.g., `registry.example.com`) or maybe +certificate for that specific domain (for example, `registry.example.com`) or maybe a wildcard certificate if hosted under a subdomain of your existing GitLab -domain (e.g., `registry.gitlab.example.com`). +domain, for example, `registry.gitlab.example.com`. -NOTE: **Note:** As well as manually generated SSL certificates (explained here), certificates automatically generated by Let's Encrypt are also [supported in Omnibus installs](https://docs.gitlab.com/omnibus/settings/ssl.html#host-services). @@ -210,19 +202,19 @@ Let's assume that you want the container Registry to be accessible at chmod 600 /etc/gitlab/ssl/registry.gitlab.example.com.* ``` -1. Once the TLS certificate is in place, edit `/etc/gitlab/gitlab.rb` with: +1. After the TLS certificate is in place, edit `/etc/gitlab/gitlab.rb` with: ```ruby registry_external_url 'https://registry.gitlab.example.com' ``` - Note how the `registry_external_url` is listening on HTTPS. + The `registry_external_url` is listening on HTTPS. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -If you have a [wildcard certificate](https://en.wikipedia.org/wiki/Wildcard_certificate), you need to specify the path to the -certificate in addition to the URL, in this case `/etc/gitlab/gitlab.rb` will -look like: +If you have a [wildcard certificate](https://en.wikipedia.org/wiki/Wildcard_certificate), you must specify the path to the +certificate in addition to the URL, in this case `/etc/gitlab/gitlab.rb` +looks like: ```ruby registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/certificate.pem" @@ -252,9 +244,8 @@ docker login registry.gitlab.example.com ## Disable Container Registry site-wide -NOTE: **Note:** -Disabling the Registry in the Rails GitLab application as set by the following -steps, will not remove any existing Docker images. This is handled by the +When you disable the Registry by following these steps, you do not +remove any existing Docker images. This is handled by the Registry application itself. **Omnibus GitLab** @@ -331,8 +322,7 @@ The different supported drivers are: | swift | OpenStack Swift Object Storage | | oss | Aliyun OSS | -NOTE: **Note:** -Although most S3 compatible services (like [MinIO](https://min.io/)) should work with the registry, we only guarantee support for AWS S3. Because we cannot assert the correctness of third-party S3 implementations, we can debug issues, but we cannot patch the registry unless an issue is reproducible against an AWS S3 bucket. +Although most S3 compatible services (like [MinIO](https://min.io/)) should work with the Container Registry, we only guarantee support for AWS S3. Because we cannot assert the correctness of third-party S3 implementations, we can debug issues, but we cannot patch the registry unless an issue is reproducible against an AWS S3 bucket. Read more about the individual driver's configuration options in the [Docker Registry docs](https://docs.docker.com/registry/configuration/#storage). @@ -347,8 +337,7 @@ This path is accessible to: - The user running the Container Registry daemon. - The user running GitLab. -CAUTION: **Warning:** -You should confirm that all GitLab, Registry and web server users +All GitLab, Registry, and web server users must have access to this directory. **Omnibus GitLab installations** @@ -387,13 +376,10 @@ driver for the Container Registry. [Read more about using object storage with GitLab](../object_storage.md). CAUTION: **Warning:** -GitLab will not backup Docker images that are not stored on the -filesystem. Remember to enable backups with your object storage provider if +GitLab does not back up Docker images that are not stored on the +filesystem. Enable backups with your object storage provider if desired. -NOTE: **Note:** -`regionendpoint` is only required when configuring an S3 compatible service such as MinIO. It takes a URL such as `http://127.0.0.1:9000`. - **Omnibus GitLab installations** To configure the `s3` storage driver in Omnibus: @@ -411,15 +397,15 @@ To configure the `s3` storage driver in Omnibus: } } ``` + + - `regionendpoint` is only required when configuring an S3 compatible service such as MinIO. It takes a URL such as `http://127.0.0.1:9000`. + - `your-s3-bucket` should be the name of a bucket that exists, and can't include subdirectories. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -NOTE: **Note:** -`your-s3-bucket` should only be the name of a bucket that exists, and can't include subdirectories. - **Installations from source** -Configuring the storage driver is done in your registry configuration YML file created +Configuring the storage driver is done in the registry configuration YML file created when you [deployed your Docker registry](https://docs.docker.com/registry/deploying/). `s3` storage driver example: @@ -438,8 +424,7 @@ storage: enabled: true ``` -NOTE: **Note:** -`your-s3-bucket` should only be the name of a bucket that exists, and can't include subdirectories. +`your-s3-bucket` should be the name of a bucket that exists, and can't include subdirectories. #### Migrate to object storage without downtime @@ -489,7 +474,7 @@ you can pull from the Container Registry, but you cannot push. DANGER: **Danger:** The [`--delete`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) - flag will delete files that exist in the destination but not in the source. + flag deletes files that exist in the destination but not in the source. Make sure not to swap the source and destination, or you will delete all data in the Registry. 1. Verify all Container Registry files have been uploaded to object storage @@ -565,10 +550,6 @@ configurable in future releases. ## Change the registry's internal port -NOTE: **Note:** -This is not to be confused with the port that GitLab itself uses to expose -the Registry to the world. - The Registry server listens on localhost at port `5000` by default, which is the address for which the Registry server should accept connections. In the examples below we set the Registry's port to `5001`. @@ -589,7 +570,7 @@ In the examples below we set the Registry's port to `5001`. [`http:addr`](https://docs.docker.com/registry/configuration/#http) value: ```yaml - http + http: addr: localhost:5001 ``` @@ -603,9 +584,8 @@ on how to achieve that. ## Use an external container registry with GitLab as an auth endpoint -NOTE: **Note:** -In using an external container registry, some features associated with the -container registry may be unavailable or have [inherent risks](./../../user/packages/container_registry/index.md#use-with-external-container-registries) +If you use an external container registry, some features associated with the +container registry may be unavailable or have [inherent risks](./../../user/packages/container_registry/index.md#use-with-external-container-registries). **Omnibus GitLab** @@ -619,10 +599,9 @@ You can use GitLab as an auth endpoint with an external container registry. gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer" ``` - NOTE: **Note:** `gitlab_rails['registry_enabled'] = true` is needed to enable GitLab's Container Registry features and authentication endpoint. GitLab's bundled - Container Registry service will not be started even with this enabled. + Container Registry service does not start, even with this enabled. 1. A certificate-key pair is required for GitLab and the external container registry to communicate securely. You will need to create a certificate-key @@ -641,11 +620,10 @@ You can use GitLab as an auth endpoint with an external container registry. gitlab_rails['registry_key_path'] = "/custom/path/to/registry-key.key" ``` - NOTE: **Note:** - The file specified at `registry_key_path` gets populated with the - content specified by `internal_key`, each time reconfigure is executed. If - no file is specified, Omnibus GitLab will default it to - `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and will populate + Each time reconfigure is executed, the file specified at `registry_key_path` + gets populated with the content specified by `internal_key`. If + no file is specified, Omnibus GitLab defaults it to + `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and populates it. 1. To change the container registry URL displayed in the GitLab Container @@ -686,8 +664,7 @@ response to events happening within the registry. Read more about the Container Registry notifications configuration options in the [Docker Registry notifications documentation](https://docs.docker.com/registry/notifications/). -NOTE: **Note:** -Multiple endpoints can be configured for the Container Registry. +You can configure multiple endpoints for the Container Registry. **Omnibus GitLab installations** @@ -761,22 +738,10 @@ project.container_repositories.find_each do |repo| end ``` -NOTE: **Note:** You can also [run cleanup on a schedule](../../user/packages/container_registry/index.md#cleanup-policy). ## Container Registry garbage collection -NOTE: **Note:** -The garbage collection tools are only available when you've installed GitLab -via an Omnibus package or the [cloud native chart](https://docs.gitlab.com/charts/charts/registry/#garbage-collection). - -DANGER: **Danger:** -By running the built-in garbage collection command, it will cause downtime to -the Container Registry. If you run this command on an instance in an environment -where one of your other instances is still writing to the Registry storage, -referenced manifests will be removed. To avoid that, make sure Registry is set to -[read-only mode](#performing-garbage-collection-without-downtime) before proceeding. - Container Registry can use considerable amounts of disk space. To clear up some unused layers, the registry includes a garbage collect command. @@ -791,6 +756,15 @@ it only unlinks tags from manifests and image blobs. To recycle the Container Registry data in the whole GitLab instance, you can use the built-in command provided by `gitlab-ctl`. +Prerequisites: + +- You must have installed GitLab by using an Omnibus package or the + [cloud native chart](https://docs.gitlab.com/charts/charts/registry/#garbage-collection). +- You must set the Registry to [read-only mode](#performing-garbage-collection-without-downtime). + Running garbage collection causes downtime for the Container Registry. If you run this command + on an instance in an environment where another instances is still writing to the Registry storage, + referenced manifests will be removed. + ### Understanding the content-addressable layers Consider the following example, where you first build the image: @@ -818,15 +792,14 @@ no longer directly accessible via the `:latest` tag. > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/987) in Omnibus GitLab 8.12. -There are a couple of considerations you need to note before running the -built-in command: +Before you run the built-in command, note the following: -- The built-in command will stop the registry before it starts the garbage collection. +- The built-in command stops the registry before it starts the garbage collection. - The garbage collect command takes some time to complete, depending on the amount of data that exists. -- If you changed the location of registry configuration file, you will need to +- If you changed the location of registry configuration file, you must specify its path. -- After the garbage collection is done, the registry should start up automatically. +- After the garbage collection is done, the registry should start automatically. If you did not change the default location of the configuration file, run: @@ -882,7 +855,6 @@ During this time, you will be able to pull from the Container Registry, but you will not be able to push. -NOTE: **Note:** By default, the [registry storage path](#configure-storage-for-the-container-registry) is `/var/opt/gitlab/gitlab-rails/shared/registry`. @@ -1065,7 +1037,7 @@ Start with a value between `25000000` (25MB) and `50000000` (50MB). s3: accesskey: 'AKIAKIAKI' secretkey: 'secret123' - bucket: 'gitlab-registry-bucket-AKIAKIAKI' + bucket: 'gitlab-registry-bucket-AKIAKIAKI' chunksize: 25000000 ``` @@ -1166,12 +1138,7 @@ curl localhost:5001/debug/vars ### Advanced Troubleshooting -NOTE: **Note:** -The following section is only recommended for experts. - -Sometimes it's not obvious what is wrong, and you may need to dive deeper into -the communication between the Docker client and the Registry to find out -what's wrong. We will use a concrete example in the past to illustrate how to +We will use a concrete example in the past to illustrate how to diagnose a problem with the S3 setup. #### Unexpected 403 error during push diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 2f9cfecc9d4..fba3d51f741 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab can be utilized as a dependency proxy for a variety of common package managers. This is the administration documentation. If you want to learn how to use the -dependency proxies, see the [user guide](../../user/group/dependency_proxy/index.md). +dependency proxies, see the [user guide](../../user/packages/dependency_proxy/index.md). ## Enabling the Dependency Proxy feature @@ -135,28 +135,28 @@ This section describes the earlier configuration format. ## ## The location where build dependency_proxy are stored (default: shared/dependency_proxy). ## - #storage_path: shared/dependency_proxy + # storage_path: shared/dependency_proxy object_store: enabled: false - remote_directory: dependency_proxy # The bucket name. - #direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). - #background_upload: true # Temporary option to limit automatic upload (Default: true). - #proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + remote_directory: dependency_proxy # The bucket name. + # direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + # background_upload: true # Temporary option to limit automatic upload (Default: true). + # proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. connection: + ## + ## If the provider is AWS S3, use the following + ## + provider: AWS + region: us-east-1 + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY ## - ## If the provider is AWS S3, uncomment the following + ## If the provider is other than AWS (an S3-compatible one), comment out the previous 4 lines and use the following instead: ## - #provider: AWS - #region: us-east-1 - #aws_access_key_id: AWS_ACCESS_KEY_ID - #aws_secret_access_key: AWS_SECRET_ACCESS_KEY - ## - ## If the provider is other than AWS (an S3-compatible one), uncomment the following - ## - #host: 's3.amazonaws.com' # default: s3.amazonaws.com. - #aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. - #endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. - #path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + # host: 's3.amazonaws.com' # default: s3.amazonaws.com. + # aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + # endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + # path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source "How to restart GitLab") for the changes to take effect. diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 1061f3c33db..6b9e9ae3d5f 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -142,33 +142,33 @@ We recommend using the [consolidated object storage settings](../object_storage. 1. Edit the `packages` section in `config/gitlab.yml` (uncomment where necessary): ```yaml - packages: - enabled: true + packages: + enabled: true + ## + ## The location where build packages are stored (default: shared/packages). + ## + # storage_path: shared/packages + object_store: + enabled: false + remote_directory: packages # The bucket name. + # direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + # background_upload: true # Temporary option to limit automatic upload (Default: true). + # proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + connection: ## - ## The location where build packages are stored (default: shared/packages). + ## If the provider is AWS S3, use the following: ## - #storage_path: shared/packages - object_store: - enabled: false - remote_directory: packages # The bucket name. - #direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). - #background_upload: true # Temporary option to limit automatic upload (Default: true). - #proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. - connection: - ## - ## If the provider is AWS S3, uncomment the following - ## - #provider: AWS - #region: us-east-1 - #aws_access_key_id: AWS_ACCESS_KEY_ID - #aws_secret_access_key: AWS_SECRET_ACCESS_KEY - ## - ## If the provider is other than AWS (an S3-compatible one), uncomment the following - ## - #host: 's3.amazonaws.com' # default: s3.amazonaws.com. - #aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. - #endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. - #path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + provider: AWS + region: us-east-1 + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + ## + ## If the provider is other than AWS (an S3-compatible one), comment out the previous 4 lines and use the following instead: + ## + # host: 's3.amazonaws.com' # default: s3.amazonaws.com. + # aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + # endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + # path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. ``` 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 3c0030be629..53b45c0ac83 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -370,8 +370,8 @@ Pages access control is disabled by default. To enable it: 1. Users can now configure it in their [projects' settings](../../user/project/pages/pages_access_control.md). NOTE: **Important:** -For multi-node setups, in order for this setting to be effective, it has to be applied -to all the App nodes as well as the Sidekiq nodes. +For this setting to be effective with multi-node setups, it has to be applied to +all the App nodes and Sidekiq nodes. #### Disabling public access to all Pages websites @@ -397,8 +397,7 @@ redeployed. This issue will be resolved by ### Running behind a proxy Like the rest of GitLab, Pages can be used in those environments where external -internet connectivity is gated by a proxy. In order to use a proxy for GitLab -pages: +internet connectivity is gated by a proxy. To use a proxy for GitLab Pages: 1. Configure in `/etc/gitlab/gitlab.rb`: @@ -508,7 +507,8 @@ To override the global maximum pages size for a specific group: ## Running GitLab Pages on a separate server -You can run the GitLab Pages daemon on a separate server in order to decrease the load on your main application server. +You can run the GitLab Pages daemon on a separate server to decrease the load on +your main application server. To configure GitLab Pages on a separate server: diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index 632b68fb014..4a164c66578 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Configure GitLab using an external PostgreSQL service If you're hosting GitLab on a cloud provider, you can optionally use a diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index 2720d8e696b..c7ec46db654 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -1,11 +1,14 @@ --- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- # Configuring PostgreSQL for scaling In this section, you'll be guided through configuring a PostgreSQL database to -be used with GitLab in one of our [Scalable and Highly Available Setups](../reference_architectures/index.md). +be used with GitLab in one of our [reference architectures](../reference_architectures/index.md). There are essentially three setups to choose from. ## PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 9db3e017359..b946c0949c4 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -1,4 +1,7 @@ --- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index bc2af167e6c..6b84b7ac725 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -1,10 +1,16 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** -This document will focus only on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. -If you are a Community Edition or Starter user, consider using a cloud hosted solution. -This document will not cover installations from source. +This document focuses on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. +If you're a Community Edition or Starter user, consider using a cloud hosted solution. +This document doesn't cover installations from source. -If a setup with replication and failover is not what you were looking for, see +If a setup with replication and failover isn't what you were looking for, see the [database configuration document](https://docs.gitlab.com/omnibus/settings/database.html) for the Omnibus GitLab packages. @@ -87,9 +93,9 @@ information. #### Network information -PostgreSQL does not listen on any network interface by default. It needs to know -which IP address to listen on in order to be accessible to other services. -Similarly, PostgreSQL access is controlled based on the network source. +PostgreSQL doesn't listen on any network interface by default. It needs to know +which IP address to listen on to be accessible to other services. Similarly, +PostgreSQL access is controlled based on the network source. This is why you will need: @@ -1348,3 +1354,93 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with 1. Repeat the last two steps for all replica nodes. `gitlab.rb` should look the same on all nodes. 1. Optional: You can remove `gitlab_repmgr` database and role on the primary. + +### Upgrading PostgreSQL major version in a Patroni cluster + +As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus GitLab. GitLab still +uses PostgreSQL 11 by default. Therefore `gitlab-ctl pg-upgrade` does not automatically upgrade +to PostgreSQL 12. If you want to upgrade to PostgreSQL 12, you must ask for it explicitly. + +CAUTION: **Warning:** +The procedure for upgrading PostgreSQL in a Patroni cluster is different than when upgrading using repmgr. +The following outlines the key differences and important considerations that need to be accounted for when +upgrading PostgreSQL. + +Here are a few key facts that you must consider before upgrading PostgreSQL: + +- The main point is that you will have to **shut down the Patroni cluster**. This means that your + GitLab deployment will be down for the duration of database upgrade or, at least, as long as your leader + node is upgraded. This can be **a significant downtime depending on the size of your database**. + +- Upgrading PostgreSQL creates a new data directory with a new control data. From Patroni's perspective + this is a new cluster that needs to be bootstrapped again. Therefore, as part of the upgrade procedure, + the cluster state, which is stored in Consul, will be wiped out. Once the upgrade is completed, Patroni + will be instructed to bootstrap a new cluster. **Note that this will change your _cluster ID_**. + +- The procedures for upgrading leader and replicas are not the same. That is why it is important to use the + right procedure on each node. + +- Upgrading a replica node **deletes the data directory and resynchronizes it** from the leader using the + configured replication method (currently `pg_basebackup` is the only available option). It might take some + time for replica to catch up with the leader, depending on the size of your database. + +- An overview of the upgrade procedure is outlined in [Patoni's documentation](https://patroni.readthedocs.io/en/latest/existing_data.html#major-upgrade-of-postgresql-version). + You can still use `gitlab-ctl pg-upgrade` which implements this procedure with a few adjustments. + +Considering these, you should carefully plan your PostgreSQL upgrade: + +1. Find out which node is the leader and which node is a replica: + + ```shell + gitlab-ctl patroni members + ``` + + NOTE: **Note:** + `gitlab-ctl pg-upgrade` tries to detect the role of the node. If for any reason the auto-detection + does not work or you believe it did not detect the role correctly, you can use the `--leader` or `--replica` + arguments to manually override it. + +1. Stop Patroni **only on replicas**. + + ```shell + sudo gitlab-ctl stop patroni + ``` + +1. Enable the maintenance mode on the **application node**: + + ```shell + sudo gitlab-ctl deploy-page up + ``` + +1. Upgrade PostgreSQL on **the leader node** and make sure that the upgrade is completed successfully: + + ```shell + sudo gitlab-ctl pg-upgrade -V 12 + ``` + +1. Check the status of the leader and cluster. You can only proceed if you have a healthy leader: + + ```shell + gitlab-ctl patroni check-leader + + # OR + + gitlab-ctl patroni members + ``` + +1. You can now disable the maintenance mode on the **application node**: + + ```shell + sudo gitlab-ctl deploy-page down + ``` + +1. Upgrade PostgreSQL **on replicas** (you can do this in parallel on all of them): + + ```shell + sudo gitlab-ctl pg-upgrade -V 12 + ``` + +CAUTION: **Warning:** +Reverting PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same considerations as +`gitlab-ctl pg-upgrade`. It can be complicated and may involve deletion of the data directory. +If you need to do that, please contact GitLab support. diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md index 2747749066e..2ac74e8a4a0 100644 --- a/doc/administration/postgresql/standalone.md +++ b/doc/administration/postgresql/standalone.md @@ -1,15 +1,21 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Standalone PostgreSQL using Omnibus GitLab **(CORE ONLY)** If you wish to have your database service hosted separately from your GitLab -application server(s), you can do this using the PostgreSQL binaries packaged +application servers, you can do this using the PostgreSQL binaries packaged together with Omnibus GitLab. This is recommended as part of our [reference architecture for up to 2,000 users](../reference_architectures/2k_users.md). ## Setting it up -1. SSH into the PostgreSQL server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. +1. SSH in to the PostgreSQL server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using *steps 1 and 2* from the GitLab downloads page. - Do not complete any other steps on the download page. 1. Generate a password hash for PostgreSQL. This assumes you will use the default username of `gitlab` (recommended). The command will request a password diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 15014fffd01..d13e6328b2f 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -154,3 +154,34 @@ If the issue persists, try triggering `gc` via the p = Project.find_by_path("project-name") Projects::HousekeepingService.new(p, :gc).execute ``` + +### Delete references to missing remote uploads + +`gitlab-rake gitlab:uploads:check VERBOSE=1` detects remote objects that do not exist because they were +deleted externally but their references still exist in the GitLab database. + +Example output with error message: + +```shell +$ sudo gitlab-rake gitlab:uploads:check VERBOSE=1 +Checking integrity of Uploads +- 100..434: Failures: 2 +- Upload: 100: Remote object does not exist +- Upload: 101: Remote object does not exist +Done! +``` + +To delete these references to remote uploads that were deleted externally, open the [GitLab Rails Console](../troubleshooting/navigating_gitlab_via_rails_console.md#starting-a-rails-console-session) +and run: +[Rails Console](../troubleshooting/navigating_gitlab_via_rails_console.md#starting-a-rails-console-session): + +```ruby +uploads_deleted=0 +Upload.find_each do |upload| + next if upload.retrieve_uploader.file.exists? + uploads_deleted=uploads_deleted + 1 + p upload ### allow verification before destroy + # p upload.destroy! ### uncomment to actually destroy +end +p "#{uploads_deleted} remote objects were destroyed." +``` diff --git a/doc/administration/raketasks/doctor.md b/doc/administration/raketasks/doctor.md index 62d0af70706..c97aa5a4de1 100644 --- a/doc/administration/raketasks/doctor.md +++ b/doc/administration/raketasks/doctor.md @@ -47,9 +47,8 @@ I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! ### Verbose mode -In order to get more detailed information about which -rows and columns cannot be decrypted, you can pass a VERBOSE -environment variable: +To get more detailed information about which rows and columns can't be +decrypted, you can pass a `VERBOSE` environment variable: **Omnibus Installation** diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index a46a2b34687..7f673a2c850 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -2,9 +2,8 @@ > [Introduced]( https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10308) in GitLab 9.1. -In order to retrieve and import GitHub repositories, you will need a -[GitHub personal access token](https://github.com/settings/tokens). -A username should be passed as the second argument to the Rake task +To retrieve and import GitHub repositories, you need a [GitHub personal access token](https://github.com/settings/tokens). +A username should be passed as the second argument to the Rake task, which will become the owner of the project. You can resume an import with the same command. diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md new file mode 100644 index 00000000000..681102a8c39 --- /dev/null +++ b/doc/administration/read_only_gitlab.md @@ -0,0 +1,125 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Place GitLab into a read-only state **(CORE ONLY)** + +CAUTION: **Warning:** +This document should be used as a temporary solution. +There's work in progress to make this +[possible with Geo](https://gitlab.com/groups/gitlab-org/-/epics/2149). + +In some cases, you might want to place GitLab under a read-only state. +The configuration for doing so depends on your desired outcome. + +## Make the repositories read-only + +The first thing you'll want to accomplish is to ensure that no changes can be +made to your repositories. There's two ways you can accomplish that: + +- Either stop Unicorn/Puma to make the internal API unreachable: + + ```shell + sudo gitlab-ctl stop puma # or unicorn + ``` + +- Or, open up a Rails console: + + ```shell + sudo gitlab-rails console + ``` + + And set the repositories for all projects read-only: + + ```ruby + Project.all.find_each { |project| project.update!(repository_read_only: true) } + ``` + + When you're ready to revert this, you can do so with the following command: + + ```ruby + Project.all.find_each { |project| project.update!(repository_read_only: false) } + ``` + +## Shut down the GitLab UI + +If you don't mind shutting down the GitLab UI, then the easiest approach is to +stop `sidekiq` and `puma`/`unicorn`, and you'll effectively ensure that no +changes can be made to GitLab: + +```shell +sudo gitlab-ctl stop sidekiq +sudo gitlab-ctl stop puma # or unicorn +``` + +When you're ready to revert this: + +```shell +sudo gitlab-ctl start sidekiq +sudo gitlab-ctl start puma # or unicorn +``` + +## Make the database read-only + +If you want to allow users to use the GitLab UI, then you'll need to ensure that +the database is read-only: + +1. Take a [GitLab backup](../raketasks/backup_restore.md#back-up-gitlab) + in case things don't go as expected. +1. Enter PostgreSQL on the console as an admin user: + + ```shell + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/postgresql gitlabhq_production + ``` + +1. Create the `gitlab_read_only` user. Note that the password is set to `mypassword`, + change that to your liking: + + ```sql + -- NOTE: Use the password defined earlier + CREATE USER gitlab_read_only WITH password 'mypassword'; + GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_read_only; + GRANT USAGE ON SCHEMA public TO gitlab_read_only; + GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_read_only; + GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_read_only; + + -- Tables created by "gitlab" should be made read-only for "gitlab_read_only" + -- automatically. + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_read_only; + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_read_only; + ``` + +1. Get the hashed password of the `gitlab_read_only` user and copy the result: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_read_only + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the password from the previous step: + + ```ruby + postgresql['sql_user_password'] = 'a2e20f823772650f039284619ab6f239' + postgresql['sql_user'] = "gitlab_read_only" + ``` + +1. Reconfigure GitLab and restart PostgreSQL: + + ```shell + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart postgresql + ``` + +When you're ready to revert the read-only state, you'll need to remove the added +lines in `/etc/gitlab/gitlab.rb`, and reconfigure GitLab and restart PostgreSQL: + +```shell +sudo gitlab-ctl reconfigure +sudo gitlab-ctl restart postgresql +``` + +Once you verify all works as expected, you can remove the `gitlab_read_only` +user from the database. diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index ca041adb1d8..72a52ba0a1f 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -124,9 +124,9 @@ each other over the network. ### Sentinel setup overview Sentinels watch both other Sentinels and Redis nodes. Whenever a Sentinel -detects that a Redis node is not responding, it will announce that to the -other Sentinels. They have to reach the **quorum**, that is the minimum amount -of Sentinels that agrees a node is down, in order to be able to start a failover. +detects that a Redis node isn't responding, it announces the node's status to +the other Sentinels. The Sentinels have to reach a _quorum_ (the minimum amount +of Sentinels agreeing a node is down) to be able to start a failover. Whenever the **quorum** is met, the **majority** of all known Sentinel nodes need to be available and reachable, so that they can elect the Sentinel **leader** diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index ce452d30fc2..680d36b6fde 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -228,13 +228,13 @@ which ideally should not have Redis or Sentinels in the same machine: sentinels: - host: 10.0.0.1 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port - host: 10.0.0.2 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port - host: 10.0.0.3 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -353,13 +353,13 @@ or a failover promotes a different **Primary** node. sentinels: - host: 10.0.0.1 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port - host: 10.0.0.2 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port - host: 10.0.0.3 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index 402b60e5b7b..c9976ac791d 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -146,13 +146,13 @@ production: sentinels: - host: 10.0.0.1 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port - host: 10.0.0.2 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port - host: 10.0.0.3 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port ``` When in doubt, read the [Redis Sentinel documentation](https://redis.io/topics/sentinel). diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 5f8ab6683a9..2f32cf9fb04 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -17,21 +17,21 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|-----------------|-------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Consul | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis - Cache | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | -| Gitaly | 2 (minimum) | 16 vCPU, 60GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| Sidekiq | 4 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 3 | 32 vCPU, 28.8GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Object Storage | n/a | n/a | n/a | n/a | n/a | -| NFS Server | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Gitaly | 2 (minimum) | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -40,41 +40,43 @@ or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -For data objects (such as LFS, Uploads, or Artifacts), an -[object storage service](#configure-the-object-storage) is recommended instead -of NFS where possible, due to better performance and availability. Since this -doesn't require a node to be set up, *Object Storage* is noted as not -applicable (n/a) in the previous table. +Due to better performance and availability, for data objects (such as LFS, +uploads, or artifacts), using an [object storage service](#configure-the-object-storage) +is recommended instead of using NFS. Using an object storage service also +doesn't require you to provision and maintain a node. ## Setup components To set up GitLab and its components to accommodate up to 10,000 users: 1. [Configure the external load balancing node](#configure-the-external-load-balancer) - that will handle the load balancing of the three GitLab application services nodes. + to handle the load balancing of the GitLab application services nodes. 1. [Configure Consul](#configure-consul). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Redis](#configure-redis). 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git - over HTTP/SSH). -1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. -1. [Configure the Object Storage](#configure-the-object-storage) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + requests (which include UI, API, and Git over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab + environment. +1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure NFS (Optional)](#configure-nfs-optional) - to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although - not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using - that feature. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. You can skip this step if you're not using GitLab Pages (which + requires NFS). -We start with all servers on the same 10.6.0.0/24 private network range, they -can connect to each other freely on those addresses. +The servers start on the same 10.6.0.0/24 private network range, and can +connect to each other freely on these addresses. -Here is a list and description of each machine and the assigned IP: +The following list includes descriptions of each server and its assigned IP: - `10.6.0.10`: External Load Balancer - `10.6.0.11`: Consul 1 @@ -1752,6 +1754,7 @@ On each node perform the following: roles ['application_role'] gitaly['enable'] = false nginx['enable'] = true + sidekiq['enable'] = false ## PostgreSQL connection details # Disable PostgreSQL on the application node @@ -1795,7 +1798,6 @@ On each node perform the following: # Set the network addresses that the exporters used for monitoring will listen on node_exporter['listen_address'] = '0.0.0.0:9100' gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' - sidekiq['listen_address'] = "0.0.0.0" puma['listen'] = '0.0.0.0' # Add the monitoring node's IP address to the monitoring whitelist and allow it to @@ -1836,7 +1838,7 @@ On each node perform the following: 1. Specify the necessary NFS mounts in `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose - to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + to configure your NFS server. See the [NFS documentation](../nfs.md) for examples and the various options. 1. Create the shared directories. These may be different depending on your NFS @@ -1889,18 +1891,22 @@ for more information. ### GitLab Rails post-configuration -Initialize the GitLab database, by running the following in one of the Rails nodes: +1. Designate one application node for running database migrations during + installation and updates. Initialize the GitLab database and ensure all + migrations ran: -```shell -sudo gitlab-rake gitlab:db:configure -``` + ```shell + sudo gitlab-rake gitlab:db:configure + ``` -NOTE: **Note:** -If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to -PostgreSQL it may be that your PgBouncer node's IP address is missing from -PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See -[PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) -in the Troubleshooting section before proceeding. + NOTE: **Note:** + If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to + PostgreSQL it may be that your PgBouncer node's IP address is missing from + PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) + in the Troubleshooting section before proceeding. + +1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2033,13 +2039,32 @@ work. </a> </div> +## Configure Advanced Search **(STARTER ONLY)** + +NOTE: **Note:** +Elasticsearch cluster design and requirements are dependent on your specific data. +For recommended best practices on how to set up your Elasticsearch cluster +alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +You can leverage Elasticsearch and enable Advanced Search for faster, more +advanced code search across your entire GitLab instance. + +[Learn how to set it up.](../../integration/elasticsearch.md) + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). -See how to [configure NFS](../high_availability/nfs.md). +See how to [configure NFS](../nfs.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index d376c1b7575..f54a25b7a9b 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -20,8 +20,8 @@ many organizations . | Users | Configuration | GCP | AWS | Azure | |--------------|-------------------------|----------------|-----------------|----------------| -| Up to 500 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Up to 1,000 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Up to 500 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Up to 1,000 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -30,9 +30,9 @@ or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -In addition to the stated configurations, we recommend having at least 2GB of +In addition to the stated configurations, we recommend having at least 2 GB of swap on your server, even if you currently have enough available memory. Having -swap will help reduce the chance of errors occurring if your available memory +swap helps to reduce the chance of errors occurring if your available memory changes. We also recommend configuring the kernel's swappiness setting to a lower value (such as `10`) to make the most of your memory, while still having the swap available when needed. @@ -45,5 +45,18 @@ For this default reference architecture, to install GitLab use the standard NOTE: **Note:** You can also optionally configure GitLab to use an [external PostgreSQL service](../postgresql/external.md) or an -[external object storage service](../high_availability/object_storage.md) for +[external object storage service](../object_storage.md) for added performance and reliability at a reduced complexity cost. + +## Configure Advanced Search **(STARTER ONLY)** + +NOTE: **Note:** +Elasticsearch cluster design and requirements are dependent on your specific data. +For recommended best practices on how to set up your Elasticsearch cluster +alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +You can leverage Elasticsearch and enable Advanced Search for faster, more +advanced code search across your entire GitLab instance. + +[Learn how to set it up.](../../integration/elasticsearch.md) diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 2ef555bff29..55de38645d7 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -17,21 +17,21 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|-------------|----------| -| External load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Consul | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 8 vCPU, 30GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis - Cache | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | -| Gitaly | 2 (minimum) | 32 vCPU, 120GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | -| Sidekiq | 4 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 5 | 32 vCPU, 28.8GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Object Storage | n/a | n/a | n/a | n/a | n/a | -| NFS Server | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.large | F2s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Gitaly | 2 (minimum) | 32 vCPU, 120 GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -40,41 +40,43 @@ or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -For data objects (such as LFS, Uploads, or Artifacts), an -[object storage service](#configure-the-object-storage) is recommended instead -of NFS where possible, due to better performance and availability. Since this -doesn't require a node to be set up, *Object Storage* is noted as not -applicable (n/a) in the previous table. +Due to better performance and availability, for data objects (such as LFS, +uploads, or artifacts), using an [object storage service](#configure-the-object-storage) +is recommended instead of using NFS. Using an object storage service also +doesn't require you to provision and maintain a node. ## Setup components To set up GitLab and its components to accommodate up to 25,000 users: 1. [Configure the external load balancing node](#configure-the-external-load-balancer) - that will handle the load balancing of the three GitLab application services nodes. + to handle the load balancing of the GitLab application services nodes. 1. [Configure Consul](#configure-consul). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Redis](#configure-redis). 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git - over HTTP/SSH). -1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. -1. [Configure the Object Storage](#configure-the-object-storage) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + requests (which include UI, API, and Git over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab + environment. +1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure NFS (Optional)](#configure-nfs-optional) - to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although - not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using - that feature. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. You can skip this step if you're not using GitLab Pages (which + requires NFS). -We start with all servers on the same 10.6.0.0/24 private network range, they -can connect to each other freely on those addresses. +The servers start on the same 10.6.0.0/24 private network range, and can +connect to each other freely on these addresses. -Here is a list and description of each machine and the assigned IP: +The following list includes descriptions of each server and its assigned IP: - `10.6.0.10`: External Load Balancer - `10.6.0.11`: Consul 1 @@ -1752,6 +1754,7 @@ On each node perform the following: roles ['application_role'] gitaly['enable'] = false nginx['enable'] = true + sidekiq['enable'] = false ## PostgreSQL connection details # Disable PostgreSQL on the application node @@ -1795,7 +1798,6 @@ On each node perform the following: # Set the network addresses that the exporters used for monitoring will listen on node_exporter['listen_address'] = '0.0.0.0:9100' gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' - sidekiq['listen_address'] = "0.0.0.0" puma['listen'] = '0.0.0.0' # Add the monitoring node's IP address to the monitoring whitelist and allow it to @@ -1836,7 +1838,7 @@ On each node perform the following: 1. Specify the necessary NFS mounts in `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose - to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + to configure your NFS server. See the [NFS documentation](../nfs.md) for examples and the various options. 1. Create the shared directories. These may be different depending on your NFS @@ -1889,18 +1891,22 @@ for more information. ### GitLab Rails post-configuration -Initialize the GitLab database, by running the following in one of the Rails nodes: +1. Designate one application node for running database migrations during + installation and updates. Initialize the GitLab database and ensure all + migrations ran: -```shell -sudo gitlab-rake gitlab:db:configure -``` + ```shell + sudo gitlab-rake gitlab:db:configure + ``` -NOTE: **Note:** -If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to -PostgreSQL it may be that your PgBouncer node's IP address is missing from -PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See -[PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) -in the Troubleshooting section before proceeding. + NOTE: **Note:** + If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to + PostgreSQL it may be that your PgBouncer node's IP address is missing from + PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) + in the Troubleshooting section before proceeding. + +1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2033,13 +2039,32 @@ work. </a> </div> +## Configure Advanced Search **(STARTER ONLY)** + +NOTE: **Note:** +Elasticsearch cluster design and requirements are dependent on your specific data. +For recommended best practices on how to set up your Elasticsearch cluster +alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +You can leverage Elasticsearch and enable Advanced Search for faster, more +advanced code search across your entire GitLab instance. + +[Learn how to set it up.](../../integration/elasticsearch.md) + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). -See how to [configure NFS](../high_availability/nfs.md). +See how to [configure NFS](../nfs.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 34b90964fbf..1c174fc6e03 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -17,14 +17,14 @@ For a full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|--------|-------------------------|----------------|--------------|---------| -| Load balancer | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 1 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| Redis | 1 | 1 vCPU, 3.75GB memory | n1-standard-1 | m5.large | D2s v3 | -| Gitaly | 1 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 2 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Load balancer | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 1 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| Redis | 1 | 1 vCPU, 3.75 GB memory | n1-standard-1 | m5.large | D2s v3 | +| Gitaly | 1 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -43,7 +43,7 @@ doesn't require you to provision and maintain a node. To set up GitLab and its components to accommodate up to 2,000 users: 1. [Configure the external load balancing node](#configure-the-external-load-balancer) - to handle the load balancing of the two GitLab application services nodes. + to handle the load balancing of the GitLab application services nodes. 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure Redis](#configure-redis). 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git @@ -55,6 +55,8 @@ To set up GitLab and its components to accommodate up to 2,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. 1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) to have shared disk storage service as an alternative to Gitaly or object storage. You can skip this step if you're not using GitLab Pages (which @@ -664,6 +666,11 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` +1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two + application nodes and install it on the other application node and the + [Gitaly node](#configure-gitaly) and + [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + NOTE: **Note:** When you specify `https` in the `external_url`, as in the example above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If @@ -671,6 +678,25 @@ certificates are not present, NGINX will fail to start. See the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for more information. +### GitLab Rails post-configuration + +1. Designate one application node for running database migrations during + installation and updates. Initialize the GitLab database and ensure all + migrations ran: + + ```shell + sudo gitlab-rake gitlab:db:configure + ``` + + NOTE: **Note:** + If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to + PostgreSQL it may be that your PgBouncer node's IP address is missing from + PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) + in the Troubleshooting section before proceeding. + +1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -851,6 +877,25 @@ functioning backups is encountered. </a> </div> +## Configure Advanced Search **(STARTER ONLY)** + +NOTE: **Note:** +Elasticsearch cluster design and requirements are dependent on your specific data. +For recommended best practices on how to set up your Elasticsearch cluster +alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +You can leverage Elasticsearch and enable Advanced Search for faster, more +advanced code search across your entire GitLab instance. + +[Learn how to set it up.](../../integration/elasticsearch.md) + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure NFS (optional) For improved performance, [object storage](#configure-the-object-storage), diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index be944586e43..f69e19ffef8 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -24,18 +24,18 @@ costly-to-operate environment by using the | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-----------------------|----------------|-------------|---------| -| External load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly | 2 (minimum) | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Sidekiq | 4 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| GitLab Rails | 3 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Object Storage | n/a | n/a | n/a | n/a | n/a | -| NFS Server (optional, not recommended) | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly | 2 (minimum) | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -44,41 +44,43 @@ or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -For data objects (such as LFS, Uploads, or Artifacts), an -[object storage service](#configure-the-object-storage) is recommended instead -of NFS where possible, due to better performance and availability. Since this -doesn't require a node to be set up, *Object Storage* is noted as not -applicable (n/a) in the previous table. +Due to better performance and availability, for data objects (such as LFS, +uploads, or artifacts), using an [object storage service](#configure-the-object-storage) +is recommended instead of using NFS. Using an object storage service also +doesn't require you to provision and maintain a node. ## Setup components To set up GitLab and its components to accommodate up to 3,000 users: 1. [Configure the external load balancing node](#configure-the-external-load-balancer) - that will handle the load balancing of the two GitLab application services nodes. + to handle the load balancing of the GitLab application services nodes. 1. [Configure Redis](#configure-redis). 1. [Configure Consul and Sentinel](#configure-consul-and-sentinel). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git - over HTTP/SSH). -1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. -1. [Configure the Object Storage](#configure-the-object-storage) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + requests (which include UI, API, and Git over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab + environment. +1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure NFS (Optional)](#configure-nfs-optional) - to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although - not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using - that feature. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. You can skip this step if you're not using GitLab Pages (which + requires NFS). -We start with all servers on the same 10.6.0.0/16 private network range, they -can connect to each other freely on those addresses. +The servers start on the same 10.6.0.0/24 private network range, and can +connect to each other freely on these addresses. -Here is a list and description of each machine and the assigned IP: +The following list includes descriptions of each server and its assigned IP: - `10.6.0.10`: External Load Balancer - `10.6.0.61`: Redis Primary @@ -1582,6 +1584,11 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` +1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two + application nodes and install it on the other application node and the + [Gitaly node](#configure-gitaly) and + [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Verify the GitLab services are running: ```shell @@ -1759,6 +1766,25 @@ work. </a> </div> +## Configure Advanced Search **(STARTER ONLY)** + +NOTE: **Note:** +Elasticsearch cluster design and requirements are dependent on your specific data. +For recommended best practices on how to set up your Elasticsearch cluster +alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +You can leverage Elasticsearch and enable Advanced Search for faster, more +advanced code search across your entire GitLab instance. + +[Learn how to set it up.](../../integration/elasticsearch.md) + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index e812eed0227..7ea571108fb 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -17,21 +17,21 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|--------------|----------| -| External load balancing node | 1 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Consul | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 16 vCPU, 60GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Redis - Cache | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | -| Gitaly | 2 (minimum) | 64 vCPU, 240GB memory | n1-standard-64 | m5.16xlarge | D64s v3 | -| Sidekiq | 4 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 12 | 32 vCPU, 28.8GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Object Storage | n/a | n/a | n/a | n/a | n/a | -| NFS Server | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Gitaly | 2 (minimum) | 64 vCPU, 240 GB memory | n1-standard-64 | m5.16xlarge | D64s v3 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -40,41 +40,43 @@ or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -For data objects (such as LFS, Uploads, or Artifacts), an -[object storage service](#configure-the-object-storage) is recommended instead -of NFS where possible, due to better performance and availability. Since this -doesn't require a node to be set up, *Object Storage* is noted as not -applicable (n/a) in the previous table. +Due to better performance and availability, for data objects (such as LFS, +uploads, or artifacts), using an [object storage service](#configure-the-object-storage) +is recommended instead of using NFS. Using an object storage service also +doesn't require you to provision and maintain a node. ## Setup components To set up GitLab and its components to accommodate up to 50,000 users: 1. [Configure the external load balancing node](#configure-the-external-load-balancer) - that will handle the load balancing of the three GitLab application services nodes. + to handle the load balancing of the GitLab application services nodes. 1. [Configure Consul](#configure-consul). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Redis](#configure-redis). 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git - over HTTP/SSH). -1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. -1. [Configure the Object Storage](#configure-the-object-storage) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + requests (which include UI, API, and Git over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab + environment. +1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure NFS (Optional)](#configure-nfs-optional) - to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although - not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using - that feature. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. You can skip this step if you're not using GitLab Pages (which + requires NFS). -We start with all servers on the same 10.6.0.0/24 private network range, they -can connect to each other freely on those addresses. +The servers start on the same 10.6.0.0/24 private network range, and can +connect to each other freely on these addresses. -Here is a list and description of each machine and the assigned IP: +The following list includes descriptions of each server and its assigned IP: - `10.6.0.10`: External Load Balancer - `10.6.0.11`: Consul 1 @@ -1752,6 +1754,7 @@ On each node perform the following: roles ['application_role'] gitaly['enable'] = false nginx['enable'] = true + sidekiq['enable'] = false ## PostgreSQL connection details # Disable PostgreSQL on the application node @@ -1795,7 +1798,6 @@ On each node perform the following: # Set the network addresses that the exporters used for monitoring will listen on node_exporter['listen_address'] = '0.0.0.0:9100' gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' - sidekiq['listen_address'] = "0.0.0.0" puma['listen'] = '0.0.0.0' # Add the monitoring node's IP address to the monitoring whitelist and allow it to @@ -1836,7 +1838,7 @@ On each node perform the following: 1. Specify the necessary NFS mounts in `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose - to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + to configure your NFS server. See the [NFS documentation](../nfs.md) for examples and the various options. 1. Create the shared directories. These may be different depending on your NFS @@ -1889,18 +1891,22 @@ for more information. ### GitLab Rails post-configuration -Initialize the GitLab database, by running the following in one of the Rails nodes: +1. Designate one application node for running database migrations during + installation and updates. Initialize the GitLab database and ensure all + migrations ran: -```shell -sudo gitlab-rake gitlab:db:configure -``` + ```shell + sudo gitlab-rake gitlab:db:configure + ``` -NOTE: **Note:** -If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to -PostgreSQL it may be that your PgBouncer node's IP address is missing from -PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See -[PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) -in the Troubleshooting section before proceeding. + NOTE: **Note:** + If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to + PostgreSQL it may be that your PgBouncer node's IP address is missing from + PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) + in the Troubleshooting section before proceeding. + +1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2033,13 +2039,32 @@ work. </a> </div> +## Configure Advanced Search **(STARTER ONLY)** + +NOTE: **Note:** +Elasticsearch cluster design and requirements are dependent on your specific data. +For recommended best practices on how to set up your Elasticsearch cluster +alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +You can leverage Elasticsearch and enable Advanced Search for faster, more +advanced code search across your entire GitLab instance. + +[Learn how to set it up.](../../integration/elasticsearch.md) + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). -See how to [configure NFS](../high_availability/nfs.md). +See how to [configure NFS](../nfs.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 6dfa588b092..b32560d7055 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -24,18 +24,18 @@ costly-to-operate environment by using the | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|----------------|-------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly | 2 (minimum) | 8 vCPU, 30GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| Sidekiq | 4 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| GitLab Rails | 3 | 16 vCPU, 14.4GB memory | n1-highcpu-16 | c5.4xlarge | F16s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Object Storage | n/a | n/a | n/a | n/a | n/a | -| NFS Server (optional, not recommended) | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly | 2 (minimum) | 8 vCPU, 30 GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | n1-highcpu-16 | c5.4xlarge | F16s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -44,41 +44,43 @@ or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -For data objects (such as LFS, Uploads, or Artifacts), an -[object storage service](#configure-the-object-storage) is recommended instead -of NFS where possible, due to better performance and availability. Since this -doesn't require a node to be set up, *Object Storage* is noted as not -applicable (n/a) in the previous table. +Due to better performance and availability, for data objects (such as LFS, +uploads, or artifacts), using an [object storage service](#configure-the-object-storage) +is recommended instead of using NFS. Using an object storage service also +doesn't require you to provision and maintain a node. ## Setup components To set up GitLab and its components to accommodate up to 5,000 users: 1. [Configure the external load balancing node](#configure-the-external-load-balancer) - that will handle the load balancing of the two GitLab application services nodes. + to handle the load balancing of the GitLab application services nodes. 1. [Configure Redis](#configure-redis). 1. [Configure Consul and Sentinel](#configure-consul-and-sentinel). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git - over HTTP/SSH). -1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. -1. [Configure the Object Storage](#configure-the-object-storage) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + requests (which include UI, API, and Git over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab + environment. +1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure NFS (Optional)](#configure-nfs-optional) - to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although - not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using - that feature. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. You can skip this step if you're not using GitLab Pages (which + requires NFS). -We start with all servers on the same 10.6.0.0/16 private network range, they -can connect to each other freely on those addresses. +The servers start on the same 10.6.0.0/24 private network range, and can +connect to each other freely on these addresses. -Here is a list and description of each machine and the assigned IP: +The following list includes descriptions of each server and its assigned IP: - `10.6.0.10`: External Load Balancer - `10.6.0.61`: Redis Primary @@ -1581,6 +1583,11 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` +1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two + application nodes and install it on the other application node and the + [Gitaly node](#configure-gitaly) and + [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Verify the GitLab services are running: ```shell @@ -1758,6 +1765,25 @@ work. </a> </div> +## Configure Advanced Search **(STARTER ONLY)** + +NOTE: **Note:** +Elasticsearch cluster design and requirements are dependent on your specific data. +For recommended best practices on how to set up your Elasticsearch cluster +alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +You can leverage Elasticsearch and enable Advanced Search for faster, more +advanced code search across your entire GitLab instance. + +[Learn how to set it up.](../../integration/elasticsearch.md) + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 3964b8daeb7..8816d0eecf4 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -105,7 +105,7 @@ is the least complex to setup. This provides a point-in-time recovery of a prede > - Supported tiers: [GitLab Starter, Premium, and Ultimate](https://about.gitlab.com/pricing/) This requires separating out GitLab into multiple application nodes with an added -[load balancer](../high_availability/load_balancer.md). The load balancer will distribute traffic +[load balancer](../load_balancer.md). The load balancer will distribute traffic across GitLab application nodes. Meanwhile, each application node connects to a shared file server and database systems on the back end. This way, if one of the application servers fails, the workflow is not interrupted. diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index f950134889d..6d34772ad24 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -71,7 +71,7 @@ The instructions make the assumption that you will be using the email address `i sudo postfix start ``` -1. Send the new `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: +1. Send the new `incoming` user an email to test SMTP, by entering the following into the SMTP prompt: ```plaintext ehlo localhost @@ -251,7 +251,7 @@ Courier, which we will install later to add IMAP authentication, requires mailbo If you get a `Connection refused` error instead, make sure your firewall is set up to allow inbound traffic on port 25. - 1. Send the `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: + 1. Send the `incoming` user an email to test SMTP, by entering the following into the SMTP prompt: ```plaintext ehlo gitlab.example.com diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index b1e7e349978..6ded7fdd7d0 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -75,8 +75,8 @@ extensions), which contain the following in a single encrypted file: - Intermediate certificates (if any) - Private key -In order to export the required files in PEM encoding from the PKCS#12 file, -the `openssl` command can be used: +To export the required files in PEM encoding from the PKCS#12 file, the +`openssl` command can be used: ```shell #-- Extract private key in PEM encoding (no password, unencrypted) diff --git a/doc/administration/snippets/index.md b/doc/administration/snippets/index.md index 95de3b8c183..5e61d20c683 100644 --- a/doc/administration/snippets/index.md +++ b/doc/administration/snippets/index.md @@ -20,8 +20,8 @@ abuse of the feature. The default value is **52428800 Bytes** (50 MB). The content size limit will be applied when a snippet is created or updated. -In order not to break any existing snippets, the limit doesn't have any -effect on them until a snippet is edited again and the content changes. +This limit doesn't affect existing snippets until they're updated and their +content changes. ### Snippets size limit configuration diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index e13261e3074..9b805226e2d 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -206,7 +206,7 @@ The best place to start is to determine if the issue is with creating an empty i If it is, check on the Elasticsearch side to determine if the `gitlab-production` (the name for the GitLab index) exists. If it exists, manually delete it on the Elasticsearch side and attempt to recreate it from the -[`recreate_index`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) +[`recreate_index`](../../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) Rake task. If you still encounter issues, try creating an index manually on the Elasticsearch @@ -225,8 +225,8 @@ during the indexing of projects. If errors do occur, they will either stem from If the indexing process does not present errors, you will want to check the status of the indexed projects. You can do this via the following Rake tasks: -- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) (shows the overall status) -- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) (shows specific projects that are not indexed) +- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows the overall status) +- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows specific projects that are not indexed) If: diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 9a23a115765..b7bcde52a42 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -525,8 +525,8 @@ conflicting_permanent_redirects.destroy_all ### Close a merge request properly (if merged but still marked as open) ```ruby -p = Project.find_by_full_path('') -m = project.merge_requests.find_by(iid: ) +p = Project.find_by_full_path('<full/path/to/project>') +m = p.merge_requests.find_by(iid: <iid>) u = User.find_by_username('') MergeRequests::PostMergeService.new(p, u).execute(m) ``` diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 01532032b49..cbcccd96d86 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -84,8 +84,7 @@ and they will assist you with any issues you are having. ## GitLab-specific Kubernetes information -- Minimal config that can be used to test a Kubernetes Helm chart can be found - [here](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/620). +- Minimal config that can be used to [test a Kubernetes Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/620). - Tailing logs of a separate pod. An example for a Webservice pod: diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index e6c081e1eea..dc3c08b61ac 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -205,6 +205,6 @@ Some of these errors come from the Excon Ruby gem, and could be generated in cir where GitLab is configured to initiate an HTTPS session to a remote server that is serving just HTTP. -One scenario is that you're using [object storage](../high_availability/object_storage.md) +One scenario is that you're using [object storage](../object_storage.md) which is not served under HTTPS. GitLab is misconfigured and attempts a TLS handshake, but the object storage will respond with plain HTTP. diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 71a41719003..d9679971fa5 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -68,9 +68,9 @@ For source installations the following settings are nested under `uploads:` and |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where Uploads will be stored| | -| `direct_upload` | Set to true to remove Puma from the Upload path. Workhorse handles the actual Artifact Upload to Object Storage while Puma does minimal processing to keep track of the upload. There is no need for local shared storage. The option may be removed if support for a single storage type for all files is introduced. Read more on [direct upload](../development/uploads.md#direct-upload). | `false` | -| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 (if `direct_upload` is set to `true` it will override `background_upload`) | `true` | -| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | +| `direct_upload` | Set to `true` to remove Puma from the Upload path. Workhorse handles the actual Artifact Upload to Object Storage while Puma does minimal processing to keep track of the upload. There is no need for local shared storage. The option may be removed if support for a single storage type for all files is introduced. Read more on [direct upload](../development/uploads.md#direct-upload). | `false` | +| `background_upload` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 (if `direct_upload` is set to `true` it will override `background_upload`) | `true` | +| `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | ### Connection settings diff --git a/doc/api/README.md b/doc/api/README.md index 53df4114a71..c3605db16eb 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -158,9 +158,21 @@ for example, without needing to explicitly pass an access token. With a few API endpoints you can use a [GitLab CI/CD job token](../user/project/new_ci_build_permissions_model.md#job-token) to authenticate with the API: +- Packages: + - [Composer Repository](../user/packages/composer_repository/index.md) + - [Conan Repository](../user/packages/conan_repository/index.md) + - [Container Registry](../user/packages/container_registry/index.md) (`$CI_REGISTRY_PASSWORD` is actually `$CI_JOB_TOKEN`, but this may change in the future) + - [Go Proxy](../user/packages/go_proxy/index.md) + - [Maven Repository](../user/packages/maven_repository/index.md#authenticating-with-a-ci-job-token) + - [NPM Repository](../user/packages/npm_registry/index.md#authenticating-with-a-ci-job-token) + - [Nuget Repository](../user/packages/nuget_repository/index.md) + - [PyPI Repository](../user/packages/pypi_repository/index.md#using-gitlab-ci-with-pypi-packages) - [Get job artifacts](job_artifacts.md#get-job-artifacts) -- [Pipeline triggers](pipeline_triggers.md) +- [Pipeline triggers](pipeline_triggers.md) (via `token=` parameter) - [Release creation](releases/index.md#create-a-release) +- [Terraform plan](../user/infrastructure/index.md) + +The token is valid as long as the job is running. ### Impersonation tokens @@ -297,21 +309,22 @@ The following table gives an overview of how the API functions generally behave. The following table shows the possible return codes for API requests. -| Return values | Description | -| ------------- | ----------- | -| `200 OK` | The `GET`, `PUT` or `DELETE` request was successful, the resource(s) itself is returned as JSON. | -| `204 No Content` | The server has successfully fulfilled the request and that there is no additional content to send in the response payload body. | -| `201 Created` | The `POST` request was successful and the resource is returned as JSON. | -| `304 Not Modified` | Indicates that the resource has not been modified since the last request. | -| `400 Bad Request` | A required attribute of the API request is missing, e.g., the title of an issue is not given. | -| `401 Unauthorized` | The user is not authenticated, a valid [user token](#authentication) is necessary. | -| `403 Forbidden` | The request is not allowed, e.g., the user is not allowed to delete a project. | -| `404 Not Found` | A resource could not be accessed, e.g., an ID for a resource could not be found. | +| Return values | Description | +| ------------------------ | ----------- | +| `200 OK` | The `GET`, `PUT` or `DELETE` request was successful, the resource(s) itself is returned as JSON. | +| `204 No Content` | The server has successfully fulfilled the request and that there is no additional content to send in the response payload body. | +| `201 Created` | The `POST` request was successful and the resource is returned as JSON. | +| `304 Not Modified` | Indicates that the resource has not been modified since the last request. | +| `400 Bad Request` | A required attribute of the API request is missing, e.g., the title of an issue is not given. | +| `401 Unauthorized` | The user is not authenticated, a valid [user token](#authentication) is necessary. | +| `403 Forbidden` | The request is not allowed, e.g., the user is not allowed to delete a project. | +| `404 Not Found` | A resource could not be accessed, e.g., an ID for a resource could not be found. | | `405 Method Not Allowed` | The request is not supported. | -| `409 Conflict` | A conflicting resource already exists, e.g., creating a project with a name that already exists. | -| `412` | Indicates the request was denied. May happen if the `If-Unmodified-Since` header is provided when trying to delete a resource, which was modified in between. | -| `422 Unprocessable` | The entity could not be processed. | -| `500 Server Error` | While handling the request something went wrong server-side. | +| `409 Conflict` | A conflicting resource already exists, e.g., creating a project with a name that already exists. | +| `412` | Indicates the request was denied. May happen if the `If-Unmodified-Since` header is provided when trying to delete a resource, which was modified in between. | +| `422 Unprocessable` | The entity could not be processed. | +| `429 Too Many Requests` | The user exceeded the [application rate limits](../administration/instance_limits.md#rate-limits). | +| `500 Server Error` | While handling the request, something went wrong server-side. | ## Pagination @@ -588,7 +601,7 @@ Such errors appear in two cases: - A required attribute of the API request is missing, e.g., the title of an issue is not given -- An attribute did not pass the validation, e.g., user bio is too long +- An attribute did not pass the validation, e.g., the user bio is too long When an attribute is missing, you will get something like: diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md index 898aa713331..3d448b2a65f 100644 --- a/doc/api/api_resources.md +++ b/doc/api/api_resources.md @@ -9,7 +9,7 @@ Available resources for the [GitLab API](README.md) can be grouped in the follow See also: - [V3 to V4](v3_to_v4.md). -- Adding [deploy keys for multiple projects](deploy_key_multiple_projects.md). +- Adding [deploy keys for multiple projects](deploy_keys.md#adding-deploy-keys-to-multiple-projects). - [API Resources for various templates](#templates-api-resources). ## Project resources diff --git a/doc/api/container_registry.md b/doc/api/container_registry.md index e4687994017..2fe72f53d87 100644 --- a/doc/api/container_registry.md +++ b/doc/api/container_registry.md @@ -24,7 +24,6 @@ GET /projects/:id/registry/repositories | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) accessible by the authenticated user. | | `tags` | boolean | no | If the parameter is included as true, each repository will include an array of `"tags"` in the response. | -| `name` | string | no | Returns a list of repositories with a name that matches the value. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29763) in GitLab 13.0). | | `tags_count` | boolean | no | If the parameter is included as true, each repository will include `"tags_count"` in the response ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/32141) in GitLab 13.1). | ```shell @@ -41,7 +40,8 @@ Example response: "path": "group/project", "project_id": 9, "location": "gitlab.example.com:5000/group/project", - "created_at": "2019-01-10T13:38:57.391Z" + "created_at": "2019-01-10T13:38:57.391Z", + "cleanup_policy_started_at": "2020-01-10T15:40:57.391Z" }, { "id": 2, @@ -49,7 +49,8 @@ Example response: "path": "group/project/releases", "project_id": 9, "location": "gitlab.example.com:5000/group/project/releases", - "created_at": "2019-01-10T13:39:08.229Z" + "created_at": "2019-01-10T13:39:08.229Z", + "cleanup_policy_started_at": "2020-08-17T03:12:35.489Z" } ] ``` @@ -66,7 +67,6 @@ GET /groups/:id/registry/repositories | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) accessible by the authenticated user. | | `tags` | boolean | no | If the parameter is included as true, each repository will include an array of `"tags"` in the response. | -| `name` | string | no | Returns a list of repositories with a name that matches the value. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29763) in GitLab 13.0). | | `tags_count` | boolean | no | If the parameter is included as true, each repository will include `"tags_count"` in the response ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/32141) in GitLab 13.1). | ```shell @@ -84,6 +84,7 @@ Example response: "project_id": 9, "location": "gitlab.example.com:5000/group/project", "created_at": "2019-01-10T13:38:57.391Z", + "cleanup_policy_started_at": "2020-08-17T03:12:35.489Z", "tags_count": 1, "tags": [ { @@ -100,6 +101,7 @@ Example response: "project_id": 11, "location": "gitlab.example.com:5000/group/other_project", "created_at": "2019-01-10T13:39:08.229Z", + "cleanup_policy_started_at": "2020-01-10T15:40:57.391Z", "tags_count": 3, "tags": [ { @@ -228,7 +230,7 @@ DELETE /projects/:id/registry/repositories/:repository_id/tags/:tag_name curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags/v10.0.0" ``` -This action does not delete blobs. In order to delete them and recycle disk space, +This action doesn't delete blobs. To delete them and recycle disk space, [run the garbage collection](https://docs.gitlab.com/omnibus/maintenance/README.html#removing-unused-layers-not-referenced-by-manifests). ## Delete registry repository tags in bulk @@ -254,18 +256,19 @@ DELETE /projects/:id/registry/repositories/:repository_id/tags This API call performs the following operations: -1. It orders all tags by creation date. The creation date is the time of the - manifest creation, not the time of tag push. -1. It removes only the tags matching the given `name_regex_delete` (or deprecated `name_regex`), keeping any that match `name_regex_keep`. -1. It never removes the tag named `latest`. -1. It keeps N latest matching tags (if `keep_n` is specified). -1. It only removes tags that are older than X amount of time (if `older_than` is specified). -1. It schedules the asynchronous job to be executed in the background. - -These operations are executed asynchronously and it might -take time to get executed. You can run this at most -once an hour for a given container repository. -This action does not delete blobs. In order to delete them and recycle disk space, +- It orders all tags by creation date. The creation date is the time of the + manifest creation, not the time of tag push. +- It removes only the tags matching the given `name_regex_delete` (or deprecated + `name_regex`), keeping any that match `name_regex_keep`. +- It never removes the tag named `latest`. +- It keeps N latest matching tags (if `keep_n` is specified). +- It only removes tags that are older than X amount of time (if `older_than` is + specified). +- It schedules the asynchronous job to be executed in the background. + +These operations are executed asynchronously and can take time to get executed. +You can run this at most once an hour for a given container repository. This +action doesn't delete blobs. To delete them and recycle disk space, [run the garbage collection](https://docs.gitlab.com/omnibus/maintenance/README.html#removing-unused-layers-not-referenced-by-manifests). NOTE: **Note:** diff --git a/doc/api/epics.md b/doc/api/epics.md index 91ea92c8589..5c7366c8457 100644 --- a/doc/api/epics.md +++ b/doc/api/epics.md @@ -267,6 +267,7 @@ POST /groups/:id/epics | `labels` | string | no | The comma separated list of labels | | `description` | string | no | The description of the epic. Limited to 1,048,576 characters. | | `confidential` | boolean | no | Whether the epic should be confidential | +| `created_at` | string | no | When the epic was created. Date time string, ISO 8601 formatted, for example `2016-03-11T03:45:40Z` . Requires administrator or project/group owner privileges ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/255309) in GitLab 13.5) | | `start_date_is_fixed` | boolean | no | Whether start date should be sourced from `start_date_fixed` or from milestones (since 11.3) | | `start_date_fixed` | string | no | The fixed start date of an epic (since 11.3) | | `due_date_is_fixed` | boolean | no | Whether due date should be sourced from `due_date_fixed` or from milestones (since 11.3) | @@ -349,6 +350,7 @@ PUT /groups/:id/epics/:epic_iid | `description` | string | no | The description of an epic. Limited to 1,048,576 characters. | | `confidential` | boolean | no | Whether the epic should be confidential | | `labels` | string | no | The comma separated list of labels | +| `updated_at` | string | no | When the epic was updated. Date time string, ISO 8601 formatted, for example `2016-03-11T03:45:40Z` . Requires administrator or project/group owner privileges ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/255309) in GitLab 13.5) | | `start_date_is_fixed` | boolean | no | Whether start date should be sourced from `start_date_fixed` or from milestones (since 11.3) | | `start_date_fixed` | string | no | The fixed start date of an epic (since 11.3) | | `due_date_is_fixed` | boolean | no | Whether due date should be sourced from `due_date_fixed` or from milestones (since 11.3) | @@ -422,10 +424,10 @@ DELETE /groups/:id/epics/:epic_iid curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/epics/5" ``` -## Create a to-do +## Create a to do -Manually creates a to-do for the current user on an epic. If -there already exists a to-do for the user on that epic, status code `304` is +Manually creates a to do for the current user on an epic. If +there already exists a to do for the user on that epic, status code `304` is returned. ```plaintext diff --git a/doc/api/feature_flags.md b/doc/api/feature_flags.md index 1088154b599..6ea01a34a25 100644 --- a/doc/api/feature_flags.md +++ b/doc/api/feature_flags.md @@ -151,7 +151,7 @@ POST /projects/:id/feature_flags | `description` | string | no | The description of the feature flag. | | `active` | boolean | no | The active state of the flag. Defaults to true. [Supported](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38350) in GitLab 13.3 and later. | | `strategies` | JSON | no | The feature flag [strategies](../operations/feature_flags.md#feature-flag-strategies). | -| `strategies:name` | JSON | no | The strategy name. | +| `strategies:name` | JSON | no | The strategy name. Can be `default`, `gradualRolloutUserId`, `userWithId`, or `gitlabUserList`. In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/36380) and later, can be [`flexibleRollout`](https://unleash.github.io/docs/activation_strategy#flexiblerollout). | | `strategies:parameters` | JSON | no | The strategy parameters. | | `strategies:scopes` | JSON | no | The scopes for the strategy. | | `strategies:scopes:environment_scope` | string | no | The environment spec for the scope. | diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md index 8d2052f7373..064bd26ee72 100644 --- a/doc/api/geo_nodes.md +++ b/doc/api/geo_nodes.md @@ -1,7 +1,7 @@ # Geo Nodes API **(PREMIUM ONLY)** -In order to interact with Geo node endpoints, you need to authenticate yourself -as an admin. +To interact with Geo node endpoints, you need to authenticate yourself as an +admin. ## Create a new Geo node @@ -460,12 +460,12 @@ Example response: "package_files_registry_count": 10, "package_files_synced_count": 6, "package_files_failed_count": 3, - "terraform_states_count": 10, - "terraform_states_checksummed_count": 10, - "terraform_states_checksum_failed_count": 0, - "terraform_states_registry_count": 10, - "terraform_states_synced_count": 6, - "terraform_states_failed_count": 3 + "terraform_state_versions_count": 10, + "terraform_state_versions_checksummed_count": 10, + "terraform_state_versions_checksum_failed_count": 0, + "terraform_state_versions_registry_count": 10, + "terraform_state_versions_synced_count": 6, + "terraform_state_versions_failed_count": 3, "snippet_repositories_count": 10, "snippet_repositories_checksummed_count": 10, "snippet_repositories_checksum_failed_count": 0, diff --git a/doc/api/graphql/reference/gitlab_schema.graphql b/doc/api/graphql/reference/gitlab_schema.graphql index 01d5044057a..84ac8d30660 100644 --- a/doc/api/graphql/reference/gitlab_schema.graphql +++ b/doc/api/graphql/reference/gitlab_schema.graphql @@ -435,6 +435,16 @@ Values for sorting alerts """ enum AlertManagementAlertSort { """ + Created at ascending order + """ + CREATED_ASC + + """ + Created at descending order + """ + CREATED_DESC + + """ Created time by ascending order """ CREATED_TIME_ASC @@ -495,6 +505,16 @@ enum AlertManagementAlertSort { STATUS_DESC """ + Updated at ascending order + """ + UPDATED_ASC + + """ + Updated at descending order + """ + UPDATED_DESC + + """ Created time by ascending order """ UPDATED_TIME_ASC @@ -507,22 +527,22 @@ enum AlertManagementAlertSort { """ Created at ascending order """ - created_asc + created_asc @deprecated(reason: "Use CREATED_ASC. Deprecated in 13.5") """ Created at descending order """ - created_desc + created_desc @deprecated(reason: "Use CREATED_DESC. Deprecated in 13.5") """ Updated at ascending order """ - updated_asc + updated_asc @deprecated(reason: "Use UPDATED_ASC. Deprecated in 13.5") """ Updated at descending order """ - updated_desc + updated_desc @deprecated(reason: "Use UPDATED_DESC. Deprecated in 13.5") } """ @@ -1035,7 +1055,7 @@ type Board { Returns the last _n_ elements from the list. """ last: Int - ): EpicConnection + ): BoardEpicConnection """ Whether or not backlog list is hidden. @@ -1134,6 +1154,478 @@ type BoardEdge { } """ +Represents an epic on an issue board +""" +type BoardEpic implements CurrentUserTodos & Noteable { + """ + Author of the epic + """ + author: User! + + """ + Children (sub-epics) of the epic + """ + children( + """ + Returns the elements in the list that come after the specified cursor. + """ + after: String + + """ + Filter epics by author + """ + authorUsername: String + + """ + Returns the elements in the list that come before the specified cursor. + """ + before: String + + """ + List items within a time frame where items.end_date is between startDate and + endDate parameters (startDate parameter must be present) + """ + endDate: Time + + """ + Returns the first _n_ elements from the list. + """ + first: Int + + """ + IID of the epic, e.g., "1" + """ + iid: ID + + """ + Filter epics by iid for autocomplete + """ + iidStartsWith: String + + """ + List of IIDs of epics, e.g., [1, 2] + """ + iids: [ID!] + + """ + Filter epics by labels + """ + labelName: [String!] + + """ + Returns the last _n_ elements from the list. + """ + last: Int + + """ + Filter epics by milestone title, computed from epic's issues + """ + milestoneTitle: String + + """ + Search query for epic title or description + """ + search: String + + """ + List epics by sort order + """ + sort: EpicSort + + """ + List items within a time frame where items.start_date is between startDate + and endDate parameters (endDate parameter must be present) + """ + startDate: Time + + """ + Filter epics by state + """ + state: EpicState + ): EpicConnection + + """ + Timestamp of when the epic was closed + """ + closedAt: Time + + """ + Indicates if the epic is confidential + """ + confidential: Boolean + + """ + Timestamp of when the epic was created + """ + createdAt: Time + + """ + Todos for the current user + """ + currentUserTodos( + """ + Returns the elements in the list that come after the specified cursor. + """ + after: String + + """ + Returns the elements in the list that come before the specified cursor. + """ + before: String + + """ + Returns the first _n_ elements from the list. + """ + first: Int + + """ + Returns the last _n_ elements from the list. + """ + last: Int + + """ + State of the todos + """ + state: TodoStateEnum + ): TodoConnection! + + """ + Number of open and closed descendant epics and issues + """ + descendantCounts: EpicDescendantCount + + """ + Total weight of open and closed issues in the epic and its descendants + """ + descendantWeightSum: EpicDescendantWeights + + """ + Description of the epic + """ + description: String + + """ + All discussions on this noteable + """ + discussions( + """ + Returns the elements in the list that come after the specified cursor. + """ + after: String + + """ + Returns the elements in the list that come before the specified cursor. + """ + before: String + + """ + Returns the first _n_ elements from the list. + """ + first: Int + + """ + Returns the last _n_ elements from the list. + """ + last: Int + ): DiscussionConnection! + + """ + Number of downvotes the epic has received + """ + downvotes: Int! + + """ + Due date of the epic + """ + dueDate: Time + + """ + Fixed due date of the epic + """ + dueDateFixed: Time + + """ + Inherited due date of the epic from milestones + """ + dueDateFromMilestones: Time + + """ + Indicates if the due date has been manually set + """ + dueDateIsFixed: Boolean + + """ + Group to which the epic belongs + """ + group: Group! + + """ + Indicates if the epic has children + """ + hasChildren: Boolean! + + """ + Indicates if the epic has direct issues + """ + hasIssues: Boolean! + + """ + Indicates if the epic has a parent epic + """ + hasParent: Boolean! + + """ + Current health status of the epic + """ + healthStatus: EpicHealthStatus + + """ + ID of the epic + """ + id: ID! + + """ + Internal ID of the epic + """ + iid: ID! + + """ + A list of issues associated with the epic + """ + issues( + """ + Returns the elements in the list that come after the specified cursor. + """ + after: String + + """ + Returns the elements in the list that come before the specified cursor. + """ + before: String + + """ + Returns the first _n_ elements from the list. + """ + first: Int + + """ + Returns the last _n_ elements from the list. + """ + last: Int + ): EpicIssueConnection + + """ + Labels assigned to the epic + """ + labels( + """ + Returns the elements in the list that come after the specified cursor. + """ + after: String + + """ + Returns the elements in the list that come before the specified cursor. + """ + before: String + + """ + Returns the first _n_ elements from the list. + """ + first: Int + + """ + Returns the last _n_ elements from the list. + """ + last: Int + ): LabelConnection + + """ + All notes on this noteable + """ + notes( + """ + Returns the elements in the list that come after the specified cursor. + """ + after: String + + """ + Returns the elements in the list that come before the specified cursor. + """ + before: String + + """ + Returns the first _n_ elements from the list. + """ + first: Int + + """ + Returns the last _n_ elements from the list. + """ + last: Int + ): NoteConnection! + + """ + Parent epic of the epic + """ + parent: Epic + + """ + List of participants for the epic + """ + participants( + """ + Returns the elements in the list that come after the specified cursor. + """ + after: String + + """ + Returns the elements in the list that come before the specified cursor. + """ + before: String + + """ + Returns the first _n_ elements from the list. + """ + first: Int + + """ + Returns the last _n_ elements from the list. + """ + last: Int + ): UserConnection + + """ + Internal reference of the epic. Returned in shortened format by default + """ + reference( + """ + Indicates if the reference should be returned in full + """ + full: Boolean = false + ): String! + + """ + URI path of the epic-issue relationship + """ + relationPath: String + + """ + The relative position of the epic in the epic tree + """ + relativePosition: Int + + """ + Start date of the epic + """ + startDate: Time + + """ + Fixed start date of the epic + """ + startDateFixed: Time + + """ + Inherited start date of the epic from milestones + """ + startDateFromMilestones: Time + + """ + Indicates if the start date has been manually set + """ + startDateIsFixed: Boolean + + """ + State of the epic + """ + state: EpicState! + + """ + Indicates the currently logged in user is subscribed to the epic + """ + subscribed: Boolean! + + """ + Title of the epic + """ + title: String + + """ + Timestamp of when the epic was updated + """ + updatedAt: Time + + """ + Number of upvotes the epic has received + """ + upvotes: Int! + + """ + Permissions for the current user on the resource + """ + userPermissions: EpicPermissions! + + """ + User preferences for the epic on the issue board + """ + userPreferences: BoardEpicUserPreferences + + """ + Web path of the epic + """ + webPath: String! + + """ + Web URL of the epic + """ + webUrl: String! +} + +""" +The connection type for BoardEpic. +""" +type BoardEpicConnection { + """ + A list of edges. + """ + edges: [BoardEpicEdge] + + """ + A list of nodes. + """ + nodes: [BoardEpic] + + """ + Information to aid in pagination. + """ + pageInfo: PageInfo! +} + +""" +An edge in a connection. +""" +type BoardEpicEdge { + """ + A cursor for use in pagination. + """ + cursor: String! + + """ + The item at the end of the edge. + """ + node: BoardEpic +} + +""" +Represents user preferences for a board epic +""" +type BoardEpicUserPreferences { + """ + Indicates epic should be displayed as collapsed + """ + collapsed: Boolean! +} + +""" Identifier of Board """ scalar BoardID @@ -2914,14 +3406,19 @@ input CreateRequirementInput { clientMutationId: String """ - The project full path the requirement is associated with + Description of the requirement + """ + description: String + + """ + Full project path the requirement is associated with """ projectPath: ID! """ Title of the requirement """ - title: String! + title: String } """ @@ -2939,7 +3436,7 @@ type CreateRequirementPayload { errors: [String!]! """ - The requirement after mutation + Requirement after mutation """ requirement: Requirement } @@ -3920,6 +4417,11 @@ A collection of designs """ type DesignCollection { """ + Copy state of the design collection + """ + copyState: DesignCollectionCopyState + + """ Find a specific design """ design( @@ -4047,6 +4549,26 @@ type DesignCollection { } """ +Copy state of a DesignCollection +""" +enum DesignCollectionCopyState { + """ + The DesignCollection encountered an error during a copy + """ + ERROR + + """ + The DesignCollection is being copied + """ + IN_PROGRESS + + """ + The DesignCollection has no copy in progress + """ + READY +} + +""" The connection type for Design. """ type DesignConnection { @@ -4471,6 +4993,41 @@ input DestroyBoardInput { } """ +Autogenerated input type of DestroyBoardList +""" +input DestroyBoardListInput { + """ + A unique identifier for the client performing the mutation. + """ + clientMutationId: String + + """ + Global ID of the list to destroy. Only label lists are accepted. + """ + listId: ListID! +} + +""" +Autogenerated return type of DestroyBoardList +""" +type DestroyBoardListPayload { + """ + A unique identifier for the client performing the mutation. + """ + clientMutationId: String + + """ + Errors encountered during execution of the mutation. + """ + errors: [String!]! + + """ + The list after mutation. + """ + list: BoardList +} + +""" Autogenerated return type of DestroyBoard """ type DestroyBoardPayload { @@ -5197,7 +5754,7 @@ type Epic implements CurrentUserTodos & Noteable { ): EpicConnection """ - Timestamp of the epic's closure + Timestamp of when the epic was closed """ closedAt: Time @@ -5207,7 +5764,7 @@ type Epic implements CurrentUserTodos & Noteable { confidential: Boolean """ - Timestamp of the epic's creation + Timestamp of when the epic was created """ createdAt: Time @@ -5502,7 +6059,7 @@ type Epic implements CurrentUserTodos & Noteable { title: String """ - Timestamp of the epic's last activity + Timestamp of when the epic was updated """ updatedAt: Time @@ -6328,6 +6885,37 @@ type GeoNode { internalUrl: String """ + Find merge request diff registries on this Geo node. Available only when + feature flag `geo_merge_request_diff_replication` is enabled + """ + mergeRequestDiffRegistries( + """ + Returns the elements in the list that come after the specified cursor. + """ + after: String + + """ + Returns the elements in the list that come before the specified cursor. + """ + before: String + + """ + Returns the first _n_ elements from the list. + """ + first: Int + + """ + Filters registries by their ID + """ + ids: [ID!] + + """ + Returns the last _n_ elements from the list. + """ + last: Int + ): MergeRequestDiffRegistryConnection + + """ The interval (in days) in which the repository verification is valid. Once expired, it will be reverified """ minimumReverificationInterval: Int @@ -6338,7 +6926,7 @@ type GeoNode { name: String """ - Package file registries of the GeoNode. Available only when feature flag `geo_package_file_replication` is enabled + Package file registries of the GeoNode """ packageFileRegistries( """ @@ -6418,10 +7006,9 @@ type GeoNode { syncObjectStorage: Boolean """ - Find terraform state registries on this Geo node. Available only when feature - flag `geo_terraform_state_replication` is enabled + Find terraform state version registries on this Geo node """ - terraformStateRegistries( + terraformStateVersionRegistries( """ Returns the elements in the list that come after the specified cursor. """ @@ -6446,7 +7033,7 @@ type GeoNode { Returns the last _n_ elements from the list. """ last: Int - ): TerraformStateRegistryConnection + ): TerraformStateVersionRegistryConnection """ The user-facing URL for this Geo node @@ -6507,9 +7094,9 @@ type Group { """ board( """ - Find a board by its ID + The board's ID """ - id: ID + id: ID! ): Board """ @@ -6781,6 +7368,16 @@ type Group { assigneeUsername: String """ + Usernames of users assigned to the issue + """ + assigneeUsernames: [String!] + + """ + Username of the author of the issue + """ + authorUsername: String + + """ Returns the elements in the list that come before the specified cursor. """ before: String @@ -8617,6 +9214,16 @@ Values for sorting issues """ enum IssueSort { """ + Created at ascending order + """ + CREATED_ASC + + """ + Created at descending order + """ + CREATED_DESC + + """ Due date by ascending order """ DUE_DATE_ASC @@ -8657,11 +9264,41 @@ enum IssueSort { PRIORITY_DESC """ + Published issues shown last + """ + PUBLISHED_ASC + + """ + Published issues shown first + """ + PUBLISHED_DESC + + """ Relative position by ascending order """ RELATIVE_POSITION_ASC """ + Severity from less critical to more critical + """ + SEVERITY_ASC + + """ + Severity from more critical to less critical + """ + SEVERITY_DESC + + """ + Updated at ascending order + """ + UPDATED_ASC + + """ + Updated at descending order + """ + UPDATED_DESC + + """ Weight by ascending order """ WEIGHT_ASC @@ -8674,22 +9311,22 @@ enum IssueSort { """ Created at ascending order """ - created_asc + created_asc @deprecated(reason: "Use CREATED_ASC. Deprecated in 13.5") """ Created at descending order """ - created_desc + created_desc @deprecated(reason: "Use CREATED_DESC. Deprecated in 13.5") """ Updated at ascending order """ - updated_asc + updated_asc @deprecated(reason: "Use UPDATED_ASC. Deprecated in 13.5") """ Updated at descending order """ - updated_desc + updated_desc @deprecated(reason: "Use UPDATED_DESC. Deprecated in 13.5") } """ @@ -9251,6 +9888,11 @@ Identifier of Label scalar LabelID """ +Identifier of List +""" +scalar ListID + +""" List limit metric setting """ enum ListLimitMetric { @@ -9319,6 +9961,26 @@ enum MeasurementIdentifier { PIPELINES """ + Pipeline count with canceled status + """ + PIPELINES_CANCELED + + """ + Pipeline count with failed status + """ + PIPELINES_FAILED + + """ + Pipeline count with skipped status + """ + PIPELINES_SKIPPED + + """ + Pipeline count with success status + """ + PIPELINES_SUCCEEDED + + """ Project count """ PROJECTS @@ -10019,6 +10681,86 @@ type MergeRequestCreatePayload { } """ +Represents the Geo sync and verification state of a Merge Request diff +""" +type MergeRequestDiffRegistry { + """ + Timestamp when the MergeRequestDiffRegistry was created + """ + createdAt: Time + + """ + ID of the MergeRequestDiffRegistry + """ + id: ID! + + """ + Error message during sync of the MergeRequestDiffRegistry + """ + lastSyncFailure: String + + """ + Timestamp of the most recent successful sync of the MergeRequestDiffRegistry + """ + lastSyncedAt: Time + + """ + ID of the Merge Request diff + """ + mergeRequestDiffId: ID! + + """ + Timestamp after which the MergeRequestDiffRegistry should be resynced + """ + retryAt: Time + + """ + Number of consecutive failed sync attempts of the MergeRequestDiffRegistry + """ + retryCount: Int + + """ + Sync state of the MergeRequestDiffRegistry + """ + state: RegistryState +} + +""" +The connection type for MergeRequestDiffRegistry. +""" +type MergeRequestDiffRegistryConnection { + """ + A list of edges. + """ + edges: [MergeRequestDiffRegistryEdge] + + """ + A list of nodes. + """ + nodes: [MergeRequestDiffRegistry] + + """ + Information to aid in pagination. + """ + pageInfo: PageInfo! +} + +""" +An edge in a connection. +""" +type MergeRequestDiffRegistryEdge { + """ + A cursor for use in pagination. + """ + cursor: String! + + """ + The item at the end of the edge. + """ + node: MergeRequestDiffRegistry +} + +""" An edge in a connection. """ type MergeRequestEdge { @@ -10368,6 +11110,16 @@ Values for sorting merge requests """ enum MergeRequestSort { """ + Created at ascending order + """ + CREATED_ASC + + """ + Created at descending order + """ + CREATED_DESC + + """ Label priority by ascending order """ LABEL_PRIORITY_ASC @@ -10408,24 +11160,34 @@ enum MergeRequestSort { PRIORITY_DESC """ + Updated at ascending order + """ + UPDATED_ASC + + """ + Updated at descending order + """ + UPDATED_DESC + + """ Created at ascending order """ - created_asc + created_asc @deprecated(reason: "Use CREATED_ASC. Deprecated in 13.5") """ Created at descending order """ - created_desc + created_desc @deprecated(reason: "Use CREATED_DESC. Deprecated in 13.5") """ Updated at ascending order """ - updated_asc + updated_asc @deprecated(reason: "Use UPDATED_ASC. Deprecated in 13.5") """ Updated at descending order """ - updated_desc + updated_desc @deprecated(reason: "Use UPDATED_DESC. Deprecated in 13.5") } """ @@ -10805,6 +11567,7 @@ type Mutation { designManagementMove(input: DesignManagementMoveInput!): DesignManagementMovePayload designManagementUpload(input: DesignManagementUploadInput!): DesignManagementUploadPayload destroyBoard(input: DestroyBoardInput!): DestroyBoardPayload + destroyBoardList(input: DestroyBoardListInput!): DestroyBoardListPayload destroyNote(input: DestroyNoteInput!): DestroyNotePayload destroySnippet(input: DestroySnippetInput!): DestroySnippetPayload @@ -10812,7 +11575,7 @@ type Mutation { Toggles the resolved state of a discussion """ discussionToggleResolve(input: DiscussionToggleResolveInput!): DiscussionToggleResolvePayload - dismissVulnerability(input: DismissVulnerabilityInput!): DismissVulnerabilityPayload + dismissVulnerability(input: DismissVulnerabilityInput!): DismissVulnerabilityPayload @deprecated(reason: "Use vulnerabilityDismiss. Deprecated in 13.5") epicAddIssue(input: EpicAddIssueInput!): EpicAddIssuePayload epicSetSubscription(input: EpicSetSubscriptionInput!): EpicSetSubscriptionPayload epicTreeReorder(input: EpicTreeReorderInput!): EpicTreeReorderPayload @@ -10847,6 +11610,7 @@ type Mutation { pipelineRetry(input: PipelineRetryInput!): PipelineRetryPayload removeAwardEmoji(input: RemoveAwardEmojiInput!): RemoveAwardEmojiPayload @deprecated(reason: "Use awardEmojiRemove. Deprecated in 13.2") removeProjectFromSecurityDashboard(input: RemoveProjectFromSecurityDashboardInput!): RemoveProjectFromSecurityDashboardPayload + revertVulnerabilityToDetected(input: RevertVulnerabilityToDetectedInput!): RevertVulnerabilityToDetectedPayload @deprecated(reason: "Use vulnerabilityRevertToDetected. Deprecated in 13.5") runDastScan(input: RunDASTScanInput!): RunDASTScanPayload @deprecated(reason: "Use DastOnDemandScanCreate. Deprecated in 13.4") todoMarkDone(input: TodoMarkDoneInput!): TodoMarkDonePayload todoRestore(input: TodoRestoreInput!): TodoRestorePayload @@ -10875,7 +11639,10 @@ type Mutation { updateNote(input: UpdateNoteInput!): UpdateNotePayload updateRequirement(input: UpdateRequirementInput!): UpdateRequirementPayload updateSnippet(input: UpdateSnippetInput!): UpdateSnippetPayload + vulnerabilityConfirm(input: VulnerabilityConfirmInput!): VulnerabilityConfirmPayload + vulnerabilityDismiss(input: VulnerabilityDismissInput!): VulnerabilityDismissPayload vulnerabilityResolve(input: VulnerabilityResolveInput!): VulnerabilityResolvePayload + vulnerabilityRevertToDetected(input: VulnerabilityRevertToDetectedInput!): VulnerabilityRevertToDetectedPayload } """ @@ -11409,7 +12176,7 @@ type PackageEdge { } """ -Represents the sync and verification state of a package file +Represents the Geo sync and verification state of a package file """ type PackageFileRegistry { """ @@ -11505,6 +12272,11 @@ enum PackageTypeEnum { GENERIC """ + Packages from the golang package manager + """ + GOLANG + + """ Packages from the maven package manager """ MAVEN @@ -11959,9 +12731,9 @@ type Project { """ board( """ - Find a board by its ID + The board's ID """ - id: ID + id: ID! ): Board """ @@ -12249,6 +13021,16 @@ type Project { assigneeUsername: String """ + Usernames of users assigned to the issue + """ + assigneeUsernames: [String!] + + """ + Username of the author of the issue + """ + authorUsername: String + + """ Issues closed after this date """ closedAfter: Time @@ -12339,6 +13121,16 @@ type Project { assigneeUsername: String """ + Usernames of users assigned to the issue + """ + assigneeUsernames: [String!] + + """ + Username of the author of the issue + """ + authorUsername: String + + """ Issues closed after this date """ closedAfter: Time @@ -12419,6 +13211,16 @@ type Project { assigneeUsername: String """ + Usernames of users assigned to the issue + """ + assigneeUsernames: [String!] + + """ + Username of the author of the issue + """ + authorUsername: String + + """ Returns the elements in the list that come before the specified cursor. """ before: String @@ -13017,7 +13819,7 @@ type Project { requestAccessEnabled: Boolean """ - Find a single requirement. Available only when feature flag `requirements_management` is enabled. + Find a single requirement """ requirement( """ @@ -13057,7 +13859,7 @@ type Project { requirementStatesCount: RequirementStatesCount """ - Find requirements. Available only when feature flag `requirements_management` is enabled. + Find requirements """ requirements( """ @@ -13257,6 +14059,31 @@ type Project { tagList: String """ + Terraform states associated with the project + """ + terraformStates( + """ + Returns the elements in the list that come after the specified cursor. + """ + after: String + + """ + Returns the elements in the list that come before the specified cursor. + """ + before: String + + """ + Returns the first _n_ elements from the list. + """ + first: Int + + """ + Returns the last _n_ elements from the list. + """ + last: Int + ): TerraformStateConnection + + """ Permissions for the current user on the resource """ userPermissions: ProjectPermissions! @@ -14001,6 +14828,11 @@ type Query { Search query for project name, path, or description """ search: String + + """ + Include namespace in project search + """ + searchNamespaces: Boolean ): ProjectConnection """ @@ -14249,6 +15081,16 @@ type Query { """ startDate: ISO8601Date! ): VulnerabilitiesCountByDayAndSeverityConnection @deprecated(reason: "Use `vulnerabilitiesCountByDay`. Deprecated in 13.3") + + """ + Find a vulnerability + """ + vulnerability( + """ + The Global ID of the Vulnerability + """ + id: VulnerabilityID! + ): Vulnerability } """ @@ -14840,6 +15682,16 @@ type Requirement { createdAt: Time! """ + Description of the requirement + """ + description: String + + """ + The GitLab Flavored Markdown rendering of `description` + """ + descriptionHtml: String + + """ ID of the requirement """ id: ID! @@ -14900,6 +15752,11 @@ type Requirement { title: String """ + The GitLab Flavored Markdown rendering of `title` + """ + titleHtml: String + + """ Timestamp of when the requirement was last updated """ updatedAt: Time! @@ -15020,6 +15877,41 @@ interface ResolvableInterface { resolvedBy: User } +""" +Autogenerated input type of RevertVulnerabilityToDetected +""" +input RevertVulnerabilityToDetectedInput { + """ + A unique identifier for the client performing the mutation. + """ + clientMutationId: String + + """ + ID of the vulnerability to be reverted + """ + id: VulnerabilityID! +} + +""" +Autogenerated return type of RevertVulnerabilityToDetected +""" +type RevertVulnerabilityToDetectedPayload { + """ + A unique identifier for the client performing the mutation. + """ + clientMutationId: String + + """ + Errors encountered during execution of the mutation. + """ + errors: [String!]! + + """ + The vulnerability after revert + """ + vulnerability: Vulnerability +} + type RootStorageStatistics { """ The CI artifacts size in bytes @@ -15273,6 +16165,26 @@ type SastCiConfigurationAnalyzersEntityEdge { } """ +Represents the analyzers entity in SAST CI configuration +""" +input SastCiConfigurationAnalyzersEntityInput { + """ + State of the analyzer + """ + enabled: Boolean! + + """ + Name of analyzer + """ + name: String! + + """ + List of variables for the analyzer + """ + variables: [SastCiConfigurationEntityInput!] +} + +""" Represents an entity in SAST CI configuration """ type SastCiConfigurationEntity { @@ -15397,6 +16309,11 @@ Represents a CI configuration of SAST """ input SastCiConfigurationInput { """ + List of analyzers and related variables for the SAST configuration + """ + analyzers: [SastCiConfigurationAnalyzersEntityInput!] + + """ List of global entities related to SAST configuration """ global: [SastCiConfigurationEntityInput!] @@ -16183,7 +17100,32 @@ type Snippet implements Noteable { """ Snippet blobs """ - blobs: [SnippetBlob!]! + blobs( + """ + Returns the elements in the list that come after the specified cursor. + """ + after: String + + """ + Returns the elements in the list that come before the specified cursor. + """ + before: String + + """ + Returns the first _n_ elements from the list. + """ + first: Int + + """ + Returns the last _n_ elements from the list. + """ + last: Int + + """ + Paths of the blobs + """ + paths: [String!] + ): SnippetBlobConnection """ Timestamp this snippet was created @@ -16407,6 +17349,41 @@ input SnippetBlobActionInputType { } """ +The connection type for SnippetBlob. +""" +type SnippetBlobConnection { + """ + A list of edges. + """ + edges: [SnippetBlobEdge] + + """ + A list of nodes. + """ + nodes: [SnippetBlob] + + """ + Information to aid in pagination. + """ + pageInfo: PageInfo! +} + +""" +An edge in a connection. +""" +type SnippetBlobEdge { + """ + A cursor for use in pagination. + """ + cursor: String! + + """ + The item at the end of the edge. + """ + node: SnippetBlob +} + +""" Represents how the blob content should be displayed """ type SnippetBlobViewer { @@ -16520,22 +17497,42 @@ enum Sort { """ Created at ascending order """ - created_asc + CREATED_ASC + + """ + Created at descending order + """ + CREATED_DESC + + """ + Updated at ascending order + """ + UPDATED_ASC + + """ + Updated at descending order + """ + UPDATED_DESC + + """ + Created at ascending order + """ + created_asc @deprecated(reason: "Use CREATED_ASC. Deprecated in 13.5") """ Created at descending order """ - created_desc + created_desc @deprecated(reason: "Use CREATED_DESC. Deprecated in 13.5") """ Updated at ascending order """ - updated_asc + updated_asc @deprecated(reason: "Use UPDATED_ASC. Deprecated in 13.5") """ Updated at descending order """ - updated_desc + updated_desc @deprecated(reason: "Use UPDATED_DESC. Deprecated in 13.5") } type Submodule implements Entry { @@ -16630,64 +17627,131 @@ type TaskCompletionStatus { count: Int! } +type TerraformState { + """ + Timestamp the Terraform state was created + """ + createdAt: Time! + + """ + ID of the Terraform state + """ + id: ID! + + """ + Timestamp the Terraform state was locked + """ + lockedAt: Time + + """ + The user currently holding a lock on the Terraform state + """ + lockedByUser: User + + """ + Name of the Terraform state + """ + name: String! + + """ + Timestamp the Terraform state was updated + """ + updatedAt: Time! +} + +""" +The connection type for TerraformState. +""" +type TerraformStateConnection { + """ + A list of edges. + """ + edges: [TerraformStateEdge] + + """ + A list of nodes. + """ + nodes: [TerraformState] + + """ + Information to aid in pagination. + """ + pageInfo: PageInfo! +} + +""" +An edge in a connection. +""" +type TerraformStateEdge { + """ + A cursor for use in pagination. + """ + cursor: String! + + """ + The item at the end of the edge. + """ + node: TerraformState +} + """ -Represents the sync and verification state of a terraform state +Represents the Geo sync and verification state of a terraform state version """ -type TerraformStateRegistry { +type TerraformStateVersionRegistry { """ - Timestamp when the TerraformStateRegistry was created + Timestamp when the TerraformStateVersionRegistry was created """ createdAt: Time """ - ID of the TerraformStateRegistry + ID of the TerraformStateVersionRegistry """ id: ID! """ - Error message during sync of the TerraformStateRegistry + Error message during sync of the TerraformStateVersionRegistry """ lastSyncFailure: String """ - Timestamp of the most recent successful sync of the TerraformStateRegistry + Timestamp of the most recent successful sync of the TerraformStateVersionRegistry """ lastSyncedAt: Time """ - Timestamp after which the TerraformStateRegistry should be resynced + Timestamp after which the TerraformStateVersionRegistry should be resynced """ retryAt: Time """ - Number of consecutive failed sync attempts of the TerraformStateRegistry + Number of consecutive failed sync attempts of the TerraformStateVersionRegistry """ retryCount: Int """ - Sync state of the TerraformStateRegistry + Sync state of the TerraformStateVersionRegistry """ state: RegistryState """ - ID of the TerraformState + ID of the terraform state version """ - terraformStateId: ID! + terraformStateVersionId: ID! } """ -The connection type for TerraformStateRegistry. +The connection type for TerraformStateVersionRegistry. """ -type TerraformStateRegistryConnection { +type TerraformStateVersionRegistryConnection { """ A list of edges. """ - edges: [TerraformStateRegistryEdge] + edges: [TerraformStateVersionRegistryEdge] """ A list of nodes. """ - nodes: [TerraformStateRegistry] + nodes: [TerraformStateVersionRegistry] """ Information to aid in pagination. @@ -16698,7 +17762,7 @@ type TerraformStateRegistryConnection { """ An edge in a connection. """ -type TerraformStateRegistryEdge { +type TerraformStateVersionRegistryEdge { """ A cursor for use in pagination. """ @@ -16707,7 +17771,7 @@ type TerraformStateRegistryEdge { """ The item at the end of the edge. """ - node: TerraformStateRegistry + node: TerraformStateVersionRegistry } """ @@ -17943,6 +19007,11 @@ input UpdateRequirementInput { clientMutationId: String """ + Description of the requirement + """ + description: String + + """ The iid of the requirement to update """ iid: String! @@ -17953,7 +19022,7 @@ input UpdateRequirementInput { lastTestReportState: TestReportState """ - The project full path the requirement is associated with + Full project path the requirement is associated with """ projectPath: ID! @@ -17983,7 +19052,7 @@ type UpdateRequirementPayload { errors: [String!]! """ - The requirement after mutation + Requirement after mutation """ requirement: Requirement } @@ -18666,7 +19735,7 @@ type VulnerabilitiesCountByDayEdge { """ Represents a vulnerability """ -type Vulnerability { +type Vulnerability implements Noteable { """ Description of the vulnerability """ @@ -18678,6 +19747,31 @@ type Vulnerability { detectedAt: Time! """ + All discussions on this noteable + """ + discussions( + """ + Returns the elements in the list that come after the specified cursor. + """ + after: String + + """ + Returns the elements in the list that come before the specified cursor. + """ + before: String + + """ + Returns the first _n_ elements from the list. + """ + first: Int + + """ + Returns the last _n_ elements from the list. + """ + last: Int + ): DiscussionConnection! + + """ GraphQL ID of the vulnerability """ id: ID! @@ -18723,6 +19817,31 @@ type Vulnerability { location: VulnerabilityLocation """ + All notes on this noteable + """ + notes( + """ + Returns the elements in the list that come after the specified cursor. + """ + after: String + + """ + Returns the elements in the list that come before the specified cursor. + """ + before: String + + """ + Returns the first _n_ elements from the list. + """ + first: Int + + """ + Returns the last _n_ elements from the list. + """ + last: Int + ): NoteConnection! + + """ Primary identifier of the vulnerability. """ primaryIdentifier: VulnerabilityIdentifier @@ -18781,6 +19900,41 @@ type Vulnerability { } """ +Autogenerated input type of VulnerabilityConfirm +""" +input VulnerabilityConfirmInput { + """ + A unique identifier for the client performing the mutation. + """ + clientMutationId: String + + """ + ID of the vulnerability to be confirmed + """ + id: VulnerabilityID! +} + +""" +Autogenerated return type of VulnerabilityConfirm +""" +type VulnerabilityConfirmPayload { + """ + A unique identifier for the client performing the mutation. + """ + clientMutationId: String + + """ + Errors encountered during execution of the mutation. + """ + errors: [String!]! + + """ + The vulnerability after state change + """ + vulnerability: Vulnerability +} + +""" The connection type for Vulnerability. """ type VulnerabilityConnection { @@ -18801,6 +19955,46 @@ type VulnerabilityConnection { } """ +Autogenerated input type of VulnerabilityDismiss +""" +input VulnerabilityDismissInput { + """ + A unique identifier for the client performing the mutation. + """ + clientMutationId: String + + """ + Reason why vulnerability should be dismissed + """ + comment: String + + """ + ID of the vulnerability to be dismissed + """ + id: ID! +} + +""" +Autogenerated return type of VulnerabilityDismiss +""" +type VulnerabilityDismissPayload { + """ + A unique identifier for the client performing the mutation. + """ + clientMutationId: String + + """ + Errors encountered during execution of the mutation. + """ + errors: [String!]! + + """ + The vulnerability after dismissal + """ + vulnerability: Vulnerability +} + +""" An edge in a connection. """ type VulnerabilityEdge { @@ -19141,7 +20335,7 @@ input VulnerabilityResolveInput { clientMutationId: String """ - ID of the vulnerability to be resolveed + ID of the vulnerability to be resolved """ id: VulnerabilityID! } @@ -19167,6 +20361,41 @@ type VulnerabilityResolvePayload { } """ +Autogenerated input type of VulnerabilityRevertToDetected +""" +input VulnerabilityRevertToDetectedInput { + """ + A unique identifier for the client performing the mutation. + """ + clientMutationId: String + + """ + ID of the vulnerability to be reverted + """ + id: VulnerabilityID! +} + +""" +Autogenerated return type of VulnerabilityRevertToDetected +""" +type VulnerabilityRevertToDetectedPayload { + """ + A unique identifier for the client performing the mutation. + """ + clientMutationId: String + + """ + Errors encountered during execution of the mutation. + """ + errors: [String!]! + + """ + The vulnerability after revert + """ + vulnerability: Vulnerability +} + +""" Represents a vulnerability scanner """ type VulnerabilityScanner { @@ -19278,6 +20507,26 @@ Vulnerability sort values """ enum VulnerabilitySort { """ + Detection timestamp in ascending order + """ + detected_asc + + """ + Detection timestamp in descending order + """ + detected_desc + + """ + Report Type in ascending order + """ + report_type_asc + + """ + Report Type in descending order + """ + report_type_desc + + """ Severity in ascending order """ severity_asc @@ -19286,6 +20535,16 @@ enum VulnerabilitySort { Severity in descending order """ severity_desc + + """ + Title in ascending order + """ + title_asc + + """ + Title in descending order + """ + title_desc } """ diff --git a/doc/api/graphql/reference/gitlab_schema.json b/doc/api/graphql/reference/gitlab_schema.json index 6458a676612..15aa205bed3 100644 --- a/doc/api/graphql/reference/gitlab_schema.json +++ b/doc/api/graphql/reference/gitlab_schema.json @@ -1227,23 +1227,47 @@ { "name": "updated_desc", "description": "Updated at descending order", + "isDeprecated": true, + "deprecationReason": "Use UPDATED_DESC. Deprecated in 13.5" + }, + { + "name": "updated_asc", + "description": "Updated at ascending order", + "isDeprecated": true, + "deprecationReason": "Use UPDATED_ASC. Deprecated in 13.5" + }, + { + "name": "created_desc", + "description": "Created at descending order", + "isDeprecated": true, + "deprecationReason": "Use CREATED_DESC. Deprecated in 13.5" + }, + { + "name": "created_asc", + "description": "Created at ascending order", + "isDeprecated": true, + "deprecationReason": "Use CREATED_ASC. Deprecated in 13.5" + }, + { + "name": "UPDATED_DESC", + "description": "Updated at descending order", "isDeprecated": false, "deprecationReason": null }, { - "name": "updated_asc", + "name": "UPDATED_ASC", "description": "Updated at ascending order", "isDeprecated": false, "deprecationReason": null }, { - "name": "created_desc", + "name": "CREATED_DESC", "description": "Created at descending order", "isDeprecated": false, "deprecationReason": null }, { - "name": "created_asc", + "name": "CREATED_ASC", "description": "Created at ascending order", "isDeprecated": false, "deprecationReason": null @@ -2764,7 +2788,7 @@ ], "type": { "kind": "OBJECT", - "name": "EpicConnection", + "name": "BoardEpicConnection", "ofType": null }, "isDeprecated": false, @@ -3042,6 +3066,1263 @@ "possibleTypes": null }, { + "kind": "OBJECT", + "name": "BoardEpic", + "description": "Represents an epic on an issue board", + "fields": [ + { + "name": "author", + "description": "Author of the epic", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "User", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "children", + "description": "Children (sub-epics) of the epic", + "args": [ + { + "name": "startDate", + "description": "List items within a time frame where items.start_date is between startDate and endDate parameters (endDate parameter must be present)", + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "endDate", + "description": "List items within a time frame where items.end_date is between startDate and endDate parameters (startDate parameter must be present)", + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "iid", + "description": "IID of the epic, e.g., \"1\"", + "type": { + "kind": "SCALAR", + "name": "ID", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "iids", + "description": "List of IIDs of epics, e.g., [1, 2]", + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "ID", + "ofType": null + } + } + }, + "defaultValue": null + }, + { + "name": "state", + "description": "Filter epics by state", + "type": { + "kind": "ENUM", + "name": "EpicState", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "search", + "description": "Search query for epic title or description", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "sort", + "description": "List epics by sort order", + "type": { + "kind": "ENUM", + "name": "EpicSort", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "authorUsername", + "description": "Filter epics by author", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "labelName", + "description": "Filter epics by labels", + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + } + }, + "defaultValue": null + }, + { + "name": "milestoneTitle", + "description": "Filter epics by milestone title, computed from epic's issues", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "iidStartsWith", + "description": "Filter epics by iid for autocomplete", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "after", + "description": "Returns the elements in the list that come after the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "before", + "description": "Returns the elements in the list that come before the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "first", + "description": "Returns the first _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "last", + "description": "Returns the last _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + } + ], + "type": { + "kind": "OBJECT", + "name": "EpicConnection", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "closedAt", + "description": "Timestamp of when the epic was closed", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "confidential", + "description": "Indicates if the epic is confidential", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Boolean", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "createdAt", + "description": "Timestamp of when the epic was created", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "currentUserTodos", + "description": "Todos for the current user", + "args": [ + { + "name": "after", + "description": "Returns the elements in the list that come after the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "before", + "description": "Returns the elements in the list that come before the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "first", + "description": "Returns the first _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "last", + "description": "Returns the last _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "state", + "description": "State of the todos", + "type": { + "kind": "ENUM", + "name": "TodoStateEnum", + "ofType": null + }, + "defaultValue": null + } + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "TodoConnection", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "descendantCounts", + "description": "Number of open and closed descendant epics and issues", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "EpicDescendantCount", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "descendantWeightSum", + "description": "Total weight of open and closed issues in the epic and its descendants", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "EpicDescendantWeights", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "description", + "description": "Description of the epic", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "discussions", + "description": "All discussions on this noteable", + "args": [ + { + "name": "after", + "description": "Returns the elements in the list that come after the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "before", + "description": "Returns the elements in the list that come before the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "first", + "description": "Returns the first _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "last", + "description": "Returns the last _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + } + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "DiscussionConnection", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "downvotes", + "description": "Number of downvotes the epic has received", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "dueDate", + "description": "Due date of the epic", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "dueDateFixed", + "description": "Fixed due date of the epic", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "dueDateFromMilestones", + "description": "Inherited due date of the epic from milestones", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "dueDateIsFixed", + "description": "Indicates if the due date has been manually set", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Boolean", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "group", + "description": "Group to which the epic belongs", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "Group", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "hasChildren", + "description": "Indicates if the epic has children", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "Boolean", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "hasIssues", + "description": "Indicates if the epic has direct issues", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "Boolean", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "hasParent", + "description": "Indicates if the epic has a parent epic", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "Boolean", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "healthStatus", + "description": "Current health status of the epic", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "EpicHealthStatus", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "id", + "description": "ID of the epic", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "ID", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "iid", + "description": "Internal ID of the epic", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "ID", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "issues", + "description": "A list of issues associated with the epic", + "args": [ + { + "name": "after", + "description": "Returns the elements in the list that come after the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "before", + "description": "Returns the elements in the list that come before the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "first", + "description": "Returns the first _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "last", + "description": "Returns the last _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + } + ], + "type": { + "kind": "OBJECT", + "name": "EpicIssueConnection", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "labels", + "description": "Labels assigned to the epic", + "args": [ + { + "name": "after", + "description": "Returns the elements in the list that come after the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "before", + "description": "Returns the elements in the list that come before the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "first", + "description": "Returns the first _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "last", + "description": "Returns the last _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + } + ], + "type": { + "kind": "OBJECT", + "name": "LabelConnection", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "notes", + "description": "All notes on this noteable", + "args": [ + { + "name": "after", + "description": "Returns the elements in the list that come after the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "before", + "description": "Returns the elements in the list that come before the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "first", + "description": "Returns the first _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "last", + "description": "Returns the last _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + } + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "NoteConnection", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "parent", + "description": "Parent epic of the epic", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "Epic", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "participants", + "description": "List of participants for the epic", + "args": [ + { + "name": "after", + "description": "Returns the elements in the list that come after the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "before", + "description": "Returns the elements in the list that come before the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "first", + "description": "Returns the first _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "last", + "description": "Returns the last _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + } + ], + "type": { + "kind": "OBJECT", + "name": "UserConnection", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "reference", + "description": "Internal reference of the epic. Returned in shortened format by default", + "args": [ + { + "name": "full", + "description": "Indicates if the reference should be returned in full", + "type": { + "kind": "SCALAR", + "name": "Boolean", + "ofType": null + }, + "defaultValue": "false" + } + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "relationPath", + "description": "URI path of the epic-issue relationship", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "relativePosition", + "description": "The relative position of the epic in the epic tree", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "startDate", + "description": "Start date of the epic", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "startDateFixed", + "description": "Fixed start date of the epic", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "startDateFromMilestones", + "description": "Inherited start date of the epic from milestones", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "startDateIsFixed", + "description": "Indicates if the start date has been manually set", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Boolean", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "state", + "description": "State of the epic", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "ENUM", + "name": "EpicState", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "subscribed", + "description": "Indicates the currently logged in user is subscribed to the epic", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "Boolean", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "title", + "description": "Title of the epic", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "updatedAt", + "description": "Timestamp of when the epic was updated", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "upvotes", + "description": "Number of upvotes the epic has received", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "userPermissions", + "description": "Permissions for the current user on the resource", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "EpicPermissions", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "userPreferences", + "description": "User preferences for the epic on the issue board", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "BoardEpicUserPreferences", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "webPath", + "description": "Web path of the epic", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "webUrl", + "description": "Web URL of the epic", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + { + "kind": "INTERFACE", + "name": "Noteable", + "ofType": null + }, + { + "kind": "INTERFACE", + "name": "CurrentUserTodos", + "ofType": null + } + ], + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "BoardEpicConnection", + "description": "The connection type for BoardEpic.", + "fields": [ + { + "name": "edges", + "description": "A list of edges.", + "args": [ + + ], + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "BoardEpicEdge", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "nodes", + "description": "A list of nodes.", + "args": [ + + ], + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "BoardEpic", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "pageInfo", + "description": "Information to aid in pagination.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "PageInfo", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "BoardEpicEdge", + "description": "An edge in a connection.", + "fields": [ + { + "name": "cursor", + "description": "A cursor for use in pagination.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "node", + "description": "The item at the end of the edge.", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "BoardEpic", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "BoardEpicUserPreferences", + "description": "Represents user preferences for a board epic", + "fields": [ + { + "name": "collapsed", + "description": "Indicates epic should be displayed as collapsed", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "Boolean", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { "kind": "SCALAR", "name": "BoardID", "description": "Identifier of Board", @@ -7872,19 +9153,25 @@ "name": "title", "description": "Title of the requirement", "type": { - "kind": "NON_NULL", - "name": null, - "ofType": { - "kind": "SCALAR", - "name": "String", - "ofType": null - } + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "description", + "description": "Description of the requirement", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null }, "defaultValue": null }, { "name": "projectPath", - "description": "The project full path the requirement is associated with", + "description": "Full project path the requirement is associated with", "type": { "kind": "NON_NULL", "name": null, @@ -7958,7 +9245,7 @@ }, { "name": "requirement", - "description": "The requirement after mutation", + "description": "Requirement after mutation", "args": [ ], @@ -8373,6 +9660,11 @@ "possibleTypes": [ { "kind": "OBJECT", + "name": "BoardEpic", + "ofType": null + }, + { + "kind": "OBJECT", "name": "Design", "ofType": null }, @@ -10790,6 +12082,20 @@ "description": "A collection of designs", "fields": [ { + "name": "copyState", + "description": "Copy state of the design collection", + "args": [ + + ], + "type": { + "kind": "ENUM", + "name": "DesignCollectionCopyState", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { "name": "design", "description": "Find a specific design", "args": [ @@ -11107,6 +12413,35 @@ "possibleTypes": null }, { + "kind": "ENUM", + "name": "DesignCollectionCopyState", + "description": "Copy state of a DesignCollection", + "fields": null, + "inputFields": null, + "interfaces": null, + "enumValues": [ + { + "name": "READY", + "description": "The DesignCollection has no copy in progress", + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "IN_PROGRESS", + "description": "The DesignCollection is being copied", + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "ERROR", + "description": "The DesignCollection encountered an error during a copy", + "isDeprecated": false, + "deprecationReason": null + } + ], + "possibleTypes": null + }, + { "kind": "OBJECT", "name": "DesignConnection", "description": "The connection type for Design.", @@ -12358,6 +13693,108 @@ "possibleTypes": null }, { + "kind": "INPUT_OBJECT", + "name": "DestroyBoardListInput", + "description": "Autogenerated input type of DestroyBoardList", + "fields": null, + "inputFields": [ + { + "name": "listId", + "description": "Global ID of the list to destroy. Only label lists are accepted.", + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "ListID", + "ofType": null + } + }, + "defaultValue": null + }, + { + "name": "clientMutationId", + "description": "A unique identifier for the client performing the mutation.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + } + ], + "interfaces": null, + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "DestroyBoardListPayload", + "description": "Autogenerated return type of DestroyBoardList", + "fields": [ + { + "name": "clientMutationId", + "description": "A unique identifier for the client performing the mutation.", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "errors", + "description": "Errors encountered during execution of the mutation.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + } + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "list", + "description": "The list after mutation.", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "BoardList", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { "kind": "OBJECT", "name": "DestroyBoardPayload", "description": "Autogenerated return type of DestroyBoard", @@ -14559,7 +15996,7 @@ }, { "name": "closedAt", - "description": "Timestamp of the epic's closure", + "description": "Timestamp of when the epic was closed", "args": [ ], @@ -14587,7 +16024,7 @@ }, { "name": "createdAt", - "description": "Timestamp of the epic's creation", + "description": "Timestamp of when the epic was created", "args": [ ], @@ -15354,7 +16791,7 @@ }, { "name": "updatedAt", - "description": "Timestamp of the epic's last activity", + "description": "Timestamp of when the epic was updated", "args": [ ], @@ -17689,6 +19126,77 @@ "deprecationReason": null }, { + "name": "mergeRequestDiffRegistries", + "description": "Find merge request diff registries on this Geo node. Available only when feature flag `geo_merge_request_diff_replication` is enabled", + "args": [ + { + "name": "ids", + "description": "Filters registries by their ID", + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "ID", + "ofType": null + } + } + }, + "defaultValue": null + }, + { + "name": "after", + "description": "Returns the elements in the list that come after the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "before", + "description": "Returns the elements in the list that come before the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "first", + "description": "Returns the first _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "last", + "description": "Returns the last _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + } + ], + "type": { + "kind": "OBJECT", + "name": "MergeRequestDiffRegistryConnection", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { "name": "minimumReverificationInterval", "description": "The interval (in days) in which the repository verification is valid. Once expired, it will be reverified", "args": [ @@ -17718,7 +19226,7 @@ }, { "name": "packageFileRegistries", - "description": "Package file registries of the GeoNode. Available only when feature flag `geo_package_file_replication` is enabled", + "description": "Package file registries of the GeoNode", "args": [ { "name": "ids", @@ -17919,8 +19427,8 @@ "deprecationReason": null }, { - "name": "terraformStateRegistries", - "description": "Find terraform state registries on this Geo node. Available only when feature flag `geo_terraform_state_replication` is enabled", + "name": "terraformStateVersionRegistries", + "description": "Find terraform state version registries on this Geo node", "args": [ { "name": "ids", @@ -17983,7 +19491,7 @@ ], "type": { "kind": "OBJECT", - "name": "TerraformStateRegistryConnection", + "name": "TerraformStateVersionRegistryConnection", "ofType": null }, "isDeprecated": false, @@ -18185,11 +19693,15 @@ "args": [ { "name": "id", - "description": "Find a board by its ID", + "description": "The board's ID", "type": { - "kind": "SCALAR", - "name": "ID", - "ofType": null + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "ID", + "ofType": null + } }, "defaultValue": null } @@ -18849,6 +20361,16 @@ "defaultValue": null }, { + "name": "authorUsername", + "description": "Username of the author of the issue", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { "name": "assigneeUsername", "description": "Username of a user assigned to the issue", "type": { @@ -18859,6 +20381,24 @@ "defaultValue": null }, { + "name": "assigneeUsernames", + "description": "Usernames of users assigned to the issue", + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + } + }, + "defaultValue": null + }, + { "name": "assigneeId", "description": "ID of a user assigned to the issues, \"none\" and \"any\" values supported", "type": { @@ -23861,23 +25401,47 @@ { "name": "updated_desc", "description": "Updated at descending order", + "isDeprecated": true, + "deprecationReason": "Use UPDATED_DESC. Deprecated in 13.5" + }, + { + "name": "updated_asc", + "description": "Updated at ascending order", + "isDeprecated": true, + "deprecationReason": "Use UPDATED_ASC. Deprecated in 13.5" + }, + { + "name": "created_desc", + "description": "Created at descending order", + "isDeprecated": true, + "deprecationReason": "Use CREATED_DESC. Deprecated in 13.5" + }, + { + "name": "created_asc", + "description": "Created at ascending order", + "isDeprecated": true, + "deprecationReason": "Use CREATED_ASC. Deprecated in 13.5" + }, + { + "name": "UPDATED_DESC", + "description": "Updated at descending order", "isDeprecated": false, "deprecationReason": null }, { - "name": "updated_asc", + "name": "UPDATED_ASC", "description": "Updated at ascending order", "isDeprecated": false, "deprecationReason": null }, { - "name": "created_desc", + "name": "CREATED_DESC", "description": "Created at descending order", "isDeprecated": false, "deprecationReason": null }, { - "name": "created_asc", + "name": "CREATED_ASC", "description": "Created at ascending order", "isDeprecated": false, "deprecationReason": null @@ -23937,6 +25501,18 @@ "deprecationReason": null }, { + "name": "SEVERITY_ASC", + "description": "Severity from less critical to more critical", + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "SEVERITY_DESC", + "description": "Severity from more critical to less critical", + "isDeprecated": false, + "deprecationReason": null + }, + { "name": "WEIGHT_ASC", "description": "Weight by ascending order", "isDeprecated": false, @@ -23947,6 +25523,18 @@ "description": "Weight by descending order", "isDeprecated": false, "deprecationReason": null + }, + { + "name": "PUBLISHED_ASC", + "description": "Published issues shown last", + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "PUBLISHED_DESC", + "description": "Published issues shown first", + "isDeprecated": false, + "deprecationReason": null } ], "possibleTypes": null @@ -25672,6 +27260,16 @@ "possibleTypes": null }, { + "kind": "SCALAR", + "name": "ListID", + "description": "Identifier of List", + "fields": null, + "inputFields": null, + "interfaces": null, + "enumValues": null, + "possibleTypes": null + }, + { "kind": "ENUM", "name": "ListLimitMetric", "description": "List limit metric setting", @@ -25845,6 +27443,30 @@ "description": "Pipeline count", "isDeprecated": false, "deprecationReason": null + }, + { + "name": "PIPELINES_SUCCEEDED", + "description": "Pipeline count with success status", + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "PIPELINES_FAILED", + "description": "Pipeline count with failed status", + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "PIPELINES_CANCELED", + "description": "Pipeline count with canceled status", + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "PIPELINES_SKIPPED", + "description": "Pipeline count with skipped status", + "isDeprecated": false, + "deprecationReason": null } ], "possibleTypes": null @@ -27832,6 +29454,251 @@ }, { "kind": "OBJECT", + "name": "MergeRequestDiffRegistry", + "description": "Represents the Geo sync and verification state of a Merge Request diff", + "fields": [ + { + "name": "createdAt", + "description": "Timestamp when the MergeRequestDiffRegistry was created", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "id", + "description": "ID of the MergeRequestDiffRegistry", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "ID", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "lastSyncFailure", + "description": "Error message during sync of the MergeRequestDiffRegistry", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "lastSyncedAt", + "description": "Timestamp of the most recent successful sync of the MergeRequestDiffRegistry", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "mergeRequestDiffId", + "description": "ID of the Merge Request diff", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "ID", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "retryAt", + "description": "Timestamp after which the MergeRequestDiffRegistry should be resynced", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "retryCount", + "description": "Number of consecutive failed sync attempts of the MergeRequestDiffRegistry", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "state", + "description": "Sync state of the MergeRequestDiffRegistry", + "args": [ + + ], + "type": { + "kind": "ENUM", + "name": "RegistryState", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "MergeRequestDiffRegistryConnection", + "description": "The connection type for MergeRequestDiffRegistry.", + "fields": [ + { + "name": "edges", + "description": "A list of edges.", + "args": [ + + ], + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "MergeRequestDiffRegistryEdge", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "nodes", + "description": "A list of nodes.", + "args": [ + + ], + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "MergeRequestDiffRegistry", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "pageInfo", + "description": "Information to aid in pagination.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "PageInfo", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "MergeRequestDiffRegistryEdge", + "description": "An edge in a connection.", + "fields": [ + { + "name": "cursor", + "description": "A cursor for use in pagination.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "node", + "description": "The item at the end of the edge.", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "MergeRequestDiffRegistry", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", "name": "MergeRequestEdge", "description": "An edge in a connection.", "fields": [ @@ -28873,23 +30740,47 @@ { "name": "updated_desc", "description": "Updated at descending order", + "isDeprecated": true, + "deprecationReason": "Use UPDATED_DESC. Deprecated in 13.5" + }, + { + "name": "updated_asc", + "description": "Updated at ascending order", + "isDeprecated": true, + "deprecationReason": "Use UPDATED_ASC. Deprecated in 13.5" + }, + { + "name": "created_desc", + "description": "Created at descending order", + "isDeprecated": true, + "deprecationReason": "Use CREATED_DESC. Deprecated in 13.5" + }, + { + "name": "created_asc", + "description": "Created at ascending order", + "isDeprecated": true, + "deprecationReason": "Use CREATED_ASC. Deprecated in 13.5" + }, + { + "name": "UPDATED_DESC", + "description": "Updated at descending order", "isDeprecated": false, "deprecationReason": null }, { - "name": "updated_asc", + "name": "UPDATED_ASC", "description": "Updated at ascending order", "isDeprecated": false, "deprecationReason": null }, { - "name": "created_desc", + "name": "CREATED_DESC", "description": "Created at descending order", "isDeprecated": false, "deprecationReason": null }, { - "name": "created_asc", + "name": "CREATED_ASC", "description": "Created at ascending order", "isDeprecated": false, "deprecationReason": null @@ -31031,6 +32922,33 @@ "deprecationReason": null }, { + "name": "destroyBoardList", + "description": null, + "args": [ + { + "name": "input", + "description": null, + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "INPUT_OBJECT", + "name": "DestroyBoardListInput", + "ofType": null + } + }, + "defaultValue": null + } + ], + "type": { + "kind": "OBJECT", + "name": "DestroyBoardListPayload", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { "name": "destroyNote", "description": null, "args": [ @@ -31135,8 +33053,8 @@ "name": "DismissVulnerabilityPayload", "ofType": null }, - "isDeprecated": false, - "deprecationReason": null + "isDeprecated": true, + "deprecationReason": "Use vulnerabilityDismiss. Deprecated in 13.5" }, { "name": "epicAddIssue", @@ -31949,6 +33867,33 @@ "deprecationReason": null }, { + "name": "revertVulnerabilityToDetected", + "description": null, + "args": [ + { + "name": "input", + "description": null, + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "INPUT_OBJECT", + "name": "RevertVulnerabilityToDetectedInput", + "ofType": null + } + }, + "defaultValue": null + } + ], + "type": { + "kind": "OBJECT", + "name": "RevertVulnerabilityToDetectedPayload", + "ofType": null + }, + "isDeprecated": true, + "deprecationReason": "Use vulnerabilityRevertToDetected. Deprecated in 13.5" + }, + { "name": "runDastScan", "description": null, "args": [ @@ -32408,6 +34353,60 @@ "deprecationReason": null }, { + "name": "vulnerabilityConfirm", + "description": null, + "args": [ + { + "name": "input", + "description": null, + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "INPUT_OBJECT", + "name": "VulnerabilityConfirmInput", + "ofType": null + } + }, + "defaultValue": null + } + ], + "type": { + "kind": "OBJECT", + "name": "VulnerabilityConfirmPayload", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "vulnerabilityDismiss", + "description": null, + "args": [ + { + "name": "input", + "description": null, + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "INPUT_OBJECT", + "name": "VulnerabilityDismissInput", + "ofType": null + } + }, + "defaultValue": null + } + ], + "type": { + "kind": "OBJECT", + "name": "VulnerabilityDismissPayload", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { "name": "vulnerabilityResolve", "description": null, "args": [ @@ -32433,6 +34432,33 @@ }, "isDeprecated": false, "deprecationReason": null + }, + { + "name": "vulnerabilityRevertToDetected", + "description": null, + "args": [ + { + "name": "input", + "description": null, + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "INPUT_OBJECT", + "name": "VulnerabilityRevertToDetectedInput", + "ofType": null + } + }, + "defaultValue": null + } + ], + "type": { + "kind": "OBJECT", + "name": "VulnerabilityRevertToDetectedPayload", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null } ], "inputFields": null, @@ -33768,6 +35794,11 @@ }, { "kind": "OBJECT", + "name": "BoardEpic", + "ofType": null + }, + { + "kind": "OBJECT", "name": "Design", "ofType": null }, @@ -33795,6 +35826,11 @@ "kind": "OBJECT", "name": "Snippet", "ofType": null + }, + { + "kind": "OBJECT", + "name": "Vulnerability", + "ofType": null } ] }, @@ -34030,7 +36066,7 @@ { "kind": "OBJECT", "name": "PackageFileRegistry", - "description": "Represents the sync and verification state of a package file", + "description": "Represents the Geo sync and verification state of a package file", "fields": [ { "name": "createdAt", @@ -34321,6 +36357,12 @@ "description": "Packages from the generic package manager", "isDeprecated": false, "deprecationReason": null + }, + { + "name": "GOLANG", + "description": "Packages from the golang package manager", + "isDeprecated": false, + "deprecationReason": null } ], "possibleTypes": null @@ -35637,11 +37679,15 @@ "args": [ { "name": "id", - "description": "Find a board by its ID", + "description": "The board's ID", "type": { - "kind": "SCALAR", - "name": "ID", - "ofType": null + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "ID", + "ofType": null + } }, "defaultValue": null } @@ -36366,6 +38412,16 @@ "defaultValue": null }, { + "name": "authorUsername", + "description": "Username of the author of the issue", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { "name": "assigneeUsername", "description": "Username of a user assigned to the issue", "type": { @@ -36376,6 +38432,24 @@ "defaultValue": null }, { + "name": "assigneeUsernames", + "description": "Usernames of users assigned to the issue", + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + } + }, + "defaultValue": null + }, + { "name": "assigneeId", "description": "ID of a user assigned to the issues, \"none\" and \"any\" values supported", "type": { @@ -36577,6 +38651,16 @@ "defaultValue": null }, { + "name": "authorUsername", + "description": "Username of the author of the issue", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { "name": "assigneeUsername", "description": "Username of a user assigned to the issue", "type": { @@ -36587,6 +38671,24 @@ "defaultValue": null }, { + "name": "assigneeUsernames", + "description": "Usernames of users assigned to the issue", + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + } + }, + "defaultValue": null + }, + { "name": "assigneeId", "description": "ID of a user assigned to the issues, \"none\" and \"any\" values supported", "type": { @@ -36754,6 +38856,16 @@ "defaultValue": null }, { + "name": "authorUsername", + "description": "Username of the author of the issue", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { "name": "assigneeUsername", "description": "Username of a user assigned to the issue", "type": { @@ -36764,6 +38876,24 @@ "defaultValue": null }, { + "name": "assigneeUsernames", + "description": "Usernames of users assigned to the issue", + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + } + }, + "defaultValue": null + }, + { "name": "assigneeId", "description": "ID of a user assigned to the issues, \"none\" and \"any\" values supported", "type": { @@ -38129,7 +40259,7 @@ }, { "name": "requirement", - "description": "Find a single requirement. Available only when feature flag `requirements_management` is enabled.", + "description": "Find a single requirement", "args": [ { "name": "iid", @@ -38232,7 +40362,7 @@ }, { "name": "requirements", - "description": "Find requirements. Available only when feature flag `requirements_management` is enabled.", + "description": "Find requirements", "args": [ { "name": "iid", @@ -38727,6 +40857,59 @@ "deprecationReason": null }, { + "name": "terraformStates", + "description": "Terraform states associated with the project", + "args": [ + { + "name": "after", + "description": "Returns the elements in the list that come after the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "before", + "description": "Returns the elements in the list that come before the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "first", + "description": "Returns the first _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "last", + "description": "Returns the last _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + } + ], + "type": { + "kind": "OBJECT", + "name": "TerraformStateConnection", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { "name": "userPermissions", "description": "Permissions for the current user on the resource", "args": [ @@ -40982,6 +43165,16 @@ "defaultValue": null }, { + "name": "searchNamespaces", + "description": "Include namespace in project search", + "type": { + "kind": "SCALAR", + "name": "Boolean", + "ofType": null + }, + "defaultValue": null + }, + { "name": "after", "description": "Returns the elements in the list that come after the specified cursor.", "type": { @@ -41617,6 +43810,33 @@ }, "isDeprecated": true, "deprecationReason": "Use `vulnerabilitiesCountByDay`. Deprecated in 13.3" + }, + { + "name": "vulnerability", + "description": "Find a vulnerability", + "args": [ + { + "name": "id", + "description": "The Global ID of the Vulnerability", + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "VulnerabilityID", + "ofType": null + } + }, + "defaultValue": null + } + ], + "type": { + "kind": "OBJECT", + "name": "Vulnerability", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null } ], "inputFields": null, @@ -43218,6 +45438,34 @@ "deprecationReason": null }, { + "name": "description", + "description": "Description of the requirement", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "descriptionHtml", + "description": "The GitLab Flavored Markdown rendering of `description`", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { "name": "id", "description": "ID of the requirement", "args": [ @@ -43381,6 +45629,20 @@ "deprecationReason": null }, { + "name": "titleHtml", + "description": "The GitLab Flavored Markdown rendering of `title`", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { "name": "updatedAt", "description": "Timestamp of when the requirement was last updated", "args": [ @@ -43790,6 +46052,108 @@ ] }, { + "kind": "INPUT_OBJECT", + "name": "RevertVulnerabilityToDetectedInput", + "description": "Autogenerated input type of RevertVulnerabilityToDetected", + "fields": null, + "inputFields": [ + { + "name": "id", + "description": "ID of the vulnerability to be reverted", + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "VulnerabilityID", + "ofType": null + } + }, + "defaultValue": null + }, + { + "name": "clientMutationId", + "description": "A unique identifier for the client performing the mutation.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + } + ], + "interfaces": null, + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "RevertVulnerabilityToDetectedPayload", + "description": "Autogenerated return type of RevertVulnerabilityToDetected", + "fields": [ + { + "name": "clientMutationId", + "description": "A unique identifier for the client performing the mutation.", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "errors", + "description": "Errors encountered during execution of the mutation.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + } + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "vulnerability", + "description": "The vulnerability after revert", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "Vulnerability", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { "kind": "OBJECT", "name": "RootStorageStatistics", "description": null, @@ -44479,6 +46843,63 @@ "possibleTypes": null }, { + "kind": "INPUT_OBJECT", + "name": "SastCiConfigurationAnalyzersEntityInput", + "description": "Represents the analyzers entity in SAST CI configuration", + "fields": null, + "inputFields": [ + { + "name": "name", + "description": "Name of analyzer", + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + }, + "defaultValue": null + }, + { + "name": "enabled", + "description": "State of the analyzer", + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "Boolean", + "ofType": null + } + }, + "defaultValue": null + }, + { + "name": "variables", + "description": "List of variables for the analyzer", + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "INPUT_OBJECT", + "name": "SastCiConfigurationEntityInput", + "ofType": null + } + } + }, + "defaultValue": null + } + ], + "interfaces": null, + "enumValues": null, + "possibleTypes": null + }, + { "kind": "OBJECT", "name": "SastCiConfigurationEntity", "description": "Represents an entity in SAST CI configuration", @@ -44848,6 +47269,24 @@ } }, "defaultValue": null + }, + { + "name": "analyzers", + "description": "List of analyzers and related variables for the SAST configuration", + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "INPUT_OBJECT", + "name": "SastCiConfigurationAnalyzersEntityInput", + "ofType": null + } + } + }, + "defaultValue": null } ], "interfaces": null, @@ -47364,24 +49803,69 @@ "name": "blobs", "description": "Snippet blobs", "args": [ - - ], - "type": { - "kind": "NON_NULL", - "name": null, - "ofType": { - "kind": "LIST", - "name": null, - "ofType": { - "kind": "NON_NULL", + { + "name": "paths", + "description": "Paths of the blobs", + "type": { + "kind": "LIST", "name": null, "ofType": { - "kind": "OBJECT", - "name": "SnippetBlob", - "ofType": null + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } } - } + }, + "defaultValue": null + }, + { + "name": "after", + "description": "Returns the elements in the list that come after the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "before", + "description": "Returns the elements in the list that come before the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "first", + "description": "Returns the first _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "last", + "description": "Returns the last _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null } + ], + "type": { + "kind": "OBJECT", + "name": "SnippetBlobConnection", + "ofType": null }, "isDeprecated": false, "deprecationReason": null @@ -48037,6 +50521,118 @@ }, { "kind": "OBJECT", + "name": "SnippetBlobConnection", + "description": "The connection type for SnippetBlob.", + "fields": [ + { + "name": "edges", + "description": "A list of edges.", + "args": [ + + ], + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "SnippetBlobEdge", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "nodes", + "description": "A list of nodes.", + "args": [ + + ], + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "SnippetBlob", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "pageInfo", + "description": "Information to aid in pagination.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "PageInfo", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "SnippetBlobEdge", + "description": "An edge in a connection.", + "fields": [ + { + "name": "cursor", + "description": "A cursor for use in pagination.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "node", + "description": "The item at the end of the edge.", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "SnippetBlob", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", "name": "SnippetBlobViewer", "description": "Represents how the blob content should be displayed", "fields": [ @@ -48414,23 +51010,47 @@ { "name": "updated_desc", "description": "Updated at descending order", + "isDeprecated": true, + "deprecationReason": "Use UPDATED_DESC. Deprecated in 13.5" + }, + { + "name": "updated_asc", + "description": "Updated at ascending order", + "isDeprecated": true, + "deprecationReason": "Use UPDATED_ASC. Deprecated in 13.5" + }, + { + "name": "created_desc", + "description": "Created at descending order", + "isDeprecated": true, + "deprecationReason": "Use CREATED_DESC. Deprecated in 13.5" + }, + { + "name": "created_asc", + "description": "Created at ascending order", + "isDeprecated": true, + "deprecationReason": "Use CREATED_ASC. Deprecated in 13.5" + }, + { + "name": "UPDATED_DESC", + "description": "Updated at descending order", "isDeprecated": false, "deprecationReason": null }, { - "name": "updated_asc", + "name": "UPDATED_ASC", "description": "Updated at ascending order", "isDeprecated": false, "deprecationReason": null }, { - "name": "created_desc", + "name": "CREATED_DESC", "description": "Created at descending order", "isDeprecated": false, "deprecationReason": null }, { - "name": "created_asc", + "name": "CREATED_ASC", "description": "Created at ascending order", "isDeprecated": false, "deprecationReason": null @@ -48764,12 +51384,237 @@ }, { "kind": "OBJECT", - "name": "TerraformStateRegistry", - "description": "Represents the sync and verification state of a terraform state", + "name": "TerraformState", + "description": null, + "fields": [ + { + "name": "createdAt", + "description": "Timestamp the Terraform state was created", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "id", + "description": "ID of the Terraform state", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "ID", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "lockedAt", + "description": "Timestamp the Terraform state was locked", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "lockedByUser", + "description": "The user currently holding a lock on the Terraform state", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "User", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "name", + "description": "Name of the Terraform state", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "updatedAt", + "description": "Timestamp the Terraform state was updated", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "Time", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "TerraformStateConnection", + "description": "The connection type for TerraformState.", + "fields": [ + { + "name": "edges", + "description": "A list of edges.", + "args": [ + + ], + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "TerraformStateEdge", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "nodes", + "description": "A list of nodes.", + "args": [ + + ], + "type": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "TerraformState", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "pageInfo", + "description": "Information to aid in pagination.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "PageInfo", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "TerraformStateEdge", + "description": "An edge in a connection.", + "fields": [ + { + "name": "cursor", + "description": "A cursor for use in pagination.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "node", + "description": "The item at the end of the edge.", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "TerraformState", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "TerraformStateVersionRegistry", + "description": "Represents the Geo sync and verification state of a terraform state version", "fields": [ { "name": "createdAt", - "description": "Timestamp when the TerraformStateRegistry was created", + "description": "Timestamp when the TerraformStateVersionRegistry was created", "args": [ ], @@ -48783,7 +51628,7 @@ }, { "name": "id", - "description": "ID of the TerraformStateRegistry", + "description": "ID of the TerraformStateVersionRegistry", "args": [ ], @@ -48801,7 +51646,7 @@ }, { "name": "lastSyncFailure", - "description": "Error message during sync of the TerraformStateRegistry", + "description": "Error message during sync of the TerraformStateVersionRegistry", "args": [ ], @@ -48815,7 +51660,7 @@ }, { "name": "lastSyncedAt", - "description": "Timestamp of the most recent successful sync of the TerraformStateRegistry", + "description": "Timestamp of the most recent successful sync of the TerraformStateVersionRegistry", "args": [ ], @@ -48829,7 +51674,7 @@ }, { "name": "retryAt", - "description": "Timestamp after which the TerraformStateRegistry should be resynced", + "description": "Timestamp after which the TerraformStateVersionRegistry should be resynced", "args": [ ], @@ -48843,7 +51688,7 @@ }, { "name": "retryCount", - "description": "Number of consecutive failed sync attempts of the TerraformStateRegistry", + "description": "Number of consecutive failed sync attempts of the TerraformStateVersionRegistry", "args": [ ], @@ -48857,7 +51702,7 @@ }, { "name": "state", - "description": "Sync state of the TerraformStateRegistry", + "description": "Sync state of the TerraformStateVersionRegistry", "args": [ ], @@ -48870,8 +51715,8 @@ "deprecationReason": null }, { - "name": "terraformStateId", - "description": "ID of the TerraformState", + "name": "terraformStateVersionId", + "description": "ID of the terraform state version", "args": [ ], @@ -48897,8 +51742,8 @@ }, { "kind": "OBJECT", - "name": "TerraformStateRegistryConnection", - "description": "The connection type for TerraformStateRegistry.", + "name": "TerraformStateVersionRegistryConnection", + "description": "The connection type for TerraformStateVersionRegistry.", "fields": [ { "name": "edges", @@ -48911,7 +51756,7 @@ "name": null, "ofType": { "kind": "OBJECT", - "name": "TerraformStateRegistryEdge", + "name": "TerraformStateVersionRegistryEdge", "ofType": null } }, @@ -48929,7 +51774,7 @@ "name": null, "ofType": { "kind": "OBJECT", - "name": "TerraformStateRegistry", + "name": "TerraformStateVersionRegistry", "ofType": null } }, @@ -48964,7 +51809,7 @@ }, { "kind": "OBJECT", - "name": "TerraformStateRegistryEdge", + "name": "TerraformStateVersionRegistryEdge", "description": "An edge in a connection.", "fields": [ { @@ -48993,7 +51838,7 @@ ], "type": { "kind": "OBJECT", - "name": "TerraformStateRegistry", + "name": "TerraformStateVersionRegistry", "ofType": null }, "isDeprecated": false, @@ -52552,38 +55397,48 @@ "defaultValue": null }, { - "name": "state", - "description": "State of the requirement", + "name": "description", + "description": "Description of the requirement", "type": { - "kind": "ENUM", - "name": "RequirementState", + "kind": "SCALAR", + "name": "String", "ofType": null }, "defaultValue": null }, { - "name": "iid", - "description": "The iid of the requirement to update", + "name": "projectPath", + "description": "Full project path the requirement is associated with", "type": { "kind": "NON_NULL", "name": null, "ofType": { "kind": "SCALAR", - "name": "String", + "name": "ID", "ofType": null } }, "defaultValue": null }, { - "name": "projectPath", - "description": "The project full path the requirement is associated with", + "name": "state", + "description": "State of the requirement", + "type": { + "kind": "ENUM", + "name": "RequirementState", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "iid", + "description": "The iid of the requirement to update", "type": { "kind": "NON_NULL", "name": null, "ofType": { "kind": "SCALAR", - "name": "ID", + "name": "String", "ofType": null } }, @@ -52661,7 +55516,7 @@ }, { "name": "requirement", - "description": "The requirement after mutation", + "description": "Requirement after mutation", "args": [ ], @@ -54606,6 +57461,63 @@ "deprecationReason": null }, { + "name": "discussions", + "description": "All discussions on this noteable", + "args": [ + { + "name": "after", + "description": "Returns the elements in the list that come after the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "before", + "description": "Returns the elements in the list that come before the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "first", + "description": "Returns the first _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "last", + "description": "Returns the last _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + } + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "DiscussionConnection", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { "name": "id", "description": "GraphQL ID of the vulnerability", "args": [ @@ -54731,6 +57643,63 @@ "deprecationReason": null }, { + "name": "notes", + "description": "All notes on this noteable", + "args": [ + { + "name": "after", + "description": "Returns the elements in the list that come after the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "before", + "description": "Returns the elements in the list that come before the specified cursor.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "first", + "description": "Returns the first _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "last", + "description": "Returns the last _n_ elements from the list.", + "type": { + "kind": "SCALAR", + "name": "Int", + "ofType": null + }, + "defaultValue": null + } + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "OBJECT", + "name": "NoteConnection", + "ofType": null + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { "name": "primaryIdentifier", "description": "Primary identifier of the vulnerability.", "args": [ @@ -54899,6 +57868,112 @@ ], "inputFields": null, "interfaces": [ + { + "kind": "INTERFACE", + "name": "Noteable", + "ofType": null + } + ], + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "INPUT_OBJECT", + "name": "VulnerabilityConfirmInput", + "description": "Autogenerated input type of VulnerabilityConfirm", + "fields": null, + "inputFields": [ + { + "name": "id", + "description": "ID of the vulnerability to be confirmed", + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "VulnerabilityID", + "ofType": null + } + }, + "defaultValue": null + }, + { + "name": "clientMutationId", + "description": "A unique identifier for the client performing the mutation.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + } + ], + "interfaces": null, + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "VulnerabilityConfirmPayload", + "description": "Autogenerated return type of VulnerabilityConfirm", + "fields": [ + { + "name": "clientMutationId", + "description": "A unique identifier for the client performing the mutation.", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "errors", + "description": "Errors encountered during execution of the mutation.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + } + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "vulnerability", + "description": "The vulnerability after state change", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "Vulnerability", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ ], "enumValues": null, @@ -54972,6 +58047,118 @@ "possibleTypes": null }, { + "kind": "INPUT_OBJECT", + "name": "VulnerabilityDismissInput", + "description": "Autogenerated input type of VulnerabilityDismiss", + "fields": null, + "inputFields": [ + { + "name": "id", + "description": "ID of the vulnerability to be dismissed", + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "ID", + "ofType": null + } + }, + "defaultValue": null + }, + { + "name": "comment", + "description": "Reason why vulnerability should be dismissed", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + }, + { + "name": "clientMutationId", + "description": "A unique identifier for the client performing the mutation.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + } + ], + "interfaces": null, + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "VulnerabilityDismissPayload", + "description": "Autogenerated return type of VulnerabilityDismiss", + "fields": [ + { + "name": "clientMutationId", + "description": "A unique identifier for the client performing the mutation.", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "errors", + "description": "Errors encountered during execution of the mutation.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + } + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "vulnerability", + "description": "The vulnerability after dismissal", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "Vulnerability", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { "kind": "OBJECT", "name": "VulnerabilityEdge", "description": "An edge in a connection.", @@ -56005,7 +59192,7 @@ "inputFields": [ { "name": "id", - "description": "ID of the vulnerability to be resolveed", + "description": "ID of the vulnerability to be resolved", "type": { "kind": "NON_NULL", "name": null, @@ -56100,6 +59287,108 @@ "possibleTypes": null }, { + "kind": "INPUT_OBJECT", + "name": "VulnerabilityRevertToDetectedInput", + "description": "Autogenerated input type of VulnerabilityRevertToDetected", + "fields": null, + "inputFields": [ + { + "name": "id", + "description": "ID of the vulnerability to be reverted", + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "VulnerabilityID", + "ofType": null + } + }, + "defaultValue": null + }, + { + "name": "clientMutationId", + "description": "A unique identifier for the client performing the mutation.", + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "defaultValue": null + } + ], + "interfaces": null, + "enumValues": null, + "possibleTypes": null + }, + { + "kind": "OBJECT", + "name": "VulnerabilityRevertToDetectedPayload", + "description": "Autogenerated return type of VulnerabilityRevertToDetected", + "fields": [ + { + "name": "clientMutationId", + "description": "A unique identifier for the client performing the mutation.", + "args": [ + + ], + "type": { + "kind": "SCALAR", + "name": "String", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "errors", + "description": "Errors encountered during execution of the mutation.", + "args": [ + + ], + "type": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "LIST", + "name": null, + "ofType": { + "kind": "NON_NULL", + "name": null, + "ofType": { + "kind": "SCALAR", + "name": "String", + "ofType": null + } + } + } + }, + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "vulnerability", + "description": "The vulnerability after revert", + "args": [ + + ], + "type": { + "kind": "OBJECT", + "name": "Vulnerability", + "ofType": null + }, + "isDeprecated": false, + "deprecationReason": null + } + ], + "inputFields": null, + "interfaces": [ + + ], + "enumValues": null, + "possibleTypes": null + }, + { "kind": "OBJECT", "name": "VulnerabilityScanner", "description": "Represents a vulnerability scanner", @@ -56443,6 +59732,42 @@ "description": "Severity in ascending order", "isDeprecated": false, "deprecationReason": null + }, + { + "name": "title_desc", + "description": "Title in descending order", + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "title_asc", + "description": "Title in ascending order", + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "detected_desc", + "description": "Detection timestamp in descending order", + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "detected_asc", + "description": "Detection timestamp in ascending order", + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "report_type_desc", + "description": "Report Type in descending order", + "isDeprecated": false, + "deprecationReason": null + }, + { + "name": "report_type_asc", + "description": "Report Type in ascending order", + "isDeprecated": false, + "deprecationReason": null } ], "possibleTypes": null diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index fc27298aff2..4a3675df15d 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -209,6 +209,57 @@ Represents a project or group board. | `name` | String | Name of the board | | `weight` | Int | Weight of the board. | +### BoardEpic + +Represents an epic on an issue board. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `author` | User! | Author of the epic | +| `closedAt` | Time | Timestamp of when the epic was closed | +| `confidential` | Boolean | Indicates if the epic is confidential | +| `createdAt` | Time | Timestamp of when the epic was created | +| `descendantCounts` | EpicDescendantCount | Number of open and closed descendant epics and issues | +| `descendantWeightSum` | EpicDescendantWeights | Total weight of open and closed issues in the epic and its descendants | +| `description` | String | Description of the epic | +| `downvotes` | Int! | Number of downvotes the epic has received | +| `dueDate` | Time | Due date of the epic | +| `dueDateFixed` | Time | Fixed due date of the epic | +| `dueDateFromMilestones` | Time | Inherited due date of the epic from milestones | +| `dueDateIsFixed` | Boolean | Indicates if the due date has been manually set | +| `group` | Group! | Group to which the epic belongs | +| `hasChildren` | Boolean! | Indicates if the epic has children | +| `hasIssues` | Boolean! | Indicates if the epic has direct issues | +| `hasParent` | Boolean! | Indicates if the epic has a parent epic | +| `healthStatus` | EpicHealthStatus | Current health status of the epic | +| `id` | ID! | ID of the epic | +| `iid` | ID! | Internal ID of the epic | +| `parent` | Epic | Parent epic of the epic | +| `reference` | String! | Internal reference of the epic. Returned in shortened format by default | +| `relationPath` | String | URI path of the epic-issue relationship | +| `relativePosition` | Int | The relative position of the epic in the epic tree | +| `startDate` | Time | Start date of the epic | +| `startDateFixed` | Time | Fixed start date of the epic | +| `startDateFromMilestones` | Time | Inherited start date of the epic from milestones | +| `startDateIsFixed` | Boolean | Indicates if the start date has been manually set | +| `state` | EpicState! | State of the epic | +| `subscribed` | Boolean! | Indicates the currently logged in user is subscribed to the epic | +| `title` | String | Title of the epic | +| `updatedAt` | Time | Timestamp of when the epic was updated | +| `upvotes` | Int! | Number of upvotes the epic has received | +| `userPermissions` | EpicPermissions! | Permissions for the current user on the resource | +| `userPreferences` | BoardEpicUserPreferences | User preferences for the epic on the issue board | +| `webPath` | String! | Web path of the epic | +| `webUrl` | String! | Web URL of the epic | + +### BoardEpicUserPreferences + +Represents user preferences for a board epic. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `collapsed` | Boolean! | Indicates epic should be displayed as collapsed | + ### BoardList Represents a list for an issue board. @@ -499,7 +550,7 @@ Autogenerated return type of CreateRequirement. | ----- | ---- | ----------- | | `clientMutationId` | String | A unique identifier for the client performing the mutation. | | `errors` | String! => Array | Errors encountered during execution of the mutation. | -| `requirement` | Requirement | The requirement after mutation | +| `requirement` | Requirement | Requirement after mutation | ### CreateSnippetPayload @@ -685,6 +736,7 @@ A collection of designs. | Field | Type | Description | | ----- | ---- | ----------- | +| `copyState` | DesignCollectionCopyState | Copy state of the design collection | | `design` | Design | Find a specific design | | `designAtVersion` | DesignAtVersion | Find a design as of a version | | `issue` | Issue! | Issue associated with the design collection | @@ -739,6 +791,16 @@ A specific version in which designs were added, modified or deleted. | `id` | ID! | ID of the design version | | `sha` | ID! | SHA of the design version | +### DestroyBoardListPayload + +Autogenerated return type of DestroyBoardList. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `clientMutationId` | String | A unique identifier for the client performing the mutation. | +| `errors` | String! => Array | Errors encountered during execution of the mutation. | +| `list` | BoardList | The list after mutation. | + ### DestroyBoardPayload Autogenerated return type of DestroyBoard. @@ -878,9 +940,9 @@ Represents an epic. | Field | Type | Description | | ----- | ---- | ----------- | | `author` | User! | Author of the epic | -| `closedAt` | Time | Timestamp of the epic's closure | +| `closedAt` | Time | Timestamp of when the epic was closed | | `confidential` | Boolean | Indicates if the epic is confidential | -| `createdAt` | Time | Timestamp of the epic's creation | +| `createdAt` | Time | Timestamp of when the epic was created | | `descendantCounts` | EpicDescendantCount | Number of open and closed descendant epics and issues | | `descendantWeightSum` | EpicDescendantWeights | Total weight of open and closed issues in the epic and its descendants | | `description` | String | Description of the epic | @@ -907,7 +969,7 @@ Represents an epic. | `state` | EpicState! | State of the epic | | `subscribed` | Boolean! | Indicates the currently logged in user is subscribed to the epic | | `title` | String | Title of the epic | -| `updatedAt` | Time | Timestamp of the epic's last activity | +| `updatedAt` | Time | Timestamp of when the epic was updated | | `upvotes` | Int! | Number of upvotes the epic has received | | `userPermissions` | EpicPermissions! | Permissions for the current user on the resource | | `webPath` | String! | Web path of the epic | @@ -1486,6 +1548,21 @@ Autogenerated return type of MergeRequestCreate. | `errors` | String! => Array | Errors encountered during execution of the mutation. | | `mergeRequest` | MergeRequest | The merge request after mutation | +### MergeRequestDiffRegistry + +Represents the Geo sync and verification state of a Merge Request diff. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `createdAt` | Time | Timestamp when the MergeRequestDiffRegistry was created | +| `id` | ID! | ID of the MergeRequestDiffRegistry | +| `lastSyncFailure` | String | Error message during sync of the MergeRequestDiffRegistry | +| `lastSyncedAt` | Time | Timestamp of the most recent successful sync of the MergeRequestDiffRegistry | +| `mergeRequestDiffId` | ID! | ID of the Merge Request diff | +| `retryAt` | Time | Timestamp after which the MergeRequestDiffRegistry should be resynced | +| `retryCount` | Int | Number of consecutive failed sync attempts of the MergeRequestDiffRegistry | +| `state` | RegistryState | Sync state of the MergeRequestDiffRegistry | + ### MergeRequestPermissions Check permissions for the current user on a merge request. @@ -1702,7 +1779,7 @@ Represents a package. ### PackageFileRegistry -Represents the sync and verification state of a package file. +Represents the Geo sync and verification state of a package file. | Field | Type | Description | | ----- | ---- | ----------- | @@ -1837,7 +1914,7 @@ Autogenerated return type of PipelineRetry. | `removeSourceBranchAfterMerge` | Boolean | Indicates if `Delete source branch` option should be enabled by default for all new merge requests of the project | | `repository` | Repository | Git repository of the project | | `requestAccessEnabled` | Boolean | Indicates if users can request member access to the project | -| `requirement` | Requirement | Find a single requirement. Available only when feature flag `requirements_management` is enabled. | +| `requirement` | Requirement | Find a single requirement | | `requirementStatesCount` | RequirementStatesCount | Number of requirements for the project by their state | | `sastCiConfiguration` | SastCiConfiguration | SAST CI configuration for the project | | `securityDashboardPath` | String | Path to project's security dashboard | @@ -2049,12 +2126,15 @@ Represents a requirement. | ----- | ---- | ----------- | | `author` | User! | Author of the requirement | | `createdAt` | Time! | Timestamp of when the requirement was created | +| `description` | String | Description of the requirement | +| `descriptionHtml` | String | The GitLab Flavored Markdown rendering of `description` | | `id` | ID! | ID of the requirement | | `iid` | ID! | Internal ID of the requirement | | `lastTestReportState` | TestReportState | Latest requirement test report state | | `project` | Project! | Project to which the requirement belongs | | `state` | RequirementState! | State of the requirement | | `title` | String | Title of the requirement | +| `titleHtml` | String | The GitLab Flavored Markdown rendering of `title` | | `updatedAt` | Time! | Timestamp of when the requirement was last updated | | `userPermissions` | RequirementPermissions! | Permissions for the current user on the resource | @@ -2079,6 +2159,16 @@ Counts of requirements by their state. | `archived` | Int | Number of archived requirements | | `opened` | Int | Number of opened requirements | +### RevertVulnerabilityToDetectedPayload + +Autogenerated return type of RevertVulnerabilityToDetected. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `clientMutationId` | String | A unique identifier for the client performing the mutation. | +| `errors` | String! => Array | Errors encountered during execution of the mutation. | +| `vulnerability` | Vulnerability | The vulnerability after revert | + ### RootStorageStatistics | Field | Type | Description | @@ -2302,7 +2392,6 @@ Represents a snippet entry. | ----- | ---- | ----------- | | `author` | User | The owner of the snippet | | `blob` **{warning-solid}** | SnippetBlob! | **Deprecated:** Use `blobs`. Deprecated in 13.3 | -| `blobs` | SnippetBlob! => Array | Snippet blobs | | `createdAt` | Time! | Timestamp this snippet was created | | `description` | String | Description of the snippet | | `descriptionHtml` | String | The GitLab Flavored Markdown rendering of `description` | @@ -2384,20 +2473,31 @@ Completion status of tasks. | `completedCount` | Int! | Number of completed tasks | | `count` | Int! | Number of total tasks | -### TerraformStateRegistry +### TerraformState + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `createdAt` | Time! | Timestamp the Terraform state was created | +| `id` | ID! | ID of the Terraform state | +| `lockedAt` | Time | Timestamp the Terraform state was locked | +| `lockedByUser` | User | The user currently holding a lock on the Terraform state | +| `name` | String! | Name of the Terraform state | +| `updatedAt` | Time! | Timestamp the Terraform state was updated | + +### TerraformStateVersionRegistry -Represents the sync and verification state of a terraform state. +Represents the Geo sync and verification state of a terraform state version. | Field | Type | Description | | ----- | ---- | ----------- | -| `createdAt` | Time | Timestamp when the TerraformStateRegistry was created | -| `id` | ID! | ID of the TerraformStateRegistry | -| `lastSyncFailure` | String | Error message during sync of the TerraformStateRegistry | -| `lastSyncedAt` | Time | Timestamp of the most recent successful sync of the TerraformStateRegistry | -| `retryAt` | Time | Timestamp after which the TerraformStateRegistry should be resynced | -| `retryCount` | Int | Number of consecutive failed sync attempts of the TerraformStateRegistry | -| `state` | RegistryState | Sync state of the TerraformStateRegistry | -| `terraformStateId` | ID! | ID of the TerraformState | +| `createdAt` | Time | Timestamp when the TerraformStateVersionRegistry was created | +| `id` | ID! | ID of the TerraformStateVersionRegistry | +| `lastSyncFailure` | String | Error message during sync of the TerraformStateVersionRegistry | +| `lastSyncedAt` | Time | Timestamp of the most recent successful sync of the TerraformStateVersionRegistry | +| `retryAt` | Time | Timestamp after which the TerraformStateVersionRegistry should be resynced | +| `retryCount` | Int | Number of consecutive failed sync attempts of the TerraformStateVersionRegistry | +| `state` | RegistryState | Sync state of the TerraformStateVersionRegistry | +| `terraformStateVersionId` | ID! | ID of the terraform state version | ### TestReport @@ -2611,7 +2711,7 @@ Autogenerated return type of UpdateRequirement. | ----- | ---- | ----------- | | `clientMutationId` | String | A unique identifier for the client performing the mutation. | | `errors` | String! => Array | Errors encountered during execution of the mutation. | -| `requirement` | Requirement | The requirement after mutation | +| `requirement` | Requirement | Requirement after mutation | ### UpdateSnippetPayload @@ -2700,6 +2800,26 @@ Represents a vulnerability. | `userPermissions` | VulnerabilityPermissions! | Permissions for the current user on the resource | | `vulnerabilityPath` | String | URL to the vulnerability's details page | +### VulnerabilityConfirmPayload + +Autogenerated return type of VulnerabilityConfirm. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `clientMutationId` | String | A unique identifier for the client performing the mutation. | +| `errors` | String! => Array | Errors encountered during execution of the mutation. | +| `vulnerability` | Vulnerability | The vulnerability after state change | + +### VulnerabilityDismissPayload + +Autogenerated return type of VulnerabilityDismiss. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `clientMutationId` | String | A unique identifier for the client performing the mutation. | +| `errors` | String! => Array | Errors encountered during execution of the mutation. | +| `vulnerability` | Vulnerability | The vulnerability after dismissal | + ### VulnerabilityIdentifier Represents a vulnerability identifier. @@ -2812,6 +2932,16 @@ Autogenerated return type of VulnerabilityResolve. | `errors` | String! => Array | Errors encountered during execution of the mutation. | | `vulnerability` | Vulnerability | The vulnerability after state change | +### VulnerabilityRevertToDetectedPayload + +Autogenerated return type of VulnerabilityRevertToDetected. + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `clientMutationId` | String | A unique identifier for the client performing the mutation. | +| `errors` | String! => Array | Errors encountered during execution of the mutation. | +| `vulnerability` | Vulnerability | The vulnerability after revert | + ### VulnerabilityScanner Represents a vulnerability scanner. @@ -2890,6 +3020,8 @@ Values for sorting alerts. | Value | Description | | ----- | ----------- | +| `CREATED_ASC` | Created at ascending order | +| `CREATED_DESC` | Created at descending order | | `CREATED_TIME_ASC` | Created time by ascending order | | `CREATED_TIME_DESC` | Created time by descending order | | `ENDED_AT_ASC` | End time by ascending order | @@ -2902,12 +3034,14 @@ Values for sorting alerts. | `STARTED_AT_DESC` | Start time by descending order | | `STATUS_ASC` | Status by order: Ignored > Resolved > Acknowledged > Triggered | | `STATUS_DESC` | Status by order: Triggered > Acknowledged > Resolved > Ignored | +| `UPDATED_ASC` | Updated at ascending order | +| `UPDATED_DESC` | Updated at descending order | | `UPDATED_TIME_ASC` | Created time by ascending order | | `UPDATED_TIME_DESC` | Created time by descending order | -| `created_asc` | Created at ascending order | -| `created_desc` | Created at descending order | -| `updated_asc` | Updated at ascending order | -| `updated_desc` | Updated at descending order | +| `created_asc` **{warning-solid}** | **Deprecated:** Use CREATED_ASC. Deprecated in 13.5 | +| `created_desc` **{warning-solid}** | **Deprecated:** Use CREATED_DESC. Deprecated in 13.5 | +| `updated_asc` **{warning-solid}** | **Deprecated:** Use UPDATED_ASC. Deprecated in 13.5 | +| `updated_desc` **{warning-solid}** | **Deprecated:** Use UPDATED_DESC. Deprecated in 13.5 | ### AlertManagementSeverity @@ -3007,6 +3141,16 @@ Mode of a commit action. | `PASSED_VALIDATION` | Site validation process finished successfully | | `PENDING_VALIDATION` | Site validation process has not started | +### DesignCollectionCopyState + +Copy state of a DesignCollection. + +| Value | Description | +| ----- | ----------- | +| `ERROR` | The DesignCollection encountered an error during a copy | +| `IN_PROGRESS` | The DesignCollection is being copied | +| `READY` | The DesignCollection has no copy in progress | + ### DesignVersionEvent Mutation event of a design within a version. @@ -3115,6 +3259,8 @@ Values for sorting issues. | Value | Description | | ----- | ----------- | +| `CREATED_ASC` | Created at ascending order | +| `CREATED_DESC` | Created at descending order | | `DUE_DATE_ASC` | Due date by ascending order | | `DUE_DATE_DESC` | Due date by descending order | | `LABEL_PRIORITY_ASC` | Label priority by ascending order | @@ -3123,13 +3269,19 @@ Values for sorting issues. | `MILESTONE_DUE_DESC` | Milestone due date by descending order | | `PRIORITY_ASC` | Priority by ascending order | | `PRIORITY_DESC` | Priority by descending order | +| `PUBLISHED_ASC` | Published issues shown last | +| `PUBLISHED_DESC` | Published issues shown first | | `RELATIVE_POSITION_ASC` | Relative position by ascending order | +| `SEVERITY_ASC` | Severity from less critical to more critical | +| `SEVERITY_DESC` | Severity from more critical to less critical | +| `UPDATED_ASC` | Updated at ascending order | +| `UPDATED_DESC` | Updated at descending order | | `WEIGHT_ASC` | Weight by ascending order | | `WEIGHT_DESC` | Weight by descending order | -| `created_asc` | Created at ascending order | -| `created_desc` | Created at descending order | -| `updated_asc` | Updated at ascending order | -| `updated_desc` | Updated at descending order | +| `created_asc` **{warning-solid}** | **Deprecated:** Use CREATED_ASC. Deprecated in 13.5 | +| `created_desc` **{warning-solid}** | **Deprecated:** Use CREATED_DESC. Deprecated in 13.5 | +| `updated_asc` **{warning-solid}** | **Deprecated:** Use UPDATED_ASC. Deprecated in 13.5 | +| `updated_desc` **{warning-solid}** | **Deprecated:** Use UPDATED_DESC. Deprecated in 13.5 | ### IssueState @@ -3184,6 +3336,10 @@ Possible identifier types for a measurement. | `ISSUES` | Issue count | | `MERGE_REQUESTS` | Merge request count | | `PIPELINES` | Pipeline count | +| `PIPELINES_CANCELED` | Pipeline count with canceled status | +| `PIPELINES_FAILED` | Pipeline count with failed status | +| `PIPELINES_SKIPPED` | Pipeline count with skipped status | +| `PIPELINES_SUCCEEDED` | Pipeline count with success status | | `PROJECTS` | Project count | | `USERS` | User count | @@ -3193,6 +3349,8 @@ Values for sorting merge requests. | Value | Description | | ----- | ----------- | +| `CREATED_ASC` | Created at ascending order | +| `CREATED_DESC` | Created at descending order | | `LABEL_PRIORITY_ASC` | Label priority by ascending order | | `LABEL_PRIORITY_DESC` | Label priority by descending order | | `MERGED_AT_ASC` | Merge time by ascending order | @@ -3201,10 +3359,12 @@ Values for sorting merge requests. | `MILESTONE_DUE_DESC` | Milestone due date by descending order | | `PRIORITY_ASC` | Priority by ascending order | | `PRIORITY_DESC` | Priority by descending order | -| `created_asc` | Created at ascending order | -| `created_desc` | Created at descending order | -| `updated_asc` | Updated at ascending order | -| `updated_desc` | Updated at descending order | +| `UPDATED_ASC` | Updated at ascending order | +| `UPDATED_DESC` | Updated at descending order | +| `created_asc` **{warning-solid}** | **Deprecated:** Use CREATED_ASC. Deprecated in 13.5 | +| `created_desc` **{warning-solid}** | **Deprecated:** Use CREATED_DESC. Deprecated in 13.5 | +| `updated_asc` **{warning-solid}** | **Deprecated:** Use UPDATED_ASC. Deprecated in 13.5 | +| `updated_desc` **{warning-solid}** | **Deprecated:** Use UPDATED_DESC. Deprecated in 13.5 | ### MergeRequestState @@ -3259,6 +3419,7 @@ Values for sorting projects. | `COMPOSER` | Packages from the composer package manager | | `CONAN` | Packages from the conan package manager | | `GENERIC` | Packages from the generic package manager | +| `GOLANG` | Packages from the golang package manager | | `MAVEN` | Packages from the maven package manager | | `NPM` | Packages from the npm package manager | | `NUGET` | Packages from the nuget package manager | @@ -3428,10 +3589,14 @@ Common sort values. | Value | Description | | ----- | ----------- | -| `created_asc` | Created at ascending order | -| `created_desc` | Created at descending order | -| `updated_asc` | Updated at ascending order | -| `updated_desc` | Updated at descending order | +| `CREATED_ASC` | Created at ascending order | +| `CREATED_DESC` | Created at descending order | +| `UPDATED_ASC` | Updated at ascending order | +| `UPDATED_DESC` | Updated at descending order | +| `created_asc` **{warning-solid}** | **Deprecated:** Use CREATED_ASC. Deprecated in 13.5 | +| `created_desc` **{warning-solid}** | **Deprecated:** Use CREATED_DESC. Deprecated in 13.5 | +| `updated_asc` **{warning-solid}** | **Deprecated:** Use UPDATED_ASC. Deprecated in 13.5 | +| `updated_desc` **{warning-solid}** | **Deprecated:** Use UPDATED_DESC. Deprecated in 13.5 | ### TestReportState @@ -3558,8 +3723,14 @@ Vulnerability sort values. | Value | Description | | ----- | ----------- | +| `detected_asc` | Detection timestamp in ascending order | +| `detected_desc` | Detection timestamp in descending order | +| `report_type_asc` | Report Type in ascending order | +| `report_type_desc` | Report Type in descending order | | `severity_asc` | Severity in ascending order | | `severity_desc` | Severity in descending order | +| `title_asc` | Title in ascending order | +| `title_desc` | Title in descending order | ### VulnerabilityState diff --git a/doc/api/groups.md b/doc/api/groups.md index ae3300e24fb..a74bb64816a 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -167,6 +167,89 @@ GET /groups/:id/subgroups ] ``` +## List a group's descendant groups + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217115) in GitLab 13.5 + +Get a list of visible descendant groups of this group. +When accessed without authentication, only public groups are returned. + +By default, this request returns 20 results at a time because the API results [are paginated](README.md#pagination). + +Parameters: + +| Attribute | Type | Required | Description | +| ------------------------ | ----------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) of the immediate parent group | +| `skip_groups` | array of integers | no | Skip the group IDs passed | +| `all_available` | boolean | no | Show all the groups you have access to (defaults to `false` for authenticated users, `true` for admin). Attributes `owned` and `min_access_level` have precedence | +| `search` | string | no | Return the list of authorized groups matching the search criteria | +| `order_by` | string | no | Order groups by `name`, `path`, or `id`. Default is `name` | +| `sort` | string | no | Order groups in `asc` or `desc` order. Default is `asc` | +| `statistics` | boolean | no | Include group statistics (admins only) | +| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (admins only) | +| `owned` | boolean | no | Limit to groups explicitly owned by the current user | +| `min_access_level` | integer | no | Limit to groups where current user has at least this [access level](members.md#valid-access-levels) | + +```plaintext +GET /groups/:id/descendant_groups +``` + +```json +[ + { + "id": 2, + "name": "Bar Group", + "path": "foo/bar", + "description": "A subgroup of Foo Group", + "visibility": "public", + "share_with_group_lock": false, + "require_two_factor_authentication": false, + "two_factor_grace_period": 48, + "project_creation_level": "developer", + "auto_devops_enabled": null, + "subgroup_creation_level": "owner", + "emails_disabled": null, + "mentions_disabled": null, + "lfs_enabled": true, + "default_branch_protection": 2, + "avatar_url": "http://gitlab.example.com/uploads/group/avatar/1/bar.jpg", + "web_url": "http://gitlab.example.com/groups/foo/bar", + "request_access_enabled": false, + "full_name": "Bar Group", + "full_path": "foo/bar", + "file_template_project_id": 1, + "parent_id": 123, + "created_at": "2020-01-15T12:36:29.590Z" + }, + { + "id": 3, + "name": "Baz Group", + "path": "foo/bar/baz", + "description": "A subgroup of Bar Group", + "visibility": "public", + "share_with_group_lock": false, + "require_two_factor_authentication": false, + "two_factor_grace_period": 48, + "project_creation_level": "developer", + "auto_devops_enabled": null, + "subgroup_creation_level": "owner", + "emails_disabled": null, + "mentions_disabled": null, + "lfs_enabled": true, + "default_branch_protection": 2, + "avatar_url": "http://gitlab.example.com/uploads/group/avatar/1/baz.jpg", + "web_url": "http://gitlab.example.com/groups/foo/bar/baz", + "request_access_enabled": false, + "full_name": "Baz Group", + "full_path": "foo/bar/baz", + "file_template_project_id": 1, + "parent_id": 123, + "created_at": "2020-01-15T12:36:29.590Z" + } +] +``` + ## List a group's projects Get a list of projects in this group. When accessed without authentication, only public projects are returned. @@ -696,7 +779,7 @@ This is similar to creating a [New group](#new-group). You'll need the `parent_i ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"path": "<subgroup_path>", "name": "<subgroup_name>", "parent_id": <parent_group_id> } \ + --data '{"path": "<subgroup_path>", "name": "<subgroup_name>", "parent_id": <parent_group_id> }' \ "https://gitlab.example.com/api/v4/groups/" ``` diff --git a/doc/api/issue_links.md b/doc/api/issue_links.md index b6502bf099c..757910d0946 100644 --- a/doc/api/issue_links.md +++ b/doc/api/issue_links.md @@ -1,8 +1,11 @@ -# Issue links API **(STARTER)** +# Issue links API **(CORE)** + +> The simple "relates to" relationship [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212329) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.4. ## List issue relations -Get a list of related issues of a given issue, sorted by the relationship creation datetime (ascending). +Get a list of a given issue's [related issues](../user/project/issues/related_issues.md), +sorted by the relationship creation datetime (ascending). Issues will be filtered according to the user authorizations. ```plaintext @@ -55,19 +58,20 @@ Parameters: ## Create an issue link -Creates a two-way relation between two issues. User must be allowed to update both issues in order to succeed. +Creates a two-way relation between two issues. The user must be allowed to +update both issues to succeed. ```plaintext POST /projects/:id/issues/:issue_iid/links ``` -| Attribute | Type | Required | Description | -|-------------|---------|----------|--------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `issue_iid` | integer | yes | The internal ID of a project's issue | +| Attribute | Type | Required | Description | +|---------------------|----------------|----------|--------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `issue_iid` | integer | yes | The internal ID of a project's issue | | `target_project_id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) of a target project | -| `target_issue_iid` | integer/string | yes | The internal ID of a target project's issue | -| `link_type` | string | no | The type of the relation ("relates_to", "blocks", "is_blocked_by"), defaults to "relates_to"). | +| `target_issue_iid` | integer/string | yes | The internal ID of a target project's issue | +| `link_type` | string | no | The type of the relation ("relates_to", "blocks", "is_blocked_by"), defaults to "relates_to"). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/4/issues/1/links?target_project_id=5&target_issue_iid=1" diff --git a/doc/api/issues.md b/doc/api/issues.md index d8249869cab..b50ea7b42be 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -16,7 +16,7 @@ are paginated. Read more on [pagination](README.md#pagination). -CAUTION: **Deprecation:** +DANGER: **Deprecated:** The `reference` attribute in responses is deprecated in favor of `references`. Introduced in [GitLab 12.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20354). @@ -33,19 +33,19 @@ use parameter `scope=all`. ```plaintext GET /issues -GET /issues?state=opened -GET /issues?state=closed +GET /issues?assignee_id=5 +GET /issues?author_id=5 +GET /issues?confidential=true +GET /issues?iids[]=42&iids[]=43 GET /issues?labels=foo GET /issues?labels=foo,bar GET /issues?labels=foo,bar&state=opened GET /issues?milestone=1.0.0 GET /issues?milestone=1.0.0&state=opened -GET /issues?iids[]=42&iids[]=43 -GET /issues?author_id=5 -GET /issues?assignee_id=5 GET /issues?my_reaction_emoji=star GET /issues?search=foo&in=title -GET /issues?confidential=true +GET /issues?state=closed +GET /issues?state=opened ``` | Attribute | Type | Required | Description | @@ -194,7 +194,7 @@ the `health_status` parameter: ] ``` -NOTE: **Note:** +DANGER: **Deprecated:** The `assignee` column is deprecated. We now show it as a single-sized array `assignees` to conform to the GitLab EE API. @@ -212,19 +212,19 @@ The preferred way to do this, is by using [personal access tokens](../user/profi ```plaintext GET /groups/:id/issues -GET /groups/:id/issues?state=opened -GET /groups/:id/issues?state=closed +GET /groups/:id/issues?assignee_id=5 +GET /groups/:id/issues?author_id=5 +GET /groups/:id/issues?confidential=true +GET /groups/:id/issues?iids[]=42&iids[]=43 GET /groups/:id/issues?labels=foo GET /groups/:id/issues?labels=foo,bar GET /groups/:id/issues?labels=foo,bar&state=opened GET /groups/:id/issues?milestone=1.0.0 GET /groups/:id/issues?milestone=1.0.0&state=opened -GET /groups/:id/issues?iids[]=42&iids[]=43 -GET /groups/:id/issues?search=issue+title+or+description -GET /groups/:id/issues?author_id=5 -GET /groups/:id/issues?assignee_id=5 GET /groups/:id/issues?my_reaction_emoji=star -GET /groups/:id/issues?confidential=true +GET /groups/:id/issues?search=issue+title+or+description +GET /groups/:id/issues?state=closed +GET /groups/:id/issues?state=opened ``` | Attribute | Type | Required | Description | @@ -372,7 +372,7 @@ the `health_status` parameter: ] ``` -NOTE: **Note:** +DANGER: **Deprecated:** The `assignee` column is deprecated. We now show it as a single-sized array `assignees` to conform to the GitLab EE API. NOTE: **Note:** @@ -389,19 +389,19 @@ The preferred way to do this, is by using [personal access tokens](../user/profi ```plaintext GET /projects/:id/issues -GET /projects/:id/issues?state=opened -GET /projects/:id/issues?state=closed +GET /projects/:id/issues?assignee_id=5 +GET /projects/:id/issues?author_id=5 +GET /projects/:id/issues?confidential=true +GET /projects/:id/issues?iids[]=42&iids[]=43 GET /projects/:id/issues?labels=foo GET /projects/:id/issues?labels=foo,bar GET /projects/:id/issues?labels=foo,bar&state=opened GET /projects/:id/issues?milestone=1.0.0 GET /projects/:id/issues?milestone=1.0.0&state=opened -GET /projects/:id/issues?iids[]=42&iids[]=43 -GET /projects/:id/issues?search=issue+title+or+description -GET /projects/:id/issues?author_id=5 -GET /projects/:id/issues?assignee_id=5 GET /projects/:id/issues?my_reaction_emoji=star -GET /projects/:id/issues?confidential=true +GET /projects/:id/issues?search=issue+title+or+description +GET /projects/:id/issues?state=closed +GET /projects/:id/issues?state=opened ``` | Attribute | Type | Required | Description | @@ -555,7 +555,7 @@ the `health_status` parameter: ] ``` -NOTE: **Note:** +DANGER: **Deprecated:** The `assignee` column is deprecated. We now show it as a single-sized array `assignees` to conform to the GitLab EE API. NOTE: **Note:** @@ -574,7 +574,7 @@ GET /issues/:id | Attribute | Type | Required | Description | |-------------|---------|----------|--------------------------------------| -| `id` | integer | yes | The ID of the issue | +| `id` | integer | yes | The ID of the issue | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/issues/41" @@ -663,10 +663,10 @@ Example response: "weight": null, "has_tasks": false, "_links": { - "self": "http://gitlab.dummy:3000/api/v4/projects/1/issues/1", - "notes": "http://gitlab.dummy:3000/api/v4/projects/1/issues/1/notes", - "award_emoji": "http://gitlab.dummy:3000/api/v4/projects/1/issues/1/award_emoji", - "project": "http://gitlab.dummy:3000/api/v4/projects/1" + "self": "http://gitlab.example:3000/api/v4/projects/1/issues/1", + "notes": "http://gitlab.example:3000/api/v4/projects/1/issues/1/notes", + "award_emoji": "http://gitlab.example:3000/api/v4/projects/1/issues/1/award_emoji", + "project": "http://gitlab.example:3000/api/v4/projects/1" }, "references": { "short": "#1", @@ -712,19 +712,19 @@ the `epic` property: } ``` -NOTE: **Note:** +DANGER: **Deprecated:** The `assignee` column is deprecated. We now show it as a single-sized array `assignees` to conform to the GitLab EE API. +DANGER: **Deprecated:** +The `epic_iid` attribute is deprecated, and [will be removed in version 5](https://gitlab.com/gitlab-org/gitlab/-/issues/35157). +Please use `iid` of the `epic` attribute instead. + NOTE: **Note:** The `closed_by` attribute was [introduced in GitLab 10.6](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17042). This value is only present for issues closed after GitLab 10.6 and if the user account that closed the issue still exists. -NOTE: **Note:** -The `epic_iid` attribute is deprecated, and [will be removed in version 5](https://gitlab.com/gitlab-org/gitlab/-/issues/35157). -Please use `iid` of the `epic` attribute instead. - ## Single project issue Get a single project issue. @@ -874,17 +874,17 @@ property: ] ``` -NOTE: **Note:** +DANGER: **Deprecated:** The `assignee` column is deprecated. We now show it as a single-sized array `assignees` to conform to the GitLab EE API. +DANGER: **Deprecated:** +The `epic_iid` attribute is deprecated and [will be removed in version 5](https://gitlab.com/gitlab-org/gitlab/-/issues/35157). +Please use `iid` of the `epic` attribute instead. + NOTE: **Note:** The `closed_by` attribute was [introduced in GitLab 10.6](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17042). This value is only present for issues closed after GitLab 10.6 and if the user account that closed the issue still exists. -NOTE: **Note:** -The `epic_iid` attribute is deprecated and [will be removed in version 5](https://gitlab.com/gitlab-org/gitlab/-/issues/35157). -Please use `iid` of the `epic` attribute instead. - ## New issue Creates a new project issue. @@ -895,21 +895,21 @@ POST /projects/:id/issues | Attribute | Type | Required | Description | |-------------------------------------------|----------------|----------|--------------| +| `assignee_ids` | integer array | no | The ID of the user(s) to assign the issue to. | +| `confidential` | boolean | no | Set an issue to be confidential. Default is `false`. | +| `created_at` | string | no | When the issue was created. Date time string, ISO 8601 formatted, for example `2016-03-11T03:45:40Z`. Requires administrator or project/group owner rights. | +| `description` | string | no | The description of an issue. Limited to 1,048,576 characters. | +| `discussion_to_resolve` | string | no | The ID of a discussion to resolve. This fills out the issue with a default description and mark the discussion as resolved. Use in combination with `merge_request_to_resolve_discussions_of`. | +| `due_date` | string | no | The due date. Date time string in the format YEAR-MONTH-DAY, for example `2016-03-11` | +| `epic_id` **(PREMIUM)** | integer | no | ID of the epic to add the issue to. Valid values are greater than or equal to 0. | +| `epic_iid` **(PREMIUM)** | integer | no | IID of the epic to add the issue to. Valid values are greater than or equal to 0. (deprecated, [will be removed in version 5](https://gitlab.com/gitlab-org/gitlab/-/issues/35157)) | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `iid` | integer/string | no | The internal ID of the project's issue (requires administrator or project owner rights) | -| `title` | string | yes | The title of an issue | -| `description` | string | no | The description of an issue. Limited to 1,048,576 characters. | -| `confidential` | boolean | no | Set an issue to be confidential. Default is `false`. | -| `assignee_ids` | integer array | no | The ID of the user(s) to assign the issue to. | -| `milestone_id` | integer | no | The global ID of a milestone to assign issue | | `labels` | string | no | Comma-separated label names for an issue | -| `created_at` | string | no | Date time string, ISO 8601 formatted, for example `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | -| `due_date` | string | no | Date time string in the format YEAR-MONTH-DAY, for example `2016-03-11` | | `merge_request_to_resolve_discussions_of` | integer | no | The IID of a merge request in which to resolve all issues. This fills out the issue with a default description and mark all discussions as resolved. When passing a description or title, these values take precedence over the default values.| -| `discussion_to_resolve` | string | no | The ID of a discussion to resolve. This fills out the issue with a default description and mark the discussion as resolved. Use in combination with `merge_request_to_resolve_discussions_of`. | +| `milestone_id` | integer | no | The global ID of a milestone to assign issue | +| `title` | string | yes | The title of an issue | | `weight` **(STARTER)** | integer | no | The weight of the issue. Valid values are greater than or equal to 0. | -| `epic_id` **(PREMIUM)** | integer | no | ID of the epic to add the issue to. Valid values are greater than or equal to 0. | -| `epic_iid` **(PREMIUM)** | integer | no | IID of the epic to add the issue to. Valid values are greater than or equal to 0. (deprecated, [will be removed in version 5](https://gitlab.com/gitlab-org/gitlab/-/issues/35157)) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/4/issues?title=Issues%20with%20auth&labels=bug" @@ -1002,7 +1002,7 @@ the `health_status` parameter: ] ``` -NOTE: **Note:** +DANGER: **Deprecated:** The `assignee` column is deprecated. We now show it as a single-sized array `assignees` to conform to the GitLab EE API. NOTE: **Note:** @@ -1019,29 +1019,43 @@ See [Issues rate limits](../user/admin_area/settings/rate_limit_on_issues_creati Updates an existing project issue. This call is also used to mark an issue as closed. +At least one of the following parameters is required for the request to be successful: + +- `:assignee_id` +- `:assignee_ids` +- `:confidential` +- `:created_at` +- `:description` +- `:discussion_locked` +- `:due_date` +- `:labels` +- `:milestone_id` +- `:state_event` +- `:title` + ```plaintext PUT /projects/:id/issues/:issue_iid ``` | Attribute | Type | Required | Description | |----------------|---------|----------|------------------------------------------------------------------------------------------------------------| +| `add_labels` | string | no | Comma-separated label names to add to an issue. | +| `assignee_ids` | integer array | no | The ID of the user(s) to assign the issue to. Set to `0` or provide an empty value to unassign all assignees. | +| `confidential` | boolean | no | Updates an issue to be confidential | +| `description` | string | no | The description of an issue. Limited to 1,048,576 characters. | +| `discussion_locked` | boolean | no | Flag indicating if the issue's discussion is locked. If the discussion is locked only project members can add or edit comments. | +| `due_date` | string | no | The due date. Date time string in the format YEAR-MONTH-DAY, for example `2016-03-11` | +| `epic_id` **(PREMIUM)** | integer | no | ID of the epic to add the issue to. Valid values are greater than or equal to 0. | +| `epic_iid` **(PREMIUM)** | integer | no | IID of the epic to add the issue to. Valid values are greater than or equal to 0. (deprecated, [will be removed in version 5](https://gitlab.com/gitlab-org/gitlab/-/issues/35157)) | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `issue_iid` | integer | yes | The internal ID of a project's issue | -| `title` | string | no | The title of an issue | -| `description` | string | no | The description of an issue. Limited to 1,048,576 characters. | -| `confidential` | boolean | no | Updates an issue to be confidential | -| `assignee_ids` | integer array | no | The ID of the user(s) to assign the issue to. Set to `0` or provide an empty value to unassign all assignees. | -| `milestone_id` | integer | no | The global ID of a milestone to assign the issue to. Set to `0` or provide an empty value to unassign a milestone.| | `labels` | string | no | Comma-separated label names for an issue. Set to an empty string to unassign all labels. | -| `add_labels` | string | no | Comma-separated label names to add to an issue. | +| `milestone_id` | integer | no | The global ID of a milestone to assign the issue to. Set to `0` or provide an empty value to unassign a milestone.| | `remove_labels`| string | no | Comma-separated label names to remove from an issue. | | `state_event` | string | no | The state event of an issue. Set `close` to close the issue and `reopen` to reopen it | -| `updated_at` | string | no | Date time string, ISO 8601 formatted, for example `2016-03-11T03:45:40Z` (requires administrator or project owner rights). Empty string or null values are not accepted.| -| `due_date` | string | no | Date time string in the format YEAR-MONTH-DAY, for example `2016-03-11` | +| `title` | string | no | The title of an issue | +| `updated_at` | string | no | When the issue was updated. Date time string, ISO 8601 formatted, for example `2016-03-11T03:45:40Z` (requires administrator or project owner rights). Empty string or null values are not accepted.| | `weight` **(STARTER)** | integer | no | The weight of the issue. Valid values are greater than or equal to 0. 0 | -| `discussion_locked` | boolean | no | Flag indicating if the issue's discussion is locked. If the discussion is locked only project members can add or edit comments. | -| `epic_id` **(PREMIUM)** | integer | no | ID of the epic to add the issue to. Valid values are greater than or equal to 0. | -| `epic_iid` **(PREMIUM)** | integer | no | IID of the epic to add the issue to. Valid values are greater than or equal to 0. (deprecated, [will be removed in version 5](https://gitlab.com/gitlab-org/gitlab/-/issues/35157)) | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/4/issues/85?state_event=close" @@ -1142,15 +1156,12 @@ the `health_status` parameter: ``` NOTE: **Note:** -At least one of following parameters is required to be passed for the request to be successful: `:assignee_id`, `:assignee_ids`, `:confidential`, `:created_at`, `:description`, `:discussion_locked`, `:due_date`, `:labels`, `:milestone_id`, `:state_event`, or `:title`. - -NOTE: **Note:** -`assignee` column is deprecated. We now show it as a single-sized array `assignees` to conform to the GitLab EE API. - -NOTE: **Note:** The `closed_by` attribute was [introduced in GitLab 10.6](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17042). This value is only present for issues closed after GitLab 10.6 and if the user account that closed the issue still exists. +DANGER: **Deprecated:** +`assignee` column is deprecated. We now show it as a single-sized array `assignees` to conform to the GitLab EE API. + ## Delete an issue Only for admins and project owners. Deletes the issue in question. @@ -1309,7 +1320,7 @@ the `health_status` parameter: ] ``` -NOTE: **Note:** +DANGER: **Deprecated:** The `assignee` column is deprecated. We now show it as a single-sized array `assignees` to conform to the GitLab EE API. NOTE: **Note:** @@ -1418,7 +1429,7 @@ the `weight` parameter: } ``` -NOTE: **Note:** +DANGER: **Deprecated:** The `assignee` column is deprecated. We now show it as a single-sized array `assignees` to conform to the GitLab EE API. NOTE: **Note:** @@ -1496,10 +1507,10 @@ Example response: } ``` -## Create a to-do +## Create a to do -Manually creates a to-do for the current user on an issue. If -there already exists a to-do for the user on that issue, status code `304` is +Manually creates a to do for the current user on an issue. If +there already exists a to do for the user on that issue, status code `304` is returned. ```plaintext @@ -1608,7 +1619,7 @@ Example response: } ``` -NOTE: **Note:** +DANGER: **Deprecated:** The `assignee` column is deprecated. We now show it as a single-sized array `assignees` to conform to the GitLab EE API. NOTE: **Note:** @@ -1625,9 +1636,9 @@ POST /projects/:id/issues/:issue_iid/time_estimate | Attribute | Type | Required | Description | |-------------|---------|----------|------------------------------------------| +| `duration` | string | yes | The duration in human format. e.g: 3h30m | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `issue_iid` | integer | yes | The internal ID of a project's issue | -| `duration` | string | yes | The duration in human format. e.g: 3h30m | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/issues/93/time_estimate?duration=3h30m" @@ -1682,9 +1693,9 @@ POST /projects/:id/issues/:issue_iid/add_spent_time | Attribute | Type | Required | Description | |-------------|---------|----------|------------------------------------------| +| `duration` | string | yes | The duration in human format. e.g: 3h30m | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `issue_iid` | integer | yes | The internal ID of a project's issue | -| `duration` | string | yes | The duration in human format. e.g: 3h30m | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/issues/93/add_spent_time?duration=1h" diff --git a/doc/api/license.md b/doc/api/license.md index 71e95fc3202..dcdf019059b 100644 --- a/doc/api/license.md +++ b/doc/api/license.md @@ -1,7 +1,7 @@ # License **(CORE ONLY)** -In order to interact with license endpoints, you need to authenticate yourself -as an admin. +To interact with license endpoints, you need to authenticate yourself as an +admin. ## Retrieve information about the current license diff --git a/doc/api/lint.md b/doc/api/lint.md index f4d8a0bc011..df7198ac5c6 100644 --- a/doc/api/lint.md +++ b/doc/api/lint.md @@ -4,11 +4,14 @@ group: Continuous Integration info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Validate the `.gitlab-ci.yml` (API) +# CI Lint API + +## Validate the CI YAML config > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5953) in GitLab 8.12. -Checks if your `.gitlab-ci.yml` file is valid. +Checks if CI/CD YAML configuration is valid. This endpoint validates basic CI/CD +configuration syntax. It doesn't have any namespace specific context. ```plaintext POST /ci/lint @@ -16,13 +19,15 @@ POST /ci/lint | Attribute | Type | Required | Description | | ---------- | ------- | -------- | -------- | -| `content` | string | yes | the `.gitlab-ci.yaml` content| +| `content` | string | yes | The CI/CD configuration content. | +| `include_merged_yaml` | boolean | no | If the [expanded CI/CD configuration](#yaml-expansion) should be included in the response. | ```shell curl --header "Content-Type: application/json" "https://gitlab.example.com/api/v4/ci/lint" --data '{"content": "{ \"image\": \"ruby:2.6\", \"services\": [\"postgres\"], \"before_script\": [\"bundle install\", \"bundle exec rake db:create\"], \"variables\": {\"DB_NAME\": \"postgres\"}, \"types\": [\"test\", \"deploy\", \"notify\"], \"rspec\": { \"script\": \"rake spec\", \"tags\": [\"ruby\", \"postgres\"], \"only\": [\"branches\"]}}"}' ``` -Be sure to copy paste the exact contents of `.gitlab-ci.yml` as YAML is very picky about indentation and spaces. +Be sure to paste the exact contents of your GitLab CI/CD YAML config because YAML +is very sensitive about indentation and spacing. Example responses: @@ -53,3 +58,39 @@ Example responses: "error": "content is missing" } ``` + +### YAML expansion + +The expansion only works for CI configurations that don't have local [includes](../ci/yaml/README.md#include). + +Example contents of a `.gitlab-ci.yml` passed to the CI Lint API with +`include_merged_yaml` set as true: + +```yaml +include: + remote: 'https://example.com/remote.yaml' + +test: + stage: test + script: + - echo 1 +``` + +Example contents of `https://example.com/remote.yaml`: + +```yaml +another_test: + stage: test + script: + - echo 2 +``` + +Example response: + +```json +{ + "status": "valid", + "errors": [], + "merged_config": "---\n:another_test:\n :stage: test\n :script: echo 2\n:test:\n :stage: test\n :script: echo 1\n" +} +``` diff --git a/doc/api/members.md b/doc/api/members.md index 76d63b277c4..43012a5cb80 100644 --- a/doc/api/members.md +++ b/doc/api/members.md @@ -223,6 +223,58 @@ Example response: } ``` +## List all billable members of a group + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217384) in GitLab 13.5. + +Gets a list of group members who counts as billable, including members in the sub group/project. + +This function takes [pagination](README.md#pagination) parameters `page` and `per_page` to restrict the list of users. + +```plaintext +GET /groups/:id/billable_members +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/billable_members" +``` + +Example response: + +```json +[ + { + "id": 1, + "username": "raymond_smith", + "name": "Raymond Smith", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root", + }, + { + "id": 2, + "username": "john_doe", + "name": "John Doe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root", + "email": "john@example.com" + }, + { + "id": 3, + "username": "foo_bar", + "name": "Foo bar", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root" + } +] +``` + ## Add a member to a group or project Adds a member to a group or project. diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index 643d03b6fb8..89e4224c735 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -60,7 +60,7 @@ POST /projects/:id/approvals | `disable_overriding_approvers_per_merge_request` | boolean | no | Allow/Disallow overriding approvers per MR | | `merge_requests_author_approval` | boolean | no | Allow/Disallow authors from self approving merge requests; `true` means authors can self approve | | `merge_requests_disable_committers_approval` | boolean | no | Allow/Disallow committers from self approving merge requests | -| `require_password_to_approve` | boolean | no | Require approver to enter a password in order to authenticate before adding the approval | +| `require_password_to_approve` | boolean | no | Require approver to enter a password to authenticate before adding the approval | ```json { diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index faefc445210..944a5530100 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -2085,10 +2085,10 @@ the `approvals_before_merge` parameter: } ``` -## Create a to-do +## Create a to do -Manually creates a to-do for the current user on a merge request. -If there already exists a to-do for the user on that merge request, +Manually creates a to do for the current user on a merge request. +If there already exists a to do for the user on that merge request, status code `304` is returned. ```plaintext diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index ba59d467bc8..35a8226b4d8 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -87,6 +87,23 @@ the `plan` parameter associated with a namespace: ] ``` +Users on GitLab.com will also see a `max_seats_used` parameter. `max_seats_used` +is the highest number of users the group had. + +`max_seats_used` will be non-zero only for namespaces on paid plans. + +```json +[ + { + "id": 1, + "name": "user1", + "billable_members_count": 2, + "max_seats_used": 3, + ... + } +] +``` + NOTE: **Note:** Only group maintainers/owners are presented with `members_count_with_descendants`, as well as `plan` **(BRONZE ONLY)**. @@ -123,6 +140,7 @@ Example response: "web_url": "https://gitlab.example.com/groups/twitter", "members_count_with_descendants": 2, "billable_members_count": 2, + "max_seats_used": 0, "plan": "default", "trial_ends_on": null, "trial": false @@ -162,6 +180,7 @@ Example response: "web_url": "https://gitlab.example.com/groups/group1", "members_count_with_descendants": 2, "billable_members_count": 2, + "max_seats_used": 0, "plan": "default", "trial_ends_on": null, "trial": false @@ -188,6 +207,7 @@ Example response: "web_url": "https://gitlab.example.com/groups/group1", "members_count_with_descendants": 2, "billable_members_count": 2, + "max_seats_used": 0, "plan": "default", "trial_ends_on": null, "trial": false diff --git a/doc/api/openapi/openapi.yaml b/doc/api/openapi/openapi.yaml index 8aa4de62501..8c46804d86f 100644 --- a/doc/api/openapi/openapi.yaml +++ b/doc/api/openapi/openapi.yaml @@ -9,9 +9,9 @@ info: When viewing this on gitlab.com, you can test API calls directly from the browser against the `gitlab.com` instance, if you are logged in. The feature uses the current [GitLab session cookie](https://docs.gitlab.com/ee/api/README.html#session-cookie), - so each request is made using your account. + so each request is made using your account. - Read more at <https://docs.gitlab.com/ee/development/documentation/styleguide.html#restful-api>. + Read more at <https://docs.gitlab.com/ee/development/documentation/restful_api_styleguide.html>. version: "v4" title: "GitLab API" termsOfService: "https://about.gitlab.com/terms/" diff --git a/doc/api/packages.md b/doc/api/packages.md index cf65b518844..d4e69b9bc66 100644 --- a/doc/api/packages.md +++ b/doc/api/packages.md @@ -26,7 +26,7 @@ GET /projects/:id/packages | `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `order_by`| string | no | The field to use as order. One of `created_at` (default), `name`, `version`, or `type`. | | `sort` | string | no | The direction of the order, either `asc` (default) for ascending order or `desc` for descending order. | -| `package_type` | string | no | Filter the returned packages by type. One of `conan`, `maven`, `npm`, `pypi`, `composer`, or `nuget`. (_Introduced in GitLab 12.9_) +| `package_type` | string | no | Filter the returned packages by type. One of `conan`, `maven`, `npm`, `pypi`, `composer`, `nuget`, or `golang`. (_Introduced in GitLab 12.9_) | `package_name` | string | no | Filter the project packages with a fuzzy search by name. (_Introduced in GitLab 12.9_) ```shell @@ -50,6 +50,19 @@ Example response: "version": "1.0.3", "package_type": "npm", "created_at": "2019-11-27T03:37:38.711Z" + }, + { + "id": 3, + "name": "Hello/0.1@mycompany/stable", + "conan_package_name": "Hello", + "version": "0.1", + "package_type": "conan", + "_links": { + "web_path": "/foo/bar/-/packages/3", + "delete_api_path": "https://gitlab.example.com/api/v4/projects/1/packages/3" + }, + "created_at": "2029-12-16T20:33:34.316Z", + "tags": [] } ] ``` @@ -73,7 +86,7 @@ GET /groups/:id/packages | `exclude_subgroups` | boolean | false | If the parameter is included as true, packages from projects from subgroups are not listed. Default is `false`. | | `order_by`| string | no | The field to use as order. One of `created_at` (default), `name`, `version`, `type`, or `project_path`. | | `sort` | string | no | The direction of the order, either `asc` (default) for ascending order or `desc` for descending order. | -| `package_type` | string | no | Filter the returned packages by type. One of `conan`, `maven`, `npm`, `pypi`, `composer`, or `nuget`. (_Introduced in GitLab 12.9_) | +| `package_type` | string | no | Filter the returned packages by type. One of `conan`, `maven`, `npm`, `pypi`, `composer`, `nuget`, or `golang`. (_Introduced in GitLab 12.9_) | | `package_name` | string | no | Filter the project packages with a fuzzy search by name. (_[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/30980) in GitLab 13.0_) ```shell diff --git a/doc/api/releases/index.md b/doc/api/releases/index.md index 357f7e7a125..54dc2ecaa33 100644 --- a/doc/api/releases/index.md +++ b/doc/api/releases/index.md @@ -365,7 +365,7 @@ POST /projects/:id/releases | `assets:links` | array of hash | no | An array of assets links. | | `assets:links:name`| string | required by: `assets:links` | The name of the link. | | `assets:links:url` | string | required by: `assets:links` | The URL of the link. | -| `assets:links:filepath` | string | no | Optional path for a [Direct Asset link](../../user/project/releases.md). +| `assets:links:filepath` | string | no | Optional path for a [Direct Asset link](../../user/project/releases/index.md). | `assets:links:link_type` | string | no | The type of the link: `other`, `runbook`, `image`, `package`. Defaults to `other`. | `released_at` | datetime | no | The date when the release will be/was ready. Defaults to the current time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). | diff --git a/doc/api/repositories.md b/doc/api/repositories.md index 305216f853a..7e94ff7b7f2 100644 --- a/doc/api/repositories.md +++ b/doc/api/repositories.md @@ -24,7 +24,8 @@ Parameters: - `path` (optional) - The path inside repository. Used to get content of subdirectories - `ref` (optional) - The name of a repository branch or tag or if not given the default branch - `recursive` (optional) - Boolean value used to get a recursive tree (false by default) -- `per_page` (optional) - Number of results to show per page. If not specified, defaults to `20` +- `per_page` (optional) - Number of results to show per page. If not specified, defaults to `20`. + Read more on [pagination](README.md#pagination). ```json [ diff --git a/doc/api/resource_iteration_events.md b/doc/api/resource_iteration_events.md index f774cdfe9c7..47f4d70fdf1 100644 --- a/doc/api/resource_iteration_events.md +++ b/doc/api/resource_iteration_events.md @@ -6,14 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Resource iteration events API **(STARTER)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40850) in [GitLab Starter](https://about.gitlab.com/pricing/) 13.4 -> - It's [deployed behind a feature flag](../user/feature_flags.md), enabled by default. -> - It's enabled on GitLab.com. -> - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-iterations-events-tracking). - -NOTE: **Note:** -This feature might not be available to you. Check the **version history** note above for details. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229463) in [GitLab Starter](https://about.gitlab.com/pricing/) 13.4. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/229463) in [GitLab Starter](https://about.gitlab.com/pricing/) 13.5. Resource iteration events keep track of what happens to GitLab [issues](../user/project/issues/). @@ -154,22 +148,3 @@ Example response: "action": "remove" } ``` - -### Enable or disable iterations events tracking **(STARTER)** - -Iterations events tracking is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:track_iteration_change_events) -``` - -To disable it: - -```ruby -Feature.disable(:track_iteration_change_events) -``` diff --git a/doc/api/runners.md b/doc/api/runners.md index 436abe0a706..55dcf1dc3ee 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -271,22 +271,6 @@ Example response: } ``` -## Remove a runner - -Remove a runner. - -```plaintext -DELETE /runners/:id -``` - -| Attribute | Type | Required | Description | -|-----------|---------|----------|---------------------| -| `id` | integer | yes | The ID of a runner | - -```shell -curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/6" -``` - ## List runner's jobs > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/15432) in GitLab 10.3. @@ -466,7 +450,7 @@ Example response: Disable a specific runner from the project. It works only if the project isn't the only project associated with the specified runner. If so, an error is -returned. Use the [Remove a runner](#remove-a-runner) call instead. +returned. Use the call to [delete a runner](#delete-a-runner) instead. ```plaintext DELETE /projects/:id/runners/:runner_id @@ -580,9 +564,32 @@ Example response: } ``` -## Delete a registered runner +## Delete a runner + +There are two ways to delete a runner: + +- By specifying the runner ID. +- By specifying the runner's authentication token. -Deletes a registered runner. +### Delete a runner by ID + +To delete the runner by ID, use your access token with the runner's ID: + +```plaintext +DELETE /runners/:id +``` + +| Attribute | Type | Required | Description | +|-------------|---------|----------|---------------------| +| `id` | integer | yes | The ID of a runner. The ID is visible in the UI under **Settings > CI/CD**. Expand **Runners**, and below the **Remove Runner** button is an ID preceded by the pound sign, for example, `#6`. | + +```shell +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/6" +``` + +### Delete a runner by authentication token + +To delete the runner by using its authentication token: ```plaintext DELETE /runners @@ -590,7 +597,7 @@ DELETE /runners | Attribute | Type | Required | Description | |-------------|---------|----------|---------------------| -| `token` | string | yes | Runner's [authentication token](#registration-and-authentication-tokens). | +| `token` | string | yes | The runner's [authentication token](#registration-and-authentication-tokens). | ```shell curl --request DELETE "https://gitlab.example.com/api/v4/runners" --form "token=<authentication_token>" diff --git a/doc/api/search.md b/doc/api/search.md index cb90b9a064c..ae663f64755 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -27,7 +27,7 @@ GET /search Search the expression within the specified scope. Currently these scopes are supported: projects, issues, merge_requests, milestones, snippet_titles, users. -If Elasticsearch is enabled additional scopes available are blobs, wiki_blobs and commits. Find more about [the feature](../integration/elasticsearch.md). **(STARTER)** +If Elasticsearch is enabled additional scopes available are blobs, wiki_blobs, notes, and commits. Find more about [the feature](../integration/elasticsearch.md). **(STARTER)** The response depends on the requested scope. @@ -362,6 +362,40 @@ Example response: NOTE: **Note:** `filename` is deprecated in favor of `path`. Both return the full path of the file inside the repository, but in the future `filename` will be only the file name and not the full path. For details, see [issue 34521](https://gitlab.com/gitlab-org/gitlab/-/issues/34521). +### Scope: notes **(STARTER)** + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/6/search?scope=notes&search=maxime" +``` + +Example response: + +```json +[ + { + "id": 191, + "body": "Harum maxime consequuntur et et deleniti assumenda facilis.", + "attachment": null, + "author": { + "id": 23, + "name": "User 1", + "username": "user1", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/111d68d06e2d317b5a59c2c6c5bad808?s=80&d=identicon", + "web_url": "http://localhost:3000/user1" + }, + "created_at": "2017-09-05T08:01:32.068Z", + "updated_at": "2017-09-05T08:01:32.068Z", + "system": false, + "noteable_id": 22, + "noteable_type": "Issue", + "noteable_iid": 2 + } +] +``` + ### Scope: users ```shell @@ -402,7 +436,7 @@ GET /groups/:id/search Search the expression within the specified scope. Currently these scopes are supported: projects, issues, merge_requests, milestones, users. -If Elasticsearch is enabled additional scopes available are blobs, wiki_blobs and commits. Find more about [the feature](../integration/elasticsearch.md). **(STARTER)** +If Elasticsearch is enabled additional scopes available are blobs, wiki_blobs, notes, and commits. Find more about [the feature](../integration/elasticsearch.md). **(STARTER)** The response depends on the requested scope. @@ -706,6 +740,40 @@ Example response: NOTE **Note:** `filename` is deprecated in favor of `path`. Both return the full path of the file inside the repository, but in the future `filename` will be only the file name and not the full path. For details, see [issue 34521](https://gitlab.com/gitlab-org/gitlab/-/issues/34521). +### Scope: notes **(STARTER)** + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/6/search?scope=notes&search=maxime" +``` + +Example response: + +```json +[ + { + "id": 191, + "body": "Harum maxime consequuntur et et deleniti assumenda facilis.", + "attachment": null, + "author": { + "id": 23, + "name": "User 1", + "username": "user1", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/111d68d06e2d317b5a59c2c6c5bad808?s=80&d=identicon", + "web_url": "http://localhost:3000/user1" + }, + "created_at": "2017-09-05T08:01:32.068Z", + "updated_at": "2017-09-05T08:01:32.068Z", + "system": false, + "noteable_id": 22, + "noteable_type": "Issue", + "noteable_iid": 2 + } +] +``` + ### Scope: users ```shell diff --git a/doc/api/services.md b/doc/api/services.md index 405047a433d..7c01e43a4d8 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -659,7 +659,7 @@ Parameters: | `webhook` | string | true | The Hangouts Chat webhook. For example, `https://chat.googleapis.com/v1/spaces...`. | | `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | | `notify_only_default_branch` | boolean | false | DEPRECATED: This parameter has been replaced with `branches_to_be_notified` | -| `branches_to_be_notified` | string | all | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected" | +| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected" | | `push_events` | boolean | false | Enable notifications for push events | | `issues_events` | boolean | false | Enable notifications for issue events | | `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | @@ -714,7 +714,7 @@ Parameters: | `merge_requests_events` | boolean | false | Enable notifications for merge request events | | `tag_push_events` | boolean | false | Enable notifications for tag push events | | `note_events` | boolean | false | Enable notifications for note events | -| `confidental_note_events` | boolean | false | Enable notifications for confidential note events | +| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | | `pipeline_events` | boolean | false | Enable notifications for pipeline events | ### Delete HipChat service diff --git a/doc/api/settings.md b/doc/api/settings.md index c8a466d1fcd..deff2535f9c 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -2,8 +2,8 @@ These API calls allow you to read and modify GitLab instance [application settings](#list-of-settings-that-can-be-accessed-via-api-calls) -as appear in `/admin/application_settings/general`. You have to be an -administrator in order to perform this action. +as they appear in `/admin/application_settings/general`. You must be an +administrator to perform this action. ## Get current application settings @@ -185,13 +185,14 @@ Example responses: **(PREMIUM ONLY)** ## List of settings that can be accessed via API calls -In general, all settings are optional. Certain settings though, if enabled, will -require other settings to be set in order to function properly. These requirements -are listed in the descriptions of the relevant settings. +In general, all settings are optional. Certain settings though, if enabled, +require other settings to be set to function properly. These requirements are +listed in the descriptions of the relevant settings. -| Attribute | Type | Required | Description | -| --------- | ---- | :------: | ----------- | -| `admin_notification_email` | string | no | Abuse reports will be sent to this address if it is set. Abuse reports are always available in the Admin Area. | +| Attribute | Type | Required | Description | +|------------------------------------------|------------------|:------------------------------------:|-------------| +| `admin_notification_email` | string | no | Deprecated: Use `abuse_notification_email` instead. If set, [abuse reports](../user/admin_area/abuse_reports.md) are sent to this address. Abuse reports are always available in the Admin Area. | +| `abuse_notification_email` | string | no | If set, [abuse reports](../user/admin_area/abuse_reports.md) are sent to this address. Abuse reports are always available in the Admin Area. | | `after_sign_out_path` | string | no | Where to redirect users after logout. | | `after_sign_up_text` | string | no | Text shown to the user after signing up | | `akismet_api_key` | string | required by: `akismet_enabled` | API key for Akismet spam protection. | @@ -255,10 +256,10 @@ are listed in the descriptions of the relevant settings. | `external_auth_client_cert` | string | no | (**If enabled, requires:** `external_auth_client_key`) The certificate to use to authenticate with the external authorization service | | `external_auth_client_key_pass` | string | no | Passphrase to use for the private key when authenticating with the external service this is encrypted when stored | | `external_auth_client_key` | string | required by: `external_auth_client_cert` | Private key for the certificate when authentication is required for the external authorization service, this is encrypted when stored | -| `external_authorization_service_default_label` | string | required by: `external_authorization_service_enabled` | The default classification label to use when requesting authorization and no classification label has been specified on the project | +| `external_authorization_service_default_label` | string | required by:<br>`external_authorization_service_enabled` | The default classification label to use when requesting authorization and no classification label has been specified on the project. | | `external_authorization_service_enabled` | boolean | no | (**If enabled, requires:** `external_authorization_service_default_label`, `external_authorization_service_timeout` and `external_authorization_service_url`) Enable using an external authorization service for accessing projects | -| `external_authorization_service_timeout` | float | required by: `external_authorization_service_enabled` | The timeout after which an authorization request is aborted, in seconds. When a request times out, access is denied to the user. (min: 0.001, max: 10, step: 0.001) | -| `external_authorization_service_url` | string | required by: `external_authorization_service_enabled` | URL to which authorization requests will be directed | +| `external_authorization_service_timeout` | float | required by:<br>`external_authorization_service_enabled` | The timeout after which an authorization request is aborted, in seconds. When a request times out, access is denied to the user. (min: 0.001, max: 10, step: 0.001). | +| `external_authorization_service_url` | string | required by:<br>`external_authorization_service_enabled` | URL to which authorization requests are directed. | | `file_template_project_id` | integer | no | **(PREMIUM)** The ID of a project to load custom file templates from | | `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday, `1` for Monday, and `6` for Saturday. | | `geo_node_allowed_ips` | string | yes | **(PREMIUM)** Comma-separated list of IPs and CIDRs of allowed secondary nodes. For example, `1.1.1.1, 2.2.2.0/24`. | @@ -321,7 +322,7 @@ are listed in the descriptions of the relevant settings. | `receive_max_input_size` | integer | no | Maximum push size (MB). | | `repository_checks_enabled` | boolean | no | GitLab will periodically run `git fsck` in all project and wiki repositories to look for silent disk corruption issues. | | `repository_size_limit` | integer | no | **(PREMIUM)** Size limit per repository (MB) | -| `repository_storages_weighted` | hash of strings to integers | no | (GitLab 13.1 and later) Hash of names of taken from `gitlab.yml` to weights. New projects are created in one of these stores, chosen by a weighted random selection. | +| `repository_storages_weighted` | hash of strings to integers | no | (GitLab 13.1 and later) Hash of names of taken from `gitlab.yml` to [weights](../administration/repository_storage_paths.md#choose-where-new-repositories-will-be-stored). New projects are created in one of these stores, chosen by a weighted random selection. | | `repository_storages` | array of strings | no | (GitLab 13.0 and earlier) List of names of enabled storage paths, taken from `gitlab.yml`. New projects are created in one of these stores, chosen at random. | | `require_two_factor_authentication` | boolean | no | (**If enabled, requires:** `two_factor_grace_period`) Require all users to set up Two-factor authentication. | | `restricted_visibility_levels` | array of strings | no | Selected levels cannot be used by non-admin users for groups, projects or snippets. Can take `private`, `internal` and `public` as a parameter. Default is `null` which means there is no restriction. | @@ -351,14 +352,14 @@ are listed in the descriptions of the relevant settings. | `terminal_max_session_time` | integer | no | Maximum time for web terminal websocket connection (in seconds). Set to `0` for unlimited time. | | `terms` | text | required by: `enforce_terms` | (**Required by:** `enforce_terms`) Markdown content for the ToS. | | `throttle_authenticated_api_enabled` | boolean | no | (**If enabled, requires:** `throttle_authenticated_api_period_in_seconds` and `throttle_authenticated_api_requests_per_period`) Enable authenticated API request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). | -| `throttle_authenticated_api_period_in_seconds` | integer | required by: `throttle_authenticated_api_enabled` | Rate limit period in seconds. | -| `throttle_authenticated_api_requests_per_period` | integer | required by: `throttle_authenticated_api_enabled` | Max requests per period per user. | +| `throttle_authenticated_api_period_in_seconds` | integer | required by:<br>`throttle_authenticated_api_enabled` | Rate limit period in seconds. | +| `throttle_authenticated_api_requests_per_period` | integer | required by:<br>`throttle_authenticated_api_enabled` | Max requests per period per user. | | `throttle_authenticated_web_enabled` | boolean | no | (**If enabled, requires:** `throttle_authenticated_web_period_in_seconds` and `throttle_authenticated_web_requests_per_period`) Enable authenticated web request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). | -| `throttle_authenticated_web_period_in_seconds` | integer | required by: `throttle_authenticated_web_enabled` | Rate limit period in seconds. | -| `throttle_authenticated_web_requests_per_period` | integer | required by: `throttle_authenticated_web_enabled` | Max requests per period per user. | +| `throttle_authenticated_web_period_in_seconds` | integer | required by:<br>`throttle_authenticated_web_enabled` | Rate limit period in seconds. | +| `throttle_authenticated_web_requests_per_period` | integer | required by:<br>`throttle_authenticated_web_enabled` | Max requests per period per user. | | `throttle_unauthenticated_enabled` | boolean | no | (**If enabled, requires:** `throttle_unauthenticated_period_in_seconds` and `throttle_unauthenticated_requests_per_period`) Enable unauthenticated request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). | -| `throttle_unauthenticated_period_in_seconds` | integer | required by: `throttle_unauthenticated_enabled` | Rate limit period in seconds. | -| `throttle_unauthenticated_requests_per_period` | integer | required by: `throttle_unauthenticated_enabled` | Max requests per period per IP. | +| `throttle_unauthenticated_period_in_seconds` | integer | required by:<br>`throttle_unauthenticated_enabled` | Rate limit period in seconds. | +| `throttle_unauthenticated_requests_per_period` | integer | required by:<br>`throttle_unauthenticated_enabled` | Max requests per period per IP. | | `time_tracking_limit_to_hours` | boolean | no | Limit display of time tracking units to hours. Default is `false`. | | `two_factor_grace_period` | integer | required by: `require_two_factor_authentication` | Amount of time (in hours) that users are allowed to skip forced configuration of two-factor authentication. | | `unique_ips_limit_enabled` | boolean | no | (**If enabled, requires:** `unique_ips_limit_per_user` and `unique_ips_limit_time_window`) Limit sign in from multiple ips. | diff --git a/doc/api/todos.md b/doc/api/todos.md index ebe10ecbd49..ab36021d694 100644 --- a/doc/api/todos.md +++ b/doc/api/todos.md @@ -4,13 +4,13 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# To-dos API +# To dos API > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/3188) in GitLab 8.10. -## Get a list of to-dos +## Get a list of to dos -Returns a list of to-dos. When no filter is applied, it returns all pending to-dos +Returns a list of to dos. When no filter is applied, it returns all pending to dos for the current user. Different filters allow the user to precise the request. ```plaintext @@ -25,8 +25,8 @@ Parameters: | `author_id` | integer | no | The ID of an author | | `project_id` | integer | no | The ID of a project | | `group_id` | integer | no | The ID of a group | -| `state` | string | no | The state of the to-do. Can be either `pending` or `done` | -| `type` | string | no | The type of a to-do. Can be either `Issue`, `MergeRequest`, `DesignManagement::Design` or `AlertManagement::Alert` | +| `state` | string | no | The state of the to do. Can be either `pending` or `done` | +| `type` | string | no | The type of a to do. Can be either `Issue`, `MergeRequest`, `DesignManagement::Design` or `AlertManagement::Alert` | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/todos" @@ -187,10 +187,10 @@ Example Response: ] ``` -## Mark a to-do as done +## Mark a to do as done -Marks a single pending to-do given by its ID for the current user as done. The -to-do marked as done is returned in the response. +Marks a single pending to do given by its ID for the current user as done. The +to do marked as done is returned in the response. ```plaintext POST /todos/:id/mark_as_done @@ -200,7 +200,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer | yes | The ID of a to-do | +| `id` | integer | yes | The ID of a to do | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/todos/130/mark_as_done" @@ -285,9 +285,9 @@ Example Response: } ``` -## Mark all to-dos as done +## Mark all to dos as done -Marks all pending to-dos for the current user as done. It returns the HTTP status code `204` with an empty response. +Marks all pending to dos for the current user as done. It returns the HTTP status code `204` with an empty response. ```plaintext POST /todos/mark_as_done diff --git a/doc/api/users.md b/doc/api/users.md index 634e0bd0842..dfd00571bcb 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -950,7 +950,7 @@ Returns `204 No Content` on success, or `404 Not found` if the key cannot be fou ## List all GPG keys for given user -Get a list of a specified user's GPG keys. Available only for admins. +Get a list of a specified user's GPG keys. This endpoint can be accessed without authentication. ```plaintext GET /users/:id/gpg_keys diff --git a/doc/api/vulnerabilities.md b/doc/api/vulnerabilities.md index 73c765e2ccc..f89f2bda2eb 100644 --- a/doc/api/vulnerabilities.md +++ b/doc/api/vulnerabilities.md @@ -220,3 +220,53 @@ Example response: "closed_at": null } ``` + +## Revert vulnerability to detected state + +Reverts a given vulnerability to detected state. Returns status code `304` if the vulnerability is already in detected state. + +If an authenticated user does not have permission to +[revert vulnerability to detected state](../user/permissions.md#project-members-permissions), +this request will result in a `403` status code. + +```plaintext +POST /vulnerabilities/:id/revert +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer or string | yes | The ID of a vulnerability to revert to detected state | + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/vulnerabilities/5/dismiss" +``` + +Example response: + +```json +{ + "id": 2, + "title": "Predictable pseudorandom number generator", + "description": null, + "state": "detected", + "severity": "medium", + "confidence": "medium", + "report_type": "sast", + "project": { + "id": 32, + "name": "security-reports", + "full_path": "/gitlab-examples/security/security-reports", + "full_name": "gitlab-examples / security / security-reports" + }, + "author_id": 1, + "updated_by_id": null, + "last_edited_by_id": null, + "closed_by_id": null, + "start_date": null, + "due_date": null, + "created_at": "2019-10-13T15:08:40.219Z", + "updated_at": "2019-10-13T15:09:40.382Z", + "last_edited_at": null, + "closed_at": null +} +``` diff --git a/doc/api/vulnerability_exports.md b/doc/api/vulnerability_exports.md index d19d41b647e..0ee8a18a46a 100644 --- a/doc/api/vulnerability_exports.md +++ b/doc/api/vulnerability_exports.md @@ -198,18 +198,18 @@ The response will be `404 Not Found` if the vulnerability export is not finished Example response: ```csv -Group Name,Project Name,Scanner Type,Scanner Name,Status,Vulnerability,Details,Additional Info,Severity,CVE -Gitlab.org,Defend,container_scanning,Clair,confirmed,CVE-2017-16997 in glibc,,CVE-2017-16997 in glibc,critical,CVE-2017-16997 +Group Name,Project Name,Scanner Type,Scanner Name,Status,Vulnerability,Details,Additional Info,Severity,CVE,CWE,Other Identifiers +Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2017-16997 in glibc,,CVE-2017-16997 in glibc,critical,CVE-2017-16997 Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2017-18269 in glibc,,CVE-2017-18269 in glibc,critical,CVE-2017-18269 Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2018-1000001 in glibc,,CVE-2018-1000001 in glibc,high,CVE-2018-1000001 Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2016-10228 in glibc,,CVE-2016-10228 in glibc,medium,CVE-2016-10228 -Gitlab.org,Defend,container_scanning,Clair,confirmed,CVE-2010-4052 in glibc,,CVE-2010-4052 in glibc,low,CVE-2010-4052 +Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2010-4052 in glibc,,CVE-2010-4052 in glibc,low,CVE-2010-4052 Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2018-18520 in elfutils,,CVE-2018-18520 in elfutils,low,CVE-2018-18520 -Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2018-16869 in nettle,,CVE-2018-16869 in nettle,unknown,CVE-2018-16869 -Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Regular Expression Denial of Service in debug,,Regular Expression Denial of Service in debug,unknown,yarn.lock:debug:gemnasium:37283ed4-0380-40d7-ada7-2d994afcc62a -Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Authentication bypass via incorrect DOM traversal and canonicalization in saml2-js,,Authentication bypass via incorrect DOM traversal and canonicalization in saml2-js,unknown,yarn.lock:saml2-js:gemnasium:9952e574-7b5b-46fa-a270-aeb694198a98 -Gitlab.org,Defend,sast,Find Security Bugs,detected,Predictable pseudorandom number generator,,Predictable pseudorandom number generator,medium,818bf5dacb291e15d9e6dc3c5ac32178:PREDICTABLE_RANDOM:src/main/java/com/gitlab/security_products/tests/App.java:47 -Gitlab.org,Defend,sast,Find Security Bugs,detected,Cipher with no integrity,,Cipher with no integrity,medium,e6449b89335daf53c0db4c0219bc1634:CIPHER_INTEGRITY:src/main/java/com/gitlab/security_products/tests/App.java:29 -Gitlab.org,Defend,sast,Find Security Bugs,detected,Predictable pseudorandom number generator,,Predictable pseudorandom number generator,medium,e8ff1d01f74cd372f78da8f5247d3e73:PREDICTABLE_RANDOM:src/main/java/com/gitlab/security_products/tests/App.java:41 -Gitlab.org,Defend,sast,Find Security Bugs,confirmed,ECB mode is insecure 2,,ECB mode is insecure,medium,ea0f905fc76f2739d5f10a1fd1e37a10:ECB_MODE:src/main/java/com/gitlab/security_products/tests/App.java:29 -Gitlab.org,Defend,``` +Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2018-16869 in nettle,,CVE-2018-16869 in nettle,unknown,CVE-2018-16869,CWE-1 +Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Regular Expression Denial of Service in debug,,Regular Expression Denial of Service in debug,unknown,CVE-2021-1234,CWE-2,"""yarn.lock:debug:gemnasium:37283ed4-0380-40d7-ada7-2d994afcc62a""" +Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Authentication bypass via incorrect DOM traversal and canonicalization in saml2-js,,Authentication bypass via incorrect DOM traversal and canonicalization in saml2-js,unknown,,,"""yarn.lock:saml2-js:gemnasium:9952e574-7b5b-46fa-a270-aeb694198a98""" +Gitlab.org,Defend,sast,Find Security Bugs,detected,Predictable pseudorandom number generator,,Predictable pseudorandom number generator,medium,,,"""818bf5dacb291e15d9e6dc3c5ac32178:PREDICTABLE_RANDOM:src/main/java/com/gitlab/security_products/tests/App.java:47""" +Gitlab.org,Defend,sast,Find Security Bugs,detected,Cipher with no integrity,,Cipher with no integrity,medium,,,"""e6449b89335daf53c0db4c0219bc1634:CIPHER_INTEGRITY:src/main/java/com/gitlab/security_products/tests/App.java:29""" +Gitlab.org,Defend,sast,Find Security Bugs,detected,Predictable pseudorandom number generator,,Predictable pseudorandom number generator,medium,,,"""e8ff1d01f74cd372f78da8f5247d3e73:PREDICTABLE_RANDOM:src/main/java/com/gitlab/security_products/tests/App.java:41""" +Gitlab.org,Defend,sast,Find Security Bugs,detected,ECB mode is insecure,,ECB mode is insecure,medium,,,"""ea0f905fc76f2739d5f10a1fd1e37a10:ECB_MODE:src/main/java/com/gitlab/security_products/tests/App.java:29""" +``` diff --git a/doc/architecture/blueprints/cloud_native_build_logs/index.md b/doc/architecture/blueprints/cloud_native_build_logs/index.md new file mode 100644 index 00000000000..0b02da21109 --- /dev/null +++ b/doc/architecture/blueprints/cloud_native_build_logs/index.md @@ -0,0 +1,137 @@ +--- +comments: false +description: 'Next iteration of build logs architecture at GitLab' +--- + +# Cloud Native Build Logs + +Cloud native and the adoption of Kubernetes has been recognised by GitLab to be +one of the top two biggest tailwinds that are helping us grow faster as a +company behind the project. + +This effort is described in a more details [in the infrastructure team +handbook](https://about.gitlab.com/handbook/engineering/infrastructure/production/kubernetes/gitlab-com/). + +## Traditional build logs + +Traditional job logs depend a lot on availability of a local shared storage. + +Every time a GitLab Runner sends a new partial build output, we write this +output to a file on a disk. This is simple, but this mechanism depends on +shared local storage - the same file needs to be available on every GitLab web +node machine, because GitLab Runner might connect to a different one every time +it performs an API request. Sidekiq also needs access to the file because when +a job is complete, a trace file contents will be sent to the object store. + +## New architecture + +New architecture writes data to Redis instead of writing build logs into a +file. + +In order to make this performant and resilient enough, we implemented a chunked +I/O mechanism - we store data in Redis in chunks, and migrate them to an object +store once we reach a desired chunk size. + +Simplified sequence diagram is available below. + +```mermaid +sequenceDiagram + autonumber + participant U as User + participant R as Runner + participant G as GitLab (rails) + participant I as Redis + participant D as Database + participant O as Object store + + loop incremental trace update sent by a runner + Note right of R: Runner appends a build trace + R->>+G: PATCH trace [build.id, offset, data] + G->>+D: find or create chunk [chunk.index] + D-->>-G: chunk [id, index] + G->>I: append chunk data [chunk.index, data] + G-->>-R: 200 OK + end + + Note right of R: User retrieves a trace + U->>+G: GET build trace + loop every trace chunk + G->>+D: find chunk [index] + D-->>-G: chunk [id] + G->>+I: read chunk data [chunk.index] + I-->>-G: chunk data [data, size] + end + G-->>-U: build trace + + Note right of R: Trace chunk is full + R->>+G: PATCH trace [build.id, offset, data] + G->>+D: find or create chunk [chunk.index] + D-->>-G: chunk [id, index] + G->>I: append chunk data [chunk.index, data] + G->>G: chunk full [index] + G-->>-R: 200 OK + G->>+I: read chunk data [chunk.index] + I-->>-G: chunk data [data, size] + G->>O: send chunk data [data, size] + G->>+D: update data store type [chunk.id] + G->>+I: delete chunk data [chunk.index] +``` + +## NFS coupling + +In 2017, we experienced serious problems of scaling our NFS infrastructure. We +even tried to replace NFS with +[CephFS](https://docs.ceph.com/docs/master/cephfs/) - unsuccessfully. + +Since that time it has become apparent that the cost of operations and +maintenance of a NFS cluster is significant and that if we ever decide to +migrate to Kubernetes [we need to decouple GitLab from a shared local storage +and +NFS](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/426#note_375646396). + +1. NFS might be a single point of failure +1. NFS can only be reliably scaled vertically +1. Moving to Kubernetes means increasing the number of mount points by an order + of magnitude +1. NFS depends on extremely reliable network which can be difficult to provide + in Kubernetes environment +1. Storing customer data on NFS involves additional security risks + +Moving GitLab to Kubernetes without NFS decoupling would result in an explosion +of complexity, maintenance cost and enormous, negative impact on availability. + +## Iterations + +1. ✓ Implement the new architecture in way that it does not depend on shared local storage +1. ✓ Evaluate performance and edge-cases, iterate to improve the new architecture +1. ✓ Design cloud native build logs correctness verification mechanisms +1. ✓ Build observability mechanisms around performance and correctness +1. Rollout the feature into production environment incrementally + +The work needed to make the new architecture production ready and enabled on +GitLab.com is being tracked in [Cloud Native Build Logs on +GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/4275) epic. + +Enabling this feature on GitLab.com is a subtask of [making the new +architecture generally +available](https://gitlab.com/groups/gitlab-org/-/epics/3791) for everyone. + +## Who + +Proposal: + +| Role | Who +|------------------------------|-------------------------| +| Author | Grzegorz Bizon | +| Architecture Evolution Coach | Gerardo Lopez-Fernandez | +| Engineering Leader | Darby Frey | +| Domain Expert | Kamil Trzciński | +| Domain Expert | Sean McGivern | + +DRIs: + +| Role | Who +|------------------------------|------------------------| +| Product | Jason Yavorska | +| Leadership | Darby Frey | +| Engineering | Grzegorz Bizon | diff --git a/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md b/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md new file mode 100644 index 00000000000..7aa7e10ace3 --- /dev/null +++ b/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md @@ -0,0 +1,131 @@ +--- +comments: false +description: 'Making GitLab Pages a Cloud Native application - architecture blueprint.' +--- + +# GitLab Pages New Architecture + +GitLab Pages is an important component of the GitLab product. It is mostly +being used to serve static content, and has a limited set of well defined +responsibilities. That being said, unfortunately it has become a blocker for +GitLab.com Kubernetes migration. + +Cloud Native and the adoption of Kubernetes has been recognised by GitLab to be +one of the top two biggest tailwinds that are helping us grow faster as a +company behind the project. + +This effort is described in more detail [in the infrastructure team handbook +page](https://about.gitlab.com/handbook/engineering/infrastructure/production/kubernetes/gitlab-com/). + +GitLab Pages is tightly coupled with NFS and in order to unblock Kubernetes +migration a significant change to GitLab Pages' architecture is required. This +is an ongoing work that we have started more than a year ago. This blueprint +might be useful to understand why it is important, and what is the roadmap. + +## How GitLab Pages Works + +GitLab Pages is a daemon designed to serve static content, written in +[Go](https://golang.org/). + +Initially, GitLab Pages has been designed to store static content on a local +shared block storage (NFS) in a hierarchical group > project directory +structure. Each directory, representing a project, was supposed to contain a +configuration file and static content that GitLab Pages daemon was supposed to +read and serve. + +```mermaid +graph LR + A(GitLab Rails) -- Writes new pages deployment --> B[(NFS)] + C(GitLab Pages) -. Reads static content .-> B +``` + +This initial design has become outdated because of a few reasons - NFS coupling +being one of them - and we decided to replace it with more "decoupled +service"-like architecture. The new architecture, that we are working on, is +described in this blueprint. + +## NFS coupling + +In 2017, we experienced serious problems of scaling our NFS infrastructure. We +even tried to replace NFS with +[CephFS](https://docs.ceph.com/docs/master/cephfs/) - unsuccessfully. + +Since that time it has become apparent that the cost of operations and +maintenance of a NFS cluster is significant and that if we ever decide to +migrate to Kubernetes [we need to decouple GitLab from a shared local storage +and +NFS](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/426#note_375646396). + +1. NFS might be a single point of failure +1. NFS can only be reliably scaled vertically +1. Moving to Kubernetes means increasing the number of mount points by an order + of magnitude +1. NFS depends on extremely reliable network which can be difficult to provide + in Kubernetes environment +1. Storing customer data on NFS involves additional security risks + +Moving GitLab to Kubernetes without NFS decoupling would result in an explosion +of complexity, maintenance cost and enormous, negative impact on availability. + +## New GitLab Pages Architecture + +- GitLab Pages is going to source domains' configuration from GitLab's internal + API, instead of reading `config.json` files from a local shared storage. +- GitLab Pages is going to serve static content from Object Storage. + +```mermaid +graph TD + A(User) -- Pushes pages deployment --> B{GitLab} + C((GitLab Pages)) -. Reads configuration from API .-> B + C -. Reads static content .-> D[(Object Storage)] + C -- Serves static content --> E(Visitors) +``` + +This new architecture has been briefly described in [the blog +post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/) +too. + +## Iterations + +1. ✓ Redesign GitLab Pages configuration source to use GitLab's API +1. ✓ Evaluate performance and build reliable caching mechanisms +1. ✓ Incrementally rollout the new source on GitLab.com +1. ✓ Make GitLab Pages API domains config source enabled by default +1. Enable experimentation with different servings through feature flags +1. Triangulate object store serving design through meaningful experiments +1. Design pages migration mechanisms that can work incrementally +1. Gradually migrate towards object storage serving on GitLab.com + +[GitLab Pages Architecture](https://gitlab.com/groups/gitlab-org/-/epics/1316) +epic with detailed roadmap is also available. + +## Who + +Proposal: + +| Role | Who +|------------------------------|-------------------------| +| Author | Grzegorz Bizon | +| Architecture Evolution Coach | Kamil Trzciński | +| Engineering Leader | Daniel Croft | +| Domain Expert | Grzegorz Bizon | +| Domain Expert | Vladimir Shushlin | +| Domain Expert | Jaime Martinez | + +DRIs: + +| Role | Who +|------------------------------|------------------------| +| Product | Jackie Porter | +| Leadership | Daniel Croft | +| Engineering | Kamil Trzciński | + +Domain Experts: + +| Role | Who +|------------------------------|------------------------| +| Domain Expert | Kamil Trzciński | +| Domain Expert | Grzegorz Bizon | +| Domain Expert | Vladimir Shushlin | +| Domain Expert | Jaime Martinez | +| Domain Expert | Krasimir Angelov | diff --git a/doc/architecture/blueprints/feature_flags_development/index.md b/doc/architecture/blueprints/feature_flags_development/index.md new file mode 100644 index 00000000000..9e6fd452dcb --- /dev/null +++ b/doc/architecture/blueprints/feature_flags_development/index.md @@ -0,0 +1,136 @@ +--- +comments: false +description: 'Internal usage of Feature Flags for GitLab development' +--- + +# Usage of Feature Flags for GitLab development + +Usage of feature flags become crucial for the development of GitLab. The +feature flags are a convenient way to ship changes early, and safely rollout +them to wide audience ensuring that feature is stable and performant. + +Since the presence of feature is controlled with a dedicated condition, a +developer can decide for a best time for testing the feature, ensuring that +feature is not enable prematurely. + +## Challenges + +The extensive usage of feature flags poses a few challenges + +- Each feature flag that we add to codebase is a ~"technical debt" as it adds a + matrix of configurations. +- Testing each combination of feature flags is close to impossible, so we + instead try to optimise our testing of feature flags to the most common + scenarios. +- There's a growing challenge of maintaining a growing number of feature flags. + We sometimes forget how our feature flags are configured or why we haven't + yet removed the feature flag. +- The usage of feature flags can also be confusing to people outside of + development that might not fully understand dependence of ~feature or ~bug + fix on feature flag and how this feature flag is configured. Or if the feature + should be announced as part of release post. +- Maintaining feature flags poses additional challenge of having to manage + different configurations across different environments/target. We have + different configuration of feature flags for testing, for development, for + staging, for production and what is being shipped to our customers as part of + on-premise offering. + +## Goals + +The biggest challenge today with our feature flags usage is their implicit +nature. Feature flags are part of the codebase, making them hard to understand +outside of development function. + +We should aim to make our feature flag based development to be accessible to +any interested party. + +- developer / engineer + - can easily add a new feature flag, and configure it's state + - can quickly find who to reach if touches another feature flag + - can quickly find stale feature flags +- engineering manager + - can understand what feature flags her/his group manages +- engineering manager and director + - can understand how much ~"technical debt" is inflicted due to amount of feature flags that we have to manage + - can understand how many feature flags are added and removed in each release +- product manager and documentation writer + - can understand what features are gated by what feature flags + - can understand if feature and thus feature flag is generally available on GitLab.com + - can understand if feature and thus feature flag is enabled by default for on-premise installations +- delivery engineer + - can understand what feature flags are introduced and changed between subsequent deployments +- support and reliability engineer + - can understand how feature flags changed between releases: what feature flags become enabled, what removed + - can quickly find relevant information about feature flag to know individuals which might help with an ongoing support request or incident + +## Proposal + +To help with above goals we should aim to make our feature flags usage explicit +and understood by all involved parties. + +Introduce a YAML-described `feature-flags/<name-of-feature.yml>` that would +allow us to have: + +1. A central place where all feature flags are documented, +1. A description of why the given feature flag was introduced, +1. A what relevant issue and merge request it was introduced by, +1. Build automated documentation with all feature flags in the codebase, +1. Track how many feature flags are per given group +1. Track how many feature flags are added and removed between releases +1. Make this information easily accessible for all +1. Allow our customers to easily discover how to enable features and quickly + find out information what did change between different releases + +### The `YAML` + +```yaml +--- +name: ci_disallow_to_create_merge_request_pipelines_in_target_project +introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40724 +rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/235119 +group: group::progressive delivery +type: development +default_enabled: false +``` + +## Reasons + +These are reason why these changes are needed: + +- we have around 500 different feature flags today +- we have hard time tracking their usage +- we have ambiguous usage of feature flag with different `default_enabled:` and + different `actors` used +- we lack a clear indication who owns what feature flag and where to find + relevant informations +- we do not emphasise the desire to create feature flag rollout issue to + indicate that feature flag is in fact a ~"technical debt" +- we don't know exactly what feature flags we have in our codebase +- we don't know exactly how our feature flags are configured for different + environments: what is being used for `test`, what we ship for `on-premise`, + what is our settings for `staging`, `qa` and `production` + +## Iterations + +This work is being done as part of dedicated epic: [Improve internal usage of +Feature Flags](https://gitlab.com/groups/gitlab-org/-/epics/3551). This epic +describes a meta reasons for making these changes. + +## Who + +Proposal: + +| Role | Who +|------------------------------|-------------------------| +| Author | Kamil Trzciński | +| Architecture Evolution Coach | Gerardo Lopez-Fernandez | +| Engineering Leader | Kamil Trzciński | +| Domain Expert | Shinya Maeda | + +DRIs: + +| Role | Who +|------------------------------|------------------------| +| Product | Kenny Johnston | +| Leadership | Craig Gomes | +| Engineering | Kamil Trzciński | diff --git a/doc/architecture/index.md b/doc/architecture/index.md new file mode 100644 index 00000000000..0a2ade6b7b0 --- /dev/null +++ b/doc/architecture/index.md @@ -0,0 +1,9 @@ +--- +comments: false +description: 'Architecture Practice at GitLab' +--- + +# Architecture at GitLab + +- [Architecture at GitLab](https://about.gitlab.com/handbook/engineering/architecture/) +- [Architecture Workflow](https://about.gitlab.com/handbook/engineering/architecture/workflow/) diff --git a/doc/ci/README.md b/doc/ci/README.md index 9d3f7f2a8f2..ec8697224b8 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -24,7 +24,7 @@ webcast to learn about continuous methods and how GitLab’s built-in CI can hel ## Overview Continuous Integration works by pushing small code chunks to your -application's code base hosted in a Git repository, and, to every +application's code base hosted in a Git repository, and to every push, run a pipeline of scripts to build, test, and validate the code changes before merging them into the main branch. diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index 74c48f087b2..f8e2a2b7eb0 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -9,7 +9,7 @@ type: howto GitLab CI/CD can be used with Bitbucket Cloud by: -1. Creating a [CI/CD project](../../user/project/ci_cd_for_external_repo.md). +1. Creating a [CI/CD project](index.md). 1. Connecting your Git repository via URL. To use GitLab CI/CD with a Bitbucket Cloud repository: diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 045fcd39c4d..a319c5f09ab 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -48,7 +48,6 @@ The simplest approach is to install GitLab Runner in `shell` execution mode. GitLab Runner then executes job scripts as the `gitlab-runner` user. 1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/#installation). - 1. During GitLab Runner installation select `shell` as method of executing job scripts or use command: ```shell @@ -90,7 +89,6 @@ GitLab Runner then executes job scripts as the `gitlab-runner` user. 1. You can now use `docker` command (and **install** `docker-compose` if needed). -NOTE: **Note:** By adding `gitlab-runner` to the `docker` group you are effectively granting `gitlab-runner` full root permissions. For more information please read [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful). @@ -101,7 +99,6 @@ The second approach is to use the special Docker-in-Docker (dind) (`docker`) and run the job script in context of that image in privileged mode. -NOTE: **Note:** `docker-compose` is not part of Docker-in-Docker (dind). To use `docker-compose` in your CI builds, follow the `docker-compose` [installation instructions](https://docs.docker.com/compose/install/). @@ -149,22 +146,17 @@ released. #### TLS enabled -NOTE: **Note:** -Requires GitLab Runner 11.11 or later, but is not supported if GitLab -Runner is installed using the [Helm -chart](https://docs.gitlab.com/runner/install/kubernetes.html). See the -[related -issue](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/83) for -details. - The Docker daemon supports connection over TLS and it's done by default for Docker 19.03.12 or higher. This is the **suggested** way to use the Docker-in-Docker service and [GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners) support this. -1. Install [GitLab Runner](https://docs.gitlab.com/runner/install/). +GitLab Runner 11.11 or later is required, but it is not supported if GitLab +Runner is installed using the [Helm chart](https://docs.gitlab.com/runner/install/kubernetes.html). +See the [related issue](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/83) for details. +1. Install [GitLab Runner](https://docs.gitlab.com/runner/install/). 1. Register GitLab Runner from the command line to use `docker` and `privileged` mode: @@ -225,7 +217,7 @@ support this. # The 'docker' hostname is the alias of the service container as described at # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services. # - # Note that if you're using GitLab Runner 12.7 or earlier with the Kubernetes executor and Kubernetes 1.6 or earlier, + # If you're using GitLab Runner 12.7 or earlier with the Kubernetes executor and Kubernetes 1.6 or earlier, # the variable must be set to tcp://localhost:2376 because of how the # Kubernetes executor connects services to the job container # DOCKER_HOST: tcp://localhost:2376 @@ -287,7 +279,7 @@ variables: # The 'docker' hostname is the alias of the service container as described at # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services # - # Note that if you're using GitLab Runner 12.7 or earlier with the Kubernetes executor and Kubernetes 1.6 or earlier, + # If you're using GitLab Runner 12.7 or earlier with the Kubernetes executor and Kubernetes 1.6 or earlier, # the variable must be set to tcp://localhost:2375 because of how the # Kubernetes executor connects services to the job container # DOCKER_HOST: tcp://localhost:2375 @@ -324,7 +316,6 @@ are done to the services as well, making these incompatible. In order to do that, follow the steps: 1. Install [GitLab Runner](https://docs.gitlab.com/runner/install/). - 1. Register GitLab Runner from the command line to use `docker` and share `/var/run/docker.sock`: ```shell @@ -506,7 +497,6 @@ environment = ["DOCKER_DRIVER=overlay2"] If you're running multiple runners, you have to modify all configuration files. -NOTE: **Note:** Read more about the [runner configuration](https://docs.gitlab.com/runner/configuration/) and [using the OverlayFS storage driver](https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/). diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 0fcd95c41ed..185de159416 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -138,7 +138,6 @@ still succeeds even if that warning was printed. For example: As it was mentioned before, this feature is designed to provide **network accessible** services. A database is the simplest example of such a service. -NOTE: **Note:** The services feature is not designed to, and does not add any software from the defined `services` image(s) to the job's container. @@ -186,7 +185,6 @@ access to it from your build container under two hostnames to choose from: - `tutum-wordpress` - `tutum__wordpress` -NOTE: **Note:** Hostnames with underscores are not RFC valid and may cause problems in 3rd party applications. @@ -364,10 +362,9 @@ For example, the following two definitions are equal: | `name` | yes, when used with any other option | 9.4 | Full name of the image that should be used. It should contain the Registry part if needed. | | `entrypoint` | no | 9.4 |Command or script that should be executed as the container's entrypoint. It's translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | | `command` | no | 9.4 |Command or script that should be used as the container's command. It's translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`](https://docs.docker.com/engine/reference/builder/#cmd) directive, where each shell token is a separate string in the array. | -| `alias` | no | 9.4 |Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | +| `alias` (1) | no | 9.4 |Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | -NOTE: **Note:** -Alias support for the Kubernetes executor was [introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2229) in GitLab Runner 12.8, and is only available for Kubernetes version 1.7 or later. +(1) Alias support for the Kubernetes executor was [introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2229) in GitLab Runner 12.8, and is only available for Kubernetes version 1.7 or later. ### Starting multiple services from the same image @@ -532,7 +529,6 @@ To define which should be used, the GitLab Runner process reads the configuratio If the `--user` flag is provided to run the GitLab Runner child processes as unprivileged user, the home directory of the main GitLab Runner process user is used. -NOTE: **Note:** GitLab Runner reads this configuration **only** from `config.toml` and ignores it if it's provided as an environment variable. This is because GitLab Runner uses **only** `config.toml` configuration and does not interpolate **ANY** environment variables at @@ -547,6 +543,7 @@ runtime. at least version **1.8** if you want to use private registries. - Available for [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html) in GitLab Runner 13.1 and later. +- [Credentials Store](#using-credentials-store) and [Credential Helpers](#using-credential-helpers) require binaries to be added to the GitLab Runner's `$PATH`, and require access to do so. Therefore, these features are not available on shared runners or any other runner where the user does not have access to the environment where the runner is installed. ### Using statically-defined credentials @@ -600,7 +597,7 @@ There are two ways to determine the value of `DOCKER_AUTH_CONFIG`: Open a terminal and execute the following command: ```shell - # Note the use of "-n" - it prevents encoding a newline in the password. + # The use of "-n" - prevents encoding a newline in the password. echo -n "my_username:my_password" | base64 # Example output to copy @@ -650,7 +647,6 @@ follow these steps: You can add configuration for as many registries as you want, adding more registries to the `"auths"` hash as described above. -NOTE: **Note:** The full `hostname:port` combination is required everywhere for the runner to match the `DOCKER_AUTH_CONFIG`. For example, if `registry.example.com:5000/namespace/image:tag` is specified in `.gitlab-ci.yml`, @@ -679,17 +675,14 @@ To add `DOCKER_AUTH_CONFIG` to a runner: environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"registry.example.com:5000\":{\"auth\":\"bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=\"}}}"] ``` -1. Restart the runner service. - -NOTE: **Note:** -The double quotes included in the `DOCKER_AUTH_CONFIG` -data must be escaped with backslashes. This prevents them from being -interpreted as TOML. + - The double quotes included in the `DOCKER_AUTH_CONFIG` + data must be escaped with backslashes. This prevents them from being + interpreted as TOML. + - The `environment` option is a list. Your runner may + have existing entries and you should add this to the list, not replace + it. -NOTE: **Note:** -The `environment` option is a list. So your runner may -have existing entries and you should add this to the list, not replace -it. +1. Restart the runner service. ### Using Credentials Store @@ -717,10 +710,9 @@ To configure credentials store, follow these steps: `${GITLAB_RUNNER_HOME}/.docker/config.json`. GitLab Runner reads this configuration file and uses the needed helper for this specific repository. -NOTE: **Note:** `credsStore` is used to access ALL the registries. -If you want to use both images from private registry and public images from DockerHub, -pulling from DockerHub would fail, because Docker daemon tries to use the same credentials for **ALL** the registries. +If you want to use both images from private registry and public images from Docker Hub, +pulling from Docker Hub would fail, because Docker daemon tries to use the same credentials for **ALL** the registries. ### Using Credential Helpers @@ -732,10 +724,8 @@ image which is private and requires you to log in into a private container regis To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow these steps: 1. Make sure `docker-credential-ecr-login` is available in GitLab Runner's `$PATH`. - 1. Have any of the following [AWS credentials setup](https://github.com/awslabs/amazon-ecr-credential-helper#aws-credentials). Make sure that GitLab Runner can access the credentials. - 1. Make GitLab Runner use it. There are two ways to accomplish this. Either: - Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) @@ -791,7 +781,6 @@ service containers. For all possible configuration variables check the documentation of each image provided in their corresponding Docker hub page. -NOTE: **Note:** All variables are passed to all services containers. It's not designed to distinguish which variable should go where. diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 07d0dac6163..589607fbed8 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -118,11 +118,9 @@ With this configuration, we: - Ensure that our app is able to be built successfully. - Lastly we deploy to the staging server. -NOTE: **Note:** -The `environment` keyword defines where the app is deployed. -The environment `name` and `url` is exposed in various places -within GitLab. Each time a job that has an environment specified -succeeds, a deployment is recorded, along with the Git SHA, and environment name. +Note that the `environment` keyword defines where the app is deployed. The environment `name` and +`url` is exposed in various places within GitLab. Each time a job that has an environment specified +succeeds, a deployment is recorded along with the Git SHA and environment name. CAUTION: **Caution:** Some characters are not allowed in environment names. Use only letters, @@ -288,12 +286,11 @@ You can find the "play" button in the pipelines, environments, deployments, and | Deployments |  | | Jobs |  | -Clicking on the play button in any view will trigger the `deploy_prod` job, and the -deployment will be recorded as a new environment named `production`. +Clicking the play button in any view triggers the `deploy_prod` job. The deployment is recorded as a +new environment named `production`. -NOTE: **Note:** -If your environment's name is `production` (all lowercase), -it will get recorded in [Value Stream Analytics](../../user/project/cycle_analytics.md). +If your environment's name is `production` (all lowercase), it's recorded in +[Value Stream Analytics](../../user/analytics/value_stream_analytics.md). ### Configuring dynamic environments @@ -371,9 +368,8 @@ For the value of: the example above: `https://$CI_COMMIT_REF_NAME.example.com`, which would give a URL of `https://100-do-the-thing.example.com`. -NOTE: **Note:** -You are not required to use the same prefix or only slashes (`/`) in the dynamic environments' -names. However, using this format will enable the [grouping similar environments](#grouping-similar-environments) +You aren't required to use the same prefix or only slashes (`/`) in the dynamic environments' names. +However, using this format enables the [grouping similar environments](#grouping-similar-environments) feature. ### Configuring Kubernetes deployments @@ -384,6 +380,12 @@ If you are deploying to a [Kubernetes cluster](../../user/project/clusters/index associated with your project, you can configure these deployments from your `gitlab-ci.yml` file. +NOTE: **Note:** +Kubernetes configuration isn't supported for Kubernetes clusters that are +[managed by GitLab](../../user/project/clusters/index.md#gitlab-managed-clusters). +To follow progress on support for GitLab-managed clusters, see the +[relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). + The following configuration options are supported: - [`namespace`](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) @@ -411,12 +413,6 @@ trace on the deployment job page:  -NOTE: **Note:** -Kubernetes configuration is not supported for Kubernetes clusters -that are [managed by GitLab](../../user/project/clusters/index.md#gitlab-managed-clusters). -To follow progress on support for GitLab-managed clusters, see the -[relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). - #### Configuring incremental rollouts Learn how to release production changes to only a portion of your Kubernetes pods with @@ -514,9 +510,8 @@ review_app: This example requires that NGINX and GitLab Runner are set up on the server this job will run on. -NOTE: **Note:** -See the [limitations](#limitations) section for some edge cases regarding the naming of -your branches and Review Apps. +See the [limitations](#limitations) section for some edge cases regarding the naming of your +branches and Review Apps. The complete example provides the following workflow to developers: @@ -617,13 +612,12 @@ To retry or rollback a deployment: #### What to expect with a rollback -Pressing the **Rollback** button on a specific commit will trigger a _new_ deployment with its -own unique job ID. - -This means that you will see a new deployment that points to the commit you are rolling back to. +Pressing the **Rollback** button on a specific commit triggers a _new_ deployment with its own +unique job ID. This means that you will see a new deployment that points to the commit you're +rolling back to. -NOTE: **Note:** -The defined deployment process in the job's `script` determines whether the rollback succeeds or not. +Note that the defined deployment process in the job's `script` determines whether the rollback +succeeds. ### Using the environment URL @@ -662,9 +656,8 @@ Stopping an environment: This is often used when multiple developers are working on a project at the same time, each of them pushing to their own branches, causing many dynamic environments to be created. -NOTE: **Note:** -Starting with GitLab 8.14, dynamic environments are stopped automatically -when their associated branch is deleted. +Starting with GitLab 8.14, dynamic environments stop automatically when their associated branch is +deleted. #### Automatically stopping an environment @@ -721,29 +714,25 @@ You can read more in the [`.gitlab-ci.yml` reference](../yaml/README.md#environm > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20956) in GitLab 12.8. -You can set a expiry time to environments and stop them automatically after a certain period. +You can set an expiry time for environments and stop them automatically after a certain period. -For example, consider the use of this feature with Review Apps environments. -When you set up Review Apps, sometimes they keep running for a long time -because some merge requests are left as open. An example for this situation is when the author of the merge -request is not actively working on it, due to priority changes or a different approach was decided on, and the merge requests was simply forgotten. -Idle environments waste resources, therefore they -should be terminated as soon as possible. +For example, consider the use of this feature with Review App environments. When you set up Review +Apps, sometimes they keep running for a long time because some merge requests are left open and +forgotten. Such idle environments waste resources and should be terminated as soon as possible. -To address this problem, you can specify an optional expiration date for -Review Apps environments. When the expiry time is reached, GitLab will automatically trigger a job -to stop the environment, eliminating the need of manually doing so. In case an environment is updated, the expiration is renewed -ensuring that only active merge requests keep running Review Apps. +To address this problem, you can specify an optional expiration date for Review App environments. +When the expiry time is reached, GitLab automatically triggers a job to stop the environment, +eliminating the need of manually doing so. In case an environment is updated, the expiration is +renewed ensuring that only active merge requests keep running Review Apps. -To enable this feature, you need to specify the [`environment:auto_stop_in`](../yaml/README.md#environmentauto_stop_in) keyword in `.gitlab-ci.yml`. -You can specify a human-friendly date as the value, such as `1 hour and 30 minutes` or `1 day`. -`auto_stop_in` uses the same format of [`artifacts:expire_in` docs](../yaml/README.md#artifactsexpire_in). +To enable this feature, you must specify the [`environment:auto_stop_in`](../yaml/README.md#environmentauto_stop_in) +keyword in `.gitlab-ci.yml`. You can specify a human-friendly date as the value, such as +`1 hour and 30 minutes` or `1 day`. `auto_stop_in` uses the same format of +[`artifacts:expire_in` docs](../yaml/README.md#artifactsexpire_in). -NOTE: **Note:** -Due to the resource limitation, a background worker for stopping environments only -runs once every hour. This means environments will not be stopped at the exact -timestamp as the specified period, but will be stopped when the hourly cron worker -detects expired environments. +Note that due to resource limitation, a background worker for stopping environments only runs once +every hour. This means that environments aren't stopped at the exact timestamp specified, but are +instead stopped when the hourly cron worker detects expired environments. ##### Auto-stop example @@ -903,7 +892,6 @@ you can monitor the behavior of your app running in each environment. For the mo dashboard to appear, you need to Configure Prometheus to collect at least one [supported metric](../../user/project/integrations/prometheus_library/index.md). -NOTE: **Note:** Since GitLab 9.2, all deployments to an environment are shown directly on the monitoring dashboard. Once configured, GitLab will attempt to retrieve [supported performance metrics](../../user/project/integrations/prometheus_library/index.md) @@ -938,6 +926,11 @@ This is a powerful feature that allows you to debug issues without leaving the c of your web browser. To enable it, just follow the instructions given in the service integration documentation. +NOTE: **Note:** +Container-based deployments often lack basic tools (like an editor), and may +be stopped or restarted at any time. If this happens, you will lose all your +changes. Treat this as a debugging tool, not a comprehensive online IDE. + Once enabled, your environments will gain a "terminal" button:  @@ -961,11 +954,6 @@ by your deployment so you can: You can open multiple terminals to the same environment, they each get their own shell session and even a multiplexer like `screen` or `tmux`. -NOTE: **Note:** -Container-based deployments often lack basic tools (like an editor), and may -be stopped or restarted at any time. If this happens, you will lose all your -changes. Treat this as a debugging tool, not a comprehensive online IDE. - ### Check out deployments locally Since GitLab 8.13, a reference in the Git repository is saved for each deployment, so @@ -1024,9 +1012,8 @@ As you can see, you can use specific matching for selecting a particular environ and also use wildcard matching (`*`) for selecting a particular environment group, such as [Review Apps](../review_apps/index.md) (`review/*`). -NOTE: **Note:** -The most _specific_ spec takes precedence over the other wildcard matching. -In this case, `review/feature-1` spec takes precedence over `review/*` and `*` specs. +Note that the most _specific_ spec takes precedence over the other wildcard matching. In this case, +the `review/feature-1` spec takes precedence over `review/*` and `*` specs. ### Environments Dashboard **(PREMIUM)** diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index 5dda0cc81f6..87bced29906 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -9,8 +9,6 @@ type: concepts, howto > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6303) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.3. -## Overview - [Environments](../environments/index.md) can be used for different reasons: - Some of them are just for testing. diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 73896547675..4a0ff2fa6ac 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -41,15 +41,14 @@ The JWT's payload looks like this: "nbf": 1585798372, # Not valid before "exp": 1585713886, # Expire at "sub": "job_1212", # Subject (job id) - "namespace_id": "1", - "namespace_path": "mygroup", - "project_id": "22", - "project_path": "mygroup/myproject", - "user_id": "42", - "user_login": "myuser", - "user_email": "myuser@example.com", - "pipeline_id": "1212", - "job_id": "1212", + "namespace_id": "1", # Use this to scope to group or user level namespace by id + "namespace_path": "mygroup", # Use this to scope to group or user level namespace by path + "project_id": "22", # + "project_path": "mygroup/myproject", # + "user_id": "42", # Id of the user executing the job + "user_email": "myuser@example.com", # Email of the user executing the job + "pipeline_id": "1212", # + "job_id": "1212", # "ref": "auto-deploy-2020-04-01", # Git ref for this job "ref_type": "branch", # Git ref type, branch or tag "ref_protected": "true" # true if this git ref is protected, false otherwise diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index 1f6c81a68aa..2866a34899a 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -9,7 +9,7 @@ type: tutorial [Review Apps](../../review_apps/index.md) are great: for every merge request (or branch, for that matter), the new code can be copied and deployed to a fresh production-like live -environment, making it incredibly low-effort to assess the impact of the changes. Thus, when we use a dependency manager like +environment, reducing the effort to assess the impact of changes. Thus, when we use a dependency manager like [Dependencies.io](https://www.dependencies.io/), it can submit a merge request with an updated dependency, and it will immediately be clear that the application can still be properly built and deployed. After all, you can _see_ it running! diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 8927c5c3480..d3f2d8223ef 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -397,8 +397,6 @@ To be able to build, test, and deploy our app with GitLab CI/CD, we need to prep To do that, we'll use a Docker image which has the minimum requirements that a Laravel app needs to run. [There are other ways](../php.md#test-php-projects-using-the-docker-executor) to do that as well, but they may lead our builds run slowly, which is not what we want when there are faster options to use. -With Docker images our builds run incredibly faster! - ### Create a Container Image Let's create a [Dockerfile](https://gitlab.com/mehranrasulian/laravel-sample/blob/master/Dockerfile) in the root directory of our app with the following content: @@ -441,9 +439,9 @@ On your GitLab project repository navigate to the **Registry** tab.  -You may need to [enable Container Registry](../../../user/packages/container_registry/index.md#enable-the-container-registry-for-your-project) to your project to see this tab. You'll find it under your project's **Settings > General > Visibility, project features, permissions**. +You may need to enable the Container Registry for your project to see this tab. You'll find it under your project's **Settings > General > Visibility, project features, permissions**. -To start using Container Registry on our machine, we first need to login to the GitLab registry using our GitLab username and password: +To start using Container Registry on our machine, we first need to sign in to the GitLab registry using our GitLab username and password: ```shell docker login registry.gitlab.com diff --git a/doc/ci/img/ci_lint.png b/doc/ci/img/ci_lint.png Binary files differindex e62de011293..fdc3868cdce 100644 --- a/doc/ci/img/ci_lint.png +++ b/doc/ci/img/ci_lint.png diff --git a/doc/ci/img/ci_lint_dry_run.png b/doc/ci/img/ci_lint_dry_run.png Binary files differindex 4092b66d534..61d6379f70e 100644 --- a/doc/ci/img/ci_lint_dry_run.png +++ b/doc/ci/img/ci_lint_dry_run.png diff --git a/doc/ci/img/gitlab_vault_workflow_v13_4.png b/doc/ci/img/gitlab_vault_workflow_v13_4.png Binary files differnew file mode 100644 index 00000000000..80d07362bf4 --- /dev/null +++ b/doc/ci/img/gitlab_vault_workflow_v13_4.png diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index db2749233e8..12df0751c47 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -154,7 +154,7 @@ commits to a feature branch in a remote repository in GitLab, the CI/CD pipeline set for your project is triggered. By doing so, GitLab CI/CD: -- Runs automated scripts (sequential or parallel) to: +- Runs automated scripts (sequentially or in parallel) to: - Build and test your app. - Preview the changes per merge request with Review Apps, as you would see in your `localhost`. diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index 4f4471225a0..53e097760e6 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -9,8 +9,6 @@ type: reference > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9788) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. Requires GitLab Runner 11.10 and above. -## Overview - GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [Unit test reports](unit_test_reports.md), [code quality](../user/project/merge_requests/code_quality.md), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. You can configure your job to use custom Metrics Reports, and GitLab will display a report on the merge request so that it's easier and faster to identify changes without having to check the entire log. diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 8dff99f7244..1130c11f472 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -23,7 +23,7 @@ that were able to quickly complete this migration: 1. Use the [Jenkins Wrapper](#jenkinsfile-wrapper) to temporarily maintain fragile Jenkins jobs. 1. Migrate the build and CI jobs and configure them to show results directly in your merge requests. They can use [Auto DevOps](../../topics/autodevops/index.md) as a starting point, and [customize](../../topics/autodevops/customize.md) or [decompose](../../topics/autodevops/customize.md#using-components-of-auto-devops) the configuration as needed. 1. Add [Review Apps](../review_apps/index.md). - 1. Migrate the deployment jobs using [cloud deployment templates](../cloud_deployment/index.md), adding [environments](../environments/index.md), and [deploy boards](../..//user/project/deploy_boards.md). + 1. Migrate the deployment jobs using [cloud deployment templates](../cloud_deployment/index.md), adding [environments](../environments/index.md), and [deploy boards](../../user/project/deploy_boards.md). 1. Work to unwrap any jobs still running with the use of the Jenkins wrapper. 1. Take stock of any common CI/CD job definitions then create and share [templates](#templates) for them. 1. Check the [pipeline efficiency documentation](../pipelines/pipeline_efficiency.md) @@ -83,7 +83,7 @@ There are some high level differences between the products worth mentioning: - You can control which jobs run in which cases, depending on how they are triggered, with the [`rules` syntax](../yaml/README.md#rules). -- GitLab [pipeline scheduling concepts](../pipelines/schedules.md) are also different than with Jenkins. +- GitLab [pipeline scheduling concepts](../pipelines/schedules.md) are also different from Jenkins. - You can reuse pipeline configurations using the [`include` keyword](../yaml/README.md#include) and [templates](#templates). Your templates can be kept in a central repository (with different permissions), and then any project can use them. This central project could also diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index 3f9a00b6cc8..a379a6e8211 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -280,3 +280,11 @@ Any pipelines that complete successfully for new tags in the subscribed project will now trigger a pipeline on the current project's default branch. The maximum number of upstream pipeline subscriptions is 2 by default, for both the upstream and downstream projects. This [application limit](../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project) can be changed on self-managed instances by a GitLab administrator. + +## Downstream private projects confidentiality concern + +If you trigger a pipeline in a downstream private project, the name of the project +and the status of the pipeline is visible in the upstream project's pipelines page. + +If you have a public project that can trigger downstream pipelines in a private +project, make sure to check that there are no confidentiality problems. diff --git a/doc/ci/parent_child_pipelines.md b/doc/ci/parent_child_pipelines.md index 83fa1d355e6..f2a4020cc4a 100644 --- a/doc/ci/parent_child_pipelines.md +++ b/doc/ci/parent_child_pipelines.md @@ -68,7 +68,7 @@ microservice_a: trigger: include: - local: path/to/microservice_a.yml - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml ``` NOTE: **Note:** @@ -82,7 +82,7 @@ microservice_a: trigger: include: - local: path/to/microservice_a.yml - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml strategy: depend ``` diff --git a/doc/ci/pipelines/img/ci_efficiency_pipeline_dag_critical_path.png b/doc/ci/pipelines/img/ci_efficiency_pipeline_dag_critical_path.png Binary files differindex 1715e8224ab..421fddaf38d 100644 --- a/doc/ci/pipelines/img/ci_efficiency_pipeline_dag_critical_path.png +++ b/doc/ci/pipelines/img/ci_efficiency_pipeline_dag_critical_path.png diff --git a/doc/ci/pipelines/img/ci_efficiency_pipeline_health_grafana_dashboard.png b/doc/ci/pipelines/img/ci_efficiency_pipeline_health_grafana_dashboard.png Binary files differindex 0956e76804e..59276bda727 100644 --- a/doc/ci/pipelines/img/ci_efficiency_pipeline_health_grafana_dashboard.png +++ b/doc/ci/pipelines/img/ci_efficiency_pipeline_health_grafana_dashboard.png diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index 1b9048089bd..f172032630d 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -461,6 +461,28 @@ this line should be hidden when collapsed section_end:1560896353:my_first_section\r\e[0K ``` +#### Pre-collapse sections + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198413) in GitLab 13.5. + +You can make the job log automatically collapse collapsible sections by adding the `collapsed` option to the section start. +Add `[collapsed=true]` after the section name and before the `\r`. The section end marker +remains unchanged: + +- Section start marker with `[collapsed=true]`: `section_start:UNIX_TIMESTAMP:SECTION_NAME[collapsed=true]\r\e[0K` + `TEXT_OF_SECTION_HEADER` +- Section end marker: `section_end:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` + +Add the updated section start text to the CI configuration. For example, +using `echo`: + +```yaml +job1: + script: + - echo -e "section_start:`date +%s`:my_first_section[collapsed=true]\r\e[0KHeader of the 1st collapsible section" + - echo 'this line should be hidden automatically after loading the job log' + - echo -e "section_end:`date +%s`:my_first_section\r\e[0K" +``` + ## Visualize pipelines > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5742) in GitLab 8.11. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index 750a76bfaa0..39633163ac7 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -431,6 +431,17 @@ To erase a job: In order to retrieve a job artifact of a different project, you might need to use a private token in order to [authenticate and download](../../api/job_artifacts.md#get-job-artifacts) the artifacts. +## Troubleshooting + +### Error message `No files to upload` + +This is often preceded by other errors or warnings that specify the filename and why it wasn't +generated in the first place. Please check the entire job log for such messages. + +If you find no helpful messages, please retry the failed job after activating +[CI debug logging](../variables/README.md#debug-logging). +This provides useful information to investigate further. + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 849eb66d07f..4dcdf469e5c 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -37,10 +37,10 @@ in `.gitlab-ci.yml`. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/28919) in GitLab 12.0. NOTE: **Note:** -As of GitLab 12.0, newly created projects will automatically have a default +As of GitLab 12.0, newly created projects automatically have a default `git depth` value of `50`. -It is possible to limit the number of changes that GitLab CI/CD will fetch when cloning +It is possible to limit the number of changes that GitLab CI/CD fetches when cloning a repository. Setting a limit to `git depth` can speed up Pipelines execution. Maximum allowed value is `1000`. @@ -75,7 +75,7 @@ For information about setting a maximum artifact size for a project, see > - [Support for external `.gitlab-ci.yml` locations](https://gitlab.com/gitlab-org/gitlab/-/issues/14376) introduced in GitLab 12.6. By default we look for the `.gitlab-ci.yml` file in the project's root -directory. If needed, you can specify an alternate path and file name, including locations outside the project. +directory. If needed, you can specify an alternate path and filename, including locations outside the project. To customize the path: @@ -93,12 +93,12 @@ paths and file names include: - `my/path/.gitlab-ci.yml` - `my/path/.my-custom-file.yml` -If the CI configuration will be hosted on an external site, the URL link must end with `.yml`: +If hosting the CI configuration on an external site, the URL link must end with `.yml`: - `http://example.com/generate/ci/config.yml` -If the CI configuration will be hosted in a different project within GitLab, the path must be relative -to the root directory in the other project, with the group and project name added to the end: +If hosting the CI configuration in a different project within GitLab, the path must be relative +to the root directory in the other project. Include the group and project name at the end: - `.gitlab-ci.yml@mygroup/another-project` - `my/path/.my-custom-file.yml@mygroup/another-project` @@ -109,7 +109,7 @@ configuration file. For example: - Create a public project to host the configuration file. - Give write permissions on the project only to users who are allowed to edit the file. -Other users and projects will be able to access the configuration file without being +Other users and projects can access the configuration file without being able to edit it. ## Test coverage parsing @@ -125,8 +125,8 @@ can use <https://rubular.com> to test your regex. The regex returns the **last** match found in the output. If the pipeline succeeds, the coverage is shown in the merge request widget and -in the jobs table. If multiple jobs in the pipeline have coverage reports, they will -be averaged. +in the jobs table. If multiple jobs in the pipeline have coverage reports, they are +averaged.  @@ -140,7 +140,7 @@ in the pipelines settings page. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209121) the ability to download a `.csv` in GitLab 12.10. > - [Graph introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33743) in GitLab 13.1. -If you want to see the evolution of your project code coverage over time, +To see the evolution of your project code coverage over time, you can view a graph or download a CSV file with this data. From your project: 1. Go to **{chart}** **Project Analytics > Repository** to see the historic data for each job listed in the dropdown above the graph. @@ -150,13 +150,13 @@ you can view a graph or download a CSV file with this data. From your project: ### Removing color codes -Some test coverage tools output with ANSI color codes that won't be -parsed correctly by the regular expression and will cause coverage +Some test coverage tools output with ANSI color codes that aren't +parsed correctly by the regular expression. This causes coverage parsing to fail. -If your coverage tool doesn't provide an option to disable color -codes in the output, you can pipe the output of the coverage tool through a -small one line script that will strip the color codes off. +Some coverage tools don't provide an option to disable color +codes in the output. If so, pipe the output of the coverage tool through a +small one line script that strips the color codes off. For example: @@ -172,7 +172,7 @@ Pipeline visibility is determined by: - The **Public pipelines** project setting under your project's **Settings > CI/CD > General pipelines**. NOTE: **Note:** -If the project visibility is set to **Private**, the [**Public pipelines** setting will have no effect](../enable_or_disable_ci.md#per-project-user-setting). +If the project visibility is set to **Private**, the [**Public pipelines** setting has no effect](../enable_or_disable_ci.md#per-project-user-setting). This also determines the visibility of these related features: @@ -204,16 +204,14 @@ If **Public pipelines** is disabled: > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9362) in GitLab 9.1. -If you want all pending non-HEAD pipelines on branches to auto-cancel each time -a new pipeline is created, such as after a Git push or manually from the UI, -you can enable this in the project settings: +You can set pending or running pipelines to cancel automatically when a new pipeline runs on the same branch. You can enable this in the project settings: 1. Go to **Settings > CI / CD**. 1. Expand **General Pipelines**. 1. Check the **Auto-cancel redundant, pending pipelines** checkbox. 1. Click **Save changes**. -Note that only jobs with [interruptible](../yaml/README.md#interruptible) set to `true` will be cancelled. +Note that only jobs with [interruptible](../yaml/README.md#interruptible) set to `true` are cancelled. ## Skip outdated deployment jobs @@ -232,18 +230,18 @@ To avoid this scenario: 1. Check the **Skip outdated deployment jobs** checkbox. 1. Click **Save changes**. -The pending deployment jobs will be skipped. +When enabled, any older deployments job are skipped when a new deployment starts. For more information, see [Deployment safety](../environments/deployment_safety.md). ## Pipeline Badges In the pipelines settings page you can find pipeline status and test coverage -badges for your project. The latest successful pipeline will be used to read +badges for your project. The latest successful pipeline is used to read the pipeline status and test coverage values. Visit the pipelines settings page in your project to see the exact link to -your badges, as well as ways to embed the badge image in your HTML or Markdown +your badges. You can also see ways to embed the badge image in your HTML or Markdown pages.  @@ -276,8 +274,8 @@ https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg?ig ### Test coverage report badge -GitLab makes it possible to define the regular expression for [coverage report](#test-coverage-parsing), -that each job log will be matched against. This means that each job in the +GitLab makes it possible to define the regular expression for the [coverage report](#test-coverage-parsing), +that each job log is matched against. This means that each job in the pipeline can have the test coverage percentage value defined. The test coverage badge can be accessed using following link: @@ -288,7 +286,7 @@ https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg If you would like to get the coverage report from a specific job, you can add the `job=coverage_job_name` parameter to the URL. For example, the following -Markdown code will embed the test coverage report badge of the `coverage` job +Markdown code embeds the test coverage report badge of the `coverage` job into your `README.md`: ```markdown @@ -297,7 +295,7 @@ into your `README.md`: ### Badge styles -Pipeline badges can be rendered in different styles by adding the `style=style_name` parameter to the URL. Currently two styles are available: +Pipeline badges can be rendered in different styles by adding the `style=style_name` parameter to the URL. Two styles are available: #### Flat (default) diff --git a/doc/ci/review_apps/img/toolbar_feeback_form.png b/doc/ci/review_apps/img/toolbar_feeback_form.png Binary files differdeleted file mode 100644 index fe1c7e6e611..00000000000 --- a/doc/ci/review_apps/img/toolbar_feeback_form.png +++ /dev/null diff --git a/doc/ci/review_apps/img/toolbar_feedback_form_v13_5.png b/doc/ci/review_apps/img/toolbar_feedback_form_v13_5.png Binary files differnew file mode 100644 index 00000000000..dc4a13b2152 --- /dev/null +++ b/doc/ci/review_apps/img/toolbar_feedback_form_v13_5.png diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index e2d5cbcbea4..7110117709f 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -33,7 +33,7 @@ In the above example: ## How Review Apps work A Review App is a mapping of a branch with an [environment](../environments/index.md). -Access to the Review App is made available as a link on the [merge request](../../user/project/merge_requests.md) relevant to the branch. +Access to the Review App is made available as a link on the [merge request](../../user/project/merge_requests/index.md) relevant to the branch. The following is an example of a merge request with an environment set dynamically. @@ -188,20 +188,35 @@ Once you have the route mapping set up, it will take effect in the following loc ## Visual Reviews **(STARTER)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10761) in GitLab Starter 12.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10761) in GitLab Starter 12.0. +> - It's [deployed behind a feature flag](../../user/feature_flags.md), enabled by default. +> - It's enabled on GitLab.com. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-visual-reviews). **(STARTER ONLY)** -With Visual Reviews, you can provide a feedback form to your Review Apps so -that reviewers can post comments directly from the app back to the merge request -that spawned the Review App. +With Visual Reviews, members of any team (Product, Design, Quality, and so on) can provide feedback comments through a form in your review apps. The comments are added to the merge request that triggered the review app. -### Configuring Visual Reviews +### Using Visual Reviews -Ensure that the `anonymous_visual_review_feedback` feature flag is enabled. -Administrators can enable with a Rails console as follows: +After Visual Reviews has been [configured](#configure-review-apps-for-visual-reviews) for the +Review App, the Visual Reviews feedback form is overlaid on the right side of every page. -```ruby -Feature.enable(:anonymous_visual_review_feedback) -``` + + +To use the feedback form to make a comment in the merge request: + +1. Click the **Review** tab on the right side of a page. +1. Make a comment on the visual review. You can make use of all the + [Markdown annotations](../../user/markdown.md) that are also available in + merge request comments. +1. Enter your personal information: + - If [`data-require-auth`](#authentication-for-visual-reviews) is `true`, you must enter your [personal access token](../../user/profile/personal_access_tokens.md). + - Otherwise, enter your name, and optionally your email. +1. Click **Send feedback**. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +To see Visual reviews in action, see the [Visual Reviews Walk through](https://youtu.be/1_tvWTlPfM4). + +### Configure Review Apps for Visual Reviews The feedback form is served through a script you add to pages in your Review App. If you have [Developer permissions](../../user/permissions.md) to the project, @@ -212,18 +227,18 @@ if [route maps](#route-maps) are configured in the project.  The provided script should be added to the `<head>` of your application and -consists of some project and merge request specific values. Here's what it -looks like: +consists of some project and merge request specific values. Here's how it +looks for a project with code hosted in a project on GitLab.com: ```html <script data-project-id='11790219' data-merge-request-id='1' - data-mr-url='https://gitlab.example.com' + data-mr-url='https://gitlab.com' data-project-path='sarah/review-app-tester' data-require-auth='true' id='review-app-toolbar-script' - src='https://gitlab.example.com/assets/webpack/visual_review_toolbar.js'> + src='https://gitlab.com/assets/webpack/visual_review_toolbar.js'> </script> ``` @@ -239,21 +254,21 @@ to replace those values at runtime when each review app is created: - `data-mr-url` is the URL of the GitLab instance and will be the same for all review apps. - `data-project-path` is the project's path, which can be found by `CI_PROJECT_PATH`. -- `data-require-auth` is optional for public projects but required for [private and internal ones](#visual-reviews-in-private-or-internal-projects). If this is set to `true`, the user will be required to enter their [personal access token](../../user/profile/personal_access_tokens.md) instead of their name and email. +- `data-require-auth` is optional for public projects but required for [private and internal ones](#authentication-for-visual-reviews). If this is set to `true`, the user will be required to enter their [personal access token](../../user/profile/personal_access_tokens.md) instead of their name and email. - `id` is always `review-app-toolbar-script`, you don't need to change that. - `src` is the source of the review toolbar script, which resides in the respective GitLab instance and will be the same for all review apps. -For example, in a Ruby application, you would need to have this script: +For example, in a Ruby application with code hosted on in a project GitLab.com, you would need to have this script: ```html <script data-project-id="ENV['CI_PROJECT_ID']" data-merge-request-id="ENV['CI_MERGE_REQUEST_IID']" - data-mr-url='https://gitlab.example.com' + data-mr-url='https://gitlab.com' data-project-path="ENV['CI_PROJECT_PATH']" id='review-app-toolbar-script' - src='https://gitlab.example.com/assets/webpack/visual_review_toolbar.js'> + src='https://gitlab.com/assets/webpack/visual_review_toolbar.js'> </script> ``` @@ -273,33 +288,34 @@ can supply the ID by either: - Dynamically adding the `data-merge-request-id` value during the build of the app. - Supplying it manually through the visual review form in the app. -### Visual Reviews in private or internal projects +### Enable or disable Visual Reviews **(STARTER ONLY)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/42750#note_317271120) in GitLab 12.10. +Visual Reviews is deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can opt to disable it. -To enable visual reviews for private and internal projects, set the -[`data-require-auth` variable](#configuring-visual-reviews) to `true`. When enabled, -the user must enter a [personal access token](../../user/profile/personal_access_tokens.md) -with `api` scope before submitting feedback. +To disable it: -### Using Visual Reviews +```ruby +Feature.disable(:anonymous_visual_review_feedback) +``` -After Visual Reviews has been [enabled](#configuring-visual-reviews) for the -Review App, the Visual Reviews feedback form is overlaid on the app's pages at -the bottom-right corner. +To enable it: - +```ruby +Feature.enable(:anonymous_visual_review_feedback) +``` -To use the feedback form: +### Authentication for Visual Reviews -1. Make a comment on the visual review. You can make use of all the - [Markdown annotations](../../user/markdown.md) that are also available in - merge request comments. -1. If `data-require-auth` is `true`, you must enter your [personal access token](../../user/profile/personal_access_tokens.md). Otherwise, you must enter your name, and optionally, your email. -1. Finally, click **Send feedback**. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/42750#note_317271120) in GitLab 12.10. + +To enable visual reviews for private and internal projects, set the +[`data-require-auth` variable](#enable-or-disable-visual-reviews) to `true`. When enabled, +the user must enter a [personal access token](../../user/profile/personal_access_tokens.md) +with `api` scope before submitting feedback. -After you make and submit a comment in the visual review box, it will appear -automatically in the respective merge request. +This same method can be used to require authentication for any public projects. ## Limitations diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index 32561e6b98c..afcbd68aaf5 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -39,11 +39,10 @@ multiple projects. If you are using a self-managed instance of GitLab: -- Your administrator can install and register shared runners by viewing the instructions - [here](https://docs.gitlab.com/runner/install/index.html). +- Your administrator can install and register shared runners by [following the documentation](https://docs.gitlab.com/runner/install/index.html). <!-- going to your project's <!-- **Settings > CI / CD**, expanding the **Runners** section, and clicking **Show runner installation instructions**.--> - <!-- These instructions are also available [here](https://docs.gitlab.com/runner/install/index.html).--> + <!-- These instructions are also available [in the documentation](https://docs.gitlab.com/runner/install/index.html).--> - The administrator can also configure a maximum number of shared runner [pipeline minutes for each group](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota). diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index 6d561fe00a3..fb1183cd68b 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -17,23 +17,36 @@ Unlike CI variables, which are always presented to a job, secrets must be explic required by a job. Read [GitLab CI/CD pipeline configuration reference](../yaml/README.md#secrets) for more information about the syntax. -GitLab has selected [Vault by Hashicorp](https://www.vaultproject.io) as the +GitLab has selected [Vault by HashiCorp](https://www.vaultproject.io) as the first supported provider, and [KV-V2](https://www.vaultproject.io/docs/secrets/kv/kv-v2) as the first supported secrets engine. GitLab authenticates using Vault's -[JWT Auth method](https://www.vaultproject.io/docs/auth/jwt#jwt-authentication), using +[JSON Web Token (JWT) authentication method](https://www.vaultproject.io/docs/auth/jwt#jwt-authentication), using the [JSON Web Token](https://gitlab.com/gitlab-org/gitlab/-/issues/207125) (`CI_JOB_JWT`) introduced in GitLab 12.10. You must [configure your Vault server](#configure-your-vault-server) before you can use [use Vault secrets in a CI job](#use-vault-secrets-in-a-ci-job). +The flow for using GitLab with HashiCorp Vault +is summarized by this diagram: + + + +1. Configure your vault and secrets. +1. Generate your JWT and provide it to your CI job. +1. Runner contacts HashiCorp Vault and authenticates using the JWT. +1. HashiCorp Vault verifies the JWT. +1. HashiCorp Vault checks the bounded claims and attaches policies. +1. HashiCorp Vault returns the token. +1. Runner reads secrets from the HashiCorp Vault. + NOTE: **Note:** -Read the [Authenticating and Reading Secrets With Hashicorp Vault](../examples/authenticating-with-hashicorp-vault/index.md) -tutorial for a version of this feature that is available to all +Read the [Authenticating and Reading Secrets With HashiCorp Vault](../examples/authenticating-with-hashicorp-vault/index.md) +tutorial for a version of this feature. It's available to all subscription levels, supports writing secrets to and deleting secrets from Vault, -and multiple secrets engines. +and supports multiple secrets engines. ## Configure your Vault server @@ -90,7 +103,7 @@ the secrets stored in Vault by defining them with the `vault` keyword: ```yaml secrets: DATABASE_PASSWORD: - vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password` + vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password` ``` In this example: @@ -149,7 +162,7 @@ generated by this GitLab instance may be allowed to authenticate using this role For a full list of `CI_JOB_JWT` claims, read the [How it works](../examples/authenticating-with-hashicorp-vault/index.md#how-it-works) section of the -[Authenticating and Reading Secrets With Hashicorp Vault](../examples/authenticating-with-hashicorp-vault/index.md) tutorial. +[Authenticating and Reading Secrets With HashiCorp Vault](../examples/authenticating-with-hashicorp-vault/index.md) tutorial. You can also specify some attributes for the resulting Vault tokens, such as time-to-live, IP address range, and number of uses. The full list of options is available in diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 2b006b8779b..d88a6afbadb 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -95,9 +95,9 @@ Read more about the [jobs API](../../api/job_artifacts.md#download-the-artifacts ## Adding a new trigger -You can add a new trigger by going to your project's -**Settings ➔ CI/CD** under **Triggers**. The **Add trigger** button will -create a new token which you can then use to trigger a rerun of this +Go to your +**Settings ➔ CI/CD** under **Triggers** to add a new trigger. The **Add trigger** button creates +a new token which you can then use to trigger a rerun of this particular project's pipeline. Every new trigger you create, gets assigned a different token which you can @@ -121,7 +121,7 @@ POST /projects/:id/trigger/pipeline ``` The required parameters are the [trigger's `token`](#authentication-tokens) -and the Git `ref` on which the trigger will be performed. Valid refs are +and the Git `ref` on which the trigger is performed. Valid refs are branches or tags. The `:id` of a project can be found by [querying the API](../../api/projects.md) or by visiting the **CI/CD** settings page which provides self-explanatory examples. @@ -146,7 +146,7 @@ curl --request POST \ https://gitlab.example.com/api/v4/projects/9/trigger/pipeline ``` -In this case, the project with ID `9` will get rebuilt on `master` branch. +In this case, the project with ID `9` gets rebuilt on `master` branch. Alternatively, you can pass the `token` and `ref` arguments in the query string: @@ -169,9 +169,9 @@ build_docs: - tags ``` -This means that whenever a new tag is pushed on project A, the job will run and the -`build_docs` job will be executed, triggering a rebuild of project B. The -`stage: deploy` ensures that this job will run only after all jobs with +This means that whenever a new tag is pushed on project A, the job runs and the +`build_docs` job is executed, triggering a rebuild of project B. The +`stage: deploy` ensures that this job runs only after all jobs with `stage: test` complete successfully. ## Triggering a pipeline from a webhook @@ -190,7 +190,7 @@ trigger in the source repository. `ref` should be URL-encoded if it contains sla ## Making use of trigger variables You can pass any number of arbitrary variables in the trigger API call and they -will be available in GitLab CI/CD so that they can be used in your `.gitlab-ci.yml` +are available in GitLab CI/CD so that they can be used in your `.gitlab-ci.yml` file. The parameter is of the form: ```plaintext @@ -237,7 +237,7 @@ upload_package: ``` You can then trigger a rebuild while you pass the `UPLOAD_TO_S3` variable -and the script of the `upload_package` job will run: +and the script of the `upload_package` job is run: ```shell curl --request POST \ @@ -266,7 +266,7 @@ branch of project with ID `9` every night at `00:30`: ## Legacy triggers -Old triggers, created before GitLab 9.0 will be marked as legacy. +Old triggers, created before GitLab 9.0 are marked as legacy. Triggers with the legacy label do not have an associated user and only have access to the current project. They are considered deprecated and will be diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index 5a59a175a89..79da54ccffe 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -10,8 +10,6 @@ type: reference > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. > - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39737) from JUnit test reports to Unit test reports in GitLab 13.4. -## Overview - It is very common that a [CI/CD pipeline](pipelines/index.md) contains a test job that will verify your code. If the tests fail, the pipeline fails and users get notified. The person that diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 1a982fa4e19..c83065aebaa 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -103,8 +103,8 @@ variables: You can then call its value in your script: ```yaml - script: - - echo "$TEST" +script: + - echo "$TEST" ``` For more details, see [`.gitlab-ci.yml` defined variables](#gitlab-ciyml-defined-variables). @@ -571,7 +571,7 @@ in which you wish to use it. ## Where variables can be used -Click [here](where_variables_can_be_used.md) for a section that describes where and how the different types of variables can be used. +[This section](where_variables_can_be_used.md) describes where and how the different types of variables can be used. ## Advanced use @@ -609,8 +609,8 @@ then available as environment variables on the running application container. CAUTION: **Caution:** -Variables with multi-line values are not currently supported due to -limitations with the current Auto DevOps scripting environment. +Variables with multi-line values are not supported due to +limitations with the Auto DevOps scripting environment. ### Override a variable by manually running a pipeline @@ -785,7 +785,7 @@ Examples: ##### Enable or disable parenthesis support for variables **(CORE ONLY)** -The feature is currently deployed behind a feature flag that is **enabled by default**. +The feature is deployed behind a feature flag that is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) can opt to disable it for your instance. @@ -820,8 +820,7 @@ NOTE: **Note:** The available regular expression syntax is limited. See [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/35438) for more details. -If needed, you can use a test pipeline to determine whether a regular expression will -work in a variable. The example below tests the `^mast.*` regular expression directly, +If needed, you can use a test pipeline to determine whether a regular expression works in a variable. The example below tests the `^mast.*` regular expression directly, as well as from within a variable: ```yaml diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 915041b71a6..7341c7a917d 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -69,7 +69,8 @@ Kubernetes-specific environment variables are detailed in the | `CI_JOB_MANUAL` | 8.12 | all | The flag to indicate that job was manually started | | `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | | `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | -| `CI_JOB_TOKEN` | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry](../../user/packages/container_registry/index.md), downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories), and accessing [GitLab-managed Terraform state](../../user/infrastructure/index.md#gitlab-managed-terraform-state). | +| `CI_JOB_STATUS` | all | 13.5 | The state of the job as each runner stage is executed. Use with [`after_script`](../yaml/README.md#before_script-and-after_script) where `CI_JOB_STATUS` can be either: `success`, `failed` or `canceled`. | +| `CI_JOB_TOKEN` | 9.0 | 1.2 | Token used for authenticating with [a few API endpoints](../../api/README.md#gitlab-ci-job-token) and downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories). The token is valid as long as the job is running. | | `CI_JOB_JWT` | 12.10 | all | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../secrets/index.md). | | `CI_JOB_URL` | 11.1 | 0.5 | Job details URL | | `CI_KUBERNETES_ACTIVE` | 13.0 | all | Included with the value `true` only if the pipeline has a Kubernetes cluster available for deployments. Not included if no cluster is available. Can be used as an alternative to [`only:kubernetes`/`except:kubernetes`](../yaml/README.md#onlykubernetesexceptkubernetes) with [`rules:if`](../yaml/README.md#rulesif) | diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 40df9c7c986..98ff5c22e7c 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -33,11 +33,8 @@ We have complete examples of configuring pipelines: > - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Learn how [Verizon reduced rebuilds](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) > from 30 days to under 8 hours with GitLab. -NOTE: **Note:** If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository), -you may need to enable pipeline triggering. Go to your project's - -**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. +you may need to enable pipeline triggering. Go to your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. ## Introduction @@ -961,7 +958,7 @@ GitLab performs a reverse deep merge based on the keys. GitLab: - Merges the `rspec` contents into `.tests` recursively. - Doesn't merge the values of the keys. -The result is this `rspec` job: +The result is this `rspec` job, where `script: rake test` is overwritten by `script: rake rspec`: ```yaml rspec: @@ -974,9 +971,6 @@ rspec: - $RSPEC ``` -NOTE: **Note:** -Note that `script: rake test` has been overwritten by `script: rake rspec`. - If you do want to include the `rake test`, see [`before_script` and `after_script`](#before_script-and-after_script). `.tests` in this example is a [hidden job](#hide-jobs), but it's @@ -1547,11 +1541,11 @@ docker build: - Dockerfile - docker/scripts/* when: manual - # - when: never would be redundant here, this is implied any time rules are listed. + # - when: never would be redundant here, this is implied any time rules are listed. ``` -Keywords such as `branches` or `refs` that are currently available for -`only`/`except` are not yet available in `rules` as they are being individually +Keywords such as `branches` or `refs` that are available for +`only`/`except` are not available in `rules`. They are being individually considered for their usage and behavior in this context. Future keyword improvements are being discussed in our [epic for improving `rules`](https://gitlab.com/groups/gitlab-org/-/epics/2783), where anyone can add suggestions or requests. @@ -1754,7 +1748,7 @@ the pipeline if the following is true: In the example below, the `test` job will `only` be created when **all** of the following are true: -- The pipeline has been [scheduled](../../user/project/pipelines/schedules.md) **or** runs for `master`. +- The pipeline has been [scheduled](../pipelines/schedules.md) **or** runs for `master`. - The `variables` keyword matches. - The `kubernetes` service is active on the project. @@ -3578,6 +3572,11 @@ job split into three separate jobs. Use `matrix:` to configure different variables for jobs that are running in parallel. There can be from 2 to 50 jobs. +In GitLab 13.5 and later, you can have one-dimensional matrices with a single job. +The ability to have one-dimensional matrices is [deployed behind a feature flag](../../user/feature_flags.md), +disabled by default. It's disabled on GitLab.com. To use it in a GitLab self-managed +instance, ask a GitLab administrator to [enable the `one_dimensional_matrix:` feature flag](../../administration/feature_flags.md). **(CORE-ONLY)** + Every job gets the same `CI_NODE_TOTAL` [environment variable](../variables/README.md#predefined-environment-variables) value, and a unique `CI_NODE_INDEX` value. ```yaml @@ -4056,7 +4055,9 @@ release-cli create --name "Release $CI_COMMIT_SHA" --description "Created using > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33014) in GitLab 13.4. `secrets` indicates the [CI Secrets](../secrets/index.md) this job needs. It should be a hash, -and the keys should be the names of the environment variables the job needs to access the secrets. +and the keys should be the names of the environment variables that are made available to the job. +The value of each secret is saved in a temporary file. This file's path is stored in these +environment variables. #### `secrets:vault` **(PREMIUM)** @@ -4072,7 +4073,7 @@ field to fetch the value for: job: secrets: DATABASE_PASSWORD: - vault: production/db/password # translates to secret `kv-v2/data/production/db`, field `password` + vault: production/db/password # translates to secret `kv-v2/data/production/db`, field `password` ``` You can specify a custom secrets engine path by adding a suffix starting with `@`: @@ -4081,7 +4082,7 @@ You can specify a custom secrets engine path by adding a suffix starting with `@ job: secrets: DATABASE_PASSWORD: - vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password` + vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password` ``` In the detailed form of the syntax, you can specify all details explicitly: diff --git a/doc/customization/welcome_message.md b/doc/customization/welcome_message.md index 45f1fed355e..9667e5e380a 100644 --- a/doc/customization/welcome_message.md +++ b/doc/customization/welcome_message.md @@ -1,5 +1,5 @@ --- -redirect_to: 'branded_login_page.md' +redirect_to: '../user/admin_area/appearance.md#sign-in--sign-up-pages' --- -This document was moved to [another location](branded_login_page.md). +This document was moved to [another location](../user/admin_area/appearance.md#sign-in--sign-up-pages). diff --git a/doc/development/README.md b/doc/development/README.md index abdd5c662f3..ee4643ed5a5 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -43,6 +43,7 @@ For information on how to install, configure, update, and upgrade your own GitLa **Must-reads:** +- [Guide on adapting existing and introducing new components](architecture.md#adapting-existing-and-introducing-new-components) - [Code review guidelines](code_review.md) for reviewing code and having code reviewed - [Database review guidelines](database_review.md) for reviewing database-related changes and complex SQL queries, and having them reviewed - [Secure coding guidelines](secure_coding_guidelines.md) diff --git a/doc/development/adding_service_component.md b/doc/development/adding_service_component.md index 2801e27145d..bb5af286c1d 100644 --- a/doc/development/adding_service_component.md +++ b/doc/development/adding_service_component.md @@ -46,7 +46,7 @@ TIP: **Tip:** **For services that depend on the existing GitLab codebase:** -The first iteration should be opt-in, either through the `gitlab.yml` configuration or through [feature flags](feature_flags.md). For these types of services it is often necessary to [bundle the service and its dependencies with GitLab](#bundling-a-service-with-gitlab) as part of the initial integration. +The first iteration should be opt-in, either through the `gitlab.yml` configuration or through [feature flags](feature_flags/index.md). For these types of services it is often necessary to [bundle the service and its dependencies with GitLab](#bundling-a-service-with-gitlab) as part of the initial integration. TIP: **Tip:** [ActionCable](https://docs.gitlab.com/omnibus/settings/actioncable.html) is an example of a service that has been added this way. diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 18fc0fb7d33..d083ed8f67e 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -49,6 +49,20 @@ See also: - [Exposing Global IDs](#exposing-global-ids). - [Mutation arguments](#object-identifier-arguments). +We have a custom scalar type (`Types::GlobalIDType`) which should be used as the +type of input and output arguments when the value is a `GlobalID`. The benefits +of using this type instead of `ID` are: + +- it validates that the value is a `GlobalID` +- it parses it into a `GlobalID` before passing it to user code +- it can be parameterized on the type of the object (e.g. + `GlobalIDType[Project]`) which offers even better validation and security. + +Consider using this type for all new arguments and result types. Remember that +it is perfectly possible to parameterize this type with a concern or a +supertype, if you want to accept a wider range of objects (e.g. +`GlobalIDType[Issuable]` vs `GlobalIDType[Issue]`). + ## Types We use a code-first schema, and we declare what type everything is in Ruby. @@ -780,6 +794,25 @@ to advertise the need for lookahead: For an example of real world use, please see [`ResolvesMergeRequests`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/resolvers/concerns/resolves_merge_requests.rb). +### Negated arguments + +Negated filters can filter some resources (for example, find all issues that +have the `bug` label, but don't have the `bug2` label assigned). The `not` +argument is the preferred syntax to pass negated arguments: + +```graphql +issues(labelName: "bug", not: {labelName: "bug2"}) { + nodes { + id + title + } +} +``` + +To avoid duplicated argument definitions, you can place these arguments in a reusable module (or +class, if the arguments are nested). Alternatively, you can consider to add a +[helper resolver method](https://gitlab.com/gitlab-org/gitlab/-/issues/258969). + ## Pass a parent object into a child Presenter Sometimes you need to access the resolved query parent in a child context to compute fields. Usually the parent is only @@ -827,10 +860,39 @@ mutation. ### Building Mutations -Mutations live in `app/graphql/mutations` ideally grouped per +Mutations are stored in `app/graphql/mutations`, ideally grouped per resources they are mutating, similar to our services. They should inherit `Mutations::BaseMutation`. The fields defined on the mutation -will be returned as the result of the mutation. +are returned as the result of the mutation. + +#### Update mutation granularity + +GitLab's service-oriented architecture means that most mutations call a Create, Delete, or Update +service, for example `UpdateMergeRequestService`. +For Update mutations, a you might want to only update one aspect of an object, and thus only need a +_fine-grained_ mutation, for example `MergeRequest::SetWip`. + +It's acceptable to have both fine-grained mutations and coarse-grained mutations, but be aware +that too many fine-grained mutations can lead to organizational challenges in maintainability, code +comprehensibility, and testing. +Each mutation requires a new class, which can lead to technical debt. +It also means the schema becomes very big, and we want users to easily navigate our schema. +As each new mutation also needs tests (including slower request integration tests), adding mutations +slows down the test suite. + +To minimize changes: + +- Use existing mutations, such as `MergeRequest::Update`, when available. +- Expose existing services as a coarse-grained mutation. + +When a fine-grained mutation might be more appropriate: + +- Modifying a property that requires specific permissions or other specialized logic. +- Exposing a state-machine-like transition (locking issues, merging MRs, closing epics, etc). +- Accepting nested properties (where we accept properties for a child object). +- The semantics of the mutation can be expressed clearly and concisely. + +See [issue #233063](https://gitlab.com/gitlab-org/gitlab/-/issues/233063) for further context. ### Naming conventions @@ -1361,5 +1423,4 @@ For information on generating GraphQL documentation and schema files, see [updating the schema documentation](rake_tasks.md#update-graphql-documentation-and-schema-definitions). To help our readers, you should also add a new page to our [GraphQL API](../api/graphql/index.md) documentation. -For guidance, see the [GraphQL API](documentation/styleguide.md#graphql-api) section -of our documentation style guide. +For guidance, see the [GraphQL API](documentation/graphql_styleguide.md) page. diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 400752c69e9..47c06377d88 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -17,7 +17,7 @@ Each new or updated API endpoint must come with documentation, unless it is inte The docs should be in the same merge request, or, if strictly necessary, in a follow-up with the same milestone as the original merge request. -See the [Documentation Style Guide RESTful API section](documentation/styleguide.md#restful-api) for details on documenting API resources in Markdown as well as in OpenAPI definition files. +See the [Documentation Style Guide RESTful API page](documentation/restful_api_styleguide.md) for details on documenting API resources in Markdown as well as in OpenAPI definition files. ## Methods and parameters description diff --git a/doc/development/architecture.md b/doc/development/architecture.md index f6b1c8cd914..2646ad7052b 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -1,32 +1,91 @@ -# GitLab Architecture Overview +# GitLab architecture overview ## Software delivery -There are two software distributions of GitLab: the open source [Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/) (CE), and the open core [Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/) (EE). GitLab is available under [different subscriptions](https://about.gitlab.com/pricing/). +There are two software distributions of GitLab: -New versions of GitLab are released in stable branches and the master branch is for bleeding edge development. +- The open source [Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/) (CE). +- The open core [Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/) (EE). -For information, see the [GitLab Release Process](https://gitlab.com/gitlab-org/release/docs/-/tree/master#gitlab-release-process). +GitLab is available under [different subscriptions](https://about.gitlab.com/pricing/). -Both EE and CE require some add-on components called GitLab Shell and Gitaly. These components are available from the [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/master) and [Gitaly](https://gitlab.com/gitlab-org/gitaly/-/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with the exception of informal security updates deemed critical. +New versions of GitLab are released from stable branches, and the `master` branch is used for +bleeding-edge development. + +For more information, visit the [GitLab Release Process](https://about.gitlab.com/handbook/engineering/releases/). + +Both distributions require additional components. These components are described in the +[Component details](#components) section, and all have their own repositories. +New versions of each dependent component are usually tags, but staying on the `master` branch of the +GitLab codebase gives you the latest stable version of those components. New versions are +generally released around the same time as GitLab releases, with the exception of informal security +updates deemed critical. ## Components -A typical install of GitLab will be on GNU/Linux. It uses NGINX or Apache as a web front end to proxypass the Unicorn web server. By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. The web front end accesses `/home/git/gitlab/public` bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and pre-compiled assets. GitLab serves web pages and the [GitLab API](../api/README.md) using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses Redis as a non-persistent database backend for job information, meta data, and incoming jobs. +A typical install of GitLab is on GNU/Linux, but growing number of deployments also use the +Kubernetes platform. The largest known GitLab instance is on GitLab.com, which is deployed using our +[official GitLab Helm chart](https://docs.gitlab.com/charts/) and the [official Linux package](https://about.gitlab.com/install/). + +A typical installation uses NGINX or Apache as a web server to proxy through +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) and into the [Puma](https://puma.io) +application server. GitLab serves web pages and the [GitLab API](../api/README.md) using the Puma +application server. It uses Sidekiq as a job queue which, in turn, uses Redis as a non-persistent +database backend for job information, metadata, and incoming jobs. -We also support deploying GitLab on Kubernetes using our [GitLab Helm chart](https://docs.gitlab.com/charts/). +By default, communication between Puma and Workhorse is via a Unix domain socket, but forwarding +requests via TCP is also supported. Workhorse accesses the `gitlab/public` directory, bypassing the +Puma application server to serve static pages, uploads (for example, avatar images or attachments), +and pre-compiled assets. -The GitLab web app uses PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare Git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. +The GitLab application uses PostgreSQL for persistent database information (for example, users, +permissions, issues, or other metadata). GitLab stores the bare Git repositories in the location +defined in [the configuration file, `repositories:` section](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example). +It also keeps default branch and hook information with the bare repository. -When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving Git objects. +When serving repositories over HTTP/HTTPS GitLab uses the GitLab API to resolve authorization and +access and to serve Git objects. -The add-on component GitLab Shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. GitLab Shell accesses the bare repositories through Gitaly to serve Git objects and communicates with Redis to submit jobs to Sidekiq for GitLab to process. GitLab Shell queries the GitLab API to determine authorization and access. +The add-on component GitLab Shell serves repositories over SSH. It manages the SSH keys within the +location defined in [the configuration file, `GitLab Shell` section](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example). +The file in that location should never be manually edited. GitLab Shell accesses the bare +repositories through Gitaly to serve Git objects, and communicates with Redis to submit jobs to +Sidekiq for GitLab to process. GitLab Shell queries the GitLab API to determine authorization and access. -Gitaly executes Git operations from GitLab Shell and the GitLab web app, and provides an API to the GitLab web app to get attributes from Git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files). +Gitaly executes Git operations from GitLab Shell and the GitLab web app, and provides an API to the +GitLab web app to get attributes from Git (for example, title, branches, tags, or other metadata), +and to get blobs (for example, diffs, commits, or files). You may also be interested in the [production architecture of GitLab.com](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/). -### Simplified Component Overview +## Adapting existing and introducing new components + +There are fundamental differences in how the application behaves when it is installed on a +traditional Linux machine compared to a containerized platform, such as Kubernetes. + +Compared to [our official installation methods](https://about.gitlab.com/install/), some of the +notable differences are: + +- Official Linux packages can access files on the same file system with different services. + [Shared files](shared_files.md) are not an option for the application running on the Kubernetes + platform. +- Official Linux packages by default have services that have access to the shared configuration and + network. This is not the case for services running in Kubernetes, where services might be running + in complete isolation, or only accessible through specific ports. + +In other words, the shared state between services needs to be carefully considered when +architecting new features and adding new components. Services that need to have access to the same +files, need to be able to exchange information through the appropriate APIs. Whenever possible, +this should not be done with files. + +Since components written with the API-first philosophy in mind are compatible with both methods, all +new features and services must be written to consider Kubernetes compatibility **first**. + +The simplest way to ensure this, is to add support for your feature or service to +[the official GitLab Helm chart](https://docs.gitlab.com/charts/) or reach out to +[the Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/#how-to-work-with-distribution). + +### Simplified component overview This is a simplified architecture diagram that can be used to understand GitLab's architecture. @@ -51,15 +110,15 @@ SSH -- TCP 22 --> GitLabShell[GitLab Shell] SMTP[SMTP Gateway] Geo[GitLab Geo Node] -- TCP 22, 80, 443 --> NGINX -GitLabShell --TCP 8080 -->Unicorn["Unicorn (GitLab Rails)"] +GitLabShell --TCP 8080 -->Puma["Puma (GitLab Rails)"] GitLabShell --> Praefect -Unicorn --> PgBouncer[PgBouncer] -Unicorn --> Redis -Unicorn --> Praefect +Puma --> PgBouncer[PgBouncer] +Puma --> Redis +Puma --> Praefect Sidekiq --> Redis Sidekiq --> PgBouncer Sidekiq --> Praefect -GitLabWorkhorse[GitLab Workhorse] --> Unicorn +GitLabWorkhorse[GitLab Workhorse] --> Puma GitLabWorkhorse --> Redis GitLabWorkhorse --> Praefect Praefect --> Gitaly @@ -67,7 +126,7 @@ NGINX --> GitLabWorkhorse NGINX -- TCP 8090 --> GitLabPages[GitLab Pages] NGINX --> Grafana[Grafana] Grafana -- TCP 9090 --> Prometheus[Prometheus] -Prometheus -- TCP 80, 443 --> Unicorn +Prometheus -- TCP 80, 443 --> Puma RedisExporter[Redis Exporter] --> Redis Prometheus -- TCP 9121 --> RedisExporter PostgreSQLExporter[PostgreSQL Exporter] --> PostgreSQL @@ -83,27 +142,27 @@ PgBouncer --> Consul PostgreSQL --> Consul PgBouncer --> PostgreSQL NGINX --> Registry -Unicorn --> Registry +Puma --> Registry NGINX --> Mattermost -Mattermost --- Unicorn +Mattermost --- Puma Prometheus --> Alertmanager Migrations --> PostgreSQL Runner -- TCP 443 --> NGINX -Unicorn -- TCP 9200 --> Elasticsearch +Puma -- TCP 9200 --> Elasticsearch Sidekiq -- TCP 9200 --> Elasticsearch Sidekiq -- TCP 80, 443 --> Sentry -Unicorn -- TCP 80, 443 --> Sentry +Puma -- TCP 80, 443 --> Sentry Sidekiq -- UDP 6831 --> Jaeger -Unicorn -- UDP 6831 --> Jaeger +Puma -- UDP 6831 --> Jaeger Gitaly -- UDP 6831 --> Jaeger GitLabShell -- UDP 6831 --> Jaeger GitLabWorkhorse -- UDP 6831 --> Jaeger Alertmanager -- TCP 25 --> SMTP Sidekiq -- TCP 25 --> SMTP -Unicorn -- TCP 25 --> SMTP -Unicorn -- TCP 369 --> LDAP +Puma -- TCP 25 --> SMTP +Puma -- TCP 369 --> LDAP Sidekiq -- TCP 369 --> LDAP -Unicorn -- TCP 443 --> ObjectStorage["Object Storage"] +Puma -- TCP 443 --> ObjectStorage["Object Storage"] Sidekiq -- TCP 443 --> ObjectStorage GitLabWorkhorse -- TCP 443 --> ObjectStorage Registry -- TCP 443 --> ObjectStorage @@ -121,7 +180,7 @@ click Gitaly "./architecture.html#gitaly" click Jaeger "./architecture.html#jaeger" click GitLabWorkhorse "./architecture.html#gitlab-workhorse" click LDAP "./architecture.html#ldap-authentication" -click Unicorn "./architecture.html#unicorn" +click Puma "./architecture.html#puma" click GitLabShell "./architecture.html#gitlab-shell" click SSH "./architecture.html#ssh-request-22" click Sidekiq "./architecture.html#sidekiq" @@ -201,7 +260,7 @@ Table description links: | [Runner](#gitlab-runner) | Executes GitLab CI/CD jobs | ⤓ | ✅ | ⚙ | ✅ | ⚙ | ⚙ | CE & EE | | [Sentry integration](#sentry) | Error tracking for deployed apps | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | CE & EE | | [Sidekiq](#sidekiq) | Background jobs processor | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | CE & EE | -| [Unicorn (GitLab Rails)](#unicorn) | Handles requests for the web interface and API | ✅ | ✅ | ✅ | ✅ | ⚙ | ✅ | CE & EE | +| [Puma (GitLab Rails)](#puma) | Handles requests for the web interface and API | ✅ | ✅ | ✅ | ✅ | ⚙ | ✅ | CE & EE | ### Component details @@ -271,7 +330,7 @@ Consul is a tool for service discovery and configuration. Consul is distributed, - [Source](../integration/elasticsearch.md) - [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/elasticsearch.md) - Layer: Core Service (Data) -- GitLab.com: [Get Advanced Global Search working on GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/153) epic. +- GitLab.com: [Get Advanced Search working on GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/153) epic. Elasticsearch is a distributed RESTful search engine built for the cloud. @@ -368,13 +427,13 @@ GitLab CI/CD is the open-source continuous integration service included with Git - [Project page](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/README.md) - Configuration: - [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) - - [Charts](https://docs.gitlab.com/charts/charts/gitlab/unicorn/) + - [Charts](https://docs.gitlab.com/charts/charts/gitlab/webservice/) - [Source](../install/installation.md#install-gitlab-workhorse) - Layer: Core Service (Processor) - Process: `gitlab-workhorse` - GitLab.com: [Service Architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#service-architecture) -[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from Unicorn. You can read more about the [historical reasons for developing](https://about.gitlab.com/blog/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from Puma. You can read more about the [historical reasons for developing](https://about.gitlab.com/blog/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. #### Grafana @@ -411,7 +470,8 @@ For monitoring deployed apps, see [Jaeger tracing documentation](../operations/t - Layer: Core Service - Process: `logrotate` -GitLab is comprised of a large number of services that all log. We started bundling our own logrotate as of 7.4 to make sure we were logging responsibly. This is just a packaged version of the common open source offering. +GitLab is comprised of a large number of services that all log. We started bundling our own Logrotate +as of GitLab 7.4 to make sure we were logging responsibly. This is just a packaged version of the common open source offering. #### Mattermost @@ -603,8 +663,30 @@ For monitoring deployed apps, see the [Sentry integration docs](../operations/er Sidekiq is a Ruby background job processor that pulls jobs from the Redis queue and processes them. Background jobs allow GitLab to provide a faster request/response cycle by moving work into the background. +#### Puma + +NOTE: **Note:** +Starting with GitLab 13.0, Puma is the default web server and Unicorn has been +disabled by default. + +- [Project page](https://gitlab.com/gitlab-org/gitlab/blob/master/README.md) +- Configuration: + - [Omnibus](https://docs.gitlab.com/omnibus/settings/puma.html) + - [Charts](https://docs.gitlab.com/charts/charts/gitlab/webservice/) + - [Source](../install/installation.md#configure-it) + - [GDK](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) +- Layer: Core Service (Processor) +- Process: `puma` +- GitLab.com: [Puma](../user/gitlab_com/index.md#puma) + +[Puma](https://puma.io/) is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often process output you will see this as `bundle` or `config.ru` depending on the GitLab version. + #### Unicorn +NOTE: **Note:** +Starting with GitLab 13.0, Puma is the default web server and Unicorn has been +disabled by default. + - [Project page](https://gitlab.com/gitlab-org/gitlab/blob/master/README.md) - Configuration: - [Omnibus](https://docs.gitlab.com/omnibus/settings/unicorn.html) @@ -669,7 +751,7 @@ You can install them after you create a cluster. This includes: - [JupyterHub](https://jupyter.org) - [Knative](https://cloud.google.com/knative/) -## GitLab by Request Type +## GitLab by request type GitLab provides two "interfaces" for end users to access the service: @@ -678,20 +760,20 @@ GitLab provides two "interfaces" for end users to access the service: It's important to understand the distinction as some processes are used in both and others are exclusive to a specific request type. -### GitLab Web HTTP Request Cycle +### GitLab Web HTTP request cycle When making a request to an HTTP Endpoint (think `/users/sign_in`) the request will take the following path through the GitLab Service: - NGINX - Acts as our first line reverse proxy. -- GitLab Workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Unicorn. -- Unicorn - Since this is a web request, and it needs to access the application it will go to Unicorn. +- GitLab Workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Puma. +- Puma - Since this is a web request, and it needs to access the application it will go to Puma. - PostgreSQL/Gitaly/Redis - Depending on the type of request, it may hit these services to store or retrieve data. -### GitLab Git Request Cycle +### GitLab Git request cycle Below we describe the different paths that HTTP vs. SSH Git requests will take. There is some overlap with the Web Request Cycle but also some differences. -### Web Request (80/443) +### Web request (80/443) Git operations over HTTP use the stateless "smart" protocol described in the [Git documentation](https://git-scm.com/docs/http-protocol), but responsibility @@ -736,7 +818,7 @@ sequenceDiagram The sequence is similar for `git push`, except `git-receive-pack` is used instead of `git-upload-pack`. -### SSH Request (22) +### SSH request (22) Git operations over SSH can use the stateful protocol described in the [Git documentation](https://git-scm.com/docs/pack-protocol#_ssh_transport), but @@ -801,7 +883,7 @@ except there is no round-trip into Gitaly - Rails performs the action as part of the [internal API](internal_api.md) call, and GitLab Shell streams the response back to the user directly. -## System Layout +## System layout When referring to `~git` in the pictures it means the home directory of the Git user which is typically `/home/git`. @@ -811,9 +893,9 @@ The bare repositories are located in `/home/git/repositories`. GitLab is a Ruby To serve repositories over SSH there's an add-on application called GitLab Shell which is installed in `/home/git/gitlab-shell`. -### Installation Folder Summary +### Installation folder summary -To summarize here's the [directory structure of the `git` user home directory](../install/structure.md). +To summarize here's the [directory structure of the `git` user home directory](../install/installation.md#gitlab-directory-structure). ### Processes @@ -823,12 +905,12 @@ ps aux | grep '^git' GitLab has several components to operate. It requires a persistent database (PostgreSQL) and Redis database, and uses Apache `httpd` or NGINX to proxypass -Unicorn. All these components should run as different system users to GitLab -(e.g., `postgres`, `redis` and `www-data`, instead of `git`). +Puma. All these components should run as different system users to GitLab +(for example, `postgres`, `redis`, and `www-data`, instead of `git`). -As the `git` user it starts Sidekiq and Unicorn (a simple Ruby HTTP server +As the `git` user it starts Sidekiq and Puma (a simple Ruby HTTP server running on port `8080` by default). Under the GitLab user there are normally 4 -processes: `unicorn_rails master` (1 process), `unicorn_rails worker` +processes: `puma master` (1 process), `puma cluster worker` (2 processes), `sidekiq` (1 process). ### Repository access @@ -841,7 +923,7 @@ See the README for more information. ### Init scripts of the services -The GitLab init script starts and stops Unicorn and Sidekiq: +The GitLab init script starts and stops Puma and Sidekiq: ```plaintext /etc/init.d/gitlab @@ -881,9 +963,9 @@ Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [v ### Log locations of the services -GitLab (includes Unicorn and Sidekiq logs): +GitLab (includes Puma and Sidekiq logs): -- `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `unicorn.stdout.log`, `git_json.log` and `unicorn.stderr.log` normally. +- `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `puma.stdout.log`, `git_json.log` and `puma.stderr.log` normally. GitLab Shell: @@ -914,17 +996,18 @@ PostgreSQL: ### GitLab specific configuration files -GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced config files include: +GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced +configuration files include: -- `gitlab.yml` - GitLab configuration. -- `unicorn.rb` - Unicorn web server settings. -- `database.yml` - Database connection settings. +- `gitlab.yml` - GitLab configuration +- `puma.rb` - Puma web server settings +- `database.yml` - Database connection settings GitLab Shell has a configuration file at `/home/git/gitlab-shell/config.yml`. -### Maintenance Tasks +### Maintenance tasks -[GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master) provides Rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See [maintenance Rake tasks](../raketasks/maintenance.md). +[GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master) provides Rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See [maintenance Rake tasks](../administration/raketasks/maintenance.md). In a nutshell, do the following: ```shell @@ -934,7 +1017,8 @@ bundle exec rake gitlab:env:info RAILS_ENV=production bundle exec rake gitlab:check RAILS_ENV=production ``` -Note: It is recommended to log into the `git` user using `sudo -i -u git` or `sudo su - git`. While the sudo commands provided by GitLab work in Ubuntu they do not always work in RHEL. +Note: It is recommended to log into the `git` user using `sudo -i -u git` or `sudo su - git`. While +the `sudo` commands provided by GitLab work in Ubuntu they do not always work in RHEL. ## GitLab.com diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index 4b58758b5c7..e5cc2ae4d1d 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -1,4 +1,11 @@ -# Background Migrations +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + +# Background migrations Background migrations can be used to perform data migrations that would otherwise take a very long time (hours, days, years, etc) to complete. For @@ -92,6 +99,20 @@ bulk_migrate_async( ) ``` +Note that this will queue a Sidekiq job immediately: if you have a large number +of records, this may not be what you want. You can use the function +`queue_background_migration_jobs_by_range_at_intervals` to split the job into +batches: + +```ruby +queue_background_migration_jobs_by_range_at_intervals( + ClassName, + BackgroundMigrationClassName, + 2.minutes, + batch_size: 10_000 + ) +``` + You'll also need to make sure that newly created data is either migrated, or saved in both the old and new version upon creation. For complex and time consuming migrations it's best to schedule a background job using an diff --git a/doc/development/changelog.md b/doc/development/changelog.md index f57e666540c..d992f394de3 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -35,8 +35,7 @@ the `author` field. GitLab team members **should not**. - [Security fixes](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md) **must** have a changelog entry, without `merge_request` value and with `type` set to `security`. -- Any user-facing change **should** have a changelog entry. Example: "GitLab now - uses system fonts for all text." +- Any user-facing change **should** have a changelog entry. This includes both visual changes (regardless of how minor), and changes to the rendered DOM which impact how a screen reader may announce the content. - Performance improvements **should** have a changelog entry. - Changes that need to be documented in the Telemetry [Event Dictionary](telemetry/event_dictionary.md) also require a changelog entry. diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 2e319efa5f3..436acb1e4d7 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -67,6 +67,10 @@ page, with these behaviors: contains the string 'OOO', or the emoji is `:palm_tree:` or `:beach:`. 1. [Trainee maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#trainee-maintainer) are three times as likely to be picked as other reviewers. +1. People whose [GitLab status](../user/profile/index.md#current-status) emoji + is `:large_blue_circle:` are more likely to be picked. This applies to both reviewers and trainee maintainers. + - Reviewers with `:large_blue_circle:` are two times as likely to be picked as other reviewers. + - Trainee maintainers with `:large_blue_circle:` are four times as likely to be picked as other reviewers. 1. It always picks the same reviewers and maintainers for the same branch name (unless their OOO status changes, as in point 1). It removes leading `ce-` and `ee-`, and trailing `-ce` and `-ee`, so @@ -100,6 +104,7 @@ with [domain expertise](#domain-experts). by a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors)**. 1. If your merge request only includes end-to-end changes (*3*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)** 1. If your merge request includes a new or updated [application limit](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits), it must be **approved by a [product manager](https://about.gitlab.com/company/team/)**. +1. If your merge request includes Product Analytics (telemetry) changes, it should be reviewed and approved by a [Product analytics engineer](https://gitlab.com/gitlab-org/growth/telemetry/engineers). - (*1*): Please note that specs other than JavaScript specs are considered backend code. - (*2*): We encourage you to seek guidance from a database maintainer if your merge diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index bb7b4713a5e..42c67b97a35 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -320,6 +320,11 @@ look for issues labeled `~"Accepting merge requests"` with a [weight of 1](https More experienced contributors are very welcome to tackle [any of them](https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=None). +For more complex features that have a weight of 2 or more and clear scope, we recommend looking at issues +with the [label `~"Community Challenge"`](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=Accepting%20merge%20requests&label_name[]=Community%20challenge). +If your MR for the `~"Community Challenge"` issue gets merged, you will also have a chance to win a custom +GitLab merchandise. + If you've decided that you would like to work on an issue, please @-mention the [appropriate product manager](https://about.gitlab.com/handbook/product/#who-to-talk-to-for-what) as soon as possible. The product manager will then pull in appropriate GitLab team diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index d88b159b666..b26a8a775b3 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -218,7 +218,7 @@ requirements. 1. [Secure coding guidelines](https://gitlab.com/gitlab-com/gl-security/security-guidelines) have been followed. 1. [Documented](../documentation/index.md) in the `/doc` directory. 1. [Changelog entry added](../changelog.md), if necessary. -1. Reviewed by relevant (UX/FE/BE/tech writing) reviewers and all concerns are addressed. +1. Reviewed by relevant reviewers and all concerns are addressed for Availability, Regressions, Security. Documentation reviews should take place as soon as possible, but they should not block a merge request. 1. Merged by a project maintainer. 1. Create an issue in the [infrastructure issue tracker](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues) to inform the Infrastructure department when your contribution is changing default settings or introduces a new setting, if relevant. 1. Confirmed to be working in the [Canary stage](https://about.gitlab.com/handbook/engineering/#canary-testing) or on GitLab.com once the contribution is deployed. diff --git a/doc/development/database/index.md b/doc/development/database/index.md index 9ea5b6fcaac..0ae39ed7ca7 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -24,6 +24,7 @@ - [Background migrations](../background_migrations.md) - [Swapping tables](../swapping_tables.md) - [Deleting migrations](../deleting_migrations.md) +- [Partitioning tables](table_partitioning.md) ## Debugging diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md new file mode 100644 index 00000000000..00981ef28a2 --- /dev/null +++ b/doc/development/database/table_partitioning.md @@ -0,0 +1,253 @@ +# Database table partitioning + +Table partitioning is a powerful database feature that allows a table's +data to be split into smaller physical tables that act as a single large +table. If the application is designed to work with partitioning in mind, +there can be multiple benefits, such as: + +- Query performance can be improved greatly, because the database can +cheaply eliminate much of the data from the search space, while still +providing full SQL capabilities. + +- Bulk deletes can be achieved with minimal impact on the database by +dropping entire partitions. This is a natural fit for features that need +to periodically delete data that falls outside the retention window. + +- Administrative tasks like `VACUUM` and index rebuilds can operate on +individual partitions, rather than across a single massive table. + +Unfortunately, not all models fit a partitioning scheme, and there are +significant drawbacks if implemented incorrectly. Additionally, tables +can only be partitioned at their creation, making it nontrivial to apply +partitioning to a busy database. A suite of migration tools are available +to enable backend developers to partition existing tables, but the +migration process is rather heavy, taking multiple steps split across +several releases. Due to the limitations of partitioning and the related +migrations, you should understand how partitioning fits your use case +before attempting to leverage this feature. + +## Determining when to use partitioning + +While partitioning can be very useful when properly applied, it's +imperative to identify if the data and workload of a table naturally fit a +partitioning scheme. There are a few details you'll have to understand +in order to decide if partitioning is a good fit for your particular +problem. + +First, a table is partitioned on a partition key, which is a column or +set of columns which determine how the data will be split across the +partitions. The partition key is used by the database when reading or +writing data, to decide which partition(s) need to be accessed. The +partition key should be a column that would be included in a `WHERE` +clause on almost all queries accessing that table. + +Second, it's necessary to understand the strategy the database will +use to split the data across the partitions. The scheme supported by the +GitLab migration helpers is date-range partitioning, where each partition +in the table contains data for a single month. In this case, the partitioning +key would need to be a timestamp or date column. In order for this type of +partitioning to work well, most queries would need to access data within a +certain date range. + +For a more concrete example, the `audit_events` table can be used, which +was the first table to be partitioned in the application database +(scheduled for deployment with the GitLab 13.5 release). This +table tracks audit entries of security events that happen in the +application. In almost all cases, users want to see audit activity that +occurs in a certain timeframe. As a result, date-range partitioning +was a natural fit for how the data would be accessed. + +To look at this in more detail, imagine a simplified `audit_events` schema: + +```sql +CREATE TABLE audit_events ( + id SERIAL NOT NULL PRIMARY KEY, + author_id INT NOT NULL, + details jsonb NOT NULL, + created_at timestamptz NOT NULL); +``` + +Now imagine typical queries in the UI would display the data within a +certain date range, like a single week: + +```sql +SELECT * +FROM audit_events +WHERE created_at >= '2020-01-01 00:00:00' + AND created_at < '2020-01-08 00:00:00' +ORDER BY created_at DESC +LIMIT 100 +``` + +If the table is partioned on the `created_at` column the base table would +look like: + +```sql +CREATE TABLE audit_events ( + id SERIAL NOT NULL, + author_id INT NOT NULL, + details jsonb NOT NULL, + created_at timestamptz NOT NULL, + PRIMARY KEY (id, created_at)) +PARTITION BY RANGE(created_at); +``` + +NOTE: **Note:** +The primary key of a partitioned table must include the partition key as +part of the primary key definition. + +And we might have a list of partitions for the table, such as: + +```sql +audit_events_202001 FOR VALUES FROM ('2020-01-01') TO ('2020-02-01') +audit_events_202002 FOR VALUES FROM ('2020-02-01') TO ('2020-03-01') +audit_events_202003 FOR VALUES FROM ('2020-03-01') TO ('2020-04-01') +``` + +Each partition is a separate physical table, with the same structure as +the base `audit_events` table, but contains only data for rows where the +partition key falls in the specified range. For example, the partition +`audit_events_202001` contains rows where the `created_at` column is +greater than or equal to `2020-01-01` and less than `2020-02-01`. + +Now, if we look at the previous example query again, the database can +use the `WHERE` to recognize that all matching rows will be in the +`audit_events_202001` partition. Rather than searching all of the data +in all of the partitions, it can search only the single month's worth +of data in the appropriate partition. In a large table, this can +dramatically reduce the amount of data the database needs to access. +However, imagine a query that does not filter based on the partitioning +key, such as: + +```sql +SELECT * +FROM audit_events +WHERE author_id = 123 +ORDER BY created_at DESC +LIMIT 100 +``` + +In this example, the database can't prune any partitions from the search, +because matching data could exist in any of them. As a result, it has to +query each partition individually, and aggregate the rows into a single result +set. Since `author_id` would be indexed, the performance impact could +likely be acceptable, but on more complex queries the overhead can be +substantial. Partitioning should only be leveraged if the access patterns +of the data support the partitioning strategy, otherwise performance will +suffer. + +## Partitioning a table + +Unfortunately, tables can only be partitioned at their creation, making +it nontrivial to apply to a busy database. A suite of migration +tools have been developed to enable backend developers to partition +existing tables. This migration process takes multiple steps which must +be split across several releases. + +### Caveats + +The partitioning migration helpers work by creating a partitioned duplicate +of the original table and using a combination of a trigger and a background +migration to copy data into the new table. Changes to the original table +schema can be made in parallel with the partitioning migration, but they +must take care to not break the underlying mechanism that makes the migration +work. For example, if a column is added to the table that is being +partitioned, both the partitioned table and the trigger definition need to +be updated to match. + +### Step 1: Creating the partitioned copy (Release N) + +The first step is to add a migration to create the partitioned copy of +the original table. This migration will also create the appropriate +partitions based on the data in the original table, and install a +trigger that will sync writes from the original table into the +partitioned copy. + +An example migration of partitioning the `audit_events` table by its +`created_at` column would look like: + +```ruby +class PartitionAuditEvents < ActiveRecord::Migration[6.0] + include Gitlab::Database::PartitioningMigrationHelpers + + def up + partition_table_by_date :audit_events, :created_at + end + + def down + drop_partitioned_table_for :audit_events + end +end +``` + +Once this has executed, any inserts, updates or deletes in the +original table will also be duplicated in the new table. For updates and +deletes, the operation will only have an effect if the corresponding row +exists in the partitioned table. + +### Step 2: Backfill the partitioned copy (Release N) + +The second step is to add a post-deployment migration that will schedule +the background jobs that will backfill existing data from the original table +into the partitioned copy. + +Continuing the above example, the migration would look like: + +```ruby +class BackfillPartitionAuditEvents < ActiveRecord::Migration[6.0] + include Gitlab::Database::PartitioningMigrationHelpers + + def up + enqueue_partitioning_data_migration :audit_events + end + + def down + cleanup_partitioning_data_migration :audit_events + end +end +``` + +This step uses the same mechanism as any background migration, so you +may want to read the [Background Migration](../background_migrations.md) +guide for details on that process. Background jobs are scheduled every +2 minutes and copy `50_000` records at a time, which can be used to +estimate the timing of the background migration portion of the +partitioning migration. + +### Step 3: Post-backfill cleanup (Release N+1) + +The third step must occur at least one release after the release that +includes the background migration. This gives time for the background +migration to execute properly in self-managed installations. In this step, +add another post-deployment migration that will cleanup after the +background migration. This includes forcing any remaining jobs to +execute, and copying data that may have been missed, due to dropped or +failed jobs. + +Once again, continuing the example, this migration would look like: + +```ruby +class CleanupPartitionedAuditEventsBackfill < ActiveRecord::Migration[6.0] + include Gitlab::Database::PartitioningMigrationHelpers + + def up + finalize_backfilling_partitioned_table :audit_events + end + + def down + # no op + end +end +``` + +After this migration has completed, the original table and partitioned +table should contain identical data. The trigger installed on the +original table guarantees that the data will remain in sync going +forward. + +### Step 4: Swap the partitioned and non-partitioned tables (Release N+1) + +The final step of the migration will make the partitioned table ready +for use by the application. This section will be updated when the +migration helper is ready, for now development can be followed in the +[Tracking Issue](https://gitlab.com/gitlab-org/gitlab/-/issues/241267). diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index 61e8ac60bfe..e3430204561 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -19,7 +19,7 @@ If you just want to delete everything and start over with an empty DB (approxima bundle exec rake db:reset RAILS_ENV=development ``` -If you just want to delete everything and start over with dummy data (approximately 4 minutes). This +If you just want to delete everything and start over with sample data (approximately 4 minutes). This also does `db:reset` and runs DB-specific migrations: ```shell diff --git a/doc/development/database_review.md b/doc/development/database_review.md index f56ffdbad21..8a10391419a 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -184,10 +184,6 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac - [Check query plans](understanding_explain_plans.md) and suggest improvements to queries (changing the query, schema or adding indexes and similar) - General guideline is for queries to come in below 100ms execution time - - If queries rely on prior migrations that are not present yet on production - (eg indexes, columns), you can use a [one-off instance from the restore - pipeline](https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore/postgres-gprd) - in order to establish a proper testing environment. If you don't have access to this project, reach out to #database on Slack to get advice on how to proceed. - Avoid N+1 problems and minimalize the [query count](merge_request_performance_guidelines.md#query-counts). ### Timing guidelines for migrations diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index a4a6ee2fa0f..0f03ceeb4b5 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -52,7 +52,7 @@ For feature flags disabled by default, if they can be used by end users: do not say anything about it. - Say whether it's recommended for production use. - Document how to enable and disable it. -- Add a warning to the user saying that the feature is disabled. +- Add a warning to the user saying that the feature might be disabled. For example, for a feature disabled by default, disabled on GitLab.com, cannot be enabled for a single project, and is not ready for production use: @@ -250,7 +250,7 @@ be enabled by project, and is ready for production use: > - [Introduced](link-to-issue) in GitLab 12.0. > - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. > - It's enabled on GitLab.com. -> - It can be enabled or disable for a single project. +> - It can be enabled or disabled for a single project. > - It's recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(CORE ONLY)** diff --git a/doc/development/documentation/graphql_styleguide.md b/doc/development/documentation/graphql_styleguide.md new file mode 100644 index 00000000000..53a645a2936 --- /dev/null +++ b/doc/development/documentation/graphql_styleguide.md @@ -0,0 +1,92 @@ +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +description: "Writing styles, markup, formatting, and other standards for GraphQL API's GitLab Documentation." +--- + +# GraphQL API + +GraphQL APIs are different from [RESTful APIs](restful_api_styleguide.md). Reference +information is generated in our [GraphQL reference](../../api/graphql/reference/index.md). + +However, it's helpful to include examples on how to use GraphQL for different +*use cases*, with samples that readers can use directly in the +[GraphiQL explorer](../api_graphql_styleguide.md#graphiql). + +This section describes the steps required to add your GraphQL examples to +GitLab documentation. + +## Add a dedicated GraphQL page + +To create a dedicated GraphQL page, create a new `.md` file in the +`doc/api/graphql/` directory. Give that file a functional name, such as +`import_from_specific_location.md`. + +## Start the page with an explanation + +Include a page title that describes the GraphQL functionality in a few words, +such as: + +```markdown +# Search for [substitute kind of data] +``` + +Describe the search. One sentence may be all you need. More information may +help readers learn how to use the example for their GitLab deployments. + +## Include a procedure using the GraphiQL explorer + +The GraphiQL explorer can help readers test queries with working deployments. +Set up the section with the following: + +- Use the following title: + + ```markdown + ## Set up the GraphiQL explorer + ``` + +- Include a code block with the query that anyone can include in their + instance of the GraphiQL explorer: + + ````markdown + ```graphql + query { + <insert queries here> + } + ``` + ```` + +- Tell the user what to do: + + ```markdown + 1. Open the GraphiQL explorer tool in the following URL: `https://gitlab.com/-/graphql-explorer`. + 1. Paste the `query` listed above into the left window of your GraphiQL explorer tool. + 1. Select **Play** to get the result shown here: + ``` + +- Include a screenshot of the result in the GraphiQL explorer. Follow the naming + convention described in the [Save the image](styleguide.md#save-the-image) section of the Documentation style guide. +- Follow up with an example of what you can do with the output. Make sure the + example is something that readers can do on their own deployments. +- Include a link to the [GraphQL API resources](../../api/graphql/reference/index.md). + +## Add the GraphQL example to the global navigation + +You should include a link for your new document in the global navigation (the list on the +left side of the documentation website). To do so, open a second MR, against the +[GitLab documentation repository](https://gitlab.com/gitlab-org/gitlab-docs/). + +We store our global navgation in the [`default-nav.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/content/_data/default-nav.yaml) file, in the +`content/_data` subdirectory. You can find the GraphQL section under the +following line: + +```yaml +- category_title: GraphQL +``` + +Be aware that CI tests for that second MR will fail with a bad link until the +main MR that adds the new GraphQL page is merged. Therefore, only merge the MR against the +[`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) repository after the content has +been merged and live on `docs.gitlab.com`. diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index d6f24d6374d..3a02ea5aa83 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -1,4 +1,7 @@ --- +stage: none +group: Documentation Guidelines +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: Learn how to contribute to GitLab Documentation. --- @@ -52,9 +55,9 @@ docs-only merge requests using the following guide: [Contributions to GitLab docs](workflow.md) are welcome from the entire GitLab community. -To ensure that GitLab docs are current, there are special processes and responsibilities for all [feature changes](feature-change-workflow.md), that is development work that impacts the appearance, usage, or administration of a feature. +To ensure that GitLab docs are current, there are special processes and responsibilities for all [feature changes](workflow.md), that is development work that impacts the appearance, usage, or administration of a feature. -However, anyone can contribute [documentation improvements](improvement-workflow.md) that are not associated with a feature change. For example, adding a new doc on how to accomplish a use case that's already possible with GitLab or with third-party tools and GitLab. +However, anyone can contribute [documentation improvements](workflow.md) that are not associated with a feature change. For example, adding a new doc on how to accomplish a use case that's already possible with GitLab or with third-party tools and GitLab. ## Markdown and styles @@ -128,7 +131,7 @@ The following metadata should be added when a page is moved to another location: - `redirect_to`: The relative path and filename (with an `.md` extension) of the location to which visitors should be redirected for a moved page. - [Learn more](#changing-document-location). + [Learn more](#move-or-rename-a-page). - `disqus_identifier`: Identifier for Disqus commenting system. Used to keep comments with a page that's been moved to a new URL. [Learn more](#redirections-for-pages-with-disqus-comments). @@ -156,17 +159,18 @@ Nanoc layout), which will be displayed at the top of the page if defined: [algorithm](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/lib/helpers/reading_time.rb) to calculate the reading time based on the number of words. -## Changing document location +## Move or rename a page + +Moving or renaming a document is the same as changing its location. This process +requires specific steps to ensure that visitors can find the new +documentation page, whether they're using `/help` from a GitLab instance or by +visiting <https://docs.gitlab.com>. -Changing a document's location requires specific steps to ensure that -users can seamlessly access the new doc page, whether they are accessing content -on a GitLab instance domain at `/help` or at <https://docs.gitlab.com>. Be sure to assign a -technical writer if you have any questions during the process (such as -whether the move is necessary), and ensure that a technical writer reviews this -change prior to merging. +Be sure to assign a technical writer to a page move or rename MR. Technical +Writers can help with any questions and can review your change. -If you indeed need to change a document's location, do not remove the old -document, but instead replace all of its content with the following: +To change a document's location, don't remove the old document, but instead +replace all of its content with the following: ```markdown --- @@ -176,14 +180,18 @@ redirect_to: '../path/to/file/index.md' This document was moved to [another location](../path/to/file/index.md). ``` -Where `../path/to/file/index.md` is usually the relative path to the old document. +Replace `../path/to/file/index.md` with the relative path to the old document. + +The `redirect_to` variable supports both full and relative URLs; for example: + +- `https://docs.gitlab.com/ee/path/to/file.html` +- `../path/to/file.html` +- `path/to/file.md` -The `redirect_to` variable supports both full and relative URLs, for example -`https://docs.gitlab.com/ee/path/to/file.html`, `../path/to/file.html`, `path/to/file.md`. -It ensures that the redirect will work for <https://docs.gitlab.com> and any `*.md` paths -will be compiled to `*.html`. -The new line underneath the front matter informs the user that the document -changed location and is useful for someone that browses that file from the repository. +The redirect works for <https://docs.gitlab.com>, and any `*.md` paths are +changed to `*.html`. The description line following the `redirect_to` code +informs the visitor that the document changed location if the redirect process +doesn't complete successfully. For example, if you move `doc/workflow/lfs/index.md` to `doc/administration/lfs.md`, then the steps would be: @@ -462,7 +470,7 @@ If you want to know the in-depth details, here's what's really happening: The following GitLab features are used among others: - [Manual actions](../../ci/yaml/README.md#whenmanual) -- [Multi project pipelines](../../ci/multi_project_pipeline_graphs.md) +- [Multi project pipelines](../../ci/multi_project_pipelines.md) - [Review Apps](../../ci/review_apps/index.md) - [Artifacts](../../ci/yaml/README.md#artifacts) - [Specific runner](../../ci/runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md new file mode 100644 index 00000000000..b12578b5d98 --- /dev/null +++ b/doc/development/documentation/restful_api_styleguide.md @@ -0,0 +1,180 @@ +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +description: "Writing styles, markup, formatting, and other standards for GitLab's RESTful APIs." +--- + +# RESTful API + +REST API resources are documented in Markdown under +[`/doc/api`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api). Each +resource has its own Markdown file, which is linked from `api_resources.md`. + +When modifying the Markdown, also update the corresponding +[OpenAPI definition](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api/openapi) +if one exists for the resource. If not, consider creating one. Match the latest +[OpenAPI 3.0.x specification](https://swagger.io/specification/). (For more +information, see the discussion in this +[issue](https://gitlab.com/gitlab-org/gitlab/-/issues/16023#note_370901810).) + +In the Markdown doc for a resource (AKA endpoint): + +- Every method must have the REST API request. For example: + + ```plaintext + GET /projects/:id/repository/branches + ``` + +- Every method must have a detailed [description of the parameters](#method-description). +- Every method must have a cURL example. +- Every method must have a response body (in JSON format). + +## API topic template + +The following can be used as a template to get started: + +````markdown +## Descriptive title + +One or two sentence description of what endpoint does. + +```plaintext +METHOD /endpoint +``` + +| Attribute | Type | Required | Description | +|:------------|:---------|:---------|:----------------------| +| `attribute` | datatype | yes/no | Detailed description. | +| `attribute` | datatype | yes/no | Detailed description. | + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/endpoint?parameters" +``` + +Example response: + +```json +[ + { + } +] +``` +```` + +## Method description + +Use the following table headers to describe the methods. Attributes should +always be in code blocks using backticks (`` ` ``). + +```markdown +| Attribute | Type | Required | Description | +|:----------|:-----|:---------|:------------| +``` + +Rendered example: + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:--------------------| +| `user` | string | yes | The GitLab username. | + +## cURL commands + +- Use `https://gitlab.example.com/api/v4/` as an endpoint. +- Wherever needed use this personal access token: `<your_access_token>`. +- Always put the request first. `GET` is the default so you don't have to + include it. +- Wrap the URL in double quotes (`"`). +- Prefer to use examples using the personal access token and don't pass data of + username and password. + +| Methods | Description | +|:------------------------------------------- |:------------------------------------------------------| +| `--header "PRIVATE-TOKEN: <your_access_token>"` | Use this method as is, whenever authentication needed. | +| `--request POST` | Use this method when creating new objects | +| `--request PUT` | Use this method when updating existing objects | +| `--request DELETE` | Use this method when removing existing objects | + +## cURL Examples + +The following sections include a set of [cURL](https://curl.haxx.se) examples +you can use in the API documentation. + +CAUTION: **Caution:** +Do not use information for real users, URLs, or tokens. For documentation, refer to our +relevant style guide sections on [Fake user information](styleguide.md#fake-user-information), +[Fake URLs](styleguide.md#fake-urls), and [Fake tokens](styleguide.md#fake-tokens). + +### Simple cURL command + +Get the details of a group: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/gitlab-org" +``` + +### cURL example with parameters passed in the URL + +Create a new project under the authenticated user's namespace: + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects?name=foo" +``` + +### Post data using cURL's `--data` + +Instead of using `--request POST` and appending the parameters to the URI, you +can use cURL's `--data` option. The example below will create a new project +`foo` under the authenticated user's namespace. + +```shell +curl --data "name=foo" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects" +``` + +### Post data using JSON content + +NOTE: **Note:** +In this example we create a new group. Watch carefully the single and double +quotes. + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"path": "my-group", "name": "My group"}' "https://gitlab.example.com/api/v4/groups" +``` + +### Post data using form-data + +Instead of using JSON or urlencode you can use multipart/form-data which +properly handles data encoding: + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "title=ssh-key" --form "key=ssh-rsa AAAAB3NzaC1yc2EA..." "https://gitlab.example.com/api/v4/users/25/keys" +``` + +The above example is run by and administrator and will add an SSH public key +titled `ssh-key` to user's account which has an ID of 25. + +### Escape special characters + +Spaces or slashes (`/`) may sometimes result to errors, thus it is recommended +to escape them when possible. In the example below we create a new issue which +contains spaces in its title. Observe how spaces are escaped using the `%20` +ASCII code. + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/42/issues?title=Hello%20Dude" +``` + +Use `%2F` for slashes (`/`). + +### Pass arrays to API calls + +The GitLab API sometimes accepts arrays of strings or integers. For example, to +exclude specific users when requesting a list of users for a project, you would +do something like this: + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "skip_users[]=<user_id>" --data "skip_users[]=<user_id>" "https://gitlab.example.com/api/v4/projects/<project_id>/users" +``` diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index 984c64b9e9e..e262a2c4fbe 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -251,7 +251,7 @@ The table below shows what kind of documentation goes where. | `doc/legal/` | Legal documents about contributing to GitLab. | | `doc/install/` | Contains instructions for installing GitLab. | | `doc/update/` | Contains instructions for updating GitLab. | -| `doc/topics/` | Indexes per topic (`doc/topics/topic-name/index.md`): all resources for that topic. | +| `doc/topics/` | Indexes per topic (`doc/topics/topic_name/index.md`): all resources for that topic. | ### Work with directories and files @@ -369,7 +369,7 @@ create an issue or an MR to propose a change to the user interface text. - milestones - reorder issues - runner, runners, shared runners - - a to-do, to-dos + - a to-do item, to dos - *Some features are capitalized*, typically nouns naming GitLab-specific capabilities or tools. For example: - GitLab CI/CD @@ -410,7 +410,7 @@ Use forms of *sign in*, instead of *log in* or *login*. For example: Exceptions to this rule include the concept of *single sign-on* and references to user interface elements. For example: -- To sign in to product X, enter your credentials, and then click **Log in**. +- To sign in to product X, enter your credentials, and then select **Log in**. ### Inclusive language @@ -418,8 +418,11 @@ We strive to create documentation that is inclusive. This section includes guidance and examples in the following categories: - [Gender-specific wording](#avoid-gender-specific-wording). + (Tested in [`InclusionGender.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionGender.yml).) - [Ableist language](#avoid-ableist-language). + (Tested in [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml).) - [Cultural sensitivity](#culturally-sensitive-language). + (Tested in [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml).) We write our developer documentation with inclusivity and diversity in mind. This page is not an exhaustive reference, but describes some general guidelines and @@ -433,12 +436,16 @@ a gender-neutral pronoun. Avoid the use of gender-specific pronouns, unless referring to a specific person. +<!-- vale gitlab.InclusionGender = NO --> + | Use | Avoid | |-----------------------------------|---------------------------------| | People, humanity | Mankind | | GitLab Team Members | Manpower | | You can install; They can install | He can install; She can install | +<!-- vale gitlab.InclusionGender = YES --> + If you need to set up [Fake user information](#fake-user-information), use diverse or non-gendered names with common surnames. @@ -446,6 +453,8 @@ diverse or non-gendered names with common surnames. Avoid terms that are also used in negative stereotypes for different groups. +<!-- vale gitlab.InclusionAbleism = NO --> + | Use | Avoid | |------------------------|----------------------| | Check for completeness | Sanity check | @@ -455,6 +464,8 @@ Avoid terms that are also used in negative stereotypes for different groups. | Active/Inactive | Enabled/Disabled | | On/Off | Enabled/Disabled | +<!-- vale gitlab.InclusionAbleism = YES --> + Credit: [Avoid ableist language](https://developers.google.com/style/inclusive-documentation#ableist-language) in the Google Developer Style Guide. @@ -464,13 +475,57 @@ Avoid terms that reflect negative cultural stereotypes and history. In most cases, you can replace terms such as `master` and `slave` with terms that are more precise and functional, such as `primary` and `secondary`. +<!-- vale gitlab.InclusionCultural = NO --> + | Use | Avoid | |----------------------|-----------------------| | Primary / secondary | Master / slave | | Allowlist / denylist | Blacklist / whitelist | +<!-- vale gitlab.InclusionCultural = YES --> + For more information see the following [Internet Draft specification](https://tools.ietf.org/html/draft-knodel-terminology-02). +### Fake user information + +You may need to include user information in entries such as a REST call or user profile. +**Do not** use real user information or email addresses in GitLab documentation. For email +addresses and names, do use: + +- **Email addresses**: Use an email address ending in `example.com`. +- **Names**: Use strings like `example_username`. Alternatively, use diverse or + non-gendered names with common surnames, such as `Sidney Jones`, `Zhang Wei`, + or `Maria Garcia`. + +### Fake URLs + +When including sample URLs in the documentation, use: + +- `example.com` when the domain name is generic. +- `gitlab.example.com` when referring to self-managed instances of GitLab. + +### Fake tokens + +There may be times where a token is needed to demonstrate an API call using +cURL or a variable used in CI. It is strongly advised not to use real tokens in +documentation even if the probability of a token being exploited is low. + +You can use the following fake tokens as examples: + +| Token type | Token value | +|:----------------------|:-------------------------------------------------------------------| +| Private user token | `<your_access_token>` | +| Personal access token | `n671WNGecHugsdEDPsyo` | +| Application ID | `2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6` | +| Application secret | `04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df` | +| CI/CD variable | `Li8j-mLUVA3eZYjPfd_H` | +| Specific runner token | `yrnZW46BrtBFqM7xDzE7dddd` | +| Shared runner token | `6Vk7ZsosqQyfreAxXTZr` | +| Trigger token | `be20d8dcc028677c931e04f3871a9b` | +| Webhook secret token | `6XhDroRcYPM5by_h-HLY` | +| Health check token | `Tu7BgjR9qeZTEyRzGG2P` | +| Request profile token | `7VgpS4Ax5utVD2esNstz` | + ### Language to avoid When creating documentation, limit or avoid the use of the following verb @@ -832,8 +887,10 @@ When creating tables of lists of features (such as whether or not features are available to certain roles on the [Permissions](../../user/permissions.md#project-members-permissions) page), use the following phrases (based on the SVG icons): -- *No*: **{dotted-circle}** No -- *Yes*: **{check-circle}** Yes +| Option | Markdown | Displayed result | +|--------|--------------------------|------------------------| +| No | `**{dotted-circle}** No` | **{dotted-circle}** No | +| Yes | `**{check-circle}** Yes` | **{check-circle}** Yes | ## Quotes @@ -890,8 +947,8 @@ search engine optimization (SEO), use the imperative, where possible. For guidelines on capitalizing headings, see the section on [capitalization](#capitalization). NOTE: **Note:** -If you change an existing title, be careful. Any such changes may affect not -only [links](#anchor-links) within the page, but may also affect links to the +If you change an existing title, be careful. These changes might affect not +only [links](#anchor-links) within the page, but might also affect links to the GitLab documentation from both the GitLab application and external sites. ### Anchor links @@ -1095,14 +1152,26 @@ document to ensure it links to the most recent version of the file. ## Navigation -To indicate the steps of navigation through the user interface: +When documenting navigation through the user interface: -- Use the exact word as shown in the UI, including any capital letters as-is. +- Use the exact wording as shown in the UI, including any capital letters as-is. - Use bold text for navigation items and the char "greater than" (`>`) as a - separator (for example, `Navigate to your project's **Settings > CI/CD**` ). + separator. For example: `Navigate to your project's **Settings > CI/CD**`. - If there are any expandable menus, make sure to mention that the user needs to - expand the tab to find the settings you're referring to (for example, - `Navigate to your project's **Settings > CI/CD** and expand **General pipelines**`). + expand the tab to find the settings you're referring to. For example: + `Navigate to your project's **Settings > CI/CD** and expand **General pipelines**`. + +### Navigational elements + +Use the following terms when referring to the main GitLab user interface +elements: + +- **Top menu**: This is the top menu that spans the width of the user interface. + It includes the GitLab logo, search field, counters, and the user's avatar. +- **Left sidebar**: This is the navigation sidebar on the left of the user + interface, specific to the project or group. +- **Right sidebar**: This is the navigation sidebar on the right of the user + interface, specific to the open issue, merge request, or epic. ## Images @@ -1162,9 +1231,6 @@ that: - Are accurate, succinct, and unique. - Don't use *image of …* or *graphic of…* to describe the image. -Also, if a heading immediately follows an image, be sure to add three dashes -(`---`) between the image and the heading. - ### Remove image shadow All images displayed on the [GitLab documentation site](https://docs.gitlab.com) @@ -1249,7 +1315,7 @@ reviewed and approved by a technical writer. 1. In YouTube, visit the video URL you want to display. Copy the regular URL from your browser (`https://www.youtube.com/watch?v=VIDEO-ID`) and replace the video title and link in the line under `<div class="video-fallback">`. -1. In YouTube, click **Share**, and then click **Embed**. +1. In YouTube, select **Share**, and then select **Embed**. 1. Copy the `<iframe>` source (`src`) **URL only** (`https://www.youtube.com/embed/VIDEO-ID`), and paste it, replacing the content of the `src` field in the @@ -1292,6 +1358,9 @@ the documentation site, but will be displayed on GitLab's `/help`. added to code blocks. To make things easier for the user, always add a full code block for things that can be useful to copy and paste, as they can easily do it with the button on code blocks. +- HTTP methods (`HTTP POST`) and HTTP status codes, both full (`404 File Not Found`) + and abbreviated (`404`), should be wrapped in inline code blocks when used in sentences. + For example: Send a `DELETE` request to delete the runner. Send a `POST` request to create one. - Add a blank line above and below code blocks. - When providing a shell command and its output, prefix the shell command with `$` and leave a blank line between the command and the output. @@ -1416,17 +1485,17 @@ interface: Use an icon when you find yourself having to describe an interface element. For example: -- Do: Click the Admin Area icon ( **{admin}** ). -- Don't: Click the Admin Area icon (the wrench icon). +- Do: Select the Admin Area icon ( **{admin}** ). +- Don't: Select the Admin Area icon (the wrench icon). ## Alert boxes When you need to call special attention to particular sentences, use the following markup to create highlighted alert boxes. -Note that the alert boxes only work for one paragraph only. Multiple paragraphs, -lists, headers and so on, will not render correctly. For multiple lines, use -[blockquotes](#blockquotes) instead. +Alert boxes work for one paragraph only. Multiple paragraphs, lists, and headers +won't render correctly. For multiple lines, use [blockquotes](#blockquotes) +instead. Alert boxes render only on the GitLab documentation site (<https://docs.gitlab.com>). Within GitLab itself, they will appear as plain Markdown text (like the examples @@ -1444,23 +1513,20 @@ guidelines, but for consistency you should try to use these values: ### Note -Notes catch the eye of most readers, and therefore should be used very sparingly. -In most cases, content considered for a note should be included: +Notes indicate additional information that is of special use to the reader. +Notes are most effective when used _sparingly_. -- As just another sentence in the previous paragraph or the most-relevant - paragraph. -- As its own standalone paragraph. -- As content under a new subheading that introduces the topic, making it more - visible or findable. +Try to avoid them. Too many notes can impact the scannability of a topic and +create an overly busy page. -#### When to use +Instead of adding a note, try one of these alternatives: -Use a note when there is a reason that most or all readers who browse the -section should see the content. That is, if missed, it’s likely to cause major -trouble for a minority of users or significant trouble for a majority of users. +- Re-write the sentence as part of the most-relevant paragraph. +- Put the information into its own standalone paragraph. +- Put the content under a new subheading that introduces the topic, which makes + it more visible. -Weigh the costs of distracting users to whom the content is not relevant against -the cost of users missing the content if it were not expressed as a note. +If you must use a note, use the following formatting: ```markdown NOTE: **Note:** @@ -1582,12 +1648,11 @@ application: The following are recommended verbs for specific uses with user interface elements: -| Recommended | Used for | Replaces | -|:--------------------|:---------------------------|:---------------------------| -| *click* | buttons, links, menu items | "hit", "press", "select" | -| *select* or *clear* | checkboxes | "enable", "click", "press" | -| *select* | dropdowns | "pick" | -| *expand* | expandable sections | "open" | +| Recommended | Used for | Replaces | +|:--------------------|:--------------------------------------|:---------------------------| +| *select* | buttons, links, menu items, dropdowns | "click, "press," "hit" | +| *select* or *clear* | checkboxes | "enable", "click", "press" | +| *expand* | expandable sections | "open" | ### Other Verbs @@ -1614,6 +1679,15 @@ heading level. ### Text for documentation requiring version text +When a feature is new or updated, you can add version information as a bulleted +item in the **Version history**, or as an inline reference with related text. + +#### Version text in the **Version History** + +If all content in a section is related, add version text in a bulleted list +following the heading for the section. To render correctly, it must be on its +own line and surrounded by blank lines. + - For features that need to declare the GitLab version that the feature was introduced. Text similar to the following should be added immediately below the heading as a blockquote: @@ -1663,9 +1737,20 @@ heading level. and replaced by [Feature name](link-to-feature-documentation). ``` -NOTE: **Note:** -Version text must be on its own line and surrounded by blank lines to render -correctly. +#### Inline version text + +If you're adding content to an existing topic, you can add version information +inline with the existing text. + +In this case, add `([introduced/deprecated](<link-to-issue>) in GitLab X.X)`. +If applicable, include the paid tier: `([introduced/deprecated](<link-to-issue>) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.4)` + +Including the issue link is encouraged, but isn't a requirement. For example: + +```markdown +The voting strategy (introduced in GitLab 13.4) requires +the primary and secondary voters to agree. +``` ### Versions in the past or future @@ -1894,216 +1979,9 @@ steps aren't required, consider setting up a [table](#tables) with headers of Learn how to [document features deployed behind flags](feature_flags.md). For guidance on developing GitLab with feature flags, see [Feature flags in development of GitLab](../feature_flags/index.md). -## RESTful API - -REST API resources are documented in Markdown under -[`/doc/api`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api). Each -resource has its own Markdown file, which is linked from `api_resources.md`. - -When modifying the Markdown, also update the corresponding -[OpenAPI definition](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api/openapi) -if one exists for the resource. If not, consider creating one. Match the latest -[OpenAPI 3.0.x specification](https://swagger.io/specification/). (For more -information, see the discussion in this -[issue](https://gitlab.com/gitlab-org/gitlab/-/issues/16023#note_370901810).) - -In the Markdown doc for a resource (AKA endpoint): - -- Every method must have the REST API request. For example: - - ```plaintext - GET /projects/:id/repository/branches - ``` - -- Every method must have a detailed [description of the parameters](#method-description). -- Every method must have a cURL example. -- Every method must have a response body (in JSON format). - -### API topic template - -The following can be used as a template to get started: - -````markdown -## Descriptive title - -One or two sentence description of what endpoint does. - -```plaintext -METHOD /endpoint -``` - -| Attribute | Type | Required | Description | -|:------------|:---------|:---------|:----------------------| -| `attribute` | datatype | yes/no | Detailed description. | -| `attribute` | datatype | yes/no | Detailed description. | - -Example request: - -```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/endpoint?parameters" -``` - -Example response: - -```json -[ - { - } -] -``` -```` - -### Fake user information - -You may need to demonstrate an API call or a cURL command that includes the name -and email address of a user. Don't use real user information in API calls: - -- **Email addresses**: Use an email address ending in `example.com`. -- **Names**: Use strings like `Example Username`. Alternatively, use diverse or - non-gendered names with common surnames, such as `Sidney Jones`, `Zhang Wei`, - or `Maria Garcia`. - -### Fake URLs - -When including sample URLs in the documentation, use: - -- `example.com` when the domain name is generic. -- `gitlab.example.com` when referring to self-managed instances of GitLab. - -### Fake tokens - -There may be times where a token is needed to demonstrate an API call using -cURL or a variable used in CI. It is strongly advised not to use real tokens in -documentation even if the probability of a token being exploited is low. - -You can use the following fake tokens as examples: - -| Token type | Token value | -|:----------------------|:-------------------------------------------------------------------| -| Private user token | `<your_access_token>` | -| Personal access token | `n671WNGecHugsdEDPsyo` | -| Application ID | `2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6` | -| Application secret | `04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df` | -| CI/CD variable | `Li8j-mLUVA3eZYjPfd_H` | -| Specific runner token | `yrnZW46BrtBFqM7xDzE7dddd` | -| Shared runner token | `6Vk7ZsosqQyfreAxXTZr` | -| Trigger token | `be20d8dcc028677c931e04f3871a9b` | -| Webhook secret token | `6XhDroRcYPM5by_h-HLY` | -| Health check token | `Tu7BgjR9qeZTEyRzGG2P` | -| Request profile token | `7VgpS4Ax5utVD2esNstz` | - -### Method description - -Use the following table headers to describe the methods. Attributes should -always be in code blocks using backticks (`` ` ``). - -```markdown -| Attribute | Type | Required | Description | -|:----------|:-----|:---------|:------------| -``` - -Rendered example: - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:--------------------| -| `user` | string | yes | The GitLab username | - -### cURL commands - -- Use `https://gitlab.example.com/api/v4/` as an endpoint. -- Wherever needed use this personal access token: `<your_access_token>`. -- Always put the request first. `GET` is the default so you don't have to - include it. -- Wrap the URL in double quotes (`"`). -- Prefer to use examples using the personal access token and don't pass data of - username and password. - -| Methods | Description | -|:------------------------------------------- |:------------------------------------------------------| -| `--header "PRIVATE-TOKEN: <your_access_token>"` | Use this method as is, whenever authentication needed | -| `--request POST` | Use this method when creating new objects | -| `--request PUT` | Use this method when updating existing objects | -| `--request DELETE` | Use this method when removing existing objects | - -### cURL Examples - -The following sections include a set of [cURL](https://curl.haxx.se) examples -you can use in the API documentation. - -#### Simple cURL command - -Get the details of a group: - -```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/gitlab-org" -``` - -#### cURL example with parameters passed in the URL - -Create a new project under the authenticated user's namespace: - -```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects?name=foo" -``` - -#### Post data using cURL's `--data` - -Instead of using `--request POST` and appending the parameters to the URI, you -can use cURL's `--data` option. The example below will create a new project -`foo` under the authenticated user's namespace. - -```shell -curl --data "name=foo" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects" -``` - -#### Post data using JSON content - -NOTE: **Note:** -In this example we create a new group. Watch carefully the single and double -quotes. - -```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"path": "my-group", "name": "My group"}' "https://gitlab.example.com/api/v4/groups" -``` - -#### Post data using form-data - -Instead of using JSON or urlencode you can use multipart/form-data which -properly handles data encoding: - -```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "title=ssh-key" --form "key=ssh-rsa AAAAB3NzaC1yc2EA..." "https://gitlab.example.com/api/v4/users/25/keys" -``` - -The above example is run by and administrator and will add an SSH public key -titled `ssh-key` to user's account which has an ID of 25. - -#### Escape special characters - -Spaces or slashes (`/`) may sometimes result to errors, thus it is recommended -to escape them when possible. In the example below we create a new issue which -contains spaces in its title. Observe how spaces are escaped using the `%20` -ASCII code. - -```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/42/issues?title=Hello%20Dude" -``` - -Use `%2F` for slashes (`/`). - -#### Pass arrays to API calls - -The GitLab API sometimes accepts arrays of strings or integers. For example, to -exclude specific users when requesting a list of users for a project, you would -do something like this: - -```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "skip_users[]=<user_id>" --data "skip_users[]=<user_id>" "https://gitlab.example.com/api/v4/projects/<project_id>/users" -``` - ## GraphQL API -GraphQL APIs are different from [RESTful APIs](#restful-api). Reference +GraphQL APIs are different from [RESTful APIs](restful_api_styleguide.md). Reference information is generated in our [GraphQL reference](../../api/graphql/reference/index.md). However, it's helpful to include examples on how to use GraphQL for different @@ -2158,7 +2036,7 @@ Set up the section with the following: ```markdown 1. Open the GraphiQL explorer tool in the following URL: `https://gitlab.com/-/graphql-explorer`. 1. Paste the `query` listed above into the left window of your GraphiQL explorer tool. - 1. Click Play to get the result shown here: + 1. Select Play to get the result shown here: ``` - Include a screenshot of the result in the GraphiQL explorer. Follow the naming diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index e70cf456101..639759e5014 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w This area is to maintain a compendium of useful information when working with Elasticsearch. Information on how to enable Elasticsearch and perform the initial indexing is in -the [Elasticsearch integration documentation](../integration/elasticsearch.md#enabling-elasticsearch). +the [Elasticsearch integration documentation](../integration/elasticsearch.md#enabling-advanced-search). ## Deep Dive diff --git a/doc/development/emails.md b/doc/development/emails.md index cf7f49ee834..de9607fef64 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -27,7 +27,7 @@ Please note that [S/MIME signed](../administration/smime_signing_email.md) email ## Mailer previews Rails provides a way to preview our mailer templates in HTML and plaintext using -dummy data. +sample data. The previews live in [`app/mailers/previews`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/app/mailers/previews) and can be viewed at [`/rails/mailers`](http://localhost:3000/rails/mailers). diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index 07b803603a5..00a39ac0fc6 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -16,46 +16,97 @@ In either case, an outcome of the experiment should be posted to the issue with ## Code reviews -Since the code of experiments will not be part of the codebase for a long time and we want to iterate fast to retrieve data, the code quality of experiments might sometimes not fulfill our standards but should not negatively impact the availability of GitLab whether the experiment is running or not. -Initially experiments will only be deployed to a fraction of users but we still want a flawless experience for those users. Therefore, experiments still require tests. - -For reviewers and maintainers: if you find code that would usually not make it through the review, but is temporarily acceptable, please mention your concerns but note that it's not necessary to change. -The author then adds a comment to this piece of code and adds a link to the issue that resolves the experiment. If the experiment is successful and becomes part of the product these follow up issues should be addressed. +Experiments' code quality can fail our standards for several reasons. These +reasons can include not being added to the codebase for a long time, or because +of fast iteration to retrieve data. However, having the experiment run (or not +run) shouldn't impact GitLab's availability. To avoid or identify issues, +experiments are initially deployed to a small number of users. Regardless, +experiments still need tests. + +If, as a reviewer or maintainer, you find code that would usually fail review +but is acceptable for now, mention your concerns with a note that there's no +need to change the code. The author can then add a comment to this piece of code +and link to the issue that resolves the experiment. If the experiment is +successful and becomes part of the product, any follow up issues should be +addressed. ## How to create an A/B test -- Add the experiment to the `Gitlab::Experimentation::EXPERIMENTS` hash in [`experimentation.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib%2Fgitlab%2Fexperimentation.rb): - - ```ruby - EXPERIMENTS = { - other_experiment: { - #... - }, - # Add your experiment here: - signup_flow: { - environment: ::Gitlab.dev_env_or_com?, # Target environment, defaults to enabled for development and GitLab.com - tracking_category: 'Growth::Acquisition::Experiment::SignUpFlow' # Used for providing the category when setting up tracking data - } - }.freeze - ``` - -- Use the experiment in a controller: - - ```ruby - class RegistrationController < ApplicationController - def show - # experiment_enabled?(:feature_name) is also available in views and helpers - if experiment_enabled?(:signup_flow) - # render the experiment - else - # render the original version - end - end - end - ``` - -- Track necessary events. See the [telemetry guide](../telemetry/index.md) for details. -- After the merge request is merged, use [`chatops`](../../ci/chatops/README.md) in the +### Implementation + +1. Add the experiment to the `Gitlab::Experimentation::EXPERIMENTS` hash in [`experimentation.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib%2Fgitlab%2Fexperimentation.rb): + + ```ruby + EXPERIMENTS = { + other_experiment: { + #... + }, + # Add your experiment here: + signup_flow: { + environment: ::Gitlab.dev_env_or_com?, # Target environment, defaults to enabled for development and GitLab.com + tracking_category: 'Growth::Acquisition::Experiment::SignUpFlow' # Used for providing the category when setting up tracking data + } + }.freeze + ``` + +1. Use the experiment in the code. + + - Use this standard for the experiment in a controller: + + ```ruby + class RegistrationController < ApplicationController + def show + # experiment_enabled?(:experiment_key) is also available in views and helpers + if experiment_enabled?(:signup_flow) + # render the experiment + else + # render the original version + end + end + end + ``` + + - Make the experiment available to the frontend in a controller: + + ```ruby + before_action do + push_frontend_experiment(:signup_flow) + end + ``` + + The above will check whether the experiment is enabled and push the result to the frontend. + + You can check the state of the feature flag in JavaScript: + + ```javascript + import { isExperimentEnabled } from '~/experimentation'; + + if ( isExperimentEnabled('signupFlow') ) { + // ... + } + ``` + + - It is also possible to run an experiment outside of the controller scope, for example in a worker: + + ```ruby + class SomeWorker + def perform + # Check if the experiment is enabled at all (the percentage_of_time_value > 0) + return unless Gitlab::Experimentation.enabled?(:experiment_key) + + # Since we cannot access cookies in a worker, we need to bucket models based on a unique, unchanging attribute instead. + # Use the following method to check if the experiment is enabled for a certain attribute, for example a username or email address: + if Gitlab::Experimentation.enabled_for_attribute?(:experiment_key, some_attribute) + # execute experimental code + else + # execute control code + end + end + end + ``` + +1. Track necessary events. See the [telemetry guide](../telemetry/index.md) for details. +1. After the merge request is merged, use [`chatops`](../../ci/chatops/README.md) in the [appropriate channel](../feature_flags/controls.md#communicate-the-change) to start the experiment for 10% of the users. The feature flag should have the name of the experiment with the `_experiment_percentage` suffix appended. For visibility, please also share any commands run against production in the `#s_growth` channel: @@ -69,3 +120,19 @@ For visibility, please also share any commands run against production in the `#s ```shell /chatops run feature delete signup_flow_experiment_percentage ``` + +### Tests and test helpers + +Use the following in Jest to test the experiment is enabled. + +```javascript +import { withGonExperiment } from 'helpers/experimentation_helper'; + +describe('given experiment is enabled', () => { + withGonExperiment('signupFlow'); + + it('should do the experimental thing', () => { + expect(wrapper.find('.js-some-experiment-triggered-element')).toEqual(expect.any(Element)); + }); +}); +``` diff --git a/doc/development/fe_guide/architecture.md b/doc/development/fe_guide/architecture.md index 3d27f67a8a6..c7e1ba59f23 100644 --- a/doc/development/fe_guide/architecture.md +++ b/doc/development/fe_guide/architecture.md @@ -15,5 +15,5 @@ You can find the Frontend Architecture experts on the [team page](https://about. ## Examples -You can find documentation about the desired architecture for a new feature -built with Vue.js [here](vue.md). +You can find [documentation about the desired architecture](vue.md) for a new +feature built with Vue.js. diff --git a/doc/development/fe_guide/dependencies.md b/doc/development/fe_guide/dependencies.md index 7f078df887d..ba97e7e39be 100644 --- a/doc/development/fe_guide/dependencies.md +++ b/doc/development/fe_guide/dependencies.md @@ -18,9 +18,9 @@ automatically create merge requests for updating dependencies of several project up-to-date list of projects managed by the renovate bot in the project’s README. Some key dependencies updated using renovate are: -- [`@gitlab/ui`](https://gitlab.com/gitlab-org/gitlab-ui/) -- [`@gitlab/svgs`](https://gitlab.com/gitlab-org/gitlab-svgs/) -- [`@gitlab/eslint-config`](https://gitlab.com/gitlab-org/gitlab-eslint-config) +- [`@gitlab/ui`](https://gitlab.com/gitlab-org/gitlab-ui) +- [`@gitlab/svgs`](https://gitlab.com/gitlab-org/gitlab-svgs) +- [`@gitlab/eslint-plugin`](https://gitlab.com/gitlab-org/frontend/eslint-plugin) ### Blocked dependencies diff --git a/doc/development/fe_guide/icons.md b/doc/development/fe_guide/icons.md index b539293e9cf..67add5709d9 100644 --- a/doc/development/fe_guide/icons.md +++ b/doc/development/fe_guide/icons.md @@ -1,9 +1,10 @@ # Icons and SVG Illustrations -We manage our own Icon and Illustration library in the [`gitlab-svgs`](https://gitlab.com/gitlab-org/gitlab-svgs) repository. -This repository is published on [npm](https://www.npmjs.com/package/@gitlab/svgs) and managed as a dependency via yarn. -You can browse all available Icons and Illustrations [here](https://gitlab-org.gitlab.io/gitlab-svgs). -To upgrade to a new version run `yarn upgrade @gitlab/svgs`. +We manage our own icon and illustration library in the [`gitlab-svgs`](https://gitlab.com/gitlab-org/gitlab-svgs) +repository. This repository is published on [npm](https://www.npmjs.com/package/@gitlab/svgs), +and is managed as a dependency with yarn. You can browse all available +[icons and illustrations](https://gitlab-org.gitlab.io/gitlab-svgs). To upgrade +to a new version run `yarn upgrade @gitlab/svgs`. ## Icons @@ -21,10 +22,11 @@ To use a sprite Icon in HAML or Rails we use a specific helper function : sprite_icon(icon_name, size: nil, css_class: '') ``` -- **icon_name** Use the icon_name that you can find in the SVG Sprite - ([Overview is available here](https://gitlab-org.gitlab.io/gitlab-svgs)). -- **size (optional)** Use one of the following sizes : 16, 24, 32, 48, 72 (this will be translated into a `s16` class) -- **css_class (optional)** If you want to add additional CSS classes +- **icon_name**: Use the icon_name for the SVG sprite in the list of + ([GitLab SVGs](https://gitlab-org.gitlab.io/gitlab-svgs)). +- **size (optional)**: Use one of the following sizes : 16, 24, 32, 48, 72 (this + will be translated into a `s16` class) +- **css_class (optional)**: If you want to add additional CSS classes. **Example** @@ -66,10 +68,12 @@ export default { </template> ``` -- **name** Name of the Icon in the SVG Sprite ([Overview is available here](https://gitlab-org.gitlab.io/gitlab-svgs)). -- **size (optional)** Number value for the size which is then mapped to a specific CSS class - (Available Sizes: 8, 12, 16, 18, 24, 32, 48, 72 are mapped to `sXX` CSS classes) -- **class (optional)** Additional CSS Classes to add to the SVG tag. +- **name**: Name of the icon of the SVG sprite, as listed in the + ([GitLab SVG Previewer](https://gitlab-org.gitlab.io/gitlab-svgs)). +- **size: (optional)** Number value for the size which is then mapped to a + specific CSS class (Available sizes: 8, 12, 16, 18, 24, 32, 48, 72 are mapped + to `sXX` CSS classes) +- **class (optional)**: Additional CSS classes to add to the SVG tag. ### Usage in HTML/JS diff --git a/doc/development/fe_guide/tooling.md b/doc/development/fe_guide/tooling.md index 5685ac5abcd..b93c0a9736b 100644 --- a/doc/development/fe_guide/tooling.md +++ b/doc/development/fe_guide/tooling.md @@ -101,7 +101,10 @@ Our code is automatically formatted with [Prettier](https://prettier.io) to foll ### Editor -The easiest way to include prettier in your workflow is by setting up your preferred editor (all major editors are supported) accordingly. We suggest setting up prettier to run automatically when each file is saved. Find [here](https://prettier.io/docs/en/editors.html) the best way to set it up in your preferred editor. +The recommended method to include Prettier in your workflow is to set up your +preferred editor (all major editors are supported) accordingly. We suggest +setting up Prettier to run when each file is saved. For instructions about using +Prettier in your preferred editor, see the [Prettier documentation](https://prettier.io/docs/en/editors.html). Please take care that you only let Prettier format the same file types as the global Yarn script does (`.js`, `.vue`, `.graphql`, and `.scss`). In VSCode by example you can easily exclude file formats in your settings file: diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 4badf3f0845..528dcb3b7f4 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -32,8 +32,9 @@ When using Vuex at GitLab, separate these concerns into different files to impro └── mutation_types.js # mutation types ``` -The following example shows an application that lists and adds users to the state. -(For a more complex example implementation take a look at the security applications store in [here](https://gitlab.com/gitlab-org/gitlab/tree/master/ee/app/assets/javascripts/vue_shared/security_reports/store)) +The following example shows an application that lists and adds users to the +state. (For a more complex example implementation, review the security +applications stored in this [repository](https://gitlab.com/gitlab-org/gitlab/tree/master/ee/app/assets/javascripts/vue_shared/security_reports/store)). ### `index.js` diff --git a/doc/development/feature_categorization/index.md b/doc/development/feature_categorization/index.md index 3fd402ebe84..31ec9ee65b6 100644 --- a/doc/development/feature_categorization/index.md +++ b/doc/development/feature_categorization/index.md @@ -75,38 +75,23 @@ A feature category can be specified on an entire controller using: ```ruby -class Projects::MergeRequestsController < ApplicationController - feature_category :source_code_management +class Boards::ListsController < ApplicationController + feature_category :kanban_boards end ``` The feature category can be limited to a list of actions using the -`only` argument, actions can be excluded using the `except` argument. +second argument: ```ruby -class Projects::MergeRequestsController < ApplicationController - feature_category :code_testing, only: [:metrics_reports] - feature_category :source_code_management, except: [:test_reports, :coverage_reports] +class DashboardController < ApplicationController + feature_category :issue_tracking, [:issues, :issues_calendar] + feature_category :code_review, [:merge_requests] end ``` -`except` and `only` arguments can not be combined. - -When specifying `except` all other actions will get the specified -category assigned. - -The assignment can also be scoped using `if` and `unless` procs: - -```ruby -class Projects::MergeRequestsController < ApplicationController - feature_category :source_code_management, - unless: -> (action) { action.include?("reports") } - if: -> (action) { action.include?("widget") } -end -``` - -In this case, both procs need to be satisfied for the action to get -the category assigned. +These forms cannot be mixed: if a controller has more than one category, +every single action must be listed. ### Excluding controller actions from feature categorization @@ -125,6 +110,5 @@ The `spec/controllers/every_controller_spec.rb` will iterate over all defined routes, and check the controller to see if a category is assigned to all actions. -The spec also validates if the used feature categories are known. And -if the actions used in `only` and `except` configuration still exist -as routes. +The spec also validates if the used feature categories are known. And if +the actions used in configuration still exist as routes. diff --git a/doc/development/feature_flags/development.md b/doc/development/feature_flags/development.md index 29bd0ca0a7e..a75f1f818dd 100644 --- a/doc/development/feature_flags/development.md +++ b/doc/development/feature_flags/development.md @@ -52,10 +52,10 @@ invocations: ```ruby # Check if feature flag is enabled -Feature.enabled?(:my_ops_flag, project, type: ops) +Feature.enabled?(:my_ops_flag, project, type: :ops) # Check if feature flag is disabled -Feature.disabled?(:my_ops_flag, project, type: ops) +Feature.disabled?(:my_ops_flag, project, type: :ops) # Push feature flag to Frontend push_frontend_feature_flag(:my_ops_flag, project, type: :ops) diff --git a/doc/development/frontend.md b/doc/development/frontend.md index f46cc377f95..8bb5cf7af62 100644 --- a/doc/development/frontend.md +++ b/doc/development/frontend.md @@ -1,4 +1,5 @@ +--- +redirect_to: 'fe_guide/index.md' +--- -# Frontend Development Guidelines - -This page has moved [here](fe_guide/index.md). +This document was moved to [another location](fe_guide/index.md). diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index b720a6ca47e..bc7a5c57916 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -91,8 +91,6 @@ module Geo ::Packages::PackageFile end - # Change this to `true` to release replication of this model. Then remove - # this override in the next release. # The feature flag follows the format `geo_#{replicable_name}_replication`, # so here it would be `geo_package_file_replication` def self.replication_enabled_by_default? @@ -220,8 +218,6 @@ For example, to add support for files referenced by a `Widget` model with a model_record.file end - # Change this to `true` to release replication of this model. Then remove - # this override in the next release. # The feature flag follows the format `geo_#{replicable_name}_replication`, # so here it would be `geo_widget_replication` def self.replication_enabled_by_default? @@ -637,7 +633,7 @@ the Admin Area UI, and Prometheus! include ::Types::Geo::RegistryType graphql_name 'WidgetRegistry' - description 'Represents the sync and verification state of a widget' + description 'Represents the Geo sync and verification state of a widget' field :widget_id, GraphQL::ID_TYPE, null: false, description: 'ID of the Widget' end @@ -676,6 +672,12 @@ the Admin Area UI, and Prometheus! } ``` +1. Update the GraphQL reference documentation: + + ```shell + bundle exec rake gitlab:graphql:compile_docs + ``` + Individual widget synchronization and verification data should now be available via the GraphQL API! @@ -693,3 +695,33 @@ To do: This should be done as part of Widget sync and verification data (aggregate and individual) should now be available in the Admin UI! + +#### Releasing the feature + +1. In `ee/app/replicators/geo/widget_replicator.rb`, delete the `self.replication_enabled_by_default?` method: + + ```ruby + module Geo + class WidgetReplicator < Gitlab::Geo::Replicator + ... + + # REMOVE THIS METHOD + def self.replication_enabled_by_default? + false + end + # REMOVE THIS METHOD + + ... + end + end + ``` + +1. In `ee/app/graphql/types/geo/geo_node_type.rb`, remove the `feature_flag` option for the released type: + + ```ruby + field :widget_registries, ::Types::Geo::WidgetRegistryType.connection_type, + null: true, + resolver: ::Resolvers::Geo::WidgetRegistriesResolver, + description: 'Find widget registries on this Geo node', + feature_flag: :geo_widget_replication # REMOVE THIS LINE + ``` diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index f7b44e74c17..1044b7edcb8 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -3,6 +3,21 @@ The purpose of this guide is to document potential "gotchas" that contributors might encounter or should avoid during development of GitLab CE and EE. +## Do not read files from app/assets directory + +Since GitLab 10.8, Omnibus has [dropped the `app/assets` directory](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2456), +after asset compilation. The `ee/app/assets`, `vendor/assets` directories are dropped as well. + +This means that reading files from that directory will fail in Omnibus-installed GitLab instances: + +```ruby +file = Rails.root.join('app/assets/images/logo.svg') + +# This file does not exist, read will fail with: +# Errno::ENOENT: No such file or directory @ rb_sysopen +File.read(file) +``` + ## Do not assert against the absolute value of a sequence-generated attribute Consider the following factory: diff --git a/doc/development/graphql_guide/index.md b/doc/development/graphql_guide/index.md index 62650f7cd3f..9d7fb5ba0a8 100644 --- a/doc/development/graphql_guide/index.md +++ b/doc/development/graphql_guide/index.md @@ -8,6 +8,6 @@ feedback, and suggestions. - [GraphQL API development style guide](../api_graphql_styleguide.md): development style guide for GraphQL. -- [GraphQL API documentation style guide](../documentation/styleguide.md#graphql-api): documentation +- [GraphQL API documentation style guide](../documentation/graphql_styleguide.md): documentation style guide for GraphQL. - [GraphQL API](../../api/graphql/index.md): user documentation for the GitLab GraphQL API. diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index b5c5a199b1e..cc711589916 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -432,7 +432,7 @@ To avoid this error, use the applicable HTML entity code (`<` or `>`) inst - In JavaScript: ```javascript - import { sanitize } from 'dompurify'; + import { sanitize } from '~/lib/dompurify'; const i18n = { LESS_THAN_ONE_HOUR: sanitize(__('In < 1 hour'), { ALLOWED_TAGS: [] }) }; diff --git a/doc/development/i18n/index.md b/doc/development/i18n/index.md index 929eded3f8e..2d84fe4536f 100644 --- a/doc/development/i18n/index.md +++ b/doc/development/i18n/index.md @@ -38,10 +38,10 @@ Voting for translations is also valuable, helping to confirm good and flag inacc See [Translation guidelines](translation.md). -### Proof reading +### Proofreading -Proof reading helps ensure the accuracy and consistency of translations. All -translations are proof read before being accepted. If a translations requires +Proofreading helps ensure the accuracy and consistency of translations. All +translations are proofread before being accepted. If a translations requires changes, you will be notified with a comment explaining why. See [Proofreading Translations](proofreader.md) for more information on who's diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index dcfd0f40bf0..b085f1244c1 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -374,12 +374,16 @@ which is shared by the analyzers that GitLab maintains. You can [contribute](htt new generic identifiers to if needed. Analyzers may also produce vendor-specific or product-specific identifiers, which don't belong in the [common library](https://gitlab.com/gitlab-org/security-products/analyzers/common). -The first item of the `identifiers` array is called the primary identifier. +The first item of the `identifiers` array is called the [primary +identifier](../../user/application_security/terminology/#primary-identifier). The primary identifier is particularly important, because it is used to [track vulnerabilities](#tracking-and-merging-vulnerabilities) as new commits are pushed to the repository. Identifiers are also used to [merge duplicate vulnerabilities](#tracking-and-merging-vulnerabilities) reported for the same commit, except for `CWE` and `WASC`. +Not all vulnerabilities have CVEs, and a CVE can be identified multiple times. As a result, a CVE +isn't a stable identifier and you shouldn't assume it as such when tracking vulnerabilities. + ### Location The `location` indicates where the vulnerability has been detected. diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index 830cb84e257..19fd86f4bf6 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -36,7 +36,7 @@ best place to integrate your own product and its results into GitLab. - Pipeline jobs serve a variety of purposes. Jobs can do scanning for and have implications for app security, corporate policy, or compliance. When complete, the job reports back on its status and creates a - [job artifact](../../user/project/pipelines/job_artifacts.md) as a result. + [job artifact](../../ci/pipelines/job_artifacts.md) as a result. - The [Merge Request Security Widget](../../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports) displays the results of the pipeline's security checks and the developer can review them. The developer can review both a summary and a detailed version @@ -44,7 +44,7 @@ best place to integrate your own product and its results into GitLab. - If certain policies (such as [merge request approvals](../../user/project/merge_requests/merge_request_approvals.md)) are in place for a project, developers must resolve specific findings or get an approval from a specific list of people. -- The [security dashboard](../../user/application_security/security_dashboard/index.md#gitlab-security-dashboard) +- The [security dashboard](../../user/application_security/security_dashboard/index.md) also shows results which can developers can use to quickly see all the vulnerabilities that need to be addressed in the code. - When the developer reads the details about a vulnerability, they are @@ -78,7 +78,7 @@ and complete an integration with the Secure stage. to successfully display your own product's results with the rest of GitLab. - See detailed [technical directions](secure.md) for this step. - Read more about [job report artifacts](../../ci/pipelines/job_artifacts.md#artifactsreports). - - Read about [job artifacts](../../user/project/pipelines/job_artifacts.md). + - Read about [job artifacts](../../ci/pipelines/job_artifacts.md). - Your report artifact must be in one of our currently supported formats. For more information, see the [documentation on reports](secure.md#report). - Documentation for [SAST reports](../../user/application_security/sast/index.md#reports-json-format). @@ -89,7 +89,7 @@ and complete an integration with the Secure stage. and add the label `devops::secure`. - Once the job is completed, the data can be seen: - In the [Merge Request Security Report](../../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports) ([MR Security Report data flow](https://gitlab.com/snippets/1910005#merge-request-view)). - - While [browsing a Job Artifact](../../user/project/pipelines/job_artifacts.md). + - While [browsing a Job Artifact](../../ci/pipelines/job_artifacts.md). - In the [Security Dashboard](../../user/application_security/security_dashboard/index.md) ([Dashboard data flow](https://gitlab.com/snippets/1910005#project-and-group-dashboards)). 1. Optional: Provide a way to interact with results as Vulnerabilities: - Users can interact with the findings from your artifact within their workflow. They can dismiss the findings or accept them and create a backlog issue. diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md index 64089d51c7b..a54798b4c35 100644 --- a/doc/development/kubernetes.md +++ b/doc/development/kubernetes.md @@ -15,14 +15,14 @@ This document provides various guidelines when developing for GitLab's Some Kubernetes operations, such as creating restricted project namespaces are performed on the GitLab Rails application. These -operations are performed using a [client library](#client-library). -These operations will carry an element of risk as the operations will be -run as the same user running the GitLab Rails application, see the -[security](#security) section below. +operations are performed using a [client library](#client-library), +and carry an element of risk. The operations are +run as the same user running the GitLab Rails application. For more information, +read the [security](#security) section below. Some Kubernetes operations, such as installing cluster applications are performed on one-off pods on the Kubernetes cluster itself. These -installation pods are currently named `install-<application_name>` and +installation pods are named `install-<application_name>` and are created within the `gitlab-managed-apps` namespace. In terms of code organization, we generally add objects that represent @@ -33,33 +33,32 @@ Kubernetes resources in We use the [`kubeclient`](https://rubygems.org/gems/kubeclient) gem to perform Kubernetes API calls. As the `kubeclient` gem does not support -different API Groups (e.g. `apis/rbac.authorization.k8s.io`) from a +different API Groups (such as `apis/rbac.authorization.k8s.io`) from a single client, we have created a wrapper class, [`Gitlab::Kubernetes::KubeClient`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/kubernetes/kube_client.rb) -that will enable you to achieve this. +that enable you to achieve this. -Selected Kubernetes API groups are currently supported. Do add support +Selected Kubernetes API groups are supported. Do add support for new API groups or methods to [`Gitlab::Kubernetes::KubeClient`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/kubernetes/kube_client.rb) if you need to use them. New API groups or API group versions can be -added to `SUPPORTED_API_GROUPS` - internally, this will create an +added to `SUPPORTED_API_GROUPS` - internally, this creates an internal client for that group. New methods can be added as a delegation to the relevant internal client. ### Performance considerations -All calls to the Kubernetes API must be in a background process. Do not -perform Kubernetes API calls within a web request as this will block -Unicorn and can easily lead to a Denial Of Service (DoS) attack in GitLab as +All calls to the Kubernetes API must be in a background process. Don't +perform Kubernetes API calls within a web request. This blocks +Unicorn, and can lead to a denial-of-service (DoS) attack in GitLab as the Kubernetes cluster response times are outside of our control. The easiest way to ensure your calls happen a background process is to delegate any such work to happen in a [Sidekiq worker](sidekiq_style_guide.md). -There are instances where you would like to make calls to Kubernetes and -return the response and as such a background worker does not seem to be -a good fit. For such cases you should make use of [reactive -caching](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/reactive_caching.rb). +You may want to make calls to Kubernetes and return the response, but a background +worker isn't a good fit. Consider using +[reactive caching](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/reactive_caching.rb). For example: ```ruby @@ -76,7 +75,7 @@ For example: ### Testing -We have some Webmock stubs in +We have some WebMock stubs in [`KubernetesHelpers`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/support/helpers/kubernetes_helpers.rb) which can help with mocking out calls to Kubernetes API in your tests. @@ -86,8 +85,8 @@ This section outlines the process for allowing a GitLab instance to create EKS c The following prerequisites are required: -A `Customer` AWS account. This is the account in which the -EKS cluster will be created. The following resources must be present: +A `Customer` AWS account. The EKS cluster is created in this account. The following +resources must be present: - A provisioning role that has permissions to create the cluster and associated resources. It must list the `GitLab` AWS account @@ -114,7 +113,7 @@ The process for creating a cluster is as follows: which takes somewhere between 10 and 15 minutes in most cases. 1. When the stack is ready, GitLab stores the cluster details and generates another set of temporary credentials, this time to allow connecting to - the cluster via Kubeclient. These credentials are valid for one minute. + the cluster via `kubeclient`. These credentials are valid for one minute. 1. GitLab configures the worker nodes so that they are able to authenticate to the cluster, and creates a service account for itself for future operations. 1. Credentials that are no longer required are removed. This deletes the following @@ -126,7 +125,7 @@ The process for creating a cluster is as follows: ## Security -### SSRF +### Server Side Request Forgery (SSRF) attacks As URLs for Kubernetes clusters are user controlled it is easily susceptible to Server Side Request Forgery (SSRF) attacks. You should @@ -153,11 +152,11 @@ Mitigation strategies include: app.make_errored!("Kubernetes error: #{e.error_code}") ``` -## Debugging +## Debugging Kubernetes integrations Logs related to the Kubernetes integration can be found in [`kubernetes.log`](../administration/logs.md#kuberneteslog). On a local -GDK install, this will be present in `log/kubernetes.log`. +GDK install, these logs are present in `log/kubernetes.log`. Some services such as [`Clusters::Applications::InstallService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/clusters/applications/install_service.rb#L18) @@ -176,7 +175,9 @@ kubectl logs <pod_name> --follow -n gitlab-managed-apps ## GitLab Managed Apps -GitLab provides [GitLab Managed Apps](../user/clusters/applications.md), a one-click install for various applications which can be added directly to your configured cluster. +GitLab provides [GitLab Managed Apps](../user/clusters/applications.md), a one-click +install for various applications which can be added directly to your configured cluster. **<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview of how to add a new GitLab-managed app, see [How to add GitLab-managed-apps to Kubernetes integration](https://youtu.be/mKm-jkranEk).** +For an overview of how to add a new GitLab-managed app, see +[How to add GitLab-managed-apps to Kubernetes integration](https://youtu.be/mKm-jkranEk).** diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 207dd02d258..740ac54c415 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -647,7 +647,7 @@ tables: `namespaces`. This can be translated to: ```sql ALTER TABLE namespaces ALTER COLUMN request_access_enabled -DEFAULT false +SET DEFAULT false ``` In this particular case, the default value exists and we're just changing the metadata for diff --git a/doc/development/new_fe_guide/development/accessibility.md b/doc/development/new_fe_guide/development/accessibility.md index b9ee5c3a549..9c63ccad6e1 100644 --- a/doc/development/new_fe_guide/development/accessibility.md +++ b/doc/development/new_fe_guide/development/accessibility.md @@ -12,7 +12,7 @@ WAI-ARIA (the Accessible Rich Internet Applications specification) defines a way The `role` attribute describes the role the element plays in the context of the document. -Check the list of WAI-ARIA roles [here](https://www.w3.org/TR/wai-aria-1.1/#landmark_roles) +Review the list of [WAI-ARIA roles](https://www.w3.org/TR/wai-aria-1.1/#landmark_roles). ## Icons diff --git a/doc/development/new_fe_guide/development/index.md b/doc/development/new_fe_guide/development/index.md index 5dced3dc466..119dbc58012 100644 --- a/doc/development/new_fe_guide/development/index.md +++ b/doc/development/new_fe_guide/development/index.md @@ -12,6 +12,6 @@ Learn how to implement an accessible frontend. Learn how to keep our frontend performant. -## [Testing](testing.md) +## [Testing](../../testing_guide/frontend_testing.md) Learn how to keep our frontend tested. diff --git a/doc/development/packages.md b/doc/development/packages.md index 55e22d4bb5f..9504d5802c3 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -87,7 +87,7 @@ Composer package naming scope is Instance Level. To avoid name conflict for instance-level endpoints you will need to define a package naming convention that gives a way to identify the project that the package belongs to. This generally involves using the project ID or full project path in the package name. See -[Conan's naming convention](../user/packages/conan_repository/index.md#package-recipe-naming-convention-for-instance-level-remote) as an example. +[Conan's naming convention](../user/packages/conan_repository/index.md#package-recipe-naming-convention-for-instance-remotes) as an example. For group and project-level endpoints, naming can be less constrained and it will be up to the group and project members to be certain that there is no conflict between two package names. However, the system should prevent diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 7756ef376fc..d12220d8c95 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -122,7 +122,6 @@ graph RL; click 2_2-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8404303&udv=0" subgraph "Needs `setup-test-env` & `compile-test-assets`"; 2_2-2 & 2_2-4 & 2_2-5 --> 1-6 & 1-3; - 2_2-3 --> 1-6 & 1-4; end 2_3-1["build-assets-image (2.5 minutes)"]; @@ -228,7 +227,6 @@ graph RL; click 2_2-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8404303&udv=0" subgraph "Needs `setup-test-env` & `compile-test-assets`"; 2_2-2 & 2_2-4 & 2_2-5 --> 1-6 & 1-3; - 2_2-3 --> 1-6 & 1-4; end 2_3-1["build-assets-image (2.5 minutes)"]; diff --git a/doc/development/profiling.md b/doc/development/profiling.md index f7ead94d725..f5a4d1edb92 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -91,9 +91,6 @@ that builds on this to add some additional niceties, such as allowing configuration with a single YAML file for multiple URLs, and uploading of the profile and log output to S3. -For GitLab.com, you can find the latest results here (restricted to GitLab Team members only): -`https://redash.gitlab.com/dashboard/gitlab-profiler-statistics` - ## Sherlock Sherlock is a custom profiling tool built into GitLab. Sherlock is _only_ diff --git a/doc/development/prometheus.md b/doc/development/prometheus.md index 902d4e6a1d0..b8cf5b997ba 100644 --- a/doc/development/prometheus.md +++ b/doc/development/prometheus.md @@ -47,3 +47,13 @@ using a Prometheus managed application in Kubernetes: ``` 1. Open `localhost:9090` in your browser to display the Prometheus user interface. + +## Script access to Prometheus + +You can script the access to Prometheus, extracting the name of the pod automatically like this: + +```shell +POD_INFORMATION=$(kubectl get pods -n gitlab-managed-apps | grep 'prometheus-prometheus-server') +POD_NAME=$(echo $POD_INFORMATION | awk '{print $1;}') +kubectl port-forward $POD_NAME 9090:9090 -n gitlab-managed-apps +``` diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index f3386305e93..cf125c46565 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -85,9 +85,7 @@ The ReactiveCaching concern can be used in models as well as `project_services` 1. Implement the `calculate_reactive_cache` method in your model/service. 1. Call `with_reactive_cache` in your model/service where the cached value is needed. -1. If the `calculate_reactive_cache` method above submits requests to external services -(e.g. Prometheus, K8s), make sure to change the -[`reactive_cache_work_type` accordingly](#selfreactive_cache_work_type). +1. Set the [`reactive_cache_work_type` accordingly](#selfreactive_cache_work_type). ### In controllers @@ -252,7 +250,7 @@ self.reactive_cache_hard_limit = 5.megabytes - This is the type of work performed by the `calculate_reactive_cache` method. Based on this attribute, it's able to pick the right worker to process the caching job. Make sure to set it as `:external_dependency` if the work performs any external request -(e.g. Kubernetes, Sentry). +(e.g. Kubernetes, Sentry); otherwise set it to `:no_dependency`. #### `self.reactive_cache_worker_finder` diff --git a/doc/development/redis.md b/doc/development/redis.md index d205082b9c6..530a97c33a7 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -68,6 +68,10 @@ which is enabled for the `cache` and `shared_state` ## Redis in structured logging +For GitLab Team Members: There are [basic](https://www.youtube.com/watch?v=Uhdj19Dc6vU) and +[advanced](https://youtu.be/jw1Wv2IJxzs) videos that show how you can work with the Redis +structured logging fields on GitLab.com. + Our [structured logging](logging.md#use-structured-json-logging) for web requests and Sidekiq jobs contains fields for the duration, call count, bytes written, and bytes read per Redis instance, along with a total for @@ -96,10 +100,14 @@ requests that read the most data from the cache, we can just sort by ### The slow log +TIP: **Tip:** +There is a [video showing how to see the slow log](https://youtu.be/BBI68QuYRH8) (GitLab internal) +on GitLab.com + On GitLab.com, entries from the [Redis slow log](https://redis.io/commands/slowlog) are available in the `pubsub-redis-inf-gprd*` index with the [`redis.slowlog` -tag](https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(json.type,json.command,json.exec_time),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWSQX_Vf93rHTYrsexmk,key:json.tag,negate:!f,params:(query:redis.slowlog),type:phrase),query:(match:(json.tag:(query:redis.slowlog,type:phrase))))),index:AWSQX_Vf93rHTYrsexmk)). +tag](https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(json.type,json.command,json.exec_time_s),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWSQX_Vf93rHTYrsexmk,key:json.tag,negate:!f,params:(query:redis.slowlog),type:phrase),query:(match:(json.tag:(query:redis.slowlog,type:phrase))))),index:AWSQX_Vf93rHTYrsexmk)). This shows commands that have taken a long time and may be a performance concern. diff --git a/doc/development/reference_processing.md b/doc/development/reference_processing.md index 527fb94f228..cf587043cae 100644 --- a/doc/development/reference_processing.md +++ b/doc/development/reference_processing.md @@ -77,6 +77,22 @@ a minimum implementation of `AbstractReferenceFilter` should define: and an identifier, find the object. For example, this in a reference filter for merge requests, this might be `project.merge_requests.where(iid: iid)`. +### Add a new reference prefix and filter + +For reference filters for new objects, use a prefix format following the pattern +`^<object_type>#`, because: + +1. Varied single-character prefixes are hard for users to track. Especially for + lower-use object types, this can diminish value for the feature. +1. Suitable single-character prefixes are limited. +1. Following a consistent pattern allows users to infer the existence of new features. + +To add a reference prefix for a new object `apple`,which has both a name and ID, +format the reference as: + +- `^apple#123` for identification by ID. +- `^apple#"Granny Smith"` for identification by name. + ### Performance #### Find object optimization diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 1961d1dcc34..8e7cc1ec9f2 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -82,7 +82,8 @@ This Ruby Regex specialty can have security impact, as often regular expressions #### Examples -GitLab specific examples can be found [here](https://gitlab.com/gitlab-org/gitlab/-/issues/36029#note_251262187) and [there](https://gitlab.com/gitlab-org/gitlab/-/issues/33569). +GitLab-specific examples can be found in the following [path traversal](https://gitlab.com/gitlab-org/gitlab/-/issues/36029#note_251262187) +and [open redirect](https://gitlab.com/gitlab-org/gitlab/-/issues/33569) issues. Another example would be this fictional Ruby on Rails controller: diff --git a/doc/development/shell_scripting_guide/index.md b/doc/development/shell_scripting_guide/index.md index 622c90d7a97..704b46c7efa 100644 --- a/doc/development/shell_scripting_guide/index.md +++ b/doc/development/shell_scripting_guide/index.md @@ -1,7 +1,5 @@ # Shell scripting standards and style guidelines -## Overview - GitLab consists of many various services and sub-projects. The majority of their backend code is written in [Ruby](https://www.ruby-lang.org) and [Go](https://golang.org). However, some of them use shell scripts for @@ -15,7 +13,7 @@ should be eventually harmonized with this guide. If there are any per-project deviations from this guide, they should be described in the `README.md` or `PROCESS.md` file for such a project. -### Avoid using shell scripts +## Avoid using shell scripts CAUTION: **Caution:** This is a must-read section. diff --git a/doc/development/sql.md b/doc/development/sql.md index 3b969c7d27a..55a8192578b 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -220,7 +220,7 @@ Project.select(:id, :user_id).joins(:merge_requests) Never use ActiveRecord's `pluck` to pluck a set of values into memory only to use them as an argument for another query. For example, this will execute an -extra unecessary database query and load a lot of unecessary data into memory: +extra unnecessary database query and load a lot of unnecessary data into memory: ```ruby projects = Project.all.pluck(:id) diff --git a/doc/development/telemetry/snowplow.md b/doc/development/telemetry/snowplow.md index f427d7d1488..d6e3f0de2c8 100644 --- a/doc/development/telemetry/snowplow.md +++ b/doc/development/telemetry/snowplow.md @@ -297,6 +297,11 @@ class Projects::CreateService < BaseService end ``` +### Unit testing + +Use the `expect_snowplow_event` helper when testing backend Snowplow events. See [testing best practices]( +https://docs.gitlab.com/ee/development/testing_guide/best_practices.html#test-snowplow-events) for details. + ### Performance We use the [AsyncEmitter](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#52-the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. diff --git a/doc/development/telemetry/usage_ping.md b/doc/development/telemetry/usage_ping.md index ff4e7e0797b..a671ac17621 100644 --- a/doc/development/telemetry/usage_ping.md +++ b/doc/development/telemetry/usage_ping.md @@ -31,7 +31,7 @@ More useful links: - The usage data is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. In addition to counts, other facts that help us classify and understand GitLab installations are collected. - Usage ping is important to GitLab as we use it to calculate our Stage Monthly Active Users (SMAU) which helps us measure the success of our stages and features. -- Once usage ping is enabled, GitLab will gather data from the other instances and will be able to show usage statistics of your instance to your users. +- While usage ping is enabled, GitLab will gather data from the other instances and will be able to show usage statistics of your instance to your users. ### Why should we enable Usage Ping? @@ -126,10 +126,11 @@ happened over time, such as how many CI pipelines have run. They are monotonic a Observations are facts collected from one or more GitLab instances and can carry arbitrary data. There are no general guidelines around how to collect those, due to the individual nature of that data. -There are four types of counters which are all found in `usage_data.rb`: +There are several types of counters which are all found in `usage_data.rb`: - **Ordinary Batch Counters:** Simple count of a given ActiveRecord_Relation - **Distinct Batch Counters:** Distinct count of a given ActiveRecord_Relation on given column +- **Sum Batch Counters:** Sum the values of a given ActiveRecord_Relation on given column - **Alternative Counters:** Used for settings and configurations - **Redis Counters:** Used for in-memory counts. @@ -200,6 +201,47 @@ distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: :: distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') ``` +### Sum Batch Counters + +Handles `ActiveRecord::StatementInvalid` error + +Sum the values of a given ActiveRecord_Relation on given column and handles errors. + +Method: `sum(relation, column, batch_size: nil, start: nil, finish: nil)` + +Arguments: + +- `relation` the ActiveRecord_Relation to perform the operation +- `column` the column to sum on +- `batch_size`: if none set it will use default value 1000 from `Gitlab::Database::BatchCounter` +- `start`: custom start of the batch counting in order to avoid complex min calculations +- `end`: custom end of the batch counting in order to avoid complex min calculations + +Examples: + +```ruby +sum(JiraImportState.finished, :imported_issues_count) +``` + +### Grouping & Batch Operations + +The `count`, `distinct_count`, and `sum` batch counters can accept an `ActiveRecord::Relation` +object, which groups by a specified column. With a grouped relation, the methods do batch counting, +handle errors, and returns a hash table of key-value pairs. + +Examples: + +```ruby +count(Namespace.group(:type)) +# returns => {nil=>179, "Group"=>54} + +distinct_count(Project.group(:visibility_level), :creator_id) +# returns => {0=>1, 10=>1, 20=>11} + +sum(Issue.group(:state_id), :weight)) +# returns => {1=>3542, 2=>6820} +``` + ### Redis Counters Handles `::Redis::CommandError` and `Gitlab::UsageDataCounters::BaseCounter::UnknownEvent` @@ -360,7 +402,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF | `event` | string | yes | The event name it should be tracked | Response - +w Return 200 if tracking failed for any reason. - `200` if event was tracked or any errors @@ -368,6 +410,18 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF - `401 Unauthorized` if user is not authenticated - `403 Forbidden` for invalid CSRF token provided +1. Track events using JavaScript/Vue API helper which calls the API above + + Example usage for an existing event already defined in [known events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml): + + Note that `usage_data_api` and `usage_data_#{event_name}` should be enabled in order to be able to track events + + ```javascript + import api from '~/api'; + + api.trackRedisHllUserEvent('my_already_defined_event_name'), + ``` + 1. Track event using base module `Gitlab::UsageDataCounters::HLLRedisCounter.track_event(entity_id, event_name)`. Arguments: @@ -396,15 +450,6 @@ Recommendations: All events added in [`known_events.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml) are automatically added to usage data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209). For each event we add metrics for the weekly and monthly time frames, and totals for each where applicable: -- `#{event_name}_weekly` data for 7 days for daily [aggregation](#adding-new-events) events and data for last complete week for weekly [aggregation](#adding-new-events) events. -- `#{event_name}_monthly` data for 28 days for daily [aggregation](#adding-new-events) events and data for last 4 complete weeks for weekly [aggregation](#adding-new-events) events. -- `#{category}_total_unique_counts_weekly` total unique counts for events in same category for last 7 days or last complete week, if events are in the same Redis slot and if we have more than one metric. -- `#{event_name}_weekly` - Data for 7 days for daily [aggregation](#adding-new-events) events and data for the last complete week for weekly [aggregation](#adding-new-events) events. -- `#{event_name}_monthly` - Data for 28 days for daily [aggregation](#adding-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#adding-new-events) events. -- `#{category}_total_unique_counts_weekly` - Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric. -- `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#adding-new-events) events and data for last complete week for weekly [aggregation](#adding-new-events) events. -- `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#adding-new-events) events and data for last 4 complete weeks for weekly [aggregation](#adding-new-events) events. -- `#{category}_total_unique_counts_weekly` total unique counts for events in same category for last 7 days or last complete week, if events are in the same Redis slot and if we have more than one metric. - `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#adding-new-events) events and data for the last complete week for weekly [aggregation](#adding-new-events) events. - `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#adding-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#adding-new-events) events. - `#{category}_total_unique_counts_weekly`: Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric. diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 6ef9be381b4..53aa84cffcb 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -151,11 +151,14 @@ In order to reuse a single object for all calls to a named factory in implicit p can be used: ```ruby +RSpec.describe API::Search, factory_default: :keep do let_it_be(:namespace) { create_default(:namespace) } ``` Then every project we create will use this `namespace`, without us having to pass -it as `namespace: namespace`. +it as `namespace: namespace`. In order to make it work along with `let_it_be`, `factory_default: :keep` +must be explicitly specified. That will keep the default factory for every example in a suite instead of +recreating it for each example. Maybe we don't need to create 208 different projects - we can create one and reuse it. In addition, we can see that only about 1/3 of the @@ -497,10 +500,9 @@ let_it_be(:project, refind: true) { create(:project) } ### Time-sensitive tests -[Timecop](https://github.com/travisjeffery/timecop) is available in our -Ruby-based tests for verifying things that are time-sensitive. Any test that -exercises or verifies something time-sensitive should make use of Timecop to -prevent transient test failures. +[`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) +can be used to verify things that are time-sensitive. Any test that exercises or verifies something time-sensitive +should make use of these helpers to prevent transient test failures. Example: @@ -508,7 +510,7 @@ Example: it 'is overdue' do issue = build(:issue, due_date: Date.tomorrow) - Timecop.freeze(3.days.from_now) do + travel_to(3.days.from_now) do expect(issue).to be_overdue end end diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md index 6917639454c..8091142410c 100644 --- a/doc/development/testing_guide/ci.md +++ b/doc/development/testing_guide/ci.md @@ -28,9 +28,6 @@ After that, the next pipeline will use the up-to-date `knapsack/report-master.js The GitLab test suite is [monitored](../performance.md#rspec-profiling) for the `master` branch, and any branch that includes `rspec-profile` in their name. -A [public dashboard](https://redash.gitlab.com/public/dashboards/l1WhHXaxrCWM5Ai9D7YDqHKehq6OU3bx5gssaiWe?org_slug=default) is available for everyone to see. Feel free to look at the -slowest test files and try to improve them. - ## CI setup - Rails logging to `log/test.log` is disabled by default in CI [for diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 36cb49256a6..866a949d795 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -310,3 +310,56 @@ end # Using native mouse click events in the case of a mask/overlay click_element_coordinates(:title) ``` + +## Ensure `expect` statements wait efficiently + +In general, we use an `expect` statement to check that something _is_ as we expect it. For example: + +```ruby +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).to have_job("a_job") +end +``` + +### Ensure `expect` checks for negation efficiently + +However, sometimes we want to check that something is _not_ as we _don't_ want it to be. In other +words, we want to make sure something is absent. In such a case we should use an appropriate +predicate method that returns quickly, rather than waiting for a state that won't appear. + +It's most efficient to use a predicate method that returns immediately when there is no job, or waits +until it disappears: + +```ruby +# Good +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).to have_no_job("a_job") +end +``` + +### Problematic alternatives + +Alternatively, if we want to check that a job doesn't exist it might be tempting to use `not_to`: + +```ruby +# Bad +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).not_to have_job("a_job") +end +``` + +For this statement to pass, `have_job("a_job")` has to return `false` so that `not_to` can negate it. +The problem is that `have_job("a_job")` waits up to ten seconds for `"a job"` to appear before +returning `false`. Under the expected condition this test will take ten seconds longer than it needs to. + +Instead, we could force no wait: + +```ruby +# Not as bad but potentially flaky +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).not_to have_job("a_job", wait: 0) +end +``` + +The problem is that if `"a_job"` is present and we're waiting for it to disappear, this statement +will fail. diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 87a9738b313..e571774167d 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -1,29 +1,70 @@ # Testing with feature flags -To run a specific test with a feature flag enabled you can use the `QA::Runtime::Feature` class to enable and disable feature flags ([via the API](../../../api/features.md)). +To run a specific test with a feature flag enabled you can use the `QA::Runtime::Feature` class to +enable and disable feature flags ([via the API](../../../api/features.md)). -Note that administrator authorization is required to change feature flags. `QA::Runtime::Feature` will automatically authenticate as an administrator as long as you provide an appropriate access token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME` and `GITLAB_ADMIN_PASSWORD`. +Note that administrator authorization is required to change feature flags. `QA::Runtime::Feature` +will automatically authenticate as an administrator as long as you provide an appropriate access +token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME` +and `GITLAB_ADMIN_PASSWORD`. -Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments where admin access is not available. +Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments +where admin access is not available. + +CAUTION: **Caution:** +You are strongly advised to [enable feature flags only for a group, project, user](../../feature_flags/development.md#feature-actors), +or [feature group](../../feature_flags/development.md#feature-groups). This makes it possible to +test a feature in a shared environment without affecting other users. + +For example, the code below would enable a feature flag named `:feature_flag_name` for the project +created by the test: ```ruby RSpec.describe "with feature flag enabled", :requires_admin do + let(:project) { Resource::Project.fabricate_via_api! } + before do - Runtime::Feature.enable('feature_flag_name') + Runtime::Feature.enable(:feature_flag_name, project: project) end it "feature flag test" do - # Execute a test with a feature flag enabled + # Execute the test with the feature flag enabled. + # It will only affect the project created in this test. end after do - Runtime::Feature.disable('feature_flag_name') + Runtime::Feature.disable(:feature_flag_name, project: project) end end ``` +Note that the `enable` and `disable` methods first set the flag and then check that the updated +value is returned by the API. + +Similarly, you can enable a feature for a group, user, or feature group: + +```ruby +group = Resource::Group.fabricate_via_api! +Runtime::Feature.enable(:feature_flag_name, group: group) + +user = Resource::User.fabricate_via_api! +Runtime::Feature.enable(:feature_flag_name, user: user) + +feature_group = "a_feature_group" +Runtime::Feature.enable(:feature_flag_name, feature_group: feature_group) +``` + +If no scope is provided, the feature flag will be set instance-wide: + +```ruby +# This will affect all users! +Runtime::Feature.enable(:feature_flag_name) +``` + ## Running a scenario with a feature flag enabled -It's also possible to run an entire scenario with a feature flag enabled, without having to edit existing tests or write new ones. +It's also possible to run an entire scenario with a feature flag enabled, without having to edit +existing tests or write new ones. -Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) for details. +Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) +for details. diff --git a/doc/gitlab-basics/README.md b/doc/gitlab-basics/README.md index a7f18e13f74..3850655aaed 100644 --- a/doc/gitlab-basics/README.md +++ b/doc/gitlab-basics/README.md @@ -31,7 +31,7 @@ The following are guides to basic GitLab functionality: - [Add a file](add-file.md), to add new files to a project's repository. - [Create an issue](../user/project/issues/managing_issues.md#create-a-new-issue), to start collaborating within a project. -- [Create a merge request](add-merge-request.md), to request changes made in a branch +- [Create a merge request](../user/project/merge_requests/creating_merge_requests.md), to request changes made in a branch be merged into a project's repository. - See how these features come together in the [GitLab Flow introduction video](https://youtu.be/InKNIvky2KE) and [GitLab Flow page](../topics/gitlab_flow.md). diff --git a/doc/gitlab-basics/create-branch.md b/doc/gitlab-basics/create-branch.md index 0ad5cb53e97..13f20869b51 100644 --- a/doc/gitlab-basics/create-branch.md +++ b/doc/gitlab-basics/create-branch.md @@ -9,11 +9,11 @@ type: howto A branch is an independent line of development in a [project](../user/project/index.md). -When you create a new branch (in your [terminal](basic-git-commands.md) or with +When you create a new branch (in your [terminal](start-using-git.md) or with [the web interface](../user/project/repository/web_editor.md#create-a-new-branch)), you are creating a snapshot of a certain branch, usually the main `master` branch, at it's current state. From there, you can start to make your own changes without affecting the main codebase. The history of your changes will be tracked in your branch. When your changes are ready, you then merge them into the rest of the codebase with a -[merge request](add-merge-request.md). +[merge request](../user/project/merge_requests/creating_merge_requests.md). diff --git a/doc/gitlab-basics/create-project.md b/doc/gitlab-basics/create-project.md index f411ac769c0..c8db1664b89 100644 --- a/doc/gitlab-basics/create-project.md +++ b/doc/gitlab-basics/create-project.md @@ -20,7 +20,7 @@ To create a project in GitLab: - Create a [blank project](#blank-projects). - Create a project using with one of the available [project templates](#project-templates). - [Import a project](../user/project/import/index.md) from a different repository, - if enabled on your GitLab instance. Contact your GitLab admin if this is unavailable. + if enabled on your GitLab instance. Contact your GitLab administrator if this is unavailable. - Run [CI/CD pipelines for external repositories](../ci/ci_cd_for_external_repos/index.md). **(PREMIUM)** NOTE: **Note:** @@ -129,7 +129,7 @@ To use a custom project template on the **New project** page: > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/26388) in GitLab 10.5. When you create a new repository locally, instead of going to GitLab to manually -create a new project and then [clone the repo](start-using-git.md#clone-a-repository) +create a new project and then [clone the repository](start-using-git.md#clone-a-repository) locally, you can directly push it to GitLab to create the new project, all without leaving your terminal. If you have access rights to the associated namespace, GitLab will automatically create a new project under that GitLab namespace with its visibility diff --git a/doc/gitlab-geo/after_setup.md b/doc/gitlab-geo/after_setup.md deleted file mode 100644 index c8a7b9d1096..00000000000 --- a/doc/gitlab-geo/after_setup.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/replication/using_a_geo_server.md' ---- - -This document was moved to [another location](../administration/geo/replication/using_a_geo_server.md). diff --git a/doc/gitlab-geo/bring-primary-back.md b/doc/gitlab-geo/bring-primary-back.md deleted file mode 100644 index 8c43f4d805f..00000000000 --- a/doc/gitlab-geo/bring-primary-back.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/disaster_recovery/bring_primary_back.md' ---- - -This document was moved to [another location](../administration/geo/disaster_recovery/bring_primary_back.md). diff --git a/doc/gitlab-geo/configuration.md b/doc/gitlab-geo/configuration.md deleted file mode 100644 index b46a2caea4a..00000000000 --- a/doc/gitlab-geo/configuration.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/replication/configuration.md' ---- - -This document was moved to [another location](../administration/geo/replication/configuration.md). diff --git a/doc/gitlab-geo/configuration_source.md b/doc/gitlab-geo/configuration_source.md deleted file mode 100644 index b46a2caea4a..00000000000 --- a/doc/gitlab-geo/configuration_source.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/replication/configuration.md' ---- - -This document was moved to [another location](../administration/geo/replication/configuration.md). diff --git a/doc/gitlab-geo/database.md b/doc/gitlab-geo/database.md deleted file mode 100644 index 2f68068d95b..00000000000 --- a/doc/gitlab-geo/database.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/setup/database.md' ---- - -This document was moved to [another location](../administration/geo/setup/database.md). diff --git a/doc/gitlab-geo/database_source.md b/doc/gitlab-geo/database_source.md deleted file mode 100644 index 2f68068d95b..00000000000 --- a/doc/gitlab-geo/database_source.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/setup/database.md' ---- - -This document was moved to [another location](../administration/geo/setup/database.md). diff --git a/doc/gitlab-geo/disaster-recovery.md b/doc/gitlab-geo/disaster-recovery.md deleted file mode 100644 index d42e815a879..00000000000 --- a/doc/gitlab-geo/disaster-recovery.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/disaster_recovery/index.md' ---- - -This document was moved to [another location](../administration/geo/disaster_recovery/index.md). diff --git a/doc/gitlab-geo/docker_registry.md b/doc/gitlab-geo/docker_registry.md deleted file mode 100644 index 26a708f6845..00000000000 --- a/doc/gitlab-geo/docker_registry.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/replication/docker_registry.md' ---- - -This document was moved to [another location](../administration/geo/replication/docker_registry.md). diff --git a/doc/gitlab-geo/faq.md b/doc/gitlab-geo/faq.md deleted file mode 100644 index f1952bc7e4c..00000000000 --- a/doc/gitlab-geo/faq.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/replication/faq.md' ---- - -This document was moved to [another location](../administration/geo/replication/faq.md). diff --git a/doc/gitlab-geo/ha.md b/doc/gitlab-geo/ha.md deleted file mode 100644 index 0be70791d45..00000000000 --- a/doc/gitlab-geo/ha.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/replication/multiple_servers.md' ---- - -This document was moved to [another location](../administration/geo/replication/multiple_servers.md). diff --git a/doc/gitlab-geo/object_storage.md b/doc/gitlab-geo/object_storage.md deleted file mode 100644 index 1f29b7b7e8c..00000000000 --- a/doc/gitlab-geo/object_storage.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/replication/object_storage.md' ---- - -This document was moved to [another location](../administration/geo/replication/object_storage.md). diff --git a/doc/gitlab-geo/planned-failover.md b/doc/gitlab-geo/planned-failover.md deleted file mode 100644 index 720b6bc9424..00000000000 --- a/doc/gitlab-geo/planned-failover.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/disaster_recovery/planned_failover.md' ---- - -This document was moved to [another location](../administration/geo/disaster_recovery/planned_failover.md). diff --git a/doc/gitlab-geo/security-review.md b/doc/gitlab-geo/security-review.md deleted file mode 100644 index a0a5b0e536c..00000000000 --- a/doc/gitlab-geo/security-review.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/replication/security_review.md' ---- - -This document was moved to [another location](../administration/geo/replication/security_review.md). diff --git a/doc/gitlab-geo/ssh.md b/doc/gitlab-geo/ssh.md deleted file mode 100644 index 4f8db687850..00000000000 --- a/doc/gitlab-geo/ssh.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/operations/fast_ssh_key_lookup.md' ---- - -This document was moved to [another location](../administration/operations/fast_ssh_key_lookup.md). diff --git a/doc/gitlab-geo/troubleshooting.md b/doc/gitlab-geo/troubleshooting.md deleted file mode 100644 index 25fe1372c69..00000000000 --- a/doc/gitlab-geo/troubleshooting.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/replication/troubleshooting.md' ---- - -This document was moved to [another location](../administration/geo/replication/troubleshooting.md). diff --git a/doc/gitlab-geo/tuning.md b/doc/gitlab-geo/tuning.md deleted file mode 100644 index 84ac40f99db..00000000000 --- a/doc/gitlab-geo/tuning.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/replication/tuning.md' ---- - -This document was moved to [another location](../administration/geo/replication/tuning.md). diff --git a/doc/gitlab-geo/updating_the_geo_nodes.md b/doc/gitlab-geo/updating_the_geo_nodes.md deleted file mode 100644 index 28234ec02ed..00000000000 --- a/doc/gitlab-geo/updating_the_geo_nodes.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/replication/updating_the_geo_nodes.md' ---- - -This document was moved to [another location](../administration/geo/replication/updating_the_geo_nodes.md). diff --git a/doc/gitlab-geo/using_a_geo_server.md b/doc/gitlab-geo/using_a_geo_server.md deleted file mode 100644 index c8a7b9d1096..00000000000 --- a/doc/gitlab-geo/using_a_geo_server.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../administration/geo/replication/using_a_geo_server.md' ---- - -This document was moved to [another location](../administration/geo/replication/using_a_geo_server.md). diff --git a/doc/install/README.md b/doc/install/README.md index f1ef368685e..1f9ccc8115d 100644 --- a/doc/install/README.md +++ b/doc/install/README.md @@ -1,4 +1,7 @@ --- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers comments: false description: Read through the GitLab installation methods. type: index @@ -68,7 +71,7 @@ GitLab maintains a set of official Docker images based on the Omnibus GitLab pac If the Omnibus GitLab package is not available in your distribution, you can install GitLab from source: Useful for unsupported systems like \*BSD. For an -overview of the directory structure, read the [structure documentation](structure.md). +overview of the directory structure, read the [structure documentation](installation.md#gitlab-directory-structure). [**> Install GitLab from source.**](installation.md) diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index aba76ecf50e..bad01175363 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -1,4 +1,7 @@ --- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: howto --- diff --git a/doc/install/azure/index.md b/doc/install/azure/index.md index b6e3025a0e0..b5e72a9b84a 100644 --- a/doc/install/azure/index.md +++ b/doc/install/azure/index.md @@ -422,7 +422,7 @@ on any cloud service you choose. ## Where to next? -Check out our other [Technical Articles](../../articles/index.md) or browse the [GitLab Documentation](../../README.md) to learn more about GitLab. +Check out our other [Technical Articles](../../topics/index.md) or browse the [GitLab Documentation](../../README.md) to learn more about GitLab. ### Useful links diff --git a/doc/install/docker.md b/doc/install/docker.md index c2d7655d526..ca780caa563 100644 --- a/doc/install/docker.md +++ b/doc/install/docker.md @@ -1,4 +1,7 @@ --- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: index --- diff --git a/doc/install/google_cloud_platform/index.md b/doc/install/google_cloud_platform/index.md index 8ca5c5c266a..da2b30df476 100644 --- a/doc/install/google_cloud_platform/index.md +++ b/doc/install/google_cloud_platform/index.md @@ -1,4 +1,7 @@ --- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: 'Learn how to install a GitLab instance on Google Cloud Platform.' type: howto --- diff --git a/doc/install/installation.md b/doc/install/installation.md index e2c77073983..bfdf797cc77 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -1,4 +1,7 @@ --- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: howto --- diff --git a/doc/install/openshift_and_gitlab/index.md b/doc/install/openshift_and_gitlab/index.md index 022e2095c68..7ac5aa6667a 100644 --- a/doc/install/openshift_and_gitlab/index.md +++ b/doc/install/openshift_and_gitlab/index.md @@ -1,6 +1,8 @@ --- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: howto -date: 2016-06-28 --- # How to install GitLab on OpenShift Origin 3 @@ -372,7 +374,7 @@ running scaled to 2. Upping the GitLab pods is actually like adding new application servers to your cluster. You can see how that would work if you didn't use GitLab with -OpenShift by following the [HA documentation](../../administration/high_availability/gitlab.md) for the application servers. +OpenShift by following the [HA documentation](../../administration/reference_architectures/index.md) for the application servers. Bare in mind that you may need more resources (CPU, RAM, disk space) when you scale up. If a pod is in pending state for too long, you can navigate to diff --git a/doc/install/relative_url.md b/doc/install/relative_url.md index 09069a2d1fd..82cf134f968 100644 --- a/doc/install/relative_url.md +++ b/doc/install/relative_url.md @@ -1,4 +1,7 @@ --- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/install/requirements.md b/doc/install/requirements.md index da0128fecc3..bc320bcd335 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -1,4 +1,7 @@ --- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/install/structure.md b/doc/install/structure.md index 87ef11c60fe..ca90a3de1b8 100644 --- a/doc/install/structure.md +++ b/doc/install/structure.md @@ -2,4 +2,4 @@ redirect_to: installation.md#gitlab-directory-structure --- -This page was moved to [another location](#installation.md#gitlab-directory-structure). +This page was moved to [another location](installation.md#gitlab-directory-structure). diff --git a/doc/integration/README.md b/doc/integration/README.md index c5c21644d1c..c22c383410f 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -26,7 +26,7 @@ GitLab can be configured to authenticate access requests with the following auth - Enable sign in with [Bitbucket](bitbucket.md) accounts. - Configure GitLab to sign in using [CAS](cas.md). - Integrate with [Kerberos](kerberos.md). -- Enable sign in via [LDAP](ldap.md). +- Enable sign in via [LDAP](../administration/auth/ldap/index.md). - Enable [OAuth2 provider](oauth_provider.md) application creation. - Use [OmniAuth](omniauth.md) to enable sign in via Twitter, GitHub, GitLab.com, Google, Bitbucket, Facebook, Shibboleth, SAML, Crowd, Azure or Authentiq ID. diff --git a/doc/integration/cas.md b/doc/integration/cas.md index cf4e501e772..eee801350eb 100644 --- a/doc/integration/cas.md +++ b/doc/integration/cas.md @@ -45,10 +45,10 @@ To enable the CAS OmniAuth provider you must register your application with your - { name: 'cas3', label: 'cas', args: { - url: 'CAS_SERVER', - login_url: '/CAS_PATH/login', - service_validate_url: '/CAS_PATH/p3/serviceValidate', - logout_url: '/CAS_PATH/logout'} } + url: 'CAS_SERVER', + login_url: '/CAS_PATH/login', + service_validate_url: '/CAS_PATH/p3/serviceValidate', + logout_url: '/CAS_PATH/logout' } } ``` 1. Change 'CAS_PATH' to the root of your CAS instance (ie. `cas`). diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index f40955ad8ff..54375db1831 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -10,8 +10,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/109 "Elasticsearch Merge Request") in GitLab [Starter](https://about.gitlab.com/pricing/) 8.4. > - Support for [Amazon Elasticsearch](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1305) in GitLab [Starter](https://about.gitlab.com/pricing/) 9.0. -This document describes how to set up Elasticsearch with GitLab. After -Elasticsearch is enabled, you'll have the benefit of fast search response times +This document describes how to enable Advanced Search. After +Advanced Search is enabled, you'll have the benefit of fast search response times and the advantage of the following special searches: - [Advanced Search](../user/search/advanced_global_search.md) @@ -80,7 +80,7 @@ replicas can not be as there is no other node to which Elasticsearch can assign replica. After the data is added to the database or repository and [Elasticsearch is -enabled in the Admin Area](#enabling-elasticsearch) the search index will be +enabled in the Admin Area](#enabling-advanced-search) the search index will be updated automatically. ## Elasticsearch repository indexer @@ -155,15 +155,20 @@ Example: PREFIX=/usr sudo -E make install ``` -After installation, be sure to [enable Elasticsearch](#enabling-elasticsearch). +After installation, be sure to [enable Elasticsearch](#enabling-advanced-search). -## Enabling Elasticsearch +NOTE: **Note:** +If you see an error such as `Permission denied - /home/git/gitlab-elasticsearch-indexer/` while indexing, you +may need to set the `production -> elasticsearch -> indexer_path` setting in your `gitlab.yml` file to +`/usr/local/bin/gitlab-elasticsearch-indexer`, which is where the binary is installed. + +## Enabling Advanced Search NOTE: **Note:** For GitLab instances with more than 50GB repository data you can follow the instructions for [Indexing large instances](#indexing-large-instances) below. -To enable Elasticsearch, you need to have admin access to GitLab: +To enable Advanced Search, you need to have admin access to GitLab: 1. Navigate to **Admin Area** (wrench icon), then **Settings > General** and expand the **Advanced Search** section. @@ -172,7 +177,7 @@ To enable Elasticsearch, you need to have admin access to GitLab: To see the Advanced Search section, you need an active Starter [license](../user/admin_area/license.md). -1. Configure the [Elasticsearch settings](#elasticsearch-configuration) for +1. Configure the [Advanced Search settings](#advanced-search-configuration) for your Elasticsearch cluster. Do not enable **Elasticsearch indexing** or **Search with Elasticsearch enabled** yet. 1. Click **Save changes** for the changes to take effect. @@ -206,7 +211,7 @@ To enable Elasticsearch, you need to have admin access to GitLab: **Admin Area > Settings > General > Advanced Search** and click **Save changes**. -### Elasticsearch configuration +### Advanced Search configuration The following Elasticsearch settings are available: @@ -238,14 +243,14 @@ If you select `Limit namespaces and projects that can be indexed`, more options You can select namespaces and projects to index exclusively. Note that if the namespace is a group it will include any sub-groups and projects belonging to those sub-groups to be indexed as well. -Elasticsearch only provides cross-group code/commit search (global) if all name-spaces are indexed. In this particular scenario where only a subset of namespaces are indexed, a global search will not provide a code or commit scope. This will be possible only in the scope of an indexed namespace. Currently there is no way to code/commit search in multiple indexed namespaces (when only a subset of namespaces has been indexed). For example if two groups are indexed, there is no way to run a single code search on both. You can only run a code search on the first group and then on the second. +Advanced Search only provides cross-group code/commit search (global) if all name-spaces are indexed. In this particular scenario where only a subset of namespaces are indexed, a global search will not provide a code or commit scope. This will be possible only in the scope of an indexed namespace. Currently there is no way to code/commit search in multiple indexed namespaces (when only a subset of namespaces has been indexed). For example if two groups are indexed, there is no way to run a single code search on both. You can only run a code search on the first group and then on the second. You can filter the selection dropdown by writing part of the namespace or project name you're interested in.  NOTE: **Note:** -If no namespaces or projects are selected, no Elasticsearch indexing will take place. +If no namespaces or projects are selected, no Advanced Search indexing will take place. CAUTION: **Warning:** If you have already indexed your instance, you will have to regenerate the index in order to delete all existing data @@ -253,7 +258,7 @@ for filtering to work correctly. To do this run the Rake tasks `gitlab:elastic:r `gitlab:elastic:clear_index_status`. Afterwards, removing a namespace or a project from the list will delete the data from the Elasticsearch index as expected. -## Disabling Elasticsearch +## Disabling Advanced Search To disable the Elasticsearch integration: @@ -283,7 +288,7 @@ alias such as we can change the underlying index at will. NOTE: **Note:** Any index attached to the production alias is deemed a `primary` and will be -used by the GitLab Elasticsearch integration. +used by the GitLab Advanced Search integration. ### Pause the indexing @@ -421,12 +426,12 @@ After the reindexing is completed, the original index will be scheduled to be de While the reindexing is running, you will be able to follow its progress under that same section. -## GitLab Elasticsearch Rake tasks +## GitLab Advanced Search Rake tasks Rake tasks are available to: - [Build and install](#building-and-installing) the indexer. -- Delete indexes when [disabling Elasticsearch](#disabling-elasticsearch). +- Delete indexes when [disabling Elasticsearch](#disabling-advanced-search). - Add GitLab data to an index. The following are some available Rake tasks: @@ -467,7 +472,7 @@ Indexing project repositories...I, [2019-03-04T21:27:03.083410 #3384] INFO -- : I, [2019-03-04T21:27:05.215266 #3384] INFO -- : Indexing GitLab User / test (ID=33) is done! ``` -## Elasticsearch index scopes +## Advanced Search index scopes When performing a search, the GitLab index will use the following scopes: @@ -499,7 +504,7 @@ For basic guidance on choosing a cluster configuration you may refer to [Elastic - `refresh_interval` is a per index setting. You may want to adjust that from default `1s` to a bigger value if you don't need data in realtime. This will change how soon you will see fresh results. If that's important for you, you should leave it as close as possible to the default value. - You might want to raise [`indices.memory.index_buffer_size`](https://www.elastic.co/guide/en/elasticsearch/reference/current/indexing-buffer.html) to 30% or 40% if you have a lot of heavy indexing operations. -### Elasticsearch integration settings guidance +### Advanced Search integration settings guidance - The `Number of Elasticsearch shards` setting usually corresponds with the number of CPUs available in your cluster. For example, if you have a 3-node cluster with 4 cores each, this means you will benefit from having at least 3*4=12 shards in the cluster. Please note, it's only possible to change the shards number by using [Split index API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-split-index.html) or by reindexing to a different index with a changed number of shards. - The `Number of Elasticsearch replicas` setting should most of the time be equal to `1` (each shard will have 1 replica). Using `0` is not recommended, because losing one node will corrupt the index. @@ -507,7 +512,7 @@ For basic guidance on choosing a cluster configuration you may refer to [Elastic ### Indexing large instances This section may be helpful in the event that the other -[basic instructions](#enabling-elasticsearch) cause problems +[basic instructions](#enabling-advanced-search) cause problems due to large volumes of data being indexed. CAUTION: **Warning:** @@ -516,7 +521,7 @@ Make sure to prepare for this task by having a [Scalable and Highly Available Setup](../administration/reference_architectures/index.md) or creating [extra Sidekiq processes](../administration/operations/extra_sidekiq_processes.md). -1. [Configure your Elasticsearch host and port](#enabling-elasticsearch). +1. [Configure your Elasticsearch host and port](#enabling-advanced-search). 1. Create empty indexes: ```shell @@ -537,7 +542,7 @@ Sidekiq processes](../administration/operations/extra_sidekiq_processes.md). bundle exec rake gitlab:elastic:clear_index_status RAILS_ENV=production ``` -1. [Enable **Elasticsearch indexing**](#enabling-elasticsearch). +1. [Enable **Elasticsearch indexing**](#enabling-advanced-search). 1. Indexing large Git repositories can take a while. To speed up the process, you can [tune for indexing speed](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#tune-for-indexing-speed): - You can temporarily disable [`refresh`](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html), the operation responsible for making changes to an index available to search. @@ -664,7 +669,7 @@ Sidekiq processes](../administration/operations/extra_sidekiq_processes.md). } }' ``` -1. After the indexing has completed, enable [**Search with Elasticsearch enabled**](#enabling-elasticsearch). +1. After the indexing has completed, enable [**Search with Elasticsearch enabled**](#enabling-advanced-search). ### Deleted documents @@ -698,96 +703,114 @@ However, some larger installations may wish to tune the merge policy settings: ## Troubleshooting -### Common issues +Here are some common pitfalls and how to overcome them. + +### How can I verify that my GitLab instance is using Elasticsearch? + +There are a couple of ways to achieve that: -Here are some common pitfalls and how to overcome them: +- Whenever you perform a search there will be a link on the search results page + in the top right hand corner saying "Advanced search functionality is enabled". + This is always correctly identifying whether the current project/namespace + being searched is using Elasticsearch. -- **How can I verify my GitLab instance is using Elasticsearch?** +- From the admin area under **Settings > General > Elasticsearch** check that the + Advanced Search settings are checked. - The easiest method is via the rails console (`sudo gitlab-rails console`) by running the following: + Those same settings there can be obtained from the Rails console if necessary: ```ruby - u = User.find_by_username('your-username') - s = SearchService.new(u, {:search => 'search_term'}) - pp s.search_objects.class.name + ::Gitlab::CurrentSettings.elasticsearch_search? # Whether or not searches will use Elasticsearch + ::Gitlab::CurrentSettings.elasticsearch_indexing? # Whether or not content will be indexed in Elasticsearch + ::Gitlab::CurrentSettings.elasticsearch_limit_indexing? # Whether or not Elasticsearch is limited only to certain projects/namespaces ``` - If you see `"ActiveRecord::Relation"`, you are **not** using Elasticsearch. - - If you see `"Kaminari::PaginatableArray"` you are using Elasticsearch. +- If Elasticsearch is limited to specific namespaces and you need to know if + Elasticsearch is being used for a specific project or namespace, you can use + the Rails console: - NOTE: **Note:** - The above instructions are used to verify that GitLab is using Elasticsearch only when indexing all namespaces. This is not to be used for scenarios that only index a [subset of namespaces](#limiting-namespaces-and-projects). + ```ruby + ::Gitlab::CurrentSettings.search_using_elasticsearch?(scope: Namespace.find_by_full_path("/my-namespace")) + ::Gitlab::CurrentSettings.search_using_elasticsearch?(scope: Project.find_by_full_path("/my-namespace/my-project")) + ``` -- **I updated GitLab and now I can't find anything** +### I updated GitLab and now I can't find anything - We continuously make updates to our indexing strategies and aim to support - newer versions of Elasticsearch. When indexing changes are made, it may - be necessary for you to [reindex](#zero-downtime-reindexing) after updating GitLab. +We continuously make updates to our indexing strategies and aim to support +newer versions of Elasticsearch. When indexing changes are made, it may +be necessary for you to [reindex](#zero-downtime-reindexing) after updating GitLab. -- **I indexed all the repositories but I can't find anything** +### I indexed all the repositories but I can't get any hits for my search term in the UI - Make sure you indexed all the database data [as stated above](#enabling-elasticsearch). +Make sure you indexed all the database data [as stated above](#enabling-advanced-search). - Beyond that, check via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to see if the data shows up on the Elasticsearch side. +If there aren't any results (hits) in the UI search, check if you are seeing the same results via the rails console (`sudo gitlab-rails console`): - If it shows up via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), check that it shows up via the rails console (`sudo gitlab-rails console`): +```ruby +u = User.find_by_username('your-username') +s = SearchService.new(u, {:search => 'search_term', :scope => 'blobs'}) +pp s.search_objects.to_a +``` - ```ruby - u = User.find_by_username('your-username') - s = SearchService.new(u, {:search => 'search_term', :scope => 'blobs'}) - pp s.search_objects.to_a - ``` +Beyond that, check via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to see if the data shows up on the Elasticsearch side: - NOTE: **Note:** - The above instructions are not to be used for scenarios that only index a [subset of namespaces](#limiting-namespaces-and-projects). +```shell +curl --request GET <elasticsearch_server_ip>:9200/gitlab-production/_search?q=<search_term> +``` - See [Elasticsearch Index Scopes](#elasticsearch-index-scopes) for more information on searching for specific types of data. +More [complex Elasticsearch API calls](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-filter-context.html) are also possible. -- **I indexed all the repositories but then switched Elasticsearch servers and now I can't find anything** +It is important to understand at which level the problem is manifesting (UI, Rails code, Elasticsearch side) to be able to [troubleshoot further](../administration/troubleshooting/elasticsearch.md#search-results-workflow). - You will need to re-run all the Rake tasks to reindex the database, repositories, and wikis. +NOTE: **Note:** +The above instructions are not to be used for scenarios that only index a [subset of namespaces](#limiting-namespaces-and-projects). -- **The indexing process is taking a very long time** +See [Elasticsearch Index Scopes](#advanced-search-index-scopes) for more information on searching for specific types of data. - The more data present in your GitLab instance, the longer the indexing process takes. +### I indexed all the repositories but then switched Elasticsearch servers and now I can't find anything -- **There are some projects that weren't indexed, but we don't know which ones** +You will need to re-run all the Rake tasks to reindex the database, repositories, and wikis. - You can run `sudo gitlab-rake gitlab:elastic:projects_not_indexed` to display projects that aren't indexed. +### The indexing process is taking a very long time -- **No new data is added to the Elasticsearch index when I push code** +The more data present in your GitLab instance, the longer the indexing process takes. - NOTE: **Note:** - This was [fixed in GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35936) and the Rake task is not available for versions greater than that. +### There are some projects that weren't indexed, but I don't know which ones - When performing the initial indexing of blobs, we lock all projects until the project finishes indexing. It could happen that an error during the process causes one or multiple projects to remain locked. In order to unlock them, run: +You can run `sudo gitlab-rake gitlab:elastic:projects_not_indexed` to display projects that aren't indexed. - ```shell - sudo gitlab-rake gitlab:elastic:clear_locked_projects - ``` +### No new data is added to the Elasticsearch index when I push code -- **"Can't specify parent if no parent field has been configured"** +NOTE: **Note:** +This was [fixed in GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35936) and the Rake task is not available for versions greater than that. - If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indexes you will get - exception in lots of different cases: +When performing the initial indexing of blobs, we lock all projects until the project finishes indexing. It could happen that an error during the process causes one or multiple projects to remain locked. In order to unlock them, run: - ```plaintext - Elasticsearch::Transport::Transport::Errors::BadRequest([400] { - "error": { - "root_cause": [{ - "type": "illegal_argument_exception", - "reason": "Can't specify parent if no parent field has been configured" - }], - "type": "illegal_argument_exception", - "reason": "Can't specify parent if no parent field has been configured" - }, - "status": 400 - }): - ``` +```shell +sudo gitlab-rake gitlab:elastic:clear_locked_projects +``` + +### `Can't specify parent if no parent field has been configured` error + +If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indexes you will get +exception in lots of different cases: + +```plaintext +Elasticsearch::Transport::Transport::Errors::BadRequest([400] { + "error": { + "root_cause": [{ + "type": "illegal_argument_exception", + "reason": "Can't specify parent if no parent field has been configured" + }], + "type": "illegal_argument_exception", + "reason": "Can't specify parent if no parent field has been configured" + }, + "status": 400 +}): +``` - This is because we changed the index mapping in GitLab 8.12 and the old indexes should be removed and built from scratch again, - see details in the [update guide](../update/upgrading_from_source.md). +This is because we changed the index mapping in GitLab 8.12 and the old indexes should be removed and built from scratch again, +see details in the [update guide](../update/upgrading_from_source.md). - Exception `Elasticsearch::Transport::Transport::Errors::BadRequest` @@ -809,50 +832,51 @@ Here are some common pitfalls and how to overcome them: for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of the underlying instance. -- **My single node Elasticsearch cluster status never goes from `yellow` to `green` even though everything seems to be running properly** +### My single node Elasticsearch cluster status never goes from `yellow` to `green` even though everything seems to be running properly - **For a single node Elasticsearch cluster the functional cluster health status will be yellow** (will never be green) because the primary shard is allocated but replicas can not be as there is no other node to which Elasticsearch can assign a replica. This also applies if you are using the -[Amazon Elasticsearch](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html#aes-handling-errors-yellow-cluster-status) service. +**For a single node Elasticsearch cluster the functional cluster health status will be yellow** (never green) because the primary shard is allocated but replicas cannot be as there is no other node to which Elasticsearch can assign a replica. This also applies if you are using the [Amazon Elasticsearch](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html#aes-handling-errors-yellow-cluster-status) service. - CAUTION: **Warning:** - Setting the number of replicas to `0` is not something that we recommend (this is not allowed in the GitLab Elasticsearch Integration menu). If you are planning to add more Elasticsearch nodes (for a total of more than 1 Elasticsearch) the number of replicas will need to be set to an integer value larger than `0`. Failure to do so will result in lack of redundancy (losing one node will corrupt the index). +CAUTION: **Warning:** +Setting the number of replicas to `0` is discouraged (this is not allowed in the GitLab Elasticsearch Integration menu). If you are planning to add more Elasticsearch nodes (for a total of more than 1 Elasticsearch) the number of replicas will need to be set to an integer value larger than `0`. Failure to do so will result in lack of redundancy (losing one node will corrupt the index). - If you have a **hard requirement to have a green status for your single node Elasticsearch cluster**, please make sure you understand the risks outlined in the previous paragraph and then simply run the following query to set the number of replicas to `0`(the cluster will no longer try to create any shard replicas): +If you have a **hard requirement to have a green status for your single node Elasticsearch cluster**, please make sure you understand the risks outlined in the previous paragraph and then run the following query to set the number of replicas to `0`(the cluster will no longer try to create any shard replicas): - ```shell - curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{ - "index" : { - "number_of_replicas" : 0 - } - }' - ``` +```shell +curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{ +"index" : { + "number_of_replicas" : 0 + } +}' +``` -- **I'm getting a `health check timeout: no Elasticsearch node available` error in Sidekiq during the indexing process** +### `health check timeout: no Elasticsearch node available` error in Sidekiq - ```plaintext - Gitlab::Elastic::Indexer::Error: time="2020-01-23T09:13:00Z" level=fatal msg="health check timeout: no Elasticsearch node available" - ``` +If you're getting a `health check timeout: no Elasticsearch node available` error in Sidekiq during the indexing process: + +```plaintext +Gitlab::Elastic::Indexer::Error: time="2020-01-23T09:13:00Z" level=fatal msg="health check timeout: no Elasticsearch node available" +``` - You probably have not used either `http://` or `https://` as part of your value in the **"URL"** field of the Elasticsearch Integration Menu. Please make sure you are using either `http://` or `https://` in this field as the [Elasticsearch client for Go](https://github.com/olivere/elastic) that we are using [needs the prefix for the URL to be accepted as valid](https://github.com/olivere/elastic/commit/a80af35aa41856dc2c986204e2b64eab81ccac3a). - Once you have corrected the formatting of the URL, delete the index (via the [dedicated Rake task](#gitlab-elasticsearch-rake-tasks)) and [reindex the content of your instance](#enabling-elasticsearch). +You probably have not used either `http://` or `https://` as part of your value in the **"URL"** field of the Elasticsearch Integration Menu. Please make sure you are using either `http://` or `https://` in this field as the [Elasticsearch client for Go](https://github.com/olivere/elastic) that we are using [needs the prefix for the URL to be accepted as valid](https://github.com/olivere/elastic/commit/a80af35aa41856dc2c986204e2b64eab81ccac3a). +Once you have corrected the formatting of the URL, delete the index (via the [dedicated Rake task](#gitlab-advanced-search-rake-tasks)) and [reindex the content of your instance](#enabling-advanced-search). ### Low-level troubleshooting There is a [more structured, lower-level troubleshooting document](../administration/troubleshooting/elasticsearch.md) for when you experience other issues, including poor performance. -### Known Issues +### Known issues -- **[Elasticsearch `code_analyzer` doesn't account for all code cases](https://gitlab.com/groups/gitlab-org/-/epics/3621)** +[Elasticsearch `code_analyzer` doesn't account for all code cases](https://gitlab.com/groups/gitlab-org/-/epics/3621). - The `code_analyzer` pattern and filter configuration is being evaluated for improvement. We have fixed [most edge cases](https://gitlab.com/groups/gitlab-org/-/epics/3621#note_363429094) that were not returning expected search results due to our pattern and filter configuration. +The `code_analyzer` pattern and filter configuration is being evaluated for improvement. We have fixed [most edge cases](https://gitlab.com/groups/gitlab-org/-/epics/3621#note_363429094) that were not returning expected search results due to our pattern and filter configuration. - Improvements to the `code_analyzer` pattern and filters is being discussed in [epic 3621](https://gitlab.com/groups/gitlab-org/-/epics/3621). +Improvements to the `code_analyzer` pattern and filters are being discussed in [epic 3621](https://gitlab.com/groups/gitlab-org/-/epics/3621). ### Reverting to Basic Search Sometimes there may be issues with your Elasticsearch index data and as such GitLab will allow you to revert to "basic search" when there are no search results and assuming that basic search is supported in that scope. This "basic -search" will behave as though you don't have Elasticsearch enabled at all for +search" will behave as though you don't have Advanced Search enabled at all for your instance and search using other data sources (ie. PostgreSQL data and Git data). diff --git a/doc/integration/facebook.md b/doc/integration/facebook.md index fc65191994d..dbefb560fe7 100644 --- a/doc/integration/facebook.md +++ b/doc/integration/facebook.md @@ -83,7 +83,7 @@ To enable the Facebook OmniAuth provider you must register your application with ```yaml - { name: 'facebook', app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET' } + app_secret: 'YOUR_APP_SECRET' } ``` 1. Change 'YOUR_APP_ID' to the API key from Facebook page in step 10. diff --git a/doc/integration/github.md b/doc/integration/github.md index ccce4672cbf..4444db46853 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -71,17 +71,18 @@ Follow these steps to incorporate the GitHub OAuth 2 app in your GitLab server: ```yaml - { name: 'github', app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - args: { scope: 'user:email' } } + app_secret: 'YOUR_APP_SECRET', + args: { scope: 'user:email' } } ``` For GitHub Enterprise: ```yaml - - { name: 'github', app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - url: "https://github.example.com/", - args: { scope: 'user:email' } } + - { name: 'github', + app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + url: "https://github.example.com/", + args: { scope: 'user:email' } } ``` **Replace `https://github.example.com/` with your GitHub URL.** @@ -125,11 +126,12 @@ omnibus_gitconfig['system'] = { "http" => ["sslVerify = false"] } For installation from source: ```yaml -- { name: 'github', app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - url: "https://github.example.com/", - verify_ssl: false, - args: { scope: 'user:email' } } +- { name: 'github', + app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + url: "https://github.example.com/", + verify_ssl: false, + args: { scope: 'user:email' } } ``` You will also need to disable Git SSL verification on the server hosting GitLab. diff --git a/doc/integration/gitlab.md b/doc/integration/gitlab.md index f423863ce8e..a200f6b6470 100644 --- a/doc/integration/gitlab.md +++ b/doc/integration/gitlab.md @@ -63,9 +63,10 @@ GitLab.com will generate an application ID and secret key for you to use. For installations from source: ```yaml - - { name: 'gitlab', app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - args: { scope: 'api' } } + - { name: 'gitlab', + app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { scope: 'api' } } ``` 1. Change 'YOUR_APP_ID' to the Application ID from the GitLab.com application page. diff --git a/doc/integration/gitpod.md b/doc/integration/gitpod.md index f26483e3b5e..bd038eaf47b 100644 --- a/doc/integration/gitpod.md +++ b/doc/integration/gitpod.md @@ -72,3 +72,4 @@ To disable it: ```ruby Feature.disable(:gitpod) +``` diff --git a/doc/integration/google.md b/doc/integration/google.md index 0f848bbc7aa..4cf589c1da8 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -84,9 +84,10 @@ On your GitLab server: For installations from source: ```yaml - - { name: 'google_oauth2', app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - args: { access_type: 'offline', approval_prompt: '' } } + - { name: 'google_oauth2', + app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { access_type: 'offline', approval_prompt: '' } } ``` 1. Change `YOUR_APP_ID` to the client ID from the Google Developer page diff --git a/doc/integration/jira_development_panel.md b/doc/integration/jira_development_panel.md index 9b7aa5829c1..e91b024546e 100644 --- a/doc/integration/jira_development_panel.md +++ b/doc/integration/jira_development_panel.md @@ -148,7 +148,7 @@ GitLab. No other error messages appear in any logs. If there was an issue with SSL/TLS, this error message will be generated. -- The [GitLab Jira integration](jira.md) requires GitLab to connect to Jira. Any +- The [GitLab Jira integration](../user/project/integrations/jira.md) requires GitLab to connect to Jira. Any TLS issues that arise from a private certificate authority or self-signed certificate [are resolved on the GitLab server](https://docs.gitlab.com/omnibus/settings/ssl.html#other-certificate-authorities), as GitLab is the TLS client. diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md index 1b14b5a986f..11ea74f3430 100644 --- a/doc/integration/kerberos.md +++ b/doc/integration/kerberos.md @@ -207,9 +207,10 @@ remove the OmniAuth provider named `kerberos` from your `gitlab.yml` / ```yaml omniauth: + # Rest of configuration omitted # ... providers: - - { name: 'kerberos' } # <-- remove this line + - { name: 'kerberos' } # <-- remove this line ``` 1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect. diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index dd183ad9eb0..cf09c2f2803 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -50,7 +50,7 @@ earlier version, you'll need to explicitly enable it. - `allow_single_sign_on` allows you to specify the providers you want to allow to automatically create an account. It defaults to `false`. If `false` users must be created manually or they will not be able to sign in via OmniAuth. -- `auto_link_ldap_user` can be used if you have [LDAP / ActiveDirectory](ldap.md) +- `auto_link_ldap_user` can be used if you have [LDAP / ActiveDirectory](../administration/auth/ldap/index.md) integration enabled. It defaults to `false`. When enabled, users automatically created through an OmniAuth provider will have their LDAP identity created in GitLab as well. - `block_auto_created_users` defaults to `true`. If `true` auto created users will @@ -104,21 +104,21 @@ To change these settings: ```yaml ## OmniAuth settings - omniauth: - # Allow login via Twitter, Google, etc. using OmniAuth providers - # Versions prior to 11.4 require this to be set to true - # enabled: true + omniauth: + # Allow login via Twitter, Google, etc. using OmniAuth providers + # Versions prior to 11.4 require this to be set to true + # enabled: true - # CAUTION! - # This allows users to login without having a user account first. Define the allowed providers - # using an array, e.g. ["saml", "twitter"], or as true/false to allow all providers or none. - # User accounts will be created automatically when authentication was successful. - allow_single_sign_on: ["saml", "twitter"] + # CAUTION! + # This allows users to login without having a user account first. Define the allowed providers + # using an array, e.g. ["saml", "twitter"], or as true/false to allow all providers or none. + # User accounts will be created automatically when authentication was successful. + allow_single_sign_on: ["saml", "twitter"] - auto_link_ldap_user: true + auto_link_ldap_user: true - # Locks down those users until they have been cleared by the admin (default: true). - block_auto_created_users: true + # Locks down those users until they have been cleared by the admin (default: true). + block_auto_created_users: true ``` Now we can choose one or more of the [Supported Providers](#supported-providers) @@ -142,7 +142,7 @@ The chosen OmniAuth provider is now active and can be used to sign in to GitLab ## Automatically Link Existing Users to OmniAuth Users -> [Introduced in GitLab 13.4.](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36664) +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36664) in GitLab 13.4. You can automatically link OmniAuth users with existing GitLab users if their email addresses match. For example, the following setting is used to enable the auto link feature for both a SAML provider and the Twitter OAuth provider: diff --git a/doc/integration/salesforce.md b/doc/integration/salesforce.md index 7e0b2518e76..dbd0a03e3cf 100644 --- a/doc/integration/salesforce.md +++ b/doc/integration/salesforce.md @@ -64,7 +64,7 @@ To get the credentials (a pair of Client ID and Client Secret), you must [create - { name: 'salesforce', app_id: 'SALESFORCE_CLIENT_ID', app_secret: 'SALESFORCE_CLIENT_SECRET' - } + } ``` 1. Change `SALESFORCE_CLIENT_ID` to the Consumer Key from the Salesforce connected application page. diff --git a/doc/integration/saml.md b/doc/integration/saml.md index e7e94b21683..ee08a0026cd 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -113,16 +113,16 @@ in your SAML IdP: omniauth: providers: - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://login.example.com/idp', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' - }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" - } + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' + }, + label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } ``` 1. Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint @@ -210,7 +210,7 @@ Example: idp_sso_target_url: 'https://login.example.com/idp', issuer: 'https://gitlab.example.com', name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' - } } + } } ``` ### External Groups **(STARTER ONLY)** @@ -228,7 +228,7 @@ SAML login supports automatic identification on whether a user should be conside idp_sso_target_url: 'https://login.example.com/idp', issuer: 'https://gitlab.example.com', name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' - } } + } } ``` ### Admin Groups **(STARTER ONLY)** @@ -248,7 +248,7 @@ considered admin users. idp_sso_target_url: 'https://login.example.com/idp', issuer: 'https://gitlab.example.com', name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' - } } + } } ``` ### Auditor Groups **(STARTER ONLY)** @@ -270,7 +270,7 @@ considered auditor users. idp_sso_target_url: 'https://login.example.com/idp', issuer: 'https://gitlab.example.com', name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' - } } + } } ``` ## Bypass two factor authentication @@ -328,22 +328,22 @@ In addition to the changes in GitLab, make sure that your IdP is returning the omniauth: providers: - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://login.example.com/idp', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', - upstream_two_factor_authn_contexts: - [ - 'urn:oasis:names:tc:SAML:2.0:ac:classes:CertificateProtectedTransport', - 'urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorOTPSMS', - 'urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorIGTOKEN' - ] - }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" - } + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', + upstream_two_factor_authn_contexts: + [ + 'urn:oasis:names:tc:SAML:2.0:ac:classes:CertificateProtectedTransport', + 'urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorOTPSMS', + 'urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorIGTOKEN' + ] + }, + label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } ``` 1. Save the file and [restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect @@ -436,7 +436,7 @@ args: { issuer: 'https://gitlab.example.com', name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', attribute_statements: { email: ['EmailAddress'] }, - allowed_clock_drift: 1 # for one second clock drift + allowed_clock_drift: 1 # for one second clock drift } ``` @@ -561,10 +561,10 @@ args: { <redacted> -----END PRIVATE KEY-----', security: { - authn_requests_signed: true, # enable signature on AuthNRequest - want_assertions_signed: true, # enable the requirement of signed assertion - embed_sign: true, # embedded signature or HTTP GET parameter signature - metadata_signed: false, # enable signature on Metadata + authn_requests_signed: true, # enable signature on AuthNRequest + want_assertions_signed: true, # enable the requirement of signed assertion + embed_sign: true, # embedded signature or HTTP GET parameter signature + metadata_signed: false, # enable signature on Metadata signature_method: 'http://www.w3.org/2001/04/xmldsig-more#rsa-sha256', digest_method: 'http://www.w3.org/2001/04/xmlenc#sha256', } @@ -588,6 +588,52 @@ Refer to the documentation for your SAML Identity Provider for information on ho The [Generated passwords for users created through integrated authentication](../security/passwords_for_integrated_authentication_methods.md) guide provides an overview of how GitLab generates and sets passwords for users created via SAML. +## Configuring Group SAML on a self-managed GitLab instance **(PREMIUM ONLY)** + +For information on the GitLab.com implementation, please see the [SAML SSO for GitLab.com groups page](../user/group/saml_sso). + +Group SAML SSO helps if you need to allow access via multiple SAML identity providers, but as a multi-tenant solution is less suited to cases where you administer your own GitLab instance. + +To proceed with configuring Group SAML SSO instead, you'll need to enable the `group_saml` OmniAuth provider. This can be done from: + +- `gitlab.rb` for [Omnibus GitLab installations](#omnibus-installations). +- `gitlab/config/gitlab.yml` for [source installations](#source-installations). + +### Limitations + +Group SAML on a self-managed instance is limited when compared to the recommended +[instance-wide SAML](../user/group/saml_sso/index.md). The recommended solution allows you to take advantage of: + +- [LDAP compatibility](../administration/auth/ldap/index.md). +- [LDAP Group Sync](../user/group/index.md#manage-group-memberships-via-ldap). +- [Required groups](#required-groups). +- [Admin groups](#admin-groups). +- [Auditor groups](#auditor-groups). + +### Omnibus installations + +1. Make sure GitLab is + [configured with HTTPS](../install/installation.md#using-https). +1. Enable OmniAuth and the `group_saml` provider in `gitlab.rb`: + + ```ruby + gitlab_rails['omniauth_enabled'] = true + gitlab_rails['omniauth_providers'] = [{ name: 'group_saml' }] + ``` + +### Source installations + +1. Make sure GitLab is + [configured with HTTPS](../install/installation.md#using-https). +1. Enable OmniAuth and the `group_saml` provider in `gitlab/config/gitlab.yml`: + + ```yaml + omniauth: + enabled: true + providers: + - { name: 'group_saml' } + ``` + ## Troubleshooting You can find the base64-encoded SAML Response in the [`production_json.log`](../administration/logs.md#production_jsonlog). diff --git a/doc/integration/twitter.md b/doc/integration/twitter.md index 95de56f24d8..e501eac0c5f 100644 --- a/doc/integration/twitter.md +++ b/doc/integration/twitter.md @@ -65,7 +65,8 @@ To enable the Twitter OmniAuth provider you must register your application with For installations from source: ```yaml - - { name: 'twitter', app_id: 'YOUR_APP_ID', + - { name: 'twitter', + app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET' } ``` diff --git a/doc/intro/README.md b/doc/intro/README.md index 4bf9c766fc4..02429f387ee 100644 --- a/doc/intro/README.md +++ b/doc/intro/README.md @@ -28,7 +28,7 @@ Create issues, labels, milestones, cast your vote, and review issues. Create merge requests and review code. - [Fork a project and contribute to it](../user/project/repository/forking_workflow.md) -- [Create a new merge request](../gitlab-basics/add-merge-request.md) +- [Create a new merge request](../user/project/merge_requests/creating_merge_requests.md) - [Automatically close issues from merge requests](../user/project/issues/managing_issues.md#closing-issues-automatically) - [Automatically merge when pipeline succeeds](../user/project/merge_requests/merge_when_pipeline_succeeds.md) - [Revert any commit](../user/project/merge_requests/revert_changes.md) diff --git a/doc/operations/feature_flags.md b/doc/operations/feature_flags.md index fe7be48270a..8db1c0945c6 100644 --- a/doc/operations/feature_flags.md +++ b/doc/operations/feature_flags.md @@ -105,7 +105,8 @@ ID for the feature to be enabled. See the [Ruby example](#ruby-application-examp ### User IDs -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8240) in GitLab 12.2. [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/34363) to be defined per environment in GitLab 12.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8240) in GitLab 12.2. +> - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/34363) to be defined per environment in GitLab 12.6. Enables the feature for a list of target users. It is implemented using the Unleash [`userWithId`](https://unleash.github.io/docs/activation_strategy#userwithid) @@ -353,31 +354,9 @@ end ## Feature Flag Related Issues **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36617) in GitLab 13.2. -> - It's deployed behind a feature flag, enabled by default. -> - It's enabled on GitLab.com. -> - It can't be enabled or disabled per-project -> - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to disable it. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/251234) in GitLab 13.5. -You can link related issues to a feature flag. In the **Linked issues** section, click the `+` button and input the issue reference number or the full URL of the issue. +You can link related issues to a feature flag. In the **Linked issues** section, +click the `+` button and input the issue reference number or the full URL of the issue. This feature is similar to the [related issues](../user/project/issues/related_issues.md) feature. - -### Enable or disable Feature Flag Related Issues **(CORE ONLY)** - -Feature Flag Related Issues is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md) -can opt to disable it for your instance. - -To disable it: - -```ruby -Feature.disable(:feature_flags_issue_links) -``` - -To enable it: - -```ruby -Feature.enable(:feature_flags_issue_links) -``` diff --git a/doc/operations/incident_management/alert_details.md b/doc/operations/incident_management/alert_details.md index 860e6d32ae4..459331ea0a5 100644 --- a/doc/operations/incident_management/alert_details.md +++ b/doc/operations/incident_management/alert_details.md @@ -1,200 +1,5 @@ --- -stage: Monitor -group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +redirect_to: alerts.md --- -# Alert details page - -Navigate to the Alert details view by visiting the -[Alert list](./alerts.md) and selecting an alert from the -list. You need least Developer [permissions](../../user/permissions.md) to access -alerts. - -TIP: **Tip:** -To review live examples of GitLab alerts, visit the -[alert list](https://gitlab.com/gitlab-examples/ops/incident-setup/everyone/tanuki-inc/-/alert_management) -for this demo project. Click any alert in the list to examine its alert details -page. - -Alerts provide **Overview** and **Alert details** tabs to give you the right -amount of information you need. - -## Alert overview tab - -The **Overview** tab provides basic information about the alert: - - - -## Alert details tab - - - -### Update an alert's status - -The Alert detail view enables you to update the Alert Status. -See [Create and manage alerts in GitLab](./alerts.md) for more details. - -### Create an issue from an alert - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1. - -The Alert detail view enables you to create an issue with a -description automatically populated from an alert. To create the issue, -click the **Create Issue** button. You can then view the issue from the -alert by clicking the **View Issue** button. - -Closing a GitLab issue associated with an alert changes the alert's status to Resolved. -See [Create and manage alerts in GitLab](alerts.md) for more details about alert statuses. - -### Update an alert's assignee - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab 13.1. - -The Alert detail view allows users to update the Alert assignee. - -In large teams, where there is shared ownership of an alert, it can be difficult -to track who is investigating and working on it. The Alert detail view -enables you to update the Alert assignee: - -NOTE: **Note:** -GitLab currently only supports a single assignee per alert. - -1. To display the list of current alerts, click - **{cloud-gear}** **Operations > Alerts**: - -  - -1. Select your desired alert to display its **Alert Details View**: - -  - -1. If the right sidebar is not expanded, click - **{angle-double-right}** **Expand sidebar** to expand it. -1. In the right sidebar, locate the **Assignee** and click **Edit**. From the - dropdown menu, select each user you want to assign to the alert. GitLab creates - a [to-do list item](../../user/todos.md) for each user. - -  - -To remove an assignee, click **Edit** next to the **Assignee** dropdown menu and -deselect the user from the list of assignees, or click **Unassigned**. - -### Alert system notes - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab 13.1. - -When you take action on an alert, this is logged as a system note, -which is visible in the Alert Details view. This gives you a linear -timeline of the alert's investigation and assignment history. - -The following actions will result in a system note: - -- [Updating the status of an alert](#update-an-alerts-status) -- [Creating an issue based on an alert](#create-an-issue-from-an-alert) -- [Assignment of an alert to a user](#update-an-alerts-assignee) - - - -### Create a to-do from an alert - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab 13.1. - -You can manually create [To-Do list items](../../user/todos.md) for yourself from the -Alert details screen, and view them later on your **To-Do List**. To add a to-do: - -1. To display the list of current alerts, click - **{cloud-gear}** **Operations > Alerts**. -1. Select your desired alert to display its **Alert Management Details View**. -1. Click the **Add a To-Do** button in the right sidebar: - -  - -Click the **To-Do** **{todo-done}** in the navigation bar to view your current to-do list. - - - -### View an alert's metrics data - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217768) in GitLab 13.2. - -To view the metrics for an alert: - - 1. Sign in as a user with Developer or higher [permissions](../../user/permissions.md). - 1. Navigate to **{cloud-gear}** **Operations > Alerts**. - 1. Click the alert you want to view. - 1. Below the title of the alert, click the **Metrics** tab. - - - -For GitLab-managed Prometheus instances, metrics data is automatically available -for the alert, making it easy to see surrounding behavior. See -[Managed Prometheus instances](../metrics/alerts.md#managed-prometheus-instances) -for information on setting up alerts. - -For externally-managed Prometheus instances, you can configure your alerting rules to -display a chart in the alert. See -[Embedding metrics based on alerts in incident issues](../metrics/embed.md#embedding-metrics-based-on-alerts-in-incident-issues) -for information on how to appropriately configure your alerting rules. See -[External Prometheus instances](../metrics/alerts.md#external-prometheus-instances) -for information on setting up alerts for your self-managed Prometheus instance. - -## Use cases for assigning alerts - -Consider a team formed by different sections of monitoring, collaborating on a -single application. After an alert surfaces, it's extremely important to -route the alert to the team members who can address and resolve the alert. - -Assigning Alerts eases collaboration and delegation. All -assignees are shown in your team's work-flows, and all assignees receive -notifications, simplifying communication and ownership of the alert. - -After completing their portion of investigating or fixing the alert, users can -unassign their account from the alert when their role is complete. -The alert status can be updated on the [Alert list](./alerts.md) to -reflect if the alert has been resolved. - -## View an alert's logs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217768) in GitLab 13.3. - -To view the logs for an alert: - - 1. Sign in as a user with Developer or higher [permissions](../../user/permissions.md). - 1. Navigate to **{cloud-gear}** **Operations > Alerts**. - 1. Click the alert you want to view. - 1. Below the title of the alert, click the **Metrics** tab. - 1. Click the [menu](../metrics/dashboards/index.md#chart-context-menu) of the metric chart to view options. - 1. Click **View logs**. - -Read [View logs from metrics panel](#view-logs-from-metrics-panel) for additional information. - -## Embed metrics in incidents and issues - -You can embed metrics anywhere [GitLab Markdown](../../user/markdown.md) is used, such as descriptions, -comments on issues, and merge requests. Embedding metrics helps you share them -when discussing incidents or performance issues. You can output the dashboard directly -into any issue, merge request, epic, or any other Markdown text field in GitLab -by [copying and pasting the link to the metrics dashboard](../metrics/embed.md#embedding-gitlab-managed-kubernetes-metrics). - -You can embed both -[GitLab-hosted metrics](../metrics/embed.md) and -[Grafana metrics](../metrics/embed_grafana.md) -in incidents and issue templates. - -### Context menu - -You can view more details about an embedded metrics panel from the context menu. -To access the context menu, click the **{ellipsis_v}** **More actions** dropdown box -above the upper right corner of the panel. For a list of options, see -[Chart context menu](../metrics/dashboards/index.md#chart-context-menu). - -#### View logs from metrics panel - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201846) in GitLab Ultimate 12.8. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25455) to [GitLab Core](https://about.gitlab.com/pricing/) 12.9. - -Viewing logs from a metrics panel can be useful if you're triaging an application -incident and need to [explore logs](../metrics/dashboards/index.md#chart-context-menu) -from across your application. These logs help you understand what is affecting -your application's performance and resolve any problems. +This document was moved to [another location](alerts.md). diff --git a/doc/operations/incident_management/alert_integrations.md b/doc/operations/incident_management/alert_integrations.md new file mode 100644 index 00000000000..f2225ad9992 --- /dev/null +++ b/doc/operations/incident_management/alert_integrations.md @@ -0,0 +1,132 @@ +--- +stage: Monitor +group: Health +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Generic alerts integration + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13203) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.4. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42640) to [GitLab Core](https://about.gitlab.com/pricing/) in 12.8. + +GitLab can accept alerts from any source via a generic webhook receiver. +When you set up the generic alerts integration, a unique endpoint will +be created which can receive a payload in JSON format, and will in turn +create an issue with the payload in the body of the issue. You can always +[customize the payload](#customizing-the-payload) to your liking. + +The entire payload will be posted in the issue discussion as a comment +authored by the GitLab Alert Bot. + +NOTE: **Note:** +In GitLab versions 13.1 and greater, you can configure +[External Prometheus instances](../metrics/alerts.md#external-prometheus-instances) +to use this endpoint. + +## Setting up generic alerts + +To obtain credentials for setting up a generic alerts integration: + +1. Sign in to GitLab as a user with maintainer [permissions](../../user/permissions.md) + for a project. +1. Navigate to the **Operations** page for your project, depending on your + installed version of GitLab: + - *In GitLab versions 13.1 and greater,* navigate to **Settings > Operations** + in your project. + - *In GitLab versions prior to 13.1,* navigate to **Settings > Integrations** + in your project. GitLab will display a banner encouraging you to enable + the Alerts endpoint in **Settings > Operations** instead. +1. Click **Alerts endpoint**. +1. Toggle the **Active** alert setting to display the **URL** and **Authorization Key** + for the webhook configuration. + +## Customizing the payload + +You can customize the payload by sending the following parameters. All fields +other than `title` are optional: + +| Property | Type | Description | +| ------------------------- | --------------- | ----------- | +| `title` | String | The title of the incident. Required. | +| `description` | String | A high-level summary of the problem. | +| `start_time` | DateTime | The time of the incident. If none is provided, a timestamp of the issue will be used. | +| `end_time` | DateTime | For existing alerts only. When provided, the alert is resolved and the associated incident is closed. | +| `service` | String | The affected service. | +| `monitoring_tool` | String | The name of the associated monitoring tool. | +| `hosts` | String or Array | One or more hosts, as to where this incident occurred. | +| `severity` | String | The severity of the alert. Must be one of `critical`, `high`, `medium`, `low`, `info`, `unknown`. Default is `critical`. | +| `fingerprint` | String or Array | The unique identifier of the alert. This can be used to group occurrences of the same alert. | +| `gitlab_environment_name` | String | The name of the associated GitLab [environment](../../ci/environments/index.md). This can be used to associate your alert to your environment. | + +You can also add custom fields to the alert's payload. The values of extra +parameters aren't limited to primitive types (such as strings or numbers), but +can be a nested JSON object. For example: + +```json +{ "foo": { "bar": { "baz": 42 } } } +``` + +TIP: **Payload size:** +Ensure your requests are smaller than the [payload application limits](../../administration/instance_limits.md#generic-alert-json-payloads). + +Example request: + +```shell +curl --request POST \ + --data '{"title": "Incident title"}' \ + --header "Authorization: Bearer <authorization_key>" \ + --header "Content-Type: application/json" \ + <url> +``` + +The `<authorization_key>` and `<url>` values can be found when [setting up generic alerts](#setting-up-generic-alerts). + +Example payload: + +```json +{ + "title": "Incident title", + "description": "Short description of the incident", + "start_time": "2019-09-12T06:00:55Z", + "service": "service affected", + "monitoring_tool": "value", + "hosts": "value", + "severity": "high", + "fingerprint": "d19381d4e8ebca87b55cda6e8eee7385", + "foo": { + "bar": { + "baz": 42 + } + } +} +``` + +## Triggering test alerts + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab Core in 13.2. + +After a [project maintainer or owner](#setting-up-generic-alerts) +[configures generic alerts](#setting-up-generic-alerts), you can trigger a test +alert to confirm your integration works properly. + +1. Sign in as a user with Developer or greater [permissions](../../user/permissions.md). +1. Navigate to **Settings > Operations** in your project. +1. Click **Alerts endpoint** to expand the section. +1. Enter a sample payload in **Alert test payload** (valid JSON is required). +1. Click **Test alert payload**. + +GitLab displays an error or success message, depending on the outcome of your test. + +## Automatic grouping of identical alerts **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214557) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. + +In GitLab versions 13.2 and greater, GitLab groups alerts based on their +payload. When an incoming alert contains the same payload as another alert +(excluding the `start_time` and `hosts` attributes), GitLab groups these alerts +together and displays a counter on the [Alert Management List](./incidents.md) +and details pages. + +If the existing alert is already `resolved`, GitLab creates a new alert instead. + + diff --git a/doc/operations/incident_management/alert_notifications.md b/doc/operations/incident_management/alert_notifications.md new file mode 100644 index 00000000000..1bff563579c --- /dev/null +++ b/doc/operations/incident_management/alert_notifications.md @@ -0,0 +1,35 @@ +--- +stage: Monitor +group: Health +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Alert notifications + +GitLab can react to alerts triggered from your applications. When an alert is +triggered in GitLab by [managed-Prometheus](../../user/project/integrations/prometheus.md#managed-prometheus-on-kubernetes) +or triggered using an external source and received with an integration, it's +important for a responder to be notified. + +Responders can receive notifications using the methods described on this page. + +## Slack notifications + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216326) in GitLab 13.1. + +You can be alerted by a Slack message when a new alert has been triggered. + +For setup information, see the [Slack Notifications Service docs](../../user/project/integrations/slack.md). + +## Email notifications + +If a project has been [configured to create incidents for triggered alerts](incidents.md#configure-incidents), +projects members with the _Owner_ or _Maintainer_ role will be sent an email +notification. To send additional email notifications to project members with the +Developer role: + +1. Navigate to **Settings > Operations**. +1. Expand the **Incidents** section. +1. In the **Alert Integration** tab, select the **Send a separate email notification to Developers** + check box. +1. Select **Save changes**. diff --git a/doc/operations/incident_management/alerts.md b/doc/operations/incident_management/alerts.md index d908af63000..bbfbb59c644 100644 --- a/doc/operations/incident_management/alerts.md +++ b/doc/operations/incident_management/alerts.md @@ -6,34 +6,36 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Create and manage alerts in GitLab -Users with at least Developer [permissions](../../user/permissions.md) can access -the Alert Management list at **{cloud-gear}** **Operations > Alerts** in your -project's sidebar. The Alert Management list displays alerts sorted by start time, -but you can change the sort order by clicking the headers in the Alert Management list. +Users with at least Developer [permissions](../../user/permissions.md) can +access the Alert Management list at **Operations > Alerts** in your project's +sidebar. The Alert Management list displays alerts sorted by start time, but +you can change the sort order by clicking the headers in the Alert Management list. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1.) The alert list displays the following information: - + -- **Search** - The alert list supports a simple free text search on the title, +- **Search**: The alert list supports a simple free text search on the title, description, monitoring tool, and service fields. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213884) in GitLab 13.1.) -- **Severity** - The current importance of a alert and how much attention it should - receive. For a listing of all statuses, read [Alert Management severity](#alert-severity). -- **Start time** - How long ago the alert fired. This field uses the standard - GitLab pattern of `X time ago`, but is supported by a granular date/time tooltip - depending on the user's locale. -- **Alert description** - The description of the alert, which attempts to capture the most meaningful data. -- **Event count** - The number of times that an alert has fired. -- **Issue** - A link to the incident issue that has been created for the alert. -- **Status** - The current status of the alert: +- **Severity**: The current importance of a alert and how much attention it + should receive. For a listing of all statuses, read [Alert Management severity](#alert-severity). +- **Start time**: How long ago the alert fired. This field uses the standard + GitLab pattern of `X time ago`, but is supported by a granular date/time + tooltip depending on the user's locale. +- **Alert description**: The description of the alert, which attempts to + capture the most meaningful data. +- **Event count**: The number of times that an alert has fired. +- **Issue**: A link to the incident issue that has been created for the alert. +- **Status**: The current status of the alert: - **Triggered**: No one has begun investigation. - **Acknowledged**: Someone is actively investigating the problem. - **Resolved**: No further work is required. - + TIP: **Tip:** -Check out a [live example](https://gitlab.com/gitlab-examples/ops/incident-setup/everyone/tanuki-inc/-/alert_management) +Check out a live example available from the +[`tanuki-inc` project page](https://gitlab-examples-ops-incident-setup-everyone-tanuki-inc.34.69.64.147.nip.io/) in GitLab to examine alerts in action. ## Enable Alerts @@ -42,65 +44,62 @@ NOTE: **Note:** You need at least Maintainer [permissions](../../user/permissions.md) to enable the Alerts feature. -There are several ways to accept alerts into your GitLab project. -Enabling any of these methods enables the Alert list. After configuring -alerts, visit **{cloud-gear}** **Operations > Alerts** in your project's sidebar -to view the list of alerts. +There are several ways to accept alerts into your GitLab project. Enabling any +of these methods enables the Alert list. After configuring alerts, visit +**Operations > Alerts** in your project's sidebar to view the list of alerts. ### Enable GitLab-managed Prometheus alerts You can install the GitLab-managed Prometheus application on your Kubernetes -cluster. For more information, read +cluster. For more information, see [Managed Prometheus on Kubernetes](../../user/project/integrations/prometheus.md#managed-prometheus-on-kubernetes). -When GitLab-managed Prometheus is installed, the [Alerts list](alerts.md) -is also enabled. +When GitLab-managed Prometheus is installed, the Alerts list is also enabled. -To populate the alerts with data, read +To populate the alerts with data, see [GitLab-Managed Prometheus instances](../metrics/alerts.md#managed-prometheus-instances). ### Enable external Prometheus alerts You can configure an externally-managed Prometheus instance to send alerts -to GitLab. To set up this configuration, read the [configuring Prometheus](../metrics/alerts.md#external-prometheus-instances) documentation. Activating the external Prometheus -configuration also enables the [Alerts list](./alerts.md). +to GitLab. To set up this configuration, read the +[configuring Prometheus](../metrics/alerts.md#external-prometheus-instances) +documentation. Activating the external Prometheus configuration also enables +the Alerts list. -To populate the alerts with data, read -[External Prometheus instances](../metrics/alerts.md#external-prometheus-instances). +To populate the alerts with data, see [External Prometheus instances](../metrics/alerts.md#external-prometheus-instances). ### Enable a Generic Alerts endpoint -GitLab provides the Generic Alerts endpoint so you can accept alerts from a third-party -alerts service. Read the -[instructions for toggling generic alerts](generic_alerts.md#setting-up-generic-alerts) -to add this option. After configuring the endpoint, the -[Alerts list](./alerts.md) is enabled. +GitLab provides the Generic Alerts endpoint so you can accept alerts from a +third-party alerts service. Read the [instructions for toggling generic alerts](alert_integrations.md#setting-up-generic-alerts) +to add this option. After configuring the endpoint, the Alerts list is enabled. -To populate the alerts with data, read [Customizing the payload](./generic_alerts.md#customizing-the-payload) for requests to the alerts endpoint. +To populate the alerts with data, see [Customizing the payload](alert_integrations.md#customizing-the-payload) +for requests to the alerts endpoint. ### Opsgenie integration **(PREMIUM)** > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. -A new way of monitoring Alerts via a GitLab integration is with -[Opsgenie](https://www.atlassian.com/software/opsgenie). +You can monitor alerts using a GitLab integration with [Opsgenie](https://www.atlassian.com/software/opsgenie). NOTE: **Note:** -If you enable the Opsgenie integration, you can't have other GitLab alert services, -such as [Generic Alerts](./generic_alerts.md) or -Prometheus alerts, active at the same time. +If you enable the Opsgenie integration, you can't have other GitLab alert +services, such as [Generic Alerts](generic_alerts.md) or Prometheus alerts, +active at the same time. To enable Opsgenie integration: 1. Sign in as a user with Maintainer or Owner [permissions](../../user/permissions.md). -1. Navigate to **{cloud-gear}** **Operations > Alerts**. -1. In the **Integrations** select box, select Opsgenie. -1. Click the **Active** toggle. -1. In the **API URL**, enter the base URL for your Opsgenie integration, such - as `https://app.opsgenie.com/alert/list`. -1. Click **Save changes**. +1. Navigate to **Operations > Alerts**. +1. In the **Integrations** select box, select **Opsgenie**. +1. Select the **Active** toggle. +1. In the **API URL** field, enter the base URL for your Opsgenie integration, + such as `https://app.opsgenie.com/alert/list`. +1. Select **Save changes**. -After enabling the integration, navigate to the Alerts list page at -**{cloud-gear}** **Operations > Alerts**, and click **View alerts in Opsgenie**. +After you enable the integration, navigate to the Alerts list page at +**Operations > Alerts**, and then select **View alerts in Opsgenie**. ## Alert severity @@ -108,15 +107,209 @@ Each level of alert contains a uniquely shaped and color-coded icon to help you identify the severity of a particular alert. These severity icons help you immediately identify which alerts you should prioritize investigating: - + Alerts contain one of the following icons: -| Severity | Icon | Color (hexadecimal) | -|---|---|---| -| Critical | **{severity-critical}** | `#8b2615` | -| High | **{severity-high}** | `#c0341d` | -| Medium | **{severity-medium}** | `#fca429` | -| Low | **{severity-low}** | `#fdbc60` | -| Info | **{severity-info}** | `#418cd8` | -| Unknown | **{severity-unknown}** | `#bababa` | +| Severity | Icon | Color (hexadecimal) | +|----------|-------------------------|---------------------| +| Critical | **{severity-critical}** | `#8b2615` | +| High | **{severity-high}** | `#c0341d` | +| Medium | **{severity-medium}** | `#fca429` | +| Low | **{severity-low}** | `#fdbc60` | +| Info | **{severity-info}** | `#418cd8` | +| Unknown | **{severity-unknown}** | `#bababa` | + +## Alert details page + +Navigate to the Alert details view by visiting the [Alert list](./alerts.md) +and selecting an alert from the list. You need least Developer [permissions](../../user/permissions.md) +to access alerts. + +TIP: **Tip:** +To review live examples of GitLab alerts, visit the +[alert list](https://gitlab.com/gitlab-examples/ops/incident-setup/everyone/tanuki-inc/-/alert_management) +for this demo project. Select any alert in the list to examine its alert details +page. + +Alerts provide **Overview** and **Alert details** tabs to give you the right +amount of information you need. + +### Alert overview tab + +The **Overview** tab provides basic information about the alert: + + + +### Alert details tab + + + +#### Update an alert's status + +The Alert detail view enables you to update the Alert Status. +See [Create and manage alerts in GitLab](./alerts.md) for more details. + +#### Create an issue from an alert + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1. + +The Alert detail view enables you to create an issue with a +description populated from an alert. To create the issue, +select the **Create Issue** button. You can then view the issue from the +alert by selecting the **View Issue** button. + +Closing a GitLab issue associated with an alert changes the alert's status to +Resolved. See [Create and manage alerts in GitLab](alerts.md) for more details +about alert statuses. + +#### Update an alert's assignee + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab 13.1. + +The Alert detail view allows users to update the Alert assignee. + +In large teams, where there is shared ownership of an alert, it can be +difficult to track who is investigating and working on it. The Alert detail +view enables you to update the Alert assignee: + +NOTE: **Note:** +GitLab supports only a single assignee per alert. + +1. To display the list of current alerts, navigate to **Operations > Alerts**: + +  + +1. Select your desired alert to display its **Alert Details View**: + +  + +1. If the right sidebar is not expanded, select + **{angle-double-right}** **Expand sidebar** to expand it. +1. In the right sidebar, locate the **Assignee**, and then select **Edit**. + From the dropdown menu, select each user you want to assign to the alert. + GitLab creates a [to-do item](../../user/todos.md) for each user. + +  + +To remove an assignee, select **Edit** next to the **Assignee** dropdown menu +and deselect the user from the list of assignees, or select **Unassigned**. + +#### Alert system notes + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab 13.1. + +When you take action on an alert, this is logged as a system note, +which is visible in the Alert Details view. This gives you a linear +timeline of the alert's investigation and assignment history. + +The following actions will result in a system note: + +- [Updating the status of an alert](#update-an-alerts-status) +- [Creating an issue based on an alert](#create-an-issue-from-an-alert) +- [Assignment of an alert to a user](#update-an-alerts-assignee) + + + +#### Create a to do from an alert + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab 13.1. + +You can manually create [To-Do list items](../../user/todos.md) for yourself +from the Alert details screen, and view them later on your **To-Do List**. To +add a to do: + +1. To display the list of current alerts, navigate to **Operations > Alerts**. +1. Select your desired alert to display its **Alert Management Details View**. +1. Select the **Add a To-Do** button in the right sidebar: + +  + +Select the **To-Do List** **{todo-done}** in the navigation bar to view your current to-do list. + + + +#### View an alert's metrics data + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217768) in GitLab 13.2. + +To view the metrics for an alert: + + 1. Sign in as a user with Developer or higher [permissions](../../user/permissions.md). + 1. Navigate to **Operations > Alerts**. + 1. Select the alert you want to view. + 1. Below the title of the alert, select the **Metrics** tab. + + + +For GitLab-managed Prometheus instances, metrics data is available for the +alert, making it easy to see surrounding behavior. For information about +setting up alerts, see [Managed Prometheus instances](../metrics/alerts.md#managed-prometheus-instances). + +For externally-managed Prometheus instances, you can configure your alerting +rules to display a chart in the alert. For information about how to configure +your alerting rules, see [Embedding metrics based on alerts in incident issues](../metrics/embed.md#embedding-metrics-based-on-alerts-in-incident-issues). See +[External Prometheus instances](../metrics/alerts.md#external-prometheus-instances) +for information about setting up alerts for your self-managed Prometheus +instance. + +### Use cases for assigning alerts + +Consider a team formed by different sections of monitoring, collaborating on a +single application. After an alert surfaces, it's extremely important to route +the alert to the team members who can address and resolve the alert. + +Assigning Alerts eases collaboration and delegation. All assignees are shown in +your team's work-flows, and all assignees receive notifications, simplifying +communication and ownership of the alert. + +After completing their portion of investigating or fixing the alert, users can +unassign their account from the alert when their role is complete. You can +update the alert on the [Alert list](./alerts.md) to reflect if the alert has +been resolved. + +### View an alert's logs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217768) in GitLab 13.3. + +To view the logs for an alert: + + 1. Sign in as a user with Developer or higher [permissions](../../user/permissions.md). + 1. Navigate to **Operations > Alerts**. + 1. Select the alert you want to view. + 1. Below the title of the alert, select the **Metrics** tab. + 1. Select the [menu](../metrics/dashboards/index.md#chart-context-menu) of + the metric chart to view options. + 1. Select **View logs**. + +For additional information, see [View logs from metrics panel](#view-logs-from-metrics-panel). + +### Embed metrics in incidents and issues + +You can embed metrics anywhere [GitLab Markdown](../../user/markdown.md) is +used, such as descriptions, comments on issues, and merge requests. Embedding +metrics helps you share them when discussing incidents or performance issues. +You can output the dashboard directly into any issue, merge request, epic, or +any other Markdown text field in GitLab by +[copying and pasting the link to the metrics dashboard](../metrics/embed.md#embedding-gitlab-managed-kubernetes-metrics). + +You can embed both [GitLab-hosted metrics](../metrics/embed.md) and +[Grafana metrics](../metrics/embed_grafana.md) in incidents and issue +templates. + +#### Context menu + +You can view more details about an embedded metrics panel from the context +menu. To access the context menu, select the **{ellipsis_v}** **More actions** +dropdown box above the upper right corner of the panel. For a list of options, +see [Chart context menu](../metrics/dashboards/index.md#chart-context-menu). + +##### View logs from metrics panel + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201846) in GitLab Ultimate 12.8. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25455) to [GitLab Core](https://about.gitlab.com/pricing/) 12.9. + +Viewing logs from a metrics panel can be useful if you're triaging an +application incident and need to [explore logs](../metrics/dashboards/index.md#chart-context-menu) +from across your application. These logs help you understand what's affecting +your application's performance and how to resolve any problems. diff --git a/doc/operations/incident_management/generic_alerts.md b/doc/operations/incident_management/generic_alerts.md index 11d4dbc6924..a8f2f9a58a6 100644 --- a/doc/operations/incident_management/generic_alerts.md +++ b/doc/operations/incident_management/generic_alerts.md @@ -1,126 +1,5 @@ --- -stage: Monitor -group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +redirect_to: alert_notifications.md --- -# Generic alerts integration - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13203) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.4. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42640) to [GitLab Core](https://about.gitlab.com/pricing/) in 12.8. - -GitLab can accept alerts from any source via a generic webhook receiver. -When you set up the generic alerts integration, a unique endpoint will -be created which can receive a payload in JSON format, and will in turn -create an issue with the payload in the body of the issue. You can always -[customize the payload](#customizing-the-payload) to your liking. - -The entire payload will be posted in the issue discussion as a comment -authored by the GitLab Alert Bot. - -NOTE: **Note:** -In GitLab versions 13.1 and greater, you can configure -[External Prometheus instances](../metrics/alerts.md#external-prometheus-instances) -to use this endpoint. - -## Setting up generic alerts - -To obtain credentials for setting up a generic alerts integration: - -- Sign in to GitLab as a user with maintainer [permissions](../../user/permissions.md) for a project. -- Navigate to the **Operations** page for your project, depending on your installed version of GitLab: - - *In GitLab versions 13.1 and greater,* navigate to **Settings > Operations** in your project. - - *In GitLab versions prior to 13.1,* navigate to **Settings > Integrations** in your project. GitLab will display a banner encouraging you to enable the Alerts endpoint in **Settings > Operations** instead. -- Click **Alerts endpoint**. -- Toggle the **Active** alert setting to display the **URL** and **Authorization Key** for the webhook configuration. - -## Customizing the payload - -You can customize the payload by sending the following parameters. All fields other than `title` are optional: - -| Property | Type | Description | -| -------- | ---- | ----------- | -| `title` | String | The title of the incident. Required. | -| `description` | String | A high-level summary of the problem. | -| `start_time` | DateTime | The time of the incident. If none is provided, a timestamp of the issue will be used. | -| `end_time` | DateTime | For existing alerts only. When provided, the alert is resolved and the associated incident is closed. | -| `service` | String | The affected service. | -| `monitoring_tool` | String | The name of the associated monitoring tool. | -| `hosts` | String or Array | One or more hosts, as to where this incident occurred. | -| `severity` | String | The severity of the alert. Must be one of `critical`, `high`, `medium`, `low`, `info`, `unknown`. Default is `critical`. | -| `fingerprint` | String or Array | The unique identifier of the alert. This can be used to group occurrences of the same alert. | -| `gitlab_environment_name` | String | The name of the associated GitLab [environment](../../ci/environments/index.md). This can be used to associate your alert to your environment. | - -You can also add custom fields to the alert's payload. The values of extra parameters -are not limited to primitive types, such as strings or numbers, but can be a nested -JSON object. For example: - -```json -{ "foo": { "bar": { "baz": 42 } } } -``` - -TIP: **Payload size:** -Ensure your requests are smaller than the [payload application limits](../../administration/instance_limits.md#generic-alert-json-payloads). - -Example request: - -```shell -curl --request POST \ - --data '{"title": "Incident title"}' \ - --header "Authorization: Bearer <authorization_key>" \ - --header "Content-Type: application/json" \ - <url> -``` - -The `<authorization_key>` and `<url>` values can be found when [setting up generic alerts](#setting-up-generic-alerts). - -Example payload: - -```json -{ - "title": "Incident title", - "description": "Short description of the incident", - "start_time": "2019-09-12T06:00:55Z", - "service": "service affected", - "monitoring_tool": "value", - "hosts": "value", - "severity": "high", - "fingerprint": "d19381d4e8ebca87b55cda6e8eee7385", - "foo": { - "bar": { - "baz": 42 - } - } -} -``` - -## Triggering test alerts - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab Core in 13.2. - -After a [project maintainer or owner](#setting-up-generic-alerts) -[configures generic alerts](#setting-up-generic-alerts), you can trigger a -test alert to confirm your integration works properly. - -1. Sign in as a user with Developer or greater [permissions](../../user/permissions.md). -1. Navigate to **Settings > Operations** in your project. -1. Click **Alerts endpoint** to expand the section. -1. Enter a sample payload in **Alert test payload** (valid JSON is required). -1. Click **Test alert payload**. - -GitLab displays an error or success message, depending on the outcome of your test. - -## Automatic grouping of identical alerts **(PREMIUM)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214557) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. - -In GitLab versions 13.2 and greater, GitLab groups alerts based on their payload. -When an incoming alert contains the same payload as another alert (excluding the -`start_time` and `hosts` attributes), GitLab groups these alerts together and -displays a counter on the -[Alert Management List](./incidents.md) -and details pages. - -If the existing alert is already `resolved`, then a new alert will be created instead. - - +This document was moved to [another location](alert_notifications.md). diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index 3ff02b3dc6b..1086885a58b 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w While no configuration is required to use the [manual features](#create-an-incident-manually) of incident management, some simple [configuration](#configure-incidents) is needed to automate incident creation. -For users with at least Developer [permissions](../../user/permissions.md), the +For users with at least Guest [permissions](../../user/permissions.md), the Incident Management list is available at **Operations > Incidents** in your project's sidebar. The list contains the following metrics: @@ -28,7 +28,8 @@ in your project's sidebar. The list contains the following metrics: - **{severity-unknown}** **Unknown** NOTE: **Note:** - Editing incident severity on the incident details page was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229402) in GitLab 13.4. + Editing incident severity on the incident details page was + [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229402) in GitLab 13.4. - **Incident** - The description of the incident, which attempts to capture the most meaningful data. @@ -71,7 +72,7 @@ to create issues when alerts are triggered: [Trigger actions from alerts](../metrics/alerts.md#trigger-actions-from-alerts) **(ULTIMATE)**. 1. To create issues from alerts, select the template in the **Issue Template** select box. -1. To send [separate email notifications](index.md#notify-developers-of-alerts) to users +1. To send [separate email notifications](alert_notifications.md#email-notifications) to users with [Developer permissions](../../user/permissions.md), select **Send a separate email notification to Developers**. 1. Click **Save changes**. @@ -84,7 +85,7 @@ when you receive notification that the alert is resolved. ## Create an incident manually -If you have at least Developer [permissions](../../user/permissions.md), to create an Incident, you have two options. +If you have at least Guest [permissions](../../user/permissions.md), to create an Incident, you have two options. ### From the Incidents List diff --git a/doc/operations/incident_management/index.md b/doc/operations/incident_management/index.md index 28e69a6bbfe..df190f52393 100644 --- a/doc/operations/incident_management/index.md +++ b/doc/operations/incident_management/index.md @@ -14,28 +14,7 @@ being developed, efficiency and awareness can be increased. GitLab offers solutions for handling incidents in your applications and services, such as [setting up Prometheus alerts](#configure-prometheus-alerts), -[displaying metrics](./alert_details.md#embed-metrics-in-incidents-and-issues), and sending notifications. - -## Alert notifications - -### Slack Notifications - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216326) in GitLab 13.1. - -You can be alerted via a Slack message when a new alert has been received. - -See the [Slack Notifications Service docs](../../user/project/integrations/slack.md) for information on how to set this up. - -### Notify developers of alerts - -GitLab can react to the alerts triggered from your applications and services -by creating issues and alerting developers through email. By default, GitLab -sends these emails to [owners and maintainers](../../user/permissions.md) of the project. -These emails contain details of the alert, and a link for more information. - -To send separate email notifications to users with -[Developer permissions](../../user/permissions.md), see -[Configure incidents](./incidents.md#configure-incidents). +[displaying metrics](./alerts.md#embed-metrics-in-incidents-and-issues), and sending notifications. ## Configure Prometheus alerts @@ -54,7 +33,7 @@ When [configuring the generic alerts integration](./generic_alerts.md), GitLab creates a unique endpoint which receives a JSON-formatted, customizable payload. After configuration, you can manage your alerts using either the -[alerts section](./alerts.md) or the [alert details section](./alert_details.md). +[alerts section](./alerts.md) or the [alert details section](./alerts.md#alert-details-page). ## Integrate incidents with Slack diff --git a/doc/operations/index.md b/doc/operations/index.md index 7ab34502277..b49487b60e6 100644 --- a/doc/operations/index.md +++ b/doc/operations/index.md @@ -35,7 +35,7 @@ using metrics and logs, and promote the critical alerts to incidents. Are your alerts too noisy? Alerts configured on GitLab metrics can configured and fine-tuned in GitLab immediately following a fire-fight. -- [Manage alerts and incidents](../user/incident_management/index.md) in GitLab. +- [Manage alerts and incidents](incident_management/index.md) in GitLab. - [Configure alerts for metrics](metrics/alerts.md#set-up-alerts-for-prometheus-metrics) in GitLab. - Create a [status page](incident_management/status_page.md) to communicate efficiently to your users during an incident. diff --git a/doc/operations/metrics/alerts.md b/doc/operations/metrics/alerts.md index 5b880ab9746..c0a16da3702 100644 --- a/doc/operations/metrics/alerts.md +++ b/doc/operations/metrics/alerts.md @@ -15,7 +15,7 @@ your team when environment performance falls outside of the boundaries you set. ## Managed Prometheus instances -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6590) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2 for [custom metrics](index.md#adding-custom-metrics), and GitLab 11.3 for [library metrics](../../user/project/integrations/prometheus_library/metrics.md). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6590) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2 for [custom metrics](index.md#adding-custom-metrics), and GitLab 11.3 for [library metrics](../../user/project/integrations/prometheus_library/index.md). For managed Prometheus instances using auto configuration, you can [configure alerts for metrics](index.md#adding-custom-metrics) directly in the @@ -70,7 +70,8 @@ receivers: bearer_token: 9e1cbfcd546896a9ea8be557caf13a76 send_resolved: true url: http://192.168.178.31:3001/root/manual_prometheus/prometheus/alerts/notify.json - ... + # Rest of configuration omitted + # ... ``` For GitLab to associate your alerts with an [environment](../../ci/environments/index.md), diff --git a/doc/operations/metrics/dashboards/img/metrics_dashboard_panel_preview_v13_3.png b/doc/operations/metrics/dashboards/img/metrics_dashboard_panel_preview_v13_3.png Binary files differindex 4f6d3b3dfa4..3c3203265e1 100644 --- a/doc/operations/metrics/dashboards/img/metrics_dashboard_panel_preview_v13_3.png +++ b/doc/operations/metrics/dashboards/img/metrics_dashboard_panel_preview_v13_3.png diff --git a/doc/operations/metrics/dashboards/index.md b/doc/operations/metrics/dashboards/index.md index ffcb7dc92c6..42dc5bbfb6d 100644 --- a/doc/operations/metrics/dashboards/index.md +++ b/doc/operations/metrics/dashboards/index.md @@ -51,16 +51,16 @@ To create a new dashboard from the command line: - group: 'Group Title' panels: - type: area-chart - title: "Chart Title" - y_label: "Y-Axis" + title: 'Chart Title' + y_label: 'Y-Axis' y_axis: format: number precision: 0 metrics: - id: my_metric_id query_range: 'http_requests_total' - label: "Instance: {{instance}}, method: {{method}}" - unit: "count" + label: 'Instance: {{instance}}, method: {{method}}' + unit: 'count' ``` 1. Save the file, commit, and push to your repository. The file must be present in your **default** branch. diff --git a/doc/operations/metrics/dashboards/panel_types.md b/doc/operations/metrics/dashboards/panel_types.md index b2cbdcb88d9..20f6e25cb6b 100644 --- a/doc/operations/metrics/dashboards/panel_types.md +++ b/doc/operations/metrics/dashboards/panel_types.md @@ -17,16 +17,16 @@ dashboard: 'Dashboard Title' panel_groups: - group: 'Group Title' panels: - - type: area-chart # or line-chart + - type: area-chart # or line-chart title: 'Area Chart Title' - y_label: "Y-Axis" + y_label: 'Y-Axis' y_axis: format: number precision: 0 metrics: - id: area_http_requests_total query_range: 'http_requests_total' - label: "Instance: {{instance}}, Method: {{method}}" + label: 'Instance: {{instance}}, Method: {{method}}' unit: "count" ``` @@ -55,23 +55,23 @@ panel_groups: - group: 'Group Title' panels: - type: anomaly-chart - title: "Chart Title" + title: 'Chart Title' y_label: "Y-Axis" metrics: - id: anomaly_requests_normal query_range: 'http_requests_total' - label: "# of Requests" - unit: "count" + label: '# of Requests' + unit: 'count' metrics: - id: anomaly_requests_upper_limit query_range: 10000 - label: "Max # of requests" - unit: "count" + label: 'Max # of requests' + unit: 'count' metrics: - id: anomaly_requests_lower_limit query_range: 2000 - label: "Min # of requests" - unit: "count" + label: 'Min # of requests' + unit: 'count' ``` Note the following properties: @@ -93,13 +93,13 @@ panel_groups: - group: 'Group title' panels: - type: bar - title: "Http Handlers" + title: 'HTTP Handlers' x_label: 'Response Size' y_axis: - name: "Handlers" + name: 'Handlers' metrics: - id: prometheus_http_response_size_bytes_bucket - query_range: "sum(increase(prometheus_http_response_size_bytes_bucket[1d])) by (handler)" + query_range: 'sum(increase(prometheus_http_response_size_bytes_bucket[1d])) by (handler)' unit: 'Bytes' ``` @@ -121,13 +121,13 @@ dashboard: 'Dashboard Title' panel_groups: - group: 'Group title' panels: - - title: "Column" - type: "column" + - title: 'Column' + type: 'column' metrics: - id: 1024_memory query: 'avg(sum(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}) by (job)) without (job) / count(avg(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}) without (job)) /1024/1024' unit: MB - label: "Memory Usage" + label: 'Memory Usage' ``` Note the following properties: @@ -153,19 +153,19 @@ panel_groups: priority: 5 panels: - type: 'stacked-column' - title: "Stacked column" - y_label: "y label" + title: 'Stacked column' + y_label: 'y label' x_label: 'x label' metrics: - id: memory_1 query_range: 'memory_query' - label: "memory query 1" - unit: "count" + label: 'memory query 1' + unit: 'count' series_name: 'group 1' - id: memory_2 query_range: 'memory_query_2' - label: "memory query 2" - unit: "count" + label: 'memory query 2' + unit: 'count' series_name: 'group 2' ``` @@ -185,13 +185,13 @@ dashboard: 'Dashboard Title' panel_groups: - group: 'Group Title' panels: - - title: "Single Stat" - type: "single-stat" + - title: 'Single Stat' + type: 'single-stat' metrics: - id: 10 query: 'max(go_memstats_alloc_bytes{job="prometheus"})' unit: MB - label: "Total" + label: 'Total' ``` Note the following properties: @@ -215,14 +215,14 @@ dashboard: 'Dashboard Title' panel_groups: - group: 'Group Title' panels: - - title: "Single Stat" - type: "single-stat" + - title: 'Single Stat' + type: 'single-stat' max_value: 100 metrics: - id: 10 query: 'max(go_memstats_alloc_bytes{job="prometheus"})' unit: '%' - label: "Total" + label: 'Total' ``` For example, if you have a query value of `53.6`, adding `%` as the unit results in a single stat value of `53.6%`, but if the maximum expected value of the query is `120`, the value would be `44.6%`. Adding the `max_value` causes the correct percentage value to display. @@ -242,15 +242,15 @@ dashboard: 'Dashboard Title' panel_groups: - group: 'Group Title' panels: - - title: "Gauge" - type: "gauge" + - title: 'Gauge' + type: 'gauge' min_value: 0 max_value: 1000 split: 5 thresholds: values: [60, 90] - mode: "percentage" - format: "kilobytes" + mode: 'percentage' + format: 'kilobytes' metrics: - id: 10 query: 'floor(max(prometheus_http_response_size_bytes_bucket)/1000)' @@ -289,13 +289,13 @@ dashboard: 'Dashboard Title' panel_groups: - group: 'Group Title' panels: - - title: "Heatmap" - type: "heatmap" + - title: 'Heatmap' + type: 'heatmap' metrics: - id: 10 query: 'sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[60m])) by (status_code)' unit: req/sec - label: "Status code" + label: 'Status code' ``` Note the following properties: diff --git a/doc/operations/metrics/dashboards/yaml.md b/doc/operations/metrics/dashboards/yaml.md index f92ba4079e9..fac2c7b40d7 100644 --- a/doc/operations/metrics/dashboards/yaml.md +++ b/doc/operations/metrics/dashboards/yaml.md @@ -103,8 +103,8 @@ When a static label is used and a query returns multiple time series, then all t metrics: - id: my_metric_id query_range: 'http_requests_total' - label: "Time Series" - unit: "count" + label: 'Time Series' + unit: 'count' ``` This may render a legend like this: @@ -117,8 +117,8 @@ For labels to be more explicit, using variables that reflect time series labels metrics: - id: my_metric_id query_range: 'http_requests_total' - label: "Instance: {{instance}}, method: {{method}}" - unit: "count" + label: 'Instance: {{instance}}, method: {{method}}' + unit: 'count' ``` The resulting rendered legend will look like this: @@ -131,8 +131,8 @@ There is also a shorthand value for dynamic dashboard labels that make use of on metrics: - id: my_metric_id query_range: 'http_requests_total' - label: "Method" - unit: "count" + label: 'Method' + unit: 'count' ``` This works by lowercasing the value of `label` and, if there are more words separated by spaces, replacing those spaces with an underscore (`_`). The transformed value is then checked against the labels of the time series returned by the Prometheus query. If a time series label is found that is equal to the transformed value, then the label value will be used and rendered in the legend like this: diff --git a/doc/push_rules/push_rules.md b/doc/push_rules/push_rules.md index 1643c96d229..032ec1cb8ca 100644 --- a/doc/push_rules/push_rules.md +++ b/doc/push_rules/push_rules.md @@ -97,7 +97,7 @@ The following options are available. | Restrict by commit message (negative match)| **Starter** 11.1 | Only commit messages that do not match this regular expression are allowed to be pushed. Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | | Restrict by branch name | **Starter** 9.3 | Only branch names that match this regular expression are allowed to be pushed. Leave empty to allow any branch name. | | Restrict by commit author's email | **Starter** 7.10 | Only commit author's email that match this regular expression are allowed to be pushed. Leave empty to allow any email. | -| Prohibited file names | **Starter** 7.10 | Any committed filenames that match this regular expression are not allowed to be pushed. Leave empty to allow any filenames. | +| Prohibited file names | **Starter** 7.10 | Any committed filenames that match this regular expression and do not already exist in the repository are not allowed to be pushed. Leave empty to allow any filenames. See [common examples](#prohibited-file-names). | | Maximum file size | **Starter** 7.12 | Pushes that contain added or updated files that exceed this file size (in MB) are rejected. Set to 0 to allow files of any size. Files tracked by Git LFS are exempted. | TIP: **Tip:** @@ -178,6 +178,44 @@ pry.history bash_history ``` +## Prohibited file names + +> Introduced in [GitLab Starter](https://about.gitlab.com/pricing/) 7.10. + +Each file name contained in a Git push is compared to the regular expression in this field. Filenames in Git consist of both the file's name and any directory that may precede it. A singular regular expression can contain multiple independent matches used as exclusions. File names can be broadly matched to any location in the repository, or restricted to specific locations. Filenames can also be partial matches used to exclude file types by extension. + +The following examples make use of regex string boundary characters which match the beginning of a string (`^`), and the end (`$`). They also include instances where either the directory path or the filename can include `.` or `/`. Both of these special regex characters have to be escaped with a backslash `\` to be used as normal characters in a match condition. + +Example: prevent pushing any `.exe` files to any location in the repository. This is an example of a partial match, which can match any filename that contains `.exe` at the end: + +```plaintext +\.exe$ +``` + +Example: prevent a specific configuration file in the repository root from being pushed: + +```plaintext +^config\.yml$ +``` + +Example: prevent a specific configuration file in a known directory from being pushed: + +```plaintext +^directory-name\/config\.yml$ +``` + +Example: prevent the specific file named `install.exe` from being pushed to any location in the repository. Note that the parenthesized expression `(^|\/)` will match either a file following a directory separator or a file in the root directory of the repository: + +```plaintext +(^|\/)install\.exe$ +``` + +Example: combining all of the above in a single expression. Note that all of the preceding expressions rely on the end of string character `$`, so we can move that part of each expression to the end of the grouped collection of match conditions where it will be appended to all matches: + +```plaintext +(\.exe|^config\.yml|^directory-name\/config\.yml|(^|\/)install\.exe)$ +``` + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md index 523486d5137..7aec74f1243 100644 --- a/doc/raketasks/README.md +++ b/doc/raketasks/README.md @@ -21,7 +21,7 @@ The following are available Rake tasks: | [Clean up](cleanup.md) | Clean up unneeded items from GitLab instances. | | [Development](../development/rake_tasks.md) | Tasks for GitLab contributors. | | [Doctor tasks](../administration/raketasks/doctor.md) | Checks for data integrity issues. | -| [Elasticsearch](../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) **(STARTER ONLY)** | Maintain Elasticsearch in a GitLab instance. | +| [Elasticsearch](../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) **(STARTER ONLY)** | Maintain Elasticsearch in a GitLab instance. | | [Enable namespaces](features.md) | Enable usernames and namespaces for user projects. | | [General maintenance](../administration/raketasks/maintenance.md) | General maintenance and self-check tasks. | | [Geo maintenance](../administration/raketasks/geo.md) **(PREMIUM ONLY)** | [Geo](../administration/geo/index.md)-related maintenance. | diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 68b076258ce..c050931ce3b 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -10,48 +10,41 @@ of GitLab on which it was created. The best way to migrate your repositories from one server to another is through backup restore. CAUTION: **Warning:** -GitLab will not backup items that are not stored on the -filesystem. If using [object storage](../administration/object_storage.md), -remember to enable backups with your object storage provider if desired. +GitLab won't back up items that aren't stored in the filesystem. If you're +using [object storage](../administration/object_storage.md), be sure to enable +backups with your object storage provider, if desired. ## Requirements -In order to be able to backup and restore, you need one essential tool -installed on your system. +To be able to backup and restore, ensure that Rsync is installed on your +system. If you installed GitLab: -- **Rsync**: If you installed GitLab: - - Using the Omnibus package, you're all set. - - From source, make sure `rsync` is installed. For example: +- _Using the Omnibus package_, you're all set. +- _From source_, you need to determine if `rsync` is installed. For example: - ```shell - # Debian/Ubuntu - sudo apt-get install rsync + ```shell + # Debian/Ubuntu + sudo apt-get install rsync - # RHEL/CentOS - sudo yum install rsync - ``` + # RHEL/CentOS + sudo yum install rsync + ``` ## Backup timestamp -NOTE: **Note:** -In GitLab 9.2 the timestamp format was changed from `EPOCH_YYYY_MM_DD` to -`EPOCH_YYYY_MM_DD_GitLab_version`, for example `1493107454_2018_04_25` -would become `1493107454_2018_04_25_10.6.4-ce`. - The backup archive will be saved in `backup_path`, which is specified in the -`config/gitlab.yml` file. -The filename will be `[TIMESTAMP]_gitlab_backup.tar`, where `TIMESTAMP` -identifies the time at which each backup was created, plus the GitLab version. -The timestamp is needed if you need to restore GitLab and multiple backups are -available. +`config/gitlab.yml` file. The filename will be `[TIMESTAMP]_gitlab_backup.tar`, +where `TIMESTAMP` identifies the time at which each backup was created, plus +the GitLab version. The timestamp is needed if you need to restore GitLab and +multiple backups are available. For example, if the backup name is `1493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar`, -then the timestamp is `1493107454_2018_04_25_10.6.4-ce`. +the timestamp is `1493107454_2018_04_25_10.6.4-ce`. ## Back up GitLab -GitLab provides a simple command line interface to back up your whole instance. -It backs up your: +GitLab provides a command line interface to back up your entire instance, +including: - Database - Attachments @@ -63,47 +56,59 @@ It backs up your: - GitLab Pages content CAUTION: **Warning:** -GitLab does not back up any configuration files, SSL certificates, or system files. -You are highly advised to [read about storing configuration files](#storing-configuration-files). +GitLab does not back up any configuration files, SSL certificates, or system +files. You are highly advised to read about [storing configuration files](#storing-configuration-files). -Use this command if you've installed GitLab with the Omnibus package: +Depending on your version of GitLab, use the following command if you installed +GitLab using the Omnibus package: -```shell -sudo gitlab-backup create -``` +- GitLab 12.2 or later: + + ```shell + sudo gitlab-backup create + ``` -NOTE: **Note:** -For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. +- GitLab 12.1 and earlier: -Use this if you've installed GitLab from source: + ```shell + gitlab-rake gitlab:backup:create + ``` + +If you installed GitLab from source, use the following command: ```shell sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ``` -If you are running GitLab within a Docker container, you can run the backup from the host: +If you're running GitLab from within a Docker container, run the backup from +the host, based on your installed version of GitLab: -```shell -docker exec -t <container name> gitlab-backup create -``` +- GitLab 12.2 or later: + + ```shell + docker exec -t <container name> gitlab-backup create + ``` -NOTE: **Note:** -For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. +- GitLab 12.1 and earlier: -If you are using the [GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab) on a -Kubernetes cluster, you can run the backup task using `backup-utility` script on -the GitLab task runner pod via `kubectl`. Refer to [backing up a GitLab installation](https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/backup-restore/backup.md#backing-up-a-gitlab-installation) for more details: + ```shell + gitlab-rake gitlab:backup:create + ``` + +If you're using the [GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab) +on a Kubernetes cluster, you can run the backup task by using `kubectl` to run the `backup-utility` +script on the GitLab task runner pod. For more details, see +[backing up a GitLab installation](https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/backup-restore/backup.md#backing-up-a-gitlab-installation). ```shell kubectl exec -it <gitlab task-runner pod> backup-utility ``` -Similarly to the Kubernetes case, if you have scaled out your GitLab -cluster to use multiple application servers, you should pick a -designated node (that won't be auto-scaled away) for running the -backup Rake task. Because the backup Rake task is tightly coupled to -the main Rails application, this is typically a node on which you're -also running Unicorn/Puma and/or Sidekiq. +Similar to the Kubernetes case, if you have scaled out your GitLab cluster to +use multiple application servers, you should pick a designated node (that won't +be auto-scaled away) for running the backup Rake task. Because the backup Rake +task is tightly coupled to the main Rails application, this is typically a node +on which you're also running Unicorn/Puma or Sidekiq. Example output: @@ -136,11 +141,11 @@ Deleting old backups... [SKIPPING] ### Storing configuration files -The [backup Rake task](#back-up-gitlab) GitLab provides -does **not** store your configuration files. The primary reason for this is that your -database contains encrypted information for two-factor authentication, the CI/CD -'secure variables', and so on. Storing encrypted information along with its key in the -same place defeats the purpose of using encryption in the first place. +The [backup Rake task](#back-up-gitlab) GitLab provides does _not_ store your +configuration files. The primary reason for this is that your database contains +items including encrypted information for two-factor authentication and the +CI/CD _secure variables_. Storing encrypted information in the same location +as its key defeats the purpose of using encryption in the first place. CAUTION: **Warning:** The secrets file is essential to preserve your database encryption key. @@ -158,30 +163,31 @@ For installation from source: - `/home/git/gitlab/config/gitlab.yml` For [Docker installations](https://docs.gitlab.com/omnibus/docker/), you must -back up the volume where the configuration files are stored. If you have created -the GitLab container according to the documentation, it should be under -`/srv/gitlab/config`. +back up the volume where the configuration files are stored. If you created +the GitLab container according to the documentation, it should be in the +`/srv/gitlab/config` directory. -For [GitLab Helm chart Installations](https://gitlab.com/gitlab-org/charts/gitlab) on a -Kubernetes cluster, you must follow the [Backup the secrets](https://docs.gitlab.com/charts/backup-restore/backup.html#backup-the-secrets) instructions. +For [GitLab Helm chart installations](https://gitlab.com/gitlab-org/charts/gitlab) +on a Kubernetes cluster, you must follow the +[Backup the secrets](https://docs.gitlab.com/charts/backup-restore/backup.html#backup-the-secrets) +instructions. You may also want to back up any TLS keys and certificates, and your [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). -If you use Omnibus GitLab, see some additional information -[to backup your configuration](https://docs.gitlab.com/omnibus/settings/backups.html). +If you use Omnibus GitLab, review additional information to +[backup your configuration](https://docs.gitlab.com/omnibus/settings/backups.html). In the unlikely event that the secrets file is lost, see the [troubleshooting section](#when-the-secrets-file-is-lost). ### Backup options -The command line tool GitLab provides to backup your instance can take more options. +The command line tool GitLab provides to backup your instance can accept more +options. #### Backup strategy option -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8728) in GitLab 8.17. - The default backup strategy is to essentially stream data from the respective data locations to the backup using the Linux command `tar` and `gzip`. This works fine in most cases, but can cause problems when data is rapidly changing. @@ -203,8 +209,7 @@ To use the `copy` strategy instead of the default streaming strategy, specify sudo gitlab-backup create STRATEGY=copy ``` -NOTE: **Note:** -For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. #### Backup filename @@ -212,34 +217,41 @@ CAUTION: **Warning:** If you use a custom backup filename, you will not be able to [limit the lifetime of the backups](#limit-backup-lifetime-for-local-files-prune-old-backups). -By default a backup file is created according to the specification in [the Backup timestamp](#backup-timestamp) section above. You can however override the `[TIMESTAMP]` part of the filename by setting the `BACKUP` environment variable. For example: +By default, a backup file is created according to the specification in the +previous [Backup timestamp](#backup-timestamp) section. You can, however, +override the `[TIMESTAMP]` portion of the filename by setting the `BACKUP` +environment variable. For example: ```shell sudo gitlab-backup create BACKUP=dump ``` -NOTE: **Note:** -For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. -The resulting file will then be `dump_gitlab_backup.tar`. This is useful for systems that make use of rsync and incremental backups, and will result in considerably faster transfer speeds. +The resulting file will then be `dump_gitlab_backup.tar`. This is useful for +systems that make use of rsync and incremental backups, and will result in +considerably faster transfer speeds. #### Rsyncable -To make sure the generated archive is intelligently transferable by rsync, the `GZIP_RSYNCABLE=yes` option can be set. This will set the `--rsyncable` option to `gzip`. This is only useful in combination with setting [the Backup filename option](#backup-filename). +To ensure the generated archive is transferable by rsync, you can set the `GZIP_RSYNCABLE=yes` +option. This sets the `--rsyncable` option to `gzip`, which is useful only in +combination with setting [the Backup filename option](#backup-filename). -Note that the `--rsyncable` option in `gzip` is not guaranteed to be available on all distributions. To verify that it is available in your distribution you can run `gzip --help` or consult the man pages. +Note that the `--rsyncable` option in `gzip` isn't guaranteed to be available +on all distributions. To verify that it's available in your distribution, run +`gzip --help` or consult the man pages. ```shell sudo gitlab-backup create BACKUP=dump GZIP_RSYNCABLE=yes ``` -NOTE: **Note:** -For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. #### Excluding specific directories from the backup -You can choose what should be exempt from the backup up by adding the environment variable `SKIP`. -The available options are: +You can choose what should be exempt from the backup by adding the environment +variable `SKIP`. The available options are: - `db` (database) - `uploads` (attachments) @@ -252,8 +264,8 @@ The available options are: Use a comma to specify several options at the same time: -All wikis will be backed up as part of the `repositories` group. Non-existent wikis -will be skipped during a backup. +All wikis will be backed up as part of the `repositories` group. Non-existent +wikis will be skipped during a backup. For Omnibus GitLab packages: @@ -261,8 +273,7 @@ For Omnibus GitLab packages: sudo gitlab-backup create SKIP=db,uploads ``` -NOTE: **Note:** -For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. For installations from source: @@ -317,13 +328,15 @@ sudo -u git -H GITLAB_ASSUME_YES=1 bundle exec rake gitlab:backup:restore RAILS_ > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37158) in GitLab 13.3. -Repositories can be backed up concurrently to help fully utilise CPU time. The following variables -are available to modify the default behavior of the Rake task: +Repositories can be backed up concurrently to help fully utilize CPU time. The +following variables are available to modify the default behavior of the Rake +task: -- `GITLAB_BACKUP_MAX_CONCURRENCY` sets the maximum number of projects to backup at the same time. - Defaults to 1. -- `GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY` sets the maximum number of projects to backup at the same time on each storage. This allows the repository backups to be spread across storages. - Defaults to 1. +- `GITLAB_BACKUP_MAX_CONCURRENCY`: The maximum number of projects to back up at + the same time. Defaults to `1`. +- `GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY`: The maximum number of projects to + back up at the same time on each storage. This allows the repository backups + to be spread across storages. Defaults to `1`. For example, for Omnibus GitLab installations: @@ -339,12 +352,11 @@ sudo -u git -H bundle exec rake gitlab:backup:create GITLAB_BACKUP_MAX_CONCURREN #### Uploading backups to a remote (cloud) storage -Starting with GitLab 7.4 you can let the backup script upload the `.tar` file it creates. -It uses the [Fog library](http://fog.io/) to perform the upload. -In the example below we use Amazon S3 for storage, but Fog also lets you use -[other storage providers](http://fog.io/storage/). GitLab -[imports cloud drivers](https://gitlab.com/gitlab-org/gitlab/blob/30f5b9a5b711b46f1065baf755e413ceced5646b/Gemfile#L88) -for AWS, Google, OpenStack Swift, Rackspace, and Aliyun as well. A local driver is +You can let the backup script upload (using the [Fog library](http://fog.io/)) +the `.tar` file it creates. In the following example, we use Amazon S3 for +storage, but Fog also lets you use [other storage providers](http://fog.io/storage/). +GitLab also [imports cloud drivers](https://gitlab.com/gitlab-org/gitlab/blob/30f5b9a5b711b46f1065baf755e413ceced5646b/Gemfile#L88) +for AWS, Google, OpenStack Swift, Rackspace, and Aliyun. A local driver is [also available](#uploading-to-locally-mounted-shares). [Read more about using object storage with GitLab](../administration/object_storage.md). @@ -367,11 +379,12 @@ For Omnibus GitLab packages: gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' ``` -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect ##### Digital Ocean Spaces -This example can be used for a bucket in Amsterdam (AMS3). +This example can be used for a bucket in Amsterdam (AMS3): 1. Add the following to `/etc/gitlab/gitlab.rb`: @@ -386,20 +399,20 @@ This example can be used for a bucket in Amsterdam (AMS3). gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' ``` -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect -NOTE: **Note:** -If you see `400 Bad Request` by using Digital Ocean Spaces, the cause may be the -usage of backup encryption. Remove or comment the line that -contains `gitlab_rails['backup_encryption']` since Digital Ocean Spaces -doesn't support encryption. +If you see a `400 Bad Request` error message when using Digital Ocean Spaces, +the cause may be the use of backup encryption. Because Digital Ocean Spaces +doesn't support encryption, remove or comment the line that contains +`gitlab_rails['backup_encryption']`. ##### Other S3 Providers -Not all S3 providers are fully-compatible with the Fog library. For example, -if you see `411 Length Required` errors after attempting to upload, you may -need to downgrade the `aws_signature_version` value from the default value to -2 [due to this issue](https://github.com/fog/fog-aws/issues/428). +Not all S3 providers are fully compatible with the Fog library. For example, +if you see a `411 Length Required` error message after attempting to upload, +you may need to downgrade the `aws_signature_version` value from the default +value to `2`, [due to this issue](https://github.com/fog/fog-aws/issues/428). For installations from source: @@ -431,9 +444,10 @@ For installations from source: # storage_class: 'STANDARD' ``` -1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect -If you are uploading your backups to S3 you will probably want to create a new +If you're uploading your backups to S3, you'll probably want to create a new IAM user with restricted access rights. To give the upload user access only for uploading backups create the following IAM profile, replacing `my.s3.bucket` with the name of your bucket: @@ -486,16 +500,16 @@ with the name of your bucket: ##### Using Google Cloud Storage -If you want to use Google Cloud Storage to save backups, you'll have to create -an access key from the Google console first: +To use Google Cloud Storage to save backups, you must first create an +access key from the Google console: -1. Go to the storage settings page <https://console.cloud.google.com/storage/settings> -1. Select "Interoperability" and create an access key -1. Make note of the "Access Key" and "Secret" and replace them in the - configurations below -1. In the buckets advanced settings ensure the Access Control option "Set object-level - and bucket-level permissions" is selected -1. Make sure you already have a bucket created +1. Go to the [Google storage settings page](https://console.cloud.google.com/storage/settings). +1. Select **Interoperability**, and then create an access key. +1. Make note of the **Access Key** and **Secret** and replace them in the + following configurations. +1. In the buckets advanced settings ensure the Access Control option + **Set object-level and bucket-level permissions** is selected. +1. Ensure you have already created a bucket. For Omnibus GitLab packages: @@ -516,7 +530,8 @@ For Omnibus GitLab packages: gitlab_rails['backup_upload_remote_directory'] = 'my.google.bucket' ``` -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect For installations from source: @@ -532,7 +547,8 @@ For installations from source: remote_directory: 'my.google.bucket' ``` -1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect ##### Using Azure Blob storage @@ -552,7 +568,8 @@ For Omnibus GitLab packages: gitlab_rails['backup_upload_remote_directory'] = '<AZURE BLOB CONTAINER>' ``` -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect For installations from source: @@ -568,13 +585,14 @@ For installations from source: remote_directory: '<AZURE BLOB CONTAINER>' ``` -1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect -See [the table of Azure parameters](../administration/object_storage.md#azure-blob-storage) for more details. +For more details, see the [table of Azure parameters](../administration/object_storage.md#azure-blob-storage). ##### Specifying a custom directory for backups -Note: This option only works for remote storage. If you want to group your backups +This option works only for remote storage. If you want to group your backups, you can pass a `DIRECTORY` environment variable: ```shell @@ -582,26 +600,25 @@ sudo gitlab-backup create DIRECTORY=daily sudo gitlab-backup create DIRECTORY=weekly ``` -NOTE: **Note:** -For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. #### Uploading to locally mounted shares -You may also send backups to a mounted share (for example, `NFS`,`CIFS`, or `SMB`) by -using the Fog [`Local`](https://github.com/fog/fog-local#usage) storage provider. -The directory pointed to by the `local_root` key **must** be owned by the `git` -user **when mounted** (mounting with the `uid=` of the `git` user for `CIFS` and -`SMB`) or the user that you are executing the backup tasks under (for Omnibus -packages, this is the `git` user). +You may also send backups to a mounted share (for example, `NFS`,`CIFS`, or +`SMB`) by using the Fog [`Local`](https://github.com/fog/fog-local#usage) +storage provider. The directory pointed to by the `local_root` key _must_ be +owned by the `git` user _when mounted_ (mounting with the `uid=` of the `git` +user for `CIFS` and `SMB`) or the user that you are executing the backup tasks +as (for Omnibus packages, this is the `git` user). -The `backup_upload_remote_directory` **must** be set in addition to the +The `backup_upload_remote_directory` _must_ be set in addition to the `local_root` key. This is the sub directory inside the mounted directory that backups will be copied to, and will be created if it does not exist. If the directory that you want to copy the tarballs to is the root of your mounted -directory, just use `.` instead. +directory, use `.` instead. -NOTE: **Note:** -Since file system performance may affect GitLab's overall performance, we do not recommend using EFS for storage. See the [relevant documentation](../administration/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. +Because file system performance may affect GitLab's overall performance, +[GitLab doesn't recommend using EFS for storage](../administration/nfs.md#avoid-using-awss-elastic-file-system-efs). For Omnibus GitLab packages: @@ -618,7 +635,8 @@ For Omnibus GitLab packages: gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups' ``` -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. For installations from source: @@ -636,14 +654,16 @@ For installations from source: remote_directory: 'gitlab_backups' ``` -1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect. +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect. #### Backup archive permissions The backup archives created by GitLab (`1393513186_2014_02_27_gitlab_backup.tar`) -will have owner/group `git`/`git` and 0600 permissions by default. -This is meant to avoid other system users reading GitLab's data. -If you need the backup archives to have different permissions you can use the 'archive_permissions' setting. +will have owner/group `git`/`git` and 0600 permissions by default. This is +meant to avoid other system users reading GitLab's data. If you need the backup +archives to have different permissions, you can use the `archive_permissions` +setting. For Omnibus GitLab packages: @@ -653,7 +673,8 @@ For Omnibus GitLab packages: gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable ``` -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. For installations from source: @@ -664,7 +685,8 @@ For installations from source: archive_permissions: 0644 # Makes the backup archives world-readable ``` -1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect. +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect. #### Configuring cron to make daily backups @@ -689,8 +711,7 @@ For Omnibus GitLab packages: 0 2 * * * /opt/gitlab/bin/gitlab-backup create CRON=1 ``` - NOTE: **Note:** - For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. + Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. For installations from source: @@ -707,26 +728,25 @@ For installations from source: 0 2 * * * cd /home/git/gitlab && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1 ``` -NOTE: **Note:** -The `CRON=1` environment setting tells the backup script to suppress all progress output if there are no errors. -This is recommended to reduce cron spam. +The `CRON=1` environment setting directs the backup script to hide all progress +output if there aren't any errors. This is recommended to reduce cron spam. ### Limit backup lifetime for local files (prune old backups) CAUTION: **Warning:** -This will not work if you have used a [custom filename](#backup-filename) +The process described in this section will not work if you used a [custom filename](#backup-filename) for your backups. -NOTE: **Note:** -This configuration option only manages local files. GitLab does not automatically -prune old files stored in a third-party [object storage](#uploading-backups-to-a-remote-cloud-storage) -because the user may not have permission to list and delete files. It is +To prevent regular backups from using all your disk space, you may want to set a limited lifetime +for backups. The next time the backup task runs, backups older than the `backup_keep_time` are +pruned. + +This configuration option manages only local files. GitLab doesn't prune old +files stored in a third-party [object storage](#uploading-backups-to-a-remote-cloud-storage) +because the user may not have permission to list and delete files. It's recommended that you configure the appropriate retention policy for your object storage (for example, [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html)). -You may want to set a limited lifetime for backups to prevent regular -backups using all your disk space. The next time the backup task is run, backups older than the `backup_keep_time` will be pruned. - For Omnibus GitLab packages: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -736,7 +756,8 @@ For Omnibus GitLab packages: gitlab_rails['backup_keep_time'] = 604800 ``` -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. For installations from source: @@ -748,61 +769,64 @@ For installations from source: keep_time: 604800 ``` -1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect. +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect. ## Restore GitLab -GitLab provides a simple command line interface to restore your whole installation, +GitLab provides a command line interface to restore your entire installation, and is flexible enough to fit your needs. The [restore prerequisites section](#restore-prerequisites) includes crucial -information. Make sure to read and test the whole restore process at least once -before attempting to perform it in a production environment. +information. Be sure to read and test the complete restore process at least +once before attempting to perform it in a production environment. -You can only restore a backup to **exactly the same version and type (CE/EE)** of -GitLab that you created it on, for example CE 9.1.0. +You can restore a backup only to _the exact same version and type (CE/EE)_ of +GitLab that you created it on (for example CE 9.1.0). -If your backup is a different version than the current installation, you will +If your backup is a different version than the current installation, you'll need to [downgrade your GitLab installation](https://docs.gitlab.com/omnibus/update/README.html#downgrade) before restoring the backup. ### Restore prerequisites -You need to have a working GitLab installation before you can perform -a restore. This is mainly because the system user performing the -restore actions (`git`) is usually not allowed to create or delete -the SQL database it needs to import data into (`gitlabhq_production`). -All existing data will be either erased (SQL) or moved to a separate -directory (repositories, uploads). +You need to have a working GitLab installation before you can perform a +restore. This is because the system user performing the restore actions (`git`) +is usually not allowed to create or delete the SQL database needed to import +data into (`gitlabhq_production`). All existing data will be either erased +(SQL) or moved to a separate directory (such as repositories and uploads). -To restore a backup, you will also need to restore `/etc/gitlab/gitlab-secrets.json` -(for Omnibus packages) or `/home/git/gitlab/.secret` (for installations -from source). This file contains the database encryption key, +To restore a backup, you'll also need to restore `/etc/gitlab/gitlab-secrets.json` +(for Omnibus packages) or `/home/git/gitlab/.secret` (for installations from +source). This file contains the database encryption key, [CI/CD variables](../ci/variables/README.md#gitlab-cicd-environment-variables), and variables used for [two-factor authentication](../user/profile/account/two_factor_authentication.md). If you fail to restore this encryption key file along with the application data backup, users with two-factor authentication enabled and GitLab Runner will lose access to your GitLab server. -You may also want to restore any TLS keys, certificates, or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). +You may also want to restore any TLS keys, certificates, or +[SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). -Starting with GitLab 12.9 if an untarred backup (like the ones made with +Starting with GitLab 12.9, if an untarred backup (like the ones made with `SKIP=tar`) is found, and no backup is chosen with `BACKUP=<timestamp>`, the untarred backup is used. Depending on your case, you might want to run the restore command with one or more of the following options: -- `BACKUP=timestamp_of_backup` - Required if more than one backup exists. +- `BACKUP=timestamp_of_backup`: Required if more than one backup exists. Read what the [backup timestamp is about](#backup-timestamp). -- `force=yes` - Does not ask if the authorized_keys file should get regenerated and assumes 'yes' for warning that database tables will be removed, enabling the "Write to authorized_keys file" setting, and updating LDAP providers. +- `force=yes`: Doesn't ask if the authorized_keys file should get regenerated, + and assumes 'yes' for warning that database tables will be removed, + enabling the "Write to authorized_keys file" setting, and updating LDAP + providers. -If you are restoring into directories that are mount points, you will need to make -sure these directories are empty before attempting a restore. Otherwise GitLab -will attempt to move these directories before restoring the new data and this -would cause an error. +If you're restoring into directories that are mount points, you must ensure these directories are +empty before attempting a restore. Otherwise, GitLab attempts to move these directories before +restoring the new data, which causes an error. -Read more on [configuring NFS mounts](../administration/nfs.md) +Read more about [configuring NFS mounts](../administration/nfs.md) ### Restore for installation from source @@ -845,7 +869,7 @@ Restoring repositories: Deleting tmp directories...[DONE] ``` -Next, restore `/home/git/gitlab/.secret` if necessary as mentioned above. +Next, restore `/home/git/gitlab/.secret` if necessary, as previously mentioned. Restart GitLab: @@ -862,7 +886,7 @@ This procedure assumes that: - You have run `sudo gitlab-ctl reconfigure` at least once. - GitLab is running. If not, start it using `sudo gitlab-ctl start`. -First make sure your backup tar file is in the backup directory described in the +First ensure your backup tar file is in the backup directory described in the `gitlab.rb` configuration `gitlab_rails['backup_path']`. The default is `/var/opt/gitlab/backups`. It needs to be owned by the `git` user. @@ -890,15 +914,16 @@ restore: sudo gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce ``` -NOTE: **Note:** -For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:restore`. +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. CAUTION: **Warning:** -`gitlab-rake gitlab:backup:restore` does not set the right file system permissions on your Registry directory. -This is a [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759). On GitLab 12.2 or newer, you can -use `gitlab-backup restore` to avoid this issue. +`gitlab-rake gitlab:backup:restore` doesn't set the correct file system +permissions on your Registry directory. This is a [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759). +On GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this +issue. -Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary as mentioned above. +Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary, as previously +mentioned. Reconfigure, restart and check GitLab: @@ -908,29 +933,31 @@ sudo gitlab-ctl restart sudo gitlab-rake gitlab:check SANITIZE=true ``` -If there is a GitLab version mismatch between your backup tar file and the installed -version of GitLab, the restore command will abort with an error. Install the -[correct GitLab version](https://packages.gitlab.com/gitlab/) and try again. +If there's a GitLab version mismatch between your backup tar file and the +installed version of GitLab, the restore command aborts with an error +message. Install the [correct GitLab version](https://packages.gitlab.com/gitlab/), +and then try again. -NOTE: **Note:** -There is currently a [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3470) for restore not working -with `pgbouncer`. In order to workaround the issue, the Rails node will need to bypass `pgbouncer` and connect -directly to the primary database node. This can be done by setting `gitlab_rails['db_host']` and `gitlab_rails['port']` -to connect to the primary database node and [reconfiguring GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure). +There is a [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3470) +for restore not working with `pgbouncer`. To work around the issue, the Rails +node must bypass `pgbouncer` and connect directly to the primary +database node. You can do this by setting `gitlab_rails['db_host']` and +`gitlab_rails['port']` to connect to the primary database node and +[reconfiguring GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure). ### Restore for Docker image and GitLab Helm chart installations -For GitLab installations using the Docker image or the GitLab Helm chart on -a Kubernetes cluster, the restore task expects the restore directories to be empty. -However, with Docker and Kubernetes volume mounts, some system level directories -may be created at the volume roots, like `lost+found` directory found in Linux -operating systems. These directories are usually owned by `root`, which can -cause access permission errors since the restore Rake task runs as `git` user. -So, to restore a GitLab installation, users have to confirm the restore target -directories are empty. +For GitLab installations using the Docker image or the GitLab Helm chart on a +Kubernetes cluster, the restore task expects the restore directories to be +empty. However, with Docker and Kubernetes volume mounts, some system level +directories may be created at the volume roots, such as the `lost+found` +directory found in Linux operating systems. These directories are usually owned +by `root`, which can cause access permission errors since the restore Rake task +runs as the `git` user. To restore a GitLab installation, users have to confirm +the restore target directories are empty. -For both these installation types, the backup tarball has to be available in the -backup location (default location is `/var/opt/gitlab/backups`). +For both these installation types, the backup tarball has to be available in +the backup location (default location is `/var/opt/gitlab/backups`). For Docker installations, the restore task can be run from host: @@ -953,43 +980,43 @@ docker restart <name of container> docker exec -it <name of container> gitlab-rake gitlab:check SANITIZE=true ``` -NOTE: **Note:** -For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:restore`. +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. CAUTION: **Warning:** -`gitlab-rake gitlab:backup:restore` does not set the right file system permissions on your Registry directory. -This is a [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759). On GitLab 12.2 or newer, you can -use `gitlab-backup restore` to avoid this issue. +`gitlab-rake gitlab:backup:restore` doesn't set the correct file system +permissions on your Registry directory. This is a [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759). +On GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this +issue. The GitLab Helm chart uses a different process, documented in [restoring a GitLab Helm chart installation](https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/backup-restore/restore.md). ### Restoring only one or a few project(s) or group(s) from a backup -While the Rake task used to restore a GitLab instance doesn't support -restoring a single project or group, you can use a workaround by -restoring your backup to a separate, temporary GitLab instance, and -export your project or group from there: +Although the Rake task used to restore a GitLab instance doesn't support +restoring a single project or group, you can use a workaround by restoring +your backup to a separate, temporary GitLab instance, and then export your +project or group from there: 1. [Install a new GitLab](../install/README.md) instance at the same version as the backed-up instance from which you want to restore. -1. [Restore the backup](#restore-gitlab) into this new instance and +1. [Restore the backup](#restore-gitlab) into this new instance, then export your [project](../user/project/settings/import_export.md) - or [group](../user/group/settings/import_export.md). Make sure to - read the **Important Notes** on either export feature's documentation - to understand what will be exported and what not. -1. Once the export is complete, go to the old instance and import it. -1. After importing only the project(s) or group(s) that you wanted is complete, - you may delete the new, temporary GitLab instance. - -NOTE: **Note:** + or [group](../user/group/settings/import_export.md). Be sure to read the + **Important Notes** on either export feature's documentation to understand + what is and isn't exported. +1. After the export is complete, go to the old instance and then import it. +1. After importing the projects or groups that you wanted is complete, you may + delete the new, temporary GitLab instance. + A feature request to provide direct restore of individual projects or groups is being discussed in [issue #17517](https://gitlab.com/gitlab-org/gitlab/-/issues/17517). ## Alternative backup strategies -If your GitLab server contains a lot of Git repository data you may find the GitLab backup script to be too slow. -In this case you can consider using filesystem snapshots as part of your backup strategy. +If your GitLab server contains a lot of Git repository data, you may find the +GitLab backup script to be too slow. In this case you can consider using +filesystem snapshots as part of your backup strategy. Example: Amazon EBS @@ -1006,29 +1033,33 @@ Example: LVM snapshots + rsync > Now we can have a longer running rsync job which will create a consistent replica on the remote server. > The replica includes all repositories, uploads and PostgreSQL data. -If you are running GitLab on a virtualized server you can possibly also create VM snapshots of the entire GitLab server. -It is not uncommon however for a VM snapshot to require you to power down the server, so this approach is probably of limited practical use. +If you're running GitLab on a virtualized server, you can possibly also create +VM snapshots of the entire GitLab server. It's not uncommon however for a VM +snapshot to require you to power down the server, which limits this solution's +practical use. ## Additional notes -This documentation is for GitLab Community and Enterprise Edition. We backup -GitLab.com and make sure your data is secure, but you can't use these methods -to export / backup your data yourself from GitLab.com. +This documentation is for GitLab Community and Enterprise Edition. We back up +GitLab.com and ensure your data is secure. You can't, however, use these +methods to export or back up your data yourself from GitLab.com. -Issues are stored in the database. They can't be stored in Git itself. +Issues are stored in the database, and can't be stored in Git itself. -To migrate your repositories from one server to another with an up-to-date version of -GitLab, you can use the [import Rake task](import.md) to do a mass import of the -repository. Note that if you do an import Rake task, rather than a backup restore, you -will have all your repositories, but not any other data. +To migrate your repositories from one server to another with an up-to-date +version of GitLab, use the [import Rake task](import.md) to do a mass import of +the repository. If you do an import Rake task rather than a backup restore, +you get all of your repositories, but no other data. ## Troubleshooting -The following are possible problems you might encounter with possible solutions. +The following are possible problems you might encounter, along with potential +solutions. ### Restoring database backup using Omnibus packages outputs warnings -If you are using backup restore procedures you might encounter the following warnings: +If you're using backup restore procedures, you may encounter the following +warning messages: ```plaintext psql:/var/opt/gitlab/backups/db/database.sql:22: ERROR: must be owner of extension plpgsql @@ -1036,22 +1067,31 @@ psql:/var/opt/gitlab/backups/db/database.sql:2931: WARNING: no privileges could psql:/var/opt/gitlab/backups/db/database.sql:2933: WARNING: no privileges were granted for "public" (two occurrences) ``` -Be advised that, backup is successfully restored in spite of these warnings. +Be advised that the backup is successfully restored in spite of these warning +messages. -The Rake task runs this as the `gitlab` user which does not have the superuser access to the database. When restore is initiated it will also run as `gitlab` user but it will also try to alter the objects it does not have access to. -Those objects have no influence on the database backup/restore but they give this annoying warning. +The Rake task runs this as the `gitlab` user, which doesn't have superuser +access to the database. When restore is initiated, it also runs as the `gitlab` +user, but it also tries to alter the objects it doesn't have access to. +Those objects have no influence on the database backup or restore, but display +a warning message. -For more information see similar questions on PostgreSQL issue tracker [here](https://www.postgresql.org/message-id/201110220712.30886.adrian.klaver@gmail.com) and [here](https://www.postgresql.org/message-id/2039.1177339749@sss.pgh.pa.us) as well as [stack overflow](https://stackoverflow.com/questions/4368789/error-must-be-owner-of-language-plpgsql). +For more information, see: + +- PostgreSQL issue tracker: + - [Not being a superuser](https://www.postgresql.org/message-id/201110220712.30886.adrian.klaver@gmail.com). + - [Having different owners](https://www.postgresql.org/message-id/2039.1177339749@sss.pgh.pa.us). + +- Stack Overflow: [Resulting errors](https://stackoverflow.com/questions/4368789/error-must-be-owner-of-language-plpgsql). ### When the secrets file is lost -If you have failed to [back up the secrets file](#storing-configuration-files), you'll -need to perform a number of steps to get GitLab working properly again. +If you didn't [back up the secrets file](#storing-configuration-files), you +must complete several steps to get GitLab working properly again. -The secrets file is responsible for storing the encryption key for several -columns containing sensitive information. If the key is lost, GitLab will be -unable to decrypt those columns. This will break a wide range of functionality, -including (but not restricted to): +The secrets file is responsible for storing the encryption key for the columns +that contain required, sensitive information. If the key is lost, GitLab can't +decrypt those columns, preventing access to the following items: - [CI/CD variables](../ci/variables/README.md) - [Kubernetes / GCP integration](../user/project/clusters/index.md) @@ -1061,38 +1101,40 @@ including (but not restricted to): - [Project mirroring](../user/project/repository/repository_mirroring.md) - [Web hooks](../user/project/integrations/webhooks.md) -In cases like CI/CD variables and runner authentication, you might -experience some unexpected behavior such as: +In cases like CI/CD variables and runner authentication, you can experience +unexpected behaviors, such as: - Stuck jobs. - 500 errors. -In this case, you are required to reset all the tokens for CI/CD variables -and runner authentication, which is described in more detail below. After -resetting the tokens, you should be able to visit your project and the jobs -will have started running again. Use the information in the following sections at your own risk. +In this case, you must reset all the tokens for CI/CD variables and +runner authentication, which is described in more detail in the following +sections. After resetting the tokens, you should be able to visit your project +and the jobs will have started running again. + +Use the information in the following sections at your own risk. #### Check for undecryptable values -You can check whether you have undecryptable values in the database using -the [Secrets Doctor Rake task](../administration/raketasks/doctor.md). +You can determine if you have undecryptable values in the database by using the +[Secrets Doctor Rake task](../administration/raketasks/doctor.md). #### Take a backup -You will need to directly modify GitLab data to work around your lost secrets file. +You must directly modify GitLab data to work around your lost secrets file. CAUTION: **Warning:** -Make sure you've taken a backup beforehand, particularly a full database backup. +Be sure to create a full database backup before attempting any changes. #### Disable user two-factor authentication (2FA) -Users with 2FA enabled will not be able to log into GitLab. In that case, -you need to [disable 2FA for everyone](../security/two_factor_authentication.md#disabling-2fa-for-everyone) -and then users will have to reactivate 2FA from scratch. +Users with 2FA enabled can't sign in to GitLab. In that case, you must +[disable 2FA for everyone](../security/two_factor_authentication.md#disabling-2fa-for-everyone), +after which users must reactivate 2FA. #### Reset CI/CD variables -1. Enter the DB console: +1. Enter the database console: For Omnibus GitLab packages: @@ -1106,14 +1148,14 @@ and then users will have to reactivate 2FA from scratch. sudo -u git -H bundle exec rails dbconsole -e production ``` -1. Check the `ci_group_variables` and `ci_variables` tables: +1. Examine the `ci_group_variables` and `ci_variables` tables: ```sql SELECT * FROM public."ci_group_variables"; SELECT * FROM public."ci_variables"; ``` - Those are the variables that you need to delete. + These are the variables that you need to delete. 1. Drop the table: @@ -1122,12 +1164,11 @@ and then users will have to reactivate 2FA from scratch. DELETE FROM ci_variables; ``` -1. You may need to reconfigure or restart GitLab for the changes to take - effect. +You may need to reconfigure or restart GitLab for the changes to take effect. #### Reset runner registration tokens -1. Enter the DB console: +1. Enter the database console: For Omnibus GitLab packages: @@ -1141,11 +1182,11 @@ and then users will have to reactivate 2FA from scratch. sudo -u git -H bundle exec rails dbconsole -e production ``` -1. Clear all the tokens for projects, groups, and the whole instance: +1. Clear all tokens for projects, groups, and the entire instance: CAUTION: **Caution:** - The last UPDATE operation will stop the runners being able to pick up - new jobs. You must register new runners. + The final `UPDATE` operation stops the runners from being able to pick + up new jobs. You must register new runners. ```sql -- Clear project tokens @@ -1160,7 +1201,7 @@ and then users will have to reactivate 2FA from scratch. #### Reset pending pipeline jobs -1. Enter the DB console: +1. Enter the database console: For Omnibus GitLab packages: @@ -1181,19 +1222,18 @@ and then users will have to reactivate 2FA from scratch. UPDATE ci_builds SET token = null, token_encrypted = null; ``` -A similar strategy can be employed for the remaining features - by removing the -data that cannot be decrypted, GitLab can be brought back into working order, -and the lost data can be manually replaced. +A similar strategy can be employed for the remaining features. By removing the +data that can't be decrypted, GitLab can be returned to operation, and the +lost data can be manually replaced. #### Fix project integrations -If you've lost your secrets, the -[projects' integrations settings pages](../user/project/integrations/index.md) -are probably generating 500 errors. +If you've lost your secrets, the [projects' integrations settings pages](../user/project/integrations/index.md) +are probably displaying `500` error messages. The fix is to truncate the `web_hooks` table: -1. Enter the DB console: +1. Enter the database console: For Omnibus GitLab packages: @@ -1207,7 +1247,7 @@ The fix is to truncate the `web_hooks` table: sudo -u git -H bundle exec rails dbconsole -e production ``` -1. Truncate the table +1. Truncate the table: ```sql -- truncate web_hooks table @@ -1216,11 +1256,11 @@ The fix is to truncate the `web_hooks` table: ### Container Registry push failures after restoring from a backup -If you use the [Container Registry](../user/packages/container_registry/index.md), you -may see pushes to the registry fail after restoring your backup on an Omnibus -GitLab instance after restoring the registry data. +If you use the [Container Registry](../user/packages/container_registry/index.md), +pushes to the registry may fail after restoring your backup on an Omnibus GitLab +instance after restoring the registry data. -These failures will mention permission issues in the registry logs, like: +These failures mention permission issues in the registry logs, similar to: ```plaintext level=error @@ -1230,9 +1270,9 @@ err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docke err.message="unknown error" ``` -This is caused by the restore being run as the unprivileged user `git` which was -unable to assign the correct ownership to the registry files during the restore -([issue 62759](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759 "Incorrect permissions on registry filesystem after restore")). +This issue is caused by the restore running as the unprivileged user `git`, +which is unable to assign the correct ownership to the registry files during +the restore process ([issue 62759](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759 "Incorrect permissions on registry filesystem after restore")). To get your registry working again: @@ -1240,14 +1280,12 @@ To get your registry working again: sudo chown -R registry:registry /var/opt/gitlab/gitlab-rails/shared/registry/docker ``` -NOTE: **Note:** -If you have changed the default filesystem location for the registry, you will -want to run the `chown` against your custom location instead of -`/var/opt/gitlab/gitlab-rails/shared/registry/docker`. +If you changed the default filesystem location for the registry, run `chown` +against your custom location, instead of `/var/opt/gitlab/gitlab-rails/shared/registry/docker`. ### Backup fails to complete with Gzip error -While running the backup, you may receive a Gzip error: +When running the backup, you may receive a Gzip error message: ```shell sudo /opt/gitlab/bin/gitlab-backup create @@ -1259,7 +1297,8 @@ gzip: stdout: Input/output error Backup failed ``` -If this happens, check the following: +If this happens, examine the following: -1. Confirm there is sufficient disk space for the Gzip operation. -1. If NFS is being used, check if the mount option `timeout` is set. The default is `600`, and changing this to smaller values have resulted in this error. +- Confirm there is sufficient disk space for the Gzip operation. +- If NFS is being used, check if the mount option `timeout` is set. The + default is `600`, and changing this to smaller values results in this error. diff --git a/doc/security/rack_attack.md b/doc/security/rack_attack.md index d3de2222c39..b386917f399 100644 --- a/doc/security/rack_attack.md +++ b/doc/security/rack_attack.md @@ -4,8 +4,6 @@ type: reference, howto # Rack Attack initializer -## Overview - [Rack Attack](https://github.com/kickstarter/rack-attack), also known as Rack::Attack, is a Ruby gem that is meant to protect GitLab with the ability to customize throttling and to block user IP addresses. diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md index 9d49e1d3af2..0c2d616b003 100644 --- a/doc/security/two_factor_authentication.md +++ b/doc/security/two_factor_authentication.md @@ -68,6 +68,17 @@ The following are important notes about 2FA: ## Disabling 2FA for everyone +CAUTION: **Caution:** +Disabling 2FA for everyone does not disable the [enforce 2FA for all users](#enforcing-2fa-for-all-users) +or [enforce 2FA for all users in a group](#enforcing-2fa-for-all-users-in-a-group) +settings. In addition to the steps in this section, you will need to disable any enforced 2FA +settings so users aren't asked to set up 2FA again, the next time the user signs in to GitLab. +Disabling 2FA for everyone does not disable the [enforce 2FA for all users](#enforcing-2fa-for-all-users) +or [enforce 2FA for all users in a group](#enforcing-2fa-for-all-users-in-a-group) +settings if they have been configured. In addition to the steps in this section, +you will need to disable any enforced 2FA settings so users aren't asked to setup +2FA again when the next login to GitLab. + There may be some special situations where you want to disable 2FA for everyone even when forced 2FA is disabled. There is a Rake task for that: diff --git a/doc/ssh/README.md b/doc/ssh/README.md index 1c5654ae96c..6a4dc5426f8 100644 --- a/doc/ssh/README.md +++ b/doc/ssh/README.md @@ -24,7 +24,7 @@ connections to GitLab repositories. ## Requirements To support SSH, GitLab requires the installation of the OpenSSH client, which -comes pre-installed on GNU/Linux and macOS, but not on Windows. +comes pre-installed on GNU/Linux and macOS, as well as on Windows 10. Make sure that your system includes SSH version 6.5 or newer, as that excludes the now insecure MD5 signature scheme. The following command returns the version of diff --git a/doc/topics/autodevops/customize.md b/doc/topics/autodevops/customize.md index 13aa8f7e035..45083dfeb64 100644 --- a/doc/topics/autodevops/customize.md +++ b/doc/topics/autodevops/customize.md @@ -332,7 +332,7 @@ applications. | `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME` | From GitLab 11.11, used to set a username to connect to the Helm repository. Defaults to no credentials. Also set `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD`. | | `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD` | From GitLab 11.11, used to set a password to connect to the Helm repository. Defaults to no credentials. Also set `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME`. | | `AUTO_DEVOPS_DEPLOY_DEBUG` | From GitLab 13.1, if this variable is present, Helm will output debug logs. | -| `AUTO_DEVOPS_ALLOW_TO_FORCE_DEPLOY_V<N>` | From [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) v1.0.0, if this variable is present, a new major version of chart is forcibly deployed. [More details](upgrading_chart.md#ignore-warning-and-continue-deploying) | +| `AUTO_DEVOPS_ALLOW_TO_FORCE_DEPLOY_V<N>` | From [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) v1.0.0, if this variable is present, a new major version of chart is forcibly deployed. For more information, see [Ignore warnings and continue deploying](upgrading_auto_deploy_dependencies.md#ignore-warnings-and-continue-deploying). | | `AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE` | From GitLab 12.5, used in combination with [ModSecurity feature flag](../../user/clusters/applications.md#web-application-firewall-modsecurity) to toggle [ModSecurity's `SecRuleEngine`](https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#SecRuleEngine) behavior. Defaults to `DetectionOnly`. | | `BUILDPACK_URL` | Buildpack's full URL. Can point to either [a Git repository URL or a tarball URL](#custom-buildpacks). | | `CANARY_ENABLED` | From GitLab 11.0, used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments). | diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index a39f93a26e1..a7bd810abbe 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -3,10 +3,11 @@ > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37115) in GitLab 10.0. > - Generally available on GitLab 11.0. -Auto DevOps provides pre-defined CI/CD configuration allowing you to automatically -detect, build, test, deploy, and monitor your applications. Leveraging CI/CD -best practices and tools, Auto DevOps aims to simplify the setup and execution -of a mature and modern software development lifecycle. +Auto DevOps are default CI/CD templates that auto-discover the source code you have. They +enable GitLab to automatically detect, build, test, deploy, and monitor your applications. +Leveraging [CI/CD best practices](../../ci/pipelines/pipeline_efficiency.md) and tools, +Auto DevOps aims to simplify the setup and execution of a mature and modern software +development lifecycle. ## Overview @@ -83,9 +84,9 @@ project in a simple and automatic way: 1. [Auto Build](stages.md#auto-build) 1. [Auto Test](stages.md#auto-test) -1. [Auto Code Quality](stages.md#auto-code-quality) **(STARTER)** -1. [Auto SAST (Static Application Security Testing)](stages.md#auto-sast) **(ULTIMATE)** -1. [Auto Secret Detection](stages.md#auto-secret-detection) **(ULTIMATE)** +1. [Auto Code Quality](stages.md#auto-code-quality) +1. [Auto SAST (Static Application Security Testing)](stages.md#auto-sast) +1. [Auto Secret Detection](stages.md#auto-secret-detection) 1. [Auto Dependency Scanning](stages.md#auto-dependency-scanning) **(ULTIMATE)** 1. [Auto License Compliance](stages.md#auto-license-compliance) **(ULTIMATE)** 1. [Auto Container Scanning](stages.md#auto-container-scanning) **(ULTIMATE)** @@ -317,7 +318,7 @@ metadata: name: gitlab-managed-apps-default-proxy namespace: gitlab-managed-apps spec: - env: + env: - name: http_proxy value: "PUT_YOUR_HTTP_PROXY_HERE" - name: https_proxy diff --git a/doc/topics/autodevops/stages.md b/doc/topics/autodevops/stages.md index b58c369714e..44eebb748a6 100644 --- a/doc/topics/autodevops/stages.md +++ b/doc/topics/autodevops/stages.md @@ -124,7 +124,10 @@ The supported buildpacks are: If your application needs a buildpack that is not in the above list, you might want to use a [custom buildpack](customize.md#custom-buildpacks). -## Auto Code Quality **(STARTER)** +## Auto Code Quality + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1984) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.3. +> - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) in GitLab 13.2. Auto Code Quality uses the [Code Quality image](https://gitlab.com/gitlab-org/ci-cd/codequality) to run @@ -133,9 +136,10 @@ report, it's uploaded as an artifact which you can later download and check out. The merge request widget also displays any [differences between the source and target branches](../../user/project/merge_requests/code_quality.md). -## Auto SAST **(ULTIMATE)** +## Auto SAST -> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.3. +> - Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.3. +> - Select functionality made available in all tiers beginning in 13.1 Static Application Security Testing (SAST) uses the [SAST Docker image](https://gitlab.com/gitlab-org/security-products/sast) to run static @@ -151,9 +155,10 @@ warnings. To learn more about [how SAST works](../../user/application_security/sast/index.md), see the documentation. -## Auto Secret Detection **(ULTIMATE)** +## Auto Secret Detection -> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.1. +> - Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.1. +> - [Select functionality made available in all tiers](../../user/application_security/secret_detection/#making-secret-detection-available-to-all-gitlab-tiers) in 13.3 Secret Detection uses the [Secret Detection Docker image](https://gitlab.com/gitlab-org/security-products/analyzers/secrets) to run Secret Detection on the current code, and checks for leaked secrets. The @@ -466,7 +471,7 @@ application runs. ### Upgrade auto-deploy-app Chart -You can upgrade auto-deploy-app chart by following the [upgrade guide](upgrading_chart.md). +You can upgrade the auto-deploy-app chart by following the [upgrade guide](upgrading_auto_deploy_dependencies.md). ### Workers diff --git a/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md b/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md new file mode 100644 index 00000000000..1aefb6b34df --- /dev/null +++ b/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md @@ -0,0 +1,262 @@ +--- +stage: Release +group: Progressive Delivery +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference +--- + +# Upgrading deployments for newer Auto Deploy dependencies (Auto Deploy template, auto-deploy-image and auto-deploy-app chart) + +[Auto Deploy](stages.md#auto-deploy) is a feature that deploys your application to a Kubernetes cluster. +It consists of several dependencies: + +- [Auto Deploy template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) is a set of pipeline jobs and scripts that makes use of `auto-deploy-image`. +- [`auto-deploy-image`](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) is the executable image that communicates with the Kubernetes cluster. +- [`auto-deploy-app chart`](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/tree/master/assets/auto-deploy-app) is the Helm chart for deploying your application. + +The `auto-deploy-image` and `auto-deploy-app` charts use [Semantic Versioning](https://semver.org/). +By default, your Auto DevOps project keeps using the stable and non-breaking version. +However, these dependencies could be upgraded in a major version release of GitLab +with breaking changes requiring you to upgrade your deployments. + +This guide explains how to upgrade your deployments with newer or different major versions of Auto Deploy dependencies. + +## Verify dependency versions + +The process to check the current versions differs depending on which template you +are using. First verify which template is in use: + +- For self-managed instances, the [stable Auto Deploy template bundled with the GitLab package](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) + is being used. +- [The GitLab.com stable Auto Deploy template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) + is being used if **one** of the following is true: + - Your Auto DevOps project doesn't have a `.gitlab-ci.yml` file. + - Your Auto DevOps project has a `.gitlab-ci.yml` and [includes](../../ci/yaml/README.md#includetemplate) + the `Auto-DevOps.gitlab-ci.yml` template. +- [The latest Auto Deploy template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.latest.gitlab-ci.yml) + is being used if **both** of the following is true: + - Your Auto DevOps project has a `.gitlab-ci.yml` file and [includes](../../ci/yaml/README.md#includetemplate) + the `Auto-DevOps.gitlab-ci.yml` template. + - It also includes [the latest Auto Deploy template](#early-adopters) + +If you know what template is being used: + +- The `auto-deploy-image` version is in the template (for example `auto-deploy-image:v1.0.3`). +- The `auto-deploy-app` chart version is [in the auto-deploy-image repository](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/blob/v1.0.3/assets/auto-deploy-app/Chart.yaml) + (for example `version: 1.0.3`). + +## Compatibility + +The following table explains the version compatibility between GitLab and Auto Deploy dependencies: + +| GitLab version | `auto-deploy-image` version | Notes | +|------------------|-----------------------------|-------| +| v10.0 to v14.0 | v0.1.0 to v2.0.0 | v0 and v1 auto-deploy-image are backwards compatible. | +| v13.4 and higher | v2.0.0 and higher | v2 auto-deploy-image contains breaking changes, as explained in the [upgrade guide](#upgrade-deployments-to-the-v2-auto-deploy-image). | + +You can find the current stable version of auto-deploy-image in the [Auto Deploy stable template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). + +## Upgrade Guide + +Projects using Auto DevOps must use the unmodified chart managed by GitLab. +[Customized charts](customize.md#custom-helm-chart) are unsupported. + +### Upgrade deployments to the v1 `auto-deploy-image` + +The v1 chart is backward compatible with the v0 chart, so no configuration changes are needed. + +### Upgrade deployments to the v2 `auto-deploy-image` + +The v2 auto-deploy-image contains multiple dependency and architectural changes. +If your Auto DevOps project has an active environment deployed with the v1 `auto-deploy-image`, +please proceed with the following upgrade guide. Otherwise, you can skip this process. + +#### Helm 3 + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/228609) in GitLab 13.4. + +The `auto-deploy-image` uses the Helm binary to manipulate the releases. +Previously, `auto-deploy-image` used Helm v2, which used Tiller in a cluster. +In the v2 `auto-deploy-image`, it uses Helm v3 that doesn't require Tiller anymore. + +If your Auto DevOps project has an active environment that was deployed with the v1 +`auto-deploy-image`, use the following steps to upgrade to v2, which uses Helm 3: + +1. Modify your `.gitlab-ci.yml` with: + + ```yaml + include: + - template: Auto-DevOps.gitlab-ci.yml + - remote: https://gitlab.com/hfyngvason/ci-templates/-/raw/master/Helm-2to3.gitlab-ci.yml + + variables: + # If this variable is not present, the migration jobs will not show up + MIGRATE_HELM_2TO3: "true" + + .auto-deploy: + image: registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v2.0.0-beta.1 + variables: + AUTO_DEVOPS_FORCE_DEPLOY_V2: 1 + ``` + +1. Run the `<environment-name>:helm-2to3:migrate` job. +1. Deploy your environment as usual. This deployment uses Helm 3. +1. If the deployment succeeds, you can safely run `environment:helm-2to3:cleanup`. + This deletes all Helm 2 release data from the namespace. + + If you accidentally delete the Helm 2 releases before you are ready, the `<environment-name>:helm2to3:migrate` + job saves a backup for 1 week in a job artifact called `helm-2-release-backups`. + The backup is in a Kubernetes manifest file that can be restored using + `kubectl apply -f $backup`. +1. Remove the `MIGRATE_HELM_2TO3` variable. + +#### Traffic routing change for canary deployments and incremental rollouts + +> [Introduced](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/merge_requests/109) in GitLab 13.4. + +Auto Deploy supports advanced deployment strategies such as [canary deployments](customize.md#deploy-policy-for-canary-environments) +and [incremental rollouts](../../ci/environments/incremental_rollouts.md). + +Previously, `auto-deploy-image` created one service to balance the traffic between +unstable and stable tracks by changing the replica ratio. In the v2 `auto-deploy-image`, +it controls the traffic with [Canary Ingress](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary). + +For more details, see the [v2 `auto-deploy-app` chart resource architecture](#v2-chart-resource-architecture). + +If your Auto DevOps project has active `canary` or `rollout` track releases in the +`production` environment deployed with the v1 `auto-deploy-image`, use the following +steps to upgrade to v2: + +1. Verify your project is [using the v1 `auto-deploy-image`](#verify-dependency-versions). + If not, [specify the version](#use-a-specific-version-of-auto-deploy-dependencies). +1. If you're in the process of deploying `canary` or `rollout` deployments, promote + them to `production` first to delete the unstable tracks. +1. Verify your project is [using the v2 `auto-deploy-image`](#verify-dependency-versions). + If not, [specify the version](#use-a-specific-version-of-auto-deploy-dependencies). +1. Add an `AUTO_DEVOPS_FORCE_DEPLOY_V2` environment variable with a value of `true` + in the GitLab CI/CD settings. +1. Create a new pipeline and run the `production` job to renew the resource architecture + with the v2 `auto-deploy-app chart`. +1. Remove the `AUTO_DEVOPS_FORCE_DEPLOY_V2` environment variable. + +### Use a specific version of Auto Deploy dependencies + +To use a specifc version of Auto Deploy dependencies, specify the previous Auto Deploy +stable template that contains the [desired version of `auto-deploy-image` and `auto-deploy-app`](#verify-dependency-versions). + +For example, if the template is bundled in GitLab v13.3, change your `.gitlab-ci.yml` to: + +```yaml +include: + - template: Auto-DevOps.gitlab-ci.yml + - remote: https://gitlab.com/gitlab-org/gitlab/-/blob/v13.3.0-ee/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml +``` + +### Ignore warnings and continue deploying + +If you are certain that the new chart version is safe to be deployed, you can add +the `AUTO_DEVOPS_FORCE_DEPLOY_V<major-version-number>` [environment variable](customize.md#build-and-deployment) +to force the deployment to continue. + +For example, if you want to deploy the `v2.0.0` chart on a deployment that previously +used the `v0.17.0` chart, add `AUTO_DEVOPS_FORCE_DEPLOY_V2`. + +## Early adopters + +If you want to use the latest beta or unstable version of `auto-deploy-image`, include +the latest Auto Deploy template into your `.gitlab-ci.yml`: + +```yaml +include: + - template: Auto-DevOps.gitlab-ci.yml + - remote: https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.latest.gitlab-ci.yml +``` + +CAUTION: **Warning:** +Using a beta or unstable `auto-deploy-image` could cause unrecoverable damage to +your environments. Do not test it with important projects or environments. + +The next stable template update is [planned for GitLab v14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/232788). + +## Resource Architectures of the `auto-deploy-app` chart + +### v0 and v1 chart resource architecture + +```mermaid +graph TD; +subgraph gl-managed-app +Z[Nginx Ingress] +end +Z[Nginx Ingress] --> A(Ingress); +Z[Nginx Ingress] --> B(Ingress); +subgraph stg namespace +B[Ingress] --> H(...); +end +subgraph prd namespace +A[Ingress] --> D(Service); +D[Service] --> E(Deployment:Pods:app:stable); +D[Service] --> F(Deployment:Pods:app:canary); +D[Service] --> I(Deployment:Pods:app:rollout); +E(Deployment:Pods:app:stable)---id1[(Pods:Postgres)] +F(Deployment:Pods:app:canary)---id1[(Pods:Postgres)] +I(Deployment:Pods:app:rollout)---id1[(Pods:Postgres)] +end +``` + +### v2 chart resource architecture + +```mermaid +graph TD; +subgraph gl-managed-app +Z[Nginx Ingress] +end +Z[Nginx Ingress] --> A(Ingress); +Z[Nginx Ingress] --> B(Ingress); +Z[Nginx Ingress] --> |If canary is present or incremental rollout/|J(Canary Ingress); +subgraph stg namespace +B[Ingress] --> H(...); +end +subgraph prd namespace +subgraph stable track +A[Ingress] --> D[Service]; +D[Service] --> E(Deployment:Pods:app:stable); +end +subgraph canary track +J(Canary Ingress) --> K[Service] +K[Service] --> F(Deployment:Pods:app:canary); +end +E(Deployment:Pods:app:stable)---id1[(Pods:Postgres)] +F(Deployment:Pods:app:canary)---id1[(Pods:Postgres)] +end +``` + +## Troubleshooting + +### Major version mismatch warning + +If deploying a chart that has a major version that is different from the previous one, +the new chart might not be correctly deployed. This could be due to an architectural +change. If that happens, the deployment job fails with a message similar to: + +```plaintext +************************************************************************************* + [WARNING] +Detected a major version difference between the the chart that is currently deploying (auto-deploy-app-v0.7.0), and the previously deployed chart (auto-deploy-app-v1.0.0). +A new major version might not be backward compatible with the current release (production). The deployment could fail or be stuck in an unrecoverable status. +... +``` + +To clear this error message and resume deployments, you must do one of the following: + +- Manually [upgrade the chart version](#upgrade-guide). +- [Use a specific chart version](#use-a-specific-version-of-auto-deploy-dependencies). + +### Error: `missing key "app.kubernetes.io/managed-by": must be set to "Helm"` + +If your cluster has a deployment that was deployed with the v1 `auto-deploy-image`, +you might encounter the following error: + +- `Error: rendered manifests contain a resource that already exists. Unable to continue with install: Secret "production-postgresql" in namespace "<project-name>-production" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "production-postgresql"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "<project-name>-production"` + +This is because the previous deployment was deployed with Helm2, which is not compatible with Helm3. +To resolve the problem, please follow the [upgrade guide](#upgrade-deployments-to-the-v2-auto-deploy-image). diff --git a/doc/topics/autodevops/upgrading_chart.md b/doc/topics/autodevops/upgrading_chart.md index ffa485f6d2c..e4fb84d4509 100644 --- a/doc/topics/autodevops/upgrading_chart.md +++ b/doc/topics/autodevops/upgrading_chart.md @@ -1,72 +1,5 @@ -# Upgrading auto-deploy-app chart for Auto DevOps +--- +redirect_to: 'upgrading_auto_deploy_dependencies.md' +--- -Auto DevOps provides the auto-deploy-app chart for deploying your application to the -Kubernetes cluster with Helm/Tiller. Major version changes of this chart could have -a significantly different resource architecture, and may not be backwards compatible. - -This guide provides instructions on how to upgrade your deployments to use the latest -chart and resource architecture. - -## Compatibility - -The following table lists the version compatibility between GitLab and [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) (with the [auto-deploy-app chart](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/tree/master/assets/auto-deploy-app)). - -| GitLab version | auto-deploy-image version | Notes | -|------------------|---------------------------|--------------------------------------------| -| v10.0 and higher | v0.1.0 and higher | v0 and v1 charts are backwards compatible. | - -## Upgrade Guide - -The Auto DevOps project must use the unmodified chart managed by GitLab. -[Customized charts](customize.md#custom-helm-chart) are unsupported. - -### v1 chart - -The v1 chart is backward compatible with the v0 chart, so no configuration changes are needed. - -## Troubleshooting - -### Major version mismatch warning - -If deploying a chart that has a major version that is different from the previous one, -the new chart might not be correctly deployed. This could be due to an architectural -change. If that happens, the deployment job fails with a message similar to: - -```plaintext -************************************************************************************* - [WARNING] -Detected a major version difference between the the chart that is currently deploying (auto-deploy-app-v0.7.0), and the previously deployed chart (auto-deploy-app-v1.0.0). -A new major version might not be backward compatible with the current release (production). The deployment could fail or be stuck in an unrecoverable status. -... -``` - -To clear this error message and resume deployments, you must do one of the following: - -- Manually [upgrade the chart version](#upgrade-guide). -- [Use a specific chart version](#use-a-specific-chart-version). - -#### Use a specific chart version - -To use a specific chart version, you must specify a corresponding version of [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image). -Do this by [customizing the image in your `.gitlab-ci.yml`](customize.md#customizing-gitlab-ciyml). - -For example, create the following `.gitlab-ci.yml` file in the project. It configures Auto DevOps -to use [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) version `v0.17.0` -for deployment jobs. It will download the chart from [chart repository](https://charts.gitlab.io/): - -```yaml -include: - - template: Auto-DevOps.gitlab-ci.yml - -.auto-deploy: - image: "registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v0.17.0" -``` - -#### Ignore warning and continue deploying - -If you are certain that the new chart version is safe to be deployed, -you can add the `AUTO_DEVOPS_FORCE_DEPLOY_V<N>` [environment variable](customize.md#build-and-deployment) -to force the deployment to continue, where `<N>` is the major version. - -For example, if you want to deploy the v2.0.0 chart on a deployment that previously -used the v0.17.0 chart, add `AUTO_DEVOPS_FORCE_DEPLOY_V2`. +This document was moved to [another location](upgrading_auto_deploy_dependencies.md). diff --git a/doc/university/README.md b/doc/university/README.md index d029c91a19f..63252000dd7 100644 --- a/doc/university/README.md +++ b/doc/university/README.md @@ -65,7 +65,6 @@ The GitLab University curriculum is composed of GitLab videos, screencasts, pres 1. [Making GitLab Great for Everyone - Video](https://www.youtube.com/watch?v=GGC40y4vMx0) - Response to "Dear GitHub" letter 1. [Using Innersourcing to Improve Collaboration](https://about.gitlab.com/blog/2014/09/05/innersourcing-using-the-open-source-workflow-to-improve-collaboration-within-an-organization/) 1. [The Software Development Market and GitLab - Video](https://www.youtube.com/watch?v=sXlhgPK1NTY&list=PLFGfElNsQthbQu_IWlNOxul0TbS_2JH-e&index=6) - [Slides](https://docs.google.com/presentation/d/1vCU-NbZWz8NTNK8Vu3y4zGMAHb5DpC8PE5mHtw1PWfI/edit) -1. [The GitLab Book Club](bookclub/index.md) 1. [GitLab Resources](https://about.gitlab.com/resources/) ### 1.7 Community and Support @@ -76,7 +75,6 @@ The GitLab University curriculum is composed of GitLab videos, screencasts, pres - Getting Technical Support - Being part of our Great Community and Contributing to GitLab 1. [Getting Started with the GitLab Development Kit (GDK)](https://about.gitlab.com/blog/2016/06/08/getting-started-with-gitlab-development-kit/) -1. [GitLab Training Workshops](training/end-user/README.md) 1. [GitLab Professional Services](https://about.gitlab.com/services/) ### 1.8 GitLab Training Material diff --git a/doc/university/training/index.md b/doc/university/training/index.md index 69f82392027..4c54108b4fa 100644 --- a/doc/university/training/index.md +++ b/doc/university/training/index.md @@ -17,7 +17,7 @@ This section contains the following topics: - [Cherry pick](topics/cherry_picking.md). - [Code review and collaboration with Merge Requests](topics/merge_requests.md). - [Configure your environment](topics/env_setup.md). -- [Explore GitLab](topics/explore_gitlab.md). +- [Explore GitLab](../../gitlab-basics/README.md). - [Feature branching](topics/feature_branching.md). - [Getting started](topics/getting_started.md). - [GitLab flow](gitlab_flow.md). diff --git a/doc/update/10.0-ce-to-ee.md b/doc/update/10.0-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/10.0-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.0-to-10.1.md b/doc/update/10.0-to-10.1.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/10.0-to-10.1.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/10.1-ce-to-ee.md b/doc/update/10.1-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/10.1-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.1-to-10.2.md b/doc/update/10.1-to-10.2.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/10.1-to-10.2.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/10.2-ce-to-ee.md b/doc/update/10.2-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/10.2-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.2-to-10.3.md b/doc/update/10.2-to-10.3.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/10.2-to-10.3.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/10.3-ce-to-ee.md b/doc/update/10.3-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/10.3-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.3-to-10.4.md b/doc/update/10.3-to-10.4.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/10.3-to-10.4.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/10.4-ce-to-ee.md b/doc/update/10.4-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/10.4-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.4-to-10.5.md b/doc/update/10.4-to-10.5.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/10.4-to-10.5.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/10.5-ce-to-ee.md b/doc/update/10.5-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/10.5-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.5-to-10.6.md b/doc/update/10.5-to-10.6.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/10.5-to-10.6.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/10.6-ce-to-ee.md b/doc/update/10.6-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/10.6-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.6-to-10.7.md b/doc/update/10.6-to-10.7.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/10.6-to-10.7.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/10.7-ce-to-ee.md b/doc/update/10.7-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/10.7-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.7-to-10.8.md b/doc/update/10.7-to-10.8.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/10.7-to-10.8.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/10.8-ce-to-ee.md b/doc/update/10.8-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/10.8-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.8-to-11.0.md b/doc/update/10.8-to-11.0.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/10.8-to-11.0.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/11.0-ce-to-ee.md b/doc/update/11.0-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/11.0-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.0-to-11.1.md b/doc/update/11.0-to-11.1.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/11.0-to-11.1.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/11.1-ce-to-ee.md b/doc/update/11.1-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/11.1-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.1-to-11.2.md b/doc/update/11.1-to-11.2.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/11.1-to-11.2.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/11.2-ce-to-ee.md b/doc/update/11.2-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/11.2-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.2-to-11.3.md b/doc/update/11.2-to-11.3.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/11.2-to-11.3.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/11.3-ce-to-ee.md b/doc/update/11.3-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/11.3-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.3-to-11.4.md b/doc/update/11.3-to-11.4.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/11.3-to-11.4.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/11.4-ce-to-ee.md b/doc/update/11.4-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/11.4-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.4-to-11.5.md b/doc/update/11.4-to-11.5.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/11.4-to-11.5.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/11.5-ce-to-ee.md b/doc/update/11.5-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/11.5-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.5-to-11.6.md b/doc/update/11.5-to-11.6.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/11.5-to-11.6.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/11.6-ce-to-ee.md b/doc/update/11.6-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/11.6-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.6-to-11.7.md b/doc/update/11.6-to-11.7.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/11.6-to-11.7.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/11.7-ce-to-ee.md b/doc/update/11.7-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/11.7-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.7-to-11.8.md b/doc/update/11.7-to-11.8.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/11.7-to-11.8.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/11.8-ce-to-ee.md b/doc/update/11.8-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/11.8-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/2.6-to-3.0.md b/doc/update/2.6-to-3.0.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/2.6-to-3.0.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/2.9-to-3.0.md b/doc/update/2.9-to-3.0.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/2.9-to-3.0.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/3.0-to-3.1.md b/doc/update/3.0-to-3.1.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/3.0-to-3.1.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/3.1-to-4.0.md b/doc/update/3.1-to-4.0.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/3.1-to-4.0.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/4.0-to-4.1.md b/doc/update/4.0-to-4.1.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/4.0-to-4.1.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/4.1-to-4.2.md b/doc/update/4.1-to-4.2.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/4.1-to-4.2.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/4.2-to-5.0.md b/doc/update/4.2-to-5.0.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/4.2-to-5.0.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/5.0-to-5.1.md b/doc/update/5.0-to-5.1.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/5.0-to-5.1.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/5.1-to-5.2.md b/doc/update/5.1-to-5.2.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/5.1-to-5.2.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/5.1-to-5.4.md b/doc/update/5.1-to-5.4.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/5.1-to-5.4.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/5.1-to-6.0.md b/doc/update/5.1-to-6.0.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/5.1-to-6.0.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/5.2-to-5.3.md b/doc/update/5.2-to-5.3.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/5.2-to-5.3.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/5.3-to-5.4.md b/doc/update/5.3-to-5.4.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/5.3-to-5.4.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/5.4-to-6.0.md b/doc/update/5.4-to-6.0.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/5.4-to-6.0.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/6.0-ce-to-ee.md b/doc/update/6.0-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/6.0-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.0-to-6.1.md b/doc/update/6.0-to-6.1.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/6.0-to-6.1.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/6.1-ce-to-ee.md b/doc/update/6.1-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/6.1-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.1-to-6.2.md b/doc/update/6.1-to-6.2.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/6.1-to-6.2.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/6.2-ce-to-ee.md b/doc/update/6.2-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/6.2-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.2-to-6.3.md b/doc/update/6.2-to-6.3.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/6.2-to-6.3.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/6.3-ce-to-ee.md b/doc/update/6.3-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/6.3-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.3-to-6.4.md b/doc/update/6.3-to-6.4.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/6.3-to-6.4.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/6.4-ce-to-ee.md b/doc/update/6.4-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/6.4-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.4-to-6.5.md b/doc/update/6.4-to-6.5.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/6.4-to-6.5.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/6.5-ce-to-ee.md b/doc/update/6.5-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/6.5-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.5-to-6.6.md b/doc/update/6.5-to-6.6.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/6.5-to-6.6.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/6.6-ce-to-ee.md b/doc/update/6.6-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/6.6-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.6-to-6.7.md b/doc/update/6.6-to-6.7.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/6.6-to-6.7.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/6.7-ce-to-ee.md b/doc/update/6.7-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/6.7-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.7-to-6.8.md b/doc/update/6.7-to-6.8.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/6.7-to-6.8.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/6.8-ce-to-ee.md b/doc/update/6.8-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/6.8-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.8-to-6.9.md b/doc/update/6.8-to-6.9.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/6.8-to-6.9.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/6.9-ce-to-ee.md b/doc/update/6.9-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/6.9-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.9-to-7.0.md b/doc/update/6.9-to-7.0.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/6.9-to-7.0.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/6.x-or-7.x-to-7.14.md b/doc/update/6.x-or-7.x-to-7.14.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/6.x-or-7.x-to-7.14.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.0-ce-to-ee.md b/doc/update/7.0-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.0-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.0-to-7.1.md b/doc/update/7.0-to-7.1.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.0-to-7.1.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.1-ce-to-ee.md b/doc/update/7.1-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.1-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.1-to-7.2.md b/doc/update/7.1-to-7.2.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.1-to-7.2.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.10-ce-to-ee.md b/doc/update/7.10-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.10-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.10-to-7.11.md b/doc/update/7.10-to-7.11.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.10-to-7.11.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.11-ce-to-ee.md b/doc/update/7.11-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.11-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.11-to-7.12.md b/doc/update/7.11-to-7.12.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.11-to-7.12.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.12-ce-to-ee.md b/doc/update/7.12-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.12-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.12-to-7.13.md b/doc/update/7.12-to-7.13.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.12-to-7.13.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.13-ce-to-ee.md b/doc/update/7.13-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.13-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.13-to-7.14.md b/doc/update/7.13-to-7.14.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.13-to-7.14.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.14-ce-to-ee.md b/doc/update/7.14-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.14-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.14-to-8.0.md b/doc/update/7.14-to-8.0.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.14-to-8.0.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.2-to-7.3.md b/doc/update/7.2-to-7.3.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.2-to-7.3.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.3-ce-to-ee.md b/doc/update/7.3-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.3-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.3-to-7.4.md b/doc/update/7.3-to-7.4.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.3-to-7.4.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.4-ce-to-ee.md b/doc/update/7.4-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.4-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.4-to-7.5.md b/doc/update/7.4-to-7.5.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.4-to-7.5.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.5-ce-to-ee.md b/doc/update/7.5-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.5-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.5-to-7.6.md b/doc/update/7.5-to-7.6.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.5-to-7.6.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.6-ce-to-ee.md b/doc/update/7.6-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.6-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.6-to-7.7.md b/doc/update/7.6-to-7.7.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.6-to-7.7.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.7-ce-to-ee.md b/doc/update/7.7-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.7-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.7-to-7.8.md b/doc/update/7.7-to-7.8.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.7-to-7.8.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.8-ce-to-ee.md b/doc/update/7.8-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.8-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.8-to-7.9.md b/doc/update/7.8-to-7.9.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.8-to-7.9.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/7.9-ce-to-ee.md b/doc/update/7.9-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/7.9-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.9-to-7.10.md b/doc/update/7.9-to-7.10.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/7.9-to-7.10.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.0-ce-to-ee.md b/doc/update/8.0-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.0-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.0-to-8.1.md b/doc/update/8.0-to-8.1.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.0-to-8.1.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.1-ce-to-ee.md b/doc/update/8.1-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.1-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.1-to-8.2.md b/doc/update/8.1-to-8.2.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.1-to-8.2.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.10-ce-to-ee.md b/doc/update/8.10-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.10-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.10-to-8.11.md b/doc/update/8.10-to-8.11.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.10-to-8.11.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.11-ce-to-ee.md b/doc/update/8.11-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.11-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.11-to-8.12.md b/doc/update/8.11-to-8.12.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.11-to-8.12.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.12-ce-to-ee.md b/doc/update/8.12-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.12-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.12-to-8.13.md b/doc/update/8.12-to-8.13.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.12-to-8.13.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.13-ce-to-ee.md b/doc/update/8.13-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.13-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.13-to-8.14.md b/doc/update/8.13-to-8.14.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.13-to-8.14.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.14-ce-to-ee.md b/doc/update/8.14-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.14-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.14-to-8.15.md b/doc/update/8.14-to-8.15.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.14-to-8.15.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.15-ce-to-ee.md b/doc/update/8.15-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.15-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.15-to-8.16.md b/doc/update/8.15-to-8.16.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.15-to-8.16.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.16-ce-to-ee.md b/doc/update/8.16-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.16-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.16-to-8.17.md b/doc/update/8.16-to-8.17.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.16-to-8.17.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.17-ce-to-ee.md b/doc/update/8.17-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.17-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.17-to-9.0.md b/doc/update/8.17-to-9.0.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.17-to-9.0.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.2-ce-to-ee.md b/doc/update/8.2-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.2-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.2-to-8.3.md b/doc/update/8.2-to-8.3.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.2-to-8.3.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.3-ce-to-ee.md b/doc/update/8.3-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.3-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.3-to-8.4.md b/doc/update/8.3-to-8.4.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.3-to-8.4.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.4-ce-to-ee.md b/doc/update/8.4-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.4-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.4-to-8.5.md b/doc/update/8.4-to-8.5.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.4-to-8.5.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.5-ce-to-ee.md b/doc/update/8.5-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.5-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.5-to-8.6.md b/doc/update/8.5-to-8.6.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.5-to-8.6.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.6-ce-to-ee.md b/doc/update/8.6-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.6-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.6-to-8.7.md b/doc/update/8.6-to-8.7.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.6-to-8.7.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.7-ce-to-ee.md b/doc/update/8.7-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.7-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.7-to-8.8.md b/doc/update/8.7-to-8.8.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.7-to-8.8.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.8-ce-to-ee.md b/doc/update/8.8-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.8-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.8-to-8.9.md b/doc/update/8.8-to-8.9.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.8-to-8.9.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/8.9-ce-to-ee.md b/doc/update/8.9-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/8.9-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.9-to-8.10.md b/doc/update/8.9-to-8.10.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/8.9-to-8.10.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/9.0-ce-to-ee.md b/doc/update/9.0-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/9.0-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.0-to-9.1.md b/doc/update/9.0-to-9.1.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/9.0-to-9.1.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/9.1-ce-to-ee.md b/doc/update/9.1-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/9.1-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.1-to-9.2.md b/doc/update/9.1-to-9.2.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/9.1-to-9.2.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/9.2-ce-to-ee.md b/doc/update/9.2-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/9.2-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.2-to-9.3.md b/doc/update/9.2-to-9.3.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/9.2-to-9.3.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/9.3-ce-to-ee.md b/doc/update/9.3-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/9.3-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.3-to-9.4.md b/doc/update/9.3-to-9.4.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/9.3-to-9.4.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/9.4-ce-to-ee.md b/doc/update/9.4-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/9.4-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.4-to-9.5.md b/doc/update/9.4-to-9.5.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/9.4-to-9.5.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/update/9.5-ce-to-ee.md b/doc/update/9.5-ce-to-ee.md deleted file mode 100644 index 10c9e21fa81..00000000000 --- a/doc/update/9.5-ce-to-ee.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_ce_to_ee.md ---- - -This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.5-to-10.0.md b/doc/update/9.5-to-10.0.md deleted file mode 100644 index 8514aa13f48..00000000000 --- a/doc/update/9.5-to-10.0.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: upgrading_from_source.md ---- - -This document was moved to [another location](upgrading_from_source.md). diff --git a/doc/user/admin_area/abuse_reports.md b/doc/user/admin_area/abuse_reports.md index 0283f1a9eff..4c346d7dfe8 100644 --- a/doc/user/admin_area/abuse_reports.md +++ b/doc/user/admin_area/abuse_reports.md @@ -20,6 +20,9 @@ To receive notifications of new abuse reports by e-mail, follow these steps: 1. Expand the **Abuse reports** section. 1. Provide an email address. +The notification email address can also be set and retrieved +[using the API](../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls). + ## Reporting abuse To find out more about reporting abuse, see [abuse reports user @@ -31,14 +34,14 @@ To access abuse reports, go to **Admin Area > Abuse Reports**. There are 3 ways to resolve an abuse report, with a button for each method: -- Remove user & report. This will: - - [Delete the reported user](../profile/account/delete_account.md) from the +- Remove user & report. This: + - [Deletes the reported user](../profile/account/delete_account.md) from the instance. - - Remove the abuse report from the list. + - Removes the abuse report from the list. - [Block user](#blocking-users). -- Remove report. This will: - - Remove the abuse report from the list. - - Remove access restrictions for the reported user. +- Remove report. This: + - Removes the abuse report from the list. + - Removes access restrictions for the reported user. The following is an example of the **Abuse Reports** page: @@ -54,8 +57,7 @@ Blocking a user: - Leaves them in the abuse report list. - Changes the **Block user** button to a disabled **Already blocked** button. -The user will be notified with the -[following message](https://gitlab.com/gitlab-org/gitlab/blob/master/app/workers/email_receiver_worker.rb#L38): +The user is notified with the following message: ```plaintext Your account has been blocked. If you believe this is in error, contact a staff member. diff --git a/doc/user/admin_area/activating_deactivating_users.md b/doc/user/admin_area/activating_deactivating_users.md index 29f162616bf..8b3a7682841 100644 --- a/doc/user/admin_area/activating_deactivating_users.md +++ b/doc/user/admin_area/activating_deactivating_users.md @@ -39,7 +39,7 @@ A user can be deactivated from the Admin Area. To do this: Please note that for the deactivation option to be visible to an admin, the user: - Must be currently active. -- Must not have signed in, or have any activity, in the last 180 days. +- Must not have signed in, or have any activity, in the last 90 days. Users can also be deactivated using the [GitLab API](../../api/users.md#deactivate-user). diff --git a/doc/user/admin_area/analytics/img/cohorts_v13_4.png b/doc/user/admin_area/analytics/img/cohorts_v13_4.png Binary files differindex 4af1841a033..6f7dd5101f2 100644 --- a/doc/user/admin_area/analytics/img/cohorts_v13_4.png +++ b/doc/user/admin_area/analytics/img/cohorts_v13_4.png diff --git a/doc/user/admin_area/analytics/img/dev_ops_report_v13_4.png b/doc/user/admin_area/analytics/img/dev_ops_report_v13_4.png Binary files differindex 1fa070a6915..d47d86cd514 100644 --- a/doc/user/admin_area/analytics/img/dev_ops_report_v13_4.png +++ b/doc/user/admin_area/analytics/img/dev_ops_report_v13_4.png diff --git a/doc/user/admin_area/analytics/index.md b/doc/user/admin_area/analytics/index.md index b3336b471f8..f79245c7325 100644 --- a/doc/user/admin_area/analytics/index.md +++ b/doc/user/admin_area/analytics/index.md @@ -4,7 +4,8 @@ Administrators have access to instance-wide analytics, as shown in **Admin Area > Analytics**. -There are two kinds of statistics: +There are several kinds of statistics: - [DevOps Report](dev_ops_report.md): Provides an overview of your entire instance's feature usage. +- [Instance Statistics](instance_statistics.md): Shows how much data your instance contains, and how that is changing. - [User Cohorts](user_cohorts.md): Display the monthly cohorts of new users and their activities over time. diff --git a/doc/user/admin_area/analytics/instance_statistics.md b/doc/user/admin_area/analytics/instance_statistics.md new file mode 100644 index 00000000000..bac0e845d2c --- /dev/null +++ b/doc/user/admin_area/analytics/instance_statistics.md @@ -0,0 +1,18 @@ +# Instance Statistics + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235754) in GitLab 13.4. + +Instance Statistics gives you an overview of how much data your instance contains, and how quickly this volume is changing over time. + +## Total counts + +At the top of the page, Instance Statistics shows total counts for: + +- Users +- Projects +- Groups +- Issues +- Merge Requests +- Pipelines + +These figures can be useful for understanding how much data your instance contains in total. diff --git a/doc/user/admin_area/credentials_inventory.md b/doc/user/admin_area/credentials_inventory.md index 7f2d49dafea..e845f22140b 100644 --- a/doc/user/admin_area/credentials_inventory.md +++ b/doc/user/admin_area/credentials_inventory.md @@ -9,8 +9,6 @@ type: howto > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20912) in GitLab 12.6. -## Overview - GitLab administrators are responsible for the overall security of their instance. To assist, GitLab provides a Credentials inventory to keep track of all the credentials that can be used to access their self-managed instance. Using Credentials inventory, you can see all the personal access tokens (PAT) and SSH keys that exist in your GitLab instance. In addition, you can [revoke them](#revoke-a-users-personal-access-token) and see: diff --git a/doc/user/admin_area/settings/gitaly_timeouts.md b/doc/user/admin_area/settings/gitaly_timeouts.md index 2003d02c9b3..cac05678a1a 100644 --- a/doc/user/admin_area/settings/gitaly_timeouts.md +++ b/doc/user/admin_area/settings/gitaly_timeouts.md @@ -3,36 +3,28 @@ stage: Create group: Gitaly info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" type: reference -type: reference --- -# Gitaly timeouts - - - -3 timeout types can be configured to make sure that long running -Gitaly calls don't needlessly take up resources. - -- Default timeout - -This timeout is the default for most Gitaly calls. -It should be shorter than the worker timeout that can be configured -for -[Puma](https://docs.gitlab.com/omnibus/settings/puma.html#puma-settings) -or [Unicorn](https://docs.gitlab.com/omnibus/settings/unicorn.html). -This makes sure that Gitaly calls made within a web request cannot -exceed these the entire request timeout. +# Gitaly timeouts **(CORE ONLY)** -The default for this timeout is 55 seconds. +[Gitaly](../../../administration/gitaly/index.md) timeouts are configurable. The timeouts can be +configured to make sure that long running Gitaly calls don't needlessly take up resources. -- Fast timeout +To access Gitaly timeout settings: -This is the timeout for very short Gitaly calls. +1. Go to **Admin Area > Settings > Preferences**. +1. Expand the **Gitaly** section. -The default for this timeout is 10 seconds. +## Available timeouts -- Medium timeout +The following timeouts can be modified: -This timeout should be between the default and the fast timeout +- **Default Timeout Period**. This timeout is the default for most Gitaly calls. It should be shorter than the + worker timeout that can be configured for [Puma](https://docs.gitlab.com/omnibus/settings/puma.html#puma-settings) + or [Unicorn](https://docs.gitlab.com/omnibus/settings/unicorn.html). Used to make sure that Gitaly + calls made within a web request cannot exceed the entire request timeout. + Defaults to 55 seconds. -The default for this timeout is 30 seconds. +- **Fast Timeout Period**. This is the timeout for very short Gitaly calls. Defaults to 10 seconds. +- **Medium Timeout Period**. This timeout should be between the default and the fast timeout. + Defaults to 30 seconds. diff --git a/doc/user/admin_area/settings/img/gitaly_timeouts.png b/doc/user/admin_area/settings/img/gitaly_timeouts.png Binary files differdeleted file mode 100644 index 28394d238f7..00000000000 --- a/doc/user/admin_area/settings/img/gitaly_timeouts.png +++ /dev/null diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index ba9bccbf3e7..a107e607d7f 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -30,7 +30,7 @@ Access the default page for admin area settings by navigating to **Admin Area > | Option | Description | | ------ | ----------- | -| [Elasticsearch](../../../integration/elasticsearch.md#enabling-elasticsearch) | Elasticsearch integration. Elasticsearch AWS IAM. | +| [Elasticsearch](../../../integration/elasticsearch.md#enabling-advanced-search) | Elasticsearch integration. Elasticsearch AWS IAM. | | [PlantUML](../../../administration/integration/plantuml.md#gitlab) | Allow rendering of PlantUML diagrams in AsciiDoc documents. | | [Slack application](../../../user/project/integrations/gitlab_slack_application.md#configuration) **(FREE ONLY)** | Slack integration allows you to interact with GitLab via slash commands in a chat window. This option is only available on GitLab.com, though it may be [available for self-managed instances in the future](https://gitlab.com/gitlab-org/gitlab/-/issues/28164). | | [Third party offers](third_party_offers.md) | Control the display of third party offers. | @@ -102,7 +102,7 @@ Access the default page for admin area settings by navigating to **Admin Area > | Option | Description | | ------ | ----------- | | [Email](email.md) | Various email settings. | -| [Help page](../../../customization/help_message.md) | Help page text and support page URL. | +| [Help page](help_page.md) | Help page text and support page URL. | | [Pages](../../../administration/pages/index.md#custom-domain-verification) | Size and domain settings for static websites | | [Real-time features](../../../administration/polling.md) | Change this value to influence how frequently the GitLab UI polls for updates. | | [Gitaly timeouts](gitaly_timeouts.md) | Configure Gitaly timeouts. | diff --git a/doc/user/admin_area/settings/instance_template_repository.md b/doc/user/admin_area/settings/instance_template_repository.md index 97380b93295..20f2812bc39 100644 --- a/doc/user/admin_area/settings/instance_template_repository.md +++ b/doc/user/admin_area/settings/instance_template_repository.md @@ -9,8 +9,6 @@ type: reference > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5986) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.3. -## Overview - In hosted systems, enterprises often have a need to share their own templates across teams. This feature allows an administrator to pick a project to be the instance-wide collection of file templates. These templates are then exposed to diff --git a/doc/user/analytics/img/mr_throughput_chart_v13_3.png b/doc/user/analytics/img/mr_throughput_chart_v13_3.png Binary files differindex 04fa54f323c..100c9a8557c 100644 --- a/doc/user/analytics/img/mr_throughput_chart_v13_3.png +++ b/doc/user/analytics/img/mr_throughput_chart_v13_3.png diff --git a/doc/user/analytics/img/mr_throughput_table_v13_3.png b/doc/user/analytics/img/mr_throughput_table_v13_3.png Binary files differindex 63ffb9389f4..bb63770dc3f 100644 --- a/doc/user/analytics/img/mr_throughput_table_v13_3.png +++ b/doc/user/analytics/img/mr_throughput_table_v13_3.png diff --git a/doc/user/analytics/img/new_value_stream_v13_3.png b/doc/user/analytics/img/new_value_stream_v13_3.png Binary files differindex 4284b8ab194..bc8502e85a6 100644 --- a/doc/user/analytics/img/new_value_stream_v13_3.png +++ b/doc/user/analytics/img/new_value_stream_v13_3.png diff --git a/doc/user/analytics/img/vsa_filter_bar_v13.3.png b/doc/user/analytics/img/vsa_filter_bar_v13.3.png Binary files differindex 71e59892434..506765f63cb 100644 --- a/doc/user/analytics/img/vsa_filter_bar_v13.3.png +++ b/doc/user/analytics/img/vsa_filter_bar_v13.3.png diff --git a/doc/user/analytics/productivity_analytics.md b/doc/user/analytics/productivity_analytics.md index 653836de8be..290aae6395f 100644 --- a/doc/user/analytics/productivity_analytics.md +++ b/doc/user/analytics/productivity_analytics.md @@ -13,7 +13,7 @@ Track development velocity with Productivity Analytics. For many companies, the development cycle is a black box and getting an estimate of how long, on average, it takes to deliver features is an enormous endeavor. -While [Value Stream Analytics](../project/cycle_analytics.md) focuses on the entire +While [Value Stream Analytics](../analytics/value_stream_analytics.md) focuses on the entire Software Development Life Cycle (SDLC) process, Productivity Analytics provides a way for Engineering Management to drill down in a systematic way to uncover patterns and causes for success or failure at an individual, project, or group level. Productivity can slow down for many reasons ranging from degrading code base to quickly growing teams. In order to investigate, department or team leaders can start by visualizing the time it takes for merge requests to be merged. diff --git a/doc/user/analytics/value_stream_analytics.md b/doc/user/analytics/value_stream_analytics.md index 14012d4a28d..49157823d9f 100644 --- a/doc/user/analytics/value_stream_analytics.md +++ b/doc/user/analytics/value_stream_analytics.md @@ -12,26 +12,26 @@ info: To determine the technical writer assigned to the Stage/Group associated w Value Stream Analytics measures the time spent to go from an [idea to production](https://about.gitlab.com/blog/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab) -(also known as cycle time) for each of your projects. Value Stream Analytics displays the median time +(also known as cycle time) for each of your projects or groups. Value Stream Analytics displays the median time spent in each stage defined in the process. -For information on how to contribute to the development of Value Stream Analytics, see our [contributor documentation](../../development/value_stream_analytics.md). - Value Stream Analytics is useful in order to quickly determine the velocity of a given project. It points to bottlenecks in the development process, enabling management to uncover, triage, and identify the root cause of slowdowns in the software development life cycle. -Value Stream Analytics is tightly coupled with the [GitLab flow](../../topics/gitlab_flow.md) and -calculates a separate median for each stage. +For information on how to contribute to the development of Value Stream Analytics, see our [contributor documentation](../../development/value_stream_analytics.md). + +## Project Level Value Stream Analytics **CORE** + +Project Level Value Stream Analytics is available via **Project > Analytics > Value Stream**. -## Overview +## Group Level Value Stream Analytics **PREMIUM** -Value Stream Analytics is available: +From GitLab 12.9, group level Value Stream Analytics is available via **Group > Analytics > Value Stream**. -- From GitLab 12.9, at the group level via **Group > Analytics > Value Stream**. **(PREMIUM)** -- At the project level via **Project > Analytics > Value Stream**. +## Default stages -There are seven stages that are tracked as part of the Value Stream Analytics calculations. +The stages tracked by Value Stream Analytics by default represent the [GitLab flow](../../topics/gitlab_flow.md). These stages can be customized in Group Level Value Stream Analytics. - **Issue** (Tracker) - Time to schedule an issue (by milestone or by adding it to an issue board) @@ -123,7 +123,7 @@ How this works, behind the scenes: we need for the stages, like issue creation date, merge request merge time, etc. -To sum up, anything that doesn't follow [GitLab flow](../../workflow/gitlab_flow.md) will not be tracked and the +To sum up, anything that doesn't follow [GitLab flow](../../topics/gitlab_flow.md) will not be tracked and the Value Stream Analytics dashboard will not present any data for: - Merge requests that do not close an issue. diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index a6ad701360e..ead34ca227e 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -19,8 +19,9 @@ then in the left sidebar go to **Security & Compliance > Configuration**. For each security control the page displays: -- **Status** - Status of the security control: enabled, not enabled, or available. -- **Manage** - A management option or a link to the documentation. +- **Security Control:** Name, description, and a documentation link. +- **Status:** The security control's status (enabled, not enabled, or available). +- **Manage:** A management option or a documentation link. ## Status @@ -29,12 +30,11 @@ The status of each security control is determined by the project's latest defaul If a job with the expected security report artifact exists in the pipeline, the feature's status is _enabled_. -For SAST, click **View history** to see the `.gitlab-ci.yml` file’s history. - -NOTE: **Note:** If the latest pipeline used [Auto DevOps](../../../topics/autodevops/index.md), all security features are configured by default. +For SAST, click **View history** to see the `.gitlab-ci.yml` file's history. + ## Manage You can configure the following security controls: diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 880e5a3875a..47b7347deba 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -9,8 +9,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3672) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. -## Overview - Your application's Docker image may itself be based on Docker images that contain known vulnerabilities. By including an extra job in your pipeline that scans for those vulnerabilities and displays them in a merge request, you can use GitLab to audit your Docker-based apps. @@ -19,7 +17,6 @@ By default, container scanning in GitLab is based on [Clair](https://github.com/ containers. [GitLab's Klar analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/klar/) scans the containers and serves as a wrapper for Clair. -NOTE: **Note:** To integrate security scanners other than Clair and Klar into GitLab, see [Security scanner integration](../../../development/integrations/secure.md). @@ -65,7 +62,7 @@ To enable container scanning in your pipeline, you need the following: variables: IMAGE_TAG: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA script: - - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY + - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t $IMAGE_TAG . - docker push $IMAGE_TAG ``` @@ -119,7 +116,7 @@ build: IMAGE: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA script: - docker info - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t $IMAGE . - docker push $IMAGE @@ -219,8 +216,7 @@ To use container scanning in an offline environment, you need: - GitLab Runner with the [`docker` or `kubernetes` executor](#requirements). - To configure a local Docker container registry with copies of the container scanning [analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/klar) images, found in the [container scanning container registry](https://gitlab.com/gitlab-org/security-products/analyzers/klar/container_registry). -NOTE: **Note:** -GitLab Runner has a [default `pull policy` of `always`](https://docs.gitlab.com/runner/executors/docker.html#using-the-always-pull-policy), +Note that GitLab Runner has a [default `pull policy` of `always`](https://docs.gitlab.com/runner/executors/docker.html#using-the-always-pull-policy), meaning the runner tries to pull Docker images from the GitLab container registry even if a local copy is available. The GitLab Runner [`pull_policy` can be set to `if-not-present`](https://docs.gitlab.com/runner/executors/docker.html#using-the-if-not-present-pull-policy) in an offline environment if you prefer using only locally available Docker images. However, we @@ -287,7 +283,7 @@ build_latest_vulnerabilities: script: - docker pull arminc/clair-db:latest - docker tag arminc/clair-db:latest $CI_REGISTRY/namespace/clair-vulnerabilities-db - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker push $CI_REGISTRY/namespace/clair-vulnerabilities-db ``` @@ -433,3 +429,7 @@ This is a result of a bug in Docker which is now [fixed](https://github.com/cont To prevent the error, ensure the Docker version that the runner is using is `18.09.03` or higher. For more information, see [issue #10241](https://gitlab.com/gitlab-org/gitlab/-/issues/10241 "Investigate why Container Scanning is not working with NFS mounts"). + +### Getting warning message `gl-container-scanning-report.json: no matching files` + +For information on this, see the [general Application Security troubleshooting section](../../../ci/pipelines/job_artifacts.md#error-message-no-files-to-upload). diff --git a/doc/user/application_security/dast/img/dast_v13_2.png b/doc/user/application_security/dast/img/dast_v13_2.png Binary files differdeleted file mode 100644 index bbf7944eb40..00000000000 --- a/doc/user/application_security/dast/img/dast_v13_2.png +++ /dev/null diff --git a/doc/user/application_security/dast/img/dast_v13_4.png b/doc/user/application_security/dast/img/dast_v13_4.png Binary files differnew file mode 100644 index 00000000000..d9c1d1b5c66 --- /dev/null +++ b/doc/user/application_security/dast/img/dast_v13_4.png diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 73a8e727389..d83b7e34d51 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -9,17 +9,17 @@ type: reference, howto > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4348) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. -NOTE: **Note:** -The whitepaper ["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) -explains how **4 of the top 6 attacks were application based**. Download it -to learn how to protect your organization. - Running [static checks](../sast/index.md) on your code is the first step to detect vulnerabilities that can put the security of your code at risk. Yet, once deployed, your application is exposed to a new category of possible attacks, such as cross-site scripting or broken authentication flaws. This is where Dynamic Application Security Testing (DAST) comes into place. +NOTE: **Note:** +The whitepaper ["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) +explains how 4 of the top 6 attacks were application based. Download it to learn how to protect your +organization. + ## Overview If you're using [GitLab CI/CD](../../../ci/README.md), you can analyze your running web applications @@ -32,11 +32,10 @@ provided by [Auto DevOps](../../../topics/autodevops/index.md). GitLab checks the DAST report, compares the found vulnerabilities between the source and target branches, and shows the information on the merge request. -NOTE: **Note:** -This comparison logic uses only the latest pipeline executed for the target branch's base commit. -Running the pipeline on any other commit has no effect on the merge request. +Note that this comparison logic uses only the latest pipeline executed for the target branch's base +commit. Running the pipeline on any other commit has no effect on the merge request. - + By clicking on one of the detected linked vulnerabilities, you can see the details and the URL(s) affected. @@ -53,12 +52,11 @@ However, DAST can be [configured](#full-scan) to also perform an *active scan*: attack your application and produce a more extensive security report. It can be very useful combined with [Review Apps](../../../ci/review_apps/index.md). -NOTE: **Note:** -A pipeline may consist of multiple jobs, including SAST and DAST scanning. If any -job fails to finish for any reason, the security dashboard doesn't show DAST scanner -output. For example, if the DAST job finishes but the SAST job fails, the security -dashboard doesn't show DAST results. The analyzer outputs an -[exit code](../../../development/integrations/secure.md#exit-code) on failure. +Note that a pipeline may consist of multiple jobs, including SAST and DAST scanning. If any job +fails to finish for any reason, the security dashboard doesn't show DAST scanner output. For +example, if the DAST job finishes but the SAST job fails, the security dashboard doesn't show DAST +results. On failure, the analyzer outputs an +[exit code](../../../development/integrations/secure.md#exit-code). ## Use cases @@ -206,8 +204,8 @@ variables: DAST_FULL_SCAN_ENABLED: "true" ``` -NOTE: **Note:** -If your DAST job exceeds the job timeout and you need to reduce the scan duration, we shared some tips for optimizing DAST scans in a [blog post](https://about.gitlab.com/blog/2020/08/31/how-to-configure-dast-full-scans-for-complex-web-applications/). +If your DAST job exceeds the job timeout and you need to reduce the scan duration, we shared some +tips for optimizing DAST scans in a [blog post](https://about.gitlab.com/blog/2020/08/31/how-to-configure-dast-full-scans-for-complex-web-applications/). #### Domain validation @@ -398,11 +396,9 @@ variables: DAST_API_HOST_OVERRIDE: api-test.host.com ``` -NOTE: **Note:** -Using a host override is ONLY supported when importing the API -specification from a URL. It does not work and will be ignored when importing -the specification from a file. This is due to a limitation in the ZAP OpenAPI -extension. +Note that using a host override is ONLY supported when importing the API specification from a URL. +It doesn't work and is ignored when importing the specification from a file. This is due to a +limitation in the ZAP OpenAPI extension. #### Authentication using headers @@ -427,7 +423,8 @@ A URL scan allows you to specify which parts of a website are scanned by DAST. #### Define the URLs to scan -To specify the paths to be scanned, add a comma-separated list of the paths to the `DAST_PATHS` environment variable. Note that you can only scan paths of a single host. +To specify the paths to scan, add a comma-separated list of the paths to the `DAST_PATHS` +environment variable. Note that you can only scan paths of a single host. ```yaml include: @@ -437,8 +434,11 @@ variables: DAST_PATHS=/page1.html,/category1/page1.html,/page3.html ``` -NOTE: **Note:** -`DAST_AUTH_EXCLUDE_URLS` are ignored when `DAST_PATHS` is set. +When using `DAST_PATHS`, note the following: + +- The `DAST_PATHS` environment variable has a limit of about 130kb. If you have a list or paths + greater than this, you should create multiple DAST jobs and split the paths over each job. +- The `DAST_AUTH_EXCLUDE_URLS` environment variable is ignored when `DAST_PATHS` is set. #### Full Scan @@ -590,8 +590,7 @@ To use DAST in an offline environment, you need: [container image](https://gitlab.com/gitlab-org/security-products/dast), found in the [DAST container registry](https://gitlab.com/gitlab-org/security-products/dast/container_registry). -NOTE: **Note:** -GitLab Runner has a [default `pull policy` of `always`](https://docs.gitlab.com/runner/executors/docker.html#using-the-always-pull-policy), +Note that GitLab Runner has a [default `pull policy` of `always`](https://docs.gitlab.com/runner/executors/docker.html#using-the-always-pull-policy), meaning the runner tries to pull Docker images from the GitLab container registry even if a local copy is available. The GitLab Runner [`pull_policy` can be set to `if-not-present`](https://docs.gitlab.com/runner/executors/docker.html#using-the-if-not-present-pull-policy) in an offline environment if you prefer using only locally available Docker images. However, we @@ -672,11 +671,6 @@ To delete an existing site profile: ## Scanner profile > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222767) in GitLab 13.4. -> - [Deployed behind a feature flag](../../feature_flags.md), enabled by default. -> - Enabled on GitLab.com. -> - Can be enabled or disabled per-project. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can [disable this feature](#enable-or-disable-dast-scanner-profiles). A scanner profile defines the scanner settings used to run an on-demand scan: @@ -711,29 +705,6 @@ To delete a scanner profile: 1. Click **Manage** in the **DAST Profiles** row. 1. Click **{remove}** in the scanner profile's row. -### Enable or disable DAST scanner profiles - -The scanner profile feature is ready for production use. It's deployed behind a feature flag that -is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) can opt to disable it. - -To disable it: - -```ruby -# Instance-wide -Feature.disable(:security_on_demand_scans_scanner_profiles) -# or by project -Feature.disable(:security_on_demand_scans_scanner_profiles, Project.find(<project id>)) -``` - -To enable it: - -```ruby -# Instance-wide -Feature.enable(:security_on_demand_scans_scanner_profiles) -# or by project -Feature.enable(:security_on_demand_scans_scanner_profiles, Project.find(<project ID>)) -``` - ## On-demand scans > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218465) in GitLab 13.2. @@ -756,7 +727,8 @@ An on-demand DAST scan: NOTE: **Note:** You must have permission to run an on-demand DAST scan against a protected branch. -The default branch is automatically protected. For more details, see [Pipeline security on protected branches](../../../ci/pipelines/index.md#pipeline-security-on-protected-branches). +The default branch is automatically protected. For more information, see +[Pipeline security on protected branches](../../../ci/pipelines/index.md#pipeline-security-on-protected-branches). To run an on-demand DAST scan, you need: @@ -923,6 +895,10 @@ Change the number after `-Xmx` to the required memory amount. If your DAST job exceeds the job timeout and you need to reduce the scan duration, we shared some tips for optimizing DAST scans in a [blog post](https://about.gitlab.com/blog/2020/08/31/how-to-configure-dast-full-scans-for-complex-web-applications/). +### Getting warning message `gl-dast-report.json: no matching files` + +For information on this, see the [general Application Security troubleshooting section](../../../ci/pipelines/job_artifacts.md#error-message-no-files-to-upload). + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index 5cce336d04c..67d2ae2d3a7 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -9,25 +9,26 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5105) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.7. -Dependency Scanning helps to find security vulnerabilities in your dependencies automatically -while you're developing and testing your applications, such as when your -application is using an external (open source) library that is known to be vulnerable. +GitLab's Dependency Scanning feature can automatically find security vulnerabilities in your +dependencies while you're developing and testing your applications. For example, dependency scanning +lets you know if your application uses an external (open source) library that is known to be +vulnerable. You can then take action to protect your application. ## Overview -If you're using [GitLab CI/CD](../../../ci/README.md), you can analyze your dependencies for known -vulnerabilities using Dependency Scanning. -All dependencies are scanned, including the transitive dependencies (also known as nested dependencies). -You can take advantage of Dependency Scanning by either [including the Dependency Scanning template](#configuration) -in your existing `.gitlab-ci.yml` file or by implicitly using -the [Auto Dependency Scanning](../../../topics/autodevops/stages.md#auto-dependency-scanning) +If you're using [GitLab CI/CD](../../../ci/README.md), you can use dependency scanning to analyze +your dependencies for known vulnerabilities. GitLab scans all dependencies, including transitive +dependencies (also known as nested dependencies). You can take advantage of dependency scanning by +either [including the dependency scanning template](#configuration) +in your existing `.gitlab-ci.yml` file, or by implicitly using +the [auto dependency scanning](../../../topics/autodevops/stages.md#auto-dependency-scanning) provided by [Auto DevOps](../../../topics/autodevops/index.md). -GitLab checks the Dependency Scanning report, compares the found vulnerabilities +GitLab checks the dependency scanning report, compares the found vulnerabilities between the source and target branches, and shows the information on the merge request. - + The results are sorted by the severity of the vulnerability: @@ -40,7 +41,7 @@ The results are sorted by the severity of the vulnerability: ## Requirements -To run Dependency Scanning jobs, by default, you need GitLab Runner with the +To run dependency scanning jobs, by default, you need GitLab Runner with the [`docker`](https://docs.gitlab.com/runner/executors/docker.html) or [`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html) executor. If you're using the shared runners on GitLab.com, this is enabled by default. @@ -56,24 +57,24 @@ The current detection logic limits the maximum search depth to two levels. For e The following languages and dependency managers are supported: -| Language (package managers) | Supported files | Scan tool(s) | -|----------------------------- | --------------- | ------------ | -| C# .NET ([NuGet](https://www.nuget.org/) 4.9+) | [`packages.lock.json`](https://docs.microsoft.com/en-us/nuget/consume-packages/package-references-in-project-files#enabling-lock-file) | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | -| C/C++ ([Conan](https://conan.io/)) | [`conan.lock`](https://docs.conan.io/en/latest/versioning/lockfiles.html) | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | -| Java ([Gradle](https://gradle.org/), [Maven](https://maven.apache.org/)) | `build.gradle`, `build.gradle.kts`, `pom.xml` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | -| JavaScript ([npm](https://www.npmjs.com/), [yarn](https://classic.yarnpkg.com/en/)) | `package-lock.json`, `npm-shrinkwrap.json`, `yarn.lock` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [Retire.js](https://retirejs.github.io/retire.js/) | -| Go ([Golang](https://golang.org/)) | `go.sum` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | -| PHP ([Composer](https://getcomposer.org/)) | `composer.lock` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | -| Python ([setuptools](https://setuptools.readthedocs.io/en/latest/), [pip](https://pip.pypa.io/en/stable/), [Pipenv](https://pipenv.pypa.io/en/latest/)) | `setup.py`, `requirements.txt`, `requirements.pip`, `requires.txt`, `Pipfile` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | -| Ruby ([Bundler](https://bundler.io/)) | `Gemfile.lock`, `gems.locked` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [bundler-audit](https://github.com/rubysec/bundler-audit) | -| Scala ([sbt](https://www.scala-sbt.org/)) | `build.sbt` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| Package Managers | Languages | Supported files | Scan tools | +| ------------------- | --------- | --------------- | ------------ | +| [Bundler](https://bundler.io/) | Ruby | `Gemfile.lock`, `gems.locked` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [bundler-audit](https://github.com/rubysec/bundler-audit) | +| [Composer](https://getcomposer.org/) | PHP | `composer.lock` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| [Conan](https://conan.io/) | C, C++ | [`conan.lock`](https://docs.conan.io/en/latest/versioning/lockfiles.html) | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| [Golang](https://golang.org/) | Go | `go.sum` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) | Java | `build.gradle`, `build.gradle.kts`, `pom.xml` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| [npm](https://www.npmjs.com/), [yarn](https://classic.yarnpkg.com/en/) | JavaScript | `package-lock.json`, `npm-shrinkwrap.json`, `yarn.lock` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [Retire.js](https://retirejs.github.io/retire.js/) | +| [NuGet](https://www.nuget.org/) 4.9+ | .NET, C# | [`packages.lock.json`](https://docs.microsoft.com/en-us/nuget/consume-packages/package-references-in-project-files#enabling-lock-file) | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| [setuptools](https://setuptools.readthedocs.io/en/latest/), [pip](https://pip.pypa.io/en/stable/), [Pipenv](https://pipenv.pypa.io/en/latest/) | Python | `setup.py`, `requirements.txt`, `requirements.pip`, `requires.txt`, `Pipfile` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| [sbt](https://www.scala-sbt.org/) | Scala | `build.sbt` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | Plans are underway for supporting the following languages, dependency managers, and dependency files. For details, see the issue link for each. -| Language (package managers) | Supported files | Scan tool(s) | Issue | -|----------------------------- | --------------- | ------------ | ----- | -| Python ([Poetry](https://python-poetry.org/)) | `poetry.lock` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | [GitLab#7006](https://gitlab.com/gitlab-org/gitlab/-/issues/7006) | -| Python ([Pipenv](https://pipenv.pypa.io/en/latest/)) | `Pipfile.lock` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | [GitLab#11756](https://gitlab.com/gitlab-org/gitlab/-/issues/11756) | +| Package Managers | Languages | Supported files | Scan tools | +| ------------------- | --------- | --------------- | ------------ | +| [Pipenv](https://pipenv.pypa.io/en/latest/) | Python | `Pipfile.lock` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | [GitLab#11756](https://gitlab.com/gitlab-org/gitlab/-/issues/11756) | +| [Poetry](https://python-poetry.org/) | Python | `poetry.lock` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | [GitLab#7006](https://gitlab.com/gitlab-org/gitlab/-/issues/7006) | ## Contribute your scanner @@ -81,7 +82,7 @@ The [Security Scanner Integration](../../../development/integrations/secure.md) ## Configuration -To enable Dependency Scanning for GitLab 11.9 and later, you must +To enable dependency scanning for GitLab 11.9 and later, you must [include](../../../ci/yaml/README.md#includetemplate) the [`Dependency-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml) that is provided as a part of your GitLab installation. @@ -95,16 +96,16 @@ include: - template: Dependency-Scanning.gitlab-ci.yml ``` -The included template creates Dependency Scanning jobs in your CI/CD +The included template creates dependency scanning jobs in your CI/CD pipeline and scans your project's source code for possible vulnerabilities. The results are saved as a -[Dependency Scanning report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportsdependency_scanning) +[dependency scanning report artifact](../../../ci/pipelines/job_artifacts.md#artifactsreportsdependency_scanning) that you can later download and analyze. Due to implementation limitations, we -always take the latest Dependency Scanning artifact available. +always take the latest dependency scanning artifact available. -### Customizing the Dependency Scanning settings +### Customizing the dependency scanning settings -The Dependency Scanning settings can be changed through [environment variables](#available-variables) by using the +The dependency scanning settings can be changed through [environment variables](#available-variables) by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. For example: @@ -119,7 +120,7 @@ variables: Because template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline configuration, the last mention of the variable takes precedence. -### Overriding Dependency Scanning jobs +### Overriding dependency scanning jobs CAUTION: **Deprecation:** Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/README.md#onlyexcept-basic) @@ -141,10 +142,10 @@ gemnasium-dependency_scanning: ### Available variables -Dependency Scanning can be [configured](#customizing-the-dependency-scanning-settings) +Dependency scanning can be [configured](#customizing-the-dependency-scanning-settings) using environment variables. -#### Configuring Dependency Scanning +#### Configuring dependency scanning The following variables allow configuration of global dependency scanning settings. @@ -156,7 +157,7 @@ The following variables allow configuration of global dependency scanning settin | `DS_EXCLUDED_PATHS` | Exclude vulnerabilities from output based on the paths. A comma-separated list of patterns. Patterns can be globs, or file or folder paths (for example, `doc,spec`). Parent directories also match patterns. Default: `"spec, test, tests, tmp"` | | `SECURE_LOG_LEVEL` | Set the minimum logging level. Messages of this logging level or higher are output. From highest to lowest severity, the logging levels are: `fatal`, `error`, `warn`, `info`, `debug`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10880) in GitLab 13.1. Default: `info` | -#### Configuring specific analyzers used by Dependency Scanning +#### Configuring specific analyzers used by dependency scanning The following variables are used for configuring specific analyzers (used for a specific language/framework). @@ -176,7 +177,7 @@ The following variables are used for configuring specific analyzers (used for a | `MAVEN_CLI_OPTS` | `gemnasium-maven` | `"-DskipTests --batch-mode"` | List of command line arguments that are passed to `maven` by the analyzer. See an example for [using private repositories](../index.md#using-private-maven-repos). | | `GRADLE_CLI_OPTS` | `gemnasium-maven` | | List of command line arguments that are passed to `gradle` by the analyzer. | | `SBT_CLI_OPTS` | `gemnasium-maven` | | List of command-line arguments that the analyzer passes to `sbt`. | -| `BUNDLER_AUDIT_UPDATE_DISABLED` | `bundler-audit` | `"false"` | Disable automatic updates for the `bundler-audit` analyzer. Useful if you're running Dependency Scanning in an offline, air-gapped environment.| +| `BUNDLER_AUDIT_UPDATE_DISABLED` | `bundler-audit` | `"false"` | Disable automatic updates for the `bundler-audit` analyzer. Useful if you're running dependency scanning in an offline, air-gapped environment.| | `BUNDLER_AUDIT_ADVISORY_DB_URL` | `bundler-audit` | `https://github.com/rubysec/ruby-advisory-db` | URL of the advisory database used by bundler-audit. | | `BUNDLER_AUDIT_ADVISORY_DB_REF_NAME` | `bundler-audit` | `master` | Git ref for the advisory database specified by `BUNDLER_AUDIT_ADVISORY_DB_URL`. | | `RETIREJS_JS_ADVISORY_DB` | `retire.js` | `https://raw.githubusercontent.com/RetireJS/retire.js/master/repository/jsrepository.json` | Path or URL to `retire.js` JS vulnerability data file. Note that if the URL hosting the data file uses a custom SSL certificate, for example in an offline installation, you can pass the certificate in the `ADDITIONAL_CA_CERT_BUNDLE` environment variable. | @@ -214,16 +215,16 @@ For more information about the vulnerabilities database update, check the ## Dependency List -An additional benefit of Dependency Scanning is the ability to view your +An additional benefit of dependency scanning is the ability to view your project's dependencies and their known vulnerabilities. Read more about the [Dependency List](../dependency_list/index.md). ## Reports JSON format -The Dependency Scanning tool emits a JSON report file. For more information, see the +The dependency scanning tool emits a JSON report file. For more information, see the [schema for this report](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dependency-scanning-report-format.json). -Here's an example Dependency Scanning report: +Here's an example dependency scanning report: ```json-doc { @@ -342,36 +343,35 @@ You can search the [gemnasium-db](https://gitlab.com/gitlab-org/security-product to find a vulnerability in the Gemnasium database. You can also [submit new vulnerabilities](https://gitlab.com/gitlab-org/security-products/gemnasium-db/blob/master/CONTRIBUTING.md). -## Running Dependency Scanning in an offline environment +## Running dependency scanning in an offline environment For self-managed GitLab instances in an environment with limited, restricted, or intermittent access -to external resources through the internet, some adjustments are required for Dependency Scanning +to external resources through the internet, some adjustments are required for dependency scanning jobs to run successfully. For more information, see [Offline environments](../offline_deployments/index.md). -### Requirements for offline Dependency Scanning +### Requirements for offline dependency scanning -Here are the requirements for using Dependency Scanning in an offline environment: +Here are the requirements for using dependency scanning in an offline environment: - GitLab Runner with the [`docker` or `kubernetes` executor](#requirements). -- Docker Container Registry with locally available copies of Dependency Scanning [analyzer](https://gitlab.com/gitlab-org/security-products/analyzers) images. +- Docker Container Registry with locally available copies of dependency scanning [analyzer](https://gitlab.com/gitlab-org/security-products/analyzers) images. - Host an offline Git copy of the [gemnasium-db advisory database](https://gitlab.com/gitlab-org/security-products/gemnasium-db/). This is required because, in an offline environment, the Gemnasium analyzer can't fetch the latest advisories from the online repository. - _Only if scanning Ruby projects_: Host an offline Git copy of the [advisory database](https://github.com/rubysec/ruby-advisory-db). - _Only if scanning npm/yarn projects_: Host an offline copy of the [retire.js](https://github.com/RetireJS/retire.js/) [node](https://github.com/RetireJS/retire.js/blob/master/repository/npmrepository.json) and [js](https://github.com/RetireJS/retire.js/blob/master/repository/jsrepository.json) advisory databases. -NOTE: **Note:** -GitLab Runner has a [default `pull policy` of `always`](https://docs.gitlab.com/runner/executors/docker.html#using-the-always-pull-policy), +Note that GitLab Runner has a [default `pull policy` of `always`](https://docs.gitlab.com/runner/executors/docker.html#using-the-always-pull-policy), meaning the runner tries to pull Docker images from the GitLab container registry even if a local copy is available. The GitLab Runner [`pull_policy` can be set to `if-not-present`](https://docs.gitlab.com/runner/executors/docker.html#using-the-if-not-present-pull-policy) in an offline environment if you prefer using only locally available Docker images. However, we recommend keeping the pull policy setting to `always` if not in an offline environment, as this enables the use of updated scanners in your CI/CD pipelines. -### Make GitLab Dependency Scanning analyzer images available inside your Docker registry +### Make GitLab dependency scanning analyzer images available inside your Docker registry -For Dependency Scanning with all [supported languages and frameworks](#supported-languages-and-package-managers), -import the following default Dependency Scanning analyzer images from `registry.gitlab.com` into +For dependency scanning with all [supported languages and frameworks](#supported-languages-and-package-managers), +import the following default dependency scanning analyzer images from `registry.gitlab.com` into your [local Docker container registry](../../packages/container_registry/index.md): ```plaintext @@ -392,7 +392,7 @@ For details on saving and transporting Docker images as a file, see Docker's doc [`docker save`](https://docs.docker.com/engine/reference/commandline/save/), [`docker load`](https://docs.docker.com/engine/reference/commandline/load/), [`docker export`](https://docs.docker.com/engine/reference/commandline/export/), and [`docker import`](https://docs.docker.com/engine/reference/commandline/import/). -### Set Dependency Scanning CI job variables to use local Dependency Scanning analyzers +### Set dependency scanning CI job variables to use local dependency scanning analyzers Add the following configuration to your `.gitlab-ci.yml` file. You must change the value of `SECURE_ANALYZERS_PREFIX` to refer to your local Docker container registry. You must also change the @@ -479,7 +479,19 @@ As a workaround, remove the [`retire.js`](analyzers.md#selecting-specific-analyz ### `Error response from daemon: error processing tar file: docker-tar: relocation error` -This error occurs when the Docker version that runs the Dependency Scanning job is `19.03.00`. +This error occurs when the Docker version that runs the dependency scanning job is `19.03.00`. Consider updating to Docker `19.03.1` or greater. Older versions are not affected. Read more in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/13830#note_211354992 "Current SAST container fails"). + +### Getting warning message `gl-dependency-scanning-report.json: no matching files` + +For information on this, see the [general Application Security troubleshooting section](../../../ci/pipelines/job_artifacts.md#error-message-no-files-to-upload). + +### Limitation when using rules:exists + +The [dependency scanning CI template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml) +uses the [`rules:exists`](../../../ci/yaml/README.md#rulesexists) +syntax. This directive is limited to 10000 checks and always returns `true` after reaching this +number. Because of this, and depending on the number of files in your repository, a dependency +scanning job might be triggered even if the scanner doesn't support your project. diff --git a/doc/user/application_security/img/cve_request_communication.png b/doc/user/application_security/img/cve_request_communication.png Binary files differindex 0766b371c11..5c58df463ef 100644 --- a/doc/user/application_security/img/cve_request_communication.png +++ b/doc/user/application_security/img/cve_request_communication.png diff --git a/doc/user/application_security/img/cve_request_communication_publication.png b/doc/user/application_security/img/cve_request_communication_publication.png Binary files differindex 9e34c217e13..9eb6f2f8d9f 100644 --- a/doc/user/application_security/img/cve_request_communication_publication.png +++ b/doc/user/application_security/img/cve_request_communication_publication.png diff --git a/doc/user/application_security/img/new_cve_request_issue.png b/doc/user/application_security/img/new_cve_request_issue.png Binary files differindex a342c73992e..6ea7ca4a2ab 100644 --- a/doc/user/application_security/img/new_cve_request_issue.png +++ b/doc/user/application_security/img/new_cve_request_issue.png diff --git a/doc/user/application_security/img/unconfigured_security_approval_rules_and_enabled_jobs_v13_4.png b/doc/user/application_security/img/unconfigured_security_approval_rules_and_enabled_jobs_v13_4.png Binary files differindex f497b0fbc4e..7b04988afdb 100644 --- a/doc/user/application_security/img/unconfigured_security_approval_rules_and_enabled_jobs_v13_4.png +++ b/doc/user/application_security/img/unconfigured_security_approval_rules_and_enabled_jobs_v13_4.png diff --git a/doc/user/application_security/img/unconfigured_security_approval_rules_and_jobs_v13_4.png b/doc/user/application_security/img/unconfigured_security_approval_rules_and_jobs_v13_4.png Binary files differindex fc847b578f5..b9b6dd13294 100644 --- a/doc/user/application_security/img/unconfigured_security_approval_rules_and_jobs_v13_4.png +++ b/doc/user/application_security/img/unconfigured_security_approval_rules_and_jobs_v13_4.png diff --git a/doc/user/application_security/img/vulnerability-check_v13_4.png b/doc/user/application_security/img/vulnerability-check_v13_4.png Binary files differindex e0b53059b45..3e38f6eebe7 100644 --- a/doc/user/application_security/img/vulnerability-check_v13_4.png +++ b/doc/user/application_security/img/vulnerability-check_v13_4.png diff --git a/doc/user/application_security/img/vulnerability_solution.png b/doc/user/application_security/img/vulnerability_solution.png Binary files differindex d86b89a5f99..63e9c1473b6 100644 --- a/doc/user/application_security/img/vulnerability_solution.png +++ b/doc/user/application_security/img/vulnerability_solution.png diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index d509176f2b2..413a9f894e2 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -22,10 +22,10 @@ Testing (SAST), and Secret Detection by adding the following to your `.gitlab-ci ```yaml include: - - template: Dependency-Scanning.gitlab-ci.yml - - template: License-Scanning.gitlab-ci.yml - - template: SAST.gitlab-ci.yml - - template: Secret-Detection.gitlab-ci.yml + - template: Security/Dependency-Scanning.gitlab-ci.yml + - template: Security/License-Scanning.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml + - template: Security/Secret-Detection.gitlab-ci.yml ``` To add Dynamic Application Security Testing (DAST) scanning, add the following to your @@ -33,7 +33,7 @@ To add Dynamic Application Security Testing (DAST) scanning, add the following t ```yaml include: - - template: DAST.gitlab-ci.yml + - template: Security/DAST.gitlab-ci.yml variables: DAST_WEBSITE: https://staging.example.com @@ -449,7 +449,7 @@ To fix this issue, you can either: ```yaml include: - template: SAST.gitlab-ci.yml + template: Security/SAST.gitlab-ci.yml spotbugs-sast: stage: unit-tests @@ -458,6 +458,15 @@ To fix this issue, you can either: [Learn more on overriding SAST jobs](sast/index.md#overriding-sast-jobs). All the security scanning tools define their stage, so this error can occur with all of them. +### Getting warning messages `… report.json: no matching files` + +This is often followed by the [error `No files to upload`](../../ci/pipelines/job_artifacts.md#error-message-no-files-to-upload), +and preceded by other errors or warnings that indicate why the JSON report wasn't generated. Please +check the entire job log for such messages. If you don't find these messages, retry the failed job +after setting `SECURE_LOG_LEVEL: "debug"` as a +[custom environment variable](../../ci/variables/README.md#custom-environment-variables). +This provides useful information to investigate further. + ### Getting error message `sast job: config key may not be used with 'rules': only/except` When [including](../../ci/yaml/README.md#includetemplate) a `.gitlab-ci.yml` template @@ -490,7 +499,7 @@ would look similar to: ```yaml include: - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml # Ensure that the scanning is only executed on master or merge requests spotbugs-sast: @@ -505,7 +514,7 @@ would be written as follows: ```yaml include: - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml # Ensure that the scanning is only executed on master or merge requests spotbugs-sast: @@ -519,7 +528,7 @@ it would look similar to: ```yaml include: - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml # Ensure that the scanning is not executed on tags spotbugs-sast: @@ -531,7 +540,7 @@ To transition to the new `rules` syntax, the override would be rewritten as: ```yaml include: - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml # Ensure that the scanning is not executed on tags spotbugs-sast: diff --git a/doc/user/application_security/sast/analyzers.md b/doc/user/application_security/sast/analyzers.md index 727f077aa09..665cd41ab05 100644 --- a/doc/user/application_security/sast/analyzers.md +++ b/doc/user/application_security/sast/analyzers.md @@ -53,7 +53,7 @@ In `.gitlab-ci.yml` define: ```yaml include: - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml variables: SECURE_ANALYZERS_PREFIX: my-docker-registry/gl-images @@ -70,7 +70,7 @@ In `.gitlab-ci.yml` define: ```yaml include: - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml variables: SAST_DEFAULT_ANALYZERS: "bandit,flawfinder" @@ -86,7 +86,7 @@ default analyzers. In `.gitlab-ci.yml` define: ```yaml include: - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml variables: SAST_DEFAULT_ANALYZERS: "" diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index a4fc3c9e638..f950d48c6d3 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -11,8 +11,8 @@ type: reference, howto NOTE: **Note:** The whitepaper ["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) -explains how **4 of the top 6 attacks were application based**. Download it -to learn how to protect your organization. +explains how 4 of the top 6 attacks were application based. Download it to learn how to protect your +organization. If you're using [GitLab CI/CD](../../../ci/README.md), you can analyze your source code for known vulnerabilities using Static Application Security Testing (SAST). GitLab checks the SAST report and @@ -31,8 +31,10 @@ The results are sorted by the priority of the vulnerability: 1. Unknown 1. Everything else -NOTE: **Note:** -A pipeline consists of multiple jobs, including SAST and DAST scanning. If any job fails to finish for any reason, the security dashboard doesn't show SAST scanner output. For example, if the SAST job finishes but the DAST job fails, the security dashboard doesn't show SAST results. The analyzer outputs an [exit code](../../../development/integrations/secure.md#exit-code) on failure. +A pipeline consists of multiple jobs, including SAST and DAST scanning. If any job fails to finish +for any reason, the security dashboard doesn't show SAST scanner output. For example, if the SAST +job finishes but the DAST job fails, the security dashboard doesn't show SAST results. On failure, +the analyzer outputs an [exit code](../../../development/integrations/secure.md#exit-code). ## Use cases @@ -63,35 +65,33 @@ You can also [view our language roadmap](https://about.gitlab.com/direction/secu | Language (package managers) / framework | Scan tool | Introduced in GitLab Version | |--------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| .NET Core | [Security Code Scan](https://security-code-scan.github.io) | 11.0, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| .NET Framework | [Security Code Scan](https://security-code-scan.github.io) | 13.0, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| Apex (Salesforce) | [PMD](https://pmd.github.io/pmd/index.html) | 12.1, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| C/C++ | [Flawfinder](https://github.com/david-a-wheeler/flawfinder) | 10.7, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.10, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| Go | [Gosec](https://github.com/securego/gosec) | 10.7, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| Groovy ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.3 (Gradle) & 11.9 (Ant, Maven, SBT), [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| Helm Charts | [Kubesec](https://github.com/controlplaneio/kubesec) | 13.1, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| Java ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (Ant, SBT), [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| JavaScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.8, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.2 | -| Kubernetes manifests | [Kubesec](https://github.com/controlplaneio/kubesec) | 12.6, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| Node.js | [NodeJsScan](https://github.com/ajinabraham/NodeJsScan) | 11.1, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| PHP | [phpcs-security-audit](https://github.com/FloeDesignTechnologies/phpcs-security-audit) | 10.8, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| Python ([pip](https://pip.pypa.io/en/stable/)) | [bandit](https://github.com/PyCQA/bandit) | 10.3, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | -| React | [ESLint react plugin](https://github.com/yannickcr/eslint-plugin-react) | 12.5, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.2 | -| Ruby on Rails | [brakeman](https://brakemanscanner.org) | 10.3, [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.1 | -| Scala ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.0 (SBT) & 11.9 (Ant, Gradle, Maven), [moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.3 | +| .NET Core | [Security Code Scan](https://security-code-scan.github.io) | 11.0 | +| .NET Framework | [Security Code Scan](https://security-code-scan.github.io) | 13.0 | +| Apex (Salesforce) | [PMD](https://pmd.github.io/pmd/index.html) | 12.1 | +| C/C++ | [Flawfinder](https://github.com/david-a-wheeler/flawfinder) | 10.7 | +| Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.1 | +| Go | [Gosec](https://github.com/securego/gosec) | 10.7 | +| Groovy ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.3 (Gradle) & 11.9 (Ant, Maven, SBT) | +| Helm Charts | [Kubesec](https://github.com/controlplaneio/kubesec) | 13.1 | +| Java ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (Ant, SBT) | +| JavaScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.8 | +| Kubernetes manifests | [Kubesec](https://github.com/controlplaneio/kubesec) | 12.6 | +| Node.js | [NodeJsScan](https://github.com/ajinabraham/NodeJsScan) | 11.1 | +| PHP | [phpcs-security-audit](https://github.com/FloeDesignTechnologies/phpcs-security-audit) | 10.8 | +| Python ([pip](https://pip.pypa.io/en/stable/)) | [bandit](https://github.com/PyCQA/bandit) | 10.3 | +| React | [ESLint react plugin](https://github.com/yannickcr/eslint-plugin-react) | 12.5 | +| Ruby on Rails | [brakeman](https://brakemanscanner.org) | 10.3 | +| Scala ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.0 (SBT) & 11.9 (Ant, Gradle, Maven) | | TypeScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.9, [merged](https://gitlab.com/gitlab-org/gitlab/-/issues/36059) with ESLint in 13.2 | -NOTE: **Note:** -The Java analyzers can also be used for variants like the +Note that the Java analyzers can also be used for variants like the [Gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html), -[Grails](https://grails.org/) and the [Maven wrapper](https://github.com/takari/maven-wrapper). +[Grails](https://grails.org/), +and the [Maven wrapper](https://github.com/takari/maven-wrapper). ### Making SAST analyzers available to all GitLab tiers -All open source (OSS) analyzers have been moved to the GitLab Core tier. Progress can be -tracked in the corresponding -[epic](https://gitlab.com/groups/gitlab-org/-/epics/2098). +All open source (OSS) analyzers have been moved to the GitLab Core tier as of GitLab 13.3. #### Summary of features per tier @@ -147,6 +147,7 @@ always take the latest SAST artifact available. > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3659) in GitLab Ultimate 13.3. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/232862) in GitLab Ultimate 13.4. +> - [Improved](https://gitlab.com/groups/gitlab-org/-/epics/3635) in GitLab Ultimate 13.5. You can enable and configure SAST with a basic configuration using the **SAST Configuration** page: @@ -154,9 +155,11 @@ page: 1. From the project's home page, go to **Security & Compliance** > **Configuration** in the left sidebar. 1. If the project does not have a `gitlab-ci.yml` file, click **Enable** in the Static Application Security Testing (SAST) row, otherwise click **Configure**. -1. Enter the custom SAST values, then click **Create Merge Request**. +1. Enter the custom SAST values. Custom values are stored in the `.gitlab-ci.yml` file. For variables not in the SAST Configuration page, their values are left unchanged. Default values are inherited from the GitLab SAST template. +1. Optionally, expand the **SAST analyzers** section, select individual [SAST analyzers](./analyzers.md) and enter custom analyzer values. +1. Click **Create Merge Request**. 1. Review and merge the merge request. ### Customizing the SAST settings @@ -169,7 +172,7 @@ set the `SAST_GOSEC_LEVEL` variable to `2`: ```yaml include: - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml variables: SAST_GOSEC_LEVEL: 2 @@ -191,7 +194,7 @@ inclusion and specify any additional keys under it. For example, this enables `F ```yaml include: - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml spotbugs-sast: variables: @@ -222,7 +225,7 @@ Kubesec analyzer. In `.gitlab-ci.yml`, define: ```yaml include: - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml variables: SCAN_KUBERNETES_MANIFESTS: "true" @@ -248,7 +251,7 @@ stages: - test include: - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml build: stage: build @@ -270,11 +273,10 @@ spotbugs-sast: sast: gl-sast-report.json ``` -NOTE: **Note:** -The path to the vendored directory must be specified explicitly to allow -the analyzer to recognize the compiled artifacts. This configuration can vary per -analyzer but in the case of Java above, `MAVEN_REPO_PATH` can be used. -See [Analyzer settings](#analyzer-settings) for the complete list of available options. +To allow the analyzer to recognize the compiled artifacts, you must explicitly specify the path to +the vendored directory. This configuration can vary per analyzer but in the case of Java above, you +can use `MAVEN_REPO_PATH`. See +[Analyzer settings](#analyzer-settings) for the complete list of available options. ### Available variables @@ -480,7 +482,6 @@ To use SAST in an offline environment, you need: - A Docker Container Registry with locally available copies of SAST [analyzer](https://gitlab.com/gitlab-org/security-products/analyzers) images. - Configure certificate checking of packages (optional). -NOTE: **Note:** GitLab Runner has a [default `pull policy` of `always`](https://docs.gitlab.com/runner/executors/docker.html#using-the-always-pull-policy), meaning the runner tries to pull Docker images from the GitLab container registry even if a local copy is available. The GitLab Runner [`pull_policy` can be set to `if-not-present`](https://docs.gitlab.com/runner/executors/docker.html#using-the-if-not-present-pull-policy) @@ -525,7 +526,7 @@ Add the following configuration to your `.gitlab-ci.yml` file. You must replace ```yaml include: - - template: SAST.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml variables: SECURE_ANALYZERS_PREFIX: "localhost:5000/analyzers" @@ -549,3 +550,7 @@ This error occurs when the Docker version that runs the SAST job is `19.03.0`. Consider updating to Docker `19.03.1` or greater. Older versions are not affected. Read more in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/13830#note_211354992 "Current SAST container fails"). + +### Getting warning message `gl-sast-report.json: no matching files` + +For information on this, see the [general Application Security troubleshooting section](../../../ci/pipelines/job_artifacts.md#error-message-no-files-to-upload). diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index f3e411cdc16..aea9b91d9f2 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -9,8 +9,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://about.gitlab.com/releases/2019/03/22/gitlab-11-9-released/#detect-secrets-and-credentials-in-the-repository) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.9. -## Overview - A recurring problem when developing applications is that developers may unintentionally commit secrets and credentials to their remote repositories. If other people have access to the source, or if the project is public, the sensitive information is then exposed and can be leveraged by @@ -67,26 +65,27 @@ as shown in the following table: ## Configuration -NOTE: **Note:** -With GitLab 13.1 Secret Detection was split into its own CI/CD template. +> GitLab 13.1 splits Secret Detection from the [SAST configuration](../sast#configuration) into its own CI/CD template. If you're using GitLab 13.0 or earlier and SAST is enabled, then Secret Detection is already enabled. Secret Detection is performed by a [specific analyzer](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Secret-Detection.gitlab-ci.yml) -during the `secret-detection` job. It runs regardless of the programming -language of your app. +during the `secret-detection` job. It runs regardless of your app's programming language. -The Secret Detection analyzer includes [Gitleaks](https://github.com/zricethezav/gitleaks) and [TruffleHog](https://github.com/dxa4481/truffleHog) checks. +The Secret Detection analyzer includes [Gitleaks](https://github.com/zricethezav/gitleaks) and +[TruffleHog](https://github.com/dxa4481/truffleHog) checks. -NOTE: **Note:** -The Secret Detection analyzer will ignore "Password in URL" vulnerabilities if the password begins -with a dollar sign (`$`) as this likely indicates the password being used is an environment -variable. For example, `https://username:$password@example.com/path/to/repo` won't be -detected, whereas `https://username:password@example.com/path/to/repo` would be detected. +Note that the Secret Detection analyzer ignores Password-in-URL vulnerabilities if the password +begins with a dollar sign (`$`), as this likely indicates the password is an environment variable. +For example, `https://username:$password@example.com/path/to/repo` isn't detected, while +`https://username:password@example.com/path/to/repo` is. NOTE: **Note:** -You don't have to configure Secret Detection manually as shown in this section if you're using [Auto Secret Detection](../../../topics/autodevops/stages.md#auto-secret-detection) +You don't have to configure Secret Detection manually as shown in this section if you're using +[Auto Secret Detection](../../../topics/autodevops/stages.md#auto-secret-detection) provided by [Auto DevOps](../../../topics/autodevops/index.md). -To enable Secret Detection for GitLab 13.1 and later, you must include the `Secret-Detection.gitlab-ci.yml` template that’s provided as a part of your GitLab installation. For GitLab versions earlier than 11.9, you can copy and use the job as defined in that template. +To enable Secret Detection for GitLab 13.1 and later, you must include the +`Secret-Detection.gitlab-ci.yml` template that's provided as a part of your GitLab installation. For +GitLab versions earlier than 11.9, you can copy and use the job as defined in that template. Add the following to your `.gitlab-ci.yml` file: @@ -103,30 +102,6 @@ The results are saved as a that you can later download and analyze. Due to implementation limitations, we always take the latest Secret Detection artifact available. -### Using the SAST Template - -Prior to GitLab 13.1, Secret Detection was part of [SAST configuration](../sast#configuration). -If you already have SAST enabled for your app configured before GitLab 13.1, -you don't need to manually configure it. - -CAUTION: **Planned Deprecation:** -In a future GitLab release, configuring Secret Detection with the SAST template will be deprecated. Please begin using `Secret-Detection.gitlab-ci.yml` -to prevent future issues. We have made a -[video to guide you through the process of transitioning](https://www.youtube.com/watch?v=W2tjcQreDwQ) -to this new template. - -<div class="video-fallback"> - See the video: <a href="https://www.youtube.com/watch?v=W2tjcQreDwQ">Walkthrough of historical secret scan</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube.com/embed/W2tjcQreDwQ" frameborder="0" allowfullscreen="true"> </iframe> -</figure> - -When using the SAST template, Secret Detection is performed by a [specific analyzer](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml#L180) -during the `sast` job. It runs regardless of the programming -language of your app, and you don't need to change your -CI/CD configuration file to enable it. Results are available in the SAST report. - ### Customizing settings The Secret Detection scan settings can be changed through [environment variables](#available-variables) @@ -197,3 +172,9 @@ We have created a [short video walkthrough](https://youtu.be/wDtc_K00Y0A) showca <figure class="video-container"> <iframe src="https://www.youtube.com/embed/wDtc_K00Y0A" frameborder="0" allowfullscreen="true"> </iframe> </figure> + +## Troubleshooting + +### Getting warning message `gl-secret-detection-report.json: no matching files` + +For information on this, see the [general Application Security troubleshooting section](../../../ci/pipelines/job_artifacts.md#error-message-no-files-to-upload). diff --git a/doc/user/application_security/security_dashboard/img/group_vulnerability_report_v13_4.png b/doc/user/application_security/security_dashboard/img/group_vulnerability_report_v13_4.png Binary files differindex 67a7bb5f368..0310ef3ea0a 100644 --- a/doc/user/application_security/security_dashboard/img/group_vulnerability_report_v13_4.png +++ b/doc/user/application_security/security_dashboard/img/group_vulnerability_report_v13_4.png diff --git a/doc/user/application_security/security_dashboard/img/instance_security_center_settings_v13_4.png b/doc/user/application_security/security_dashboard/img/instance_security_center_settings_v13_4.png Binary files differnew file mode 100644 index 00000000000..4223494c294 --- /dev/null +++ b/doc/user/application_security/security_dashboard/img/instance_security_center_settings_v13_4.png diff --git a/doc/user/application_security/security_dashboard/img/instance_security_dashboard_empty_v13_4.png b/doc/user/application_security/security_dashboard/img/instance_security_dashboard_empty_v13_4.png Binary files differindex 3c618090be8..5edceb32e5c 100644 --- a/doc/user/application_security/security_dashboard/img/instance_security_dashboard_empty_v13_4.png +++ b/doc/user/application_security/security_dashboard/img/instance_security_dashboard_empty_v13_4.png diff --git a/doc/user/application_security/security_dashboard/img/instance_security_dashboard_v13_4.png b/doc/user/application_security/security_dashboard/img/instance_security_dashboard_v13_4.png Binary files differindex d010adcc90c..5379b5c6e5d 100644 --- a/doc/user/application_security/security_dashboard/img/instance_security_dashboard_v13_4.png +++ b/doc/user/application_security/security_dashboard/img/instance_security_dashboard_v13_4.png diff --git a/doc/user/application_security/security_dashboard/img/project_security_dashboard_dismissal_v13_4.png b/doc/user/application_security/security_dashboard/img/project_security_dashboard_dismissal_v13_4.png Binary files differnew file mode 100644 index 00000000000..eb1dfe6c6f4 --- /dev/null +++ b/doc/user/application_security/security_dashboard/img/project_security_dashboard_dismissal_v13_4.png diff --git a/doc/user/application_security/security_dashboard/img/project_security_dashboard_v13_3.png b/doc/user/application_security/security_dashboard/img/project_security_dashboard_v13_3.png Binary files differindex 34c64f830ba..adae37e0190 100644 --- a/doc/user/application_security/security_dashboard/img/project_security_dashboard_v13_3.png +++ b/doc/user/application_security/security_dashboard/img/project_security_dashboard_v13_3.png diff --git a/doc/user/application_security/security_dashboard/img/project_security_dashboard_v13_4.png b/doc/user/application_security/security_dashboard/img/project_security_dashboard_v13_4.png Binary files differnew file mode 100644 index 00000000000..ea4f188c80e --- /dev/null +++ b/doc/user/application_security/security_dashboard/img/project_security_dashboard_v13_4.png diff --git a/doc/user/application_security/security_dashboard/img/vulnerability_list_table_v13_4.png b/doc/user/application_security/security_dashboard/img/vulnerability_list_table_v13_4.png Binary files differindex eb91cfc47ad..760942c3239 100644 --- a/doc/user/application_security/security_dashboard/img/vulnerability_list_table_v13_4.png +++ b/doc/user/application_security/security_dashboard/img/vulnerability_list_table_v13_4.png diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index 51d9b4f45cd..f21f53e6895 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -5,21 +5,26 @@ group: Threat Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# GitLab Security Dashboard **(ULTIMATE)** +# GitLab Security Dashboard, Security Center, and Vulnerability Reports **(ULTIMATE)** -The Security Dashboard is a good place to get an overview of all the security -vulnerabilities in your groups, projects, and pipelines. +GitLab provides a comprehensive set of features for viewing and managing vulnerabilities: + +- Security dashboards: An overview of the security status in your instance, groups, and projects. +- Vulnerability reports: Detailed lists of all vulnerabilities for the instance, group, project, or + pipeline. This is where you triage and manage vulnerabilities. +- Security Center: A dedicated area for vulnerability management at the instance level. This + includes a security dashboard, vulnerability report, and settings. You can also drill down into a vulnerability and get extra information. This includes the project it comes from, any related file(s), and metadata that helps you analyze the risk it poses. You can also dismiss a vulnerability or create an issue for it. -To benefit from the Security Dashboard you must first configure one of the +To benefit from these features, you must first configure one of the [security scanners](../index.md). ## Supported reports -The Security Dashboard displays vulnerabilities detected by scanners such as: +The vulnerability report displays vulnerabilities detected by scanners such as: - [Container Scanning](../container_scanning/index.md) - [Dynamic Application Security Testing](../dast/index.md) @@ -29,7 +34,7 @@ The Security Dashboard displays vulnerabilities detected by scanners such as: ## Requirements -To use the instance, group, project, or pipeline security dashboard: +To use the security dashboards and vulnerability reports: 1. At least one project inside a group must be configured with at least one of the [supported reports](#supported-reports). @@ -41,15 +46,19 @@ To use the instance, group, project, or pipeline security dashboard: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13496) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.3. -At the pipeline level, the Security section displays the vulnerabilities present in the branch of the project the pipeline was run against. +At the pipeline level, the Security section displays the vulnerabilities present in the branch of +the project the pipeline ran against.  Visit the page for any pipeline that ran any of the [supported reports](#supported-reports). To view the pipeline's security findings, select the **Security** tab when viewing the pipeline. -NOTE: **Note:** -A pipeline consists of multiple jobs, including SAST and DAST scanning. If any job fails to finish for any reason, the security dashboard will not show SAST scanner output. For example, if the SAST job finishes but the DAST job fails, the security dashboard will not show SAST results. The analyzer will output an [exit code](../../../development/integrations/secure.md#exit-code) on failure. +A pipeline consists of multiple jobs, including SAST and DAST scanning. If any job fails to finish +for any reason, the security dashboard doesn't show SAST scanner output. For example, if the SAST +job finishes but the DAST job fails, the security dashboard doesn't show SAST results. On failure, +the analyzer outputs an +[exit code](../../../development/integrations/secure.md#exit-code). ## Project Security Dashboard @@ -65,7 +74,7 @@ Critical, High, Medium, Low, Info, Unknown). Below this, a table shows each vuln and description. Clicking a vulnerability takes you to its [Vulnerability Details](../vulnerabilities) page to view more information about that vulnerability. - + You can filter the vulnerabilities by one or more of the following: @@ -78,7 +87,7 @@ You can also dismiss vulnerabilities in the table: 1. Select the checkbox for each vulnerability you want to dismiss. 1. In the menu that appears, select the reason for dismissal and click **Dismiss Selected**. - + ## Group Security Dashboard @@ -86,79 +95,99 @@ You can also dismiss vulnerabilities in the table: The group Security Dashboard gives an overview of the vulnerabilities in the default branches of the projects in a group and its subgroups. Access it by navigating to **Security > Security Dashboard** -for your group. By default, the Security Dashboard displays all detected and confirmed -vulnerabilities. +after selecting your group. By default, the Security Dashboard displays all detected and confirmed +vulnerabilities. If you don't see the vulnerabilities over time graph, the likely cause is that you +have not selected a group. -NOTE: **Note:** -The Security Dashboard only shows projects with [security reports](#supported-reports) enabled in a -group. +Note that the Security Dashboard only shows projects with +[security reports](#supported-reports) +enabled in a group.  There is a timeline chart that shows how many open -vulnerabilities your projects had at various points in time. You can filter among 30, 60, and -90 days, with the default being 90. Hover over the chart to get more details about -the open vulnerabilities at a specific time. +vulnerabilities your projects had at various points in time. You can display the vulnerability +trends over a 30, 60, or 90-day time frame (the default is 90 days). Hover over the chart to get +more details about the open vulnerabilities at a specific time. Next to the timeline chart is a list of projects, grouped and sorted by the severity of the vulnerability found: -- F: 1 or more "critical" -- D: 1 or more "high" or "unknown" -- C: 1 or more "medium" -- B: 1 or more "low" -- A: 0 vulnerabilities +- F: One or more "critical" +- D: One or more "high" or "unknown" +- C: One or more "medium" +- B: One or more "low" +- A: Zero vulnerabilities Projects with no vulnerability tests configured will not appear in the list. Additionally, dismissed -vulnerabilities are not included either. +vulnerabilities are excluded. + +Navigate to the group's [vulnerability report](#vulnerability-report) to view the vulnerabilities found. -Navigate to the group's [Vulnerability Report](#vulnerability-list) to view the vulnerabilities found. +## Instance Security Center -## Instance Security Dashboard +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3426) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.4. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6953) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.8. +The Security Center is where you manage vulnerabilities for your instance. It displays the +vulnerabilities present in the default branches of all the projects you configure. It includes the +following: -At the instance level, the Security Dashboard displays the vulnerabilities present in the default -branches of all the projects you configure to display on the dashboard. It includes all the -[group Security Dashboard's](#group-security-dashboard) -features. +- The [group security dashboard's](#group-security-dashboard) features. +- A [vulnerability report](#vulnerability-report). +- A dedicated settings area to configure which projects to display.  -You can access the Instance Security Dashboard from the menu +You can access the Instance Security Center from the menu bar at the top of the page. Under **More**, select **Security**. - + -The dashboard is empty before you add projects to it. +The dashboard and vulnerability report are empty before you add projects. - + -### Adding projects to the dashboard +### Adding projects to the Security Center -To add projects to the dashboard: +To add projects to the Security Center: 1. Click **Settings** in the left navigation bar or click the **Add projects** button. 1. Search for and add one or more projects using the **Search your projects** field. 1. Click the **Add projects** button. -After you add projects, the Security Dashboard displays the vulnerabilities found in those projects' -default branches. + + +After you add projects, the security dashboard and vulnerability report display the vulnerabilities +found in those projects' default branches. ## Export vulnerabilities > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213014) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.10. -You can export all your vulnerabilities in CSV format by clicking the **{upload}** **Export** -button located at top right of the **Security Dashboard**. After the report -is built, the CSV report downloads to your local machine. The report contains all -vulnerabilities for the projects defined in the **Security Dashboard**, -as filters don't apply to the export function. - - +You can export all your vulnerabilities in CSV (comma separated values) format by clicking the +**{upload}** **Export** button located at top right of the Security Dashboard. When the report is +ready, the CSV report downloads to your local machine. The report contains all vulnerabilities for +the projects defined in the Security Dashboard, as filters don't apply to the export function. NOTE: **Note:** It may take several minutes for the download to start if your project contains -thousands of vulnerabilities. Do not close the page until the download finishes. +thousands of vulnerabilities. Don't close the page until the download finishes. + +The fields in the export include: + +- Group Name +- Project Name +- Scanner Type +- Scanner Name +- Status +- Vulnerability +- Details +- Additional Info +- Severity +- [CVE](https://cve.mitre.org/) +- [CWE](https://cwe.mitre.org/) +- Other Identifiers + + ## Keeping the dashboards up to date @@ -191,14 +220,14 @@ When using [Auto DevOps](../../../topics/autodevops/index.md), use [special environment variables](../../../topics/autodevops/customize.md#environment-variables) to configure daily security scans. -## Vulnerability list +## Vulnerability report -Each dashboard's vulnerability list contains vulnerabilities from the latest scans that were merged +Each vulnerability report contains vulnerabilities from the latest scans that were merged into the default branch.  -You can filter which vulnerabilities the Security Dashboard displays by: +You can filter which vulnerabilities the vulnerability report displays by: - Status - Severity @@ -211,8 +240,14 @@ To create an issue associated with the vulnerability, click the **Create Issue**  -Once you create the issue, the vulnerability list contains a link to the issue and an icon whose -color indicates the issue's status (green for open issues, blue for closed issues). +Once you create the issue, the linked issue icon in the vulnerability list: + +- Indicates that an issue has been created for that vulnerability. +- Shows a tooltip that contains a link to the issue and an icon whose + color indicates the issue's status: + + - Open issues: green + - Closed issues: blue  diff --git a/doc/user/application_security/terminology/index.md b/doc/user/application_security/terminology/index.md index 8006a49ba35..f975de213ef 100644 --- a/doc/user/application_security/terminology/index.md +++ b/doc/user/application_security/terminology/index.md @@ -13,7 +13,6 @@ This terminology list for GitLab Secure and Defend aims to: - Improve the effectiveness of communication regarding GitLab's application security features. - Get new contributors up to speed faster. -NOTE: **Note:** This document defines application security terms in the specific context of GitLab's Secure and Defend products. Terms may therefore have different meanings outside of GitLab Secure and Defend. diff --git a/doc/user/application_security/threat_monitoring/index.md b/doc/user/application_security/threat_monitoring/index.md index 5414800b290..391666a077e 100644 --- a/doc/user/application_security/threat_monitoring/index.md +++ b/doc/user/application_security/threat_monitoring/index.md @@ -105,11 +105,10 @@ disabled state. Once enabled,a predefined policy deploys to the selected environment's deployment platform and you can manage it like the regular policies. -NOTE: **Note:** -If you're using [Auto DevOps](../../../topics/autodevops/index.md) and -change a policy in this section, your `auto-deploy-values.yaml` file -doesn't update. Auto DevOps users must make changes by following -the [Container Network Policy documentation](../../../topics/autodevops/stages.md#network-policy). +Note that if you're using [Auto DevOps](../../../topics/autodevops/index.md) +and change a policy in this section, your `auto-deploy-values.yaml` file doesn't update. Auto DevOps +users must make changes by following the +[Container Network Policy documentation](../../../topics/autodevops/stages.md#network-policy). ### Changing enforcement status @@ -119,12 +118,9 @@ To change a network policy's enforcement status: - Click the **Enforcement status** toggle to update the selected policy. - Click the **Apply changes** button to deploy network policy changes. -NOTE: **Note:** -Disabled network policies have the -`network-policy.gitlab.com/disabled_by: gitlab` selector inside the -`podSelector` block. This narrows the scope of such a policy and as a -result it doesn't affect any pods. The policy itself is still deployed -to the corresponding deployment namespace. +Disabled network policies have the `network-policy.gitlab.com/disabled_by: gitlab` selector inside +the `podSelector` block. This narrows the scope of such a policy and as a result it doesn't affect +any pods. The policy itself is still deployed to the corresponding deployment namespace. ### Container Network Policy editor @@ -135,10 +131,8 @@ create a new policy click the **New policy** button located in the **Policy** tab's header. To edit an existing policy, click**Edit policy** in the selected policy drawer. -NOTE: **Note:** -The policy editor only supports the -[CiliumNetworkPolicy](https://docs.cilium.io/en/v1.8/policy/)specification. Regular -Kubernetes +Note that the policy editor only supports the +[CiliumNetworkPolicy](https://docs.cilium.io/en/v1.8/policy/)specification. Regular Kubernetes [NetworkPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#networkpolicy-v1-networking-k8s-io) resources aren't supported. diff --git a/doc/user/application_security/vulnerabilities/index.md b/doc/user/application_security/vulnerabilities/index.md index ff383fdf553..ee3fd6c4dd4 100644 --- a/doc/user/application_security/vulnerabilities/index.md +++ b/doc/user/application_security/vulnerabilities/index.md @@ -14,6 +14,7 @@ Each security vulnerability in a project's [Security Dashboard](../security_dash - Details of the vulnerability. - The status of the vulnerability within the project. - Available actions for the vulnerability. +- Issues related to the vulnerability. On the vulnerability page, you can interact with the vulnerability in several different ways: @@ -23,6 +24,7 @@ several different ways: - [Create issue](#creating-an-issue-for-a-vulnerability) - Create a new issue with the title and description pre-populated with information from the vulnerability report. By default, such issues are [confidential](../../project/issues/confidential_issues.md). +- [Link issues](#link-issues-to-the-vulnerability) - Link existing issues to vulnerability. - [Solution](#automatic-remediation-for-vulnerabilities) - For some vulnerabilities, a solution is provided for how to fix the vulnerability. @@ -38,6 +40,9 @@ the following values: | Dismissed | A user has seen this vulnerability and dismissed it | | Resolved | The vulnerability has been fixed and is no longer in the codebase | +A timeline shows you when the vulnerability status has changed, +and allows you to comment on a change. + ## Creating an issue for a vulnerability You can create an issue for a vulnerability by selecting the **Create issue** button. @@ -47,6 +52,12 @@ project the vulnerability came from, and pre-populates it with useful informatio the vulnerability report. After the issue is created, GitLab redirects you to the issue page so you can edit, assign, or comment on the issue. +## Link issues to the vulnerability + +You can link one or more existing issues to the vulnerability. This allows you to +indicate that this vulnerability affects multiple issues. It also allows you to indicate +that the resolution of one issue would resolve multiple vulnerabilities. + ## Automatic remediation for vulnerabilities You can fix some vulnerabilities by applying the solution that GitLab automatically diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index 7b745577cc4..4065ee3337b 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -6,18 +6,23 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Kubernetes Agent **(PREMIUM ONLY)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223061) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223061) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. +> - It's disabled on GitLab.com. Rolling this feature out to GitLab.com is [planned](https://gitlab.com/groups/gitlab-org/-/epics/3834). -## Goals +CAUTION: **Warning:** +This feature might not be available to you. Check the **version history** note above for details. -The [GitLab Kubernetes Agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) is an active in-cluster component for solving GitLab and Kubernetes integration tasks in a secure and cloud native way. +The [GitLab Kubernetes Agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) +is an active in-cluster component for solving GitLab and Kubernetes integration +tasks in a secure and cloud-native way. It enables: -Features: +- Integrating GitLab with a Kubernetes cluster behind a firewall or NAT + (network address translation). +- Pull-based GitOps deployments by leveraging the + [GitOps Engine](https://github.com/argoproj/gitops-engine). +- Real-time access to API endpoints within a cluster. -1. Makes it possible to integrate GitLab with a Kubernetes cluster behind a firewall or NAT -1. Enables pull-based GitOps deployments by leveraging the [GitOps Engine](https://github.com/argoproj/gitops-engine) -1. Allows for real-time access to API endpoints within a cluster. -1. Many more features are planned. Please [review our roadmap](https://gitlab.com/groups/gitlab-org/-/epics/3329). +Many more features are planned. Please [review our roadmap](https://gitlab.com/groups/gitlab-org/-/epics/3329). ## Architecture @@ -39,37 +44,45 @@ sequenceDiagram end ``` -Please refer to our [full architecture documentation in the Agent project](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md#high-level-architecture). +Please refer to our [full architecture documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md#high-level-architecture) +in the Agent project. -## Getting started with GitOps using the GitLab Agent and the GitLab Cloud Native Helm chart +## Get started with GitOps and the GitLab Agent There are several components that work in concert for the Agent to accomplish GitOps deployments: -1. A Kubernetes cluster that is properly configured -1. A configuration repository that contains a `config.yaml` file. This `config.yaml` tells the Agent which repositories to synchronize with. -1. A manifest repository that contains a `manifest.yaml`. This `manifest.yaml` (which can be autogenerated) is tracked by the Agent and any changes to the file are automatically applied to the cluster. +- A properly-configured Kubernetes cluster. +- A configuration repository that contains a `config.yaml` file, which tells the + Agent which repositories to synchronize with. +- A manifest repository that contains a `manifest.yaml`, which is tracked by the + Agent and can be auto-generated. Any changes to `manifest.yaml` are applied to the cluster. -The setup process involves a few steps that, once completed, will enable GitOps deployments to work +The setup process involves a few steps to enable GitOps deployments: -1. Installing the Agent server via GitLab Helm chart -1. Defining a configuration directory -1. Creating an Agent record in GitLab -1. Generating and copying a Secret token used to connect to the Agent -1. Installing the Agent into the cluster -1. Creating a `manifest.yaml` +1. Installing the Agent server via GitLab Helm chart. +1. Defining a configuration directory. +1. Creating an Agent record in GitLab. +1. Generating and copying a Secret token used to connect to the Agent. +1. Installing the Agent into the cluster. +1. Creating a `manifest.yaml`. -### Installing the Agent server via Helm +### Install the Agent server -Currently the GitLab Kubernetes Agent can only be deployed via our [Helm chart](https://gitlab.com/gitlab-org/charts/gitlab). +The GitLab Kubernetes Agent can only be deployed through our +[Helm chart](https://gitlab.com/gitlab-org/charts/gitlab). If you don't already +have GitLab installed via Helm, please refer to our +[installation documentation](https://docs.gitlab.com/charts/installation/). -NOTE: We are working quickly to [include the Agent in Official Linux Package](https://gitlab.com/gitlab-org/gitlab/-/issues/223060). +NOTE: **Note:** +GitLab plans to include the Agent in the [official Linux Package](https://gitlab.com/gitlab-org/gitlab/-/issues/223060) and on [GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/3834). -If you don't already have GitLab installed via Helm please refer to our [installation documentation](https://docs.gitlab.com/charts/installation/) - -When installing/upgrading the GitLab Helm chart please consider the following Helm 2 example (if using Helm 3 please modify): +When installing or upgrading the GitLab Helm chart, consider the following Helm 2 example. +(If you're using Helm 3, you must modify this example.) You must set `global.kas.enabled=true` +for the Agent to be properly installed and configured: ```shell -helm upgrade --force --install gitlab . \ +helm repo update +helm upgrade --force --install gitlab gitlab/gitlab \ --timeout 600 \ --set global.hosts.domain=<YOUR_DOMAIN> \ --set global.hosts.externalIP=<YOUR_IP> \ @@ -78,15 +91,14 @@ helm upgrade --force --install gitlab . \ --set global.kas.enabled=true ``` -`global.kas.enabled=true` must be set in order for the Agent to be properly installed and configured. - -### Defining a configuration repository +### Define a configuration repository -Next you will need a GitLab repository that will contain your Agent configuration. +Next, you need a GitLab repository to contain your Agent configuration. The minimal +repository layout looks like this: -The minimal repository layout looks like this: - -`.gitlab/agents/<agent-name>/config.yaml` +```plaintext +.gitlab/agents/<agent-name>/config.yaml +``` The `config.yaml` file contents should look like this: @@ -96,31 +108,24 @@ gitops: - id: "path-to/your-awesome-project" ``` -### Creating an Agent record in GitLab - -Next you will need to create an GitLab Rails Agent record so that your GitLab project so that the Agent itself can associate with a GitLab project. This process will also yield a Secret that you will use to configure the Agent in subsequent steps. +### Create an Agent record in GitLab -There are two ways to accomplish this: +Next, create an GitLab Rails Agent record so the Agent can associate itself with +a GitLab project. Creating this record also creates a Secret needed to configure +the Agent in subsequent steps. You can create an Agent record either: -1. Via the Rails console -1. Via GraphQL +- Through the Rails console, by running `rails c`: -To do this you could either run `rails c` or via GraphQL. From `rails c`: - -```ruby + ```ruby project = ::Project.find_by_full_path("path-to/your-awesome-project") agent = ::Clusters::Agent.create(name: "<agent-name>", project: project) token = ::Clusters::AgentToken.create(agent: agent) token.token # this will print out the token you need to use on the next step -``` + ``` -or using GraphQL: +- Through GraphQL: **(PREMIUM ONLY)** -with this approach, you'll need a premium license to use this feature. - -If you are new to using the GitLab GraphQL API please refer to the [Getting started with the GraphQL API page](../../../api/graphql/getting_started.md) or check out the [GraphQL Explorer](https://gitlab.com/-/graphql-explorer). - -```json + ```graphql mutation createAgent { createClusterAgent(input: { projectPath: "path-to/your-awesome-project", name: "<agent-name>" }) { clusterAgent { @@ -141,37 +146,77 @@ If you are new to using the GitLab GraphQL API please refer to the [Getting star errors } } -``` + ``` -Note that GraphQL will only show you the token once, after you've created it. + NOTE: **Note:** + GraphQL only displays the token once, after creating it. -### Creating the Kubernetes secret + If you are new to using the GitLab GraphQL API, refer to the + [Getting started with the GraphQL API page](../../../api/graphql/getting_started.md), + or the [GraphQL Explorer](https://gitlab.com/-/graphql-explorer). -Once the token has been generated it needs to be applied to the Kubernetes cluster. +### Create the Kubernetes secret -If you didn't previously define or create a namespace you need to do that first: +After generating the token, you must apply it to the Kubernetes cluster. -```shell -kubectl create namespace <YOUR-DESIRED-NAMESPACE> -``` +1. If you haven't previous defined or created a namespace, run the following command: + + ```shell + kubectl create namespace <YOUR-DESIRED-NAMESPACE> + ``` + +1. Run the following command to create your Secret: -Run the following command to create your Secret: + ```shell + kubectl create secret generic -n <YOUR-DESIRED-NAMESPACE> gitlab-agent-token --from-literal=token='YOUR_AGENT_TOKEN' + ``` + +### Install the Agent into the cluster + +Next, install the in-cluster component of the Agent. This example file contains the +Kubernetes resources required for the Agent to be installed. You can modify this +example [`resources.yml` file](#example-resourcesyml-file) in the following ways: + +- You can replace `gitlab-agent` with `<YOUR-DESIRED-NAMESPACE>`. +- For the `kas-address` (Kubernetes Agent Server), the agent can use the WebSockets + or gRPC protocols to connect to the Agent Server. Depending on your cluster + configuration and GitLab architecture, you may need to use one or the other. + For the `gitlab-kas` Helm chart, an Ingress is created for the Agent Server using + the `/-/kubernetes-agent` endpoint. This can be used for the WebSockets protocol connection. + - Specify the `grpc` scheme (such as `grpc://gitlab-kas:5005`) to use gRPC directly. + Encrypted gRPC is not supported yet. Follow the + [Support TLS for gRPC communication issue](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/7) + for progress updates. + - Specify the `ws` scheme (such as `ws://gitlab-kas-ingress:80/-/kubernetes-agent`) + to use an unencrypted WebSockets connection. + - Specify the `wss` scheme (such as `wss://gitlab-kas-ingress:443/-/kubernetes-agent`) + to use an encrypted WebSockets connection. This is the recommended option if + installing the Agent into a separate cluster from your Agent Server. +- If you defined your own secret name, replace `gitlab-agent-token` with your secret name. + +To apply this file, run the following command: ```shell -kubectl create secret generic -n <YOUR-DESIRED-NAMESPACE> gitlab-agent-token --from-literal=token='YOUR_AGENT_TOKEN' +kubectl apply -n gitlab-agent -f ./resources.yml ``` -### Installing the Agent into the cluster - -Next you are now ready to install the in-cluster component of the Agent. The below is an example YAML file of the Kubernetes resources required for the Agent to be installed. +To review your configuration, run the following command: -Let's highlight a few of the details in the example below: +```shell +$ kubectl get pods --all-namespaces -1. You can replace `gitlab-agent` with <YOUR-DESIRED-NAMESPACE> -1. For the `kas-address` (Kubernetes Agent Server), you can replace `grpc://host.docker.internal:5005` with the address of the kas agent that was initialized via your Helm install. -1. If you defined your own secret name, then replace `gitlab-agent-token` with your secret name. +NAMESPACE NAME READY STATUS RESTARTS AGE +gitlab-agent gitlab-agent-77689f7dcb-5skqk 1/1 Running 0 51s +kube-system coredns-f9fd979d6-n6wcw 1/1 Running 0 14m +kube-system etcd-minikube 1/1 Running 0 14m +kube-system kube-apiserver-minikube 1/1 Running 0 14m +kube-system kube-controller-manager-minikube 1/1 Running 0 14m +kube-system kube-proxy-j6zdh 1/1 Running 0 14m +kube-system kube-scheduler-minikube 1/1 Running 0 14m +kube-system storage-provisioner 1/1 Running 0 14m +``` -`./resources.yml` +#### Example `resources.yml` file ```yaml apiVersion: v1 @@ -200,7 +245,7 @@ spec: args: - --token-file=/config/token - --kas-address - - grpc://host.docker.internal:5005 # {"$openapi":"kas-address"} + - grpc://host.docker.internal:5005 # {"$openapi":"kas-address"} volumeMounts: - name: token-volume mountPath: /config @@ -268,37 +313,31 @@ subjects: - name: gitlab-agent kind: ServiceAccount namespace: gitlab-agent - ``` -```shell -kubectl apply -n gitlab-agent -f ./resources.yml -``` +### Create a `manifest.yaml` + +In a previous step, you configured a `config.yaml` to point to the GitLab projects +the Agent should synchronize. In each of those projects, you must create a `manifest.yaml` +file for the Agent to monitor. You can auto-generate this `manifest.yaml` with a +templating engine or other means. + +Each time you commit and push a change to this file, the Agent logs the change: ```plaintext -$ kubectl get pods --all-namespaces -NAMESPACE NAME READY STATUS RESTARTS AGE -gitlab-agent gitlab-agent-77689f7dcb-5skqk 1/1 Running 0 51s -kube-system coredns-f9fd979d6-n6wcw 1/1 Running 0 14m -kube-system etcd-minikube 1/1 Running 0 14m -kube-system kube-apiserver-minikube 1/1 Running 0 14m -kube-system kube-controller-manager-minikube 1/1 Running 0 14m -kube-system kube-proxy-j6zdh 1/1 Running 0 14m -kube-system kube-scheduler-minikube 1/1 Running 0 14m -kube-system storage-provisioner 1/1 Running 0 14m +2020-09-15_14:09:04.87946 gitlab-k8s-agent : time="2020-09-15T10:09:04-04:00" level=info msg="Config: new commit" agent_id=1 commit_id=e6a3651f1faa2e928fe6120e254c122451be4eea ``` -### Creating a `manifest.yaml` - -In the above step, you configured a `config.yaml` to point to which GitLab projects the Agent should synchronize. Within each one of those projects, you need to create a `manifest.yaml` file which the Agent will monitor. This `manifest.yaml` can be autogenerated by a templating engine or other means. +#### Example `manifest.yaml` file -Example `manifest.yaml`: +This file creates a simple NGINX deployment. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment + namespace: gitlab-agent # Can be any namespace managed by you that the agent has access to. spec: selector: matchLabels: @@ -316,14 +355,9 @@ spec: - containerPort: 80 ``` -The above file creates a simple NGINX deployment. - -Each time you commit and push a change to the `manifest.yaml` the Agent will observe the change. Example log: - -```plaintext -2020-09-15_14:09:04.87946 gitlab-k8s-agent : time="2020-09-15T10:09:04-04:00" level=info msg="Config: new commit" agent_id=1 commit_id=e6a3651f1faa2e928fe6120e254c122451be4eea -``` - ## Example projects -Basic GitOps example deploying NGINX: [Configuration repository](https://gitlab.com/gitlab-org/configure/examples/kubernetes-agent), [Manifest repository](https://gitlab.com/gitlab-org/configure/examples/gitops-project) +This basic GitOps example deploys NGINX: + +- [Configuration repository](https://gitlab.com/gitlab-org/configure/examples/kubernetes-agent) +- [Manifest repository](https://gitlab.com/gitlab-org/configure/examples/gitops-project) diff --git a/doc/user/clusters/applications.md b/doc/user/clusters/applications.md index 2243ffa0cb1..c3b22526919 100644 --- a/doc/user/clusters/applications.md +++ b/doc/user/clusters/applications.md @@ -6,37 +6,29 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Managed Apps -GitLab provides **GitLab Managed Apps**, a one-click install for various applications which can -be added directly to your configured cluster. - -These applications are needed for [Review Apps](../../ci/review_apps/index.md) -and [deployments](../../ci/environments/index.md) when using [Auto DevOps](../../topics/autodevops/index.md). - -You can install them after you -[create a cluster](../project/clusters/add_remove_clusters.md). +GitLab provides **GitLab Managed Apps**, a one-click install for various +applications which can be added directly to your configured cluster. These +applications are needed for [Review Apps](../../ci/review_apps/index.md) and +[deployments](../../ci/environments/index.md) when using [Auto DevOps](../../topics/autodevops/index.md). +You can install them after you [create a cluster](../project/clusters/add_remove_clusters.md). ## Installing applications -Applications managed by GitLab will be installed onto the `gitlab-managed-apps` namespace. - -This namespace: +Applications managed by GitLab are installed onto the `gitlab-managed-apps` +namespace. This namespace: - Is different from the namespace used for project deployments. - Is created once. - Has a non-configurable name. -To see a list of available applications to install. For a: +To view a list of available applications to install for a: - [Project-level cluster](../project/clusters/index.md), navigate to your project's **Operations > Kubernetes**. - [Group-level cluster](../group/clusters/index.md), navigate to your group's **Kubernetes** page. -NOTE: **Note:** -As of GitLab 11.6, Helm will be upgraded to the latest version supported -by GitLab before installing any of the applications. - -The following applications can be installed: +You can install the following applications: - [Helm](#helm) - [Ingress](#ingress) @@ -49,10 +41,9 @@ The following applications can be installed: - [Elastic Stack](#elastic-stack) - [Fluentd](#fluentd) -With the exception of Knative, the applications will be installed in a dedicated +With the exception of Knative, the applications are installed in a dedicated namespace called `gitlab-managed-apps`. -NOTE: **Note:** Some applications are installable only for a project-level cluster. Support for installing these applications in a group-level cluster is planned for future releases. @@ -65,6 +56,9 @@ you should be careful as GitLab cannot detect it. In this case, installing Helm via the applications will result in the cluster having it twice, which can lead to confusion during deployments. +In GitLab versions 11.6 and greater, Helm is upgraded to the latest version +supported by GitLab before installing any of the applications. + ### Helm > - Introduced in GitLab 10.2 for project-level clusters. @@ -81,7 +75,6 @@ applications. Prior to [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/issu GitLab used an in-cluster Tiller server in the `gitlab-managed-apps` namespace. This server can now be safely removed. -NOTE: **Note:** GitLab's Helm integration does not support installing applications behind a proxy, but a [workaround](../../topics/autodevops/index.md#install-applications-behind-a-proxy) is available. @@ -90,26 +83,25 @@ is available. > Introduced in GitLab 11.6 for project- and group-level clusters. -[cert-manager](https://cert-manager.io/docs/) is a native -Kubernetes certificate management controller that helps with issuing -certificates. Installing cert-manager on your cluster will issue a -certificate by [Let's Encrypt](https://letsencrypt.org/) and ensure that -certificates are valid and up-to-date. +[cert-manager](https://cert-manager.io/docs/) is a native Kubernetes certificate +management controller that helps with issuing certificates. Installing +cert-manager on your cluster issues a certificate by [Let's Encrypt](https://letsencrypt.org/) +and ensures that certificates are valid and up-to-date. The chart used to install this application depends on the version of GitLab used. In: -- GitLab 12.3 and newer, the [jetstack/cert-manager](https://github.com/jetstack/cert-manager) +- GitLab 12.3 and newer, the [`jetstack/cert-manager`](https://github.com/jetstack/cert-manager) chart is used with a [`values.yaml`](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/cert_manager/values.yaml) file. -- GitLab 12.2 and older, the [stable/cert-manager](https://github.com/helm/charts/tree/master/stable/cert-manager) +- GitLab 12.2 and older, the [`stable/cert-manager`](https://gi2wthub.com/helm/charts/tree/master/stable/cert-manager) chart was used. -If you have installed cert-manager prior to GitLab 12.3, Let's Encrypt will -[block requests from older versions of cert-manager](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753). +If you installed cert-manager prior to GitLab 12.3, Let's Encrypt +[blocks requests](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753) +from older versions of `cert-manager`. To resolve this: -To resolve this: - -1. Uninstall cert-manager (consider [backing up any additional configuration](https://cert-manager.io/docs/tutorials/backup/)). +1. [Back up any additional configuration](https://cert-manager.io/docs/tutorials/backup/). +1. Uninstall cert-manager. 1. Install cert-manager again. ### GitLab Runner @@ -117,26 +109,21 @@ To resolve this: > - Introduced in GitLab 10.6 for project-level clusters. > - Introduced in GitLab 11.10 for group-level clusters. -[GitLab Runner](https://docs.gitlab.com/runner/) is the open source -project that is used to run your jobs and send the results back to -GitLab. It is used in conjunction with [GitLab -CI/CD](../../ci/README.md), the open-source continuous integration -service included with GitLab that coordinates the jobs. - -If the project is on GitLab.com, shared runners are available -(the first 2000 minutes are free, you can -[buy more later](../../subscriptions/gitlab_com/index.md#purchase-additional-ci-minutes)) -and you do not have to deploy one if they are enough for your needs. If a -project-specific runner is desired, or there are no shared runners, it is easy -to deploy one. - -Note that the deployed runner will be set as **privileged**, which means it will essentially -have root access to the underlying machine. This is required to build Docker images, -so it is the default. Make sure you read the -[security implications](../project/clusters/index.md#security-implications) +[GitLab Runner](https://docs.gitlab.com/runner/) is the open source project that +is used to run your jobs and send the results back to GitLab. It's used in +conjunction with [GitLab CI/CD](../../ci/README.md), the open-source continuous +integration service included with GitLab that coordinates the jobs. + +If the project is on GitLab.com, [shared runners](../gitlab_com/index.md#shared-runners) +are available. You don't have to deploy one if they are enough for your +needs. If a project-specific runner is desired, or there are no shared runners, +you can deploy one. + +The deployed runner is set as **privileged**. Root access to the underlying +server is required to build Docker images, so it is the default. Be sure to read +the [security implications](../project/clusters/index.md#security-implications) before deploying one. -NOTE: **Note:** The [`runner/gitlab-runner`](https://gitlab.com/gitlab-org/charts/gitlab-runner) chart is used to install this application, using [a preconfigured `values.yaml`](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/master/values.yaml) @@ -147,11 +134,13 @@ file. Customizing the installation by modifying this file is not supported. > - Introduced in GitLab 10.2 for project-level clusters. > - Introduced in GitLab 11.6 for group-level clusters. -[Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) provides load balancing, SSL termination, and name-based virtual hosting +[Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) +provides load balancing, SSL termination, and name-based virtual hosting out of the box. It acts as a web proxy for your applications and is useful if you want to use [Auto DevOps](../../topics/autodevops/index.md) or deploy your own web apps. -The Ingress Controller installed is [Ingress-NGINX](https://kubernetes.io/docs/concepts/services-networking/ingress/), +The Ingress Controller installed is +[Ingress-NGINX](https://kubernetes.io/docs/concepts/services-networking/ingress/), which is supported by the Kubernetes community. NOTE: **Note:** @@ -159,10 +148,10 @@ With the following procedure, a load balancer must be installed in your cluster to obtain the endpoint. You can use either Ingress, or Knative's own load balancer ([Istio](https://istio.io)) if using Knative. -In order to publish your web application, you first need to find the endpoint which will be either an IP +To publish your web application, you first need to find the endpoint, which is either an IP address or a hostname associated with your load balancer. -To install it, click on the **Install** button for Ingress. GitLab will attempt +To install it, click on the **Install** button for Ingress. GitLab attempts to determine the external endpoint and it should be available within a few minutes. #### Determining the external endpoint automatically @@ -178,11 +167,15 @@ using the `KUBE_INGRESS_BASE_DOMAIN` environment variable. If the endpoint doesn't appear and your cluster runs on Google Kubernetes Engine: -1. Check your [Kubernetes cluster on Google Kubernetes Engine](https://console.cloud.google.com/kubernetes) to ensure there are no errors on its nodes. -1. Ensure you have enough [Quotas](https://console.cloud.google.com/iam-admin/quotas) on Google Kubernetes Engine. For more information, see [Resource Quotas](https://cloud.google.com/compute/quotas). -1. Check [Google Cloud's Status](https://status.cloud.google.com/) to ensure they are not having any disruptions. +1. [Examine your Kubernetes cluster](https://console.cloud.google.com/kubernetes) + on Google Kubernetes Engine to ensure there are no errors on its nodes. +1. Ensure you have enough [Quotas](https://console.cloud.google.com/iam-admin/quotas) + on Google Kubernetes Engine. For more information, see + [Resource Quotas](https://cloud.google.com/compute/quotas). +1. Review [Google Cloud's Status](https://status.cloud.google.com/) for service + disruptions. -Once installed, you may see a `?` for "Ingress IP Address" depending on the +After installing, you may see a `?` for **Ingress IP Address** depending on the cloud provider. For EKS specifically, this is because the ELB is created with a DNS name, not an IP address. If GitLab is still unable to determine the endpoint of your Ingress or Knative application, you can @@ -208,58 +201,58 @@ The output of the following examples will show the external endpoint of your cluster. This information can then be used to set up DNS entries and forwarding rules that allow external access to your deployed applications. -If you installed Ingress via the **Applications**, run the following command: +- If you installed Ingress using the **Applications**, run the following + command: -```shell -kubectl get service --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' -``` + ```shell + kubectl get service --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' + ``` -Some Kubernetes clusters return a hostname instead, like [Amazon EKS](https://aws.amazon.com/eks/). For these platforms, run: +- Some Kubernetes clusters return a hostname instead, like + [Amazon EKS](https://aws.amazon.com/eks/). For these platforms, run: -```shell -kubectl get service --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' -``` + ```shell + kubectl get service --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' + ``` -For Istio/Knative, the command will be different: + If EKS is used, an [Elastic Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/) + is also created, which will incur additional AWS costs. -```shell -kubectl get svc --namespace=istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip} ' -``` +- For Istio/Knative, the command will be different: -Otherwise, you can list the IP addresses of all load balancers: + ```shell + kubectl get svc --namespace=istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip} ' + ``` -```shell -kubectl get svc --all-namespaces -o jsonpath='{range.items[?(@.status.loadBalancer.ingress)]}{.status.loadBalancer.ingress[*].ip} ' -``` +- Otherwise, you can list the IP addresses of all load balancers: -NOTE: **Note:** -If EKS is used, an [Elastic Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/) -will also be created, which will incur additional AWS costs. + ```shell + kubectl get svc --all-namespaces -o jsonpath='{range.items[?(@.status.loadBalancer.ingress)]}{.status.loadBalancer.ingress[*].ip} ' + ``` -NOTE: **Note:** -You may see a trailing `%` on some Kubernetes versions, **do not include it**. +You may see a trailing `%` on some Kubernetes versions. Do not include it. -The Ingress is now available at this address and will route incoming requests to -the proper service based on the DNS name in the request. To support this, a -wildcard DNS CNAME record should be created for the desired domain name. For example, +The Ingress is now available at this address, and routes incoming requests to +the proper service based on the DNS name in the request. To support this, create +a wildcard DNS CNAME record for the desired domain name. For example, `*.myekscluster.com` would point to the Ingress hostname obtained earlier. #### Using a static IP By default, an ephemeral external IP address is associated to the cluster's load balancer. If you associate the ephemeral IP with your DNS and the IP changes, -your apps will not be able to be reached, and you'd have to change the DNS -record again. In order to avoid that, you should change it into a static -reserved IP. +your apps won't be reachable, and you'd have to change the DNS record again. +To avoid that, change it into a static reserved IP. Read how to [promote an ephemeral external IP address in GKE](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#promote_ephemeral_ip). #### Pointing your DNS at the external endpoint -Once you've set up the external endpoint, you should associate it with a [wildcard DNS -record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) such as `*.example.com.` -in order to be able to reach your apps. If your external endpoint is an IP address, -use an A record. If your external endpoint is a hostname, use a CNAME record. +After you have set up the external endpoint, associate it with a +[wildcard DNS record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) (such +as `*.example.com.`) to reach your apps. If your external endpoint is an IP +address, use an A record. If your external endpoint is a hostname, use a CNAME +record. #### Web Application Firewall (ModSecurity) @@ -269,16 +262,15 @@ A Web Application Firewall (WAF) examines traffic being sent or received, and can block malicious traffic before it reaches your application. The benefits of a WAF are: -- Real-time security monitoring for your application -- Logging of all your HTTP traffic to the application -- Access control for your application -- Highly configurable logging and blocking rules +- Real-time security monitoring for your application. +- Logging of all your HTTP traffic to the application. +- Access control for your application. +- Highly configurable logging and blocking rules. -Out of the box, GitLab provides you with a WAF known as [`ModSecurity`](https://www.modsecurity.org/). - -ModSecurity is a toolkit for real-time web application monitoring, logging, -and access control. With GitLab's offering, the [OWASP's Core Rule Set](https://www.modsecurity.org/CRS/Documentation/), -which provides generic attack detection capabilities, is automatically applied. +By default, GitLab provides you with a WAF known as [`ModSecurity`](https://www.modsecurity.org/), +which is a toolkit for real-time web application monitoring, logging, and access +control. GitLab's offering applies the [OWASP's Core Rule Set](https://www.modsecurity.org/CRS/Documentation/), +which provides generic attack detection capabilities. This feature: @@ -299,57 +291,61 @@ There is a small performance overhead by enabling ModSecurity. If this is considered significant for your application, you can disable ModSecurity's rule engine for your deployed application in any of the following ways: -1. Setting [the deployment variable](../../topics/autodevops/index.md) -`AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE` to `Off`. This will prevent ModSecurity -from processing any requests for the given application or environment. - -1. Switching its respective toggle to the disabled position and applying changes through the **Save changes** button. This will reinstall -Ingress with the recent changes. +1. Set [the deployment variable](../../topics/autodevops/index.md) + `AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE` to `Off` to prevent ModSecurity + from processing any requests for the given application or environment. +1. Switch its respective toggle to the disabled position, and then apply changes + by selecting **Save changes** to reinstall Ingress with the recent changes.  ##### Logging and blocking modes To help you tune your WAF rules, you can globally set your WAF to either -**Logging** or **Blocking** mode: +*Logging* or *Blocking* mode: -- **Logging mode** - Allows traffic matching the rule to pass, and logs the event. -- **Blocking mode** - Prevents traffic matching the rule from passing, and logs the event. +- *Logging mode*: Allows traffic matching the rule to pass, and logs the event. +- *Blocking mode*: Prevents traffic matching the rule from passing, and logs the event. To change your WAF's mode: -1. [Install ModSecurity](../../topics/web_application_firewall/quick_start_guide.md) if you have not already done so. +1. If you haven't already done so, [install ModSecurity](../../topics/web_application_firewall/quick_start_guide.md). 1. Navigate to **Operations > Kubernetes**. 1. In **Applications**, scroll to **Ingress**. 1. Under **Global default**, select your desired mode. -1. Click **Save changes**. +1. Select **Save changes**. ##### WAF version updates -Enabling, disabling, or changing the logging mode for **ModSecurity** is only allowed within same version of [Ingress](#ingress) due to limitations in [Helm](https://helm.sh/) which might be overcome in future releases. +Enabling, disabling, or changing the logging mode for **ModSecurity** is only +allowed within same version of [Ingress](#ingress) due to limitations in +[Helm](https://helm.sh/) which might be overcome in future releases. -**ModSecurity** UI controls are disabled if the version deployed differs from the one available in GitLab, while actions at the [Ingress](#ingress) level, such as uninstalling, can still be performed: +**ModSecurity** user interface controls are disabled if the version deployed +differs from the one available in GitLab, while actions at the [Ingress](#ingress) +level, such as uninstalling, can still be performed:  -Updating [Ingress](#ingress) to the most recent version enables you to take advantage of bug fixes, security fixes, and performance improvements. To update [Ingress application](#ingress), you must first uninstall it, and then re-install it as described in [Install ModSecurity](../../topics/web_application_firewall/quick_start_guide.md). +Update [Ingress](#ingress) to the most recent version to take advantage of bug +fixes, security fixes, and performance improvements. To update the +[Ingress application](#ingress), you must first uninstall it, and then re-install +it as described in [Install ModSecurity](../../topics/web_application_firewall/quick_start_guide.md). ##### Viewing Web Application Firewall traffic > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14707) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.9. You can view Web Application Firewall traffic by navigating to your project's -**Security & Compliance > Threat Monitoring** page. - -From there, you can see tracked over time: +**Security & Compliance > Threat Monitoring** page. From there, you can see +tracked over time: - The total amount of traffic to your application. - The proportion of traffic that is considered anomalous by the Web Application Firewall's default [OWASP ruleset](https://www.modsecurity.org/CRS/Documentation/). -If a significant percentage of traffic is anomalous, it should be investigated -for potential threats, which can be done by -[examining the Web Application Firewall logs](#web-application-firewall-modsecurity). +If a significant percentage of traffic is anomalous, investigate it for potential threats +by [examining the Web Application Firewall logs](#web-application-firewall-modsecurity).  @@ -358,55 +354,51 @@ for potential threats, which can be done by > - Introduced in GitLab 11.0 for project-level clusters. > - Introduced in GitLab 12.3 for group and instance-level clusters. -[JupyterHub](https://jupyterhub.readthedocs.io/en/stable/) is a -multi-user service for managing notebooks across a team. [Jupyter -Notebooks](https://jupyter-notebook.readthedocs.io/en/latest/) provide a -web-based interactive programming environment used for data analysis, +[JupyterHub](https://jupyterhub.readthedocs.io/en/stable/) is a multi-user service +for managing notebooks across a team. [Jupyter Notebooks](https://jupyter-notebook.readthedocs.io/en/latest/) +provide a web-based interactive programming environment used for data analysis, visualization, and machine learning. -Authentication will be enabled only for [project -members](../project/members/index.md) for project-level clusters and group -members for group-level clusters with [Developer or -higher](../permissions.md) access to the associated project or group. +The [`jupyter/jupyterhub`](https://jupyterhub.github.io/helm-chart/) +chart is used to install this application with a +[`values.yaml`](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/jupyter/values.yaml) +file. + +Authentication is enabled only for [project members](../project/members/index.md) +for project-level clusters and group members for group-level clusters with +[Developer or higher](../permissions.md) access to the associated project or group. -We use a [custom Jupyter -image](https://gitlab.com/gitlab-org/jupyterhub-user-image/blob/master/Dockerfile) +GitLab uses a [custom Jupyter image](https://gitlab.com/gitlab-org/jupyterhub-user-image/blob/master/Dockerfile) that installs additional useful packages on top of the base Jupyter. You will also see ready-to-use DevOps Runbooks built with Nurtch's [Rubix library](https://github.com/Nurtch/rubix). -More information on -creating executable runbooks can be found in [our Runbooks -documentation](../project/clusters/runbooks/index.md#configure-an-executable-runbook-with-gitlab). Note that +More information on creating executable runbooks can be found in +[our Runbooks documentation](../project/clusters/runbooks/index.md#configure-an-executable-runbook-with-gitlab). Ingress must be installed and have an IP address assigned before JupyterHub can be installed. -NOTE: **Note:** -The [`jupyter/jupyterhub`](https://jupyterhub.github.io/helm-chart/) -chart is used to install this application with a -[`values.yaml`](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/jupyter/values.yaml) -file. - #### Jupyter Git Integration > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/28783) in GitLab 12.0 for project-level clusters. > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/32512) in GitLab 12.3 for group and instance-level clusters. -When installing JupyterHub onto your Kubernetes cluster, [JupyterLab's Git extension](https://github.com/jupyterlab/jupyterlab-git) -is automatically provisioned and configured using the authenticated user's: +When installing JupyterHub onto your Kubernetes cluster, +[JupyterLab's Git extension](https://github.com/jupyterlab/jupyterlab-git) +is provisioned and configured using the authenticated user's: - Name. - Email. - Newly created access token. -JupyterLab's Git extension enables full version control of your notebooks as well as issuance of Git commands within Jupyter. -Git commands can be issued via the **Git** tab on the left panel or via Jupyter's command line prompt. +JupyterLab's Git extension enables full version control of your notebooks, and +issuance of Git commands within Jupyter. You can issue Git commands through the +**Git** tab on the left panel, or through Jupyter's command-line prompt. -NOTE: **Note:** -JupyterLab's Git extension stores the user token in the JupyterHub DB in encrypted format -and in the single user Jupyter instance as plain text. This is because [Git requires storing -credentials as plain text](https://git-scm.com/docs/git-credential-store). Potentially, if -a nefarious user finds a way to read from the file system in the single user Jupyter instance -they could retrieve the token. +JupyterLab's Git extension stores the user token in the JupyterHub DB in encrypted +format, and in the single user Jupyter instance as plain text, because +[Git requires storing credentials as plain text](https://git-scm.com/docs/git-credential-store) +Potentially, if a nefarious user finds a way to read from the file system in the +single-user Jupyter instance, they could retrieve the token.  @@ -425,18 +417,16 @@ cluster. It is used in conjunction with, and includes [Istio](https://istio.io) to provide an external IP address for all programs hosted by Knative. -You will be prompted to enter a wildcard -domain where your applications will be exposed. Configure your DNS -server to use the external IP address for that domain. For any -application created and installed, they will be accessible as -`<program_name>.<kubernetes_namespace>.<domain_name>`. This will require -your Kubernetes cluster to have [RBAC -enabled](../project/clusters/add_remove_clusters.md#rbac-cluster-resources). - -NOTE: **Note:** The [`knative/knative`](https://storage.googleapis.com/triggermesh-charts) chart is used to install this application. +During installation, you must enter a wildcard domain where your applications +will be exposed. Configure your DNS server to use the external IP address for that +domain. Applications created and installed are accessible as +`<program_name>.<kubernetes_namespace>.<domain_name>`, which requires +your Kubernetes cluster to have +[RBAC enabled](../project/clusters/add_remove_clusters.md#rbac-cluster-resources). + ### Prometheus > - Introduced in GitLab 10.4 for project-level clusters. @@ -451,15 +441,14 @@ GitLab is able to monitor applications automatically, using the memory metrics are automatically collected, and response metrics are retrieved from NGINX Ingress as well. -To enable monitoring, simply install Prometheus into the cluster with the -**Install** button. - -NOTE: **Note:** The [`stable/prometheus`](https://github.com/helm/charts/tree/master/stable/prometheus) chart is used to install this application with a [`values.yaml`](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/prometheus/values.yaml) file. +To enable monitoring, install Prometheus into the cluster with the **Install** +button. + ### Crossplane > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34702) in GitLab 12.5 for project-level clusters. @@ -483,15 +472,14 @@ The Crossplane GitLab-managed application: PostgreSQL (for example, CloudSQL from GCP or RDS from AWS) and other services required by the application via the Auto DevOps pipeline. -For information on configuring Crossplane installed on the cluster, see -[Crossplane configuration](crossplane.md). - -NOTE: **Note:** [`alpha/crossplane`](https://github.com/crossplane/crossplane/tree/v0.4.1/cluster/charts/crossplane) chart v0.4.1 is used to install Crossplane using the [`values.yaml`](https://github.com/crossplane/crossplane/blob/master/cluster/charts/crossplane/values.yaml.tmpl) file. +For information about configuring Crossplane installed on the cluster, see +[Crossplane configuration](crossplane.md). + ### Elastic Stack > Introduced in GitLab 12.7 for project- and group-level clusters. @@ -500,37 +488,33 @@ file. log analysis solution which helps in deep searching, analyzing and visualizing the logs generated from different machines. -GitLab is able to gather logs from pods in your cluster automatically. -Filebeat will run as a DaemonSet on each node in your cluster, and it will ship container logs to Elasticsearch for querying. -GitLab will then connect to Elasticsearch for logs instead of the Kubernetes API, -and you will have access to more advanced querying capabilities. - -Log data is automatically deleted after 30 days using [Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/about.html). +GitLab can gather logs from pods in your cluster. Filebeat runs as a DaemonSet +on each node in your cluster, and ships container logs to Elasticsearch for +querying. GitLab then connects to Elasticsearch for logs, instead of the +Kubernetes API, giving you access to more advanced querying capabilities. Log +data is deleted after 30 days, using [Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/about.html). To enable log shipping: -1. Ensure your cluster contains at least 3 nodes of instance types larger than - `f1-micro`, `g1-small`, or `n1-standard-1`. +1. Ensure your cluster contains at least three nodes of instance types larger + than `f1-micro`, `g1-small`, or `n1-standard-1`. 1. Navigate to **Operations > Kubernetes**. 1. In **Kubernetes Cluster**, select a cluster. -1. In the **Applications** section, find **Elastic Stack** and click **Install**. +1. In the **Applications** section, find **Elastic Stack**, and then select + **Install**. -NOTE: **Note:** The [`gitlab/elastic-stack`](https://gitlab.com/gitlab-org/charts/elastic-stack) chart is used to install this application with a [`values.yaml`](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/elastic_stack/values.yaml) -file. +file. The chart deploys three identical Elasticsearch pods which can't be +colocated, and each requires one CPU and 2 GB of RAM, making them +incompatible with clusters containing fewer than three nodes, or consisting of +`f1-micro`, `g1-small`, `n1-standard-1`, or `*-highcpu-2` instance types. NOTE: **Note:** -The chart deploys 3 identical Elasticsearch pods which can't be colocated, and each -requires 1 CPU and 2 GB of RAM, making them incompatible with clusters containing -fewer than 3 nodes or consisting of `f1-micro`, `g1-small`, `n1-standard-1`, or -`*-highcpu-2` instance types. - -NOTE: **Note:** -The Elastic Stack cluster application is intended as a log aggregation solution and is not related to our -[Advanced Search](../search/advanced_global_search.md) functionality, which uses a separate -Elasticsearch cluster. +The Elastic Stack cluster application is intended as a log aggregation solution +and is not related to our [Advanced Search](../search/advanced_global_search.md) +functionality, which uses a separate Elasticsearch cluster. #### Optional: deploy Kibana to perform advanced queries @@ -1280,7 +1264,7 @@ You can check the default [`values.yaml`](https://gitlab.com/gitlab-org/gitlab/- You can customize the installation of Elastic Stack by defining `.gitlab/managed-apps/elastic-stack/values.yaml` file in your cluster management project. Refer to the -[chart](https://gitlab.com/gitlab-org/charts/elastic-stack) for the +[chart](https://gitlab.com/gitlab-org/charts/elastic-stack) for all available configuration options. NOTE: **Note:** @@ -1337,7 +1321,7 @@ You can customize the installation of Fluentd by defining `.gitlab/managed-apps/fluentd/values.yaml` file in your cluster management project. Refer to the [configuration chart for the current development release of Fluentd](https://github.com/helm/charts/tree/master/stable/fluentd#configuration) -for the available configuration options. +for all available configuration options. NOTE: **Note:** The configuration chart link points to the current development release, which @@ -1360,7 +1344,7 @@ knative: You can customize the installation of Knative by defining `.gitlab/managed-apps/knative/values.yaml` file in your cluster management project. Refer to the [chart](https://gitlab.com/gitlab-org/charts/knative) -for the available configuration options. +for all available configuration options. Here is an example configuration for Knative: diff --git a/doc/user/compliance/compliance_dashboard/img/compliance_dashboard_v13_3_1.png b/doc/user/compliance/compliance_dashboard/img/compliance_dashboard_v13_3_1.png Binary files differindex a06f8812b41..89f4e917567 100644 --- a/doc/user/compliance/compliance_dashboard/img/compliance_dashboard_v13_3_1.png +++ b/doc/user/compliance/compliance_dashboard/img/compliance_dashboard_v13_3_1.png diff --git a/doc/user/compliance/license_compliance/img/license-check_v13_4.png b/doc/user/compliance/license_compliance/img/license-check_v13_4.png Binary files differindex d3658cbaa18..bc80f938395 100644 --- a/doc/user/compliance/license_compliance/img/license-check_v13_4.png +++ b/doc/user/compliance/license_compliance/img/license-check_v13_4.png diff --git a/doc/user/compliance/license_compliance/index.md b/doc/user/compliance/license_compliance/index.md index 79c2d97b972..382d536be74 100644 --- a/doc/user/compliance/license_compliance/index.md +++ b/doc/user/compliance/license_compliance/index.md @@ -127,6 +127,11 @@ is used to detect the languages/frameworks and in turn analyzes the licenses. The License Compliance settings can be changed through [environment variables](#available-variables) by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. +### When License Compliance runs + +When using the GitLab `License-Scanning.gitlab-ci.yml` template, the License Compliance job doesn't +wait for other stages to complete. + ### Available variables License Compliance can be configured using environment variables. @@ -446,7 +451,7 @@ package manager. For a comprehensive list, consult [the Conan documentation](htt The default [Conan](https://conan.io/) configuration sets [`CONAN_LOGIN_USERNAME`](https://docs.conan.io/en/latest/reference/env_vars.html#conan-login-username-conan-login-username-remote-name) to `ci_user`, and binds [`CONAN_PASSWORD`](https://docs.conan.io/en/latest/reference/env_vars.html#conan-password-conan-password-remote-name) to the [`CI_JOB_TOKEN`](../../../ci/variables/predefined_variables.md) -for the running job. This allows Conan projects to fetch packages from a [GitLab Conan Repository](../../packages/conan_repository/#fetching-conan-package-information-from-the-gitlab-package-registry) +for the running job. This allows Conan projects to fetch packages from a [GitLab Conan Repository](../../packages/conan_repository/#fetch-conan-package-information-from-the-package-registry) if a GitLab remote is specified in the `.conan/remotes.json` file. To override the default credentials specify a [`CONAN_LOGIN_USERNAME_{REMOTE_NAME}`](https://docs.conan.io/en/latest/reference/env_vars.html#conan-login-username-conan-login-username-remote-name) diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index 67dd31efe2c..2adfd7532fd 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -38,6 +38,14 @@ The IP address for `mg.gitlab.com` is subject to change at any time. [See our backup strategy](https://about.gitlab.com/handbook/engineering/infrastructure/production/#backups). +There are several ways to perform backups of your content on GitLab.com. + +Projects can be backed up in their entirety by exporting them either [through the UI](../project/settings/import_export.md) or [API](../../api/project_import_export.md#schedule-an-export), the latter of which can be used to programmatically upload exports to a storage platform such as AWS S3. + +With exports, be sure to take note of [what is and is not](../project/settings/import_export.md#exported-contents), included in a project export. + +Since GitLab is built on Git, you can back up **just** the repository of a project by [cloning](../../gitlab-basics/start-using-git.md#clone-a-repository) it to another machine. Similarly, if you need to back up just the wiki of a repository it can also be cloned and all files uploaded to that wiki will come with it [if they were uploaded after 2020-08-22](../project/wiki/index.md#creating-a-new-wiki-page). + ## Alternative SSH port GitLab.com can be reached via a [different SSH port](https://about.gitlab.com/blog/2016/02/18/gitlab-dot-com-now-supports-an-alternate-git-plus-ssh-port/) for `git+ssh`. @@ -474,6 +482,10 @@ More information on this particular change can be found at of proposed changes can be found at <https://gitlab.com/gitlab-com/infrastructure/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=database&label_name[]=change>. +## Puma + +GitLab.com uses the default of 60 seconds for [Puma request timeouts](https://docs.gitlab.com/omnibus/settings/puma.html#worker-timeout). + ## Unicorn GitLab.com adjusts the memory limits for the [unicorn-worker-killer](https://rubygems.org/gems/unicorn-worker-killer) gem. diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index ebf38aef4a6..2d664da686f 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -14,6 +14,9 @@ Similar to [project-level](../../project/clusters/index.md) and group-level Kubernetes clusters allow you to connect a Kubernetes cluster to your group, enabling you to use the same cluster across multiple projects. +To view your group level Kubernetes clusters, navigate to your project and select +**Kubernetes** from the left-hand menu. + ## Installing applications GitLab can install and manage some applications in your group-level diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md index bcc6d958427..a689b7c380b 100644 --- a/doc/user/group/contribution_analytics/index.md +++ b/doc/user/group/contribution_analytics/index.md @@ -9,8 +9,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Introduced in [GitLab Starter](https://about.gitlab.com/pricing/) 8.3. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3090) for subgroups in GitLab 12.2. -## Overview - With Contribution Analytics you can get an overview of the following activity in your group: diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 32b76cf9280..2c31ef3bf0b 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -395,7 +395,7 @@ milestones. Get an overview of the vulnerabilities of all the projects in a group and its subgroups. -[Learn more about the Group Security Dashboard.](security_dashboard/index.md) +[Learn more about the Group Security Dashboard.](../application_security/security_dashboard/index.md) ## Insights **(ULTIMATE)** diff --git a/doc/user/group/repositories_analytics/index.md b/doc/user/group/repositories_analytics/index.md index b013e371ed2..323411ccbab 100644 --- a/doc/user/group/repositories_analytics/index.md +++ b/doc/user/group/repositories_analytics/index.md @@ -1,7 +1,7 @@ --- type: reference stage: Verify -group: Analytics +group: Testing info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index 57b9cc92c51..3d24e7b8d44 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -5,7 +5,7 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# SAML SSO for GitLab.com groups **(PREMIUM)** +# SAML SSO for GitLab.com groups **(SILVER ONLY)** > Introduced in GitLab 11.0. @@ -256,53 +256,6 @@ For example, to unlink the `MyOrg` account, the following **Disconnect** button | Issuer | How GitLab identifies itself to the identity provider. Also known as a "Relying party trust identifier". | | Certificate fingerprint | Used to confirm that communications over SAML are secure by checking that the server is signing communications with the correct certificate. Also known as a certificate thumbprint. | -## Configuring on a self-managed GitLab instance **(PREMIUM ONLY)** - -For self-managed GitLab instances we strongly recommend using the -[instance-wide SAML OmniAuth Provider](../../../integration/saml.md) instead. - -Group SAML SSO helps if you need to allow access via multiple SAML identity providers, but as a multi-tenant solution is less suited to cases where you administer your own GitLab instance. - -To proceed with configuring Group SAML SSO instead, you'll need to enable the `group_saml` OmniAuth provider. This can be done from: - -- `gitlab.rb` for [Omnibus GitLab installations](#omnibus-installations). -- `gitlab/config/gitlab.yml` for [source installations](#source-installations). - -### Limitations - -Group SAML on a self-managed instance is limited when compared to the recommended -[instance-wide SAML](../../../integration/saml.md). The recommended solution allows you to take advantage of: - -- [LDAP compatibility](../../../administration/auth/ldap/index.md). -- [LDAP Group Sync](../index.md#manage-group-memberships-via-ldap). -- [Required groups](../../../integration/saml.md#required-groups). -- [Admin groups](../../../integration/saml.md#admin-groups). -- [Auditor groups](../../../integration/saml.md#auditor-groups). - -### Omnibus installations - -1. Make sure GitLab is - [configured with HTTPS](../../../install/installation.md#using-https). -1. Enable OmniAuth and the `group_saml` provider in `gitlab.rb`: - - ```ruby - gitlab_rails['omniauth_enabled'] = true - gitlab_rails['omniauth_providers'] = [{ name: 'group_saml' }] - ``` - -### Source installations - -1. Make sure GitLab is - [configured with HTTPS](../../../install/installation.md#using-https). -1. Enable OmniAuth and the `group_saml` provider in `gitlab/config/gitlab.yml`: - - ```yaml - omniauth: - enabled: true - providers: - - { name: 'group_saml' } - ``` - ## Passwords for users created via SAML SSO for Groups The [Generated passwords for users created through integrated authentication](../../../security/passwords_for_integrated_authentication_methods.md) guide provides an overview of how GitLab generates and sets passwords for users created via SAML SSO for Groups. diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index 4f74e672392..985e6ec13be 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -260,15 +260,9 @@ It is important that this SCIM `id` and SCIM `externalId` are configured to the Group owners can see the list of users and the `externalId` stored for each user in the group SAML SSO Settings page. -Alternatively, the [SCIM API](../../../api/scim.md#get-a-list-of-saml-users) can be used to manually retrieve the `externalId` we have stored for users, also called the `external_uid` or `NameId`. +A possible alternative is to use the [SCIM API](../../../api/scim.md#get-a-list-of-saml-users) to manually retrieve the `externalId` we have stored for users, also called the `external_uid` or `NameId`. -For example: - -```shell -curl 'https://gitlab.example.com/api/scim/v2/groups/GROUP_NAME/Users?startIndex=1"' --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" -``` - -To see how this compares to the value returned as the SAML NameId, you can have the user use a [SAML Tracer](index.md#saml-debugging-tools). +To see how the `external_uid` compares to the value returned as the SAML NameId, you can have the user use a [SAML Tracer](index.md#saml-debugging-tools). #### Update or fix mismatched SCIM externalId and SAML NameId @@ -285,15 +279,9 @@ you can address the problem in the following ways: - You can have users unlink and relink themselves, based on the ["SAML authentication failed: User has already been taken"](./index.md#message-saml-authentication-failed-user-has-already-been-taken) section. - You can unlink all users simultaneously, by removing all users from the SAML app while provisioning is turned on. -- You can use the [SCIM API](../../../api/scim.md#update-a-single-saml-user) to manually correct the `externalId` stored for users to match the SAML `NameId`. +- It may be possible to use the [SCIM API](../../../api/scim.md#update-a-single-saml-user) to manually correct the `externalId` stored for users to match the SAML `NameId`. To look up a user, you'll need to know the desired value that matches the `NameId` as well as the current `externalId`. -It is then possible to issue a manual SCIM#update request, for example: - -```shell -curl --verbose --request PATCH 'https://gitlab.com/api/scim/v2/groups/YOUR_GROUP/Users/OLD_EXTERNAL_UID' --data '{ "Operations": [{"op":"Replace","path":"externalId","value":"NEW_EXTERNAL_UID"}] }' --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" -``` - It is important not to update these to incorrect values, since this will cause users to be unable to sign in. It is also important not to assign a value to the wrong user, as this would cause users to get signed into the wrong account. #### I need to change my SCIM app diff --git a/doc/user/img/feature_flags_history_note_info_v13_2.png b/doc/user/img/feature_flags_history_note_info_v13_2.png Binary files differindex 403a6002603..07d096b6dde 100644 --- a/doc/user/img/feature_flags_history_note_info_v13_2.png +++ b/doc/user/img/feature_flags_history_note_info_v13_2.png diff --git a/doc/user/img/version_history_notes_collapsed_v13_2.png b/doc/user/img/version_history_notes_collapsed_v13_2.png Binary files differindex 42ea11ae8ff..b85c9cb36dd 100644 --- a/doc/user/img/version_history_notes_collapsed_v13_2.png +++ b/doc/user/img/version_history_notes_collapsed_v13_2.png diff --git a/doc/user/index.md b/doc/user/index.md index ce8713591ab..32a1c235882 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -46,8 +46,9 @@ GitLab is a Git-based platform that integrates a great number of essential tools - Building, testing, and deploying with built-in [Continuous Integration](../ci/README.md). - Deploying personal and professional static websites with [GitLab Pages](project/pages/index.md). - Integrating with Docker by using [GitLab Container Registry](packages/container_registry/index.md). -- Tracking the development lifecycle by using [GitLab Value Stream Analytics](project/cycle_analytics.md). +- Tracking the development lifecycle by using [GitLab Value Stream Analytics](analytics/value_stream_analytics.md). - Provide support with [Service Desk](project/service_desk.md). +- [Export issues as CSV](project/issues/csv_export.md). With GitLab Enterprise Edition, you can also: @@ -60,8 +61,7 @@ With GitLab Enterprise Edition, you can also: - Leverage [Elasticsearch](../integration/elasticsearch.md) with [Advanced Search](search/advanced_global_search.md) and [Advanced Search Syntax](search/advanced_search_syntax.md) for faster, more advanced code search across your entire GitLab instance. - [Authenticate users with Kerberos](../integration/kerberos.md). - [Mirror a repository](project/repository/repository_mirroring.md) from elsewhere on your local server. -- [Export issues as CSV](project/issues/csv_export.md). -- View your entire CI/CD pipeline involving more than one project with [Multiple-Project Pipelines](../ci/multi_project_pipeline_graphs.md). +- View your entire CI/CD pipeline involving more than one project with [Multiple-Project Pipelines](../ci/multi_project_pipelines.md). - [Lock files](project/file_lock.md) to prevent conflicts. - View the current health and status of each CI environment running on Kubernetes with [Deploy Boards](project/deploy_boards.md). - Leverage continuous delivery method with [Canary Deployments](project/canary_deployments.md). diff --git a/doc/user/infrastructure/index.md b/doc/user/infrastructure/index.md index 227a67f8c8a..216cf9ed607 100644 --- a/doc/user/infrastructure/index.md +++ b/doc/user/infrastructure/index.md @@ -82,6 +82,10 @@ local machine, this is a simple way to get started: -backend-config="retry_wait_min=5" ``` + NOTE: **Note:** + The name of your state can contain only uppercase and lowercase letters, + decimal digits, hyphens and underscores. + You can now run `terraform plan` and `terraform apply` as you normally would. ## Get started using GitLab CI diff --git a/doc/user/instance/clusters/index.md b/doc/user/instance/clusters/index.md index 8059b8ca642..c370f9d8725 100644 --- a/doc/user/instance/clusters/index.md +++ b/doc/user/instance/clusters/index.md @@ -2,14 +2,14 @@ > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/39840) in GitLab 11.11. -## Overview - Similar to [project-level](../../project/clusters/index.md) and [group-level](../../group/clusters/index.md) Kubernetes clusters, instance-level Kubernetes clusters allow you to connect a Kubernetes cluster to the GitLab instance, which enables you to use the same cluster across multiple projects. +The instance level Kubernetes clusters can be found in the top menu by navigating to your instance's **{admin}** **Admin Area > Kubernetes**. + ## Cluster precedence GitLab will try to match clusters in the following order: diff --git a/doc/user/packages/composer_repository/img/project_id_v13_5.png b/doc/user/packages/composer_repository/img/project_id_v13_5.png Binary files differnew file mode 100644 index 00000000000..45861b6ff1b --- /dev/null +++ b/doc/user/packages/composer_repository/img/project_id_v13_5.png diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index 89e02b4847c..219ccef0a01 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -4,172 +4,165 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# GitLab Composer Repository +# Composer packages in the Package Registry > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15886) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Core in 13.3. -With the GitLab Composer Repository, every project can have its own space to store [Composer](https://getcomposer.org/) packages. +Publish [Composer](https://getcomposer.org/) packages in your project's Package Registry. +Then, install the packages whenever you need to use them as a dependency. -## Enabling the Composer Repository +## Create a Composer package -NOTE: **Note:** -This option is available only if your GitLab administrator has -[enabled support for the Package Registry](../../../administration/packages/index.md). +If you do not have a Composer package, create one and check it in to +a repository. This example shows a GitLab repository, but the repository +can be any public or private repository. -When the Composer Repository is enabled, it is available for all new projects -by default. To enable it for existing projects, or if you want to disable it: +1. Create a directory called `my-composer-package` and change to that directory: -1. Navigate to your project's **Settings > General > Visibility, project features, permissions**. -1. Find the Packages feature and enable or disable it. -1. Click on **Save changes** for the changes to take effect. + ```shell + mkdir my-composer-package && cd my-composer-package + ``` -You should then be able to see the **Packages & Registries** section on the left sidebar. +1. Run [`composer init`](https://getcomposer.org/doc/03-cli.md#init) and answer the prompts. -## Getting started + For namespace, enter your unique [namespace](../../../user/group/index.md#namespaces), like your GitLab username or group name. -This section covers creating a new example Composer package to publish. This is a -quickstart to test out the **GitLab Composer Registry**. + A file called `composer.json` is created: -To complete this section, you need a recent version of [Composer](https://getcomposer.org/). + ```json + { + "name": "<namespace>/composer-test", + "type": "library", + "license": "GPL-3.0-only", + "version": "1.0.0" + } + ``` -### Creating a package project +1. Run Git commands to tag the changes and push them to your repository: -Understanding how to create a full Composer project is outside the scope of this -guide, but you can create a small package to test out the registry. Start by -creating a new directory called `my-composer-package`: + ```shell + git init + git add composer.json + git commit -m 'Composer package test' + git tag v1.0.0 + git remote add origin git@gitlab.example.com:<namespace>/<project-name>.git + git push --set-upstream origin master + git push origin v1.0.0 + ``` -```shell -mkdir my-composer-package && cd my-composer-package -``` +The package is now in your GitLab Package Registry. -Create a new `composer.json` file inside this directory to set up the basic project: +## Publish a Composer package by using the API -```shell -touch composer.json -``` +Publish a Composer package to the Package Registry, +so that anyone who can access the project can use the package as a dependency. -Inside `composer.json`, add the following code: +Prerequisites: -```json -{ - "name": "<namespace>/composer-test", - "type": "library", - "license": "GPL-3.0-only", - "version": "1.0.0" -} -``` +- A package in a GitLab repository. +- The project ID, which is on the project's home page. +- A [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api`. -Replace `<namespace>` with a unique namespace like your GitLab username or group name. + NOTE: **Note:** + [Deploy tokens](./../../project/deploy_tokens/index.md) are + [not yet supported](https://gitlab.com/gitlab-org/gitlab/-/issues/240897) for use with Composer. -After this basic package structure is created, we need to tag it in Git and push it to the repository. +To publish the package: -```shell -git init -git add composer.json -git commit -m 'Composer package test' -git tag v1.0.0 -git remote add origin git@gitlab.com:<namespace>/<project-name>.git -git push --set-upstream origin master -git push origin v1.0.0 -``` +- Send a `POST` request to the [Packages API](../../../api/packages.md). + + For example, you can use `curl`: -### Publishing the package + ```shell + curl --data tag=<tag> "https://__token__:<personal-access-token>@gitlab.example.com/api/v4/projects/<project_id>/packages/composer" + ``` -Now that the basics of our project is completed, we can publish the package. -To publish the package, you need: + - `<personal-access-token>` is your personal access token. + - `<project_id>` is your project ID. + - `<tag>` is the Git tag name of the version you want to publish. + To publish a branch, use `branch=<branch>` instead of `tag=<tag>`. -- A personal access token or `CI_JOB_TOKEN`. +You can view the published package by going to **Packages & Registries > Package Registry** and +selecting the **Composer** tab. - ([Deploy tokens](./../../project/deploy_tokens/index.md) are not yet supported for use with Composer.) +## Publish a Composer package by using CI/CD -- Your project ID which can be found on the home page of your project. +You can publish a Composer package to the Package Registry as part of your CI/CD process. -To publish the package hosted on GitLab, make a `POST` request to the GitLab package API. -A tool like `curl` can be used to make this request: +1. Specify a `CI_JOB_TOKEN` in your `.gitlab-ci.yml` file: -You can generate a [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api` for repository authentication. For example: + ```yaml + stages: + - deploy -```shell -curl --data tag=<tag> 'https://__token__:<personal-access-token>@gitlab.com/api/v4/projects/<project_id>/packages/composer' -``` + deploy: + stage: deploy + script: + - 'curl --header "Job-Token: $CI_JOB_TOKEN" --data tag=<tag> "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/packages/composer"' + ``` -Where: +1. Run the pipeline. -- `<personal-access-token>` is your personal access token. -- `<project_id>` is your project ID. -- `<tag>` is the Git tag name of the version you want to publish. In this example it should be `v1.0.0`. Notice that instead of `tag=<tag>` you can also use `branch=<branch>` to publish branches. +You can view the published package by going to **Packages & Registries > Package Registry** and selecting the **Composer** tab. -If the above command succeeds, you now should be able to see the package under the **Packages & Registries** section of your project page. +### Use a CI/CD template -### Publishing the package with CI/CD +A more detailed Composer CI/CD file is also available as a `.gitlab-ci.yml` template: -To work with Composer commands within [GitLab CI/CD](./../../../ci/README.md), you can -publish Composer packages by using `CI_JOB_TOKEN` in your `.gitlab-ci.yml` file: +1. On the left sidebar, click **Project overview**. +1. Above the file list, click **Set up CI/CD**. If this button is not available, select **CI/CD Configuration** and then **Edit**. +1. From the **Apply a template** list, select **Composer**. -```yaml -stages: - - deploy +CAUTION: **Warning:** +Do not save unless you want to overwrite the existing CI/CD file. -deploy: - stage: deploy - script: - - 'curl --header "Job-Token: $CI_JOB_TOKEN" --data tag=<tag> "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/packages/composer"' -``` +## Install a Composer package -### Installing a package +Install a package from the Package Registry so you can use it as a dependency. -To install your package, you need: +Prerequisites: -- A personal access token. You can generate a [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api` for repository authentication. -- Your group ID which can be found on the home page of your project's group. +- A package in the Package Registry. +- The group ID, which is on the group's home page. +- A [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api`. -Add the GitLab Composer package repository to your existing project's `composer.json` file, along with the package name and version you want to install like so: + NOTE: **Note:** + [Deploy tokens](./../../project/deploy_tokens/index.md) are + [not yet supported](https://gitlab.com/gitlab-org/gitlab/-/issues/240897) for use with Composer. -```json -{ - ... - "repositories": [ - { "type": "composer", "url": "https://gitlab.com/api/v4/group/<group_id>/-/packages/composer/packages.json" } - ], - "require": { - ... - "<package_name>": "<version>" - }, - ... -} -``` +To install a package: -Where: +1. Add the Package Registry URL to your project's `composer.json` file, along with the package name and version you want to install: -- `<group_id>` is the group ID found under your project's group page. -- `<package_name>` is your package name as defined in your package's `composer.json` file. -- `<version>` is your package version (`1.0.0` in this example). + ```json + { + ... + "repositories": [ + { "type": "composer", "url": "https://gitlab.example.com/api/v4/group/<group_id>/-/packages/composer/packages.json" } + ], + "require": { + ... + "<package_name>": "<version>" + }, + ... + } + ``` -You also need to create a `auth.json` file with your GitLab credentials: + - `<group_id>` is the group ID. + - `<package_name>` is the package name defined in your package's `composer.json` file. + - `<version>` is the package version. -```json -{ - "gitlab-token": { - "gitlab.com": "<personal_access_token>" - } -} -``` +1. Create an `auth.json` file with your GitLab credentials: -Where: + ```shell + composer config gitlab-token.<DOMAIN-NAME> <personal_access_token> + ``` -- `<personal_access_token>` is your personal access token. - -With the `composer.json` and `auth.json` files configured, you can install the package by running `composer`: - -```shell -composer update -``` - -If successful, you should be able to see the output indicating that the package has been successfully installed. +Output indicates that the package has been successfully installed. CAUTION: **Important:** -Make sure to never commit the `auth.json` file to your repository. To install packages from a CI job, +Never commit the `auth.json` file to your repository. To install packages from a CI/CD job, consider using the [`composer config`](https://getcomposer.org/doc/articles/handling-private-packages-with-satis.md#authentication) tool with your personal access token stored in a [GitLab CI/CD environment variable](../../../ci/variables/README.md) or in -[Hashicorp Vault](../../../ci/secrets/index.md). +[HashiCorp Vault](../../../ci/secrets/index.md). diff --git a/doc/user/packages/conan_repository/img/conan_package_view.png b/doc/user/packages/conan_repository/img/conan_package_view.png Binary files differdeleted file mode 100644 index 742fb4195da..00000000000 --- a/doc/user/packages/conan_repository/img/conan_package_view.png +++ /dev/null diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index 7c3082e0f83..1f485de6293 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -4,153 +4,150 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# GitLab Conan Repository +# Conan packages in the Package Registry > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8248) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.6. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Core in 13.3. -With the GitLab Conan Repository, every -project can have its own space to store Conan packages. +Publish Conan packages in your project’s Package Registry. Then install the +packages whenever you need to use them as a dependency. - +To publish Conan packages to the Package Registry, add the +Package Registry as a remote and authenticate with it. -## Enabling the Conan Repository +Then you can run `conan` commands and publish your package to the Package Registry. -NOTE: **Note:** -This option is available only if your GitLab administrator has -[enabled support for the Conan Repository](../../../administration/packages/index.md). - -After the Conan Repository is enabled, it's available for all new projects -by default. To enable it for existing projects, or if you want to disable it: - -1. Navigate to your project's **Settings > General > Visibility, project features, permissions**. -1. Find the Packages feature and enable or disable it. -1. Click on **Save changes** for the changes to take effect. - -You should then be able to see the **Packages & Registries** section on the left sidebar. +## Build a Conan package -## Getting started +This section explains how to install Conan and build a package for your C/C++ project. -This section covers installing Conan and building a package for your C/C++ project. This is a quickstart if you're new -to Conan. If you already are using Conan and understand how to build your own packages, move on to the [next section](#adding-the-gitlab-package-registry-as-a-conan-remote). +If you already use Conan and know how to build your own packages, go to the [next section](#add-the-package-registry-as-a-conan-remote). -### Installing Conan +### Install Conan -Follow the instructions at [conan.io](https://conan.io/downloads.html) to download the Conan package manager to your local development environment. +Download the Conan package manager to your local development environment by following +the instructions at [conan.io](https://conan.io/downloads.html). -Once installation is complete, verify you can use Conan in your terminal by running +When installation is complete, verify you can use Conan in your terminal by running: ```shell conan --version ``` -You should see the Conan version printed in the output: +The Conan version is printed in the output: ```plaintext Conan version 1.20.5 ``` -### Installing CMake +### Install CMake + +When you develop with C++ and Conan, you have a range of compilers to choose from. +This example uses the CMake compiler. + +To install CMake: + +- For Mac, use [homebrew](https://brew.sh/) and run `brew install cmake`. +- For other operating systems, follow the instructions at [cmake.org](https://cmake.org/install/). -When developing with C++ and Conan, you have a wide range of options for compilers. This tutorial walks through using the cmake -compiler. In your terminal, run the command +When installation is complete, verify you can use CMake in your terminal by running: ```shell cmake --version ``` -You should see the cmake version printed in the output. If you see something else, you may have to install cmake. +The CMake version is printed in the output. -On a Mac, you can use [homebrew](https://brew.sh/) to install cmake by running `brew install cmake`. Otherwise, follow -instructions at [cmake.org](https://cmake.org/install/) for your operating system. +### Create a project -### Creating a project +To test the Package Registry, you need a C++ project. If you don't already have one, you can clone the +Conan [hello world starter project](https://github.com/conan-io/hello). -Understanding what is needed to create a valid and compilable C++ project is out of the scope of this guide, but if you're new to C++ and want to try out the GitLab -package registry, Conan.io has a great [hello world starter project](https://github.com/conan-io/hello) that you can clone to get started. +### Build a package -Clone the repository and it can be used for the rest of the tutorial if you don't have your own C++ project. +To build a package: -### Building a package +1. Open a terminal and navigate to your project's root folder. +1. Generate a new recipe by running `conan new` with a package name and version: -In your terminal, navigate to the root folder of your project. Generate a new recipe by running `conan new` and providing it with a -package name and version: + ```shell + conan new Hello/0.1 -t + ``` -```shell -conan new Hello/0.1 -t -``` - -Next, create a package for that recipe by running `conan create` providing the Conan user and channel: +1. Create a package for the recipe by running `conan create` with the Conan user and channel: -```shell -conan create . mycompany/beta -``` + ```shell + conan create . mycompany/beta + ``` -NOTE: **Note:** -If you are using the [instance level remote](#instance-level-remote), a specific [naming convention](#package-recipe-naming-convention-for-instance-level-remote) must be followed. + NOTE: **Note:** + If you use an [instance remote](#add-a-remote-for-your-instance), you must follow a specific [naming convention](#package-recipe-naming-convention-for-instance-remotes). -These two example commands generate a final package with the recipe `Hello/0.1@mycompany/beta`. +A package with the recipe `Hello/0.1@mycompany/beta` is created. -For more advanced details on creating and managing your packages, refer to the [Conan docs](https://docs.conan.io/en/latest/creating_packages.html). +For more details on creating and managing Conan packages, see the [Conan docs](https://docs.conan.io/en/latest/creating_packages.html). -You are now ready to upload your package to the GitLab registry. To get started, first you need to set GitLab as a remote. Then you need to add a Conan user for that remote to authenticate your requests. +## Add the Package Registry as a Conan remote -## Adding the GitLab Package Registry as a Conan remote +To run `conan` commands, you must add the Package Registry as a Conan remote for your project or instance. -You can add the GitLab Package Registry as a Conan remote at the project or instance level. - -### Project level remote +### Add a remote for your project > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11679) in GitLab 13.4. -The project level remote allows you to work with packages within a given project. -The advantage of using the project level remote is there are no restrictions to your -package name, however all GitLab Conan packages require a full recipe -with the user and channel (`package_name/version@user/channel`). +Set a remote so you can work with packages in a project without +having to specify the remote name in every command. + +When you set a remote for a project, there are no restrictions to your package names. +However, your commands must include the full recipe, including the user and channel, +for example, `package_name/version@user/channel`. To add the remote: -```shell -conan remote add gitlab https://gitlab.example.com/api/v4/projects/<project_id>/packages/conan -``` +1. In your terminal, run this command: -Once the remote is set, you can use the remote when running Conan commands by adding `--remote=gitlab` to the end of your commands. + ```shell + conan remote add gitlab https://gitlab.example.com/api/v4/projects/<project_id>/packages/conan + ``` -For example: +1. Use the remote by adding `--remote=gitlab` to the end of your Conan command. -```shell -conan search Hello* --all --remote=gitlab -``` + For example: -### Instance level remote + ```shell + conan search Hello* --all --remote=gitlab + ``` -The instance level remote allows you to use a single remote to access packages accross your entire -GitLab instance. However, when using this remote, there are certain -[package naming restrictions](#package-recipe-naming-convention-for-instance-level-remote) -that must be followed. +### Add a remote for your instance -Add a new remote to your Conan configuration: +Use a single remote to access packages across your entire GitLab instance. -```shell -conan remote add gitlab https://gitlab.example.com/api/v4/packages/conan -``` +However, when using this remote, you must follow these +[package naming restrictions](#package-recipe-naming-convention-for-instance-remotes). + +To add the remote: -Once the remote is set, you can use the remote when running Conan commands by adding `--remote=gitlab` to the end of your commands. +1. In your terminal, run this command: -For example: + ```shell + conan remote add gitlab https://gitlab.example.com/api/v4/packages/conan + ``` -```shell -conan search 'Hello*' --remote=gitlab -``` +1. Use the remote by adding `--remote=gitlab` to the end of your Conan command. -#### Package recipe naming convention for instance level remote + For example: -The standard Conan recipe convention looks like `package_name/version@user/channel`, -but if you're using the [instance level remote](#instance-level-remote), the recipe + ```shell + conan search 'Hello*' --remote=gitlab + ``` + +#### Package recipe naming convention for instance remotes + +The standard Conan recipe convention is `package_name/version@user/channel`, +but if you're using an [instance remote](#add-a-remote-for-your-instance), the recipe `user` must be the plus sign (`+`) separated project path. -The following table shows some example recipes you can give your package based on -the project name and path. +Example recipe names: | Project | Package | Supported | | ---------------------------------- | ----------------------------------------------- | --------- | @@ -159,179 +156,202 @@ the project name and path. | `gitlab-org/gitlab-ce` | `my-package/1.0.0@gitlab-org+gitlab-ce/stable` | Yes | | `gitlab-org/gitlab-ce` | `my-package/1.0.0@foo/stable` | No | -NOTE: **Note:** -[Project level remotes](#project-level-remote) allow for more flexible package names. +[Project remotes](#add-a-remote-for-your-project) have a more flexible naming convention. -## Authenticating to the GitLab Conan Repository +## Authenticate to the Package Registry -You need a personal access token or deploy token. +To authenticate to the Package Registry, you need either a personal access token or deploy token. -For repository authentication: +- If you use a [personal access token](../../../user/profile/personal_access_tokens.md), set the scope to `api`. +- If you use a [deploy token](./../../project/deploy_tokens/index.md), set the scope to `read_package_registry`, `write_package_registry`, or both. -- You can generate a [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api`. -- You can generate a [deploy token](./../../project/deploy_tokens/index.md) with the scope set to `read_package_registry`, `write_package_registry`, or both. +### Add your credentials to the GitLab remote -### Adding a Conan user to the GitLab remote +Associate your token with the GitLab remote, so that you don't have to explicitly +add a token to every Conan command. -Once you have a personal access token and have [set your Conan remote](#adding-the-gitlab-package-registry-as-a-conan-remote), you can associate the token with the remote so you don't have to explicitly add them to each Conan command you run: +Prerequisites: + +- You must have an authentication token. +- The Conan remote [must be set](#add-the-package-registry-as-a-conan-remote). + +In a terminal, run this command. In this example, the remote name is `gitlab`. Use the name of your remote. ```shell conan user <gitlab_username or deploy_token_username> -r gitlab -p <personal_access_token or deploy_token> ``` -NOTE: **Note:** -If you named your remote something other than `gitlab`, your remote name should be used in this command instead of `gitlab`. - -From now on, when you run commands using `--remote=gitlab`, your username and password is automatically included in the requests. - -NOTE: **Note:** -The personal access token is not stored locally at any moment. Conan uses JSON Web Tokens (JWT), so when you run this command, Conan requests an expirable token from GitLab using your token. The JWT does expire on a regular basis, so you need to re-enter your personal access token when that happens. +Now when you run commands with `--remote=gitlab`, your username and password are automatically included in the requests. -Alternatively, you could explicitly include your credentials in any given command. -For example: +Alternately, you can explicitly include your credentials in any given command. For example: ```shell CONAN_LOGIN_USERNAME=<gitlab_username or deploy_token_username> CONAN_PASSWORD=<personal_access_token or deploy_token> conan upload Hello/0.1@mycompany/beta --all --remote=gitlab ``` -### Setting a default remote to your project (optional) +NOTE: **Note:** +Your authentication with GitLab expires on a regular basis, +so occasionally you may need to re-enter your personal access token. + +### Set a default remote for your project (optional) + +If you want to interact with the GitLab Package Registry without having to specify a remote, +you can tell Conan to always use the Package Registry for your packages. -If you'd like Conan to always use GitLab as the registry for your package, you can tell Conan to always reference the GitLab remote for a given package recipe: +In a terminal, run this command. ```shell conan remote add_ref Hello/0.1@mycompany/beta gitlab ``` NOTE: **Note:** -The package recipe does include the version, so setting the default remote for `Hello/0.1@user/channel` will not work for `Hello/0.2@user/channel`. -This functionality is best suited for when you want to consume or install packages from the GitLab registry without having to specify a remote. +The package recipe includes the version, so the default remote for `Hello/0.1@user/channel` does not work for `Hello/0.2@user/channel`. -The rest of the example commands in this documentation assume that you've added a Conan user with your credentials to the `gitlab` remote and will not include the explicit credentials or remote option. With that said, be aware that any of the commands could be run without having added a user or default remote: +If you do not set a default user or remote, you can still include the user and remote in your commands: ```shell `CONAN_LOGIN_USERNAME=<gitlab_username or deploy_token_username> CONAN_PASSWORD=<personal_access_token or deploy_token> <conan command> --remote=gitlab ``` -## Uploading a package +## Publish a Conan package -First you need to [create your Conan package locally](https://docs.conan.io/en/latest/creating_packages/getting_started.html). +Publish a Conan package to the Package Registry, so that anyone who can access the project can use the package as a dependency. -NOTE: **Note:** -If you are using the [instance level remote](#instance-level-remote), a specific [naming convention](#package-recipe-naming-convention-for-instance-level-remote) must be followed. +Prerequisites: + +To publish a Conan package, you need: -Ensure you have a project created on GitLab and that the personal access token you're using has the correct permissions for write access to the container registry by selecting the `api` [scope](../../../user/profile/personal_access_tokens.md#limiting-scopes-of-a-personal-access-token). +- The Package Registry [set as a remote](#add-the-package-registry-as-a-conan-remote). +- [Authentication](#authenticate-to-the-package-registry) set up with the Package Registry. +- A local [Conan package](https://docs.conan.io/en/latest/creating_packages/getting_started.html). + - For an instance remote, the package must meet the [naming convention](#package-recipe-naming-convention-for-instance-remotes). +- A project ID, which is on the project's homepage. -You can upload your package to the GitLab Package Registry using the `conan upload` command: +To publish the package, use the `conan upload` command: ```shell conan upload Hello/0.1@mycompany/beta --all ``` -## Installing a package +## Publish a Conan package by using CI/CD -Conan packages are commonly installed as dependencies using the `conanfile.txt` file. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11678) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.7. -In your project where you would like to install the Conan package as a dependency, open `conanfile.txt` or create -an empty file named `conanfile.txt` in the root of your project. +To work with Conan commands in [GitLab CI/CD](./../../../ci/README.md), you can use +`CI_JOB_TOKEN` in place of the personal access token in your commands. -Add the Conan recipe to the `[requires]` section of the file: +You can provide the `CONAN_LOGIN_USERNAME` and `CONAN_PASSWORD` with each +Conan command in your `.gitlab-ci.yml` file. For example: -```ini - [requires] - Hello/0.1@mycompany/beta +```yaml +image: conanio/gcc7 - [generators] - cmake +create_package: + stage: deploy + script: + - conan remote add gitlab https://gitlab.example.com/api/v4/packages/conan + - conan new <package-name>/0.1 -t + - conan create . <group-name>+<project-name>/stable + - CONAN_LOGIN_USERNAME=ci_user CONAN_PASSWORD=${CI_JOB_TOKEN} conan upload <package-name>/0.1@<group-name>+<project-name>/stable --all --remote=gitlab ``` -Next, create a build directory from the root of your project and navigate to it: +Additional Conan images to use as the basis of your CI file are available +in the [Conan docs](https://docs.conan.io/en/latest/howtos/run_conan_in_docker.html#available-docker-images). -```shell -mkdir build && cd build -``` +## Install a Conan package -Now you can install the dependencies listed in `conanfile.txt`: +Install a Conan package from the Package Registry so you can use it as a dependency. -```shell -conan install .. -``` +Conan packages are often installed as dependencies by using the `conanfile.txt` file. + +Prerequisites: + +To install a Conan package, you need: + +- The Package Registry [set as a remote](#add-the-package-registry-as-a-conan-remote). +- [Authentication](#authenticate-to-the-package-registry) set up with the Package Registry. + +1. In the project where you want to install the package as a dependency, open `conanfile.txt`. + Or, in the root of your project, create a file called `conanfile.txt`. + +1. Add the Conan recipe to the `[requires]` section of the file: + + ```ini + [requires] + Hello/0.1@mycompany/beta + + [generators] + cmake + ``` + +1. At the root of your project, create a `build` directory and change to that directory: + + ```shell + mkdir build && cd build + ``` + +1. Install the dependencies listed in `conanfile.txt`: + + ```shell + conan install <options> + ``` NOTE: **Note:** -If you're trying to install the package you just created in this tutorial, not much will happen since that package -already exists on your local machine. +If you try to install the package you just created in this tutorial, the package +already exists on your local machine, so this command has no effect. -## Removing a package +## Remove a Conan package There are two ways to remove a Conan package from the GitLab Package Registry. -- **Using the Conan client in the command line:** +- From the command line, using the Conan client: ```shell conan remove Hello/0.2@user/channel --remote=gitlab ``` - You need to explicitly include the remote in this command, otherwise the package is only removed from your + You must explicitly include the remote in this command, otherwise the package is only removed from your local system cache. NOTE: **Note:** This command removes all recipe and binary package files from the Package Registry. -- **GitLab project interface**: in the packages view of your project page, you can delete packages by clicking the red trash icons. +- From the GitLab user interface: -## Searching the GitLab Package Registry for Conan packages + Go to your project's **Packages & Registries > Package Registry**. Remove the package by clicking the red trash icon. -The `conan search` command can be run searching by full or partial package name, or by exact recipe. +## Search for Conan packages in the Package Registry -To search using a partial name, use the wildcard symbol `*`, which should be placed at the end of your search (for example, `my-packa*`): +To search by full or partial package name, or by exact recipe, run the `conan search` command. -```shell -conan search Hello --all --remote=gitlab -conan search He* --all --remote=gitlab -conan search Hello/0.1@mycompany/beta --all --remote=gitlab -``` +- To search for all packages with a specific package name: + + ```shell + conan search Hello --remote=gitlab + ``` + +- To search for a partial name, like all packages starting with `He`: + + ```shell + conan search He* --remote=gitlab + ``` -The scope of your search includes all projects you have permission to access, this includes your private projects as well as all public projects. +The scope of your search includes all projects you have permission to access. This includes your private projects as well as all public projects. -## Fetching Conan package information from the GitLab Package Registry +## Fetch Conan package information from the Package Registry -The `conan info` command returns information about a given package: +The `conan info` command returns information about a package: ```shell conan info Hello/0.1@mycompany/beta ``` -## List of supported CLI commands +## Supported CLI commands The GitLab Conan repository supports the following Conan CLI commands: -- `conan upload`: Upload your recipe and package files to the GitLab Package Registry. -- `conan install`: Install a conan package from the GitLab Package Registry, this includes using the `conanfile.txt` file. -- `conan search`: Search the GitLab Package Registry for public packages, and private packages you have permission to view. -- `conan info`: View the information on a given package from the GitLab Package Registry. -- `conan remove`: Delete the package from the GitLab Package Registry. - -## Using GitLab CI with Conan packages - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11678) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.7. - -To work with Conan commands within [GitLab CI/CD](./../../../ci/README.md), you can use -`CI_JOB_TOKEN` in place of the personal access token in your commands. - -It's easiest to provide the `CONAN_LOGIN_USERNAME` and `CONAN_PASSWORD` with each -Conan command in your `.gitlab-ci.yml` file. For example: - -```yaml -image: conanio/gcc7 - -create_package: - stage: deploy - script: - - conan remote add gitlab https://gitlab.example.com/api/v4/packages/conan - - conan new <package-name>/0.1 -t - - conan create . <group-name>+<project-name>/stable - - CONAN_LOGIN_USERNAME=ci_user CONAN_PASSWORD=${CI_JOB_TOKEN} conan upload <package-name>/0.1@<group-name>+<project-name>/stable --all --remote=gitlab - -``` - -You can find additional Conan images to use as the base of your CI file -in the [Conan docs](https://docs.conan.io/en/latest/howtos/run_conan_in_docker.html#available-docker-images). +- `conan upload`: Upload your recipe and package files to the Package Registry. +- `conan install`: Install a Conan package from the Package Registry, this includes using the `conanfile.txt` file. +- `conan search`: Search the Package Registry for public packages, and private packages you have permission to view. +- `conan info`: View the information on a given package from the Package Registry. +- `conan remove`: Delete the package from the Package Registry. diff --git a/doc/user/packages/container_registry/img/container_registry_group_repositories_v13_1.png b/doc/user/packages/container_registry/img/container_registry_group_repositories_v13_1.png Binary files differdeleted file mode 100644 index bbbba44eb9b..00000000000 --- a/doc/user/packages/container_registry/img/container_registry_group_repositories_v13_1.png +++ /dev/null diff --git a/doc/user/packages/container_registry/img/container_registry_hover_path_13_4.png b/doc/user/packages/container_registry/img/container_registry_hover_path_13_4.png Binary files differnew file mode 100644 index 00000000000..2d16c11fe61 --- /dev/null +++ b/doc/user/packages/container_registry/img/container_registry_hover_path_13_4.png diff --git a/doc/user/packages/container_registry/img/container_registry_repositories_v13_1.png b/doc/user/packages/container_registry/img/container_registry_repositories_v13_1.png Binary files differdeleted file mode 100644 index 13a6d1a4470..00000000000 --- a/doc/user/packages/container_registry/img/container_registry_repositories_v13_1.png +++ /dev/null diff --git a/doc/user/packages/container_registry/img/container_registry_repositories_with_quickstart_v13_1.png b/doc/user/packages/container_registry/img/container_registry_repositories_with_quickstart_v13_1.png Binary files differdeleted file mode 100644 index 35a02182a77..00000000000 --- a/doc/user/packages/container_registry/img/container_registry_repositories_with_quickstart_v13_1.png +++ /dev/null diff --git a/doc/user/packages/container_registry/img/container_registry_repository_details_v13.0.png b/doc/user/packages/container_registry/img/container_registry_repository_details_v13.0.png Binary files differdeleted file mode 100644 index 088e80221de..00000000000 --- a/doc/user/packages/container_registry/img/container_registry_repository_details_v13.0.png +++ /dev/null diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index 077666bc036..690db3986f1 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -16,100 +16,44 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - The group level Container Registry was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23315) in GitLab 12.10. > - Searching by image repository name was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31322) in GitLab 13.0. -NOTE: **Note:** -This document is the user guide. To learn how to enable GitLab Container -Registry across your GitLab instance, visit the -[administrator documentation](../../../administration/packages/container_registry.md). - -With the Docker Container Registry integrated into GitLab, every project can +With the Docker Container Registry integrated into GitLab, every GitLab project can have its own space to store its Docker images. You can read more about Docker Registry at <https://docs.docker.com/registry/introduction/>. - - -## Enable the Container Registry for your project - -CAUTION: **Warning:** -The Container Registry follows the visibility settings of the project. If the project is public, so is the Container Registry. - -If you cannot find the **Packages & Registries > Container Registry** entry under your -project's sidebar, it is not enabled in your GitLab instance. Ask your -administrator to enable GitLab Container Registry following the -[administration documentation](../../../administration/packages/container_registry.md). - -If you are using GitLab.com, this is enabled by default so you can start using -the Registry immediately. Currently there is a soft (10GB) size restriction for -Registry on GitLab.com, as part of the [repository size limit](../../project/repository/index.md). - -Once enabled for your GitLab instance, to enable Container Registry for your -project: - -1. Go to your project's **Settings > General** page. -1. Expand the **Visibility, project features, permissions** section - and enable the **Container Registry** feature on your project. For new - projects this might be enabled by default. For existing projects - (prior GitLab 8.8), enable it explicitly. -1. Press **Save changes** for the changes to take effect. You should now be able - to see the **Packages & Registries > Container Registry** link in the sidebar. - -## Control Container Registry from within GitLab - -GitLab offers a simple Container Registry management panel. This management panel is available -for both projects and groups. - -### Control Container Registry for your project - -Navigate to your project's **{package}** **Packages & Registries > Container Registry**. - - - -This view allows you to: - -- Show all the image repositories that belong to the project. -- Filter image repositories by their name. -- [Delete](#delete-images-from-within-gitlab) one or more image repository. -- Navigate to the image repository details page. -- Show a **Quick start** dropdown with the most common commands to log in, build and push. -- Show a banner if the optional [cleanup policy](#cleanup-policy) is enabled for this project. - -### Control Container Registry for your group - -Navigate to your group's **{package}** **Packages & Registries > Container Registry**. - - +NOTE: **Note:** +This document is the user guide. To learn how to enable the Container +Registry for your GitLab instance, visit the +[administrator documentation](../../../administration/packages/container_registry.md). -This view allows you to: +## View the Container Registry -- Show all the image repositories of the projects that belong to this group. -- [Delete](#delete-images-from-within-gitlab) one or more image repositories. -- Navigate to a specific image repository details page. +You can view the Container Registry for a project or group. -### Image Repository details page +1. Go to your project or group. +1. Go to **Packages & Registries > Container Registry**. -Clicking on the name of any image repository navigates to the details. +You can search, sort, filter, and [delete](#delete-images-from-within-gitlab) containers on this page. - +CAUTION: **Warning:** +If a project is public, so is the Container Registry. -NOTE: **Note:** -The following page has the same functionalities both in the **Group level container registry** -and in the **Project level container registry**. +## Use images from the Container Registry -This view: +To download and run a container image hosted in the GitLab Container Registry: -- Shows all the image repository details. -- Shows all the tags of the image repository. -- Allows you to quickly copy the tag path (by clicking on the clipboard button near the tag name). -- Allows you to [delete one or more tags](#delete-images-from-within-gitlab). +1. Copy the link to your container image: + - Go to your project or group's **Packages & Registries > Container Registry** + and find the image you want. + - Next to the image name, click the **Copy** button. -## Use images from GitLab Container Registry +  -To download and run a container from images hosted in GitLab Container Registry, -use `docker run`: +1. Use `docker run` with the image link: -```shell -docker run [options] registry.example.com/group/project/image [arguments] -``` + ```shell + docker run [options] registry.example.com/group/project/image [arguments] + ``` For more information on running Docker containers, visit the [Docker documentation](https://docs.docker.com/engine/userguide/intro/). @@ -118,10 +62,10 @@ For more information on running Docker containers, visit the If you visit the **Packages & Registries > Container Registry** link under your project's menu, you can see the explicit instructions to login to the Container Registry -using your GitLab credentials. +by using your GitLab credentials. For example if the Registry's URL is `registry.example.com`, then you should be -able to login with: +able to log in with: ```shell docker login registry.example.com @@ -175,6 +119,10 @@ registry.example.com/group/project/image:latest registry.example.com/group/project/my/image:rc1 ``` +NOTE: **Note:** +Currently there is a soft (10GB) size restriction for +the Container Registry on GitLab.com, as part of the [repository size limit](../../project/repository/index.md). + ## Build and push images using GitLab CI/CD While you can build and push your images from your local machine, take @@ -183,7 +131,7 @@ You can then create workflows and automate any processes that involve testing, building, and eventually deploying your project from the Docker image you created. -Before diving into the details, some things you should be aware of: +Before diving into details, some things you should be aware of: - You must [authenticate to the container registry](#authenticating-to-the-container-registry-with-gitlab-cicd) before running any commands. You can do this in the `before_script` if multiple @@ -359,15 +307,15 @@ in addition to the steps in the Below is an example of what your `.gitlab-ci.yml` should look like: ```yaml - build: - image: $CI_REGISTRY/group/project/docker:19.03.12 - services: - - name: $CI_REGISTRY/group/project/docker:19.03.12-dind - alias: docker - stage: build - script: - - docker build -t my-docker-image . - - docker run my-docker-image /script/to/run/tests +build: + image: $CI_REGISTRY/group/project/docker:19.03.12 + services: + - name: $CI_REGISTRY/group/project/docker:19.03.12-dind + alias: docker + stage: build + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests ``` If you forget to set the service alias, the `docker:19.03.12` image is unable to find the @@ -394,7 +342,7 @@ the deleted images. To delete images from within GitLab: -1. Navigate to your project's or group's **{package}** **Packages & Registries > Container Registry**. +1. Navigate to your project's or group's **Packages & Registries > Container Registry**. 1. From the **Container Registry** page, you can select what you want to delete, by either: @@ -406,8 +354,6 @@ To delete images from within GitLab: 1. In the dialog box, click **Remove tag**. -  - ### Delete images using the API If you want to automate the process of deleting images, GitLab provides an API. For more @@ -512,6 +458,11 @@ Cleanup policies can be run on all projects, with these exceptions: for all projects (even those created before 12.8) in [GitLab application settings](../../../api/settings.md#change-application-settings) by setting `container_expiration_policies_enable_historic_entries` to true. + Alternatively, you can execute the following command in the [Rails console](../../../administration/troubleshooting/navigating_gitlab_via_rails_console.md#starting-a-rails-console-session): + + ```ruby + ApplicationSetting.last.update(container_expiration_policies_enable_historic_entries: true) + ``` There are performance risks with enabling it for all projects, especially if you are using an [external registry](./index.md#use-with-external-container-registries). @@ -636,7 +587,7 @@ you can use the Container Registry to store Helm Charts. However, due to the way and stored by Docker, it is not possible for GitLab to parse this data and meet performance standards. [This epic](https://gitlab.com/groups/gitlab-org/-/epics/2313) updates the architecture of the Container Registry to support Helm Charts. -You can read more about the above challenges [here](https://gitlab.com/gitlab-org/gitlab/-/issues/38047#note_298842890). +[Read more about the above challenges](https://gitlab.com/gitlab-org/gitlab/-/issues/38047#note_298842890). ## Limitations @@ -647,6 +598,19 @@ Container Registry, you must delete all existing images. - Prior to GitLab 12.10, any tags that use the same image ID as the `latest` tag are not deleted by the cleanup policy. +## Disable the Container Registry for a project + +The Container Registry is enabled by default. + +You can, however, remove the Container Registry for a project: + +1. Go to your project's **Settings > General** page. +1. Expand the **Visibility, project features, permissions** section + and disable **Container Registry**. +1. Click **Save changes**. + +The **Packages & Registries > Container Registry** entry is removed from the project's sidebar. + ## Troubleshooting the GitLab Container Registry ### Docker connection error @@ -661,6 +625,13 @@ To get around this, you can [change the group path](../../group/index.md#changin [change the project path](../../project/settings/index.md#renaming-a-repository) or change the branch name. +You may also get a `404 Not Found` or `Unknown Manifest` message if you are using +a Docker Engine version earlier than 17.12. Later versions of Docker Engine use +[the v2 API](https://docs.docker.com/registry/spec/manifest-v2-2/). + +The images in your GitLab Container Registry must also use the Docker v2 API. +For information on how to update your images, see the [Docker help](https://docs.docker.com/registry/spec/deprecated-schema-v1). + ### Troubleshoot as a GitLab server admin Troubleshooting the GitLab Container Registry, most of the times, requires diff --git a/doc/user/packages/dependency_proxy/img/group_dependency_proxy.png b/doc/user/packages/dependency_proxy/img/group_dependency_proxy.png Binary files differdeleted file mode 100644 index e550d296d5a..00000000000 --- a/doc/user/packages/dependency_proxy/img/group_dependency_proxy.png +++ /dev/null diff --git a/doc/user/packages/dependency_proxy/index.md b/doc/user/packages/dependency_proxy/index.md index e3ee8909165..3fa21ef486b 100644 --- a/doc/user/packages/dependency_proxy/index.md +++ b/doc/user/packages/dependency_proxy/index.md @@ -8,80 +8,80 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. -NOTE: **Note:** -This is the user guide. In order to use the dependency proxy, an administrator -must first [configure it](../../../administration/packages/dependency_proxy.md). +The GitLab Dependency Proxy is a local proxy you can use for your frequently-accessed +upstream images. -For many organizations, it is desirable to have a local proxy for frequently used -upstream images/packages. In the case of CI/CD, the proxy is responsible for -receiving a request and returning the upstream image from a registry, acting -as a pull-through cache. +In the case of CI/CD, the Dependency Proxy receives a request and returns the +upstream image from a registry, acting as a pull-through cache. -The dependency proxy is available in the group level. To access it, navigate to -a group's **Packages & Registries > Dependency Proxy**. +## Prerequisites - +To use the Dependency Proxy: -## Supported dependency proxies +- Your group must be public. Authentication for private groups is [not supported yet](https://gitlab.com/gitlab-org/gitlab/-/issues/11582). -NOTE: **Note:** -For a list of the upcoming additions to the proxies, visit the -[direction page](https://about.gitlab.com/direction/package/dependency_proxy/#top-vision-items). +### Supported images and packages -The following dependency proxies are supported. +The following images and packages are supported. -| Dependency proxy | GitLab version | +| Image/Package | GitLab version | | ---------------- | -------------- | | Docker | 11.11+ | -## Using the Docker dependency proxy +For a list of planned additions, view the +[direction page](https://about.gitlab.com/direction/package/dependency_proxy/#top-vision-items). + +## View the Dependency Proxy + +To view the Dependency Proxy: + +- Go to your group's **Packages & Registries > Dependency Proxy**. + +The Dependency Proxy is not available for projects. + +## Use the Dependency Proxy for Docker images + +You can use GitLab as a source for your Docker images. -With the Docker dependency proxy, you can use GitLab as a source for a Docker image. -To get a Docker image into the dependency proxy: +Prerequisites: -1. Find the proxy URL on your group's page under **Packages & Registries > Dependency Proxy**, - for example `gitlab.com/groupname/dependency_proxy/containers`. -1. Trigger GitLab to pull the Docker image you want (e.g., `alpine:latest` or - `linuxserver/nextcloud:latest`) and store it in the proxy storage by using - one of the following ways: +- Your images must be stored on [Docker Hub](https://hub.docker.com/). +- Docker Hub must be available. Follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/241639) + for progress on accessing images when Docker Hub is down. - - Manually pulling the Docker image: +To store a Docker image in Dependency Proxy storage: + +1. Go to your group's **Packages & Registries > Dependency Proxy**. +1. Copy the **Dependency Proxy URL**. +1. Use one of these commands. In these examples, the image is `alpine:latest`. + + - Add the URL to your [`.gitlab-ci.yml`](../../../ci/yaml/README.md#image) file: ```shell - docker pull gitlab.com/groupname/dependency_proxy/containers/alpine:latest + image: gitlab.example.com/groupname/dependency_proxy/containers/alpine:latest ``` - - From a `Dockerfile`: + - Manually pull the Docker image: ```shell - FROM gitlab.com/groupname/dependency_proxy/containers/alpine:latest + docker pull gitlab.example.com/groupname/dependency_proxy/containers/alpine:latest ``` - - In [`.gitlab-ci.yml`](../../../ci/yaml/README.md#image): + - Add the URL to a `Dockerfile`: ```shell - image: gitlab.com/groupname/dependency_proxy/containers/alpine:latest + FROM gitlab.example.com/groupname/dependency_proxy/containers/alpine:latest ``` GitLab pulls the Docker image from Docker Hub and caches the blobs on the GitLab server. The next time you pull the same image, GitLab gets the latest -information about the image from Docker Hub but serves the existing blobs +information about the image from Docker Hub, but serves the existing blobs from the GitLab server. -The blobs are kept forever, and there is no hard limit on how much data can be -stored. - -## Clearing the cache +## Clear the Dependency Proxy cache -It is possible to use the GitLab API to purge the dependency proxy cache for a -given group to gain back disk space that may be taken up by image blobs that -are no longer needed. See the [dependency proxy API documentation](../../../api/dependency_proxy.md) -for more details. - -## Limitations - -The following limitations apply: +Blobs are kept forever on the GitLab server, and there is no hard limit on how much data can be +stored. -- Only [public groups are supported](https://gitlab.com/gitlab-org/gitlab/-/issues/11582) (authentication is not supported yet). -- Only Docker Hub is supported. -- This feature requires Docker Hub being available. +To reclaim disk space used by image blobs that are no longer needed, use +the [Dependency Proxy API](../../../api/dependency_proxy.md). diff --git a/doc/user/packages/go_proxy/index.md b/doc/user/packages/go_proxy/index.md index edf1528a751..bd3b5b49ebd 100644 --- a/doc/user/packages/go_proxy/index.md +++ b/doc/user/packages/go_proxy/index.md @@ -50,7 +50,7 @@ Feature.disable(:go_proxy, Project.find(2)) ### Enable the Package Registry The Package Registry is enabled for new projects by default. If you cannot find -the **{package}** **Packages > List** entry under your project's sidebar, verify +the **Packages > List** entry under your project's sidebar, verify the following: 1. Your GitLab administrator has diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index 2a1c12c2afd..c1b2b16e39d 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -286,6 +286,22 @@ By default, when an NPM package is not found in the GitLab NPM Registry, the req Administrators can disable this behavior in the [Continuous Integration settings](../../admin_area/settings/continuous_integration.md). +### Installing packages from other organizations + +You can route package requests to organizations and users outside of GitLab. + +To do this, add lines to your `.npmrc` file, replacing `my-org` with the namespace or group that owns your project's repository. The name is case-sensitive and must match the name of your group or namespace exactly. + +```shell +@foo:registry=https://gitlab.example.com/api/v4/packages/npm/ +//gitlab.com/api/v4/packages/npm/:_authToken= "<your_token>" +//gitlab.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken= "<your_token>" + +@my-other-org:registry=https://gitlab.example.com/api/v4/packages/npm/ +//gitlab.com/api/v4/packages/npm/:_authToken= "<your_token>" +//gitlab.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken= "<your_token>" +``` + ## Removing a package In the packages view of your project page, you can delete packages by clicking @@ -418,5 +434,8 @@ npm dist-tag rm @scope/package@version my-tag # Delete a tag from the package npm install @scope/package@my-tag # Install a specific tag ``` +NOTE: **Note:** +You cannot use your `CI_JOB_TOKEN` or deploy token with the `npm dist-tag` commands. View [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/258835) for details. + CAUTION: **Warning:** Due to a bug in NPM 6.9.0, deleting dist tags fails. Make sure your NPM version is greater than 6.9.1. diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index 9fb50ce71fb..c40db409903 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -250,21 +250,21 @@ is updated: 1. Add a `deploy` job to your `.gitlab-ci.yml` file: ```yaml - image: mcr.microsoft.com/dotnet/core/sdk:3.1 - - stages: - - deploy - - deploy: - stage: deploy - script: - - dotnet restore -p:Configuration=Release - - dotnet build -c Release - - dotnet pack -c Release - - dotnet nuget add source "$CI_SERVER_URL/api/v4/projects/$CI_PROJECT_ID/packages/nuget/index.json" --name gitlab --username gitlab-ci-token --password $CI_JOB_TOKEN --store-password-in-clear-text - - dotnet nuget push "bin/Release/*.nupkg" --source gitlab - only: - - master + image: mcr.microsoft.com/dotnet/core/sdk:3.1 + + stages: + - deploy + + deploy: + stage: deploy + script: + - dotnet restore -p:Configuration=Release + - dotnet build -c Release + - dotnet pack -c Release + - dotnet nuget add source "$CI_SERVER_URL/api/v4/projects/$CI_PROJECT_ID/packages/nuget/index.json" --name gitlab --username gitlab-ci-token --password $CI_JOB_TOKEN --store-password-in-clear-text + - dotnet nuget push "bin/Release/*.nupkg" --source gitlab + only: + - master ``` 1. Commit the changes and push it to your GitLab repository to trigger a new CI build. diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index 9f954627b05..c4c00f39bae 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -17,7 +17,7 @@ packages, which can be easily consumed as a dependency in downstream projects. You can view packages for your project or group. 1. Go to the project or group. -1. Go to **{package}** **Packages & Registries > Package Registry**. +1. Go to **Packages & Registries > Package Registry**. You can search, sort, and filter packages on this page. @@ -31,7 +31,7 @@ authenticate with GitLab by using the `CI_JOB_TOKEN`. CI/CD templates, which you can use to get started, are in [this repo](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). -Learn more about [using CI/CD to build Maven packages](../maven_repository/index.md#creating-maven-packages-with-gitlab-cicd), [NPM packages](../npm_registry/index.md#publishing-a-package-with-cicd), [Composer packages](../composer_repository/index.md#publishing-the-package-with-cicd), [NuGet Packages](../nuget_repository/index.md#publishing-a-nuget-package-with-cicd), [Conan Packages](../conan_repository/index.md#using-gitlab-ci-with-conan-packages), and [PyPI packages](../pypi_repository/index.md#using-gitlab-ci-with-pypi-packages). +Learn more about [using CI/CD to build Maven packages](../maven_repository/index.md#creating-maven-packages-with-gitlab-cicd), [NPM packages](../npm_registry/index.md#publishing-a-package-with-cicd), [Composer packages](../composer_repository/index.md#publish-a-composer-package-by-using-cicd), [NuGet Packages](../nuget_repository/index.md#publishing-a-nuget-package-with-cicd), [Conan Packages](../conan_repository/index.md#publish-a-conan-package-by-using-cicd), and [PyPI packages](../pypi_repository/index.md#using-gitlab-ci-with-pypi-packages). If you use CI/CD to build a package, extended activity information is displayed when you view the package details: @@ -45,7 +45,7 @@ user who triggered it. To download a package: -1. Go to **{package}** **Packages & Registries > Package Registry**. +1. Go to **Packages & Registries > Package Registry**. 1. Click the name of the package you want to download. 1. In the **Activity** section, click the name of the package you want to download. @@ -60,7 +60,7 @@ You can delete packages by using [the API](../../../api/packages.md#delete-a-pro To delete a package in the UI, from your group or project: -1. Go to **{package}** **Packages & Registries > Package Registry**. +1. Go to **Packages & Registries > Package Registry**. 1. Find the name of the package you want to delete. 1. Click **Delete**. @@ -71,7 +71,7 @@ The package is permanently deleted. The Package Registry is automatically enabled. If you are using a self-managed instance of GitLab, your administrator can remove -the menu item, **{package}** **Packages & Registries**, from the GitLab sidebar. For more information, +the menu item, **Packages & Registries**, from the GitLab sidebar. For more information, see the [administration documentation](../../../administration/packages/index.md). You can also remove the Package Registry for your project specifically: @@ -81,7 +81,7 @@ You can also remove the Package Registry for your project specifically: **Packages** feature. 1. Click **Save changes**. -The **{package}** **Packages & Registries > Package Registry** entry is removed from the sidebar. +The **Packages & Registries > Package Registry** entry is removed from the sidebar. ## Package workflows diff --git a/doc/user/packages/pypi_repository/index.md b/doc/user/packages/pypi_repository/index.md index 97f3f69d676..7d79da7d79b 100644 --- a/doc/user/packages/pypi_repository/index.md +++ b/doc/user/packages/pypi_repository/index.md @@ -274,7 +274,7 @@ Where: Install the latest version of a package using the following command: ```shell -pip install --index-url https://__token__:<personal_access_token>@gitlab.com/api/v4/projects/<project_id>/packages/pypi/simple --no-deps <package_name> +pip install --extra-index-url https://__token__:<personal_access_token>@gitlab.com/api/v4/projects/<project_id>/packages/pypi/simple --no-deps <package_name> ``` Where: @@ -287,7 +287,7 @@ If you were following the guide above and want to test installing the `MyPyPiPackage` package, you can run the following: ```shell -pip install mypypipackage --no-deps --index-url https://__token__:<personal_access_token>@gitlab.com/api/v4/projects/<your_project_id>/packages/pypi/simple +pip install mypypipackage --no-deps --extra-index-url https://__token__:<personal_access_token>@gitlab.com/api/v4/projects/<your_project_id>/packages/pypi/simple ``` This should result in the following: diff --git a/doc/user/packages/workflows/project_registry.md b/doc/user/packages/workflows/project_registry.md index 571cda09e69..be4793f72f3 100644 --- a/doc/user/packages/workflows/project_registry.md +++ b/doc/user/packages/workflows/project_registry.md @@ -80,11 +80,11 @@ Now you can [deploy Maven packages](../maven_repository/index.md#uploading-packa #### Conan -For Conan, first you need to add GitLab as a Conan registry remote. Follow the instructions in the [GitLab Conan Repository docs](../conan_repository/index.md#adding-the-gitlab-package-registry-as-a-conan-remote) +For Conan, first you need to add GitLab as a Conan registry remote. Follow the instructions in the [GitLab Conan Repository docs](../conan_repository/index.md#add-the-package-registry-as-a-conan-remote) to do so. Then, create your package using the plus-separated (`+`) project path as your Conan user. For example, if your project is located at `https://gitlab.com/foo/bar/my-proj`, then you can [create your Conan package](../conan_repository/index.md) using `conan create . foo+bar+my-proj/channel`, where `channel` is your package channel (`stable`, `beta`, etc.). Once your package -is created, you are ready to [upload your package](../conan_repository/index.md#uploading-a-package) depending on your final package recipe. For example: +is created, you are ready to [upload your package](../conan_repository/index.md#publish-a-conan-package) depending on your final package recipe. For example: ```shell CONAN_LOGIN_USERNAME=<gitlab-username> CONAN_PASSWORD=<personal_access_token> conan upload MyPackage/1.0.0@foo+bar+my-proj/channel --all --remote=gitlab diff --git a/doc/user/permissions.md b/doc/user/permissions.md index e2baac1a962..3c11de76b81 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -8,7 +8,7 @@ Users have different abilities depending on the access level they have in a particular group or project. If a user is both in a project's group and the project itself, the highest permission level is used. -On public and internal projects the Guest role is not enforced. All users can: +On public and internal projects, the Guest role is not enforced. All users can: - Create issues. - Leave comments. @@ -72,7 +72,7 @@ The following table depicts the various user permission levels in a project. | Set issue weight | | ✓ | ✓ | ✓ | ✓ | | Lock issue threads | | ✓ | ✓ | ✓ | ✓ | | Manage issue tracker | | ✓ | ✓ | ✓ | ✓ | -| Manage related issues **(STARTER)** | | ✓ | ✓ | ✓ | ✓ | +| Manage related issues | | ✓ | ✓ | ✓ | ✓ | | Manage labels | | ✓ | ✓ | ✓ | ✓ | | Create code snippets | | ✓ | ✓ | ✓ | ✓ | | See a commit status | | ✓ | ✓ | ✓ | ✓ | @@ -116,6 +116,7 @@ The following table depicts the various user permission levels in a project. | Create vulnerability from vulnerability finding **(ULTIMATE)** | | | ✓ | ✓ | ✓ | | Resolve vulnerability **(ULTIMATE)** | | | ✓ | ✓ | ✓ | | Dismiss vulnerability **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| Revert vulnerability to detected state **(ULTIMATE)** | | | ✓ | ✓ | ✓ | | Apply code change suggestions | | | ✓ | ✓ | ✓ | | Create and edit wiki pages | | | ✓ | ✓ | ✓ | | Rewrite/remove Git tags | | | ✓ | ✓ | ✓ | @@ -159,6 +160,7 @@ The following table depicts the various user permission levels in a project. | Remove fork relationship | | | | | ✓ | | Delete project | | | | | ✓ | | Archive project | | | | | ✓ | +| Export project | | | | ✓ | ✓ | | Delete issues | | | | | ✓ | | Delete pipelines | | | | | ✓ | | Delete merge request | | | | | ✓ | @@ -175,7 +177,7 @@ The following table depicts the various user permission levels in a project. \* Owner permission is only available at the group or personal namespace level (and for instance admins) and is inherited by its projects. -1. Guest users are able to perform this action on public and internal projects, but not private projects. +1. Guest users are able to perform this action on public and internal projects, but not private projects. This doesn't apply to [external users](#external-users) where explicit access must be given even if the project is internal. 1. Guest users can only view the confidential issues they created themselves. 1. If **Public pipelines** is enabled in **Project Settings > CI/CD**. 1. Not allowed for Guest, Reporter, Developer, Maintainer, or Owner. See [Protected Branches](./project/protected_branches.md). @@ -195,7 +197,7 @@ which visibility level you select on project settings. - Disabled: disabled for everyone - Only team members: only team members can see even if your project is public or internal -- Everyone with access: everyone can see depending on your project visibility level +- Everyone with access: everyone can see depending on your project's visibility level - Everyone: enabled for everyone (only available for GitLab Pages) ### Protected branches @@ -224,7 +226,7 @@ Read through the documentation on [permissions for File Locking](project/file_lo ### Confidential Issues permissions -Confidential issues can be accessed by reporters and higher permission levels, +Confidential issues can be accessed by users with reporter and higher permission levels, as well as by guest users that create a confidential issue. To learn more, read through the documentation on [permissions and access to confidential issues](project/issues/confidential_issues.md#permissions-and-access-to-confidential-issues). @@ -283,7 +285,7 @@ group. - The [instance level](admin_area/settings/visibility_and_access_controls.md#default-project-creation-protection). - The [group level](group/index.md#default-project-creation-level). 1. Does not apply to subgroups. -1. Developers can push commits to the default branch of a new project only if the [default branch protection](group/index.md#changing-the-default-branch-protection-of-a-group) is set to "Partially protected" or "Not protected". +1. Developers can push commits to the default branch of a new project only if the [default branch protection](group/index.md#changing-the-default-branch-protection-of-a-group) is set to "Partially protected" or "Not protected". ### Subgroup permissions @@ -311,7 +313,7 @@ External users: Access can be granted by adding the user as member to the project or group. Like usual users, they receive a role in the project or group with all the abilities that are mentioned in the [permissions table above](#project-members-permissions). -For example, if an external user is added as Guest, and your project is +For example, if an external user is added as Guest, and your project is internal or private, they do not have access to the code; you need to grant the external user access at the Reporter level or above if you want them to have access to the code. You should always take into account the @@ -339,7 +341,7 @@ The **Internal users** field allows specifying an email address regex pattern to identify default internal users. New users whose email address matches the regex pattern are set to internal by default rather than an external collaborator. -The regex pattern format is Ruby, but it needs to be convertible to JavaScript, +The regex pattern format is in Ruby, but it needs to be convertible to JavaScript, and the ignore case flag is set (`/regex pattern/i`). Here are some examples: - Use `\.internal@domain\.com$` to mark email addresses ending with @@ -384,6 +386,16 @@ with the permissions described on the documentation on [auditor users permission [Read more about Auditor users.](../administration/auditor_users.md) +## Users with minimal access **(PREMIUM ONLY)** + +>[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40942) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. + +Administrators can add members with a "minimal access" role to a parent group. Such users don't +automatically have access to projects and subgroups underneath. To support such access, administrators must explicitly add these "minimal access" users to the specific subgroups/projects. + +Users with minimal access can list the group in the UI and through the API. However, they cannot see +details such as projects or subgroups. They do not have access to the group's page or list any of itssubgroups or projects. + ## Project features Project features like wiki and issues can be hidden from users depending on diff --git a/doc/user/profile/account/create_accounts.md b/doc/user/profile/account/create_accounts.md index 26c2c1bed89..09bfa7afc9e 100644 --- a/doc/user/profile/account/create_accounts.md +++ b/doc/user/profile/account/create_accounts.md @@ -31,9 +31,9 @@ You can also [create users through the API](../../../api/users.md) as an admin.  -## Create users through integrations +## Create users through authentication integrations Users will be: -- Automatically created upon first login with the [LDAP integration](../../../administration/auth/ldap/index.md). -- Created when first logging in via an [OmniAuth provider](../../../integration/omniauth.md) if the `allow_single_sign_on` setting is present. +- Automatically created upon first sign in with the [LDAP integration](../../../administration/auth/ldap/index.md). +- Created when first signing in via an [OmniAuth provider](../../../integration/omniauth.md) if the `allow_single_sign_on` setting is present. diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index 1b8c16f401c..1c0c1255ee3 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -14,7 +14,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w If you're unable to use [OAuth2](../../api/oauth2.md), you can use a personal access token to authenticate with the [GitLab API](../../api/README.md#personalproject-access-tokens). -You can also use personal access tokens with Git to authenticate over HTTP or SSH. Personal access tokens are required when [Two-Factor Authentication (2FA)](../account/two_factor_authentication.md) is enabled. In both cases, you can authenticate with a token in place of your password. +You can also use personal access tokens with Git to authenticate over HTTP or SSH. Personal access tokens are required when [Two-Factor Authentication (2FA)](account/two_factor_authentication.md) is enabled. In both cases, you can authenticate with a token in place of your password. Personal access tokens expire on the date you define, at midnight UTC. diff --git a/doc/user/project/clusters/add_eks_clusters.md b/doc/user/project/clusters/add_eks_clusters.md index b3b1b51a543..b2eb1c51745 100644 --- a/doc/user/project/clusters/add_eks_clusters.md +++ b/doc/user/project/clusters/add_eks_clusters.md @@ -61,21 +61,10 @@ To create and add a new Kubernetes cluster to your project, group, or instance: - **Admin Area > Kubernetes**, for an instance-level cluster. 1. Click **Add Kubernetes cluster**. 1. Under the **Create new cluster** tab, click **Amazon EKS**. You will be provided with an - `Account ID` and `External ID` to use in the next step. -1. In the [IAM Management Console](https://console.aws.amazon.com/iam/home), create an EKS management IAM role. - To do so, follow the [Amazon EKS cluster IAM role](https://docs.aws.amazon.com/eks/latest/userguide/service_IAM_role.html) instructions - to create a IAM role suitable for managing the AWS EKS cluster's resources on your behalf. - In addition to the policies that guide suggests, you must also include the `AmazonEKSClusterPolicy` - policy for this role in order for GitLab to manage the EKS cluster correctly. -1. In the [IAM Management Console](https://console.aws.amazon.com/iam/home), create an IAM role: - 1. From the left panel, select **Roles**. - 1. Click **Create role**. - 1. Under `Select type of trusted entity`, select **Another AWS account**. - 1. Enter the Account ID from GitLab into the `Account ID` field. - 1. Check **Require external ID**. - 1. Enter the External ID from GitLab into the `External ID` field. - 1. Click **Next: Permissions**. - 1. Click **Create Policy**, which will open a new window. + `Account ID` and `External ID` needed for later steps. +1. In the [IAM Management Console](https://console.aws.amazon.com/iam/home), create an IAM policy: + 1. From the left panel, select **Policies**. + 1. Click **Create Policy**, which opens a new window. 1. Select the **JSON** tab, and paste in the following snippet in place of the existing content: ```json @@ -131,7 +120,20 @@ To create and add a new Kubernetes cluster to your project, group, or instance: 1. Click **Review policy**. 1. Enter a suitable name for this policy, and click **Create Policy**. You can now close this window. - 1. Switch back to the "Create role" window, and select the policy you just created. + +1. In the [IAM Management Console](https://console.aws.amazon.com/iam/home), create an EKS management IAM role. + To do so, follow the [Amazon EKS cluster IAM role](https://docs.aws.amazon.com/eks/latest/userguide/service_IAM_role.html) instructions + to create a IAM role suitable for managing the AWS EKS cluster's resources on your behalf. + In addition to the policies that guide suggests, you must also include the `AmazonEKSClusterPolicy` + policy for this role in order for GitLab to manage the EKS cluster correctly. +1. In the [IAM Management Console](https://console.aws.amazon.com/iam/home), create an IAM role: + 1. From the left panel, select **Roles**. + 1. Click **Create role**. + 1. Under `Select type of trusted entity`, select **Another AWS account**. + 1. Enter the Account ID from GitLab into the `Account ID` field. + 1. Check **Require external ID**. + 1. Enter the External ID from GitLab into the `External ID` field. + 1. Click **Next: Permissions**, and select the policy you just created. 1. Click **Next: Tags**, and optionally enter any tags you wish to associate with this role. 1. Click **Next: Review**. 1. Enter a role name and optional description into the fields provided. diff --git a/doc/user/project/clusters/add_remove_clusters.md b/doc/user/project/clusters/add_remove_clusters.md index 18d9fa67ee1..21f216aa50a 100644 --- a/doc/user/project/clusters/add_remove_clusters.md +++ b/doc/user/project/clusters/add_remove_clusters.md @@ -191,8 +191,20 @@ To add a Kubernetes cluster to your project, group, or instance: ``` NOTE: **Note:** - If the command returns the entire certificate chain, you need copy the *root ca* - certificate at the bottom of the chain. + If the command returns the entire certificate chain, you must copy the Root CA + certificate and any intermediate certificates at the bottom of the chain. + A chain file has following structure: + + ```plaintext + -----BEGIN MY CERTIFICATE----- + -----END MY CERTIFICATE----- + -----BEGIN INTERMEDIATE CERTIFICATE----- + -----END INTERMEDIATE CERTIFICATE----- + -----BEGIN INTERMEDIATE CERTIFICATE----- + -----END INTERMEDIATE CERTIFICATE----- + -----BEGIN ROOT CERTIFICATE----- + -----END ROOT CERTIFICATE----- + ``` 1. **Token** - GitLab authenticates against Kubernetes using service tokens, which are @@ -257,7 +269,7 @@ To add a Kubernetes cluster to your project, group, or instance: Copy the `<authentication_token>` value from the output: - ```yaml + ```plaintext Name: gitlab-token-b5zv4 Namespace: kube-system Labels: <none> diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index 8d188f00ceb..e60e3fcd4e7 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -12,8 +12,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/39840) in > GitLab 11.11 for [instances](../../instance/clusters/index.md). -## Overview - Using the GitLab project Kubernetes integration, you can: - Use [Review Apps](../../../ci/review_apps/index.md). @@ -31,6 +29,11 @@ Besides integration at the project level, Kubernetes clusters can also be integrated at the [group level](../../group/clusters/index.md) or [GitLab instance level](../../instance/clusters/index.md). +To view your project level Kubernetes clusters, navigate to **Operations > Kubernetes** +from your project. On this page, you can [add a new cluster](#adding-and-removing-clusters) +and view information about your existing clusters, such as nodes count and rough estimates +of memory and CPU usage. + ## Setting up ### Supported cluster versions @@ -265,20 +268,43 @@ If your cluster was created before GitLab 12.2, default `KUBE_NAMESPACE` will be ### Custom namespace -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. +> - An option to use project-wide namespaces [was added](https://gitlab.com/gitlab-org/gitlab/-/issues/38054) in GitLab 13.5. + +The Kubernetes integration provides a `KUBECONFIG` with an auto-generated namespace +to deployment jobs. It defaults to using project-environment specific namespaces +of the form `<prefix>-<environment>`, where `<prefix>` is of the form +`<project_name>-<project_id>`. To learn more, read [Deployment variables](#deployment-variables). -The Kubernetes integration defaults to project-environment-specific namespaces -of the form `<project_name>-<project_id>-<environment>` (see [Deployment -variables](#deployment-variables)). +You can customize the deployment namespace in a few ways: -For **non**-GitLab-managed clusters, the namespace can be customized using -[`environment:kubernetes:namespace`](../../../ci/environments/index.md#configuring-kubernetes-deployments) -in `.gitlab-ci.yml`. +- You can choose between a **namespace per [environment](../../../ci/environments/index.md)** + or a **namespace per project**. A namespace per environment is the default and recommended + setting, as it prevents the mixing of resources between production and non-production environments. +- When using a project-level cluster, you can additionally customize the namespace prefix. + When using namespace-per-environment, the deployment namespace is `<prefix>-<environment>`, + but otherwise just `<prefix>`. +- For **non-managed** clusters, the auto-generated namespace is set in the `KUBECONFIG`, + but the user is responsible for ensuring its existence. You can fully customize + this value using + [`environment:kubernetes:namespace`](../../../ci/environments/index.md#configuring-kubernetes-deployments) + in `.gitlab-ci.yml`. NOTE: **Note:** -When using a [GitLab-managed cluster](#gitlab-managed-clusters), the -namespaces are created automatically prior to deployment and [can not be -customized](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). +When you customize the namespace, existing environments remain linked to their current +namespaces until you [clear the cluster cache](#clearing-the-cluster-cache). + +CAUTION: **Warning:** +By default, anyone who can create a deployment job can access any CI variable within +an environment's deployment job. This includes `KUBECONFIG`, which gives access to +any secret available to the associated service account in your cluster. +To keep your production credentials safe, consider using +[Protected Environments](../../../ci/environments/protected_environments.md), +combined with either + +- a GitLab-managed cluster and namespace per environment, +- *or*, an environment-scoped cluster per protected environment. The same cluster + can be added multiple times with multiple restricted service accounts. ### Integrations diff --git a/doc/user/project/clusters/serverless/aws.md b/doc/user/project/clusters/serverless/aws.md index 543ffdbce8f..d662dc4f715 100644 --- a/doc/user/project/clusters/serverless/aws.md +++ b/doc/user/project/clusters/serverless/aws.md @@ -222,7 +222,8 @@ the environment of the deployed function: ```yaml provider: - ... + # Other configuration omitted + # ... environment: A_VARIABLE: ${env:A_VARIABLE} ``` @@ -245,10 +246,10 @@ functions: hello: handler: src/handler.hello events: - - http: # Rewrite this part to enable CORS + - http: # Rewrite this part to enable CORS path: hello method: get - cors: true # <-- CORS here + cors: true # <-- CORS here ``` You also need to return CORS specific headers in your function response: diff --git a/doc/user/project/deploy_keys/index.md b/doc/user/project/deploy_keys/index.md index 81c9008c5b3..4f344554016 100644 --- a/doc/user/project/deploy_keys/index.md +++ b/doc/user/project/deploy_keys/index.md @@ -12,7 +12,7 @@ more repositories, by importing an SSH public key to your GitLab instance. This is useful for cloning repositories to your Continuous Integration (CI) server. By using deploy keys, you don't have to set up a -dummy user account. +fake user account. There are two types of deploy keys: diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index aa5987bf5f9..e0c4097d1c5 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -49,8 +49,8 @@ To create a Markdown file: 1. Click the `+` button next to `master` and click **New file**. 1. Add the name of your issue template to the **File name** text field next to `master`. - Make sure words are separated with underscores and that your file has the `.md` extension, for - example `feature_request.md`. + Make sure that your file has the `.md` extension, for + example `feature_request.md` or `Feature Request.md`. 1. Commit and push to your default branch. If you don't have a `.gitlab/issue_templates` directory in your repository, you'll need to create it. @@ -79,6 +79,9 @@ This will enable the `Bug` dropdown option when creating or editing issues. When to the issue description field. The 'Reset template' button will discard any changes you made after picking the template and return it to its initial status. +TIP: **Tip:** +You can create short-cut links to create an issue using a designated template. For example: `https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20proposal`. +  ## Setting a default template for merge requests and issues **(STARTER)** diff --git a/doc/user/project/img/issue_board_default_lists_v13_4.png b/doc/user/project/img/issue_board_default_lists_v13_4.png Binary files differnew file mode 100644 index 00000000000..23cdc9b4e22 --- /dev/null +++ b/doc/user/project/img/issue_board_default_lists_v13_4.png diff --git a/doc/user/project/img/issue_board_welcome_message.png b/doc/user/project/img/issue_board_welcome_message.png Binary files differdeleted file mode 100644 index 357dff42488..00000000000 --- a/doc/user/project/img/issue_board_welcome_message.png +++ /dev/null diff --git a/doc/user/project/import/bitbucket.md b/doc/user/project/import/bitbucket.md index 89130d5822f..56266718d12 100644 --- a/doc/user/project/import/bitbucket.md +++ b/doc/user/project/import/bitbucket.md @@ -76,6 +76,3 @@ If you've accidentally started the import process with the wrong account, follow 1. Revoke GitLab access to your Bitbucket account, essentially reversing the process in the following procedure: [Import your Bitbucket repositories](#import-your-bitbucket-repositories). 1. Sign out of the Bitbucket account. Follow the procedure linked from the previous step. - -NOTE: **Note:** -To import a repository including LFS objects from a Bitbucket server repository, use the [Repo by URL](../import/repo_by_url.md) importer. diff --git a/doc/user/project/import/bitbucket_server.md b/doc/user/project/import/bitbucket_server.md index d0499730bfe..d8ba4f86924 100644 --- a/doc/user/project/import/bitbucket_server.md +++ b/doc/user/project/import/bitbucket_server.md @@ -37,12 +37,7 @@ Import your projects from Bitbucket Server to GitLab with minimal effort. empty changes. 1. Attachments in Markdown are currently not imported. 1. Task lists are not imported. -1. Emoji reactions are not imported. -1. [LFS objects](../../../topics/git/lfs/index.md) are not imported. - - NOTE: **Note:** - To import a repository including LFS objects from a Bitbucket server repository, use the [Repo by URL](../import/repo_by_url.md) importer. - +1. Emoji reactions are not imported 1. Project filtering does not support fuzzy search (only `starts with` or `full match strings` are currently supported) @@ -69,20 +64,43 @@ namespace that started the import process. #### User assignment by username -Alternatively, user assignment by username is available behind a `bitbucket_server_user_mapping_by_username` feature flag. -The importer will try to find a user in the GitLab user database using author's `username` or `slug` or `displayName`. -Falls back to author's `email` if user is not found by username. -Similarly to user assignment by email, if no such user is available, the project creator is set as the author. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218609) in GitLab 13.4. +> - It's [deployed behind a feature flag](../../feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's not recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to enable it. + +CAUTION: **Warning:** +This feature might not be available to you. Check the **version history** note above for details. + +If you've enabled this feature, the importer tries to find a user in the GitLab user database with +the author's: + +- `username` +- `slug` +- `displayName` + +If the user is not found by any of these properties, the search falls back to the author's +`email` address. -To enable or disable user assignment by username: +Alternatively, if there is also no email address, the project creator is set as the author. -Start a [Rails console](../../../administration/troubleshooting/debug.md#starting-a-rails-console-session). +##### Enable or disable User assignment by username + +User assignment by username is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](<replace with path to>/administration/feature_flags.md) +can enable it. + +To enable it: ```ruby -# Enable Feature.enable(:bitbucket_server_user_mapping_by_username) +``` -# Disable +To disable it: + +```ruby Feature.disable(:bitbucket_server_user_mapping_by_username) ``` diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index 4cd0c9e02c7..be1641f8b16 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -35,25 +35,20 @@ The namespace is a user or group in GitLab, such as `gitlab.com/janedoe` or `git This process does not migrate or import any types of groups or organizations from GitHub to GitLab. -### If you're using GitLab.com - -If you're using GitLab.com, you can alternatively import -GitHub repositories using a [personal access token](#using-a-github-token), -but we don't recommend this method because it can't associate all user activity -(such as issues and pull requests) with matching GitLab users. - -### If you're importing from GitLab Enterprise - -If you're importing from GitHub Enterprise, you must enable [GitHub integration][gh-import]. - -### If you're using a self-managed GitLab instance - -If you're an administrator of a self-managed GitLab instance, you must enable -[GitHub integration][gh-import]. - -If you're an administrator of a self-managed GitLab instance, you can also use the -[GitHub Rake task](../../../administration/raketasks/github_import.md) to import projects from -GitHub without the constraints of a Sidekiq worker. +### Use cases + +The steps you take depend on whether you are importing from GitHub.com or GitHub Enterprise, as well as whether you are importing to GitLab.com or self-managed GitLab instance. + +- If you're importing to GitLab.com, you can alternatively import GitHub repositories + using a [personal access token](#using-a-github-token). We do not recommend + this method, as it does not associate all user activity (such as issues and + pull requests) with matching GitLab users. +- If you're importing to a self-managed GitLab instance, you can alternatively use the + [GitHub Rake task](../../../administration/raketasks/github_import.md) to import + projects without the constraints of a [Sidekiq](../../../development/sidekiq_style_guide.md) worker. +- If you're importing from GitHub Enterprise to your self-managed GitLab instance, you must first enable + [GitHub integration](../../../integration/github.md). However, you cannot import projects from GitHub Enterprise to GitLab.com. +- If you're importing from GitHub.com to your self-managed GitLab instance, you do not need to set up GitHub integration. ## How it works diff --git a/doc/user/project/import/img/manifest_status_v13_3.png b/doc/user/project/import/img/manifest_status_v13_3.png Binary files differindex 3f0063e6715..c1a55ba1f50 100644 --- a/doc/user/project/import/img/manifest_status_v13_3.png +++ b/doc/user/project/import/img/manifest_status_v13_3.png diff --git a/doc/user/project/index.md b/doc/user/project/index.md index c79f2be1d3f..a00f93bac9c 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -34,7 +34,7 @@ When you create a project in GitLab, you'll have access to a large number of - [Protected tags](protected_tags.md): Control over who has permission to create tags, and prevent accidental update or deletion - [Repository mirroring](repository/repository_mirroring.md) - - [Signing commits](gpg_signed_commits/index.md): use GPG to sign your commits + - [Signing commits](repository/gpg_signed_commits/index.md): use GPG to sign your commits - [Deploy tokens](deploy_tokens/index.md): Manage project-based deploy tokens that allow permanent access to the repository and Container Registry. - [Web IDE](web_ide/index.md) - [CVE ID Requests](../application_security/cve_id_request.md): Request a CVE identifier to track a @@ -96,9 +96,9 @@ When you create a project in GitLab, you'll have access to a large number of - [Wiki](wiki/index.md): document your GitLab project in an integrated Wiki. - [Snippets](../snippets.md): store, share and collaborate on code snippets. -- [Value Stream Analytics](cycle_analytics.md): review your development lifecycle. +- [Value Stream Analytics](../analytics/value_stream_analytics.md): review your development lifecycle. - [Insights](insights/index.md): configure the Insights that matter for your projects. **(ULTIMATE)** -- [Security Dashboard](security_dashboard.md): Security Dashboard. **(ULTIMATE)** +- [Security Dashboard](../application_security/security_dashboard/index.md): Security Dashboard. **(ULTIMATE)** - [Syntax highlighting](highlighting.md): an alternative to customize your code blocks, overriding GitLab's default choice of language. - [Badges](badges.md): badges for the project overview. diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index 29efe08e53d..d03241f98a9 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -12,7 +12,7 @@ GitLab has support for automatically detecting and monitoring Kubernetes metrics ## Requirements -The [Prometheus](../prometheus.md) and [Kubernetes](../kubernetes.md) +The [Prometheus](../prometheus.md) and [Kubernetes](../../clusters/index.md) integration services must be enabled. ## Metrics supported diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index f8172a0f988..e039e861d2c 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -8,8 +8,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5554) in [GitLab 8.11](https://about.gitlab.com/releases/2016/08/22/gitlab-8-11-released/#issue-board). -## Overview - The GitLab Issue Board is a software project management tool used to plan, organize, and visualize a workflow for a feature or product release. It can be used as a [Kanban](https://en.wikipedia.org/wiki/Kanban_(development)) or a @@ -374,27 +372,22 @@ If you're not able to do some of the things above, make sure you have the right ### First time using an issue board -The first time you open an issue board, you are presented with -the default lists (**Open** and **Closed**) and a welcome message that gives -you two options. You can either: +> The automatic creation of the **To Do** and **Doing** lists was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202144) in GitLab 13.4. -- Create a predefined set of labels (by default: **To Do** and **Doing**) and create their - corresponding lists to the issue board. -- Opt-out and use your own lists. +The first time you open an issue board, you are presented with the default lists +(**Open**, **To Do**, **Doing**, and **Closed**). - +If the **To Do** and **Doing** labels don't exist in the project or group, they are created, and +their lists appear as empty. If any of them already exists, the list is filled with the issues that +have that label. -If you choose to use and create the predefined lists, they will appear as empty -because the labels associated to them will not exist up until that moment, -which means the system has no way of populating them automatically. That's of -course if the predefined labels don't already exist. If any of them does exist, -the list will be created and filled with the issues that have that label. + ### Create a new list Create a new list by clicking the **Add list** button in the upper right corner of the issue board. - + Then, choose the label or user to create the list from. The new list will be inserted at the end of the lists, before **Done**. Moving and reordering lists is as diff --git a/doc/user/project/issues/design_management.md b/doc/user/project/issues/design_management.md index 7c9278c8403..b92d99fc4a8 100644 --- a/doc/user/project/issues/design_management.md +++ b/doc/user/project/issues/design_management.md @@ -10,8 +10,6 @@ info: "To determine the technical writer assigned to the Stage/Group associated > - Support for SVGs was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12771) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.4. > - Design Management was [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) to GitLab Core in 13.0. -## Overview - Design Management allows you to upload design assets (wireframes, mockups, etc.) to GitLab issues and keep them stored in one single place, accessed by the Design Management's page within an issue, giving product designers, product managers, and engineers a @@ -114,9 +112,6 @@ Designs with the same filename as an existing uploaded design will create a new of the design, and will replace the previous version. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.9, dropping a design on an existing uploaded design will also create a new version, provided the filenames are the same. -Designs cannot be added if the issue has been moved, or its -[discussion is locked](../../discussions/#lock-discussions). - ### Skipped designs Designs with the same filename as an existing uploaded design _and_ whose content has not changed will be skipped. @@ -185,9 +180,7 @@ viewed by browsing previous versions. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34382) in GitLab 13.3. -You can change the order of designs by dragging them to a new position: - - +You can change the order of designs by dragging them to a new position. ## Starting discussions on designs @@ -231,7 +224,7 @@ Note that your resolved comment pins will disappear from the Design to free up s However, if you need to revisit or find a resolved discussion, all of your resolved threads will be available in the **Resolved Comment** area at the bottom of the right sidebar. -## Add To-Do for Designs +## Add to dos for designs > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198439) in GitLab 13.4. > - It's [deployed behind a feature flag](../../feature_flags.md), enabled by default. @@ -242,11 +235,11 @@ available in the **Resolved Comment** area at the bottom of the right sidebar. CAUTION: **Warning:** This feature might not be available to you. Check the **version history** note above for details. -Add a to-do for a design by clicking **Add a To-Do** on the design sidebar: +Add a to do for a design by clicking **Add a To-Do** on the design sidebar:  -### Enable or disable the design To-Do button **(CORE ONLY)** +### Enable or disable the design to-do button **(CORE ONLY)** The **Add a To-Do** button for Designs is under development but ready for production use. It is deployed behind a feature flag that is **enabled by default**. @@ -268,10 +261,7 @@ Feature.disable(:design_management_todo_button) ## Referring to designs in Markdown > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217160) in **GitLab 13.1**. -> - It is deployed behind a feature flag, disabled by default. -> - It is disabled on GitLab.com. -> - It is not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-design-references). **(CORE ONLY)** +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/258662) in **GitLab 13.5** We support referring to designs in [Markdown](../../markdown.md), which is available throughout the application, including in merge request and issue descriptions, in discussions and comments, and in wiki pages. @@ -287,25 +277,6 @@ This will be rendered as: > See [#123[homescreen.png]](https://gitlab.com/your-group/your-project/-/issues/123/designs/homescreen.png) -### Enable or disable design references **(CORE ONLY)** - -Design reference parsing is -deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can disable it for your instance. - -To disable it: - -```ruby -Feature.disable(:design_management_reference_filter_gfm_pipeline) -``` - -To re-enable it: - -```ruby -Feature.enable(:design_management_reference_filter_gfm_pipeline) -``` - ## Design activity records > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33051) in GitLab 13.1. diff --git a/doc/user/project/issues/due_dates.md b/doc/user/project/issues/due_dates.md index 55b45bf9d3d..b3ebefadef0 100644 --- a/doc/user/project/issues/due_dates.md +++ b/doc/user/project/issues/due_dates.md @@ -46,7 +46,7 @@ the icon and the date colored red. You can sort issues by those that are Due dates also appear in your [to-do list](../../todos.md). - + The day before an open issue is due, an email will be sent to all participants of the issue. Like the due date, the "day before the due date" is determined by the diff --git a/doc/user/project/issues/img/designs_reordering_v13_3.gif b/doc/user/project/issues/img/designs_reordering_v13_3.gif Binary files differdeleted file mode 100644 index 496eea532e2..00000000000 --- a/doc/user/project/issues/img/designs_reordering_v13_3.gif +++ /dev/null diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index 060266a478f..434af3a4a49 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -156,12 +156,15 @@ collaborate with your team. efficiently and with less effort by tracking groups of issues that share a theme, across projects and milestones. -### Related issues **(STARTER)** +### Related issues You can mark two issues as related, so that when viewing one, the other is always listed in its [Related Issues](related_issues.md) section. This can help display important context, such as past work, dependencies, or duplicates. +Users on [GitLab Starter, GitLab Bronze, and higher tiers](https://about.gitlab.com/pricing/), can +also mark issues as blocking or blocked by another issue. + ### Crosslinking issues You can [cross-link issues](crosslinking_issues.md) by referencing an issue from another diff --git a/doc/user/project/issues/issue_data_and_actions.md b/doc/user/project/issues/issue_data_and_actions.md index 5356e6aeb40..95d7f2edac5 100644 --- a/doc/user/project/issues/issue_data_and_actions.md +++ b/doc/user/project/issues/issue_data_and_actions.md @@ -36,7 +36,7 @@ You can find all the information for that issue on one screen. - **15.** [Edit](#edit) - **16.** [Description](#description) - **17.** [Mentions](#mentions) -- **18.** [Related Issues **(STARTER)**](#related-issues) +- **18.** [Related Issues](#related-issues) - **19.** [Related Merge Requests](#related-merge-requests) - **20.** [Award emoji](#award-emoji) - **21.** [Show all activity](#show-all-activity) @@ -208,7 +208,7 @@ TIP: **Tip:** Avoid mentioning `@all` in issues and merge requests, as it sends an email notification to all the members of that project's group, which can be interpreted as spam. -### Related Issues **(STARTER)** +### Related Issues Issues that were mentioned as [related issues](related_issues.md) are listed here. You can also click the `+` to add more related issues. @@ -242,7 +242,7 @@ and selecting either: Also: - You can mention a user or a group present in your GitLab instance with - `@username` or `@groupname` and they will be notified via To-Do items + `@username` or `@groupname` and they will be notified via to-do items and email, unless they have [disabled all notifications](#notifications) in their profile settings. - Mentions for yourself (the current logged in user), will be highlighted diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index 0e0731528be..b033dc79dcc 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -127,6 +127,7 @@ field). | title | `issue[title]` | | | description | `issue[description]` | | | description template | `issuable_template` | | +| issue type | `issue[issue_type]` | Either `incident` or `issue` | | confidential | `issue[confidential]` | Parameter value must be `true` to set to confidential | Follow these examples to form your new issue URL with prefilled fields. diff --git a/doc/user/project/issues/multiple_assignees_for_issues.md b/doc/user/project/issues/multiple_assignees_for_issues.md index c11977f5bee..b1806460c08 100644 --- a/doc/user/project/issues/multiple_assignees_for_issues.md +++ b/doc/user/project/issues/multiple_assignees_for_issues.md @@ -8,8 +8,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1904) in [GitLab Starter 9.2](https://about.gitlab.com/releases/2017/05/22/gitlab-9-2-released/#multiple-assignees-for-issues). -## Overview - In large teams, where there is shared ownership of an issue, it can be difficult to track who is working on it, who already completed their contributions, who didn't even start yet. diff --git a/doc/user/project/issues/related_issues.md b/doc/user/project/issues/related_issues.md index 954e3771722..b040bcf3b03 100644 --- a/doc/user/project/issues/related_issues.md +++ b/doc/user/project/issues/related_issues.md @@ -4,33 +4,40 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Related issues **(STARTER)** +# Related issues **(CORE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1797) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1797) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.4. +> - The simple "relates to" relationship [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212329) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.4. Related issues are a bi-directional relationship between any two issues and appear in a block below the issue description. Issues can be across groups and projects. +You can set any issue as: + +- Related to another issue +- Blocking another issue **(STARTER)** +- Blocked by another issue **(STARTER)** + The relationship only shows up in the UI if the user can see both issues. When you try to close an issue that has open blockers, a warning is displayed. TIP: **Tip:** -To manage related issues through our API, see the [API documentation](../../../api/issue_links.md). +To manage related issues through our API, visit the [issue links API documentation](../../../api/issue_links.md). ## Adding a related issue > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2035) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/34239) to warn when attempting to close an issue that is blocked by others in [GitLab Starter](https://about.gitlab.com/pricing/) 13.0. -> When you try to close an issue with open blockers, you'll see a warning that you can dismiss. +> When you try to close an issue with open blockers, you see a warning that you can dismiss. 1. Relate one issue to another by clicking the related issues "+" button in the header of the related issue block. 1. Input the issue reference number or paste in the full URL of the issue. -1. Select whether the current issue relates to, blocks, or is blocked by the issues being entered. +1. **(STARTER)** Select whether the current issue relates to, blocks, or is blocked by the issues being entered.  @@ -42,11 +49,11 @@ in the header of the related issue block. - same group: `project#44` - different group: `group/project#44` - Valid references will be added to a temporary list that you can review. + Valid references are added to a temporary list that you can review. 1. When you have added all the related issues, click **Add** to submit. -When you have finished adding all related issues, you will be able to see +When you have finished adding all related issues, you can see them categorized so their relationships can be better understood visually.  @@ -56,7 +63,7 @@ them categorized so their relationships can be better understood visually. In the related issues block, click the "x" icon on the right-side of each issue token that you wish to remove. -Due to the bi-directional relationship, it will no longer appear in either issue. +Due to the bi-directional relationship, it no longer appears in either issue.  diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index 0d02b89be7e..1f2504a4f60 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Labels -## Overview - As your count of issues, merge requests, and epics grows in GitLab, it's more and more challenging to keep track of those items. Especially as your organization grows from just a few people to hundreds or thousands. This is where labels come in. They help you organize and tag your work diff --git a/doc/user/project/merge_requests/img/commit_nav_v13_4.png b/doc/user/project/merge_requests/img/commit_nav_v13_4.png Binary files differnew file mode 100644 index 00000000000..0ae6ce32693 --- /dev/null +++ b/doc/user/project/merge_requests/img/commit_nav_v13_4.png diff --git a/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md b/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md index 3a18cacde64..2e0c0d7aeeb 100644 --- a/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md +++ b/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md @@ -122,6 +122,8 @@ the commits to open the single-commit view. From there, you can navigate among t by clicking the **Prev** and **Next** buttons on the top-right of the page or by using the <kbd>X</kbd> and <kbd>C</kbd> keyboard shortcuts. + + ### Incrementally expand merge request diffs By default, the diff shows only the parts of a file which are changed. diff --git a/doc/user/project/merge_requests/test_coverage_visualization.md b/doc/user/project/merge_requests/test_coverage_visualization.md index 56b5774f15b..b81158c3653 100644 --- a/doc/user/project/merge_requests/test_coverage_visualization.md +++ b/doc/user/project/merge_requests/test_coverage_visualization.md @@ -5,13 +5,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, howto --- -# Test Coverage Visualization **(CORE ONLY)** +# Test Coverage Visualization > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. -> - [Feature flag enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/211410) in GitLab 13.4. -> - It's enabled on GitLab.com. -> - It can be disabled per-project. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enabling-the-feature). **(CORE ONLY)** +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/249811) in GitLab 13.5. With the help of [GitLab CI/CD](../../../ci/README.md), you can collect the test coverage information of your favorite testing or coverage-analysis tool, and visualize @@ -58,7 +55,9 @@ NOTE: **Note:** The Cobertura XML parser currently does not support the `sources` element and ignores it. It is assumed that the `filename` of a `class` element contains the full path relative to the project root. -## Example test coverage configuration +## Example test coverage configurations + +### JavaScript example The following [`gitlab-ci.yml`](../../../ci/yaml/README.md) example uses [Mocha](https://mochajs.org/) JavaScript testing and [NYC](https://github.com/istanbuljs/nyc) coverage-tooling to @@ -74,26 +73,84 @@ test: cobertura: coverage/cobertura-coverage.xml ``` -## Enabling the feature +### Java examples + +#### Maven example -This feature comes with the `:coverage_report_view` feature flag enabled by -default. It is enabled on GitLab.com. [GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can disable it for your instance. Test coverage visualization can be enabled or disabled per-project. +The following [`gitlab-ci.yml`](../../../ci/yaml/README.md) example for Java uses [Maven](https://maven.apache.org/) +to build the project and [Jacoco](https://www.eclemma.org/jacoco/) coverage-tooling to +generate the coverage artifact. +You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image. -To enable it: +GitLab expects the artifact in the Cobertura format, so you have to execute a few +scripts before uploading it. The `test-jdk11` job tests the code and generates an +XML artifact. The `coverage-jdk-11` job converts the artifact into a Cobertura report: -```ruby -# Instance-wide -Feature.enable(:coverage_report_view) -# or by project -Feature.enable(:coverage_report_view, Project.find(<project id>)) +```yaml +test-jdk11: + stage: test + image: maven:3.6.3-jdk-11 + script: + - 'mvn $MAVEN_CLI_OPTS clean org.jacoco:jacoco-maven-plugin:prepare-agent test jacoco:report' + artifacts: + paths: + - target/site/jacoco/jacoco.xml + +coverage-jdk11: + # Must be in a stage later than test-jdk11's stage. + # The `visualize` stage does not exist by default. + # Please define it first, or chose an existing stage like `deploy`. + stage: visualize + image: haynes/jacoco2cobertura:1.0.3 + script: + # convert report from jacoco to cobertura + - 'python /opt/cover2cover.py target/site/jacoco/jacoco.xml src/main/java > target/site/cobertura.xml' + # read the <source></source> tag and prepend the path to every filename attribute + - 'python /opt/source2filename.py target/site/cobertura.xml' + needs: ["test-jdk11"] + dependencies: + - test-jdk11 + artifacts: + reports: + cobertura: target/site/cobertura.xml ``` -To disable it: +#### Gradle example -```ruby -# Instance-wide -Feature.disable(:coverage_report_view) -# or by project -Feature.disable(:coverage_report_view, Project.find(<project id>)) +The following [`gitlab-ci.yml`](../../../ci/yaml/README.md) example for Java uses [Gradle](https://gradle.org/) +to build the project and [Jacoco](https://www.eclemma.org/jacoco/) coverage-tooling to +generate the coverage artifact. +You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image. + +GitLab expects the artifact in the Cobertura format, so you have to execute a few +scripts before uploading it. The `test-jdk11` job tests the code and generates an +XML artifact. The `coverage-jdk-11` job converts the artifact into a Cobertura report: + +```yaml +test-jdk11: + stage: test + image: gradle:6.6.1-jdk11 + script: + - 'gradle test jacocoTestReport' # jacoco must be configured to create an xml report + artifacts: + paths: + - build/jacoco/jacoco.xml + +coverage-jdk11: + # Must be in a stage later than test-jdk11's stage. + # The `visualize` stage does not exist by default. + # Please define it first, or chose an existing stage like `deploy`. + stage: visualize + image: haynes/jacoco2cobertura:1.0.3 + script: + # convert report from jacoco to cobertura + - 'python /opt/cover2cover.py build/jacoco/jacoco.xml src/main/java > build/cobertura.xml' + # read the <source></source> tag and prepend the path to every filename attribute + - 'python /opt/source2filename.py build/cobertura.xml' + needs: ["test-jdk11"] + dependencies: + - test-jdk11 + artifacts: + reports: + cobertura: build/cobertura.xml ``` diff --git a/doc/user/project/milestones/burndown_charts.md b/doc/user/project/milestones/burndown_charts.md index 0c8bba831a9..913904d962b 100644 --- a/doc/user/project/milestones/burndown_charts.md +++ b/doc/user/project/milestones/burndown_charts.md @@ -14,8 +14,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w > value, so the burndown chart considers them as closed on the milestone > `start_date`. In that case, a warning will be displayed. -## Overview - Burndown Charts are visual representations of the progress of completing a milestone.  diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index 9d02a22f91e..2290a10a35f 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -7,8 +7,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Milestones -## Overview - Milestones in GitLab are a way to track issues and merge requests created to achieve a broader goal in a certain period of time. Milestones allow you to organize issues and merge requests into a cohesive group, with an optional start date and an optional due date. diff --git a/doc/user/project/new_ci_build_permissions_model.md b/doc/user/project/new_ci_build_permissions_model.md index c3825371030..a7a72ca4d82 100644 --- a/doc/user/project/new_ci_build_permissions_model.md +++ b/doc/user/project/new_ci_build_permissions_model.md @@ -223,7 +223,7 @@ This is how an example usage can look like: ```yaml test: script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker pull $CI_REGISTRY/group/other-project:latest - docker run $CI_REGISTRY/group/other-project:latest ``` @@ -236,5 +236,4 @@ to projects and their project permissions. ### API -GitLab API cannot be used via `CI_JOB_TOKEN` but there is a [proposal](https://gitlab.com/gitlab-org/gitlab/-/issues/35067) -to support it. +GitLab API can be used via `CI_JOB_TOKEN`, see [the relevant documentation](../../api/README.md#gitlab-ci-job-token). diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md index 4b4b430b663..597d6737a8f 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md @@ -25,7 +25,7 @@ This feature covers only certificates for **custom domains**, not the wildcard c Before you can enable automatic provisioning of an SSL certificate for your domain, make sure you have: -- Created a [project](../getting_started_part_two.md) in GitLab +- Created a [project](../index.md#getting-started) in GitLab containing your website's source code. - Acquired a domain (`example.com`) and added a [DNS entry](index.md) pointing it to your Pages website. diff --git a/doc/user/project/pages/getting_started_part_three.md b/doc/user/project/pages/getting_started_part_three.md index 9bc9fe97fd3..4bf5300aa13 100644 --- a/doc/user/project/pages/getting_started_part_three.md +++ b/doc/user/project/pages/getting_started_part_three.md @@ -1 +1,5 @@ +--- +redirect_to: 'custom_domains_ssl_tls_certification/index.md' +--- + This document was moved to [another location](custom_domains_ssl_tls_certification/index.md). diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index 6c3b911d033..4f389716f08 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -124,3 +124,24 @@ If you are running a self-managed instance of GitLab (GitLab Community Edition a [follow the administration steps](../../../administration/pages/index.md) to configure Pages. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch a [video tutorial](https://www.youtube.com/watch?v=dD8c7WNcc6s) about how to get started with GitLab Pages administration. + +## Security for GitLab Pages + +If your username is `foo`, your GitLab Pages website is located at `foo.gitlab.io`. +GitLab allows usernames to contain a `.`, so a user named `bar.foo` could create +a GitLab Pages website `bar.foo.gitlab.io` that effectively is a subdomain of your +`foo.gitlab.io` website. Be careful if you use JavaScript to set cookies for your website. +The safe way to manually set cookies with JavaScript is to not specify the `domain` at all: + +```javascript +// Safe: This cookie is only visible to foo.gitlab.io +document.cookie = "key=value"; + +// Unsafe: This cookie is visible to foo.gitlab.io and its subdomains, +// regardless of the presence of the leading dot. +document.cookie = "key=value;domain=.foo.gitlab.io"; +document.cookie = "key=value;domain=foo.gitlab.io"; +``` + +This issue doesn't affect users with a custom domain, or users who don't set any +cookies manually with JavaScript. diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 518cf472b50..14ddd9dad6e 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -39,7 +39,7 @@ The following quick actions are applicable to descriptions, discussions and thre | `/copy_metadata <!merge_request>` | ✓ | ✓ | | Copy labels and milestone from another merge request in the project. | | `/copy_metadata <#issue>` | ✓ | ✓ | | Copy labels and milestone from another issue in the project. | | `/create_merge_request <branch name>` | ✓ | | | Create a new merge request starting from the current issue. | -| `/done` | ✓ | ✓ | ✓ | Mark To-Do as done. | +| `/done` | ✓ | ✓ | ✓ | Mark to do as done. | | `/due <date>` | ✓ | | | Set due date. Examples of valid `<date>` include `in 2 days`, `this Friday` and `December 31st`. | | `/duplicate <#issue>` | ✓ | | | Close this issue and mark as a duplicate of another issue. **(CORE)** Also, mark both as related. **(STARTER)** | | `/epic <epic>` | ✓ | | | Add to epic `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic`, or a URL to an epic. **(PREMIUM)** | @@ -74,7 +74,7 @@ The following quick actions are applicable to descriptions, discussions and thre | `/tableflip <comment>` | ✓ | ✓ | ✓ | Append the comment with `(╯°□°)╯︵ ┻━┻`. | | `/target_branch <local branch name>` | | ✓ | | Set target branch. | | `/title <new title>` | ✓ | ✓ | ✓ | Change title. | -| `/todo` | ✓ | ✓ | ✓ | Add a To-Do. | +| `/todo` | ✓ | ✓ | ✓ | Add a to do. | | `/unassign @user1 @user2` | ✓ | ✓ | | Remove specific assignees. **(STARTER)** | | `/unassign` | ✓ | ✓ | | Remove all assignees. | | `/unlabel ~label1 ~label2` or `/remove_label ~label1 ~label2` | ✓ | ✓ | ✓ | Remove all or specific labels. | diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index 28fdda07b05..ad79fd8a8f9 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -19,7 +19,7 @@ over [`git filter-branch`](https://git-scm.com/docs/git-filter-branch) and [BFG](https://rtyley.github.io/bfg-repo-cleaner/). DANGER: **Danger:** -Rewriting repository history is a destructive operation. Make sure to backup your repository before +Rewriting repository history is a destructive operation. Make sure to back up your repository before you begin. The best way back up a repository is to [export the project](../settings/import_export.md#exporting-a-project-and-its-data). @@ -230,6 +230,7 @@ This will: - Run `git gc` against the repository to remove unreferenced objects. Repacking your repository will temporarily cause the size of your repository to increase significantly, because the old pack files are not removed until the new pack files have been created. +- Unlink any unused LFS objects currently attached to your project, freeing up storage space. - Recalculate the size of your repository on disk. You will receive an email notification with the recalculated repository size after the cleanup has completed. @@ -266,21 +267,20 @@ You can still: - Create new issues. - Clone the project. -If you exceed the repository size limit, you might try to: +If you exceed the repository size limit, you can: 1. Remove some data. 1. Make a new commit. 1. Push back to the repository. -Perhaps you might also: +If these actions are insufficient, you can also: - Move some blobs to LFS. - Remove some old dependency updates from history. -Unfortunately, this workflow won't work. Deleting files in a commit doesn't actually reduce the size -of the repository because the earlier commits and blobs still exist. - -What you need to do is rewrite history. We recommend the open-source community-maintained tool +Unfortunately, this workflow doesn't work. Deleting files in a commit doesn't actually reduce the +size of the repository, because the earlier commits and blobs still exist. Instead, you must rewrite +history. We recommend the open-source community-maintained tool [`git filter-repo`](https://github.com/newren/git-filter-repo). NOTE: **Note:** diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md index e9b07f54b91..0a1b81a6359 100644 --- a/doc/user/project/service_desk.md +++ b/doc/user/project/service_desk.md @@ -10,8 +10,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/214839) to [GitLab Starter](https://about.gitlab.com/pricing/) in 13.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/215364) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.2. -## Overview - Service Desk is a module that allows your team to connect directly with any external party through email right inside of GitLab; no external tools required. An ongoing conversation right where your software is built ensures that user feedback ends @@ -129,7 +127,7 @@ in the email, `%{ISSUE_PATH}` placeholder which will be replaced by You can customize the email display name. Emails sent from Service Desk will have this name in the `From` header. The default display name is `GitLab Support Bot`. -### Using custom email address +### Using custom email address **(CORE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2201) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.0. diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index 395d4bf30c5..9902f208468 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -270,7 +270,7 @@ To restore a project marked for deletion: Forking is a great way to [contribute to a project](../repository/forking_workflow.md) of which you're not a member. If you want to use the fork for yourself and don't need to send -[merge requests](../merge_requests.md) to the upstream project, +[merge requests](../merge_requests/index.md) to the upstream project, you can safely remove the fork relationship. CAUTION: **Caution:** diff --git a/doc/user/project/settings/project_access_tokens.md b/doc/user/project/settings/project_access_tokens.md index 57cb610a2e9..8be2ac6185f 100644 --- a/doc/user/project/settings/project_access_tokens.md +++ b/doc/user/project/settings/project_access_tokens.md @@ -48,9 +48,10 @@ API calls made with a project access token are associated with the corresponding These users will appear in **Members** but can not be modified. Furthermore, the bot user can not be added to any other project. -When the project access token is [revoked](#revoking-a-project-access-token) the bot user will be deleted and all -records will be moved to a system-wide user with the username "Ghost User". For more information, -see [Associated Records](../../profile/account/delete_account.md#associated-records). +- The username is set to `project_{project_id}_bot` for the first access token, such as `project_123_bot`. +- The username is set to `project_{project_id}_bot{bot_count}` for further access tokens, such as `project_123_bot1`. + +When the project access token is [revoked](#revoking-a-project-access-token) the bot user is then deleted and all records are moved to a system-wide user with the username "Ghost User". For more information, see [Associated Records](../../profile/account/delete_account.md#associated-records). Project bot users are a [GitLab-created service account](../../../subscriptions/self_managed/index.md#choose-the-number-of-users), but count as a licensed seat. These users will not count against your licensed seat in the future when [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/223695) is resolved. diff --git a/doc/user/project/static_site_editor/img/front_matter_ui_v13_4.png b/doc/user/project/static_site_editor/img/front_matter_ui_v13_4.png Binary files differnew file mode 100644 index 00000000000..89864858ed3 --- /dev/null +++ b/doc/user/project/static_site_editor/img/front_matter_ui_v13_4.png diff --git a/doc/user/project/static_site_editor/index.md b/doc/user/project/static_site_editor/index.md index ce14cefba92..4984152d4eb 100644 --- a/doc/user/project/static_site_editor/index.md +++ b/doc/user/project/static_site_editor/index.md @@ -14,6 +14,7 @@ description: "The static site editor enables users to edit content on static web > - Markdown front matter hidden on the WYSIWYG editor [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216834) in GitLab 13.1. > - Support for `*.md.erb` files [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223171) in GitLab 13.2. > - Non-Markdown content blocks uneditable on the WYSIWYG mode [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216836) in GitLab 13.3. +> - Ability to edit page front matter [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235921) in GitLab 13.4. DANGER: **Danger:** In GitLab 13.0, we [introduced breaking changes](https://gitlab.com/gitlab-org/gitlab/-/issues/213282) @@ -60,6 +61,11 @@ click of a button:  +You can also edit the page's front matter both in WYSIWYG mode via the side-drawer and in Markdown +mode. + + + When an editor submits their changes, in the background, GitLab automatically creates a new branch, commits their changes, and opens a merge request. The editor lands directly on the merge request, and then they can assign it to @@ -90,6 +96,12 @@ From GitLab 13.1 onwards, the YAML front matter of Markdown files is hidden on t WYSIWYG editor to avoid unintended changes. To edit it, use the Markdown editing mode, the regular GitLab file editor, or the Web IDE. +NOTE: **Note:** +A new configuration file for the Static Site Editor was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/4267) +in GitLab 13.4. Beginning in 13.5, the `.gitlab/static-site-editor.yml` file will store additional +configuration options for the editor. When the functionality of the existing `data/config.yml` file +is replicated in the new configuration file, `data/config.yml` will be formally deprecated. + ### Use the Static Site Editor to edit your content For instance, suppose you are a recently hired technical writer at a large diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 821b42af049..4da9b5f88ff 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -32,7 +32,7 @@ file path fragments to start seeing results. ## Syntax highlighting As expected from an IDE, syntax highlighting for many languages within -the Web IDE will make your direct editing even easier. +the Web IDE makes your direct editing even easier. The Web IDE currently provides: @@ -79,7 +79,7 @@ which apply to the entire Web IDE screen. > - Support for validation based on custom schemas [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/226982) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. The Web IDE provides validation support for certain JSON and YAML files using schemas -based on the [JSON Schema Store](https://www.schemastore.org/json/). +based on the [JSON Schema Store](https://www.schemastore.org/json/). ### Predefined schemas @@ -143,7 +143,7 @@ The Web IDE supports configuration of certain editor settings by using [`.editorconfig` files](https://editorconfig.org/). When opening a file, the Web IDE looks for a file named `.editorconfig` in the current directory and all parent directories. If a configuration file is found and has settings -that match the file's path, these settings will be enforced on the opened file. +that match the file's path, these settings are enforced on the opened file. The Web IDE currently supports the following `.editorconfig` settings: @@ -166,7 +166,7 @@ review the list of changed files. Once you have finalized your changes, you can add a commit message, commit the changes and directly create a merge request. In case you don't have write -access to the selected branch, you will see a warning, but still be able to create +access to the selected branch, you see a warning, but can still create a new branch and start a merge request. To discard a change in a particular file, click the **Discard changes** button on that @@ -201,8 +201,7 @@ left. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19318) in [GitLab Core](https://about.gitlab.com/pricing/) 11.0. To switch between your authored and assigned merge requests, click the -dropdown in the top of the sidebar to open a list of merge requests. You will -need to commit or discard all your changes before switching to a different merge +dropdown in the top of the sidebar to open a list of merge requests. You need to commit or discard all your changes before switching to a different merge request. ## Switching branches @@ -211,7 +210,7 @@ request. To switch between branches of the current project repository, click the dropdown in the top of the sidebar to open a list of branches. -You will need to commit or discard all your changes before switching to a +You need to commit or discard all your changes before switching to a different branch. ## Markdown editing @@ -226,7 +225,7 @@ supports [GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-g You can also upload any local images by pasting them directly in the Markdown file. The image is uploaded to the same directory and is named `image.png` by default. If another file already exists with the same name, a numeric suffix is automatically -added to the file name. +added to the filename. ## Live Preview @@ -249,7 +248,7 @@ The Live Preview feature needs to be enabled in the GitLab instances admin settings. Live Preview is enabled for all projects on GitLab.com - + Once you have done that, you can preview projects with a `package.json` file and a `main` entry point inside the Web IDE. An example `package.json` is shown @@ -292,7 +291,7 @@ to work: [enabled](../../../administration/integration/terminal.md#enabling-and-disabling-terminal-support). **(ULTIMATE ONLY)** If you have the terminal open and the job has finished with its tasks, the -terminal will block the job from finishing for the duration configured in +terminal blocks the job from finishing for the duration configured in [`[session_server].session_timeout`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section) until you close the terminal window. @@ -308,15 +307,15 @@ In order to enable the Web IDE terminals you need to create the file file is fairly similar to the [CI configuration file](../../../ci/yaml/README.md) syntax but with some restrictions: -- No global blocks can be defined (ie: `before_script` or `after_script`) +- No global blocks can be defined (i.e., `before_script` or `after_script`) - Only one job named `terminal` can be added to this file. - Only the keywords `image`, `services`, `tags`, `before_script`, `script`, and `variables` are allowed to be used to configure the job. - To connect to the interactive terminal, the `terminal` job must be still alive - and running, otherwise the terminal won't be able to connect to the job's session. + and running, otherwise the terminal cannot connect to the job's session. By default the `script` keyword has the value `sleep 60` to prevent the job from ending and giving the Web IDE enough time to connect. This means - that, if you override the default `script` value, you'll have to add a command + that, if you override the default `script` value, you have to add a command which would keep the job running, like `sleep`. In the code below there is an example of this configuration file: @@ -333,40 +332,39 @@ terminal: NODE_ENV: "test" ``` -Once the terminal has started, the console will be displayed and we could access +Once the terminal has started, the console is displayed and we could access the project repository files. **Important**. The terminal job is branch dependent. This means that the -configuration file used to trigger and configure the terminal will be the one in +configuration file used to trigger and configure the terminal is the one in the selected branch of the Web IDE. -If there is no configuration file in a branch, an error message will be shown. +If there is no configuration file in a branch, an error message is shown. ### Running interactive terminals in the Web IDE -If Interactive Terminals are available for the current user, the **Terminal** button -will be visible in the right sidebar of the Web IDE. Click this button to open +If Interactive Terminals are available for the current user, the **Terminal** button is visible in the right sidebar of the Web IDE. Click this button to open or close the terminal tab. -Once open, the tab will show the **Start Web Terminal** button. This button may +Once open, the tab shows the **Start Web Terminal** button. This button may be disabled if the environment is not configured correctly. If so, a status -message will describe the issue. Here are some reasons why **Start Web Terminal** +message describes the issue. Here are some reasons why **Start Web Terminal** may be disabled: - `.gitlab/.gitlab-webide.yml` does not exist or is set up incorrectly. - No active private runners are available for the project. -If active, clicking the **Start Web Terminal** button will load the terminal view +If active, clicking the **Start Web Terminal** button loads the terminal view and start connecting to the runner's terminal. At any time, the **Terminal** tab -can be closed and reopened and the state of the terminal will not be affected. +can be closed and reopened and the state of the terminal is not affected. When the terminal is started and is successfully connected to the runner, then the -runner's shell prompt will appear in the terminal. From here, you can enter -commands that will be executed within the runner's environment. This is similar +runner's shell prompt appears in the terminal. From here, you can enter +commands executed within the runner's environment. This is similar to running commands in a local terminal or through SSH. While the terminal is running, it can be stopped by clicking **Stop Terminal**. -This will disconnect the terminal and stop the runner's terminal job. From here, +This disconnects the terminal and stops the runner's terminal job. From here, click **Restart Terminal** to start a new terminal session. ### File syncing to web terminal @@ -408,14 +406,14 @@ terminal: more information. - `$CI_PROJECT_DIR` is a [predefined environment variable](../../../ci/variables/predefined_variables.md) - for GitLab Runner. This is where your project's repository will be. + for GitLab Runners. This is where your project's repository resides. Once you have configured the web terminal for file syncing, then when the web -terminal is started, a **Terminal** status will be visible in the status bar. +terminal is started, a **Terminal** status is visible in the status bar.  -Changes made to your files via the Web IDE will sync to the running terminal +Changes made to your files via the Web IDE sync to the running terminal when: - <kbd>Ctrl</kbd> + <kbd>S</kbd> (or <kbd>Cmd</kbd> + <kbd>S</kbd> on Mac) @@ -425,9 +423,12 @@ when: ### Limitations -Interactive Terminals is in a beta phase and will continue to be improved upon in upcoming -releases. In the meantime, please note that the user is limited to having only one -active terminal at a time. +The Web IDE has a few limitations: + +- Interactive Terminals is in a beta phase and continues to be improved in upcoming releases. In the meantime, please note that the user is limited to having only one + active terminal at a time. + +- LFS files can be rendered and displayed but they cannot be updated and committed using the Web IDE. If an LFS file is modified and pushed to the repository, the LFS pointer in the repository will be overwritten with the modified LFS file content. ### Troubleshooting diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index 40ef5e216fd..86d7cfa5d16 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -163,48 +163,13 @@ Similar to versioned diff file views, you can see the changes made in a given Wi > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14902) in **GitLab 12.10.** > - Git events were [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216014) in **GitLab 13.0.** -> - It's enabled on GitLab.com. -> - Git access activity creation is managed by a feature flag. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-wiki-events-in-git). **(CORE ONLY)** +> - [Feature flag for Git events was removed](https://gitlab.com/gitlab-org/gitlab/-/issues/258665) in **GitLab 13.5** Wiki events (creation, deletion, and updates) are tracked by GitLab and displayed on the [user profile](../../profile/index.md#user-profile), [group](../../group/index.md#view-group-activity), and [project](../index.md#project-activity) activity pages. -### Enable or disable Wiki events in Git **(CORE ONLY)** - -Tracking wiki events through Git is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can enable it for your instance. - -To enable it: - -```ruby -Feature.enable(:wiki_events_on_git_push) -``` - -To enable for just a particular project: - -```ruby -project = Project.find_by_full_path('your-group/your-project') -Feature.enable(:wiki_events_on_git_push, project) -``` - -To disable it: - -```ruby -Feature.disable(:wiki_events_on_git_push) -``` - -To disable for just a particular project: - -```ruby -project = Project.find_by_full_path('your-group/your-project') -Feature.disable(:wiki_events_on_git_push, project) -``` - ## Adding and editing wiki pages locally Since wikis are based on Git repositories, you can clone them locally and edit diff --git a/doc/user/search/advanced_global_search.md b/doc/user/search/advanced_global_search.md index 53ec8b35631..3152229c39f 100644 --- a/doc/user/search/advanced_global_search.md +++ b/doc/user/search/advanced_global_search.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Editor +stage: Enablement +group: Global Search info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" type: reference --- diff --git a/doc/user/search/advanced_search_syntax.md b/doc/user/search/advanced_search_syntax.md index 804d4c540ac..5817852dd97 100644 --- a/doc/user/search/advanced_search_syntax.md +++ b/doc/user/search/advanced_search_syntax.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Editor +stage: Enablement +group: Global Search info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" type: reference --- @@ -62,6 +62,7 @@ The Advanced Search Syntax also supports the use of filters. The available filte - filename: Filters by filename. You can use the glob (`*`) operator for fuzzy matching. - path: Filters by path. You can use the glob (`*`) operator for fuzzy matching. - extension: Filters by extension in the filename. Please write the extension without a leading dot. Exact match only. +- blob: Filters by Git `object-id`. Exact match only. To use them, simply add them to your query in the format `<filter_name>:<value>` without any spaces between the colon (`:`) and the value. @@ -83,6 +84,7 @@ Filters can be inversed to **filter out** results from the result set, by prefix - `-filename` - `-path` - `-extension` +- `-blob` Examples: diff --git a/doc/user/search/img/basic_search.png b/doc/user/search/img/basic_search.png Binary files differnew file mode 100644 index 00000000000..234805d5a4f --- /dev/null +++ b/doc/user/search/img/basic_search.png diff --git a/doc/user/search/img/basic_search_results.png b/doc/user/search/img/basic_search_results.png Binary files differnew file mode 100644 index 00000000000..52aa2e3fe6c --- /dev/null +++ b/doc/user/search/img/basic_search_results.png diff --git a/doc/user/search/index.md b/doc/user/search/index.md index 475a72385ac..57981205691 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -129,14 +129,6 @@ characters to begin your search. For example, if you want to search for issues that have the assignee "Simone Presley", you'll need to type at least "Sim" before autocomplete gives any relevant results. -## Code search - -To search through code or other documents in a single project, you can use -the search field on the top-right of your screen while the project page is open. - - - - ## Search history You can view recent searches by clicking on the little arrow-clock icon, which is to the left of the search input. Click the search entry to run that search again. This feature is available for issues and merge requests. Searches are stored locally in your browser. @@ -155,24 +147,6 @@ Some filters can be added multiple times. These include but are not limited to a  -## Shortcut - -You'll also find a shortcut on the search field on the top-right of the project's dashboard to -quickly access issues and merge requests created or assigned to you within that project: - - - -### Autocomplete suggestions - -You can also type in this search bar to see autocomplete suggestions for: - -- Projects and groups -- Various help pages (try and type **API help**) -- Project feature pages (try and type **milestones**) -- Various settings pages (try and type **user settings**) -- Recently viewed issues (try and type some word from the title of a recently viewed issue) -- Recently viewed merge requests (try and type some word from the title of a recently merge request) - ## To-Do List Your [To-Do List](../todos.md#gitlab-to-do-list) can be searched by "to do" and "done". @@ -219,6 +193,59 @@ and **Labels**, select multiple issues to add to a list of your choice:  +## Shortcut + +You'll find a shortcut on the search field on the top-right of the project's dashboard to +quickly access issues and merge requests created or assigned to you within that project: + + + +### Autocomplete suggestions + +You can also type in this search bar to see autocomplete suggestions for: + +- Projects and groups +- Various help pages (try and type **API help**) +- Project feature pages (try and type **milestones**) +- Various settings pages (try and type **user settings**) +- Recently viewed issues (try and type some word from the title of a recently viewed issue) +- Recently viewed merge requests (try and type some word from the title of a recently merge request) + +## Basic search + +The Basic search in GitLab is a global search service that allows you to search +across the entire GitLab instance, within a group, or a single project. Basic search is +backed by the database and allows searching in: + +- Projects +- Issues +- Merge requests +- Milestones +- Users +- Code (Project only) +- Comments (Project only) +- Commits (Project only) +- Wiki (Project only) + +To start a search, type into the search bar on the top-right of the screen. You can always search +in all GitLab and may also see the options to search within a group or project if you are in the +group or project dashboard. + + + +Once the results are returned, you can modify the search, select a different type of data to +search, or choose a specific group or project. + + + +### Code search + +To search through code or other documents in a single project, you can use +the search field on the top-right of your screen while the project page is open. + + + + ## Advanced Search **(STARTER)** Leverage Elasticsearch for faster, more advanced code search across your entire diff --git a/doc/user/snippets.md b/doc/user/snippets.md index 15391f034a8..6335460ef96 100644 --- a/doc/user/snippets.md +++ b/doc/user/snippets.md @@ -131,8 +131,8 @@ There are two main ways of how you can discover snippets in GitLab. For exploring all snippets that are visible to you, you can go to the Snippets dashboard of your GitLab instance via the top navigation. For GitLab.com you can -find it [here](https://gitlab.com/dashboard/snippets). This navigates you to an -overview that shows snippets you created and allows you to explore all snippets. +navigate to an [overview]((https://gitlab.com/dashboard/snippets)) that shows snippets +you created and allows you to explore all snippets. If you want to discover snippets that belong to a specific project, you can navigate to the Snippets page via the left side navigation on the project page. diff --git a/doc/user/todos.md b/doc/user/todos.md index 1fca3c0ab64..46f589c1b11 100644 --- a/doc/user/todos.md +++ b/doc/user/todos.md @@ -15,7 +15,7 @@ spend your time. This can include taking an action, or keeping track of things do your work, being able to get started quickly is important. Your *To-Do List* offers a chronological list of items waiting for your input -(known as *to-dos*) in a single dashboard. +(known as *to do items*) in a single dashboard. The To-Do List supports tracking [actions](#what-triggers-a-to-do) related to the following: @@ -26,18 +26,18 @@ the following:  -You can access your To-Do List by clicking the **{task-done}** To-Do List icon -next to the search bar in the top navigation. If the to-do item count is: +You can access your To-Do List by clicking the To-Do List icon (**{task-done}**) +next to the search bar in the top navigation. If the to do item count is: -- *Less than 100*, the number in blue is the number of to-do items. +- *Less than 100*, the number in blue is the number of to do items. - *100 or more*, the number displays as 99+. The exact number displays in the To-Do List.  -## What triggers a to-do +## What triggers a to do -A to-do item appears on your To-Do List when: +A to do item appears on your To-Do List when: - An issue or merge request is assigned to you. - You're `@mentioned` in the description or comment of an issue or merge request @@ -57,23 +57,23 @@ A to-do item appears on your To-Do List when: and you're the user that added it. **(PREMIUM)** When several trigger actions occur for the same user on the same object (for -example, an issue), GitLab displays only the first action as a single to-do +example, an issue), GitLab displays only the first action as a single to do item. -To-do triggers aren't affected by [GitLab notification email settings](profile/notifications.md). +To do item triggers aren't affected by [GitLab notification email settings](profile/notifications.md). NOTE: **Note:** -When a user no longer has access to a resource related to a to-do (such as an -issue, merge request, project, or group), for security reasons GitLab deletes -any related to-do items within the next hour. Deletion is delayed to prevent -data loss, in the case where a user's access is accidentally revoked. +When a user no longer has access to a resource related to a to do item (such as +an issue, merge request, project, or group), for security reasons GitLab +deletes any related to do items within the next hour. Deletion is delayed to +prevent data loss, in the case where a user's access is accidentally revoked. -### Directly addressing a to-do +### Directly addressing a to do > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7926) in GitLab 9.0. -If you're mentioned at the start of a line, the to-do you receive will be listed -as *directly addressed*. For example, in the following comment: +If you're mentioned at the start of a line, the to do item you receive will be +listed as *directly addressed*. For example, in the following comment: ```markdown @alice What do you think? cc: @bob @@ -87,23 +87,23 @@ as *directly addressed*. For example, in the following comment: @erin @frank thank you! ``` -The people receiving directly addressed to-do items are `@alice`, `@erin`, and -`@frank`. Directly addressed to-do items only differ from mentions in their type +The people receiving directly addressed to do items are `@alice`, `@erin`, and +`@frank`. Directly addressed to do items only differ from mentions in their type for filtering purposes; otherwise, they appear as normal. -### Manually creating a to-do +### Manually creating a to do You can add an issue or merge request (or epic **(ULTIMATE)**) to your To-Do List by clicking its **Add a To Do** button.  -## Marking a to-do as done +## Marking a to do as done Any action to an issue or merge request (or epic **(ULTIMATE)**) will mark its -corresponding to-do as done. +corresponding to do item as done. -Actions that dismiss to-do items include: +Actions that dismiss to do items include: - Changing the assignee - Changing the milestone @@ -111,27 +111,28 @@ Actions that dismiss to-do items include: - Commenting on the issue Your To-Do List is personal, and items are only marked as done if you take -action. If you close the issue or merge request, your to-do is marked as done. +action. If you close the issue or merge request, your to do item is marked as +done. To prevent other users from closing issues without you being notified, if someone else closes, merges, or takes action on an issue or merge request (or -epic **(ULTIMATE)**), your to-do will remain pending. +epic **(ULTIMATE)**), your to do item remains pending. -There's just one to-do for each of these, so mentioning a user many times in an -issue will only trigger one to-do item. +There's just one to do item for each of these, so mentioning a user many times +in an issue only triggers one to do item. -If no action is needed, you can manually mark the to-do as done by clicking its -corresponding **Done** button to have GitLab remove the item from your -To-Do List. +If no action is needed, you can manually mark the to do item as done by +clicking its corresponding **Done** button to have GitLab remove the item from +your To-Do List. - + -You can also mark a to-do as done by clicking the **Mark as done** button in the -sidebar of an issue or merge request (or epic **(ULTIMATE)**). +You can also mark a to do item as done by clicking the **Mark as done** button +in the sidebar of an issue or merge request (or epic **(ULTIMATE)**).  -You can mark all your to-do items as done at once by clicking the +You can mark all your to do items as done at once by clicking the **Mark all as done** button. ## Filtering your To-Do List @@ -142,9 +143,9 @@ You can use the following types of filters with your To-Do List: | ------- | ---------------------------------------------------------------- | | Project | Filter by project. | | Group | Filter by group. | -| Author | Filter by the author that triggered the To Do. | +| Author | Filter by the author that triggered the to do. | | Type | Filter by issue, merge request, design, or epic. **(ULTIMATE)** | -| Action | Filter by the action that triggered the To Do. | +| Action | Filter by the action that triggered the to do. | You can also filter by more than one of these at the same time. The previously described [triggering actions](#what-triggers-a-to-do) include: |
