diff options
Diffstat (limited to 'doc')
21 files changed, 489 insertions, 247 deletions
diff --git a/doc/.vale/gitlab/Normal.yml b/doc/.vale/gitlab/Normal.yml new file mode 100644 index 00000000000..0ced9f3318e --- /dev/null +++ b/doc/.vale/gitlab/Normal.yml @@ -0,0 +1,14 @@ +--- +# Warning: gitlab.Normal +# +# Suggests alternatives for 'normal'. +# +# For a list of all options, see https://vale.sh/docs/topics/styles/ +extends: substitution +message: "Use '%s' instead of '%s'." +link: https://docs.gitlab.com/ee/development/documentation/styleguide/word_list.html +level: warning +ignorecase: true +swap: + normally: "usually' or 'typically" + normal: "typical' or 'standard" diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 2fa683ffbe1..27a69848655 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -509,15 +509,19 @@ If initially your LDAP configuration looked like: ## Updating LDAP DN and email -When an LDAP user is created in GitLab, their LDAP distinguished name (DN) is linked to their GitLab account as an identifier. +When an LDAP server creates a user in GitLab, the user's LDAP distinguished name (DN) is linked to their GitLab account +as an identifier. When a user tries to sign in with LDAP, GitLab tries to find the user using the DN saved on that user's account. -- If GitLab finds the user by the DN, and the user's email matches the GitLab account's email, GitLab does not take any further action. -- If GitLab finds the user by the DN and the user's email has changed, GitLab updates its record of the user's email to match the one in LDAP. -- If GitLab cannot find a user by their DN, it tries to find the user by their email. If GitLab finds the user by their email, GitLab updates the DN stored in the user's GitLab account. Both values now match the information stored in LDAP. - -If both the DN **and** the email address have changed, see the [user DN and email have changed](ldap-troubleshooting.md#user-dn-and-email-have-changed) section of our documentation. +- If GitLab finds the user by the DN and the user's email address: + - Matches the GitLab account's email address, GitLab does not take any further action. + - Has changed, GitLab updates its record of the user's email to match the one in LDAP. +- If GitLab cannot find a user by their DN, it tries to find the user by their email. If GitLab: + - Finds the user by their email, GitLab updates the DN stored in the user's GitLab account. Both values now + match the information stored in LDAP. + - Cannot find the user by their email address (both the DN **and** the email address have changed), see + [User DN and email have changed](ldap-troubleshooting.md#user-dn-and-email-have-changed). ## Disable anonymous LDAP authentication diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 3b7ca2f635d..870578898f9 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -626,14 +626,14 @@ does not do this: ### User DN and email have changed -If both the primary email **and** the DN change in LDAP, GitLab has -no way of identifying the correct LDAP record of a user and, as a -result, blocks that user. To fix this, update the user's existing -GitLab profile with at least one of the new primary email or DN values -so GitLab can find the LDAP record. - -The following script updates the emails for all provided users so they -aren't blocked or unable to access their accounts. +If both the primary email **and** the DN change in LDAP, GitLab cannot identify the correct LDAP record of a user. As a +result, GitLab blocks that user. So that GitLab can find the LDAP record, update the user's existing GitLab profile with +at least either: + +- The new primary email. +- DN values. + +The following script updates the emails for all provided users so they aren't blocked or unable to access their accounts. NOTE: The following script requires that any new accounts with the new diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md index a4d027101dd..03b09e00f1c 100644 --- a/doc/administration/raketasks/geo.md +++ b/doc/administration/raketasks/geo.md @@ -4,7 +4,7 @@ group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Geo Rake Tasks **(PREMIUM SELF)** +# Geo Rake tasks **(PREMIUM SELF)** The following Rake tasks are for [Geo installations](../geo/index.md). See also [troubleshooting Geo](../geo/replication/troubleshooting.md) for additional Geo Rake tasks. diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 224ed63d3e6..61f6137e1ed 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitHub import **(FREE SELF)** +# GitHub import Rake task **(FREE SELF)** To retrieve and import GitHub repositories, you need a [GitHub personal access token](https://github.com/settings/tokens). A username should be passed as the second argument to the Rake task, diff --git a/doc/administration/sidekiq/sidekiq_job_migration.md b/doc/administration/sidekiq/sidekiq_job_migration.md index d79c13c8f29..b93d86d4c86 100644 --- a/doc/administration/sidekiq/sidekiq_job_migration.md +++ b/doc/administration/sidekiq/sidekiq_job_migration.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Sidekiq job migration **(FREE SELF)** +# Sidekiq job migration Rake tasks **(FREE SELF)** WARNING: This operation should be very uncommon. We do not recommend it for the vast majority of GitLab instances. diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 0d04223d96e..e78167b6db1 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -4332,6 +4332,25 @@ Input type: `ProjectCiCdSettingsUpdateInput` | <a id="mutationprojectcicdsettingsupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationprojectcicdsettingsupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +### `Mutation.projectInitializeProductAnalytics` + +Input type: `ProjectInitializeProductAnalyticsInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationprojectinitializeproductanalyticsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationprojectinitializeproductanalyticsprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project to initialize. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationprojectinitializeproductanalyticsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationprojectinitializeproductanalyticserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationprojectinitializeproductanalyticsproject"></a>`project` | [`Project`](#project) | Project on which the initialization took place. | + ### `Mutation.projectSetComplianceFramework` Assign (or unset) a compliance framework to a project. @@ -13957,6 +13976,7 @@ Returns [`[VulnerableProjectsByGrade!]!`](#vulnerableprojectsbygrade). | Name | Type | Description | | ---- | ---- | ----------- | | <a id="groupvulnerabilitygradesincludesubgroups"></a>`includeSubgroups` | [`Boolean`](#boolean) | Include grades belonging to subgroups. | +| <a id="groupvulnerabilitygradeslettergrade"></a>`letterGrade` | [`VulnerabilityGrade`](#vulnerabilitygrade) | Filter the response by given letter grade. | ##### `Group.vulnerabilitySeveritiesCount` @@ -14184,7 +14204,6 @@ A block of time for which a participant is on-call. | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="instancesecuritydashboardvulnerabilitygrades"></a>`vulnerabilityGrades` | [`[VulnerableProjectsByGrade!]!`](#vulnerableprojectsbygrade) | Represents vulnerable project counts for each grade. | | <a id="instancesecuritydashboardvulnerabilityscanners"></a>`vulnerabilityScanners` | [`VulnerabilityScannerConnection`](#vulnerabilityscannerconnection) | Vulnerability scanners reported on the vulnerabilities from projects selected in Instance Security Dashboard. (see [Connections](#connections)) | #### Fields with arguments @@ -14221,6 +14240,18 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="instancesecuritydashboardprojectssearch"></a>`search` | [`String`](#string) | Search query, which can be for the project name, a path, or a description. | +##### `InstanceSecurityDashboard.vulnerabilityGrades` + +Represents vulnerable project counts for each grade. + +Returns [`[VulnerableProjectsByGrade!]!`](#vulnerableprojectsbygrade). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="instancesecuritydashboardvulnerabilitygradeslettergrade"></a>`letterGrade` | [`VulnerabilityGrade`](#vulnerabilitygrade) | Filter the response by given letter grade. | + ##### `InstanceSecurityDashboard.vulnerabilitySeveritiesCount` Counts for each vulnerability severity from projects selected in Instance Security Dashboard. diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index 1f58d5252e1..09240e63617 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -790,6 +790,24 @@ When the variable is **optional**: - You can set the variable. +## normal, normally + +Don't use **normal** to mean the usual, typical, or standard way of doing something. +Use those terms instead. + +Use: + +- Typically, you specify a certificate. +- Usually, you specify a certificate. +- Follow the standard Git workflow. + +Instead of: + +- Normally, you specify a certificate. +- Follow the normal Git workflow. + +([Vale](../testing.md#vale) rule: [`Normal.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Normal.yml)) + ## note that Do not use **note that** because it's wordy. diff --git a/doc/development/geo/proxying.md b/doc/development/geo/proxying.md index 67d4129a9d0..f3136890788 100644 --- a/doc/development/geo/proxying.md +++ b/doc/development/geo/proxying.md @@ -260,12 +260,7 @@ S-->>C: return Git response from primary ## Git push -### Unified URLs - -With unified URLs, a push will redirect to a local path formatted as `/-/push_from_secondary/$SECONDARY_ID/*`. Further -requests through this path will be proxied to the primary, which will handle the push. - -#### Git push over SSH +### Git push over SSH As SSH operations go through GitLab Shell instead of Workhorse, they are not proxied through the mechanism used for Workhorse requests. With SSH operations, they are proxied as Git HTTP requests to the primary site by the secondary @@ -293,7 +288,12 @@ I-->>S: <response> S-->>C: return Git response from primary ``` -#### Git push over HTTP(s) +### Git push over HTTP(S) + +#### Git push over HTTP(S) unified URLs + +With unified URLs, a push redirects to a local path formatted as `/-/push_from_secondary/$SECONDARY_ID/*`. Further +requests through this path are proxied to the primary, which will handle the push. ```mermaid sequenceDiagram @@ -326,7 +326,7 @@ W-->>Wsec: Pipe messages to the Git client Wsec-->>C: Return piped messages from Git ``` -### Git push over HTTP with Separate URLs +#### Git push over HTTP(S) with separate URLs With separate URLs, the secondary will redirect to a URL formatted like `$PRIMARY/-/push_from_secondary/$SECONDARY_ID/*`. diff --git a/doc/integration/advanced_search/elasticsearch.md b/doc/integration/advanced_search/elasticsearch.md index 741dccc31fe..9c9700412ce 100644 --- a/doc/integration/advanced_search/elasticsearch.md +++ b/doc/integration/advanced_search/elasticsearch.md @@ -46,13 +46,7 @@ If you are using a compatible version and after connecting to OpenSearch, you ge Elasticsearch requires additional resources to those documented in the [GitLab system requirements](../../install/requirements.md). -Memory, CPU, and storage resource amounts vary depending on the amount of data you index into the Elasticsearch cluster. Heavily used Elasticsearch clusters may require more resources. According to -[Elasticsearch official guidelines](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_memory), -each node should have: - -- [Memory](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_memory): 8 GiB (minimum). -- [CPU](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_cpus): Modern processor with multiple cores. GitLab has minimal CPU requirements for Elasticsearch. Multiple cores provide extra concurrency, which is more beneficial than faster CPUs. -- [Storage](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_disks): Use SSD storage. The total storage size of all Elasticsearch nodes is about 50% of the total size of your Git repositories. It includes one primary and one replica. The [`estimate_cluster_size`](#gitlab-advanced-search-rake-tasks) Rake task ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221177) in GitLab 13.10) uses total repository size to estimate the Advanced Search storage requirements. +Memory, CPU, and storage resource amounts vary depending on the amount of data you index into the Elasticsearch cluster. Heavily used Elasticsearch clusters may require more resources. The [`estimate_cluster_size`](#gitlab-advanced-search-rake-tasks) Rake task ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221177) in GitLab 13.10) uses the total repository size to estimate the Advanced Search storage requirements. ## Install Elasticsearch diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index a67fec26a9b..cd1a5b05622 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Clean up **(FREE SELF)** +# Clean up Rake tasks **(FREE SELF)** GitLab provides Rake tasks for cleaning up GitLab instances. diff --git a/doc/raketasks/generate_sample_prometheus_data.md b/doc/raketasks/generate_sample_prometheus_data.md index ec7f54a41c2..7445065c77e 100644 --- a/doc/raketasks/generate_sample_prometheus_data.md +++ b/doc/raketasks/generate_sample_prometheus_data.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Generate sample Prometheus data **(FREE SELF)** +# Generate sample Prometheus data Rake task **(FREE SELF)** This command runs Prometheus queries for each of the metrics of a specific environment for a series of time intervals to now: diff --git a/doc/raketasks/index.md b/doc/raketasks/index.md index 6d1a8d32adc..0d5a826b0a1 100644 --- a/doc/raketasks/index.md +++ b/doc/raketasks/index.md @@ -7,15 +7,13 @@ comments: false # Rake tasks **(FREE SELF)** -GitLab provides [Rake](https://ruby.github.io/rake/) tasks to assist you with -common administration and operational processes. +GitLab provides [Rake](https://ruby.github.io/rake/) tasks to assist you with common administration and operational +processes. You can perform GitLab Rake tasks by using: -- `gitlab-rake <raketask>` for [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html) - installations. -- `bundle exec rake <raketask>` for [source](../install/installation.md) - installations. +- `gitlab-rake <raketask>` for [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html) installations. +- `bundle exec rake <raketask>` for [source](../install/installation.md) installations. ## Available Rake tasks diff --git a/doc/raketasks/list_repos.md b/doc/raketasks/list_repos.md index 57d24a2942d..258d41aa190 100644 --- a/doc/raketasks/list_repos.md +++ b/doc/raketasks/list_repos.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Listing repository directories **(FREE SELF)** +# List repository directories Rake task **(FREE SELF)** You can print a list of all Git repositories on disk managed by GitLab. diff --git a/doc/raketasks/migrate_snippets.md b/doc/raketasks/migrate_snippets.md index 62d2866e499..e51edc5c133 100644 --- a/doc/raketasks/migrate_snippets.md +++ b/doc/raketasks/migrate_snippets.md @@ -4,7 +4,7 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Migration to versioned snippets **(FREE SELF)** +# Migrate to versioned snippets Rake tasks **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215861) in GitLab 13.0. diff --git a/doc/raketasks/spdx.md b/doc/raketasks/spdx.md index 31f860f45de..608139fa404 100644 --- a/doc/raketasks/spdx.md +++ b/doc/raketasks/spdx.md @@ -4,7 +4,7 @@ group: Composition Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# SPDX license list import **(ULTIMATE SELF)** +# SPDX license list import Rake task **(ULTIMATE SELF)** GitLab provides a Rake task for uploading a fresh copy of the [SPDX license list](https://spdx.org/licenses/) to a GitLab instance. This list is needed for matching the names of [License Compliance policies](../user/compliance/license_compliance/index.md). diff --git a/doc/raketasks/user_management.md b/doc/raketasks/user_management.md index 84d943e2c21..7756774e432 100644 --- a/doc/raketasks/user_management.md +++ b/doc/raketasks/user_management.md @@ -4,9 +4,10 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# User management **(FREE SELF)** +# User management Rake tasks **(FREE SELF)** -GitLab provides Rake tasks for user management. +GitLab provides Rake tasks for managing users. Administrators can also use the Admin Area to +[manage users](../user/admin_area/index.md#administering-users). ## Add user as a developer to all projects diff --git a/doc/raketasks/web_hooks.md b/doc/raketasks/web_hooks.md index b4d01d21dc7..3bd9d7e2d2e 100644 --- a/doc/raketasks/web_hooks.md +++ b/doc/raketasks/web_hooks.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Webhooks administration **(FREE SELF)** +# Webhooks administration Rake tasks **(FREE SELF)** GitLab provides Rake tasks for webhooks management. diff --git a/doc/raketasks/x509_signatures.md b/doc/raketasks/x509_signatures.md index 3404f6ae9cf..364264ae204 100644 --- a/doc/raketasks/x509_signatures.md +++ b/doc/raketasks/x509_signatures.md @@ -4,7 +4,7 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# X.509 signatures **(FREE SELF)** +# X.509 signatures Rake task **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/122159) in GitLab 12.10. diff --git a/doc/update/index.md b/doc/update/index.md index dd743daf13e..176cec159c3 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -499,6 +499,12 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap sidekiq['routing_rules'] = [['*', 'default']] ``` +- The structure of `/etc/gitlab/gitlab-secrets.json` was modified in [GitLab 15.4](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6310), + and new configuration was added to `gitlab_pages`, `grafana`, and `mattermost` sections. + In a highly available or GitLab Geo environment, secrets need to be the same on all nodes. + If you're manually syncing the secrets file across nodes, or manually specifying secrets in + `/etc/gitlab/gitlab.rb`, make sure `/etc/gitlab/gitlab-secrets.json` is the same on all nodes. + ### 15.3.3 - In GitLab 15.3.3, [SAML Group Links](../api/groups.md#saml-group-links) API `access_level` attribute type changed to `integer`. See diff --git a/doc/user/application_security/sast/customize_rulesets.md b/doc/user/application_security/sast/customize_rulesets.md index a0742eb79a7..c6915eb0d93 100644 --- a/doc/user/application_security/sast/customize_rulesets.md +++ b/doc/user/application_security/sast/customize_rulesets.md @@ -11,251 +11,423 @@ info: To determine the technical writer assigned to the Stage/Group associated w > passthrough chains. Expanded to include additional passthrough types of `file`, `git`, and `url` in GitLab 14.6. > - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) support for overriding rules in GitLab 14.8. -You can customize the default scanning rules provided by our SAST analyzers. -Ruleset customization supports the following that can be used -simultaneously: +You can customize the behavior of our SAST analyzers by [defining a ruleset configuration file](#create-the-configuration-file) in the +repository being scanned. There are two kinds of customization: -- [Disabling predefined rules](#disable-predefined-analyzer-rules). Available for all analyzers. -- [Overriding predefined rules](#override-predefined-analyzer-rules). Available for all analyzers. -- Modifying the default behavior of a given analyzer by [synthesizing and passing a custom configuration](#synthesize-a-custom-configuration). Available for only `nodejs-scan`, `gosec`, and `semgrep`. +- Modifying the behavior of **predefined rules**. This includes: + - [Disabling predefined rules](#disable-predefined-rules). Available for all analyzers. + - [Overriding predefined rules](#override-predefined-rules). Available for all analyzers. +- Replacing predefined rules by [synthesizing a custom configuration](#synthesize-a-custom-configuration) + using **passthroughs**. Available for only [nodejs-scan](https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan) + and [semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep). -To customize the default scanning rules, create a file containing custom rules. These rules -are passed through to the analyzer's underlying scanner tools. +## Disable predefined rules -To create a custom ruleset: +You can disable predefined rules for any SAST analyzer. Disabled rules won't appear +on the [Pipeline Security](../index.md#view-security-scan-information-in-the-pipeline-security-tab) +tab or the [Vulnerability Report](../index.md#view-security-scan-information-in-the-vulnerability-report). -1. Create a `.gitlab` directory at the root of your project, if one doesn't already exist. -1. Create a custom ruleset file named `sast-ruleset.toml` in the `.gitlab` directory. +Disabling rules has a retroactive effect. The analyzer continues to scan for the +vulnerability, but findings are omitted from the [`gl-sast-report.json` artifact](index.md#reports-json-format). + +See the [Schema](#schema) and [Examples](#examples) sections for information on how +to configure this behavior. + +## Override predefined rules + +Certain attributes of predefined rules can be overridden for any SAST analyzer. This +can be useful when adapting SAST to your existing workflow or tools. For example, you +might want to override the severity of a vulnerability based on organizational policy, +or choose a different message to display in the Vulnerability Report. + +See the [Schema](#schema) and [Examples](#examples) sections for information on how +to configure this behavior. + +## Synthesize a custom configuration + +You can completely replace the predefined rules of some SAST analyzers: + +- [nodejs-scan](https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan) - you + can replace the default [njsscan configuration file](https://github.com/ajinabraham/njsscan#configure-njsscan) + with your own. +- [semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) - you can replace + the [GitLab-maintained ruleset](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/tree/main/rules) + with your own. -## Disable predefined analyzer rules +You provide your customizations via passthroughs, which are composed into a +passthrough chain at runtime and evaluated to produce a complete configuration. The +underlying scanner is then executed against this new configuration. -To disable analyzer rules: +There are multiple passthrough types that let you provide configuration in different +ways, such as using a file committed to your repository or inline in the ruleset +configuration file. You can also choose how subsequent passthroughs in the chain are +handled; they can overwrite or append to previous configuration. -1. Set the `disabled` flag to `true` in the context of a `ruleset` section +See the [Schema](#schema) and [Examples](#examples) sections for information on how +to configure this behavior. -1. In one or more `ruleset.identifier` sub sections, list the rules that you want disabled. Every `ruleset.identifier` section has: +## Create the configuration file -- a `type` field, to name the predefined rule identifier that the targeted analyzer uses. -- a `value` field, to name the rule to be disabled. +To create the ruleset configuration file: -### Example: Disable predefined rules of SAST analyzers +1. Create a `.gitlab` directory at the root of your project, if one doesn't already exist. +1. Create a file named `sast-ruleset.toml` in the `.gitlab` directory. + +## Schema + +### The top-level section + +The top-level section contains one or more _configuration sections_, defined as [TOML tables](https://toml.io/en/v1.0.0#table). + +| Setting | Description | +| --------| ----------- | +| `[$analyzer]` | Declares a configuration section for an analyzer. The name follows the snake-case names defined in the list of [SAST analyzers](analyzers.md#sast-analyzers). | -In the following example, the disabled rules are assigned to `eslint` -and `sobelow` by matching the `type` and `value` of identifiers: +Configuration example: ```toml -[eslint] - [[eslint.ruleset]] - disable = true - [eslint.ruleset.identifier] - type = "eslint_rule_id" - value = "security/detect-object-injection" +[semgrep] +... +``` - [[eslint.ruleset]] - disable = true - [eslint.ruleset.identifier] - type = "cwe" - value = "185" +Avoid creating configuration sections that modify existing rules _and_ synthesize a custom ruleset, as +the latter replaces predefined rules completely. -[sobelow] - [[sobelow.ruleset]] - disable = true - [sobelow.ruleset.identifier] - type = "sobelow_rule_id" - value = "sql_injection" +### The `[$analyzer]` configuration section + +The `[$analyzer]` section lets you customize the behavior of an analyzer. Valid properties +differ based on the kind of configuration you're making. + +| Setting | Applies to | Description | +| --------| -------------- | ----------- | +| `[[$analyzer.ruleset]]` | Predefined rules | Defines modifications to an existing rule. | +| `interpolate` | All | If set to `true`, you can use `$VAR` in the configuration to evaluate environment variables. Use this feature with caution, so you don't leak secrets or tokens. (Default: `false`) | +| `description` | Passthroughs | Description of the custom ruleset. | +| `targetdir` | Passthroughs | The directory where the final configuration should be persisted. If empty, a directory with a random name is created. The directory can contain up to 100MB of files. | +| `validate` | Passthroughs | If set to `true`, the content of each passthrough is validated. The validation works for `yaml`, `xml`, `json` and `toml` content. The proper validator is identified based on the extension used in the `target` parameter of the `[[$analyzer.passthrough]]` section. (Default: `false`) | +| `timeout` | Passthroughs | The maximum time to spend to evaluate the passthrough chain, before timing out. The timeout cannot exceed 300 seconds. (Default: 60) | + +#### `interpolate` + +WARNING: +To reduce the risk of leaking secrets, use this feature with caution. + +The example below shows a configuration that uses the `$GITURL` environment variable to access a +private repository. The variable contains a username and token (for example `https://user:token@url`), so +they're not explicitly stored in the configuration file. + +```toml +[semgrep] + description = "My private Semgrep ruleset" + interpolate = true + + [[semgrep.passthrough]] + type = "git" + value = "$GITURL" + ref = "refs/heads/main" ``` -Those vulnerabilities containing the provided type and value are now disabled, meaning -they won't be displayed in Merge Request nor the Vulnerability Report. +### The `[[$analyzer.ruleset]]` section -## Override predefined analyzer rules +The `[[$analyzer.ruleset]]` section targets and modifies a single predefined rule. You can define +one to many of these sections per analyzer. -To override analyzer rules: +| Setting | Description | +| --------| ----------- | +| `disabled` | Whether the rule should be disabled. (Default: `false`) | +| `[$analyzer.ruleset.identifier]` | Selects the predefined rule to be modified. | +| `[$analyzer.ruleset.override]` | Defines the overrides for the rule. | -1. In one or more `ruleset.identifier` subsections, list the rules that you want to override. Every `ruleset.identifier` section has: +Configuration example: - - a `type` field, to name the predefined rule identifier that the targeted analyzer uses. - - a `value` field, to name the rule to be overridden. +```toml +[semgrep] + [[semgrep.ruleset]] + disable = true + ... +``` -1. In the `ruleset.override` context of a `ruleset` section, - provide the keys to override. Any combination of keys can be - overridden. Valid keys are: +### The `[$analyzer.ruleset.identifier]` section - - description - - message - - name - - severity (valid options are: Critical, High, Medium, Low, Unknown, Info) +The `[$analyzer.ruleset.identifier]` section defines the identifiers of the predefined +rule that you wish to modify. -### Example: Override predefined rules of SAST analyzers +| Setting | Description | +| --------| ----------- | +| `type` | The type of identifier used by the predefined rule. | +| `value` | The value of the identifier used by the predefined rule. | -Before adding a ruleset, we verify which vulnerability will be overwritten by viewing the [`gl-sast-report.json`](index.md#reports-json-format): +You can look up the correct values for `type` and `value` by viewing the +[`gl-sast-report.json`](index.md#reports-json-format) produced by the analyzer. +You can download this file as a job artifact from the analyzer's CI job. + +For example, the snippet below shows a finding from a `semgrep` rule with three +identifiers. The `type` and `value` keys in the JSON object correspond to the +values you should provide in this section. ```json -"identifiers": [ +... + "vulnerabilities": [ + { + "id": "7331a4b7093875f6eb9f6eb1755b30cc792e9fb3a08c9ce673fb0d2207d7c9c9", + "category": "sast", + "message": "Key Exchange without Entity Authentication", + "description": "Audit the use of ssh.InsecureIgnoreHostKey\n", + ... + "identifiers": [ { - "type": "gosec_rule_id", - "name": "Gosec Rule ID G307", - "value": "G307" + "type": "semgrep_id", + "name": "gosec.G106-1", + "value": "gosec.G106-1" }, { - "type": "CWE", - "name": "CWE-703", - "value": "703", - "url": "https://cwe.mitre.org/data/definitions/703.html" + "type": "cwe", + "name": "CWE-322", + "value": "322", + "url": "https://cwe.mitre.org/data/definitions/322.html" + }, + { + "type": "gosec_rule_id", + "name": "Gosec Rule ID G106", + "value": "G106" } ] + } + ... + ] +... ``` -In the following example, rules from `gosec` are matched by the `type` -and `value` of identifiers and then overridden: +Configuration example: ```toml -[gosec] - [[gosec.ruleset]] - [gosec.ruleset.identifier] - type = "CWE" - value = "703" - [gosec.ruleset.override] +[semgrep] + [[semgrep.ruleset]] + [semgrep.ruleset.identifier] + type = "semgrep_id" + value = "gosec.G106-1 + ... +``` + +### The `[$analyzer.ruleset.override]` section + +The `[$analyzer.ruleset.override]` section allows you to override attributes of a predefined rule. + +| Setting | Description | +| --------| ----------- | +| `description` | A detailed description of the issue. | +| `message` | (Deprecated) A description of the issue. | +| `name` | The name of the rule. | +| `severity` | The severity of the rule. Valid options are: `Critical`, `High`, `Medium`, `Low`, `Unknown`, `Info`) | + +NOTE: +While `message` is populated by the analyzers, it has been [deprecated](https://gitlab.com/gitlab-org/security-products/analyzers/report/-/blob/1d86d5f2e61dc38c775fb0490ee27a45eee4b8b3/vulnerability.go#L22) +in favor of `name` and `description`. + +Configuration example: + +```toml +[semgrep] + [[semgrep.ruleset]] + [semgrep.ruleset.override] severity = "Critical" + name = "Command injection" + ... ``` -If a vulnerability is found with a type `CWE` with a value of `703` then -the vulnerability severity is overwritten to `Critical`. +### The `[[$analyzer.passthrough]]` section -## Synthesize a custom configuration +NOTE: +This is currently supported by the `nodejs-scan` and `semgrep` analyzers only. -To create a custom configuration, you can use passthrough chains. +The `[[$analyzer.passthrough]]` section allows you to synthesize a custom configuration for an analyzer. You +can define up to 20 of these sections per analyzer. Passthroughs are composed into a _passthrough chain_ +that evaluates into a complete configuration that replaces the predefined rules of the analyzer. -A passthrough is a single step in a passthrough chain. The passthrough is evaluated -in a sequence to incrementally build a configuration. The configuration is then -passed to the target analyzer. +Passthroughs are evaluated in order. Passthroughs listed later in the chain have +a higher precedence and can overwrite or append to data yielded by previous +passthroughs (depending on the `mode`). This is useful for cases where you need +to use or modify an existing configuration. -A configuration section for an analyzer has the following -parameters: +The amount of data generated by a single passthrough is limited to 1MB. -| Parameter | Explanation | -| ------------- | ------ | -| `description` | Description about the analyzer configuration section. | -| `targetdir` | The `targetdir` parameter defines the directory where the final configuration is located. If `targetdir` is empty, the analyzer uses a random directory. The maximum size of `targetdir` is 100MB. | -| `validate` | If set to `true`, the target files for passthroughs (`raw`, `file` and `url`) are validated. The validation works for `yaml`, `xml`, `json` and `toml` files. The proper validator is identified based on the extension of the target file. By default, `validate` is set to `false`. | -| `interpolate` | If set to `true`, environment variable interpolation is enabled so that the configuration uses secrets/tokens. We advise using this feature with caution to not leak any secrets. By default, `interpolate` is set to `false`. | -| `timeout` | The total `timeout` for the evaluation of a passthrough chain is set to 60 seconds. If `timeout` is not set, the default timeout is 60 seconds. The timeout cannot exceed 300 seconds. | +| Setting | Applies to | Description | +| ------- | ---------- | ----------- | +| `type` | All | One of `file`, `raw`, `git` or `url`. | +| `target` | All | The target file to contain the data written by the passthrough evaluation. If empty, a random filename is used. | +| `mode` | All | If `overwrite`, the `target` file is overwritten. If `append`, new content is appended to the `target` file. Note that the `git` type only supports `overwrite`. (Default: `overwrite`) | +| `ref` | `type = "git"` | Contains the name of the branch or the SHA to pull. When using a branch name, specify it in the form `refs/heads/<branch>`, not `refs/remotes/<remote_name>/<branch>`. | +| `subdir` | `type = "git"` | Used to select a subdirectory of the Git repository as the configuration source. | +| `value` | All | For the `file`, `url`, and `git` types, defines the location of the file or Git repository. For the `raw` type, contains the inline configuration. | +| `validator` | All | Used to explicitly invoke validators (`xml`, `yaml`, `json`, `toml`) on the target file after the evaluation of a passthrough. | -A configuration section can include one or more passthrough sections. The maximum number of passthrough sections is 20. -There are several types of passthroughs: +#### Passthrough types | Type | Description | -| ------ | ------ | -| `file` | Use a file that is already available in the Git repository. | +| ------ | ----------- | +| `file` | Use a file that is present in the Git repository. | | `raw` | Provide the configuration inline. | | `git` | Pull the configuration from a remote Git repository. | -| `url` | Fetch the analyzer configuration through HTTP. | +| `url` | Fetch the configuration using HTTP. | -If multiple passthrough sections are defined in a passthrough chain, their -position in the chain defines the order in which they are evaluated. +WARNING: +When using the `raw` passthrough with a YAML snippet, it's recommended to format all indentation +in the `sast-ruleset.toml` file as spaces. The YAML specification mandates spaces over tabs, and the +analyzer will fail to parse your custom ruleset unless the indentation is represented accordingly. -- Passthroughs listed later in the chain sequence have a higher precedence. -- Passthroughs with a higher precedence overwrite (default) and append data - yielded by previous passthroughs. This is useful for cases where you need to - use or modify an existing configuration. +## Examples -Configure a passthrough these parameters: +### Disable predefined rules of SAST analyzers -| Parameter | Explanation | -| ------------ | ----------- | -| `type` | One of `file`, `raw`, `git` or `url`. | -| `target` | The target file that contains the data written by the passthrough evaluation. If no value is provided, a random target file is generated. | -| `mode` | `overwrite`: if `target` exists, overwrites the file; `append`: append to file instead. The default is `overwrite`. | -| `ref` | This option only applies to the `git` passthrough type and contains the name of the branch or the SHA to be used. When using a branch name, specify it in the form `refs/heads/<branch>`, not `refs/remotes/<remote_name>/<branch>`. | -| `subdir` | This option only applies to the `git` passthrough type and can be used to only consider a certain subdirectory of the source Git repository. | -| `value` | For the `file` `url` and `git` types, `value` defines the source location of the file/Git repository; for the `raw` type, `value` carries the raw content to be passed through. | -| `validator` | Can be used to explicitly invoke validators (`xml`, `yaml`, `json`, `toml`) on the target files after the application of a passthrough. Per default, no validator is set. | +With the following custom ruleset configuration, the following rules are omitted from the report: -The amount of data generated by a single passthrough is limited to 1MB. +- `semgrep` rules with a `semgrep_id` of `gosec.G106-1` or a `cwe` of `322`. +- `sobelow` rules with a `sobelow_rule_id` of `sql_injection`. +- `flawfinder` rules with a `flawfinder_func_name` of `memcpy`. -## Passthrough configuration examples +```toml +[semgrep] + [[semgrep.ruleset]] + disable = true + [semgrep.ruleset.identifier] + type = "semgrep_id" + value = "gosec.G106-1" -### Raw passthrough for nodejs-scan + [[semgrep.ruleset]] + disable = true + [semgrep.ruleset.identifier] + type = "cwe" + value = "322" -Define a custom analyzer configuration. In this example, customized rules are -defined for the `nodejs-scan` scanner: +[sobelow] + [[sobelow.ruleset]] + disable = true + [sobelow.ruleset.identifier] + type = "sobelow_rule_id" + value = "sql_injection" + +[flawfinder] + [[flawfinder.ruleset]] + disable = true + [flawfinder.ruleset.identifier] + type = "flawfinder_func_name" + value = "memcpy" +``` + +### Override predefined rules of SAST analyzers + +With the following custom ruleset configuration, vulnerabilities found with +`semgrep` with a type `CWE` and a value `322` will have their severity +overridden to `Critical`. + +```toml +[semgrep] + [[semgrep.ruleset]] + [semgrep.ruleset.identifier] + type = "CWE" + value = "322" + [semgrep.ruleset.override] + severity = "Critical" +``` + +### Synthesize a custom configuration using a raw passthrough for `nodejs-scan` + +With the following custom ruleset configuration, the predefined behavior +of the `nodejs-scan` analyzer is replaced with a custom configuration. + +The syntax used for the `value` follows the [njsscan config format](https://github.com/ajinabraham/njsscan#configure-njsscan). ```toml [nodejs-scan] - description = 'custom ruleset for nodejs-scan' + description = "My custom ruleset for nodejs-scan" [[nodejs-scan.passthrough]] type = "raw" value = ''' +--- - nodejs-extensions: - .js - + template-extensions: - .new - .hbs - '' - + ignore-filenames: -- skip.js - + - skip.js + ignore-paths: - __MACOSX - skip_dir - node_modules - + ignore-extensions: - .hbs - + ignore-rules: - regex_injection_dos - pug_jade_template - express_xss - ''' ``` -### File passthrough for Gosec +### Synthesize a custom configuration using a file passthrough for `semgrep` -Provide the name of the file containing a custom analyzer configuration. In -this example, customized rules for the `gosec` scanner are contained in the -file `gosec-config.json`: +With the following custom ruleset configuration, the predefined ruleset +of the `semgrep` analyzer is replaced with a custom ruleset contained in +a file called `my-semgrep-rules.yaml` in the repository being scanned. + +```yaml +# my-semgrep-rules.yml +--- +rules: +- id: my-custom-rule + pattern: print("Hello World") + message: | + Unauthorized use of Hello World. + severity: CRITICAL + languages: + - python +``` ```toml -[gosec] - description = 'custom ruleset for gosec' +[semgrep] + description = "My custom ruleset for Semgrep" - [[gosec.passthrough]] + [[semgrep.passthrough]] type = "file" - value = "gosec-config.json" + value = "my-semgrep-rules.yml" ``` -### Passthrough chain for Semgrep +### Synthesize a custom configuration using a passthrough chain for `semgrep` -In the below example, we generate a custom configuration under the `/sgrules` -target directory with a total `timeout` of 60 seconds. +With the following custom ruleset configuration, the predefined ruleset +of the `semgrep` analyzer is replaced with a custom ruleset produced by +evaluating a chain of four passthroughs. Each passthrough produces a file +that's written to the `/sgrules` directory within the container. A +`timeout` of 60 seconds is set in case any Git remotes are unresponsive. -Several passthrouh types generate a configuration for the target analyzer: +Different passthrough types are demonstrated in this example: -- Two `git` passthrough sections pull the head of branch - `refs/heads/test` from the `myrules` Git repository, and revision - `97f7686` from the `sast-rules` Git repository. From the `sast-rules` Git - repository, only data from the `go` subdirectory is considered. +- Two `git` passthroughs, the first pulling `refs/heads/test` from the + `myrules` Git repository, and the second pulling revision `97f7686` + from the `sast-rules` repository, and considering only files in the + `go` subdirectory. - The `sast-rules` entry has a higher precedence because it appears later in the configuration. - - If there is a filename collision between files in both repositories, files - from the `sast` repository overwrite files from the `myrules` repository, - as `sast-rules` has higher precedence. -- The `raw` entry creates a file named `insecure.yml` under `/sgrules`. The - full path is `/sgrules/insecure.yml`. -- The `url` entry fetches a configuration made available through a URL and - stores it in the `/sgrules/gosec.yml` file. + - If there's a filename collision between the two checkouts, files + from the `sast-rules` repository will overwrite files from the + `myrules` repository. +- A `raw` passthrough, which writes its `value` to `/sgrules/insecure.yml`. +- A `url` passthrough, which fetches a configuration hosted at a URL and + writes it to `/sgrules/gosec.yml`. Afterwards, Semgrep is invoked with the final configuration located under `/sgrules`. ```toml [semgrep] - description = 'semgrep custom rules configuration' + description = "My custom ruleset for Semgrep" targetdir = "/sgrules" timeout = 60 @@ -277,15 +449,15 @@ Afterwards, Semgrep is invoked with the final configuration located under rules: - id: "insecure" patterns: - - pattern: "func insecure() {...}" + - pattern: "func insecure() {...}" message: | Insecure function insecure detected metadata: cwe: "CWE-200: Exposure of Sensitive Information to an Unauthorized Actor" severity: "ERROR" languages: - - "go" - """ + - "go" +""" [[semgrep.passthrough]] type = "url" @@ -293,89 +465,93 @@ rules: target = "gosec.yml" ``` -### Interpolation - -The code snippet below shows an example configuration that uses an environment -variable `$GITURL` to access a private repositories with a Git URL. The variable contains -a username and token in the `value` field (for example `https://user:token@url`). -It does not explicitly store credentials in the configuration file. To reduce the risk of leaking secrets through created paths and files, use this feature with caution. - -```toml -[semgrep] - description = 'semgrep custom rules configuration' - targetdir = "/sgrules" - interpolate = true - - [[semgrep.passthrough]] - type = "git" - value = "$GITURL" - ref = "refs/heads/main" -``` - -### Configure the append mode for passthroughs - -To append data to previous passthroughs, use the `append` mode for the -passthrough types `file`, `url`, and `raw`. +### Configure the mode for passthroughs in a chain -Passthroughs in `override` mode overwrite files -created when preceding passthroughs in the chain find a naming -collision. If `mode` is set to `append`, a passthrough appends data to the -files created by its predecessors instead of overwriting. +You can choose how to handle filename conflicts that occur between +passthroughs in a chain. The default behavior is to overwrite +existing files with the same name, but you can choose `mode = append` +instead to append the content of later files onto earlier ones. -In the below Semgrep configuration,`/sgrules/insecure.yml` assembles two passthroughs. The rules are: +You can use the `append` mode for the `file`, `url`, and `raw` +passthrough types only. -- `insecure` -- `secret` - -These rules add a search pattern to the analyzer and extends Semgrep capabilities. - -For passthrough chains we recommend that you enable validation. To enable validation, -you can either: - -- set `validate` to `true` - -- set a passthrough `validator` to `xml`, `json`, `yaml`, or `toml`. +With the following custom ruleset configuration, two `raw` passthroughs +are used to iteratively assemble the `/sgrules/my-rules.yml` file, which +is then provided to Semgrep as the ruleset. Each passthrough appends a +single rule to the ruleset. The first passthrough is responsible for +initialising the top-level `rules` object, according to the +[Semgrep rule syntax](https://semgrep.dev/docs/writing-rules/rule-syntax/). ```toml [semgrep] - description = 'semgrep custom rules configuration' + description = "My custom ruleset for Semgrep" targetdir = "/sgrules" validate = true [[semgrep.passthrough]] type = "raw" - target = "insecure.yml" + target = "my-rules.yml" value = """ rules: - id: "insecure" patterns: - - pattern: "func insecure() {...}" + - pattern: "func insecure() {...}" message: | - Insecure function insecure detected + Insecure function 'insecure' detected metadata: - cwe: "... + cwe: "..." severity: "ERROR" languages: - - "go" + - "go" """ [[semgrep.passthrough]] type = "raw" mode = "append" - target = "insecure.yml" + target = "my-rules.yml" value = """ - id: "secret" patterns: - - pattern-either: - - pattern: "$MASK = \"...\"" - - metavariable-regex: - metavariable: "$MASK" - regex: "(password|pass|passwd|pwd|secret|token)" + - pattern-either: + - pattern: '$MASK = "..."' + - metavariable-regex: + metavariable: "$MASK" + regex: "(password|pass|passwd|pwd|secret|token)" message: | - Use of Hard-coded Password + Use of hard-coded password + metadata: cwe: "..." severity: "ERROR" languages: - - "go" + - "go" """ ``` + +```yaml +# /sgrules/my-rules.yml +rules: +- id: "insecure" + patterns: + - pattern: "func insecure() {...}" + message: | + Insecure function 'insecure' detected + metadata: + cwe: "..." + severity: "ERROR" + languages: + - "go" +- id: "secret" + patterns: + - pattern-either: + - pattern: '$MASK = "..."' + - metavariable-regex: + metavariable: "$MASK" + regex: "(password|pass|passwd|pwd|secret|token)" + message: | + Use of hard-coded password + metadata: + cwe: "..." + severity: "ERROR" + languages: + - "go" +``` |