diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-12-22 03:10:19 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-12-22 03:10:19 +0300 |
commit | 85bc1764095bf6f8d2557837b8e3dea9a2149e55 (patch) | |
tree | 9815d776274b31328b975f71aa74c42bfee9c25c /doc | |
parent | 909dcab6e1f82095408cf4adfe099b463f094301 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
-rw-r--r-- | doc/architecture/blueprints/secret_detection/index.md | 108 | ||||
-rw-r--r-- | doc/development/i18n/proofreader.md | 76 | ||||
-rw-r--r-- | doc/operations/tracing.md | 13 | ||||
-rw-r--r-- | doc/topics/offline/quick_start_guide.md | 9 | ||||
-rw-r--r-- | doc/user/application_security/policies/scan-execution-policies.md | 4 | ||||
-rw-r--r-- | doc/user/application_security/policies/scan-result-policies.md | 2 |
6 files changed, 83 insertions, 129 deletions
diff --git a/doc/architecture/blueprints/secret_detection/index.md b/doc/architecture/blueprints/secret_detection/index.md index fb77fffee40..f98c954b60d 100644 --- a/doc/architecture/blueprints/secret_detection/index.md +++ b/doc/architecture/blueprints/secret_detection/index.md @@ -29,6 +29,7 @@ job logs, and project management features such as issues, epics, and MRs. - Support platform-wide detection of tokens to avoid secret leaks - Prevent exposure by rejecting detected secrets - Provide scalable means of detection without harming end user experience +- Unified list of token patterns and masking See [target types](#target-types) for scan target priorities. @@ -39,9 +40,7 @@ during [preceive Git interactions and browser-based detection](#iterations). Secret revocation and rotation is also beyond the scope of this new capability. -Scanned object types beyond the scope of this MVC include: - -See [target types](#target-types) for scan target priorities. +Scanned object types beyond the scope of this MVC are included within [target types](#target-types). #### Management UI @@ -67,7 +66,7 @@ Target object types refer to the scanning targets prioritized for detection of l In order of priority this includes: -1. non-binary Git blobs +1. non-binary Git blobs under 1 megabyte 1. job logs 1. issuable creation (issues, MRs, epics) 1. issuable updates (issues, MRs, epics) @@ -75,30 +74,33 @@ In order of priority this includes: Targets out of scope for the initial phases include: +- non-binary Git blobs over 1 megabyte +- binary Git blobs - Media types (JPEG, PDF, ...) - Snippets - Wikis - Container images +- External media (Youtube platform videos) ### Token types -The existing Secret Detection configuration covers ~100 rules across a variety +The existing Secret Detection configuration covers 100+ rules across a variety of platforms. To reduce total cost of execution and likelihood of false positives the dedicated service targets only well-defined tokens. A well-defined token is -defined as a token with a precise definition, most often a fixed substring prefix or -suffix and fixed length. +defined as a token with a precise definition, most often a fixed substring prefix (or +suffix) and fixed length. Token types to identify in order of importance: 1. Well-defined GitLab tokens (including Personal Access Tokens and Pipeline Trigger Tokens) 1. Verified Partner tokens (including AWS) -1. Remainder tokens currently included in Secret Detection CI configuration +1. Well-defined third party tokens +1. Remainder tokens currently included in Secret Detection analyzer configuration -## Proposal - -### Decisions +In order to minimize false positives, there are no plans to introduce or alert on high-entropy, +arbitrary strings; i.e. patterns such as `3lsjkw3a22`. -- [001: Use Ruby Push Check approach within monolith](decisions/001_use_ruby_push_check_approach_within_monolith.md) +## Proposal The first iteration of the experimental capability will feature a blocking pre-receive hook implemented in the Rails application. This iteration @@ -119,6 +121,10 @@ This service must be: Platform-wide secret detection should be enabled by-default on GitLab SaaS as well as self-managed instances. +### Decisions + +- [001: Use Ruby Push Check approach within monolith](decisions/001_use_ruby_push_check_approach_within_monolith.md) + ## Challenges - Secure authentication to GitLab.com infrastructure @@ -154,6 +160,23 @@ for further background exploration. See [this thread](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105142#note_1194863310) for past discussion around scaling approaches. +### Detection engine + +Our current secret detection offering uses [Gitleaks](https://github.com/zricethezav/gitleaks/) +for all secret scanning in pipeline contexts. By using its `--no-git` configuration +we can scan arbitrary text blobs outside of a repository context and continue to +use it for non-pipeline scanning. + +Changes to the detection engine are out of scope until benchmarking unveils performance concerns. + +For the long-term direction of GitLab Secret Detection, the scope is greater than that of the Gitleaks tool. As such, we should consider feature encapsulation to limit the Gitleaks domain to the relevant build context only. + +In the case of pre-receive detection, we rely on a combination of keyword/substring matches +for pre-filtering and `re2` for regex detections. See [spike issue](https://gitlab.com/gitlab-org/gitlab/-/issues/423832) for initial benchmarks. + +Notable alternatives include high-performance regex engines such as [Hyperscan](https://github.com/intel/hyperscan) or it's portable fork [Vectorscan](https://github.com/VectorCamp/vectorscan). +These systems may be worth exploring in the future if our performance characteristics show a need to grow beyond the existing stack, however the team's velocity in building an independently scalable and generic scanning engine was prioritized, see [ADR 001](decisions/001_use_ruby_push_check_approach_within_monolith.md) for more on the implementation language considerations. + ### Phase 1 - Ruby pushcheck pre-receive integration The critical paths as outlined under [goals above](#goals) cover two major object @@ -204,7 +227,7 @@ sidekiq .[#ff8dd1]----> postgres @enduml ``` -#### Push event detection flow +#### Push Event Detection Flow ```mermaid sequenceDiagram @@ -308,7 +331,7 @@ consul .[#e76a9b]-> prsd_cluster @enduml ``` -#### Push event detection flow +#### Push Event Detection Flow ```mermaid sequenceDiagram @@ -345,7 +368,7 @@ sequenceDiagram Rails->>User: accepted ``` -### Phase 3 - Expansion beyond pre- +### Phase 3 - Expansion beyond pre-receive service The detection flow for arbitrary text blobs, such as issue comments, relies on subscribing to `Notes::PostProcessService` (or equivalent service) to enqueue @@ -364,11 +387,11 @@ In any other case of detection, the Rails application manually creates a vulnera using the `Vulnerabilities::ManuallyCreateService` to surface the finding in the existing Vulnerability Management UI. -#### Architecture +#### High-Level Architecture There is no change to the architecture defined in Phase 2, however the individual load requirements may require scaling up the node counts for the detection service. -#### Detection flow +#### Push Event Detection Flow There is no change to the push event detection flow defined in Phase 2, however the added capability to scan arbitary text blobs directly from Rails allows us to emulate a pre-receive behavior for issuable creations, @@ -403,53 +426,6 @@ sequenceDiagram Rails->>User: rejected: secret found ``` -### Target types - -Target object types refer to the scanning targets prioritized for detection of leaked secrets. - -In order of priority this includes: - -1. non-binary Git blobs -1. job logs -1. issuable creation (issues, MRs, epics) -1. issuable updates (issues, MRs, epics) -1. issuable comments (issues, MRs, epics) - -Targets out of scope for the initial phases include: - -- Media types (JPEG, PDF, ...) -- Snippets -- Wikis -- Container images - -### Token types - -The existing Secret Detection configuration covers ~100 rules across a variety -of platforms. To reduce total cost of execution and likelihood of false positives -the dedicated service targets only well-defined tokens. A well-defined token is -defined as a token with a precise definition, most often a fixed substring prefix or -suffix and fixed length. - -Token types to identify in order of importance: - -1. Well-defined GitLab tokens (including Personal Access Tokens and Pipeline Trigger Tokens) -1. Verified Partner tokens (including AWS) -1. Remainder tokens included in Secret Detection CI configuration - -### Detection engine - -Our current secret detection offering uses [Gitleaks](https://github.com/zricethezav/gitleaks/) -for all secret scanning in pipeline contexts. By using its `--no-git` configuration -we can scan arbitrary text blobs outside of a repository context and continue to -use it for non-pipeline scanning. - -In the case of pre-receive detection, we rely on a combination of keyword/substring matches -for pre-filtering and `re2` for regex detections. See [spike issue](https://gitlab.com/gitlab-org/gitlab/-/issues/423832) for initial benchmarks - -Changes to the detection engine are out of scope until benchmarking unveils performance concerns. - -Notable alternatives include high-performance regex engines such as [Hyperscan](https://github.com/intel/hyperscan) or it's portable fork [Vectorscan](https://github.com/VectorCamp/vectorscan). - ## Iterations - ✓ Define [requirements for detection coverage and actions](https://gitlab.com/gitlab-org/gitlab/-/issues/376716) @@ -459,9 +435,9 @@ Notable alternatives include high-performance regex engines such as [Hyperscan]( - [Pre-Production Performance Profiling for pre-receive PoCs](https://gitlab.com/gitlab-org/gitlab/-/issues/428499) - Profiling service capabilities - ✓ [Benchmarking regex performance between Ruby and Go approaches](https://gitlab.com/gitlab-org/gitlab/-/issues/423832) - - gRPC commit retrieval from Gitaly + - x gRPC commit retrieval from Gitaly - transfer latency, CPU, and memory footprint -- Implementation of secret scanning service MVC (targeting individual commits) +- ✓ Implementation of secret scanning gem integration MVC (targeting individual commits) - Capacity planning for addition of service component to Reference Architectures headroom - Security and readiness review - Deployment and monitoring diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 35e423b28e9..4d8221da941 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -16,7 +16,7 @@ are very appreciative of the work done by translators and proofreaders! - Albanian - Proofreaders needed. - Amharic - - Tsegaselassie Tadesse - [GitLab](https://gitlab.com/tsega), [Crowdin](https://crowdin.com/profile/tsegaselassi) + - Proofreaders needed. - Arabic - Proofreaders needed. - Basque @@ -24,51 +24,38 @@ are very appreciative of the work done by translators and proofreaders! - Belarusian - Anton Katsuba - [GitLab](https://gitlab.com/coinvariant), [Crowdin](https://crowdin.com/profile/aerialfiddle) - Bosnian - - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) + - Proofreaders needed. - Bulgarian - - Lyubomir Vasilev - [Crowdin](https://crowdin.com/profile/lyubomirv) + - Proofreaders needed. - Catalan - - David Planella - [GitLab](https://gitlab.com/dplanella), [Crowdin](https://crowdin.com/profile/dplanella) + - Proofreaders needed. - Chinese Simplified 简体中文 - - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) - - Victor Wu - [GitLab](https://gitlab.com/_victorwu_), [Crowdin](https://crowdin.com/profile/victorwu) - - Xiaogang Wen - [GitLab](https://gitlab.com/xiaogang_cn), [Crowdin](https://crowdin.com/profile/xiaogang_gitlab) - Zhiyuan Lu - [GitLab](https://gitlab.com/luzhiyuan.deer), [Crowdin](https://crowdin.com/profile/luzhiyuan.deer) - Chinese Traditional 繁體中文 - - Weizhe Ding - [GitLab](https://gitlab.com/d.weizhe), [Crowdin](https://crowdin.com/profile/d.weizhe) - - Yi-Jyun Pan - [GitLab](https://gitlab.com/pan93412), [Crowdin](https://crowdin.com/profile/pan93412) - - Victor Wu - [GitLab](https://gitlab.com/_victorwu_), [Crowdin](https://crowdin.com/profile/victorwu) - Hansel Wang - [GitLab](https://gitlab.com/airness), [Crowdin](https://crowdin.com/profile/airness) - Chinese Traditional, Hong Kong 繁體中文 (香港) - - Victor Wu - [GitLab](https://gitlab.com/_victorwu_), [Crowdin](https://crowdin.com/profile/victorwu) - - Ivan Ip - [GitLab](https://gitlab.com/lifehome), [Crowdin](https://crowdin.com/profile/lifehome) + - Proofreaders needed. - Croatian - - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) + - Proofreaders needed. - Czech - - Jan Urbanec - [GitLab](https://gitlab.com/TatranskyMedved), [Crowdin](https://crowdin.com/profile/Tatranskymedved) + - Proofreaders needed. - Danish - - Saederup92 - [GitLab](https://gitlab.com/Saederup92), [Crowdin](https://crowdin.com/profile/Saederup92) - scootergrisen - [GitLab](https://gitlab.com/scootergrisen), [Crowdin](https://crowdin.com/profile/scootergrisen) - Dutch - - Emily Hendle - [GitLab](https://gitlab.com/pundachan), [Crowdin](https://crowdin.com/profile/pandachan) + - Proofreaders needed. - English (UK) - - Andrew Smith - [GitLab](https://gitlab.com/espadav8), [Crowdin](https://crowdin.com/profile/espadav8) + - Proofreaders needed. - Esperanto - - Lyubomir Vasilev - [Crowdin](https://crowdin.com/profile/lyubomirv) + - Proofreaders needed. - Estonian - Proofreaders needed. - Filipino - - Andrei Jiroh Halili - [GitLab](https://gitlab.com/ajhalili2006), [Crowdin](https://crowdin.com/profile/AndreiJirohHaliliDev2006) + - Proofreaders needed. - French - - Davy Defaud - [GitLab](https://gitlab.com/DevDef), [Crowdin](https://crowdin.com/profile/DevDef) - - Germain Gorisse - [GitLab](https://gitlab.com/ggorisse), [Crowdin](https://crowdin.com/profile/germaingorisse) - Xavier Delatour - [GitLab](https://gitlab.com/xdelatour), [Crowdin](https://crowdin.com/profile/xdelatour) - Galician - - Antón Méixome - [Crowdin](https://crowdin.com/profile/meixome) - - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) + - Proofreaders needed. - German - - Michael Hahnle - [GitLab](https://gitlab.com/mhah), [Crowdin](https://crowdin.com/profile/mhah) - - Katrin Leinweber - [GitLab](https://gitlab.com/katrinleinweber), [Crowdin](https://crowdin.com/profile/katrinleinweber) - Vladislav Wanner - [GitLab](https://gitlab.com/RumBugen), [Crowdin](https://crowdin.com/profile/RumBugen) - Daniel Ziegenberg - [GitLab](https://gitlab.com/ziegenberg), [Crowdin](https://crowdin.com/profile/ziegenberg) - Greek @@ -80,69 +67,46 @@ are very appreciative of the work done by translators and proofreaders! - Hungarian - Proofreaders needed. - Indonesian - - Adi Ferdian - [GitLab](https://gitlab.com/adiferd), [Crowdin](https://crowdin.com/profile/adiferd) - - Ahmad Naufal Mukhtar - [GitLab](https://gitlab.com/anaufalm), [Crowdin](https://crowdin.com/profile/anaufalm) + - Proofreaders needed. - Italian - - Massimiliano Cuttini - [GitLab](https://gitlab.com/maxcuttins), [Crowdin](https://crowdin.com/profile/maxcuttins) - - Paolo Falomo - [GitLab](https://gitlab.com/paolofalomo), [Crowdin](https://crowdin.com/profile/paolo.falomo) + - Proofreaders needed. - Japanese - - Hiroyuki Sato - [GitLab](https://gitlab.com/hiroponz), [Crowdin](https://crowdin.com/profile/hiroponz) - Tomo Dote - [GitLab](https://gitlab.com/fu7mu4), [Crowdin](https://crowdin.com/profile/fu7mu4) - - Hiromi Nozawa - [GitLab](https://gitlab.com/hir0mi), [Crowdin](https://crowdin.com/profile/hir0mi) - - Takuya Noguchi - [GitLab](https://gitlab.com/tnir), [Crowdin](https://crowdin.com/profile/tnir) - Tsukasa Komatsubara - [GitLab](https://gitlab.com/tkomatsubara), [Crowdin](https://crowdin.com/profile/tkomatsubara) - Korean - - Chang-Ho Cha - [GitLab](https://gitlab.com/changho-cha), [Crowdin](https://crowdin.com/profile/zzazang) - - Ji Hun Oh - [GitLab](https://gitlab.com/Baw-Appie), [Crowdin](https://crowdin.com/profile/BawAppie) - - Jeongwhan Choi - [GitLab](https://gitlab.com/jeongwhanchoi), [Crowdin](https://crowdin.com/profile/jeongwhanchoi) - Sunjung Park - [GitLab](https://gitlab.com/sunjungp), [Crowdin](https://crowdin.com/profile/sunjungp) - Mongolian - Proofreaders needed. - Norwegian Bokmal - Imre Kristoffer Eilertsen - [GitLab](https://gitlab.com/DandelionSprout), [Crowdin](https://crowdin.com/profile/DandelionSprout) - Polish - - Filip Mech - [GitLab](https://gitlab.com/mehenz), [Crowdin](https://crowdin.com/profile/mehenz) - - Maksymilian Roman - [GitLab](https://gitlab.com/villaincandle), [Crowdin](https://crowdin.com/profile/villaincandle) - - Jakub Gładykowski - [GitLab](https://gitlab.com/gladykov), [Crowdin](https://crowdin.com/profile/gladykov) + - Proofreaders needed. - Portuguese - - Diogo Trindade - [GitLab](https://gitlab.com/luisdiogo2071317), [Crowdin](https://crowdin.com/profile/ldiogotrindade) + - Proofreaders needed. - Portuguese, Brazilian - - Paulo George Gomes Bezerra - [GitLab](https://gitlab.com/paulobezerra) - - André Gama - [GitLab](https://gitlab.com/andregamma), [Crowdin](https://crowdin.com/profile/ToeOficial) - Eduardo Addad de Oliveira - [GitLab](https://gitlab.com/eduardoaddad), [Crowdin](https://crowdin.com/profile/eduardoaddad) - - Horberlan Brito - [GitLab](https://gitlab.com/horberlan), [Crowdin](https://crowdin.com/profile/horberlan) - Romanian - - Mircea Pop - [GitLab](https://gitlab.com/eeex), [Crowdin](https://crowdin.com/profile/eex) - - Rareș Pița - [GitLab](https://gitlab.com/dlphin) - - Nicolae Liviu - [GitLab](https://gitlab.com/nicklcanada), [Crowdin](https://crowdin.com/profile/nicklcanada) + - Proofreaders needed. - Russian - - Nikita Grylov - [GitLab](https://gitlab.com/nixel2007), [Crowdin](https://crowdin.com/profile/nixel2007) - - Alexy Lustin - [GitLab](https://gitlab.com/allustin), [Crowdin](https://crowdin.com/profile/lustin) - Mark Minakou - [GitLab](https://gitlab.com/sandzhaj), [Crowdin](https://crowdin.com/profile/sandzhaj) - - NickVolynkin - [Crowdin](https://crowdin.com/profile/NickVolynkin) - Andrey Komarov - [GitLab](https://gitlab.com/elkamarado), [Crowdin](https://crowdin.com/profile/kamarado) - - Iaroslav Postovalov - [GitLab](https://gitlab.com/CMDR_Tvis), [Crowdin](https://crowdin.com/profile/CMDR_Tvis) - Serbian (Latin and Cyrillic) - - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) + - Proofreaders needed. - Sinhalese/Sinhala සිංහල - හෙළබස (HelaBasa) - [GitLab](https://gitlab.com/helabasa), [Crowdin](https://crowdin.com/profile/helabasa) - Slovak - Proofreaders needed. - Spanish - - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) - David Elizondo - [GitLab](https://gitlab.com/daelmo), [Crowdin](https://crowdin.com/profile/daelmo) - Pablo Reyes - [GitLab](https://gitlab.com/pabloryst9n), [Crowdin](https://crowdin.com/profile/pabloryst9n) - - Swedish - Johannes Nilsson - [GitLab](https://gitlab.com/nlssn), [Crowdin](https://crowdin.com/profile/nlssn) - Turkish - - Ali Demirtaş - [GitLab](https://gitlab.com/alidemirtas), [Crowdin](https://crowdin.com/profile/alidemirtas) - - Rıfat Ünalmış (Rifat Unalmis) - [GitLab](https://gitlab.com/runalmis), [Crowdin](https://crowdin.com/profile/runalmis) - - İsmail Arılık - [GitLab](https://gitlab.com/ismailarilik), [Crowdin](https://crowdin.com/profile/ismailarilik) + - Proofreaders needed. - Ukrainian - Andrew Vityuk - [GitLab](https://gitlab.com/3_1_3_u), [Crowdin](https://crowdin.com/profile/andruwa13) - Welsh - - Delyth Prys - [GitLab](https://gitlab.com/Delyth), [Crowdin](https://crowdin.com/profile/DelythPrys) + - Proofreaders needed. <!-- vale gitlab.Spelling = YES --> ## Become a proofreader diff --git a/doc/operations/tracing.md b/doc/operations/tracing.md index f71f41211d8..f6ac0b6174d 100644 --- a/doc/operations/tracing.md +++ b/doc/operations/tracing.md @@ -4,9 +4,9 @@ group: Observability info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Distributed tracing **(ULTIMATE SAAS EXPERIMENT)** +# Distributed tracing **(ULTIMATE SAAS BETA)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124966) in GitLab 16.2 [with flags](../administration/feature_flags.md) named `observability_group_tab` and `observability_tracing`. Disabled by default. This feature is an [Experiment](../policy/experiment-beta-support.md#experiment). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124966) in GitLab 16.2 [with flags](../administration/feature_flags.md) named `observability_group_tab` and `observability_tracing`. Disabled by default. This feature is in [Beta](../policy/experiment-beta-support.md#beta). > - Feature flag `observability_group_tab` [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133264) in GitLab 16.5. FLAG: @@ -16,7 +16,7 @@ The feature is not ready for production use. With distributed tracing, you can troubleshoot application performance issues by inspecting how a request moves through different services and systems, the timing of each operation, and any errors or logs as they occur. Tracing is particularly useful in the context of microservice applications, which group multiple independent services collaborating to fulfill user requests. -This feature is an [Experiment](../policy/experiment-beta-support.md). For more information, see the [group direction page](https://about.gitlab.com/direction/analytics/observability/). To leave feedback about tracing bugs or functionality, comment in the [feedback issue](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/2363) or open a [new issue](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/new). +This feature is in [Beta](../policy/experiment-beta-support.md). For more information, see the [group direction page](https://about.gitlab.com/direction/analytics/observability/). To leave feedback about tracing bugs or functionality, comment in the [feedback issue](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/2590) or open a [new issue](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/new). ## Configure distributed tracing for a project @@ -81,3 +81,10 @@ The trace details page and a list of spans are displayed. To view the attributes for a single span, select it from the list. ![tracing drawer](img/tracing_drawer_v16_7.png) + +## Tracing ingestion limits + +Tracing ingests a maximum of 102,400 bytes per minute. +After the limit is exceeded, a `429 Too Many Requests` response is returned. + +To request a limit increase to 104,8576 bytes per minute, contact GitLab support. diff --git a/doc/topics/offline/quick_start_guide.md b/doc/topics/offline/quick_start_guide.md index b6340d905d4..f5489d53ae0 100644 --- a/doc/topics/offline/quick_start_guide.md +++ b/doc/topics/offline/quick_start_guide.md @@ -258,8 +258,9 @@ Package metadata is stored in the following Google Cloud Provider (GCP) buckets: ```shell # To download the package metadata exports, an outbound connection to Google Cloud Storage bucket must be allowed. + # Skip v1 objects using -y "^v1\/" to only download v2 objects. v1 data is no longer used and deprecated since 16.3. mkdir -p "$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/$DATA_DIR" - gsutil -m rsync -r -d gs://$PKG_METADATA_BUCKET "$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/$DATA_DIR" + gsutil -m rsync -r -d -y "^v1\/" gs://$PKG_METADATA_BUCKET "$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/$DATA_DIR" # Alternatively, if the GitLab instance is not allowed to connect to the Google Cloud Storage bucket, the package metadata # exports can be downloaded using a machine with the allowed access, and then copied to the root of the GitLab Rails directory. @@ -293,7 +294,9 @@ PKG_METADATA_MANIFEST_OUTPUT_FILE="/tmp/package_metadata_${DATA_TYPE}_export_man PKG_METADATA_DOWNLOADS_OUTPUT_FILE="/tmp/package_metadata_${DATA_TYPE}_object_links.tsv" # Download the contents of the bucket -curl --silent --show-error --request GET "https://storage.googleapis.com/storage/v1/b/$PKG_METADATA_BUCKET/o?maxResults=7500" > "$PKG_METADATA_MANIFEST_OUTPUT_FILE" +# Filter results using `prefix=v2` to only download v2 objects. v1 data is no longer used and deprecated since 16.3. +# Maximum number of objects returned by the API seems to be 5000 and there are currently (2023-12-21) 2650 objects for V2 dataset. +curl --silent --show-error --request GET "https://storage.googleapis.com/storage/v1/b/$PKG_METADATA_BUCKET/o?prefix=v2%2f&maxResults=5000" > "$PKG_METADATA_MANIFEST_OUTPUT_FILE" # Parse the links and names for the bucket objects and output them into a tsv file jq -r '.items[] | [.name, .mediaLink] | @tsv' "$PKG_METADATA_MANIFEST_OUTPUT_FILE" > "$PKG_METADATA_DOWNLOADS_OUTPUT_FILE" @@ -331,7 +334,7 @@ To automatically update your local copy with the upstream changes, a cron job ca For License Scanning: ```plaintext -*/30 * * * * gsutil -m rsync -r -d gs://prod-export-license-bucket-1a6c642fc4de57d4 $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/licenses +*/30 * * * * gsutil -m rsync -r -d -y "^v1\/" gs://prod-export-license-bucket-1a6c642fc4de57d4 $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/licenses ``` For Dependency Scanning: diff --git a/doc/user/application_security/policies/scan-execution-policies.md b/doc/user/application_security/policies/scan-execution-policies.md index f299a38dff1..9a2053d43dd 100644 --- a/doc/user/application_security/policies/scan-execution-policies.md +++ b/doc/user/application_security/policies/scan-execution-policies.md @@ -46,6 +46,10 @@ to remove the `test` stage, jobs will run in the `scan-policies` stage instead. - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For a video walkthrough, see [How to set up Security Scan Policies in GitLab](https://youtu.be/ZBcqGmEwORA?si=aeT4EXtmHjosgjBY). - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Enforcing scan execution policies on projects with no GitLab CI/CD configuration](https://www.youtube.com/watch?v=sUfwQQ4-qHs). +## Requirements and limitations + +- The maximum number of scan execution policies is five per security policy project. + ## Scan execution policy editor NOTE: diff --git a/doc/user/application_security/policies/scan-result-policies.md b/doc/user/application_security/policies/scan-result-policies.md index 0fadd761fe4..51d5dff90b2 100644 --- a/doc/user/application_security/policies/scan-result-policies.md +++ b/doc/user/application_security/policies/scan-result-policies.md @@ -31,7 +31,7 @@ The following video gives you an overview of GitLab scan result policies: - You must add the respective [security scanning tools](../index.md#application-coverage). Otherwise, scan result policies do not have any effect. -- The maximum number of policies is five. +- The maximum number of scan result policies is five per security policy project. - Each policy can have a maximum of five rules. - All configured scanners must be present in the merge request's latest pipeline. If not, approvals are required even if some vulnerability criteria have not been met. - Scan result policies evaluate findings and determine approval requirements based on the job artifact reports published in a completed pipeline. However, scan result policies do not check the integrity or authenticity of the scan results generated in the artifact reports. |