diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-10-26 03:09:31 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-10-26 03:09:31 +0300 |
commit | 9a02cb29182ae369f98a8b04840144c67b544a7a (patch) | |
tree | 60570d599fcdda02ba8f4cf1dbc88da911816084 | |
parent | 098444d917f660f870daa75e9166dcb735573ad4 (diff) |
Add latest changes from gitlab-org/gitlab@master
-rw-r--r-- | config/feature_flags/development/remove_mr_blocking_constraints.yml | 8 | ||||
-rw-r--r-- | doc/api/graphql/reference/index.md | 1 | ||||
-rw-r--r-- | doc/user/application_security/get-started-security.md | 48 | ||||
-rw-r--r-- | lib/sbom/purl_type/converter.rb | 1 | ||||
-rw-r--r-- | package.json | 2 | ||||
-rw-r--r-- | qa/qa/page/project/web_ide/vscode.rb | 5 | ||||
-rw-r--r-- | qa/qa/specs/features/api/1_manage/import/import_large_github_repo_spec.rb | 55 | ||||
-rw-r--r-- | spec/lib/sbom/purl_type/converter_spec.rb | 1 | ||||
-rw-r--r-- | yarn.lock | 78 |
9 files changed, 112 insertions, 87 deletions
diff --git a/config/feature_flags/development/remove_mr_blocking_constraints.yml b/config/feature_flags/development/remove_mr_blocking_constraints.yml new file mode 100644 index 00000000000..df4631c711d --- /dev/null +++ b/config/feature_flags/development/remove_mr_blocking_constraints.yml @@ -0,0 +1,8 @@ +--- +name: remove_mr_blocking_constraints +introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133897 +rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/429428 +milestone: '16.6' +type: development +group: group::code review +default_enabled: false diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index af314f0c0df..1a84a872942 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -29244,6 +29244,7 @@ Values for package manager. | <a id="packagemanagerpip"></a>`PIP` | Package manager: pip. | | <a id="packagemanagerpipenv"></a>`PIPENV` | Package manager: pipenv. | | <a id="packagemanagerpnpm"></a>`PNPM` | Package manager: pnpm. | +| <a id="packagemanagerpoetry"></a>`POETRY` | Package manager: poetry. | | <a id="packagemanagersbt"></a>`SBT` | Package manager: sbt. | | <a id="packagemanagersetuptools"></a>`SETUPTOOLS` | Package manager: setuptools. | | <a id="packagemanageryarn"></a>`YARN` | Package manager: yarn. | diff --git a/doc/user/application_security/get-started-security.md b/doc/user/application_security/get-started-security.md index 3e73fbc5955..fd6829f5ae2 100644 --- a/doc/user/application_security/get-started-security.md +++ b/doc/user/application_security/get-started-security.md @@ -11,32 +11,42 @@ For an overview, see [Adopting GitLab application security](https://www.youtube. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an interactive reading and how-to demo playlist, see [Get Started With GitLab Application Security Playlist](https://www.youtube.com/playlist?list=PL05JrBw4t0KrUrjDoefSkgZLx5aJYFaF9) -The following steps help you get the most from GitLab application security tools. These steps are a recommended order of operations. You can choose to implement capabilities in a different order or omit features that do not apply to your specific needs. - -1. Enable [Secret Detection](secret_detection/index.md) and [Dependency Scanning](dependency_scanning/index.md) - to identify any leaked secrets and vulnerable packages in your codebase. - - - For all security scanners, enable them by updating your [`.gitlab-ci.yml`](../../ci/yaml/gitlab_ci_yaml.md) directly on your `default` branch. This creates a baseline scan of your `default` branch, which is necessary for - feature branch scans to be compared against. This allows [merge requests](../project/merge_requests/index.md) - to display only newly-introduced vulnerabilities. Otherwise, merge requests display every - vulnerability in the branch, regardless of whether it was introduced by a change in the branch. - - If you are after simplicity, enable only Secret Detection first. It only has one analyzer, - no build requirements, and relatively simple findings: is this a secret or not? - - It is good practice to enable Dependency Scanning early so you can start identifying existing - vulnerable packages in your codebase. -1. Let your team get comfortable with [vulnerability reports](vulnerability_report/index.md) and - establish a vulnerability triage workflow. -1. Consider creating [labels](../project/labels.md) and [issue boards](../project/issue_board.md) to +The following steps help introduce you to GitLab application security tools incrementally. +You can choose to enable features in a different order, or skip features that don't apply to your specific needs. +You should start with: + +- [Secret Detection](secret_detection/index.md), which works with all programming languages and creates understandable results. +- [Dependency Scanning](dependency_scanning/index.md), which finds known vulnerabilities in the dependencies your code uses. + +If it's your first time setting up GitLab security scanning, you should start with a single project. +After you've gotten familiar with how scanning works, you can then choose to: + +- Follow [the same steps](#recommended-steps) to enable scanning in more projects. +- [Enforce scanning](index.md#enforce-scan-execution) across more of your projects at once. + +## Recommended steps + +1. Choose a project to enable and test security features. Consider choosing a project: + - That uses your organization's typical programming languages and technologies, because some scanning features work differently across languages. + - Where you can try out new settings, like required approvals, without interrupting your team's daily work. + You could create a copy of a higher-traffic project for testing, or select a project that's not as busy. +1. Create a merge request to [enable Secret Detection](secret_detection/index.md#enable-secret-detection) and [enable Dependency Scanning](dependency_scanning/index.md#configuration) + to identify any leaked secrets and vulnerable packages in that project. + - Security scanners run in your project's [CI/CD pipelines](../../ci/pipelines/index.md). Creating a merge request to update your [`.gitlab-ci.yml`](../../ci/yaml/gitlab_ci_yaml.md) helps you check how the scanners work with your project before they start running in every pipeline. In the merge request, you can change relevant [Secret Detection settings](secret_detection/index.md#configure-scan-settings) or [Dependency Scanning settings](dependency_scanning/index.md#available-cicd-variables) to accommodate your project's layout or configuration. For example, you might choose to exclude a directory of third-party code from scanning. + - After you merge this MR to your [default branch](../project/repository/branches/default.md), the system creates a baseline scan. This scan identifies which vulnerabilities already exist on the default branch so [merge requests](../project/merge_requests/index.md) can highlight only newly-introduced problems. Without a baseline scan, merge requests display every + vulnerability in the branch, even if the vulnerability already exists on the default branch. +1. Let your team get comfortable with [viewing security findings in merge requests](index.md#view-security-scan-information) and the [vulnerability report](vulnerability_report/index.md). +1. Establish a vulnerability triage workflow. + - Consider creating [labels](../project/labels.md) and [issue boards](../project/issue_board.md) to help manage issues created from vulnerabilities. Issue boards allow all stakeholders to have a common view of all issues and track remediation progress. +1. Monitor the [Security Dashboard](security_dashboard/index.md) trends to gauge success in remediating existing vulnerabilities and preventing the introduction of new ones. 1. Enforce scheduled security scanning jobs by using a [scan execution policy](policies/scan-execution-policies.md). - These scheduled jobs run independently from any other security scans you may have defined in a compliance framework pipeline or in the project's `.gitlab-ci.yml` file. - Running regular dependency and [container scans](container_scanning/index.md) surface newly-discovered vulnerabilities that already exist in your repository. - Scheduled scans are most useful for projects or important branches with low development activity where pipeline scans are infrequent. 1. Create a [scan result policy](policies/index.md) to limit new vulnerabilities from being merged - into your `default` branch. -1. Monitor the [Security Dashboard](security_dashboard/index.md) trends to gauge success in - remediating existing vulnerabilities and preventing the introduction of new ones. + into your [default branch](../project/repository/branches/default.md). 1. Enable other scan types such as [SAST](sast/index.md), [DAST](dast/index.md), [Fuzz testing](coverage_fuzzing/index.md), or [Container Scanning](container_scanning/index.md). 1. Use [Compliance Pipelines](../group/compliance_frameworks.md#compliance-pipelines) diff --git a/lib/sbom/purl_type/converter.rb b/lib/sbom/purl_type/converter.rb index bfcfb414180..bc08083fdae 100644 --- a/lib/sbom/purl_type/converter.rb +++ b/lib/sbom/purl_type/converter.rb @@ -18,6 +18,7 @@ module Sbom 'nuget' => 'nuget', 'pip' => 'pypi', 'pipenv' => 'pypi', + 'poetry' => 'pypi', 'setuptools' => 'pypi', 'python-pkg' => 'pypi' # this package manager is generated by trivy }.with_indifferent_access.freeze diff --git a/package.json b/package.json index ef9148f2489..b605a4ffbbd 100644 --- a/package.json +++ b/package.json @@ -193,7 +193,7 @@ "remark-rehype": "^10.1.0", "scrollparent": "^2.0.1", "semver": "^7.3.4", - "sentrybrowser": "npm:@sentry/browser@7.75.0", + "sentrybrowser": "npm:@sentry/browser@7.75.1", "sentrybrowser5": "npm:@sentry/browser@5.30.0", "sortablejs": "^1.10.2", "string-hash": "1.1.3", diff --git a/qa/qa/page/project/web_ide/vscode.rb b/qa/qa/page/project/web_ide/vscode.rb index 0997064c19f..63dc75b523f 100644 --- a/qa/qa/page/project/web_ide/vscode.rb +++ b/qa/qa/page/project/web_ide/vscode.rb @@ -147,7 +147,7 @@ module QA end end - wait_for_requests + Support::WaitForRequests.wait_for_requests(finish_loading_wait: 30) Support::Waiter.wait_until(max_duration: 10, reload_page: page, retry_on_exception: true) do within_vscode_editor do # Check for webide file_explorer element @@ -232,8 +232,9 @@ module QA end end - def add_file_content(prompt_data) + def add_prompt_into_a_file(file_name, prompt_data) within_vscode_editor do + open_file_from_explorer(file_name) click_inside_editor_frame within_file_editor do send_keys(:enter, :enter, prompt_data) diff --git a/qa/qa/specs/features/api/1_manage/import/import_large_github_repo_spec.rb b/qa/qa/specs/features/api/1_manage/import/import_large_github_repo_spec.rb index 540ebb342fc..64150967877 100644 --- a/qa/qa/specs/features/api/1_manage/import/import_large_github_repo_spec.rb +++ b/qa/qa/specs/features/api/1_manage/import/import_large_github_repo_spec.rb @@ -3,6 +3,12 @@ require "etc" # Lifesize project import test executed from https://gitlab.com/gitlab-org/manage/import/import-metrics +# +# This test is executed using different size live projects on GitHub. +# Due to projects being active, there can be a lag between when test is fetching data from GitHub and +# when importer is fetching data. It can create extra objects in imported project compared to test expectation. +# Because of this, all expectation check for inclusion rather than exact match to avoid failures if extra issues, +# comments, events got created while import was running. # rubocop:disable Rails/Pluck module QA @@ -21,7 +27,6 @@ module QA let(:api_parallel_threads) { ENV['QA_LARGE_IMPORT_API_PARALLEL']&.to_i || Etc.nprocessors } let(:logger) { Runtime::Logger.logger } - let(:differ) { RSpec::Support::Differ.new(color: true) } let(:gitlab_address) { QA::Runtime::Scenario.gitlab_address.chomp("/") } let(:dummy_url) { "https://example.com" } # this is used to replace all dynamic urls in descriptions and comments let(:api_request_params) { { auto_paginate: true, attempts: 2 } } @@ -359,10 +364,8 @@ module QA def verify_repository_import logger.info("== Verifying repository import ==") expect(imported_project.description).to eq(gh_repo.description) - # check via include, importer creates more branches - # https://gitlab.com/gitlab-org/gitlab/-/issues/332711 expect(gl_branches).to include(*gh_branches) - expect(gl_commits).to match_array(gh_commits) + expect(gl_commits).to include(*gh_commits) end # Verify imported labels @@ -370,7 +373,6 @@ module QA # @return [void] def verify_labels_import logger.info("== Verifying label import ==") - # check via include, additional labels can be inherited from parent group expect(gl_labels).to include(*gh_labels) end @@ -379,7 +381,7 @@ module QA # @return [void] def verify_milestones_import logger.info("== Verifying milestones import ==") - expect(gl_milestones).to match_array(gh_milestones) + expect(gl_milestones).to include(*gh_milestones) end # Verify imported merge requests and mr issues @@ -426,12 +428,16 @@ module QA def verify_mrs_or_issues(type) # Compare length to have easy to read overview how many objects are missing # - expected = type == 'mr' ? mrs : gl_issues - actual = type == 'mr' ? gh_prs : gh_issues - count_msg = "Expected to contain same amount of #{type}s. Gitlab: #{expected.length}, Github: #{actual.length}" - expect(expected.length).to eq(actual.length), count_msg + expected = type == 'mr' ? gh_prs : gh_issues + actual = type == 'mr' ? mrs : gl_issues + + missing_objects = (expected.keys - actual.keys).map { |it| expected[it].slice(:title, :url) } + count_msg = <<~MSG + Expected to contain all of GitHub's #{type}s. Gitlab: #{actual.length}, Github: #{expected.length}. + Missing: #{missing_objects.map { |it| it[:url] }} + MSG + expect(expected.length <= actual.length).to be_truthy, count_msg - missing_objects = (actual.keys - expected.keys).map { |it| actual[it].slice(:title, :url) } missing_content = verify_comments_and_events(type, actual, expected) { @@ -461,39 +467,36 @@ module QA # expected_body = expected_item[:body] actual_body = actual_item[:body] - body_msg = <<~MSG - #{msg} same description. diff:\n#{differ.diff(expected_body, actual_body)} - MSG + body_msg = "#{msg} same description" expect(expected_body).to eq(actual_body), body_msg # Print amount difference first # expected_comments = expected_item[:comments] actual_comments = actual_item[:comments] - comment_count_msg = <<~MSG - #{msg} same amount of comments. Gitlab: #{expected_comments.length}, Github: #{actual_comments.length} + comment_count_msg = <<~MSG.strip + #{msg} same comments. GitHub: #{expected_comments.length}, GitLab: #{actual_comments.length} MSG - expect(expected_comments.length).to eq(actual_comments.length), comment_count_msg - expect(expected_comments).to match_array(actual_comments) + expect(actual_comments).to include(*expected_comments), comment_count_msg expected_events = expected_item[:events] actual_events = actual_item[:events] - event_count_msg = <<~MSG - #{msg} same amount of events. Gitlab: #{expected_events.length}, Github: #{actual_events.length} + event_count_msg = <<~MSG.strip + #{msg} same events. GitHub: #{expected_events.length}, GitLab: #{actual_events.length}. + Missing event: #{expected_events - actual_events} MSG - expect(expected_events.length).to eq(actual_events.length), event_count_msg - expect(expected_events).to match_array(actual_events) + expect(actual_events).to include(*expected_events), event_count_msg # Save missing comments and events # - comment_diff = actual_comments - expected_comments - event_diff = actual_events - expected_events + comment_diff = expected_comments - actual_comments + event_diff = expected_events - actual_events next if comment_diff.empty? && event_diff.empty? missing_content << { title: title, - github_url: actual_item[:url], - gitlab_url: expected_item[:url], + github_url: expected_item[:url], + gitlab_url: actual_item[:url], missing_comments: comment_diff.empty? ? nil : comment_diff, missing_events: event_diff.empty? ? nil : event_diff }.compact diff --git a/spec/lib/sbom/purl_type/converter_spec.rb b/spec/lib/sbom/purl_type/converter_spec.rb index 2eb35c4d079..d0907bf253f 100644 --- a/spec/lib/sbom/purl_type/converter_spec.rb +++ b/spec/lib/sbom/purl_type/converter_spec.rb @@ -22,6 +22,7 @@ RSpec.describe Sbom::PurlType::Converter, feature_category: :dependency_manageme 'nuget' | 'nuget' 'pip' | 'pypi' 'pipenv' | 'pypi' + 'poetry' | 'pypi' 'setuptools' | 'pypi' 'Python (python-pkg)' | 'pypi' 'analyzer (gobinary)' | 'golang' diff --git a/yarn.lock b/yarn.lock index 2c0fe97b749..dda9f763eee 100644 --- a/yarn.lock +++ b/yarn.lock @@ -1913,14 +1913,14 @@ estree-walker "^2.0.2" picomatch "^2.3.1" -"@sentry-internal/tracing@7.75.0": - version "7.75.0" - resolved "https://registry.yarnpkg.com/@sentry-internal/tracing/-/tracing-7.75.0.tgz#0d6cb4d3ff4ea6dd456f64455b2d505d7eb27656" - integrity sha512-/j4opF/jB9j8qnSiQK75/lFLtkfqXS5/MoOKc2KWK/pOaf15W+6uJzGQ8jRBHLYd9dDg6AyqsF48Wqy561/mNg== +"@sentry-internal/tracing@7.75.1": + version "7.75.1" + resolved "https://registry.yarnpkg.com/@sentry-internal/tracing/-/tracing-7.75.1.tgz#978c5ec58a704c423a9b33a58ca3e3e6521725f8" + integrity sha512-nynV+7iVcF8k3CqhvI2K7iA8h4ovJhgYHKnXR8RDDevQOqNG2AEX9+hjCj9fZM4MhKHYFqf1od2oO9lTr38kwg== dependencies: - "@sentry/core" "7.75.0" - "@sentry/types" "7.75.0" - "@sentry/utils" "7.75.0" + "@sentry/core" "7.75.1" + "@sentry/types" "7.75.1" + "@sentry/utils" "7.75.1" "@sentry/core@5.30.0": version "5.30.0" @@ -1933,13 +1933,13 @@ "@sentry/utils" "5.30.0" tslib "^1.9.3" -"@sentry/core@7.75.0": - version "7.75.0" - resolved "https://registry.yarnpkg.com/@sentry/core/-/core-7.75.0.tgz#d5477faf9afdfbf45b4ff46b809729f14d4e1b80" - integrity sha512-vXg3cdJgwzP24oTS9zFCgLW4MgTkMZqXx+ESRq7gTD9qJTpcmAmYT+Ckmvebg8K6DBThV6+0v61r50na2+XdrA== +"@sentry/core@7.75.1": + version "7.75.1" + resolved "https://registry.yarnpkg.com/@sentry/core/-/core-7.75.1.tgz#f48cc424990ee4f31541e93f2c0277bfd5be9ed3" + integrity sha512-Kw4KyKBxbxbh8OKO0S11Tm0gWP+6AaXXYrsq3hp8H338l/wOmIzyckmCbUrc/XJeoRqaFLJbdcCrcUEDZUvsVQ== dependencies: - "@sentry/types" "7.75.0" - "@sentry/utils" "7.75.0" + "@sentry/types" "7.75.1" + "@sentry/utils" "7.75.1" "@sentry/hub@5.30.0": version "5.30.0" @@ -1959,25 +1959,25 @@ "@sentry/types" "5.30.0" tslib "^1.9.3" -"@sentry/replay@7.75.0": - version "7.75.0" - resolved "https://registry.yarnpkg.com/@sentry/replay/-/replay-7.75.0.tgz#0b1d9e9a6954ecc004597456f2c82e7630b8139c" - integrity sha512-TAAlj7JCMF6hFFL71RmPzVX89ltyPYFWR+t4SuWaBmU6HmTliI2eJvK+M36oE+N7s3CkyRVTaXXRe0YMwRMuZQ== +"@sentry/replay@7.75.1": + version "7.75.1" + resolved "https://registry.yarnpkg.com/@sentry/replay/-/replay-7.75.1.tgz#7790e80f7cb4dc856f5e72f70a51febd5898e04d" + integrity sha512-MKQTDWNYs9QXCJ+irGX5gu8Kxdk/Ds5puhILy8+DnCoXgXuPFRMGob1Sxt8qXmbQmcGeogsx221MNTselsRS6g== dependencies: - "@sentry-internal/tracing" "7.75.0" - "@sentry/core" "7.75.0" - "@sentry/types" "7.75.0" - "@sentry/utils" "7.75.0" + "@sentry-internal/tracing" "7.75.1" + "@sentry/core" "7.75.1" + "@sentry/types" "7.75.1" + "@sentry/utils" "7.75.1" "@sentry/types@5.30.0": version "5.30.0" resolved "https://registry.yarnpkg.com/@sentry/types/-/types-5.30.0.tgz#19709bbe12a1a0115bc790b8942917da5636f402" integrity sha512-R8xOqlSTZ+htqrfteCWU5Nk0CDN5ApUTvrlvBuiH1DyP6czDZ4ktbZB0hAgBlVcK0U+qpD3ag3Tqqpa5Q67rPw== -"@sentry/types@7.75.0": - version "7.75.0" - resolved "https://registry.yarnpkg.com/@sentry/types/-/types-7.75.0.tgz#e171f1680785a155cb828942af890ad4ee657ca3" - integrity sha512-xG8OLADxG7HpGhMxrF4v4tKq/v/gqmLsTZ858R51pz0xCWM8SK6ZSWOKudkAGBIpRjI6RUHMnkBtRAN2aKDOkQ== +"@sentry/types@7.75.1": + version "7.75.1" + resolved "https://registry.yarnpkg.com/@sentry/types/-/types-7.75.1.tgz#48b11336a0e70433d41bbe41c617dd339d4992ea" + integrity sha512-km+ygqgMDaFfTrbQwdhrptFqx0Oq15jZABqIoIpbaOCkCAMm+tyCqrFS8dTfaq5wpCktqWOy2qU/DOpppO99Cg== "@sentry/utils@5.30.0": version "5.30.0" @@ -1987,12 +1987,12 @@ "@sentry/types" "5.30.0" tslib "^1.9.3" -"@sentry/utils@7.75.0": - version "7.75.0" - resolved "https://registry.yarnpkg.com/@sentry/utils/-/utils-7.75.0.tgz#7a638c4c027ca2018518ee8d2eead1397cb97d66" - integrity sha512-UHWKeevhUNRp+mAWDbMVFOMgseoq8t/xFgdUywO/2PC14qZKRBH+0k1BKoNkp5sOzDT06ETj2w6wYoYhy6i+dA== +"@sentry/utils@7.75.1": + version "7.75.1" + resolved "https://registry.yarnpkg.com/@sentry/utils/-/utils-7.75.1.tgz#169040ba13ff4d4ecccb7b7aa23f84724d108b97" + integrity sha512-QzW2eRjY20epD//9/tQ0FTNwdAL6XZi+LyJNUQIeK3NMnc5NgHrgpxId87gmFq8cNx47utH1Blub8RuMbKqiwQ== dependencies: - "@sentry/types" "7.75.0" + "@sentry/types" "7.75.1" "@sinclair/typebox@^0.24.1": version "0.24.40" @@ -11915,16 +11915,16 @@ send@0.17.2: "@sentry/utils" "5.30.0" tslib "^1.9.3" -"sentrybrowser@npm:@sentry/browser@7.75.0": - version "7.75.0" - resolved "https://registry.yarnpkg.com/@sentry/browser/-/browser-7.75.0.tgz#7ea88f335c7bbaf3b5eecbf4e12590785abc0ee7" - integrity sha512-DXH/69vzp2j8xjydX+lrUYasrk7a1mpbXFGA9GtnII7shMCy55+QkVxpa6cLojYUaG2K/8yFDMcrP9N395LnWg== +"sentrybrowser@npm:@sentry/browser@7.75.1": + version "7.75.1" + resolved "https://registry.yarnpkg.com/@sentry/browser/-/browser-7.75.1.tgz#70422d26e2651443bcf15ea6bb5142774926c5ae" + integrity sha512-0+jPfPA5P9HVYYRQraDokGCY2NiMknSfz11dggClK4VmjvG+hOXiEyf73SFVwLFnv/hwrkWySjoIrVCX65xXQA== dependencies: - "@sentry-internal/tracing" "7.75.0" - "@sentry/core" "7.75.0" - "@sentry/replay" "7.75.0" - "@sentry/types" "7.75.0" - "@sentry/utils" "7.75.0" + "@sentry-internal/tracing" "7.75.1" + "@sentry/core" "7.75.1" + "@sentry/replay" "7.75.1" + "@sentry/types" "7.75.1" + "@sentry/utils" "7.75.1" serialize-javascript@^2.1.2: version "2.1.2" |