Add latest changes from gitlab-org/gitlab@master

8 files changed, 102 insertions, 68 deletions

diff --git a/.gitlab/CODEOWNERS b/.gitlab/CODEOWNERS
index 6e866739b3f..5bd781f4277 100644
--- a/.gitlab/CODEOWNERS
+++ b/.gitlab/CODEOWNERS
@@ -1445,23 +1445,6 @@ ee/lib/ee/api/entities/project.rb
 /ee/app/views/shared/icons/_icon_audit_events_purple.svg
 /ee/app/views/shared/promotions/_promote_audit_events.html.haml
 /ee/app/workers/audit_events/audit_event_streaming_worker.rb
-/ee/config/events/1652263097_groups__audit_events__index_click_streams_tab.yml
-/ee/config/events/202108302307_admin_audit_logs_index_click_date_range_button.yml
-/ee/config/events/202108302307_groups__audit_events_controller_search_audit_event.yml
-/ee/config/events/202108302307_profiles_controller_search_audit_event.yml
-/ee/config/events/202108302307_projects__audit_events_controller_search_audit_event.yml
-/ee/config/events/202111041910_admin__audit_logs_controller_search_audit_event.yml
-/ee/config/metrics/counts_28d/20210216183930_g_compliance_audit_events_monthly.yml
-/ee/config/metrics/counts_28d/20210216183934_i_compliance_audit_events_monthly.yml
-/ee/config/metrics/counts_28d/20210216183942_a_compliance_audit_events_api_monthly.yml
-/ee/config/metrics/counts_28d/20211130085433_g_manage_compliance_audit_event_destinations.yml
-/ee/config/metrics/counts_7d/20210216183906_g_compliance_audit_events.yml
-/ee/config/metrics/counts_7d/20210216183908_i_compliance_audit_events.yml
-/ee/config/metrics/counts_7d/20210216183912_a_compliance_audit_events_api.yml
-/ee/config/metrics/counts_7d/20210216183928_g_compliance_audit_events_weekly.yml
-/ee/config/metrics/counts_7d/20210216183932_i_compliance_audit_events_weekly.yml
-/ee/config/metrics/counts_7d/20210216183940_a_compliance_audit_events_api_weekly.yml
-/ee/config/metrics/counts_all/20211130085433_g_manage_compliance_audit_event_destinations.yml
 /ee/lib/api/audit_events.rb
 /ee/lib/audit/
 /ee/lib/ee/api/entities/audit_event.rb
diff --git a/db/docs/activity_pub_releases_subscriptions.yml b/db/docs/activity_pub_releases_subscriptions.yml
index 8a27a51f9f3..d759aada5a9 100644
--- a/db/docs/activity_pub_releases_subscriptions.yml
+++ b/db/docs/activity_pub_releases_subscriptions.yml
@@ -7,4 +7,5 @@ feature_categories:
 description: Stores subscriptions from external users through ActivityPub for project
   releases
 introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132889
+milestone: '16.6'
 gitlab_schema: gitlab_main
diff --git a/db/docs/p_ci_job_annotations.yml b/db/docs/p_ci_job_annotations.yml
index 62a1b56abad..aae2ea67295 100644
--- a/db/docs/p_ci_job_annotations.yml
+++ b/db/docs/p_ci_job_annotations.yml
@@ -6,4 +6,5 @@ feature_categories:
 - build_artifacts
 description: Stores user provided annotations for jobs. Currently storing extra information for a given job feed by API.
 introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117319
+milestone: '16.1'
 gitlab_schema: gitlab_ci
diff --git a/doc/architecture/blueprints/secret_detection/index.md b/doc/architecture/blueprints/secret_detection/index.md
index fc97ca71d7f..81f0dd3fcb2 100644
--- a/doc/architecture/blueprints/secret_detection/index.md
+++ b/doc/architecture/blueprints/secret_detection/index.md
@@ -26,28 +26,22 @@ job logs, and project management features such as issues, epics, and MRs.
 
 ### Goals
 
-- Support asynchronous secret detection for the following scan targets:
-  - push events
-  - issuable creation
-  - issuable updates
-  - issuable comments
+- Support platform-wide detection of tokens to avoid secret leaks
+- Prevent exposure by rejecting detected secrets
+- Provide scalable means of detection without harming end user experience
 
-### Non-Goals
+See [target types](#target-types) for scan target priorities.
 
-The current proposal is limited to asynchronous detection and alerting only.
+### Non-Goals
 
-**Blocking** secrets on push events is high-risk to a critical path and
-would require extensive performance profiling before implementing. See
-[a recent example](https://gitlab.com/gitlab-org/gitlab/-/issues/246819#note_1164411983)
-of a customer incident where this was attempted.
+Initial proposal is limited to detection and alerting across plaform, with rejection only
+during preceive Git interactions.
 
 Secret revocation and rotation is also beyond the scope of this new capability.
 
 Scanned object types beyond the scope of this MVC include:
 
-- Media types (JPEGs, PDFs,...)
-- Snippets
-- Wikis
+See [target types](#target-types) for scan target priorities.
 
 #### Management UI
 
@@ -69,7 +63,13 @@ which remain focused on active detection.
 
 ## Proposal
 
-To achieve scalable secret detection for a variety of domain objects a dedicated
+The first iteration of the experimental capability will feature a blocking
+pre-receive hook implemented within the Rails application. This iteration
+will be released in an experimental state to select users and provide
+opportunity for the team to profile the capability before considering extraction
+into a dedicated service.
+
+In the future state, to achieve scalable secret detection for a variety of domain objects a dedicated
 scanning service must be created and deployed alongside the GitLab distribution.
 This is referred to as the `SecretScanningService`.
 
@@ -94,10 +94,10 @@ as self-managed instances.
 The critical paths as outlined under [goals above](#goals) cover two major object
 types: Git blobs (corresponding to push events) and arbitrary text blobs.
 
-The detection flow for push events relies on subscribing to the PostReceive hook
-to enqueue Sidekiq requests to the `SecretScanningService`. The `SecretScanningService`
-service fetches enqueued refs, queries Gitaly for the ref blob contents, scans
-the commit contents, and notifies the Rails application when a secret is detected.
+The detection flow for push events relies on subscribing to the PreReceive hook
+to scan commit data using the [PushCheck interface](https://gitlab.com/gitlab-org/gitlab/blob/3f1653f5706cd0e7bbd60ed7155010c0a32c681d/lib/gitlab/checks/push_check.rb). This `SecretScanningService`
+service fetches the specified blob contents from Gitaly, scans
+the commit contents, and rejects the push when a secret is detected.
 See [Push event detection flow](#push-event-detection-flow) for sequence.
 
 The detection flow for arbitrary text blobs, such as issue comments, relies on
@@ -112,13 +112,33 @@ storage. See discussion [in this issue](https://gitlab.com/groups/gitlab-org/-/e
 around scanning during streaming and the added complexity in buffering lookbacks
 for arbitrary trace chunks.
 
-In any case of detection, the Rails application manually creates a vulnerability
+In the case of a push detection, the commit is rejected and error returned to the end user.
+In any other case of detection, the Rails application manually creates a vulnerability
 using the `Vulnerabilities::ManuallyCreateService` to surface the finding in the
 existing Vulnerability Management UI.
 
 See [technical discovery](https://gitlab.com/gitlab-org/gitlab/-/issues/376716)
 for further background exploration.
 
+### Target types
+
+Target object types refer to the scanning targets prioritized for detection of leaked secrets.
+
+In order of priority this includes:
+
+1. non-binary Git blobs
+1. job logs
+1. issuable creation (issues, MRs, epics)
+1. issuable updates (issues, MRs, epics)
+1. issuable comments (issues, MRs, epics)
+
+Targets out of scope for the initial phases include:
+
+- Media types (JPEGs, PDFs,...)
+- Snippets
+- Wikis
+- Container images
+
 ### Token types
 
 The existing Secret Detection configuration covers ~100 rules across a variety
@@ -140,9 +160,10 @@ for all secret scanning in pipeline contexts. By using its `--no-git` configurat
 we can scan arbitrary text blobs outside of a repository context and continue to
 utilize it for non-pipeline scanning.
 
-Given our existing familiarity with the tool and its extensibility, it should
-remain our engine of choice. Changes to the detection engine are out of scope
-unless benchmarking unveils performance concerns.
+In the case of prereceive detection, we rely on a combination of keyword/substring matches
+for prefiltering and `re2` for regex detections. See [spike issue](https://gitlab.com/gitlab-org/gitlab/-/issues/423832) for initial benchmarks
+
+Changes to the detection engine are out of scope until benchmarking unveils performance concerns.
 
 Notable alternatives include high-performance regex engines such as [hyperscan](https://github.com/intel/hyperscan) or it's portable fork [vectorscan](https://github.com/VectorCamp/vectorscan).
 
@@ -167,37 +188,42 @@ for past discussion around scaling approaches.
 sequenceDiagram
     autonumber
     actor User
-    User->>+Workhorse: git push
+    User->>+Workhorse: git push with-secret
     Workhorse->>+Gitaly: tcp
-    Gitaly->>+Rails: grpc
-    Sidekiq->>+Rails: poll job
-    Rails->>-Sidekiq: PostReceive worker
-    Sidekiq-->>+Sidekiq: enqueue PostReceiveSecretScanWorker
-
-    Sidekiq->>+Rails: poll job
-    loop PostReceiveSecretScanWorker
-      Rails->>-Sidekiq: PostReceiveSecretScanWorker
-      Sidekiq->>+SecretScanningSvc: ScanBlob(ref)
-      SecretScanningSvc->>+Sidekiq: accepted
-      Note right of SecretScanningSvc: Scanning job enqueued
-      Sidekiq-->>+Rails: done
-      SecretScanningSvc->>+Gitaly: retrieve blob
-      SecretScanningSvc->>+SecretScanningSvc: scan blob
-      SecretScanningSvc->>+Rails: secret found
-    end
+    Gitaly->>+Rails: PreReceive
+    Rails->>-Gitaly: ListAllBlobs
+    Gitaly->>-Rails: ListAllBlobsResponse
+
+    Rails->>+GitLabSecretDetection: Scan(blob)
+    GitLabSecretDetection->>-Rails: found
+
+    Rails->>User: rejected: secret found
+
+    User->>+Workhorse: git push without-secret
+    Workhorse->>+Gitaly: tcp
+    Gitaly->>+Rails: PreReceive
+    Rails->>-Gitaly: ListAllBlobs
+    Gitaly->>-Rails: ListAllBlobsResponse
+
+    Rails->>+GitLabSecretDetection: Scan(blob)
+    GitLabSecretDetection->>-Rails: not_found
+
+    Rails->>User: OK
 ```
 
 ## Iterations
 
 - ✓ Define [requirements for detection coverage and actions](https://gitlab.com/gitlab-org/gitlab/-/issues/376716)
-- ✓ Implement [Clientside detection of GitLab tokens in comments/issues](https://gitlab.com/gitlab-org/gitlab/-/issues/368434)
-- PoC of secret scanning service
-  - Benchmarking of issuables, comments, job logs and blobs to gain confidence that the total costs will be viable
-  - Capacity planning for addition of service component to Reference Architectures headroom
-  - Service capabilities
+- ✓ Implement [Browser-based detection of GitLab tokens in comments/issues](https://gitlab.com/gitlab-org/gitlab/-/issues/368434)
+- ✓ [PoC of secret scanning service](https://gitlab.com/gitlab-org/secure/pocs/secret-detection-go-poc/)
+- ✓ [PoC of secret scanning gem](https://gitlab.com/gitlab-org/gitlab/-/issues/426823)
+- [Pre Production Performance Profiling for pre-receive PoCs](https://gitlab.com/gitlab-org/gitlab/-/issues/428499)
+  - Profiling service capabilities
+    - ✓ [Benchmarking regex performance between Ruby and Go approaches](https://gitlab.com/gitlab-org/gitlab/-/issues/423832)
     - gRPC commit retrieval from Gitaly
-    - blob scanning
+    - transfer latency, CPU, and memory footprint
 - Implementation of secret scanning service MVC (targeting individual commits)
+- Capacity planning for addition of service component to Reference Architectures headroom
 - Security and readiness review
 - Deployment and monitoring
 - Implementation of secret scanning service MVC (targeting arbitrary text blobs)
diff --git a/doc/user/application_security/vulnerabilities/index.md b/doc/user/application_security/vulnerabilities/index.md
index 34c57292767..326ee48eb49 100644
--- a/doc/user/application_security/vulnerabilities/index.md
+++ b/doc/user/application_security/vulnerabilities/index.md
@@ -104,7 +104,13 @@ When dismissing a vulnerability, one of the following reasons must be chosen to
 - **Used in tests**: The finding is not a vulnerability because it is part of a test or is test data.
 - **Not applicable**: The vulnerability is known, and has not been remediated or mitigated, but is considered to be in a part of the application that will not be updated.
 
-## Change status of a vulnerability
+## Change the status of a vulnerability
+
+> In GitLab 16.4 the ability for `Developers` to change the status of a vulnerability (`admin_vulnerability`) was [deprecated](../../../update/deprecations.md#deprecate-change-vulnerability-status-from-the-developer-role). The `admin_vulnerability` permission will be removed, by default, from all `Developer` roles in GitLab 17.0.
+
+Prerequisites:
+
+- You must have at least the Developer role for the project.
 
 To change a vulnerability's status from its Vulnerability Page:
 
diff --git a/doc/user/permissions.md b/doc/user/permissions.md
index a83ce6a56c6..3a08281f69e 100644
--- a/doc/user/permissions.md
+++ b/doc/user/permissions.md
@@ -201,10 +201,10 @@ The following table lists project permissions available for each role:
 | [Requirements Management](project/requirements/index.md):<br>Import / export                                                                                                         |          | ✓        | ✓         | ✓          | ✓        |
 | [Security dashboard](application_security/security_dashboard/index.md):<br>Create issue from vulnerability finding                                                                   |          |          | ✓         | ✓          | ✓        |
 | [Security dashboard](application_security/security_dashboard/index.md):<br>Create vulnerability from vulnerability finding                                                           |          |          | ✓         | ✓          | ✓        |
-| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability                                                                                     |          |          | ✓         | ✓          | ✓        |
-| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability finding                                                                             |          |          | ✓         | ✓          | ✓        |
-| [Security dashboard](application_security/security_dashboard/index.md):<br>Resolve vulnerability                                                                                     |          |          | ✓         | ✓          | ✓        |
-| [Security dashboard](application_security/security_dashboard/index.md):<br>Revert vulnerability to detected state                                                                    |          |          | ✓         | ✓          | ✓        |
+| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability                                                                                     |          |          | ✓ (24)        | ✓          | ✓        |
+| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability finding                                                                             |          |          | ✓         | ✓ (24)          | ✓        |
+| [Security dashboard](application_security/security_dashboard/index.md):<br>Resolve vulnerability                                                                                     |          |          | ✓ (24)         | ✓          | ✓        |
+| [Security dashboard](application_security/security_dashboard/index.md):<br>Revert vulnerability to detected state                                                                    |          |          | ✓ (24)         | ✓          | ✓        |
 | [Security dashboard](application_security/security_dashboard/index.md):<br>Use security dashboard                                                                                    |          |          | ✓         | ✓          | ✓        |
 | [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability                                                                                        |          |          | ✓         | ✓          | ✓        |
 | [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability findings in [dependency list](application_security/dependency_list/index.md)           |          |          | ✓         | ✓          | ✓        |
@@ -249,6 +249,7 @@ The following table lists project permissions available for each role:
 21. Authors of tasks can delete them even if they don't have the Owner role, but they have to have at least the Guest role for the project.
 22. You must have permission to [view the epic](group/epics/manage_epics.md#who-can-view-an-epic).
 23. In GitLab 15.9 and later, users with the Guest role and an Ultimate license can view private repository content if an administrator (on self-managed) or group owner (on GitLab.com) gives those users permission. The administrator or group owner can create a [custom role](custom_roles.md) through the API and assign that role to the users.
+24. In GitLab 16.4 the ability for `Developers` to change the status of a vulnerability (`admin_vulnerability`) was [deprecated](../update/deprecations.md#deprecate-change-vulnerability-status-from-the-developer-role). The `admin_vulnerability` permission will be removed, by default, from all `Developer` roles in GitLab 17.0.
 
 <!-- markdownlint-enable MD029 -->
 
diff --git a/locale/gitlab.pot b/locale/gitlab.pot
index 9dc87be9af6..3626a09e83a 100644
--- a/locale/gitlab.pot
+++ b/locale/gitlab.pot
@@ -56920,6 +56920,9 @@ msgstr ""
 msgid "is not in the group enforcing Group Managed Account"
 msgstr ""
 
+msgid "is not linked to a SAML account or has an inactive SCIM identity. For information on how to resolve this error, see the %{troubleshoot_link_start}troubleshooting SCIM documentation%{troubleshoot_link_end}."
+msgstr ""
+
 msgid "is not one of"
 msgstr ""
 
diff --git a/qa/qa/specs/features/api/1_manage/migration/gitlab_migration_large_project_spec.rb b/qa/qa/specs/features/api/1_manage/migration/gitlab_migration_large_project_spec.rb
index be9c1aab3ea..6ea2047fed7 100644
--- a/qa/qa/specs/features/api/1_manage/migration/gitlab_migration_large_project_spec.rb
+++ b/qa/qa/specs/features/api/1_manage/migration/gitlab_migration_large_project_spec.rb
@@ -48,6 +48,19 @@ module QA
         create(:sandbox, api_client: admin_api_client, path: "qa-sandbox-#{SecureRandom.hex(4)}")
       end
 
+      let!(:api_client) do
+        Runtime::API::Client.new(
+          user: user,
+          is_new_session: false,
+          personal_access_token: Resource::PersonalAccessToken.fabricate_via_api! do |pat|
+            pat.user = user
+            # importing very large project can take multiple days
+            # token must not expire while we still poll for import result
+            pat.expires_at = (Time.now.to_date + 5)
+          end.token
+        )
+      end
+
       # Source objects
       #
       let(:source_project) { source_group.projects(auto_paginate: true).find { |project| project.name == gitlab_source_project }.reload! }