Add latest changes from gitlab-org/gitlab@master

5 files changed, 152 insertions, 148 deletions

diff --git a/doc/architecture/blueprints/organization/index.md b/doc/architecture/blueprints/organization/index.md
index be99211754d..c37a8260720 100644
--- a/doc/architecture/blueprints/organization/index.md
+++ b/doc/architecture/blueprints/organization/index.md
@@ -55,7 +55,7 @@ Due to urgency of delivering Organizations as a prerequisite for Cells, it is cu
 
 ## Proposal
 
-We create Organizations as a new lightweight entity, with just the features and workflows which it requires. We already have much of the functionality present in groups and projects, and groups themselves are essentially already the top-level entity. It is unlikely that we need to add significant features to Organizations outside of some key settings, as top-level groups can continue to serve this purpose at least on SaaS.
+We create Organizations as a new lightweight entity, with just the features and workflows which it requires. We already have much of the functionality present in groups and projects, and groups themselves are essentially already the top-level entity. It is unlikely that we need to add significant features to Organizations outside of some key settings, as top-level groups can continue to serve this purpose at least on SaaS. From an infrastructure perspective, cluster-wide shared data must be both minimal (small in volume) and infrequently written.
 
 ```mermaid
 graph TD
@@ -91,7 +91,7 @@ The Organization MVC will contain the following functionality:
 - Every instance will have a default organization. Initially, all users will be managed by this default Organization.
 - Organization Owner. The creation of an Organization appoints that user as the Organization Owner. Once established, the Organization Owner can appoint other Organization Owners.
 - Organization users. A user is managed by one Organization, but can be part of multiple Organizations. Users are able to navigate between the different Organizations they are part of.
-- Setup settings. Containing the Organization name, ID, description, README, and avatar. Settings are editable by the Organization Owner.
+- Setup settings. Containing the Organization name, ID, description, and avatar. Settings are editable by the Organization Owner.
 - Setup flow. Users are able to build an Organization on top of an existing top-level group. New users are able to create an Organization from scratch and to start building top-level groups from there.
 - Visibility. Options will be `public` and `private`. A nonuser of a specific Organization will not see private Organizations in the explore section. Visibility is editable by the Organization Owner.
 - Organization settings page with the added ability to remove an Organization. Deletion of the default Organization is prevented.
@@ -123,6 +123,10 @@ Non-users are external to the Organization and can only access the public resour
 
 Today only users, projects, namespaces and container images are considered routable entities which require global uniqueness on `https://gitlab.com/<path>/-/`. Initially, Organization routes will be [unscoped](../../../development/routing.md). Organizations will follow the path `https://gitlab.com/-/organizations/org-name/` as one of the design goals is that the addition of Organizations should not change existing group and project paths.
 
+### Impact of the Organization on Other Features
+
+We want a minimal amount of infrequently written tables in the shared database. If we have high write volume or large amounts of data in the shared database then this can become a single bottleneck for scaling and we lose the horizontal scalability objective of Cells.
+
 ## Iteration Plan
 
 The following iteration plan outlines how we intend to arrive at the Organization MVC. We are following the guidelines for [Experiment, Beta, and Generally Available features](../../../policy/experiment-beta-support.md).
@@ -142,7 +146,9 @@ In iteration 1, we introduce the concept of an Organization as a way to group to
 
 In iteration 2, an Organization MVC Experiment will be released. We will test the functionality with a select set of customers and improve the MVC based on these learnings. Users will be able to build an Organization on top of their existing top-level group.
 
-- The Organization has a description and a README.
+- The Organization has a description.
+- Organizations can be deleted.
+- Users can navigate between the different Organizations they are part of.
 
 ### Iteration 3: Organization MVC Beta (FY24Q4)
 
@@ -150,19 +156,24 @@ In iteration 3, the Organization MVC Beta will be released.
 
 - Multiple Organization Owners can be assigned.
 - Enterprise users can be added to an Organization.
+- Organization Owners can change the visibility of an organization between `public` and `private`. A nonuser of a specific Organization will not see private Organizations in the explore section.
 
 ### Iteration 4: Organization MVC GA (FY25Q1)
 
+In iteration 4, the Organization MVC will be rolled out.
+
 ### Post-MVC Iterations
 
 After the initial rollout of Organizations, the following functionality will be added to address customer needs relating to their implementation of GitLab:
 
 1. Internal visibility will be made available on Organizations that are part of GitLab.com.
+1. Projects can be created from the Organization-level Projects overview.
+1. Groups can be created from the Organization-level Groups overview.
 1. Move billing from top-level group to Organization.
 1. Audit events at the Organization level.
 1. Set merge request approval rules at the Organization level and cascade to all groups and projects.
 1. Security policies at the Organization level.
-1. Vulnerability reports at the Organization level.
+1. Vulnerability Report and Dependency List at the Organization level.
 1. Cascading Organization setting to enforce security scans.
 1. Scan result policies at the Organization level.
 1. Compliance frameworks.
diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md
index 3e54a78757a..5aca74acc56 100644
--- a/doc/development/database/batched_background_migrations.md
+++ b/doc/development/database/batched_background_migrations.md
@@ -260,6 +260,9 @@ Background migration classes for EE-only features that use job arguments should
 in the GitLab FOSS class. This is required to prevent job arguments validation from failing when
 migration is scheduled in GitLab FOSS context.
 
+You can use the [generator](#generator) to generate an EE-only migration scaffold by passing
+`--ee-only` flag when generating a new batched background migration.
+
 ## Queueing
 
 Queueing a batched background migration should be done in a post-deployment
diff --git a/doc/development/gems.md b/doc/development/gems.md
index 4a332a28f62..d709e4ab223 100644
--- a/doc/development/gems.md
+++ b/doc/development/gems.md
@@ -9,38 +9,25 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 GitLab uses Gems as a tool to improve code reusability and modularity
 in a monolithic codebase.
 
-Sometimes we create libraries within our codebase that we want to
-extract, either because their functionality is highly isolated,
-we want to use them in other applications
-ourselves, or we think it would benefit the wider community.
+Sometimes we extract libraries from our codebase because their functionality
+is highly isolated and we want to use them in other applications
+ourselves or we think it would benefit the wider community.
 Extracting code to a gem also means that we can be sure that the gem
 does not contain any hidden dependencies on our application code.
 
-## When to use Gems
-
 Gems should be used always when implementing functions that can be considered isolated,
-that are decoupled from the business logic of GitLab and can be developed separately. Consider the
-following examples where Gem logic could be placed:
+that are decoupled from the business logic of GitLab and can be developed separately.
 
-The best example where we can look for opportunities to introduce new gems
+The best place in Rails where we can look for opportunities to extract new gems
 is the [lib/](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/) folder.
 
 The **lib/** folder is a mix of code that is generic/universal, GitLab-specific, and tightly integrated with the rest of the codebase.
 
-If you cannot find a good place for your code in **lib/** you should strongly
-consider creating the new Gem [In the same repo](#in-the-same-repo).
+Ask yourself the question: **can this code also be used in a Ruby project other than the current project?**.
+If the answer is **Yes** you should strongly consider starting with creating a new Gem [in the same repo](#in-the-same-repo)
+and eventually evaluate whether to extract it as a separate repository.
 
-## In the same repo
-
-**Our GitLab Gems should be always put in `gems/` of GitLab monorepo.**
-
-That gives us the advantages of gems (modular code, quicker to run tests in development).
-and prevents complexity (coordinating changes across repos, new permissions, multiple projects, etc.).
-
-Gems stored in the same repo should be referenced in `Gemfile` with the `path:` syntax.
-They should not be published to RubyGems.
-
-### Advantages
+## Advantages of using Gems
 
 Using Gems can provide several benefits for code maintenance:
 
@@ -58,11 +45,116 @@ Using Gems can provide several benefits for code maintenance:
   Since the gem is packaged, not changed too often, it also allows us to run those tests less frequently improving
   CI testing time.
 
-### To Do
+## In the same repo
+
+**Our GitLab Gems should be always put in `gems/` of GitLab monorepo.**
+
+That gives us the advantages of gems (modular code, quicker to run tests in development).
+and prevents complexity (coordinating changes across repos, new permissions, multiple projects, etc.).
+
+Gems stored in the same repo should be referenced in `Gemfile` with the `path:` syntax.
+They should not be published to RubyGems.
+
+### Create and use a new Gem
+
+You can see example adding new Gem: [!121676](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121676).
+
+1. Create a new Ruby Gem in `gems/gitlab-<name-of-gem>` with `bundle gem gems/gitlab-<name-of-gem> --no-exe --no-coc --no-ext --no-mit`.
+1. Edit `gitlab-<name-of-gem>/README.md` to provide a simple description of the Gem.
+1. Edit `gitlab-<name-of-gem>/gitlab-<name-of-gem>.gemspec` and fill the details about the Gem as in the following example:
+
+   ```ruby
+   Gem::Specification.new do |spec|
+     spec.name = "gitlab-<name-of-gem>"
+     spec.version = Gitlab::NameOfGem::VERSION
+     spec.authors = ["group::tenant-scale"]
+     spec.email = ["engineering@gitlab.com"]
+
+     spec.summary = "GitLab's RSpec extensions"
+     spec.description = "A set of useful helpers to configure RSpec with various stubs and CI configs."
+     spec.homepage = "https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-<name-of-gem>"
+     spec.required_ruby_version = ">= 2.6.0"
+   end
+   ```
+
+1. Update `gems/gitlab-<name-of-gem>/.rubocop` with:
+
+   ```yaml
+   inherit_from:
+     - ../../.rubocop.yml
+
+   CodeReuse/ActiveRecord:
+     Enabled: false
 
-#### Desired use cases
+   AllCops:
+     TargetRubyVersion: 3.0
 
-The `gitlab-utils` is a Gem containing as of set of class that implement common intrisic functions
+   Naming/FileName:
+     Exclude:
+       - spec/**/*.rb
+   ```
+
+1. Configure CI for a newly added Gem:
+
+   - Add `gems/gitlab-<name-of-gem>/.gitlab-ci.yml`:
+
+     ```yaml
+     workflow:
+       rules:
+         - if: $CI_MERGE_REQUEST_ID
+
+     rspec:
+       image: "ruby:${RUBY_VERSION}"
+       cache:
+         key: gitlab-<name-of-gem>
+         paths:
+           - gitlab-<name-of-gem>/vendor/ruby
+       before_script:
+         - cd gems/gitlab-<name-of-gem>
+         - ruby -v                                   # Print out ruby version for debugging
+         - gem install bundler --no-document         # Bundler is not installed with the image
+         - bundle config set --local path 'vendor'   # Install dependencies into ./vendor/ruby
+         - bundle config set with 'development'
+         - bundle config set --local frozen 'true'   # Disallow Gemfile.lock changes on CI
+         - bundle config                             # Show bundler configuration
+         - bundle install -j $(nproc)
+       script:
+         - bundle exec rspec
+       parallel:
+         matrix:
+           - RUBY_VERSION: ["2.7", "3.0", "3.1", "3.2"]
+     ```
+
+   - To `.gitlab/ci/rules.gitlab-ci.yml` add:
+
+     ```yaml
+     .gems:rules:gitlab-<name-of-gem>:
+       rules:
+         - <<: *if-merge-request
+           changes: ["gems/gitlab-<name-of-gem>/**/*"]
+     ```
+
+   - To `.gitlab/ci/gitlab-gems.gitlab-ci.yml` add:
+
+     ```yaml
+     gems gitlab-<name-of-gem>:
+       extends:
+         - .gems:rules:gitlab-<name-of-gem>
+       needs: []
+       trigger:
+         include: gems/gitlab-<name-of-gem>/.gitlab-ci.yml
+         strategy: depend
+     ```
+
+1. Reference Gem in `Gemfile` with:
+
+   ```ruby
+   gem 'gitlab-<name-of-gem>', path: 'gems/gitlab-<name-of-gem>'
+   ```
+
+### Examples of Gem extractions
+
+The `gitlab-utils` is a Gem containing as of set of class that implement common intrinsic functions
 used by GitLab developers, like `strong_memoize` or `Gitlab::Utils.to_boolean`.
 
 The `gitlab-database-schema-migrations` is a potential Gem containing our extensions to Rails
@@ -72,7 +164,7 @@ or potentially be upstreamed.
 
 The `gitlab-database-load-balancing` similar to previous is a potential Gem to implement GitLab specific
 load balancing to Rails database handling. Since this is rather complex and highly specific code
-maintaing it's complexity in a isolated and well tested Gem would help with removing this complexity
+maintaining its complexity in a isolated and well tested Gem would help with removing this complexity
 from a big monolithic codebase.
 
 The `gitlab-flipper` is another potential Gem implementing all our custom extensions to support feature
@@ -81,18 +173,10 @@ usage, adding consistency checks and various helpers to track owners of feature
 not really part of GitLab business logic and could be used to better track our implementation
 of Flipper and possibly much easier change it to dogfood [GitLab Feature Flags](../operations/feature_flags.md).
 
-The `gitlab-ci-reports-parsers` is a potential Gem that could implement all various parsers for various formats.
-The parsed output would be transformed into objects that could then be used by GitLab the application
-to store it in the database. This functionality could be an additional Gem since it is isolated,
-rarely changed, and GitLab Rails only consumes the data.
-
-The same pattern could be applied to all other type of parsers, like security vulnerabilities, or any
-other complex structures that need to be transformed into a form that is consumed by GitLab Rails.
-
 The `gitlab-active_record` is a gem adding GitLab specific Active Record patches.
 It is very well desired for such to be managed separately to isolate complexity.
 
-#### Other potential use cases
+### Other potential use cases
 
 The `gitlab-ci-config` is a potential Gem containing all our CI code used to parse `.gitlab-ci.yml`.
 This code is today lightly interlocked with GitLab the application due to lack of proper abstractions.
@@ -101,103 +185,6 @@ with GitLab the application. The interface would for example define an adapter t
 Once we would have a `gitlab-ci-config` Gem it could be used within GitLab and outside of GitLab Rails
 and [GitLab CLI](https://gitlab.com/gitlab-org/cli).
 
-### Create and use a new Gem
-
-You can see example adding new Gem: [!121676](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121676).
-
-1. Create a new Ruby Gem in `gems/gitlab-<name-of-gem>` with `bundle gem gems/gitlab-<name-of-gem> --no-exe --no-coc --no-ext --no-mit`.
-1. Edit or remove `gitlab-<name-of-gem>/README.md` to provide a simple one paragraph description of the Gem.
-1. Edit `gitlab-<name-of-gem>/gitlab-<name-of-gem>.gemspec` and fill the details about the Gem as in the following example:
-
-    ```ruby
-    Gem::Specification.new do |spec|
-      spec.name = "gitlab-<name-of-gem>"
-      spec.version = Gitlab::NameOfGem::VERSION
-      spec.authors = ["group::tenant-scale"]
-      spec.email = ["engineering@gitlab.com"]
-
-      spec.summary = "GitLab's RSpec extensions"
-      spec.description = "A set of useful helpers to configure RSpec with various stubs and CI configs."
-      spec.homepage = "https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-<name-of-gem>"
-      spec.required_ruby_version = ">= 2.6.0"
-    end
-    ```
-
-1. Update `gems/gitlab-<name-of-gem>/.rubocop` with:
-
-    ```yaml
-    inherit_from:
-      - ../../.rubocop.yml
-
-    CodeReuse/ActiveRecord:
-      Enabled: false
-
-    AllCops:
-      TargetRubyVersion: 3.0
-
-    Naming/FileName:
-      Exclude:
-        - spec/**/*.rb
-    ```
-
-1. Configure CI for a newly added Gem:
-
-- Add `gems/gitlab-<name-of-gem>/.gitlab-ci.yml`:
-
-    ```yaml
-    workflow:
-      rules:
-        - if: $CI_MERGE_REQUEST_ID
-
-    rspec:
-      image: "ruby:${RUBY_VERSION}"
-      cache:
-        key: gitlab-<name-of-gem>
-        paths:
-          - gitlab-<name-of-gem>/vendor/ruby
-      before_script:
-        - cd gems/gitlab-<name-of-gem>
-        - ruby -v                                   # Print out ruby version for debugging
-        - gem install bundler --no-document         # Bundler is not installed with the image
-        - bundle config set --local path 'vendor'   # Install dependencies into ./vendor/ruby
-        - bundle config set with 'development'
-        - bundle config set --local frozen 'true'   # Disallow Gemfile.lock changes on CI
-        - bundle config                             # Show bundler configuration
-        - bundle install -j $(nproc)
-      script:
-        - bundle exec rspec
-      parallel:
-        matrix:
-          - RUBY_VERSION: ["2.7", "3.0", "3.1", "3.2"]
-    ```
-
-- To `.gitlab/ci/rules.gitlab-ci.yml` add:
-
-    ```yaml
-    .gems:rules:gitlab-<name-of-gem>:
-      rules:
-        - <<: *if-merge-request
-          changes: ["gems/gitlab-<name-of-gem>/**/*"]
-    ```
-
-- To `.gitlab/ci/gitlab-gems.gitlab-ci.yml` add:
-
-    ```yaml
-    gems gitlab-<name-of-gem>:
-      extends:
-        - .gems:rules:gitlab-<name-of-gem>
-      needs: []
-      trigger:
-        include: gems/gitlab-<name-of-gem>/.gitlab-ci.yml
-        strategy: depend
-    ```
-
-1. Reference Gem in `Gemfile` with:
-
-    ```ruby
-    gem 'gitlab-<name-of-gem>', path: 'gems/gitlab-<name-of-gem>'
-    ```
-
 ## In the external repo
 
 In general, we want to think carefully before doing this as there are
diff --git a/doc/user/group/compliance_frameworks.md b/doc/user/group/compliance_frameworks.md
index 47764b0c915..0ba329a8963 100644
--- a/doc/user/group/compliance_frameworks.md
+++ b/doc/user/group/compliance_frameworks.md
@@ -104,6 +104,10 @@ However, the compliance pipeline configuration can reference the `.gitlab-ci.yml
 - Jobs and variables defined in the compliance pipeline can't be changed by variables in the labeled project's
   `.gitlab-ci.yml` file.
 
+NOTE:
+Because of a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/414004), project pipelines must be included first at the top of compliance pipeline configuration
+to prevent projects overriding settings downstream.
+
 For more information, see:
 
 - [Example configuration](#example-configuration) for help configuring a compliance pipeline that runs jobs from
@@ -151,6 +155,13 @@ The following example `.compliance-gitlab-ci.yml` includes the `include` keyword
 configuration is also executed.
 
 ```yaml
+include:  # Execute individual project's configuration (if project contains .gitlab-ci.yml)
+  - project: '$CI_PROJECT_PATH'
+    file: '$CI_CONFIG_PATH'
+    ref: '$CI_COMMIT_SHA' # Must be defined or MR pipelines always use the use default branch
+    rules:
+      - if: $CI_PROJECT_PATH != "my-group/project-1" # Must be the hardcoded path to the project that hosts this configuration.
+
 # Allows compliance team to control the ordering and interweaving of stages/jobs.
 # Stages without jobs defined will remain hidden.
 stages:
@@ -210,13 +221,6 @@ audit trail:
     - echo "running $FOO"
   after_script:
     - "# No after scripts."
-
-include:  # Execute individual project's configuration (if project contains .gitlab-ci.yml)
-  - project: '$CI_PROJECT_PATH'
-    file: '$CI_CONFIG_PATH'
-    ref: '$CI_COMMIT_SHA' # Must be defined or MR pipelines always use the use default branch
-    rules:
-      - if: $CI_PROJECT_PATH != "my-group/project-1" # Must be the hardcoded path to the project that hosts this configuration.
 ```
 
 The `rules` configuration in the `include` definition avoids circular inclusion in case the compliance pipeline must be able to run in the host project itself.
diff --git a/doc/user/project/merge_requests/approvals/index.md b/doc/user/project/merge_requests/approvals/index.md
index 68d5665139a..2a574764954 100644
--- a/doc/user/project/merge_requests/approvals/index.md
+++ b/doc/user/project/merge_requests/approvals/index.md
@@ -121,11 +121,10 @@ Whenever an approval rule cannot be satisfied, the rule is displayed as **(!) Au
 - No eligible approvers (either groups or users) have been assigned to the approval rule.
 - The number of required approvals is more than the number of eligible approvers.
 
-These rules are automatically approved to unblock their respective merge requests,
-unless they were created through a security policy.
-
-Invalid approval rules created through a security policy are presented with **(!) Action Required**
-and are not automatically approved, blocking their respective merge requests.
+These rules are automatically approved to unblock their respective merge requests, unless they were
+created through a [scan result policy](../../../application_security/policies/scan-result-policies.md).
+Invalid approval rules created through a scan result policy are presented with
+**(!) Action Required** and are not automatically approved, blocking their respective merge requests.
 
 ## Related topics