Add latest changes from gitlab-org/gitlab@master

7 files changed, 297 insertions, 50 deletions

diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md
index f1f02b606f5..c7df8249ae4 100644
--- a/doc/administration/operations/puma.md
+++ b/doc/administration/operations/puma.md
@@ -113,7 +113,7 @@ is used when Puma is enabled.
 NOTE:
 Unlike Unicorn, the `puma['worker_timeout']` setting does not set the maximum request duration.
 
-To change the worker timeout:
+To change the worker timeout to 600 seconds:
 
 1. Edit `/etc/gitlab/gitlab.rb`:
 
diff --git a/doc/architecture/blueprints/object_storage/index.md b/doc/architecture/blueprints/object_storage/index.md
new file mode 100644
index 00000000000..a79374d60bd
--- /dev/null
+++ b/doc/architecture/blueprints/object_storage/index.md
@@ -0,0 +1,220 @@
+---
+stage: none
+group: unassigned
+comments: false
+description: 'Object storage: direct_upload consolidation - architecture blueprint.'
+---
+
+# Object storage: `direct_upload` consolidation
+
+## Abstract
+
+GitLab stores three classes of user data: database records, Git
+repositories, and user-uploaded files (which are referred to as
+file storage throughout the blueprint).
+
+The user and contributor experience for our file
+storage has room for significant improvement:
+
+- Initial GitLab setup experience requires creation and setup of 13
+  buckets, instead of just 1.
+- Features using file storage require contributors to think about both local
+  storage and object storage, which leads to friction and
+  complexity. This often results in broken features and security issues.
+- Contributors who work on file storage often also have to write code
+  for Workhorse, Omnibus, and cloud native GitLab (CNG).
+
+## Problem definition
+
+Object storage is a fundamental component of GitLab, providing the
+underlying implementation for shared, distributed, highly-available
+(HA) file storage.
+
+Over time, we have built support for object storage across the
+application, solving specific problems in a [multitude of
+iterations](https://about.gitlab.com/company/team/structure/working-groups/object-storage/#company-efforts-on-uploads). This
+has led to increased complexity across the board, from development
+(new features and bug fixes) to installation:
+
+- New GitLab installations require the creation and configuration of
+  several object storage buckets instead of just one, as each group of
+  features requires its own. This has an impact on the installation
+  experience and new feature adoption, and takes us further away from
+  boring solutions.
+- The release of cloud native GitLab required the removal of NFS
+  shared storage and the development of direct upload, a feature that
+  was expanded, milestone after milestone, to several type of uploads,
+  but never enabled globally.
+- Today, GitLab supports both local storage and object storage. Local
+  storage only works on single box installations or with a NFS, which
+  [we no longer recommend](../../../administration/nfs.md) to our
+  users and is no longer in use on GitLab.com.
+- Understanding all the moving parts and the flow is extremely
+  complicated: we have CarrierWave, Fog, Golang S3/Azure SDKs, all
+  being used, and that complicates testing as well.
+- Fog and CarrierWave are not maintained to the level of the native
+  SDKs (for example, AWS S3 SDK), so we have to maintain or monkey
+  patch those tools to support requested customer features
+  (for example, [issue #242245](https://gitlab.com/gitlab-org/gitlab/-/issues/242245))
+  that would normally be "free".
+- In many cases, we copy around object storage files needlessly
+  (for example, [issue #285597](https://gitlab.com/gitlab-org/gitlab/-/issues/285597)).
+  Large files (LFS, packages, and so on) are slow to finalize or don't work
+  at all as a result.
+
+## Improvements over the current situation
+
+The following is a brief description of the main directions we can take to
+remove the pain points affecting our object storage implementation.
+
+This is also available as [a YouTube
+video](https://youtu.be/X9V_w8hsM8E) recorded for the [Object Storage
+Working
+Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/).
+
+### Simplify GitLab architecture by shipping MinIO
+
+In the beginning, object storage support was a Premium feature, not
+part of our CE distribution. Because of that, we had to support both
+local storage and object storage.
+
+With local storage, there is the assumption of a shared storage
+between components. This can be achieved by having a single box
+installation, without HA, or with a NFS, which [we no longer
+recommend](../../../administration/nfs.md).
+
+We have a testing gap on object storage. It also requires Workhorse
+and MinIO, which are not present in our pipelines, so too much is
+replaced by a mock implementation. Furthermore, the presence of a
+shared disk, both in CI and in local development, often hides broken
+implementations until we deploy on an HA environment.
+
+Shipping MinIO as part of the product will reduce the differences
+between a cloud and a local installation, standardizing our file
+storage on a single technology.
+
+The removal of local disk operations will reduce the complexity of
+development as well as mitigate several security attack vectors as
+we no longer write user-provided data on the local storage.
+
+It will also reduce human errors as we will always run a local object
+storage in development mode and any local file disk access should
+raise a red flag during the merge request review.
+
+This effort is described in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6099).
+
+### Enable direct upload by default on every upload
+
+Because every group of features requires its own bucket, we don't have
+direct upload enabled everywhere. Contributing a new upload requires
+coding it in both Ruby on Rails and Go.
+
+Implementing a new feature that does not yet have a dedicated bucket
+requires the developer to also create a merge request in Omnibus
+and CNG, as well as coordinate with SREs to configure the new bucket
+for our own environments.
+
+This also slows down feature adoptions, because our users need to
+reconfigure GitLab and prepare a new bucket in their
+infrastructure. It also makes the initial installation more complex
+feature after feature.
+
+Implementing a direct upload by default, with a
+[consolidated object storage configuration](../../../administration/object_storage.md#consolidated-object-storage-configuration)
+will reduce the number of merge requests needed to ship a new feature
+from four to only one. It will also remove the need for SRE
+intervention as the bucket will always be the same.
+
+This will simplify our development and review processes, as well as
+the GitLab configuration file. And every user will immediately have
+access to new features without infrastructure chores.
+
+### Simplify object storage code
+
+Our implementation is built on top of a 3rd-party framework where
+every object storage client is a 3rd-party library. Unfortunately some
+of them are unmaintained. [We have customers who cannot push 5GB Git
+LFS objects](https://gitlab.com/gitlab-org/gitlab/-/issues/216442),
+but with such a vital feature implemented in 3rd-party libraries we
+are slowed down in fixing it, and we also rely on external maintainers
+to merge and release fixes.
+
+Before the introduction of direct upload, using the
+[CarrierWave](https://github.com/carrierwaveuploader/carrierwave)
+library, _"a gem that provides a simple and extremely flexible way to
+upload files from Ruby applications."_, was the boring solution.
+However this is no longer our use-case, as we upload files from
+Workhorse, and we had to [patch CarrierWave's
+internals](https://gitlab.com/gitlab-org/gitlab/-/issues/285597#note_452696638)
+to support direct upload.
+
+A brief proposal covering CarrierWave removal and a new streamlined
+internal upload API is described
+[in this issue comment](https://gitlab.com/gitlab-org/gitlab/-/issues/213288#note_325358026).
+
+Ideally, we wouldn't need to duplicate object storage clients in Go
+and Ruby. By removing CarrierWave, we can make use of the officially
+supported native clients when the provider S3 compatibility level is
+not sufficient.
+
+## Iterations
+
+In this section we list some possible iterations. This is not
+intended to be the final roadmap, but is a conversation started for the
+Object Storage Working Group.
+
+1. Create a new catchall bucket and a unified internal API for
+   authorization without CarrierWave.
+1. Ship MinIO with Omnibus (CNG images already include it).
+1. Expand GitLab-QA to cover all the supported configurations.
+1. Deprecate local disk access.
+1. Deprecate configurations with multiple buckets.
+1. Implement a bucket-to-bucket migration.
+1. Migrate the current CarrierWave uploads to the new implementation.
+1. On the next major release: Remove support for local disk access and
+   configurations with multiple buckets.
+
+### Benefits of the current iteration plan
+
+The current plan is designed to provide tangible benefits from the
+first step.
+
+With the introduction of the catchall bucket, every upload currently
+not subject to direct upload will get its benefits, and new features
+could be shipped with a single merge request.
+
+Shipping MinIO with Omnibus will allow us to default new installations
+to object storage, and Omnibus could take care of creating
+buckets. This will simplify HA installation outside of Kubernetes.
+
+Then we can migrate each CarrierWave uploader to the new
+implementation, up to a point where GitLab installation will only
+require one bucket.
+
+## Additional reading materials
+
+- [Uploads development documentation: The problem description](../../../development/uploads.md#the-problem-description).
+- [Speed up the monolith, building a smart reverse proxy in Go](https://archive.fosdem.org/2020/schedule/event/speedupmonolith/): a presentation explaining a bit of workhorse history and the challenge we faced in releasing the first cloud-native installation.
+- [Object Storage improvements epic](https://gitlab.com/groups/gitlab-org/-/epics/483).
+- We are moving to GraphQL API, but [we do not support direct upload](https://gitlab.com/gitlab-org/gitlab/-/issues/280819).
+
+## Who
+
+Proposal:
+
+<!-- vale gitlab.Spelling = NO -->
+
+| Role                           | Who                     |
+|--------------------------------|-------------------------|
+| Author                         | Alessio Caiazza         |
+| Architecture Evolution Coach   | Gerardo Lopez-Fernandez |
+| Engineering Leader             | Marin Jankovski         |
+| Domain Expert / Object storage | Stan Hu                 |
+| Domain Expert / Security       | Joern Schneeweisz       |
+
+DRIs:
+
+The DRI for this blueprint is the [Object Storage Working
+Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/).
+
+<!-- vale gitlab.Spelling = YES -->
diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md
index 87d7b9f9e30..52c3e737fa4 100644
--- a/doc/ci/environments/index.md
+++ b/doc/ci/environments/index.md
@@ -878,11 +878,6 @@ To ensure the `action: stop` can always run when needed, you can:
 
 > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21182) in GitLab 14.4.
 
-FLAG:
-On self-managed GitLab, by default this bug fix is available. To hide the bug fix per project,
-ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `surface_environment_creation_failure`.
-On GitLab.com, this bug fix is available.
-
 If your project is configured to [create a dynamic environment](#create-a-dynamic-environment),
 you might encounter this error because the dynamically generated parameter can't be used for creating an environment.
 
diff --git a/doc/ci/pipelines/merge_trains.md b/doc/ci/pipelines/merge_trains.md
index 6074909a887..31e9d382534 100644
--- a/doc/ci/pipelines/merge_trains.md
+++ b/doc/ci/pipelines/merge_trains.md
@@ -35,7 +35,8 @@ If the pipeline for the merge request at the front of the train completes succes
 the changes are merged into the target branch, and the other pipelines continue to
 run.
 
-To add a merge request to a merge train, you need [permissions](../../user/permissions.md) to push to the target branch.
+To add a merge request to a merge train, you need [permissions](../../user/permissions.md) to merge or push to the 
+target branch.
 
 Each merge train can run a maximum of **twenty** pipelines in parallel.
 If more than twenty merge requests are added to the merge train, the merge requests
diff --git a/doc/ci/pipelines/parent_child_pipelines.md b/doc/ci/pipelines/parent_child_pipelines.md
index 64f4160c963..5b8729757cc 100644
--- a/doc/ci/pipelines/parent_child_pipelines.md
+++ b/doc/ci/pipelines/parent_child_pipelines.md
@@ -31,10 +31,9 @@ set of concurrently running child pipelines, but within the same project:
 - Child pipelines still execute each of their jobs according to a stage sequence, but
   would be free to continue forward through their stages without waiting for unrelated
   jobs in the parent pipeline to finish.
-- The configuration is split up into smaller child pipeline configurations, which are
+- The configuration is split up into smaller child pipeline configurations. Each child pipeline contains only relevant steps which are
   easier to understand. This reduces the cognitive load to understand the overall configuration.
 - Imports are done at the child pipeline level, reducing the likelihood of collisions.
-- Each pipeline has only relevant steps, making it easier to understand what's going on.
 
 Child pipelines work well with other GitLab CI/CD features:
 
diff --git a/doc/update/index.md b/doc/update/index.md
index ce7e594bd1f..bd90da0eb36 100644
--- a/doc/update/index.md
+++ b/doc/update/index.md
@@ -82,10 +82,10 @@ See the guide to [plan your GitLab upgrade](plan_your_upgrade.md).
 ## Checking for background migrations before upgrading
 
 Certain releases may require different migrations to be
-finished before you update to the newer version. 
+finished before you update to the newer version.
 
 [Batched migrations](#batched-background-migrations) are a migration type available in GitLab 14.0 and later.
-Background migrations and batched migrations not the same, so you should check that both are 
+Background migrations and batched migrations not the same, so you should check that both are
 complete before updating.
 
 Decrease the time required to complete these migrations by increasing the number of
@@ -362,8 +362,10 @@ or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [follo
 
 ### 14.4.0
 
-Git 2.33.x and later is required. We recommend you use the
-[Git version provided by Gitaly](../install/installation.md#git).
+- Git 2.33.x and later is required. We recommend you use the
+  [Git version provided by Gitaly](../install/installation.md#git).
+
+- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
 
 ### 14.3.0
 
@@ -390,6 +392,8 @@ for how to proceed.
   sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production
   ```
 
+- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
+
 ### 14.2.0
 
 - [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases).
@@ -416,15 +420,20 @@ for how to proceed.
   sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production
   ```
 
+- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
+
 ### 14.1.0
 
 - [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases)
   but can upgrade to 14.1.Z.
-- It is not required for instances already running 14.0.5 (or higher) to stop at 14.1.Z.
+
+  It is not required for instances already running 14.0.5 (or higher) to stop at 14.1.Z.
   14.1 is included on the upgrade path for the broadest compatibility
   with self-managed installations, and ensure 14.0.0-14.0.4 installations do not
   encounter issues with [batched background migrations](#batched-background-migrations).
 
+- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
+
 ### 14.0.0
 
 - Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances.
@@ -454,6 +463,8 @@ for how to proceed.
   You should instead follow a [supported upgrade path](#upgrade-paths).
 - The support of PostgreSQL 11 [has been dropped](../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0.
 
+- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
+
 #### Upgrading to later 14.Y releases
 
 - Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later,
@@ -464,48 +475,60 @@ for how to proceed.
   1. [Batched background migrations need to finish](#batched-background-migrations)
      before you update to a later version [and may take longer than usual](#1400).
 
+### 13.12.0
+
+See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
+
 ### 13.11.0
 
-Git 2.31.x and later is required. We recommend you use the
-[Git version provided by Gitaly](../install/installation.md#git).
+- Git 2.31.x and later is required. We recommend you use the
+  [Git version provided by Gitaly](../install/installation.md#git).
+
+- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
+
+### 13.10.0
+
+See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
 
 ### 13.9.0
 
-We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160)
-that prevents upgrades to GitLab 13.9.0, 13.9.1, 13.9.2, and 13.9.3 when following the zero-downtime steps. It is necessary
-to perform the following additional steps for the zero-downtime upgrade:
+- We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160)
+  that prevents upgrades to GitLab 13.9.0, 13.9.1, 13.9.2, and 13.9.3 when following the zero-downtime steps. It is necessary
+  to perform the following additional steps for the zero-downtime upgrade:
 
-1. Before running the final `sudo gitlab-rake db:migrate` command on the deploy node,
-   execute the following queries using the PostgreSQL console (or `sudo gitlab-psql`)
-   to drop the problematic triggers:
+  1. Before running the final `sudo gitlab-rake db:migrate` command on the deploy node,
+     execute the following queries using the PostgreSQL console (or `sudo gitlab-psql`)
+     to drop the problematic triggers:
 
-   ```sql
-   drop trigger trigger_e40a6f1858e6 on application_settings;
-   drop trigger trigger_0d588df444c8 on application_settings;
-   drop trigger trigger_1572cbc9a15f on application_settings;
-   drop trigger trigger_22a39c5c25f3 on application_settings;
-   ```
+     ```sql
+     drop trigger trigger_e40a6f1858e6 on application_settings;
+     drop trigger trigger_0d588df444c8 on application_settings;
+     drop trigger trigger_1572cbc9a15f on application_settings;
+     drop trigger trigger_22a39c5c25f3 on application_settings;
+     ```
 
-1. Run the final migrations:
+  1. Run the final migrations:
 
-   ```shell
-   sudo gitlab-rake db:migrate
-   ```
+     ```shell
+     sudo gitlab-rake db:migrate
+     ```
 
-If you have already run the final `sudo gitlab-rake db:migrate` command on the deploy node and have
-encountered the [column rename issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160), you
-see the following error:
+  If you have already run the final `sudo gitlab-rake db:migrate` command on the deploy node and have
+  encountered the [column rename issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160), you
+  see the following error:
 
-```shell
--- remove_column(:application_settings, :asset_proxy_whitelist)
-rake aborted!
-StandardError: An error has occurred, all later migrations canceled:
-PG::DependentObjectsStillExist: ERROR: cannot drop column asset_proxy_whitelist of table application_settings because other objects depend on it
-DETAIL: trigger trigger_0d588df444c8 on table application_settings depends on column asset_proxy_whitelist of table application_settings
-```
+  ```shell
+  -- remove_column(:application_settings, :asset_proxy_whitelist)
+  rake aborted!
+  StandardError: An error has occurred, all later migrations canceled:
+  PG::DependentObjectsStillExist: ERROR: cannot drop column asset_proxy_whitelist of table application_settings because other objects depend on it
+  DETAIL: trigger trigger_0d588df444c8 on table application_settings depends on column asset_proxy_whitelist of table application_settings
+  ```
 
-To work around this bug, follow the previous steps to complete the update.
-More details are available [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160).
+  To work around this bug, follow the previous steps to complete the update.
+  More details are available [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160).
+
+- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
 
 ### 13.8.8
 
@@ -615,6 +638,14 @@ After upgraded to 11.11.8 you can safely upgrade to 12.0.Z.
 See our [documentation on upgrade paths](../policy/maintenance.md#upgrade-recommendations)
 for more information.
 
+### Maintenance mode issue in GitLab 13.9 to 14.4
+
+When [Maintenance mode](../administration/maintenance_mode/index.md) is enabled, users cannot sign in with SSO, SAML, or LDAP.
+
+Users who were signed in before Maintenance mode was enabled will continue to be signed in. If the admin who enabled Maintenance mode loses their session, then they will not be able to disable Maintenance mode via the UI. In that case, you can [disable Maintenance mode via the API or Rails console](../administration/maintenance_mode/#disable-maintenance-mode).
+
+[This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in GitLab 14.5.0, and is expected to be backported to GitLab 14.3 and 14.4.
+
 ## Miscellaneous
 
 - [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating
diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md
index 6aa32b1b816..4167d30cdab 100644
--- a/doc/user/project/repository/gpg_signed_commits/index.md
+++ b/doc/user/project/repository/gpg_signed_commits/index.md
@@ -54,17 +54,18 @@ started:
 1. [Install GPG](https://www.gnupg.org/download/index.html) for your operating system.
    If your operating system has `gpg2` installed, replace `gpg` with `gpg2` in
    the following commands.
-1. Generate the private/public key pair with the following command, which will
-   spawn a series of questions:
+1. Generate the private/public key pair with the command appropriate for your version
+   of `gpg`. This command spawns a series of questions:
 
    ```shell
+   # Use this command for the default version of gpg, including
+   # Gpg4win on Windows, and most macOS versions:
+   gpg --gen-key
+
+   # Use this command for versions of GPG later than 2.1.17:
    gpg --full-gen-key
    ```
 
-   NOTE:
-   In some cases like Gpg4win on Windows and other macOS versions, the command
-   here may be `gpg --gen-key`.
-
 1. The first question is which algorithm can be used. Select the kind you want
    or press <kbd>Enter</kbd> to choose the default (RSA and RSA):