Add latest changes from gitlab-org/gitlab@master

4 files changed, 183 insertions, 46 deletions

diff --git a/doc/development/lfs.md b/doc/development/lfs.md
index 8f6d54e08e3..bb327bf5d04 100644
--- a/doc/development/lfs.md
+++ b/doc/development/lfs.md
@@ -3,12 +3,144 @@ stage: Create
 group: Source Code
 info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
 ---
-
 # Git LFS development guidelines
 
 This page contains developer-centric information for GitLab team members. For the
 user documentation, see [Git Large File Storage](../topics/git/lfs/index.md).
 
+## Controllers and Services
+
+### Repositories::GitHttpClientController
+
+The methods for authentication defined here are inherited by all the other LFS controllers.
+
+### Repositories::LfsApiController
+
+#### `#batch`
+
+After authentication the `batch` action is the first action called by the Git LFS
+client during downloads and uploads (such as pull, push, and clone).
+
+### Repositories::LfsStorageController
+
+#### `#upload_authorize`
+
+Provides payload to Workhorse including a path for Workhorse to save the file to. Could be remote object storage.
+
+#### `#upload_finalize`
+
+Handles requests from Workhorse that contain information on a file that workhorse already uploaded (see [this middleware](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/middleware/multipart.rb)) so that `gitlab` can either:
+
+- Create an `LfsObject`.
+- Connect an existing `LfsObject` to a project with an `LfsObjectsProject`.
+
+### LfsObject and LfsObjectsProject
+
+- Only one `LfsObject` is created for a file with a given `oid` (a SHA256 checksum of the file) and file size.
+- `LfsObjectsProject` associate `LfsObject`s with `Project`s. They determine if a file can be accessed through a project.
+- These objects are also used for calculating the amount of LFS storage a given project is using.
+  For more information, see
+  [`ProjectStatistics#update_lfs_objects_size`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/project_statistics.rb#L82-84).
+
+### Repositories::LfsLocksApiController
+
+Handles the lock API for LFS. Delegates mostly to corresponding services:
+
+- `Lfs::LockFileService`
+- `Lfs::UnlockFileService`
+- `Lfs::LocksFinderService`
+
+These services create and delete `LfsFileLock`.
+
+#### `#verify`
+
+- This endpoint responds with a payload that allows a client to check if there are any files being pushed that have locks that belong to another user.
+- A client-side `lfs.locksverify` configuration can be set so that the client aborts the push if locks exist that belong to another user.
+- The existence of locks belonging to other users is also [validated on the server side](https://gitlab.com/gitlab-org/gitlab/-/blob/65f0c6e59121b62c9b0f89b810ef5186969bb4d2/lib/gitlab/checks/diff_check.rb#L69).
+
+## Example authentication
+
+```mermaid
+sequenceDiagram
+autonumber
+    alt Over HTTPS
+        Git client-->>Git client: user-supplied credentials
+    else Over SSH
+        Git client->>gitlab-shell: git-lfs-authenticate
+        activate gitlab-shell
+        activate GitLab Rails
+        gitlab-shell->>GitLab Rails:  POST /api/v4/internal/lfs_authenticate
+        GitLab Rails-->>gitlab-shell: token with expiry
+        deactivate gitlab-shell
+        deactivate GitLab Rails
+    end
+```
+
+1. Clients can be configured to store credentials in a few different ways.
+   See the [Git LFS documentation on authentication](https://github.com/git-lfs/git-lfs/blob/bea0287cdd3acbc0aa9cdf67ae09b6843d3ffcf0/docs/api/authentication.md#git-credentials).
+1. Running `gitlab-lfs-authenticate` on `gitlab-shell`. See the [Git LFS documentation concerning `gitlab-lfs-authenticate`](https://github.com/git-lfs/git-lfs/blob/bea0287cdd3acbc0aa9cdf67ae09b6843d3ffcf0/docs/api/server-discovery.md#ssh).
+1. `gitlab-shell`makes a request to the GitLab API.
+1. [Responding to shell with token](https://gitlab.com/gitlab-org/gitlab/-/blob/7a2f7a31a88b6085ea89b8ba188a4d92d5fada91/lib/api/internal/base.rb#L168) which is used in subsequent requests. See [Git LFS documentation concerning authentication](https://github.com/git-lfs/git-lfs/blob/bea0287cdd3acbc0aa9cdf67ae09b6843d3ffcf0/docs/api/authentication.md).
+
+## Example clone
+
+```mermaid
+sequenceDiagram
+    Note right of Git client: Typical Git clone things happen first
+    Note right of Git client: Authentication for LFS comes next
+    activate GitLab Rails
+    autonumber
+    Git client->>GitLab Rails: POST project/namespace/info/lfs/objects/batch
+    GitLab Rails-->>Git client: payload with objects
+    deactivate GitLab Rails
+    loop each object in payload
+    Git client->>GitLab Rails: GET project/namespace/gitlab-lfs/objects/:oid/ (<- This URL is from the payload)
+    GitLab Rails->>Workhorse: SendfileUpload
+    Workhorse-->> Git client: Binary data
+    end
+```
+
+1. Git LFS requests the ability to download files with authorization header from authorization.
+1. `gitlab` responds with the list of objects and where to find them. See
+   [LfsApiController#batch](https://gitlab.com/gitlab-org/gitlab/-/blob/7a2f7a31a88b6085ea89b8ba188a4d92d5fada91/app/controllers/repositories/lfs_api_controller.rb#L25).
+1. Git LFS makes a request for each file for the `href` in the previous response. See
+   [how downloads are handled with the basic transfer mode](https://github.com/git-lfs/git-lfs/blob/bea0287cdd3acbc0aa9cdf67ae09b6843d3ffcf0/docs/api/basic-transfers.md#downloads).
+1. `gitlab` redirects to the remote URL if remote object storage is enabled. See
+   [SendFileUpload](https://gitlab.com/gitlab-org/gitlab/-/blob/7a2f7a31a88b6085ea89b8ba188a4d92d5fada91/app/controllers/concerns/send_file_upload.rb#L4).
+
+## Example push
+
+```mermaid
+sequenceDiagram
+    Note right of Git client: Typical Git push things happen first.
+    Note right of Git client: Suthentication for LFS comes next.
+    autonumber
+    activate GitLab Rails
+        Git client ->> GitLab Rails: POST project/namespace/info/lfs/objects/batch
+        GitLab Rails-->>Git client: payload with objects
+    deactivate GitLab Rails
+    loop each object in payload
+    Git client->>Workhorse: PUT project/namespace/gitlab-lfs/objects/:oid/:size (URL is from payload)
+    Workhorse->>GitLab Rails: PUT project/namespace/gitlab-lfs/objects/:oid/:size/authorize
+    GitLab Rails-->>Workhorse: response with where path to upload
+    Workhorse->>Workhorse: Upload
+    Workhorse->>GitLab Rails: PUT project/namespace/gitlab-lfs/objects/:oid/:size/finalize
+    end
+```
+
+1. Git LFS requests the ability to upload files.
+1. `gitlab` responds with the list of objects and uploads to find them. See
+   [LfsApiController#batch](https://gitlab.com/gitlab-org/gitlab/-/blob/7a2f7a31a88b6085ea89b8ba188a4d92d5fada91/app/controllers/repositories/lfs_api_controller.rb#L27).
+1. Git LFS makes a request for each file for the `href` in the previous response. See
+   [how uploads are handled with the basic transfer mode](https://github.com/git-lfs/git-lfs/blob/bea0287cdd3acbc0aa9cdf67ae09b6843d3ffcf0/docs/api/basic-transfers.md#uploads).
+1. `gitlab` responds with a payload including a path for Workhorse to save the file to.
+   Could be remote object storage. See
+   [LfsStorageController#upload_authorize](https://gitlab.com/gitlab-org/gitlab/-/blob/96250de93a410e278ef659a3d38b056f12024636/app/controllers/repositories/lfs_storage_controller.rb#L42).
+1. Workhorse does the work of saving the file.
+1. Workhorse makes a request to `gitlab` with information on the uploaded file so
+   that `gitlab` can create an `LfsObject`. See
+   [LfsStorageController#upload_finalize](https://gitlab.com/gitlab-org/gitlab/-/blob/96250de93a410e278ef659a3d38b056f12024636/app/controllers/repositories/lfs_storage_controller.rb#L51).
+
 ## Deep Dive
 
 In April 2019, Francisco Javier López hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/-/issues/1`)
diff --git a/doc/integration/oauth2_generic.md b/doc/integration/oauth2_generic.md
index 7b809e2feee..79238c78421 100644
--- a/doc/integration/oauth2_generic.md
+++ b/doc/integration/oauth2_generic.md
@@ -107,39 +107,45 @@ To configure the provider:
       helm get values gitlab > gitlab_values.yaml
       ```
 
-   1. Edit `gitlab_values.yaml`.
+   1. Put the following content in a file named `oauth2_generic.yaml` for use as a
+      [Kubernetes Secret](https://docs.gitlab.com/charts/charts/globals.html#providers):
 
-      NOTE:
-      The following example exposes the `app_secret` value in the main YAML file.
-      You're strongly advised to use
-      [Helm secrets](https://docs.gitlab.com/charts/installation/secrets.html)
-      instead.
+      ```yaml
+      name: "oauth2_generic"
+      label: "Provider name" # optional label for login button defaults to "Oauth2 Generic"
+      app_id: "<your_app_client_id>"
+      app_secret: "<your_app_client_secret>"
+      args:
+        client_options:
+          site: "<your_auth_server_url>"
+          user_info_url: "/oauth2/v1/userinfo"
+          authorize_url: "/oauth2/v1/authorize"
+          token_url: "/oauth2/v1/token"
+        user_response_structure:
+          root_path: []
+          id_path: ["sub"]
+          attributes:
+            email: "email"
+            name: "name"
+        authorize_params:
+          scope: "openid profile email"
+        strategy_class: "OmniAuth::Strategies::OAuth2Generic"
+      ```
+
+   1. Create the Kubernetes Secret:
+
+      ```shell
+      kubectl create secret generic -n <namespace> gitlab-oauth2-generic --from-file=provider=oauth2_generic.yaml
+      ```
+
+   1. Edit `gitlab_values.yaml` and add the provider configuration:
 
       ```yaml
       global:
         appConfig:
           omniauth:
-            enabled: true
             providers:
-              - name: "oauth2_generic"
-                label: "Provider name" # optional label for login button defaults to "Oauth2 Generic"
-                app_id: "<your_app_client_id>"
-                app_secret: "<your_app_client_secret>"
-                args:
-                  client_options:
-                    site: "<your_auth_server_url>"
-                    user_info_url: "/oauth2/v1/userinfo"
-                    authorize_url: "/oauth2/v1/authorize"
-                    token_url: "/oauth2/v1/token"
-                  user_response_structure:
-                    root_path: []
-                    id_path: ["sub"]
-                    attributes:
-                      email: "email"
-                      name: "name"
-                  authorize_params:
-                    scope: "openid profile email"
-                  strategy_class: "OmniAuth::Strategies::OAuth2Generic"
+              - secret: gitlab-oauth2-generic
       ```
 
    1. Save the file and apply the new values:
diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md
index 37f1f4cc65e..d1f8722b9dc 100644
--- a/doc/user/group/saml_sso/index.md
+++ b/doc/user/group/saml_sso/index.md
@@ -18,28 +18,15 @@ Users can sign in to GitLab through their SAML identity provider.
 
 You can configure SAML SSO for the top-level group only.
 
-## Configure your identity provider
-
-1. [Configure your SAML identity provider](#set-up-identity-provider).
-1. Configure the SAML response to include a [**NameID**](#nameid) that uniquely identifies each user.
-1. Configure the required [user attributes](#user-attributes), ensuring you include the user's email address.
-1. While the default is enabled for most SAML providers, ensure the app is set to have service provider
-   initiated calls to link existing GitLab accounts.
-1. Once the identity provider is set up, move on to [configuring GitLab](#configure-gitlab).
-
-![Issuer and callback for configuring SAML identity provider with GitLab.com](img/group_saml_configuration_information.png)
-
-If your account is the only owner in the group after SAML is set up, you can't unlink the account. To [unlink the account](#unlinking-accounts),
-set up another user as a group owner.
-
 ## Set up identity provider
 
 The SAML standard means that you can use a wide range of identity providers with GitLab. Your identity provider might have relevant documentation. It can be generic SAML documentation or specifically targeted for GitLab.
 
-When [configuring your identity provider](#configure-your-identity-provider), consider the notes below for specific providers to help avoid common issues and as a guide for terminology used.
+When setting up your identity provider, use the following provider-specific documentation
+to help avoid common issues and as a guide for terminology used.
 
-For providers not listed below, you can refer to the [instance SAML notes on configuring an identity provider](../../../integration/saml.md#configure-saml-on-your-idp)
-for additional guidance on information your identity provider may require.
+For identity providers not listed, you can refer to the [instance SAML notes on configuring an identity provider](../../../integration/saml.md#configure-saml-on-your-idp)
+for additional guidance on information your provider may require.
 
 GitLab provides the following information for guidance only.
 If you have any questions on configuring the SAML app, contact your provider's support.
@@ -69,6 +56,9 @@ To set up SSO with Azure as your identity provider:
    - **nameid-format** to `persistent`. For more information, see [**NameID**](#nameid).
    - **Additional claims** to [supported attributes](#user-attributes).
 
+1. Make sure the identity provider is set to have provider-initiated calls
+   to link existing GitLab accounts.
+
 1. Optional. If you use [Group Sync](#group-sync), customize the name of the
    group claim to match the required attribute.
 
@@ -109,6 +99,9 @@ To set up Google Workspace as your identity provider:
    - For **Name ID format**: `EMAIL`. For more information, see the [**NameID** format documentation](#nameid-format).
    - For **NameID**: `Basic Information > Primary email`. For more information, see [**NameID**](#nameid).
 
+1. Make sure the identity provider is set to have provider-initiated calls
+   to link existing GitLab accounts.
+
 On the GitLab SAML SSO page, when you select **Verify SAML Configuration**, disregard
 the warning that recommends setting the **NameID** format to `persistent`.
 
@@ -138,6 +131,9 @@ To set up SSO with Okta as your identity provider:
    - For **Application username (NameID)**: **Custom** `user.getInternalProperty("id")`.
    - For **Name ID Format**: `Persistent`. For more information, see [**NameID**](#nameid).
 
+1. Make sure the identity provider is set to have provider-initiated calls
+   to link existing GitLab accounts.
+
 The Okta GitLab application available in the App Catalog only supports [SCIM](scim_setup.md). Support
 for SAML is proposed in [issue 216173](https://gitlab.com/gitlab-org/gitlab/-/issues/216173).
 
@@ -171,6 +167,9 @@ To set up OneLogin as your identity provider:
 
 1. For **NameID**, use `OneLogin ID`. For more information, see [**NameID**](#nameid).
 
+1. Make sure the identity provider is set to have provider-initiated calls
+   to link existing GitLab accounts.
+
 ### Set up identity provider using metadata
 
 To configure some identity providers, you need a GitLab metadata URL.
@@ -198,7 +197,7 @@ users cannot access any of the SAML groups. To mitigate this, you can disable
 
 To change identity providers:
 
-1. [Configure](#configure-your-identity-provider) the group with the new identity provider.
+1. [Configure](#set-up-identity-provider) the group with the new identity provider.
 1. Optional. If the **NameID** is not identical, [change the **NameID** for users](#change-nameid-for-one-or-more-users).
 
 #### Change email domains
diff --git a/doc/user/group/saml_sso/troubleshooting.md b/doc/user/group/saml_sso/troubleshooting.md
index eadf385feb3..a022c299d0f 100644
--- a/doc/user/group/saml_sso/troubleshooting.md
+++ b/doc/user/group/saml_sso/troubleshooting.md
@@ -244,7 +244,7 @@ If all users are receiving a `404` when attempting to sign in using SAML, confir
 If you receive a `404` during setup when using "verify configuration", make sure you have used the correct
 [SHA-1 generated fingerprint](../../../integration/saml.md#configure-saml-on-your-idp).
 
-If a user is trying to sign in for the first time and the GitLab single sign-on URL has not [been configured](index.md#configure-your-identity-provider), they may see a 404.
+If a user is trying to sign in for the first time and the GitLab single sign-on URL has not [been configured](index.md#set-up-identity-provider), they may see a 404.
 As outlined in the [user access section](index.md#linking-saml-to-your-existing-gitlabcom-account), a group Owner needs to provide the URL to users.
 
 If all users are receiving a `404` after signing in to the identity provider (IdP):