From 787981be19177a149a901e873f199331ecb873d4 Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Mon, 28 Nov 2022 06:10:35 +0000 Subject: Add latest changes from gitlab-org/gitlab@master --- doc/api/merge_requests.md | 68 ++++++++++++++ .../pods/pods-feature-container-registry.md | 104 ++++++++++++++++++++- doc/development/testing_guide/best_practices.md | 2 + 3 files changed, 173 insertions(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 28233efdcd8..6ee74cc73ce 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -1101,6 +1101,74 @@ Supported attributes: } ``` +WARNING: +This endpoint was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/322117) in GitLab 15.7 +and will be removed in API v5. Use the [List merge request diffs](#list-merge-request-diffs) endpoint instead. + +## List merge request diffs + +List diffs of the files changed in a merge request. + +```plaintext +GET /projects/:id/merge_requests/:merge_request_iid/diffs +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +|-----------|------|----------|-------------| +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `merge_request_iid` | integer | **{check-circle}** Yes | The internal ID of the merge request. | +| `page` | integer | **{dotted-circle}** no | The page of results to return. Defaults to 1. | +| `per_page` | integer | **{dotted-circle}** no | The number of results per page. Defaults to 20. | + +If successful, returns [`200 OK`](index.md#status-codes) and the +following response attributes: + +| Attribute | Type | Description | +|:----------|:-----|:------------| +| `old_path` | string | Old path of the file. | +| `new_path` | string | New path of the file. | +| `a_mode` | string | Old file mode of the file. | +| `b_mode` | string | New file mode of the file. | +| `diff` | string | Diff representation of the changes made on the file. | +| `new_file` | boolean | Indicates if the file has just been added. | +| `renamed_file` | boolean | Indicates if the file has been renamed. | +| `deleted_file` | boolean | Indicates if the file has been removed. | + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: " "https://gitlab.example.com/api/v4/projects/1/merge_requests/1/diffs?page=1&per_page=2" +``` + +Example response: + +```json +[ + { + "old_path": "README", + "new_path": "README", + "a_mode": "100644", + "b_mode": "100644", + "diff": "--- a/README\ +++ b/README\ @@ -1 +1 @@\ -Title\ +README", + "new_file": false, + "renamed_file": false, + "deleted_file": false + }, + { + "old_path": "VERSION", + "new_path": "VERSION", + "a_mode": "100644", + "b_mode": "100644", + "diff": "--- a/VERSION\ +++ b/VERSION\ @@ -1 +1 @@\ -1.9.7\ +1.9.8", + "new_file": false, + "renamed_file": false, + "deleted_file": false + } +] +``` + ## List merge request pipelines Get a list of merge request pipelines. diff --git a/doc/architecture/blueprints/pods/pods-feature-container-registry.md b/doc/architecture/blueprints/pods/pods-feature-container-registry.md index 93aed00bd61..4a2e885a8c7 100644 --- a/doc/architecture/blueprints/pods/pods-feature-container-registry.md +++ b/doc/architecture/blueprints/pods/pods-feature-container-registry.md @@ -14,16 +14,118 @@ we can document the reasons for not choosing this approach. # Pods: Container Registry -> TL;DR +GitLab Container Registry is a feature allowing to store Docker Container Images +in GitLab. You can read about GitLab integration [here](../../../user/packages/container_registry/index.md). ## 1. Definition +GitLab Container Registry is a complex service requiring usage of PostgreSQL, Redis +and Object Storage dependencies. Right now there's undergoing work to introduce +[Container Registry Metadata](../container_registry_metadata_database/index.md) +to optimize data storage and image retention policies of Container Registry. + +GitLab Container Registry is serving as a container for stored data, +but on it's own does not authenticate `docker login`. The `docker login` +is executed with user credentials (can be `personal access token`) +or CI build credentials (ephemeral `ci_builds.token`). + +Container Registry uses data deduplication. It means that the same blob +(image layer) that is shared between many projects is stored only once. +Each layer is hashed by `sha256`. + +The `docker login` does request JWT time-limited authentication token that +is signed by GitLab, but validated by Container Registry service. The JWT +token does store all authorized scopes (`container repository images`) +and operation types (`push` or `pull`). A single JWT authentication token +can be have many authorized scopes. This allows container registry and client +to mount existing blobs from another scopes. GitLab responds only with +authorized scopes. Then it is up to GitLab Container Registry to validate +if the given operation can be performed. + +The GitLab.com pages are always scoped to project. Each project can have many +container registry images attached. + +Currently in case of GitLab.com the actual registry service is served +via `https://registry.gitlab.com`. + +The main identifiable problems are: + +- the authentication reqest (`https://gitlab.com/jwt/auth`) that is processed by GitLab.com +- the `https://registry.gitlab.com` that is run by external service and uses it's own data store +- the data deduplication, the Pods architecture with registry run in a Pod would reduce + efficiency of data storage + ## 2. Data flow +### 2.1. Authorization request that is send by `docker login` + +```shell +curl \ + --user "username:password" \ + "https://gitlab/jwt/auth?client_id=docker&offline_token=true&service=container_registry&scope=repository:gitlab-org/gitlab-build-images:push,pull" +``` + +Result is encoded and signed JWT token. Second base64 encoded string (split by `.`) contains JSON with authorized scopes. + +```json +{"auth_type":"none","access":[{"type":"repository","name":"gitlab-org/gitlab-build-images","actions":["pull"]}],"jti":"61ca2459-091c-4496-a3cf-01bac51d4dc8","aud":"container_registry","iss":"omnibus-gitlab-issuer","iat":1669309469,"nbf":166} +``` + +### 2.2. Docker client fetching tags + +```shell +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/tags/list + +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/manifests/danger-ruby-2.6.6 +``` + +### 2.3. Docker client fetching blobs and manifests + +```shell +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/blobs/sha256:a3f2e1afa377d20897e08a85cae089393daa0ec019feab3851d592248674b416 +``` + ## 3. Proposal +### 3.1. Shard Container Registry separately to Pods architecture + +Due to it's architecture it extensive architecture and in general highly scalable +horizontal architecture it should be evaluated if the GitLab Container Registry +should be run not in Pod, but in a Cluster and be scaled indepdently. + +This might be easier, but would definitely not offer the same amount of data isolation. + +### 3.2. Run Container Registry within a Pod + +It appears that except `/jwt/auth` which would likely have to be processed by Router +(to decode `scope`) the container registry could be run as a local service of a Pod. + +The actual data at least in case of GitLab.com is not forwarded via registry, +but rather served directly from Object Storage / CDN. + +Its design encodes container repository image in a URL that is easily routable. +It appears that we could re-use the same stateless Router service in front of Container Registry +to serve manifests and blobs redirect. + +The only downside is increased complexity of managing standalone registry for each Pod, +but this might be desired approach. + ## 4. Evaluation +There do not seem any theorethical problems with running GitLab Container Registry in a Pod. +Service seems that can be easily made routable to work well. + +The practical complexities are around managing complex service from infrastructure side. + ## 4.1. Pros ## 4.2. Cons diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 80a4604c341..a0bdd9f9eb4 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -859,6 +859,8 @@ it 'is overdue' do travel_to(3.days.from_now) do expect(issue).to be_overdue end + + travel_back # Returns the current time back to its original state end ``` -- cgit v1.2.3