diff options
Diffstat (limited to 'doc/architecture/blueprints')
23 files changed, 477 insertions, 131 deletions
diff --git a/doc/architecture/blueprints/ci_data_decay/index.md b/doc/architecture/blueprints/ci_data_decay/index.md index 6df37e28992..e26e7d5dbd3 100644 --- a/doc/architecture/blueprints/ci_data_decay/index.md +++ b/doc/architecture/blueprints/ci_data_decay/index.md @@ -1,8 +1,8 @@ --- -status: ready +status: ongoing creation-date: "2021-09-10" authors: [ "@grzesiek" ] -coach: "@kamil" +coach: [ "@ayufan", "@grzesiek" ] approvers: [ "@jporter", "@cheryl.li" ] owning-stage: "~devops::verify" participating-stages: [] diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 100c9e67fda..5bff794c683 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -1,8 +1,8 @@ --- status: proposed creation-date: "2022-09-14" -authors: [ "@fabio", "@grzesiek" ] -coach: "@kamil" +authors: [ "@ayufan", "@fabiopitino", "@grzesiek" ] +coach: [ "@ayufan", "@grzesiek" ] approvers: [ "@dhershkovitch", "@marknuzzo" ] owning-stage: "~devops::verify" participating-stages: [] @@ -134,6 +134,28 @@ For best experience with any systems made of components it's fundamental that co The version identifies the exact interface and behavior of the component. - **Resolvable**: when a component depends on another component, this dependency must be explicit and trackable. +### Predictable components + +Eventually, we want to make CI Catalog Components predictable. Including a +component by its path, using a fixed `@` version, should always return the same +configuration, regardless of a context from which it is getting included from. +The resulting configuration should be the same for a given component version +and the set of inputs passed using `with:` keyword, hence it should be +[deterministic](https://en.wikipedia.org/wiki/Deterministic_algorithm). + +A component should not produce side effects by being included and should be +[referentially transparent](https://en.wikipedia.org/wiki/Referential_transparency). + +Making components predictable is a process, and we may not be able to achieve +this without significantly redesigning CI templates, what could be disruptive +for users and customers right now. We initially considered restricting some +top-level keywords, like `include: remote:` to make components more +deterministic, but eventually agreed that we first need to iterate on the MVP +to better understand the design that is required to make components more +predictable. The predictability, determinism, referential transparency and +making CI components predictable is still important for us, but we may be +unable to achieve it early iterations. + ## Structure of a component A pipeline component is identified by the path to a repository or directory that defines it @@ -143,30 +165,36 @@ For example: `gitlab-org/dast@1.0`. ### The component path -A component path must contain at least the metadata YAML and optionally a related `README.md` documentation file. +A component path must contain at least the component YAML and optionally a +related `README.md` documentation file. The component path can be: -- A path to a project: `gitlab-org/dast`. In this case the 2 files are defined in the root directory of the repository. -- A path to a project subdirectory: `gitlab-org/dast/api-scan`. In this case the 2 files are defined in the `api-scan` directory. -- A path to a local directory: `/path/to/component`. This path must contain the metadata YAML that defines the component. +- A path to a project: `gitlab-org/dast`. The default component is processed. +- A path to an explicit component: `gitlab-org/dast/api-scan`. In this case the explicit `api-scan` component is processed. +- A path to a local directory: `/path/to/component`. This path must contain the component YAML that defines the component. The path must start with `/` to indicate a full path in the repository. -The metadata YAML file follows the filename convention `gitlab-<component-type>.yml` where component type is one of: +The component YAML file follows the filename convention `<type>.yml` where component type is one of: | Component type | Context | | -------------- | ------- | | `template` | For components used under `include:` keyword | | `step` | For components used under `steps:` keyword | -| `workflow` | For components used under `trigger:` keyword | Based on the context where the component is used we fetch the correct YAML file. -For example, if we are including a component `gitlab-org/dast@1.0` we expect a YAML file named `gitlab-template.yml` in the -top level directory of `gitlab-org/dast` repository. +For example: + +- if we are including a component `gitlab-org/dast@1.0` we expect a YAML file named `template.yml` in the + root directory of `gitlab-org/dast` repository. +- if we are including a component `gitlab-org/dast/api-scan@1.0` we expect a YAML file named `template.yml` inside a + directory `api-scan` of `gitlab-org/dast` repository. +- if we are using a step component `gitlab-org/dast/api-scan@1.0` we expect a YAML file named `step.yml` inside a + directory `api-scan` of `gitlab-org/dast` repository. -A `gitlab-<component-type>.yml` file: +A component YAML file: -- Must have a **name** to be referenced to and **description** for extra details. +- Must have a **name** to be referenced to. - Must specify its **type** in the filename, which defines how it can be used (raw configuration to be `include`d, child pipeline workflow, job step). - Must define its **content** based on the type. - Must specify **input parameters** that it accepts. Components should depend on input parameters for dynamic values and not environment variables. @@ -174,6 +202,7 @@ A `gitlab-<component-type>.yml` file: - Should be **validated statically** (for example: using JSON schema validators). ```yaml +--- spec: inputs: website: @@ -184,11 +213,12 @@ spec: - unit - integration - system -content: { ... } +--- +# content of the component ``` -Components that are released in the catalog must have a `README.md` file in the same directory as the -metadata YAML file. The `README.md` represents the documentation for the specific component, hence it's recommended +Components that are released in the catalog must have a `README.md` file at the root directory of the repository. +The `README.md` represents the documentation for the specific component, hence it's recommended even when not releasing versions in the catalog. ### The component version @@ -230,30 +260,28 @@ The following directory structure would support 1 component per project: ```plaintext . -├── gitlab-<type>.yml +├── template.yml ├── README.md └── .gitlab-ci.yml ``` The `.gitlab-ci.yml` is recommended for the project to ensure changes are verified accordingly. -The component is now identified by the path `myorg/rails-rspec`. In other words, this means that -the `gitlab-<type>.yml` and `README.md` are located in the root directory of the repository. +The component is now identified by the path `myorg/rails-rspec` and we expect a `template.yml` file +and `README.md` located in the root directory of the repository. The following directory structure would support multiple components per project: ```plaintext . ├── .gitlab-ci.yml +├── README.md ├── unit/ -│ ├── gitlab-workflow.yml -│ └── README.md +│ └── template.yml ├── integration/ -│ ├── gitlab-workflow.yml -│ └── README.md +│ └── template.yml └── feature/ - ├── gitlab-workflow.yml - └── README.md + └── template.yml ``` In this example we are defining multiple test profiles that are executed with RSpec. @@ -266,18 +294,20 @@ This directory structure could also support both strategies: ```plaintext . -├── gitlab-template.yml # myorg/rails-rspec +├── template.yml # myorg/rails-rspec ├── README.md +├── LICENSE ├── .gitlab-ci.yml ├── unit/ -│ ├── gitlab-workflow.yml # myorg/rails-rspec/unit -│ └── README.md +│ └── template.yml # myorg/rails-rspec/unit ├── integration/ -│ ├── gitlab-workflow.yml # myorg/rails-rspec/integration -│ └── README.md -└── feature/ - ├── gitlab-workflow.yml # myorg/rails-rspec/feature - └── README.md +│ └── template.yml # myorg/rails-rspec/integration +├── feature/ +│ └── template.yml # myorg/rails-rspec/feature +└── report/ + ├── step.yml # myorg/rails-rspec/report + ├── Dockerfile + └── ... other files ``` With the above structure we could have a top-level component that can be used as the @@ -285,14 +315,15 @@ default component. For example, `myorg/rails-rspec` could run all the test profi However, more specific test profiles could be used separately (for example `myorg/rails-rspec/integration`). NOTE: -Any nesting more than 1 level is initially not permitted. +Nesting of components is not permitted. This limitation encourages cohesion at project level and keeps complexity low. -## Input parameters `spec:inputs:` parameters +## `spec:inputs:` parameters If the component takes any input parameters they must be specified according to the following schema: ```yaml +--- spec: inputs: website: # by default all declared inputs are mandatory. @@ -303,8 +334,15 @@ spec: - unit - integration - system +--- +# content of the component +my-job: + script: echo ``` +The YAML in this case contains 2 documents. The first document represents the specifications while the +second document represents the content. + When using the component we pass the input parameters as follows: ```yaml @@ -322,27 +360,28 @@ possible [inputs provided upstream](#input-parameters-for-pipelines). Input parameters are validated as soon as possible: 1. Read the file `gitlab-template.yml` inside `org/my-component`. -1. Parse `spec:inputs` and validate the parameters against this schema. -1. If successfully validated, proceed with parsing `content:`. Return an error otherwise. -1. Interpolate input parameters inside the component's `content:`. +1. Parse `spec:inputs` from the specifications and validate the parameters against this schema. +1. If successfully validated, proceed with parsing the content. Return an error otherwise. +1. Interpolate input parameters inside the component's content. ```yaml +--- spec: inputs: environment: options: [test, staging, production] -content: - "run-tests-$[[ inputs.environment ]]": - script: ./run-test - - scan-website: - script: ./scan-website $[[ inputs.environment ]] - rules: - - if: $[[ inputs.environment ]] == 'staging' - - if: $[[ inputs.environment ]] == 'production' +--- +"run-tests-$[[ inputs.environment ]]": + script: ./run-test + +scan-website: + script: ./scan-website $[[ inputs.environment ]] + rules: + - if: $[[ inputs.environment ]] == 'staging' + - if: $[[ inputs.environment ]] == 'production' ``` -With `$[[ inputs.XXX ]]` inputs are interpolated immediately after parsing the `content:`. +With `$[[ inputs.XXX ]]` inputs are interpolated immediately after parsing the content. ### Why input parameters and not environment variables? @@ -386,17 +425,19 @@ include: foo: bar ``` -Then the configuration being included must specify the inputs: +Then the configuration being included must specify the inputs by defining a specification section in the YAML: ```yaml +--- spec: inputs: foo: - +--- # rest of the configuration ``` -If a YAML includes content using `with:` but the including YAML doesn't specify `inputs:`, an error should be raised. +If a YAML includes content using `with:` but the including YAML doesn't define `inputs:` in the specifications, +an error should be raised. |`with:`| `inputs:` | result | | --- | --- | --- | @@ -428,9 +469,10 @@ deploy-app: deploy_environment: staging ``` -To solve the problem of `Run Pipeline` UI form we could fully leverage the `spec:inputs` schema: +To solve the problem of `Run Pipeline` UI form we could fully leverage the `inputs` specifications: ```yaml +--- spec: inputs: concurrency: @@ -443,9 +485,11 @@ spec: - canary # 2nd option - production # 3rd option default: staging # selected by default in the UI. - # if `default:` is not specified, the user must explicitly select - # an option. + # if `default:` is not specified, the user must explicitly select + # an option. description: Deployment environment # optional: render as input label. +--- +# rest of the pipeline config ``` ## Limits diff --git a/doc/architecture/blueprints/ci_scale/index.md b/doc/architecture/blueprints/ci_scale/index.md index 574a79c86a5..cf7065f7c07 100644 --- a/doc/architecture/blueprints/ci_scale/index.md +++ b/doc/architecture/blueprints/ci_scale/index.md @@ -1,5 +1,5 @@ --- -status: in progress +status: ongoing creation-date: "2021-01-21" authors: [ "@grzesiek" ] coach: "@grzesiek" diff --git a/doc/architecture/blueprints/cloud_native_build_logs/index.md b/doc/architecture/blueprints/cloud_native_build_logs/index.md index 20cfb46abc4..bbce54a3fb2 100644 --- a/doc/architecture/blueprints/cloud_native_build_logs/index.md +++ b/doc/architecture/blueprints/cloud_native_build_logs/index.md @@ -2,7 +2,7 @@ status: implemented creation-date: "2020-08-26" authors: [ "@grzesiek" ] -coach: "@kamil" +coach: [ "@ayufan", "@grzesiek" ] approvers: [ "@thaoyeager", "@darbyfrey" ] owning-stage: "~devops::release" participating-stages: [] diff --git a/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md b/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md index b6f3a59dc0b..2bca1a4a4ca 100644 --- a/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md +++ b/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md @@ -2,7 +2,7 @@ status: implemented creation-date: "2019-05-16" authors: [ "@grzesiek" ] -coach: "@kamil" +coach: [ "@ayufan", "@grzesiek" ] approvers: [ "@ogolowinski", "@dcroft", "@vshushlin" ] owning-stage: "~devops::release" participating-stages: [] diff --git a/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md b/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md index 3ef98c33035..7fecbd1de71 100644 --- a/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md +++ b/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md @@ -1,14 +1,14 @@ --- status: proposed creation-date: "2021-05-19" -authors: [ "@kamil", "@mkaeppler" ] +authors: [ "@ayufan", "@mkaeppler" ] coach: "@glopezfernandez" approvers: [] owning-stage: "~devops::non_devops" participating-stages: [] --- -# Composable GitLab codebase - using Rails Engines +# Composable GitLab Codebase NOTE: Due to our focus on improving the overall availability of GitLab.com and reducing tech debt, we do not have capacity to act on this blueprint. We will re-evaluate in Q1-FY23. diff --git a/doc/architecture/blueprints/feature_flags_development/index.md b/doc/architecture/blueprints/feature_flags_development/index.md index 730daf56f0d..b2e6fd1e82c 100644 --- a/doc/architecture/blueprints/feature_flags_development/index.md +++ b/doc/architecture/blueprints/feature_flags_development/index.md @@ -1,14 +1,14 @@ --- -status: accepted +status: implemented creation-date: "2020-06-10" -authors: [ "@kamil" ] +authors: [ "@ayufan" ] coach: "@glopezfernandez" approvers: [ "@kencjohnston", "@craig-gomes" ] owning-stage: "~devops::non_devops" participating-stages: [] --- -# Architectural discussion of feature flags +# Development Feature Flags Architecture Usage of feature flags become crucial for the development of GitLab. The feature flags are a convenient way to ship changes early, and safely rollout diff --git a/doc/architecture/blueprints/gitlab_observability_backend/metrics/index.md b/doc/architecture/blueprints/gitlab_observability_backend/metrics/index.md index a6efe68310e..c5bd2440b0c 100644 --- a/doc/architecture/blueprints/gitlab_observability_backend/metrics/index.md +++ b/doc/architecture/blueprints/gitlab_observability_backend/metrics/index.md @@ -673,10 +673,10 @@ Using PromQL directly could be a steep learning curve for users. It would be rea The following section enlists how we intend to implement the aforementioned proposal around building Metrics support into GitLab Observability Service. Each corresponding document and/or issue contains further details of how each next step is planned to be executed. -- **[DONE]** [Research & draft design proposal and/or requirements](https://docs.google.com/document/d/1kHyIoWEcs14sh3CGfKGiI8QbCsdfIHeYkzVstenpsdE/edit?usp=sharing) -- **[IN-PROGRESS]** [Submit system/schema designs (proposal) & gather feedback](https://docs.google.com/document/d/1kHyIoWEcs14sh3CGfKGiI8QbCsdfIHeYkzVstenpsdE/edit?usp=sharing) -- **[IN-PROGRESS]** [Develop table definitions and/or storage interfaces](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/1666) -- **[IN-PROGRESS]** [Prototype reference implementation, instrument key metrics](https://gitlab.com/gitlab-org/opstrace/opstrace/-/merge_requests/1823) +- **DONE** [Research & draft design proposal and/or requirements](https://docs.google.com/document/d/1kHyIoWEcs14sh3CGfKGiI8QbCsdfIHeYkzVstenpsdE/edit?usp=sharing) +- **IN-PROGRESS** [Submit system/schema designs (proposal) & gather feedback](https://docs.google.com/document/d/1kHyIoWEcs14sh3CGfKGiI8QbCsdfIHeYkzVstenpsdE/edit?usp=sharing) +- **IN-PROGRESS** [Develop table definitions and/or storage interfaces](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/1666) +- **IN-PROGRESS** [Prototype reference implementation, instrument key metrics](https://gitlab.com/gitlab-org/opstrace/opstrace/-/merge_requests/1823) - [Benchmark Clickhouse and/or proposed schemas, gather expert advice from Clickhouse Inc.](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/1666) - Develop write path(s) - `remote_write` API - Develop read path(s) - `remote_read` API, `PromQL`-based querier. diff --git a/doc/architecture/blueprints/gitlab_observability_backend/metrics/supported-deployments.png b/doc/architecture/blueprints/gitlab_observability_backend/metrics/supported-deployments.png Binary files differindex 54c4ed3b48f..9dccc515129 100644 --- a/doc/architecture/blueprints/gitlab_observability_backend/metrics/supported-deployments.png +++ b/doc/architecture/blueprints/gitlab_observability_backend/metrics/supported-deployments.png diff --git a/doc/architecture/blueprints/graphql_api/index.md b/doc/architecture/blueprints/graphql_api/index.md index 4b446a78541..95ff834cd27 100644 --- a/doc/architecture/blueprints/graphql_api/index.md +++ b/doc/architecture/blueprints/graphql_api/index.md @@ -2,7 +2,7 @@ status: accepted creation-date: "2021-01-07" authors: [ "@grzesiek" ] -coach: "@kamil" +coach: [ "@ayufan", "@grzesiek" ] approvers: [ "@dsatcher", "@deuley" ] owning-stage: "~devops::manage" participating-stages: [] diff --git a/doc/architecture/blueprints/image_resizing/index.md b/doc/architecture/blueprints/image_resizing/index.md index 99a7c4ae144..b3938cbc8ab 100644 --- a/doc/architecture/blueprints/image_resizing/index.md +++ b/doc/architecture/blueprints/image_resizing/index.md @@ -2,7 +2,7 @@ status: implemented creation-date: "2020-10-21" authors: [ "@craig-gomes" ] -coach: "@kamil" +coach: "@ayufan" approvers: [ "@timzallmann", "@joshlambert" ] owning-stage: "~devops::non_devops" participating-stages: [] diff --git a/doc/architecture/blueprints/pods/index.md b/doc/architecture/blueprints/pods/index.md index 0a36de5790f..077303be30f 100644 --- a/doc/architecture/blueprints/pods/index.md +++ b/doc/architecture/blueprints/pods/index.md @@ -1,8 +1,8 @@ --- status: accepted creation-date: "2022-09-07" -authors: [ "@fzimmer", "@DylanGriffith" ] -coach: "@kamil" +authors: [ "@ayufan", "@fzimmer", "@DylanGriffith" ] +coach: "@ayufan" approvers: [ "@fzimmer" ] owning-stage: "~devops::enablement" participating-stages: [] @@ -150,6 +150,14 @@ At this moment, GitLab.com has "social-network"-like capabilities that may not f We should evaluate if the SMB and mid market segment is interested in these features, or if not having them is acceptable in most cases. +### Self-managed + +For reasons of consistency, it is expected that self-managed instances will +adopt the pods architecture as well. To expand, self-managed instances can +continue with just a single Pod while supporting the option of adding additional +Pods. Organizations, and possible User decomposition will also be adopted for +self-managed instances. + ## High-level architecture problems to solve A number of technical issues need to be resolved to implement Pods (in no particular order). This section will be expanded. @@ -325,9 +333,11 @@ This is the list of known affected features with the proposed solutions. - [Pods: Organizations](pods-feature-organizations.md) - [Pods: Router Endpoints Classification](pods-feature-router-endpoints-classification.md) - [Pods: Schema changes (Postgres and Elasticsearch migrations)](pods-feature-schema-changes.md) +- [Pods: Backups](pods-feature-backups.md) - [Pods: Global Search](pods-feature-global-search.md) - [Pods: CI Runners](pods-feature-ci-runners.md) - [Pods: Admin Area](pods-feature-admin-area.md) +- [Pods: Secrets](pods-feature-secrets.md) - [Pods: Container Registry](pods-feature-container-registry.md) - [Pods: Contributions: Forks](pods-feature-contributions-forks.md) - [Pods: Personal Namespaces](pods-feature-personal-namespaces.md) diff --git a/doc/architecture/blueprints/pods/pods-feature-backups.md b/doc/architecture/blueprints/pods/pods-feature-backups.md new file mode 100644 index 00000000000..5e4de42f473 --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-backups.md @@ -0,0 +1,61 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: Backups' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: Backups + +Each pods will take its own backups, and consequently have its own isolated +backup / restore procedure. + +## 1. Definition + +GitLab Backup takes a backup of the PostgreSQL database used by the application, +and also Git repository data. + +## 2. Data flow + +Each pod has a number of application databases to back up (e.g. `main`, and `ci`). + +Additionally, there may be cluster-wide metadata tables (e.g. `users` table) +which is directly accesible via PostgreSQL. + +## 3. Proposal + +### 3.1. Cluster-wide metadata + +It is currently unknown how cluster-wide metadata tables will be accessible. We +may choose to have cluster-wide metadata tables backed up separately, or have +each pod back up its copy of cluster-wide metdata tables. + +### 3.2 Consistency + +#### 3.2.1 Take backups independently + +As each pod will communicate with each other via API, and there will be no joins +to the users table, it should be acceptable for each pod to take a backup +independently of each other. + +#### 3.2.2 Enforce snapshots + +We can require that each pod take a snapshot for the PostgreSQL databases at +around the same time to allow for a consistent-enough backup. + +## 4. Evaluation + +As the number of pods increases, it will likely not be feasible to take a +snapshot at the same time for all pods. Hence taking backups independently is +the better option. + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/pods-feature-secrets.md b/doc/architecture/blueprints/pods/pods-feature-secrets.md new file mode 100644 index 00000000000..f18a41dc0fb --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-secrets.md @@ -0,0 +1,48 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: Secrets' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: Secrets + +Where possible, each pod should have its own distinct set of secrets. +However, there will be some secrets that will be required to be the same for all +pods in the cluster + +## 1. Definition + +GitLab has a lot of +[secrets](https://docs.gitlab.com/charts/installation/secrets.html) that needs +to be configured. + +Some secrets are for inter-component communication, e.g. `GitLab Shell secret`, +and used only within a pod. + +Some secrets are used for features, e.g. `ci_jwt_signing_key`. + +## 2. Data flow + +## 3. Proposal + +1. Secrets used for features will need to be consistent across all pods, so that the UX is consistent. + 1. This is especially true for the `db_key_base` secret which is used for + encrypting data at rest in the database - so that projects that are + transferred to another pod will continue to work. We do not want to have + to re-encrypt such rows when we move projects/groups between pods. +1. Secrets which are used for intra-pod communication only should be uniquely generated + per-pod. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/proposal-stateless-router-with-buffering-requests.md b/doc/architecture/blueprints/pods/proposal-stateless-router-with-buffering-requests.md index ab19b652f93..adc523e90c2 100644 --- a/doc/architecture/blueprints/pods/proposal-stateless-router-with-buffering-requests.md +++ b/doc/architecture/blueprints/pods/proposal-stateless-router-with-buffering-requests.md @@ -555,9 +555,9 @@ sequenceDiagram They get a 404. -### Experience for non-logged in users +### Experience for non-authenticated users -Flow is similar to logged in users except global routes like `/dashboard` will +Flow is similar to authenticated users except global routes like `/dashboard` will redirect to the login page as there is no default organization to choose from. ### A new customers signs up diff --git a/doc/architecture/blueprints/pods/proposal-stateless-router-with-routes-learning.md b/doc/architecture/blueprints/pods/proposal-stateless-router-with-routes-learning.md index c99b02a35e9..1156e65f6aa 100644 --- a/doc/architecture/blueprints/pods/proposal-stateless-router-with-routes-learning.md +++ b/doc/architecture/blueprints/pods/proposal-stateless-router-with-routes-learning.md @@ -578,7 +578,7 @@ sequenceDiagram They get a 404. -### Experience for non-logged in users +### Experience for non-authenticated users Flow is similar to logged in users except global routes like `/dashboard` will redirect to the login page as there is no default organization to choose from. diff --git a/doc/architecture/blueprints/rate_limiting/index.md b/doc/architecture/blueprints/rate_limiting/index.md index 22709a90cee..26114fce7f2 100644 --- a/doc/architecture/blueprints/rate_limiting/index.md +++ b/doc/architecture/blueprints/rate_limiting/index.md @@ -4,7 +4,7 @@ creation-date: "2022-09-08" authors: [ "@grzesiek", "@marshall007", "@fabiopitino", "@hswimelar" ] coach: "@andrewn" approvers: [ "@sgoldstein" ] -owning-stage: +owning-stage: "~devops::enablement" participating-stages: [] --- @@ -354,28 +354,33 @@ hierarchy. Choosing a proper solution will require a thoughtful research. ## Phases and iterations -**Phase 1**: Compile examples of current most important application limits — Owning Team - a. Owning Team (in collaboration with Stage Groups) compiles a list of the - most important application limits used in Rails today. -**Phase 2**: Implement Rate Limiting Framework in Rails - Owning Team - a. Triangulate rate limiting abstractions based on the data gathered in Phase 1 - b. Develop YAML model for limits. - c. Build Rails SDK. - d. Create examples showcasing usage of the new rate limits SDK. -**Phase 3**: Team fan out of Rails SDK - Stage Groups - a. Individual stage groups begin using the SDK built in Phase 2 for new limit and policies. - b. Stage groups begin replacing historical adhoc limit implementations with the SDK. - c. Provides means to monitor and observe the progress of the replacement effort. Ideally this is broken down to the `feature_category` level to drive group-level buy-in -- Owning Team. -**Phase 4**: Enable Satellite Services to Use the Rate Limiting Framework - Owning Team - a. Determine if the goals of Phase 4 are best met by either - 1. Extracting the Rails rate limiting service into a decoupled service OR - 2. Implementing a separate Go library which uses the same backend (eg, Redis) for rate limiting. -**Phase 5**: SDK for Satellite Services - Owning Team - a. Build Golang SDK. - c. Create examples showcasing usage of the new rate limits SDK. -**Phase 6**: Team fan out for Satellite Services - Stage Groups - a. Individual stage groups being using the SDK built in Phase 5 for new limit and policies. - b. Stage groups begin replacing historical adhoc limit implementations with the SDK. +1. **Compile examples of current most important application limits (Owning Team)** + - Owning Team (in collaboration with Stage Groups) compiles a list of the + most important application limits used in Rails today. + +1. **Implement Rate Limiting Framework in Rails (Owning Team)** + - Triangulate rate limiting abstractions based on the data gathered in Phase 1. + - Develop YAML model for limits. + - Build Rails SDK. + - Create examples showcasing usage of the new rate limits SDK. + +1. **Team fan out of Rails SDK (Stage Groups)** + - Individual stage groups begin using the SDK built in Phase 2 for new limit and policies. + - Stage groups begin replacing historical ad hoc limit implementations with the SDK. + - (Owning team) Provides means to monitor and observe the progress of the replacement effort. Ideally this is broken down to the `feature_category` level to drive group-level buy-in. + +1. **Enable Satellite Services to Use the Rate Limiting Framework (Owning Team)** + - Determine if the goals of Phase 4 are best met by either: + - Extracting the Rails rate limiting service into a decoupled service. + - Implementing a separate Go library which uses the same backend (for example, Redis) for rate limiting. + +1. **SDK for Satellite Services (Owning Team)** + - Build Golang SDK. + - Create examples showcasing usage of the new rate limits SDK. + +1. **Team fan out for Satellite Services (Stage Groups)** + - Individual stage groups begin using the SDK built in Phase 5 for new limit and policies. + - Stage groups begin replacing historical ad hoc limit implementations with the SDK. ## Status diff --git a/doc/architecture/blueprints/remote_development/img/remote_dev_15_7_1.png b/doc/architecture/blueprints/remote_development/img/remote_dev_15_7_1.png Binary files differindex 330873380d4..aab946923e2 100644 --- a/doc/architecture/blueprints/remote_development/img/remote_dev_15_7_1.png +++ b/doc/architecture/blueprints/remote_development/img/remote_dev_15_7_1.png diff --git a/doc/architecture/blueprints/remote_development/index.md b/doc/architecture/blueprints/remote_development/index.md index 39ea2fb2948..38c3782f5e3 100644 --- a/doc/architecture/blueprints/remote_development/index.md +++ b/doc/architecture/blueprints/remote_development/index.md @@ -1,5 +1,5 @@ --- -status: proposed +status: ongoing creation-date: "2022-11-15" authors: [ "@vtak" ] coach: "@grzesiek" diff --git a/doc/architecture/blueprints/runner_scaling/index.md b/doc/architecture/blueprints/runner_scaling/index.md index 8eb6bfd2551..53401d80e34 100644 --- a/doc/architecture/blueprints/runner_scaling/index.md +++ b/doc/architecture/blueprints/runner_scaling/index.md @@ -1,8 +1,8 @@ --- -status: accepted +status: ongoing creation-date: "2022-01-19" authors: [ "@grzesiek", "@tmaczukin", "@josephburnett" ] -coach: "@kamil" +coach: [ "@ayufan", "@grzesiek" ] approvers: [ "@DarrenEastman" ] owning-stage: "~devops::verify" participating-stages: [] diff --git a/doc/architecture/blueprints/runner_tokens/index.md b/doc/architecture/blueprints/runner_tokens/index.md index 7a282649c5c..7d21dd594ad 100644 --- a/doc/architecture/blueprints/runner_tokens/index.md +++ b/doc/architecture/blueprints/runner_tokens/index.md @@ -1,8 +1,11 @@ --- -stage: Verify -group: Runner -comments: false -description: 'Next Runner Token Architecture' +status: ready +creation-date: "2022-10-27" +authors: [ "@pedropombeiro", "@tmaczukin" ] +coach: "@ayufan" +approvers: [ "@erushton" ] +owning-stage: "~devops::verify" +participating-stages: [] --- # Next GitLab Runner Token Architecture @@ -37,7 +40,7 @@ We call this new mechanism the "next GitLab Runner Token architecture". The proposal addresses the issues of a _single token per scope_ and _token storage_ by eliminating the need for a registration token. Runner creation happens -in the GitLab Runners settings page for the given scope, in the context of the logged-in user, +in the GitLab Runners settings page for the given scope, in the context of the authenticated user, which provides traceability. The page provides instructions to configure the newly-created runner in supported environments using the existing `gitlab-runner register` command. @@ -137,7 +140,7 @@ instance (for example if the runner is offline). In addition, we should add the following columns to `ci_runners`: -- a `user_id` column to keep track of who created a runner; +- a `creator_id` column to keep track of who created a runner; - a `registration_type` enum column to `ci_runners` to signal whether a runner has been created using the legacy `register` method, or the new UI-based method. Possible values are `:registration_token` and `:authenticated_user`. @@ -145,26 +148,25 @@ In addition, we should add the following columns to `ci_runners`: future uses that may not be apparent. ```sql -CREATE TABLE ci_runner ( +CREATE TABLE ci_runners ( ... - user_id bigint + creator_id bigint registration_type int8 ) ``` -The `ci_builds_runner_session` (or `ci_builds` or `ci_builds_metadata`) shall reference -`ci_runner_machines`. +The `ci_builds_metadata` table shall reference `ci_runner_machines`. We might consider a more efficient way to store `contacted_at` than updating the existing record. ```sql -CREATE TABLE ci_builds_runner_session ( +CREATE TABLE ci_builds_metadata ( ... runner_machine_id bigint NOT NULL ); CREATE TABLE ci_runner_machines ( - id integer NOT NULL, - machine_id character varying UNIQUE NOT NULL, + id bigint NOT NULL, + machine_xid character varying UNIQUE NOT NULL, contacted_at timestamp without time zone, version character varying, revision character varying, @@ -172,6 +174,7 @@ CREATE TABLE ci_runner_machines ( architecture character varying, ip_address character varying, executor_type smallint, + config jsonb DEFAULT '{}'::jsonb NOT NULL ); ``` @@ -238,7 +241,7 @@ future after the legacy registration system is removed, and runners have been up versions. Job pings from such legacy runners results in a `ci_runner_machines` record containing a -`<legacy>` `machine_id` field value. +`<legacy>` `machine_xid` field value. Not using the unique system ID means that all connected runners with the same token are notified, instead of just the runner matching the exact system identifier. While not ideal, this is @@ -246,8 +249,13 @@ not an issue per-se. ### `ci_runner_machines` record lifetime -New records are created when the runner pings the GitLab instance for new jobs, if a record matching -the `token`+`system_id` does not already exist. +New records are created in 2 situations: + +- when the runner calls the `POST /api/v4/runners/verify` endpoint as part of the +`gitlab-runner register` command, if the specified runner token is prefixed with `glrt-`. +This allows the frontend to determine whether the user has successfully completed the registration and take an +appropriate action; +- when GitLab is pinged for new jobs and a record matching the `token`+`system_id` does not already exist. Due to the time-decaying nature of the `ci_runner_machines` records, they are automatically cleaned after 7 days after the last contact from the respective runner. @@ -306,32 +314,34 @@ using PAT tokens for example - such that every runner is associated with an owne | Component | Milestone | Changes | |------------------|----------:|---------| -| GitLab Runner | `15.x` | Ensure a sidecar TOML file exists with a `system_id` value.<br/>Log new system ID values with `INFO` level as they get assigned. | -| GitLab Runner | `15.x` | Log unique system ID in the build logs. | -| GitLab Runner | `15.x` | Label Prometheus metrics with unique system ID. | -| GitLab Runner | `15.x` | Prepare `register` command to fail if runner server-side configuration options are passed together with a new `glrt-` token. | +| GitLab Runner | `15.7` | Ensure a sidecar TOML file exists with a `system_id` value.<br/>Log new system ID values with `INFO` level as they get assigned. | +| GitLab Runner | `15.7` | Log unique system ID in the build logs. | +| GitLab Runner | `15.9` | Label Prometheus metrics with unique system ID. | +| GitLab Runner | `15.8` | Prepare `register` command to fail if runner server-side configuration options are passed together with a new `glrt-` token. | ### Stage 3 - Database changes | Component | Milestone | Changes | |------------------|----------:|---------| -| GitLab Rails app | | Create database migration to add columns to `ci_runners` table. | -| GitLab Rails app | | Create database migration to add `ci_runner_machines` table. | -| GitLab Rails app | | Create database migration to add `ci_runner_machines.machine_id` foreign key to `ci_builds_runner_session` table. | -| GitLab Rails app | | Create database migrations to add `allow_runner_registration_token` setting to `application_settings` and `namespace_settings` tables (default: `true`). | -| GitLab Runner | | Use runner token + `system_id` JSON parameters in `POST /jobs/request` request in the [heartbeat request](https://gitlab.com/gitlab-org/gitlab/blob/c73c96a8ffd515295842d72a3635a8ae873d688c/lib/api/ci/helpers/runner.rb#L14-20) to update the `ci_runner_machines` cache/table. | +| GitLab Rails app | `%15.8` | Create database migration to add columns to `ci_runners` table. | +| GitLab Rails app | `%15.8` | Create database migration to add `ci_runner_machines` table. | +| GitLab Rails app | `%15.9` | Create database migration to add `ci_runner_machines.id` foreign key to `ci_builds_metadata` table. | +| GitLab Rails app | `%15.8` | Create database migrations to add `allow_runner_registration_token` setting to `application_settings` and `namespace_settings` tables (default: `true`). | +| GitLab Rails app | `%15.8` | Create database migration to add config column to `ci_runner_machines` table. | | GitLab Runner | | Start sending `system_id` value in `POST /jobs/request` request and other follow-up requests that require identifying the unique system. | | GitLab Rails app | | Create service similar to `StaleGroupRunnersPruneCronWorker` service to clean up `ci_runner_machines` records instead of `ci_runners` records.<br/>Existing service continues to exist but focuses only on legacy runners. | +| GitLab Rails app | | Create `ci_runner_machines` record in `POST /runners/verify` request if the runner token is prefixed with `glrt-`. | +| GitLab Rails app | | Use runner token + `system_id` JSON parameters in `POST /jobs/request` request in the [heartbeat request](https://gitlab.com/gitlab-org/gitlab/blob/c73c96a8ffd515295842d72a3635a8ae873d688c/lib/api/ci/helpers/runner.rb#L14-20) to update the `ci_runner_machines` cache/table. | ### Stage 4 - New UI | Component | Milestone | Changes | |------------------|----------:|---------| -| GitLab Runner | | Implement new GraphQL user-authenticated API to create a new runner. | -| GitLab Runner | | [Add prefix to newly generated runner authentication tokens](https://gitlab.com/gitlab-org/gitlab/-/issues/383198). | -| GitLab Rails app | | Implement UI to create new runner. | -| GitLab Rails app | | GraphQL changes to `CiRunner` type. | -| GitLab Rails app | | UI changes to runner details view (listing of platform, architecture, IP address, etc.) (?) | +| GitLab Rails app | `%15.10` | Implement new GraphQL user-authenticated API to create a new runner. | +| GitLab Rails app | `%15.10` | [Add prefix to newly generated runner authentication tokens](https://gitlab.com/gitlab-org/gitlab/-/issues/383198). | +| GitLab Rails app | `%15.10` | Implement UI to create new runner. | +| GitLab Rails app | `%15.10` | GraphQL changes to `CiRunner` type. | +| GitLab Rails app | `%15.10` | UI changes to runner details view (listing of platform, architecture, IP address, etc.) (?) | ### Stage 5 - Optional disabling of registration token diff --git a/doc/architecture/blueprints/secret_detection/index.md b/doc/architecture/blueprints/secret_detection/index.md new file mode 100644 index 00000000000..26551367a7c --- /dev/null +++ b/doc/architecture/blueprints/secret_detection/index.md @@ -0,0 +1,167 @@ +--- +status: proposed +creation-date: "2022-11-25" +authors: [ "@theoretick" ] +coach: "@DylanGriffith" +approvers: [ "@connorgilbert", "@amarpatel" ] +owning-stage: "~devops::secure" +participating-stages: [] +--- + +# Secret Detection as a platform-wide experience + +## Summary + +Today's secret detection feature is built around containerized scans of repositories +within a pipeline context. This feature is quite limited compared to where leaks +or compromised tokens may appear and should be expanded to include a much wider scope. + +Secret detection as a platform-wide experience encompasses detection across +platform features with high risk of secret leakage, including repository contents, +job logs, and project management features such as issues, epics, and MRs. + +## Motivation + +### Goals + +- Support asynchronous secret detection for: + - push events + - issuable creation + - issuable updates + - issuable comments + +### Non-Goals + +The current proposal is limited to asynchronous detection and alerting only. + +**Blocking** secrets on push events is high-risk to a critical path and +would require extensive performance profiling before implementing. See +[a recent example](https://gitlab.com/gitlab-org/gitlab/-/issues/246819#note_1164411983) +of a customer incident where this was attempted. + +Secret revocation and rotation is also beyond the scope of this new capability. + +Scanned object types beyond the scope of this MVC include: + +- Media types (JPEGs, PDFs,...) +- Snippets +- Wikis + +## Proposal + +To achieve scalable secret detection for a variety of domain objects a dedicated +scanning service must be created and deployed alongside the GitLab distribution. +This is referred to as the `SecretScanningService`. + +This service must be: + +- highly performant +- horizontally scalable +- generic in domain object scanning capability + +Platform-wide secret detection should be enabled by-default on GitLab SaaS as well +as self-managed instances. + +## Challenges + +- Secure authentication to GitLab.com infrastructure +- Performance of scanning against large blobs +- Performance of scanning against volume of domain objects (such as push frequency) + +## Design and implementation details + +The critical paths as outlined under [goals above](#goals) cover two major object +types: Git blobs (corresponding to push events) and arbitrary text blobs. + +The detection flow for push events relies on subscribing to the PostReceive hook +and enqueueing Sidekiq requests to the `SecretScanningService`. The `SecretScanningService` +service fetches enqueued refs, queries Gitaly for the ref blob contents, scans +the commit contents, and notifies the Rails application when a secret is detected. +See [Push event detection flow](#push-event-detection-flow) for sequence. + +The detection flow for arbitrary text blobs, such as issue comments, relies on +subscribing to `Notes::PostProcessService` (or equivalent service) and enqueueing +Sidekiq requests to the `SecretScanningService` to process the text blob by object type +and primary key of domain object. The `SecretScanningService` service fetches the +relevant text blob, scans the contents, and notifies the Rails application when a secret +is detected. + +The detection flow for job logs requires processing the log during archive to object +storage. See discussion [in this issue](https://gitlab.com/groups/gitlab-org/-/epics/8847#note_1116647883) +around scanning during streaming and the added complexity in buffering lookbacks +for arbitrary trace chunks. + +In any case of detection, the Rails application manually creates a vulnerability +using the `Vulnerabilities::ManuallyCreateService` to surface the finding within the +existing Vulnerability Management UI. + +See [technical discovery](https://gitlab.com/gitlab-org/gitlab/-/issues/376716) +for further background exploration. + +### Token types + +The existing Secret Detection configuration covers ~100 rules across a variety +of platforms. To reduce total cost of execution and likelihood of false positives +the dedicated service targets only well-defined tokens. A well-defined token is +defined as a token with a precise definition, most often a fixed substring prefix or +suffix and fixed length. + +Token types to identify in order of importance: + +1. Well-defined GitLab tokens (including Personal Access Tokens and Pipeline Trigger Tokens) +1. Verified Partner tokens (including AWS) +1. Remainder tokens currently included in Secret Detection CI configuration + +### Detection engine + +Our current secret detection offering utilizes [Gitleaks](https://github.com/zricethezav/gitleaks/) +for all secret scanning within pipeline contexts. By using its `--no-git` configuration +we can scan arbitrary text blobs outside of a repository context and continue to +utilize it for non-pipeline scanning. + +Given our existing familiarity with the tool and its extensibility, it should +remain our engine of choice. Changes to the detection engine are out of scope +unless benchmarking unveils performance concerns. + +### Push event detection flow + +```mermaid +sequenceDiagram + autonumber + actor User + User->>+Workhorse: git push + Workhorse->>+Gitaly: tcp + Gitaly->>+Rails: grpc + Sidekiq->>+Rails: poll job + Rails->>-Sidekiq: PostReceive worker + Sidekiq-->>+Sidekiq: enqueue PostReceiveSecretScanWorker + + Sidekiq->>+Rails: poll job + loop PostReceiveSecretScanWorker + Rails->>-Sidekiq: PostReceiveSecretScanWorker + Sidekiq->>+SecretScanningSvc: ScanBlob(ref) + SecretScanningSvc->>+Sidekiq: accepted + Note right of SecretScanningSvc: Scanning job enqueued + Sidekiq-->>+Rails: done + SecretScanningSvc->>+Gitaly: retrieve blob + SecretScanningSvc->>+SecretScanningSvc: scan blob + SecretScanningSvc->>+Rails: secret found + end +``` + +## Iterations + +1. Requirements definition for detection coverage and actions +1. PoC of secret scanning service + 1. gRPC commit retrieval from Gitaly + 1. blob scanning + 1. benchmarking of issuables, comments, job logs and blobs to gain confidence that the total costs will be viable +1. Implementation of secret scanning service MVC (targeting individual commits) +1. Security and readiness review +1. Deployment and monitoring +1. Implementation of secret scanning service MVC (targeting arbitrary text blobs) +1. Deployment and monitoring +1. High priority domain object rollout (priority `TBD`) + 1. Issuable comments + 1. Issuable bodies + 1. Job logs diff --git a/doc/architecture/blueprints/work_items/index.md b/doc/architecture/blueprints/work_items/index.md index 101fdbf4c2d..058282ec2b7 100644 --- a/doc/architecture/blueprints/work_items/index.md +++ b/doc/architecture/blueprints/work_items/index.md @@ -2,7 +2,7 @@ status: accepted creation-date: "2022-09-28" authors: [ "@ntepluhina" ] -coach: "@kamil" +coach: "@ayufan" approvers: [ "@gweaver" ] owning-stage: "~devops::plan" participating-stages: [] @@ -66,6 +66,7 @@ All Work Item types share the same pool of predefined widgets and are customized | start and due date | | | status\* | | | weight | | +| [notes](https://gitlab.com/gitlab-org/gitlab/-/issues/378949) | work_items_mvc | \* status is not currently a widget, but a part of the root work item, similar to title |
