diff options
Diffstat (limited to 'doc')
-rw-r--r-- | doc/api/users.md | 4 | ||||
-rw-r--r-- | doc/development/README.md | 1 | ||||
-rw-r--r-- | doc/development/pipelines.md | 177 | ||||
-rw-r--r-- | doc/development/redis.md | 43 |
4 files changed, 159 insertions, 66 deletions
diff --git a/doc/api/users.md b/doc/api/users.md index d77df9bc6c7..e147637ca59 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -385,7 +385,7 @@ Parameters: - `skip_confirmation` (optional) - Skip confirmation - true or false (default) - `external` (optional) - Flags the user as external - true or false (default) - `avatar` (optional) - Image file for user's avatar -- `private_profile` (optional) - User's profile is private - true or false (default) +- `private_profile` (optional) - User's profile is private - true, false (default), or null (will be converted to false) - `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user **(STARTER)** - `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user **(STARTER)** @@ -423,7 +423,7 @@ Parameters: - `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user - `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user - `avatar` (optional) - Image file for user's avatar -- `private_profile` (optional) - User's profile is private - true or false (default) +- `private_profile` (optional) - User's profile is private - true, false (default), or null (will be converted to false) - `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user **(STARTER)** - `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user **(STARTER)** - `note` (optional) - Admin notes for this user **(STARTER)** diff --git a/doc/development/README.md b/doc/development/README.md index f94da66085b..ba1714b2746 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -83,6 +83,7 @@ Complementary reads: - [Cycle Analytics development guide](cycle_analytics.md) - [Issue types vs first-class types](issue_types.md) - [Application limits](application_limits.md) +- [Redis guidelines](redis.md) ## Performance guides diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 42b4e7385dc..32b4c97cb62 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -19,9 +19,6 @@ The current stages are: <https://gitlab.com/gitlab-org/gitlab-foss>. - `prepare`: This stage includes jobs that prepare artifacts that are needed by jobs in subsequent stages. -- `quick-test`: This stage includes test jobs that should run first and fail the - pipeline early (currently used to run Geo tests when the branch name starts - with `geo-`, `geo/`, or ends with `-geo`). - `test`: This stage includes most of the tests, DB/migration jobs, and static analysis jobs. - `post-test`: This stage includes jobs that build reports or gather data from the `test` stage's jobs (e.g. coverage, Knapsack metadata etc.). @@ -138,54 +135,109 @@ duplicating the `if` conditions and `changes` patterns lists since they cannot b **If you update an `if` condition or `changes` patterns list, make sure to mass-update those across all the CI config files (i.e. `.gitlab/ci/*.yml`).** -### Canonical commits only +### Canonical/security namespace merge requests only -This condition limits jobs creation to commits under the `gitlab-org/` top-level group -on GitLab.com only. This is similar to the `.only:variables-canonical-dot-com` CI definition: +This condition limits jobs creation to merge requests under the `gitlab-org/` top-level group +on GitLab.com only (i.e. this won't run for `master`, stable or auto-deploy branches). +This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [merge_requests]` +CI definitions. -```yaml -.if-canonical-gitlab: &if-canonical-gitlab - if: '$CI_SERVER_HOST == "gitlab.com" && $CI_PROJECT_NAMESPACE =~ /^gitlab-org($|\/)/' -``` +The definition for `if-canonical-dot-com-gitlab-org-groups-merge-request` can be +seen in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/docs.gitlab-ci.yml>. -### Canonical merge requests only +### Canonical/security namespace tags only -Same as the "Canonical commits only" condition above but further limits jobs creation -to merge requests only (i.e. this won't run for `master`, stable or auto-deploy branches). -This is similar to the `.only:variables-canonical-dot-com` + `.except:refs-master-tags-stable-deploy` -CI definitions: +This condition limits jobs creation to tags under the `gitlab-org/` top-level group +on GitLab.com only. +This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [tags]` CI definition: -```yaml -.if-canonical-gitlab-merge-request: &if-canonical-gitlab-merge-request - if: '$CI_SERVER_HOST == "gitlab.com" && $CI_PROJECT_NAMESPACE =~ /^gitlab-org($|\/)/ && $CI_MERGE_REQUEST_IID' -``` +The definition for `if-canonical-dot-com-gitlab-org-groups-tag` can be seen in +<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/cng.gitlab-ci.yml>. + +### Canonical namespace `master` only + +This condition limits jobs creation to `master` pipelines for the `gitlab-org` top-level group +on GitLab.com only. +This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [master]` CI definition: + +The definition for `if-canonical-dot-com-gitlab-org-group-master-refs` can be +seen in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/pages.gitlab-ci.yml>. + +### Canonical namespace schedules only + +This condition limits jobs creation to scheduled pipelines for the `gitlab-org` top-level group +on GitLab.com only. +This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [schedules]` CI definition: + +The definition for `if-canonical-dot-com-gitlab-org-group-schedule` can be seen +in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml>. + +### Not canonical/security namespace + +This condition matches if the project isn't in the canonical/security namespace. +Useful to **not** create a job if the project is a fork, or in other words, when +a job should only run in the canonical projects. + +The definition for `if-not-canonical-namespace` can be seen in +<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>. + +### Not EE + +This condition matches if the project isn't EE. Useful to **not** create a job if +the project is GitLab, or in other words, when a job should only run in the GitLab +FOSS project. + +The definition for `if-not-ee` can be seen in +<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>. + +### Default refs only + +This condition is the equivalent of `.default-only`. + +The definition for `if-default-refs` can be seen in +<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>. + +### `master` refs only + +This condition is the equivalent of `only:refs: [master]`. + +The definition for `if-master-refs` can be seen in +<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>. ### Code changes patterns Similar patterns as for `.only:changes-code`: -```yaml -.code-patterns: &code-patterns - - ... -``` +The definition for `code-patterns` can be seen in +<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml>. ### QA changes patterns Similar patterns as for `.only:changes-qa`: -```yaml -.qa-patterns: &qa-patterns - - ... -``` +The definition for `qa-patterns` can be seen in +<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml>. + +### Docs changes patterns + +Similar patterns as for `.only:changes-docs`: + +The definition for `docs-patterns` can be seen in +<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/docs.gitlab-ci.yml>. ### Code and QA changes patterns Similar patterns as for `.only:changes-code-qa`: -```yaml -.code-qa-patterns: &code-qa-patterns - - ... -``` +The definition for `code-qa-patterns` can be seen in +<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/review.gitlab-ci.yml>. + +### Code, backstage and QA changes patterns + +Similar patterns as for `.only:changes-code-backstage-qa`: + +The definition for `code-backstage-qa-patterns` can be seen in +<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>. ## Directed acyclic graph @@ -195,21 +247,21 @@ execute jobs out of order for the following jobs: ```mermaid graph RL; A[setup-test-env]; - B["gitlab:assets:compile pull-push-cache<br/>(master only)"]; - C[gitlab:assets:compile pull-cache]; + B["gitlab:assets:compile pull-push-cache<br/>(canonical master only)"]; + C["gitlab:assets:compile pull-cache<br/>(canonical default refs only)"]; D["cache gems<br/>(master and tags only)"]; E[review-build-cng]; F[build-qa-image]; G[review-deploy]; G2["schedule:review-deploy<br/>(master only)"]; - H[karma]; - I[jest]; + I["karma, jest, webpack-dev-server, static-analysis"]; + I2["karma-foss, jest-foss<br/>(EE default refs only)"]; J["compile-assets pull-push-cache<br/>(master only)"]; + J2["compile-assets pull-push-cache foss<br/>(EE master only)"]; K[compile-assets pull-cache]; - L[webpack-dev-server]; + K2["compile-assets pull-cache foss<br/>(EE default refs only)"]; M[coverage]; - N[pages]; - O[static-analysis]; + N["pages (master only)"]; Q[package-and-qa]; S["RSpec<br/>(e.g. rspec unit pg9)"] T[retrieve-tests-metadata]; @@ -220,58 +272,55 @@ subgraph "`prepare` stage" C F K + K2 J + J2 T end subgraph "`test` stage" - D --> |needs| A; - H -.-> |needs and depends on| A; - H -.-> |needs and depends on| K; + D -.-> |needs| A; I -.-> |needs and depends on| A; I -.-> |needs and depends on| K; + I2 -.-> |needs and depends on| A; + I2 -.-> |needs and depends on| K; L -.-> |needs and depends on| A; - L -.-> |needs and depends on| K; - O -.-> |needs and depends on| A; - O -.-> |needs and depends on| K; S -.-> |needs and depends on| A; S -.-> |needs and depends on| K; S -.-> |needs and depends on| T; - downtime_check --> |needs and depends on| A; - db:* --> |needs| A; - gitlab:setup --> |needs| A; - downtime_check --> |needs and depends on| A; - graphql-docs-verify --> |needs| A; + L["db:*, gitlab:setup, graphql-docs-verify, downtime_check"] -.-> |needs| A; + end + +subgraph "`post-test` stage" + M --> |happens after| S end subgraph "`review-prepare` stage" - E --> |needs| C; - X["schedule:review-build-cng<br/>(master schedule only)"] --> |needs| C; + E -.-> |needs| C; + E2["schedule:review-build-cng<br/>(master schedule only)"] -.-> |needs| C; end subgraph "`review` stage" - G - G2 + G --> |happens after| E + G2 --> |happens after| E2 end subgraph "`qa` stage" - Q --> |needs| C; - Q --> |needs| F; - review-qa-smoke -.-> |needs and depends on| G; - review-qa-all -.-> |needs and depends on| G; - review-performance -.-> |needs and depends on| G; - X2["schedule:review-performance<br/>(master only)"] -.-> |needs and depends on| G2; - dast -.-> |needs and depends on| G; + Q -.-> |needs| C; + Q -.-> |needs| F; + QA1["review-qa-smoke, review-qa-all, review-performance, dast"] -.-> |needs and depends on| G; + QA2["schedule:review-performance<br/>(master only)"] -.-> |needs and depends on| G2; end -subgraph "`post-test` stage" - M - end +subgraph "`post-qa` stage" + PQA1["parallel-spec-reports"] -.-> |depends on `review-qa-all`| QA1; + end subgraph "`pages` stage" N -.-> |depends on| C; - N -.-> |depends on| H; + N -.-> |depends on karma| I; N -.-> |depends on| M; + N --> |happens after| PQA1 end ``` diff --git a/doc/development/redis.md b/doc/development/redis.md new file mode 100644 index 00000000000..a4a87155f5a --- /dev/null +++ b/doc/development/redis.md @@ -0,0 +1,43 @@ +# Redis guidelines + +GitLab uses [Redis](https://redis.io) for three distinct purposes: + +- Caching via `Rails.cache`. +- As a job processing queue with [Sidekiq](sidekiq_style_guide.md). +- To manage the shared application state. + +Every application process is configured to use the same Redis servers, so they +can be used for inter-process communication in cases where [PostgreSQL](sql.md) +is less appropriate, for example, transient state or data that is written much +more often than it is read. + +If [Geo](geo.md) is enabled, each Geo node gets its own, independent Redis +database. + +## Key naming + +Redis is a flat namespace with no hierarchy, which means we must pay attention +to key names to avoid collisions. Typically we use colon-separated elements to +provide a semblence of structure at application level. An example might be +`projects:1:somekey`. + +Although we split our Redis usage into three separate purposes, and those may +map to separate Redis servers in a [Highly Available](../administration/high_availability/redis.md) +configuration, the default Omnibus and GDK setups share a single Redis server. +This means that keys should **always** be globally unique across the three +purposes. + +It is usually better to use immutable identifiers - project ID rather than +full path, for instance - in Redis key names. If full path is used, the key will +stop being consulted if the project is renamed. If the contents of the key are +invalidated by a name change, it is better to include a hook that will expire +the entry, instead of relying on the key changing. + +We don't use [Redis Cluster](https://redis.io/topics/cluster-tutorial) at the +moment, but may wish to in the future: [#118820](https://gitlab.com/gitlab-org/gitlab/issues/118820). + +This imposes an additional constraint on naming: where GitLab is performing +operations that require several keys to be held on the same Redis server - for +instance, diffing two sets held in Redis - the keys should ensure that by +enclosing the changeable parts in curly braces, such as, `project:{1}:set_a` and +`project:{1}:set_b`. |