Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2022-09-20 02:18:09 +0300
committerGitLab Bot <gitlab-bot@gitlab.com>2022-09-20 02:18:09 +0300
commit6ed4ec3e0b1340f96b7c043ef51d1b33bbe85fde (patch)
treedc4d20fe6064752c0bd323187252c77e0a89144b /doc/development
parent9868dae7fc0655bd7ce4a6887d4e6d487690eeed (diff)
Add latest changes from gitlab-org/gitlab@15-4-stable-eev15.4.0-rc42
Diffstat (limited to 'doc/development')
-rw-r--r--doc/development/api_graphql_styleguide.md92
-rw-r--r--doc/development/api_styleguide.md6
-rw-r--r--doc/development/application_secrets.md1
-rw-r--r--doc/development/application_slis/index.md14
-rw-r--r--doc/development/architecture.md2
-rw-r--r--doc/development/audit_event_guide/index.md44
-rw-r--r--doc/development/auto_devops.md2
-rw-r--r--doc/development/backend/create_source_code_be/index.md3
-rw-r--r--doc/development/backend/ruby_style_guide.md114
-rw-r--r--doc/development/build_test_package.md2
-rw-r--r--doc/development/changelog.md2
-rw-r--r--doc/development/cicd/index.md2
-rw-r--r--doc/development/cicd/pipeline_wizard.md15
-rw-r--r--doc/development/cicd/templates.md5
-rw-r--r--doc/development/code_intelligence/index.md2
-rw-r--r--doc/development/code_review.md142
-rw-r--r--doc/development/contributing/index.md12
-rw-r--r--doc/development/contributing/merge_request_workflow.md104
-rw-r--r--doc/development/contributing/style_guides.md37
-rw-r--r--doc/development/database/add_foreign_key_to_existing_column.md2
-rw-r--r--doc/development/database/adding_database_indexes.md16
-rw-r--r--doc/development/database/batched_background_migrations.md6
-rw-r--r--doc/development/database/ci_mirrored_tables.md4
-rw-r--r--doc/development/database/client_side_connection_pool.md4
-rw-r--r--doc/development/database/database_debugging.md2
-rw-r--r--doc/development/database/loose_foreign_keys.md2
-rw-r--r--doc/development/database/multiple_databases.md2
-rw-r--r--doc/development/database/not_null_constraints.md9
-rw-r--r--doc/development/database/ordering_table_columns.md2
-rw-r--r--doc/development/database/strings_and_the_text_data_type.md2
-rw-r--r--doc/development/database/understanding_explain_plans.md18
-rw-r--r--doc/development/database_review.md14
-rw-r--r--doc/development/deprecation_guidelines/index.md13
-rw-r--r--doc/development/development_processes.md2
-rw-r--r--doc/development/distributed_tracing.md8
-rw-r--r--doc/development/documentation/feature_flags.md3
-rw-r--r--doc/development/documentation/index.md49
-rw-r--r--doc/development/documentation/redirects.md168
-rw-r--r--doc/development/documentation/restful_api_styleguide.md40
-rw-r--r--doc/development/documentation/review_apps.md2
-rw-r--r--doc/development/documentation/site_architecture/deployment_process.md9
-rw-r--r--doc/development/documentation/site_architecture/index.md2
-rw-r--r--doc/development/documentation/structure.md398
-rw-r--r--doc/development/documentation/styleguide/index.md50
-rw-r--r--doc/development/documentation/styleguide/word_list.md44
-rw-r--r--doc/development/documentation/testing.md2
-rw-r--r--doc/development/documentation/topic_types/concept.md46
-rw-r--r--doc/development/documentation/topic_types/index.md130
-rw-r--r--doc/development/documentation/topic_types/reference.md32
-rw-r--r--doc/development/documentation/topic_types/task.md65
-rw-r--r--doc/development/documentation/topic_types/troubleshooting.md56
-rw-r--r--doc/development/documentation/versions.md9
-rw-r--r--doc/development/ee_features.md12
-rw-r--r--doc/development/elasticsearch.md8
-rw-r--r--doc/development/emails.md6
-rw-r--r--doc/development/experiment_guide/index.md2
-rw-r--r--doc/development/export_csv.md4
-rw-r--r--doc/development/fe_guide/graphql.md4
-rw-r--r--doc/development/fe_guide/index.md3
-rw-r--r--doc/development/fe_guide/storybook.md10
-rw-r--r--doc/development/fe_guide/style/vue.md12
-rw-r--r--doc/development/fe_guide/tooling.md4
-rw-r--r--doc/development/fe_guide/troubleshooting.md2
-rw-r--r--doc/development/fe_guide/view_component.md39
-rw-r--r--doc/development/fe_guide/vue.md16
-rw-r--r--doc/development/fe_guide/vuex.md4
-rw-r--r--doc/development/fe_guide/widgets.md6
-rw-r--r--doc/development/feature_development.md1
-rw-r--r--doc/development/feature_flags/controls.md4
-rw-r--r--doc/development/feature_flags/index.md14
-rw-r--r--doc/development/fips_compliance.md4
-rw-r--r--doc/development/gemfile.md23
-rw-r--r--doc/development/geo.md5
-rw-r--r--doc/development/geo/proxying.md2
-rw-r--r--doc/development/git_object_deduplication.md4
-rw-r--r--doc/development/github_importer.md40
-rw-r--r--doc/development/gitlab_flavored_markdown/specification_guide/index.md143
-rw-r--r--doc/development/go_guide/dependencies.md4
-rw-r--r--doc/development/go_guide/index.md6
-rw-r--r--doc/development/import_export.md23
-rw-r--r--doc/development/import_project.md16
-rw-r--r--doc/development/integrations/jenkins.md2
-rw-r--r--doc/development/integrations/secure.md32
-rw-r--r--doc/development/internal_api/index.md2
-rw-r--r--doc/development/internal_users.md2
-rw-r--r--doc/development/lfs.md4
-rw-r--r--doc/development/logging.md4
-rw-r--r--doc/development/merge_request_concepts/index.md28
-rw-r--r--doc/development/merge_request_performance_guidelines.md2
-rw-r--r--doc/development/migration_style_guide.md2
-rw-r--r--doc/development/newlines_styleguide.md111
-rw-r--r--doc/development/packages/new_format_development.md4
-rw-r--r--doc/development/permissions.md36
-rw-r--r--doc/development/pipelines.md29
-rw-r--r--doc/development/policies.md2
-rw-r--r--doc/development/rails_update.md2
-rw-r--r--doc/development/rake_tasks.md14
-rw-r--r--doc/development/real_time.md2
-rw-r--r--doc/development/redis/new_redis_instance.md2
-rw-r--r--doc/development/reusing_abstractions.md25
-rw-r--r--doc/development/routing.md2
-rw-r--r--doc/development/ruby3_gotchas.md37
-rw-r--r--doc/development/scalability.md18
-rw-r--r--doc/development/sec/index.md69
-rw-r--r--doc/development/secure_coding_guidelines.md2
-rw-r--r--doc/development/service_ping/implement.md15
-rw-r--r--doc/development/service_ping/index.md17
-rw-r--r--doc/development/service_ping/metrics_instrumentation.md26
-rw-r--r--doc/development/service_ping/troubleshooting.md6
-rw-r--r--doc/development/shell_scripting_guide/index.md2
-rw-r--r--doc/development/sidekiq/compatibility_across_updates.md6
-rw-r--r--doc/development/sidekiq/idempotent_jobs.md2
-rw-r--r--doc/development/snowplow/index.md2
-rw-r--r--doc/development/snowplow/troubleshooting.md8
-rw-r--r--doc/development/sql.md4
-rw-r--r--doc/development/testing_guide/best_practices.md82
-rw-r--r--doc/development/testing_guide/end_to_end/best_practices.md6
-rw-r--r--doc/development/testing_guide/end_to_end/feature_flags.md14
-rw-r--r--doc/development/testing_guide/end_to_end/index.md70
-rw-r--r--doc/development/testing_guide/end_to_end/rspec_metadata_tests.md13
-rw-r--r--doc/development/testing_guide/flaky_tests.md2
-rw-r--r--doc/development/testing_guide/frontend_testing.md17
-rw-r--r--doc/development/testing_guide/index.md2
-rw-r--r--doc/development/testing_guide/testing_migrations_guide.md2
-rw-r--r--doc/development/value_stream_analytics.md178
-rw-r--r--doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md41
-rw-r--r--doc/development/work_items.md32
-rw-r--r--doc/development/workhorse/index.md2
128 files changed, 1988 insertions, 1227 deletions
diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md
index 0b36b9b2f2f..673ec692bf4 100644
--- a/doc/development/api_graphql_styleguide.md
+++ b/doc/development/api_graphql_styleguide.md
@@ -77,6 +77,63 @@ The complexity score of a query [can itself be queried for](../api/graphql/getti
Requests time out at 30 seconds.
+### Limit maximum field call count
+
+In some cases, you want to prevent the evaluation of a specific field on multiple parent nodes
+because it results in an N+1 query problem and there is no optimal solution. This should be
+considered an option of last resort, to be used only when methods such as
+[lookahead to preload associations](#look-ahead), or [using batching](graphql_guide/batchloader.md)
+have been considered.
+
+For example:
+
+```graphql
+# This usage is expected.
+query {
+ project {
+ environments
+ }
+}
+
+# This usage is NOT expected.
+# It results in N+1 query problem. EnvironmentsResolver can't use GraphQL batch loader in favor of GraphQL pagination.
+query {
+ projects {
+ nodes {
+ environments
+ }
+ }
+}
+```
+
+To prevent this, you can use the `Gitlab::Graphql::Limit::FieldCallCount` extension on the field:
+
+```ruby
+# This allows maximum 1 call to the `environments` field. If the field is evaluated on more than one node,
+# it raises an error.
+field :environments do
+ extension(::Gitlab::Graphql::Limit::FieldCallCount, limit: 1)
+ end
+```
+
+or you can apply the extension in a resolver class:
+
+```ruby
+module Resolvers
+ class EnvironmentsResolver < BaseResolver
+ extension(::Gitlab::Graphql::Limit::FieldCallCount, limit: 1)
+ # ...
+ end
+end
+```
+
+When you add this limit, make sure that the affected field's `description` is also updated accordingly. For example,
+
+```ruby
+field :environments,
+ description: 'Environments of the project. This field can only be resolved for one project in any single request.'
+```
+
## Breaking changes
The GitLab GraphQL API is [versionless](https://graphql.org/learn/best-practices/#versioning) which means
@@ -484,7 +541,7 @@ You can also
[change or remove Alpha items at any time](#breaking-change-exemptions) without needing to deprecate them. When the flag is removed, "release"
the schema item by removing its Alpha property to make it public.
-### Descriptions for feature flagged items
+### Descriptions for feature-flagged items
When using a feature flag to toggle the value or behavior of a schema item, the
`description` of the item must:
@@ -494,7 +551,11 @@ When using a feature flag to toggle the value or behavior of a schema item, the
- State what the field returns, or behavior is, when the feature flag is disabled (or
enabled, if more appropriate).
-Example of a feature-flagged field:
+### Examples of using feature flags
+
+#### Feature-flagged field
+
+A field value is toggled based on the feature flag state. A common use is to return `null` if the feature flag is disabled:
```ruby
field :foo, GraphQL::Types::String, null: true,
@@ -507,7 +568,10 @@ def foo
end
```
-Example of a feature-flagged argument:
+#### Feature-flagged argument
+
+An argument can be ignored, or have its value changed, based on the feature flag state.
+A common use is to ignore the argument when a feature flag is disabled:
```ruby
argument :foo, type: GraphQL::Types::String, required: false,
@@ -521,11 +585,23 @@ def resolve(args)
end
```
-### `feature_flag` property (deprecated)
+#### Feature-flagged mutation
-NOTE:
-This property is deprecated and should no longer be used. The property
-has been temporarily renamed to `_deprecated_feature_flag` and support for it will be removed in [#369202](https://gitlab.com/gitlab-org/gitlab/-/issues/369202).
+A mutation that cannot be performed due to a feature flag state is handled as a
+[non-recoverable mutation error](#failure-irrelevant-to-the-user). The error is returned at the top level:
+
+```ruby
+description 'Mutates an object. Does not mutate the object if ' \
+ '`my_feature_flag` feature flag is disabled.'
+
+def resolve(id: )
+ object = authorized_find!(id: id)
+
+ raise Gitlab::Graphql::Errors::ResourceNotAvailable, '`my_feature_flag` feature flag is disabled.' \
+ if Feature.disabled?(:my_feature_flag, object)
+ # ...
+end
+```
## Deprecating schema items
@@ -1611,7 +1687,7 @@ correctly rendered to the clients.
### Errors in mutations
-We encourage following the practice of
+We encourage following the practice of
[errors as data](https://graphql-ruby.org/mutations/mutation_errors) for mutations, which
distinguishes errors by who they are relevant to, defined by who can deal with
them.
diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md
index b72ef1bffc4..7f7d78bb58e 100644
--- a/doc/development/api_styleguide.md
+++ b/doc/development/api_styleguide.md
@@ -110,14 +110,14 @@ Model.create(foo: params[:foo])
With Grape v1.3+, Array types must be defined with a `coerce_with`
block, or parameters, fails to validate when passed a string from an
-API request. See the
+API request. See the
[Grape upgrading documentation](https://github.com/ruby-grape/grape/blob/master/UPGRADING.md#ensure-that-array-types-have-explicit-coercions)
for more details.
### Automatic coercion of nil inputs
Prior to Grape v1.3.3, Array parameters with `nil` values would
-automatically be coerced to an empty Array. However, due to
+automatically be coerced to an empty Array. However, due to
[this pull request in v1.3.3](https://github.com/ruby-grape/grape/pull/2040), this
is no longer the case. For example, suppose you define a PUT `/test`
request that has an optional parameter:
@@ -259,7 +259,7 @@ In situations where the same model has multiple entities in the API
discretion with applying this scope. It may be that you optimize for the
most basic entity, with successive entities building upon that scope.
-The `with_api_entity_associations` scope also
+The `with_api_entity_associations` scope also
[automatically preloads data](https://gitlab.com/gitlab-org/gitlab/-/blob/19f74903240e209736c7668132e6a5a735954e7c/app%2Fmodels%2Ftodo.rb#L34)
for `Todo` _targets_ when returned in the [to-dos API](../api/todos.md).
diff --git a/doc/development/application_secrets.md b/doc/development/application_secrets.md
index 06a38eb238a..93e43856b34 100644
--- a/doc/development/application_secrets.md
+++ b/doc/development/application_secrets.md
@@ -17,6 +17,7 @@ This page is a development guide for application secrets.
|`db_key_base` | The base key to encrypt the data for `attr_encrypted` columns |
|`openid_connect_signing_key` | The signing key for OpenID Connect |
| `encrypted_settings_key_base` | The base key to encrypt settings files with |
+| `ci_jwt_signing_key` | The base key for encrypting the `CI_JOB_JWT` and `CI_JOB_JWT_V2` predefined CI/CD variables |
## Where the secrets are stored
diff --git a/doc/development/application_slis/index.md b/doc/development/application_slis/index.md
index 27e69ff3445..cb2eb9b8d90 100644
--- a/doc/development/application_slis/index.md
+++ b/doc/development/application_slis/index.md
@@ -25,6 +25,7 @@ to be emitted from the rails application:
## Existing SLIs
1. [`rails_request_apdex`](rails_request_apdex.md)
+1. `global_search_apdex`
## Defining a new SLI
@@ -45,8 +46,8 @@ for clarity, they define different metric names:
As shown in this example, they can share a base name (`foo` in this example). We
recommend this when they refer to the same operation.
-Before the first scrape, it is important to have
-[initialized the SLI with all possible label-combinations](https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics).
+Before the first scrape, it is important to have
+[initialized the SLI with all possible label-combinations](https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics).
This avoid confusing results when using these counters in calculations.
To initialize an SLI, use the `.initialize_sli` class method, for
@@ -135,10 +136,7 @@ After that, add the following information:
into the error budgets for stage groups.
- `description`: a Markdown string explaining the SLI. It will
be shown on dashboards and alerts.
-- `kind`: the kind of indicator. Only `sliDefinition.apdexKind` is supported at the moment.
- Reach out in
- [this issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1395)
- if you want to implement an SLI for success or error rates.
+- `kind`: the kind of indicator. For example `sliDefinition.apdexKind`.
When done, run `make generate` to generate recording rules for
the new SLI. This command creates recordings for all services
@@ -152,9 +150,9 @@ When these changes are merged, and the aggregations in
the success ratio of the new aggregated metrics. For example:
```prometheus
-sum by (environment, stage, type)(gitlab_sli_aggregation:rails_request_apdex:apdex:success:rate_1h)
+sum by (environment, stage, type)(application_sli_aggregation:rails_request:apdex:success:rate_1h)
/
-sum by (environment, stage, type)(gitlab_sli_aggregation:rails_request_apdex:apdex:weight:rate_1h)
+sum by (environment, stage, type)(application_sli_aggregation:rails_request:apdex:weight:score_1h)
```
This shows the success ratio, which can guide you to set an
diff --git a/doc/development/architecture.md b/doc/development/architecture.md
index 10d6c0ae9c9..a813072a976 100644
--- a/doc/development/architecture.md
+++ b/doc/development/architecture.md
@@ -626,7 +626,7 @@ Mattermost is an open source, private cloud, Slack-alternative from <https://mat
- Layer: Core Service (Data)
- GitLab.com: [Storage Architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#storage-architecture)
-MinIO is an object storage server released under Apache License v2.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups, and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB.
+MinIO is an object storage server released under the GNU AGPL v3.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups, and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB.
#### NGINX
diff --git a/doc/development/audit_event_guide/index.md b/doc/development/audit_event_guide/index.md
index 0c66189a6f6..50d7eeed107 100644
--- a/doc/development/audit_event_guide/index.md
+++ b/doc/development/audit_event_guide/index.md
@@ -1,5 +1,5 @@
---
-stage: Manage
+stage: Govern
group: Compliance
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
@@ -31,7 +31,7 @@ To instrument an audit event, the following attributes should be provided:
| Attribute | Type | Required? | Description |
|:-------------|:---------------------|:----------|:------------------------------------------------------------------|
-| `name` | String | false | Action name to be audited. Used for error tracking |
+| `name` | String | false | Action name to be audited. Represents the [type of the event](#event-type-definitions). Used for error tracking |
| `author` | User | true | User who authors the change |
| `scope` | User, Project, Group | true | Scope which the audit event belongs to |
| `target` | Object | true | Target object being audited |
@@ -40,17 +40,15 @@ To instrument an audit event, the following attributes should be provided:
## How to instrument new Audit Events
-There are three ways of instrumenting audit events:
+1. Create a [YAML type definition](#add-a-new-audit-event-type) for the new audit event.
+1. Call `Gitlab::Audit::Auditor.audit`, passing an action block.
+
+The following ways of instrumenting audit events are deprecated:
- Create a new class in `ee/lib/ee/audit/` and extend `AuditEventService`
- Call `AuditEventService` after a successful action
-- Call `Gitlab::Audit::Auditor.audit` passing an action block
-
-This inconsistency leads to unexpected bugs, increases maintainer effort, and worsens the
-developer experience. Therefore, we suggest you use `Gitlab::Audit::Auditor` to
-instrument new audit events.
-With new service, we can instrument audit events in two ways:
+With `Gitlab::Audit::Auditor` service, we can instrument audit events in two ways:
- Using block for multiple events.
- Using standard method call for single events.
@@ -197,6 +195,34 @@ deactivate B
In addition to recording to the database, we also write these events to
[a log file](../../administration/logs/index.md#audit_jsonlog).
+## Event type definitions
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367847) in GitLab 15.4.
+
+All new audit events must have a type definition stored in `config/audit_events/types/` that contains a single source of truth for every auditable event in GitLab.
+
+### Add a new audit event type
+
+To add a new audit event type:
+
+1. Create a new file in `config/audit_events/types/` with the filename matching the name of the event type. For example, a definition for the event type triggered when a
+ user is added to a project might be stored in `config/audit_events/types/project_add_user.yml`.
+1. Add contents to the file that conform to the [schema](#schema) defined in `config/audit_events/types/type_schema.json`.
+1. Ensure that all calls to `Gitlab::Audit::Auditor` use the `name` defined in your file.
+
+### Schema
+
+| Field | Required | Description |
+| ----- | -------- |--------------|
+| `name` | yes | Unique, lowercase and underscored name describing the type of event. Must match the filename. |
+| `description` | yes | Human-readable description of how this event is triggered |
+| `group` | yes | Name of the group that introduced this audit event. For example, `manage::compliance` |
+| `introduced_by_issue` | yes | Issue URL that proposed the addition of this type |
+| `introduced_by_mr` | yes | MR URL that added this new type |
+| `milestone` | yes | Milestone in which this type was added |
+| `saved_to_database` | yes | Indicate whether to persist events to database and JSON logs |
+| `streamed` | yes | Indicate that events should be streamed to external services (if configured) |
+
## Event streaming
All events where the entity is a `Group` or `Project` are recorded in the audit log, and also streamed to one or more
diff --git a/doc/development/auto_devops.md b/doc/development/auto_devops.md
index 55ab234cc68..b9b8770207e 100644
--- a/doc/development/auto_devops.md
+++ b/doc/development/auto_devops.md
@@ -20,7 +20,7 @@ based on your project contents. When Auto DevOps is enabled for a
project, the user does not need to explicitly include any pipeline configuration
through a [`.gitlab-ci.yml` file](../ci/yaml/index.md).
-In the absence of a `.gitlab-ci.yml` file, the
+In the absence of a `.gitlab-ci.yml` file, the
[Auto DevOps CI/CD template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml)
is used implicitly to configure the pipeline for the project. This
template is a top-level template that includes other sub-templates,
diff --git a/doc/development/backend/create_source_code_be/index.md b/doc/development/backend/create_source_code_be/index.md
index e1ee78731de..a1322b3fa25 100644
--- a/doc/development/backend/create_source_code_be/index.md
+++ b/doc/development/backend/create_source_code_be/index.md
@@ -33,6 +33,9 @@ GitLab Shell handles Git SSH sessions for GitLab and modifies the list of author
For more information, [refer to the README](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/main/README.md).
for GitLab Shell.
+To learn about the reasoning behind our creation of `gitlab-sshd`, read the blog post
+[Why we implemented our own SSHD solution](https://about.gitlab.com/blog/2022/08/17/why-we-have-implemented-our-own-sshd-solution-on-gitlab-sass/).
+
## GitLab Rails
### Gitaly touch points
diff --git a/doc/development/backend/ruby_style_guide.md b/doc/development/backend/ruby_style_guide.md
index a9fee02a15a..6ba5b8dd2c5 100644
--- a/doc/development/backend/ruby_style_guide.md
+++ b/doc/development/backend/ruby_style_guide.md
@@ -20,6 +20,17 @@ See also [guidelines for reusing abstractions](../reusing_abstractions.md).
Everything listed here can be [reopened for discussion](https://about.gitlab.com/handbook/values/#disagree-commit-and-disagree).
+## String literals quoting
+
+Due to the sheer amount of work to rectify, we do not care whether string
+literals are single, or double quoted.
+
+Previous discussions include:
+
+- <https://gitlab.com/gitlab-org/gitlab-foss/-/issues/44234>
+- <https://gitlab.com/gitlab-org/gitlab-foss/-/issues/36076>
+- <https://gitlab.com/gitlab-org/gitlab/-/issues/198046>
+
## Instance variable access using `attr_reader`
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52351) in GitLab 14.1.
@@ -72,3 +83,106 @@ end
Public attributes should only be used if they are accessed outside of the class.
There is not a strong opinion on what strategy is used when attributes are only
accessed internally, as long as there is consistency in related code.
+
+## Newlines style guide
+
+This style guide recommends best practices for newlines in Ruby code.
+
+### Rule: separate code with newlines only to group together related logic
+
+```ruby
+# bad
+def method
+ issue = Issue.new
+
+ issue.save
+
+ render json: issue
+end
+```
+
+```ruby
+# good
+def method
+ issue = Issue.new
+ issue.save
+
+ render json: issue
+end
+```
+
+### Rule: separate code and block with newlines
+
+#### Newline before block
+
+```ruby
+# bad
+def method
+ issue = Issue.new
+ if issue.save
+ render json: issue
+ end
+end
+```
+
+```ruby
+# good
+def method
+ issue = Issue.new
+
+ if issue.save
+ render json: issue
+ end
+end
+```
+
+### Rule: Newline after block
+
+```ruby
+# bad
+def method
+ if issue.save
+ issue.send_email
+ end
+ render json: issue
+end
+```
+
+```ruby
+# good
+def method
+ if issue.save
+ issue.send_email
+ end
+
+ render json: issue
+end
+```
+
+#### Exception: no need for newline when code block starts or ends right inside another code block
+
+```ruby
+# bad
+def method
+
+ if issue
+
+ if issue.valid?
+ issue.save
+ end
+
+ end
+
+end
+```
+
+```ruby
+# good
+def method
+ if issue
+ if issue.valid?
+ issue.save
+ end
+ end
+end
+```
diff --git a/doc/development/build_test_package.md b/doc/development/build_test_package.md
index 4645bd02d9e..97dd24fc522 100644
--- a/doc/development/build_test_package.md
+++ b/doc/development/build_test_package.md
@@ -13,7 +13,7 @@ pipeline that can be used to trigger a pipeline in the Omnibus GitLab repository
that will create:
- A deb package for Ubuntu 16.04, available as a build artifact, and
-- A Docker image, which is pushed to the
+- A Docker image, which is pushed to the
[Omnibus GitLab container registry](https://gitlab.com/gitlab-org/omnibus-gitlab/container_registry)
(images titled `gitlab-ce` and `gitlab-ee` respectively and image tag is the
commit which triggered the pipeline).
diff --git a/doc/development/changelog.md b/doc/development/changelog.md
index c5b234069e3..c0296a6d75e 100644
--- a/doc/development/changelog.md
+++ b/doc/development/changelog.md
@@ -190,7 +190,7 @@ editor. Once closed, Git presents you with a new text editor instance to edit
the commit message of commit B. Add the trailer, then save and quit the editor.
If all went well, commit B is now updated.
-For more information about interactive rebases, take a look at
+For more information about interactive rebases, take a look at
[the Git documentation](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History).
---
diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md
index e8e116037de..7a2dfa01d1e 100644
--- a/doc/development/cicd/index.md
+++ b/doc/development/cicd/index.md
@@ -37,7 +37,7 @@ On the left side we have the events that can trigger a pipeline based on various
- When a [merge request is created or updated](../../ci/pipelines/merge_request_pipelines.md).
- When an MR is added to a [Merge Train](../../ci/pipelines/merge_trains.md#merge-trains).
- A [scheduled pipeline](../../ci/pipelines/schedules.md).
-- When project is [subscribed to an upstream project](../../ci/pipelines/multi_project_pipelines.md#trigger-a-pipeline-when-an-upstream-project-is-rebuilt).
+- When project is [subscribed to an upstream project](../../ci/pipelines/index.md#trigger-a-pipeline-when-an-upstream-project-is-rebuilt).
- When [Auto DevOps](../../topics/autodevops/index.md) is enabled.
- When GitHub integration is used with [external pull requests](../../ci/ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests).
- When an upstream pipeline contains a [bridge job](../../ci/yaml/index.md#trigger) which triggers a downstream pipeline.
diff --git a/doc/development/cicd/pipeline_wizard.md b/doc/development/cicd/pipeline_wizard.md
index 7a0b70bd8e8..227a49d85db 100644
--- a/doc/development/cicd/pipeline_wizard.md
+++ b/doc/development/cicd/pipeline_wizard.md
@@ -58,7 +58,7 @@ consists of 2-3 steps, for a total of 3-4 steps visible to the user.
```yaml
# ~/pipeline_wizard/templates/my_template.yml
-
+id: gitlab/my-template
title: Set up my specific tech pipeline
description: Here's two or three introductory sentences that help the user understand what this wizard is going to set up.
steps:
@@ -156,12 +156,13 @@ Webpack does not parse it as an Object.
In the root element of the template file, you can define the following properties:
-| Name | Required | Type | Description |
-|---------------|------------------------|--------|---------------------------------------------------------------------------------------|
-| `title` | **{check-circle}** Yes | string | The page title as displayed to the user. It becomes an `h1` heading above the wizard. |
-| `description` | **{check-circle}** Yes | string | The page description as displayed to the user. |
-| `filename` | **{dotted-circle}** No | string | The name of the file that is being generated. Defaults to `.gitlab-ci.yml`. |
-| `steps` | **{check-circle}** Yes | list | A list of [step definitions](#step-reference). |
+| Name | Required | Type | Description |
+|---------------|------------------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `id` | **{check-circle}** Yes | string | A unique template ID. This ID should follow a namespacing pattern, with a forward slash `/` as separator. Templates committed to GitLab source code should always begin with `gitlab`. For example: `gitlab/my-template` |
+| `title` | **{check-circle}** Yes | string | The page title as displayed to the user. It becomes an `h1` heading above the wizard. |
+| `description` | **{check-circle}** Yes | string | The page description as displayed to the user. |
+| `filename` | **{dotted-circle}** No | string | The name of the file that is being generated. Defaults to `.gitlab-ci.yml`. |
+| `steps` | **{check-circle}** Yes | list | A list of [step definitions](#step-reference). |
### `step` Reference
diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md
index 4ea7a9d960c..d0c56fb18bc 100644
--- a/doc/development/cicd/templates.md
+++ b/doc/development/cicd/templates.md
@@ -465,7 +465,10 @@ To add a metric definition for a new template:
- `name:` and `performance_indicator_type:`: Delete (not needed).
- `introduced_by_url:`: The URL of the MR adding the template.
- `data_source:`: Set to `redis_hll`.
- - All other fields that have no values: Set to empty strings (`''`).
+ - `description`: Add a short description of what this metric counts, for example: `Count of pipelines using the latest Auto Deploy template`
+ - `product_*`: Set to [section, stage, group, and feature category](https://about.gitlab.com/handbook/product/categories/#devops-stages)
+ as per the [metrics dictionary guide](../service_ping/metrics_dictionary.md#metrics-definition-and-validation).
+ If you are unsure what to use for these keywords, you can ask for help in the merge request.
- Add the following to the end of each file:
```yaml
diff --git a/doc/development/code_intelligence/index.md b/doc/development/code_intelligence/index.md
index a89730383e4..87697a5e252 100644
--- a/doc/development/code_intelligence/index.md
+++ b/doc/development/code_intelligence/index.md
@@ -35,7 +35,7 @@ sequenceDiagram
Workhorse-->>-Runner: request results
```
-1. The CI/CD job generates a document in an LSIF format (usually `dump.lsif`) using
+1. The CI/CD job generates a document in an LSIF format (usually `dump.lsif`) using
[an indexer](https://lsif.dev) for the language of a project. The format
[describes](https://github.com/sourcegraph/sourcegraph/blob/main/doc/code_intelligence/explanations/writing_an_indexer.md)
interactions between a method or function and its definitions or references. The
diff --git a/doc/development/code_review.md b/doc/development/code_review.md
index e9e546c6f9b..c320540401f 100644
--- a/doc/development/code_review.md
+++ b/doc/development/code_review.md
@@ -46,19 +46,28 @@ Read more about [author responsibilities](#the-responsibility-of-the-merge-reque
Domain experts are team members who have substantial experience with a specific technology,
product feature, or area of the codebase. Team members are encouraged to self-identify as
-domain experts and add it to their [team profiles](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/team_members/person/README.md).
+domain experts and add it to their
+[team profiles](https://about.gitlab.com/handbook/engineering/workflow/code-review/#how-to-self-identify-as-a-domain-expert).
When self-identifying as a domain expert, it is recommended to assign the MR changing the `.yml` file to be merged by an already established Domain Expert or a corresponding Engineering Manager.
We make the following assumption with regards to automatically being considered a domain expert:
-- Team members working in a specific stage/group (for example, create: source code) are considered domain experts for that area of the app they work on
-- Team members working on a specific feature (for example, search) are considered domain experts for that feature
+- Team members working in a specific stage/group (for example, create: source code) are considered domain experts for that area of the app they work on.
+- Team members working on a specific feature (for example, search) are considered domain experts for that feature.
We default to assigning reviews to team members with domain expertise.
When a suitable [domain expert](#domain-experts) isn't available, you can choose any team member to review the MR, or simply follow the [Reviewer roulette](#reviewer-roulette) recommendation.
-Team members' domain expertise can be viewed on the [engineering projects](https://about.gitlab.com/handbook/engineering/projects/) page or on the [GitLab team page](https://about.gitlab.com/company/team/).
+To find a domain expert:
+
+- View the list of team members who work in the [stage or group](https://about.gitlab.com/handbook/product/categories/#devops-stages) related to the merge request.
+- View team members' domain expertise on the [engineering projects](https://about.gitlab.com/handbook/engineering/projects/) page or on the [GitLab team page](https://about.gitlab.com/company/team/). Domains are self-identified, so use your judgment to map the changes on your merge request to a domain.
+- Look for team members who have contributed to the files in the merge request. View the logs by running `git log <file>`.
+- Look for team members who have reviewed the files. You can find the relevant merge request by:
+ 1. Getting the commit SHA by using `git log <file>`.
+ 1. Navigating to `https://gitlab.com/gitlab-org/gitlab/-/commit/<SHA>`.
+ 1. Selecting the related merge request shown for the commit.
### Reviewer roulette
@@ -92,6 +101,12 @@ page, with these behaviors:
- 3️⃣ - `:three:`
- 4️⃣ - `:four:`
- 5️⃣ - `:five:`
+
+ Review requests for merge requests that do not target the default branch of any
+ project under the [security group](https://gitlab.com/gitlab-org/security/) are
+ not counted. These MRs are usually backports, and maintainers or reviewers usually
+ do not need much time reviewing them.
+
- Team members whose Slack or [GitLab status](../user/profile/index.md#set-your-current-status) emoji
is 🔵 `:large_blue_circle:` are more likely to be picked. This applies to both reviewers and trainee maintainers.
- Reviewers with 🔵 `:large_blue_circle:` are two times as likely to be picked as other reviewers.
@@ -117,7 +132,7 @@ As an experiment, we want to introduce a `local` reviewer status for database re
focusing on work from a team/stage, but not outside of it. This helps to focus and build great domain
knowledge. We are not introducing changes to the reviewer roulette till we evaluate the impact and feedback from this
experiment. We ask to respect reviewers who decline reviews based on their focus on `local` reviews. For tracking purposes,
-please use in your personal YAML file entry: `- reviewer database local` instead of `- reviewer database`.
+please use in your personal YAML file entry: `- reviewer database local` instead of `- reviewer database`.
### Approval guidelines
@@ -125,40 +140,26 @@ As described in the section on the responsibility of the maintainer below, you
are recommended to get your merge request approved and merged by maintainers
with [domain expertise](#domain-experts).
-1. If your merge request includes `~backend` changes (*1*), it must be
- **approved by a [backend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_backend)**.
-1. If your merge request includes database migrations or changes to expensive queries (*2*), it must be
- **approved by a [database maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_database)**.
- Read the [database review guidelines](database_review.md) for more details.
-1. If your merge request includes `~frontend` changes (*1*), it must be
- **approved by a [frontend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_frontend)**.
-1. If your merge request includes (`~UX`) user-facing changes (*3*), it must be
- **approved by a [Product Designer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_UX)**.
- See the [design and user interface guidelines](contributing/design.md) for details.
-1. If your merge request includes adding a new JavaScript library (*1*)...
- - If the library significantly increases the
- [bundle size](https://gitlab.com/gitlab-org/frontend/playground/webpack-memory-metrics/-/blob/master/doc/report.md), it must
- be **approved by a [frontend foundations member](https://about.gitlab.com/direction/ecosystem/foundations/)**.
- - If the license used by the new library hasn't been approved for use in
- GitLab, the license must be **approved by a [legal department member](https://about.gitlab.com/handbook/legal/)**.
- More information about license compatibility can be found in our
- [GitLab Licensing and Compatibility documentation](licensing.md).
-1. If your merge request includes a new dependency or a file system change, it must be
- **approved by a [Distribution team member](https://about.gitlab.com/company/team/)**. See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/systems/distribution/#how-to-work-with-distribution) for more details.
-1. If your merge request includes documentation changes, it must be **approved
- by a [Technical writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments)**,
- based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages).
-1. If your merge request includes changes to development guidelines, follow the [review process](development_processes.md#development-guidelines-review) and get the approvals accordingly.
-1. If your merge request includes end-to-end **and** non-end-to-end changes (*4*), it must be **approved
- by a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors)**.
-1. If your merge request only includes end-to-end changes (*4*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)**
-1. If your merge request includes a new or updated [application limit](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits), it must be **approved by a [product manager](https://about.gitlab.com/company/team/)**.
-1. If your merge request includes Product Intelligence (telemetry or analytics) changes, it should be reviewed and approved by a [Product Intelligence engineer](https://gitlab.com/gitlab-org/analytics-section/product-intelligence/engineers).
-1. If your merge request includes an addition of, or changes to a [Feature spec](testing_guide/testing_levels.md#frontend-feature-tests), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa) or [Quality reviewer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_qa)**.
-1. If your merge request introduces a new service to GitLab (Puma, Sidekiq, Gitaly are examples), it must be **approved by a [product manager](https://about.gitlab.com/company/team/)**. See the [process for adding a service component to GitLab](adding_service_component.md) for details.
-1. If your merge request includes changes related to authentication or authorization, it must be **approved by a [Manage:Authentication and Authorization team member](https://about.gitlab.com/company/team/)**. Check the [code review section on the group page](https://about.gitlab.com/handbook/engineering/development/dev/manage/authentication-and-authorization/#additional-considerations) for more details. Patterns for files known to require review from the team are listed in the in the `Authentication and Authorization` section of the [`CODEOWNERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/CODEOWNERS) file, and the team will be listed in the approvers section of all merge requests that modify these files.
-
-- (*1*): Specs other than JavaScript specs are considered `~backend` code. Haml markup is considered `~frontend` code. However, Ruby code within Haml templates is considered `~backend` code.
+| If your merge request includes | It must be approved by a |
+| ------------------------------- | ------------------------ |
+| `~backend` changes (*1*) | [Backend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_backend). |
+| `~database` migrations or changes to expensive queries (*2*) | [Database maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_database). Refer to the [database review guidelines](database_review.md) for more details. |
+| `~workhorse` changes | [Workhorse maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_workhorse). |
+| `~frontend` changes (*1*) | [Frontend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_frontend). |
+| `~UX` user-facing changes (*3*) | [Product Designer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_UX). Refer to the [design and user interface guidelines](contributing/design.md) for details. |
+| Adding a new JavaScript library (*1*) | - [Frontend foundations member](https://about.gitlab.com/direction/ecosystem/foundations/) if the library significantly increases the [bundle size](https://gitlab.com/gitlab-org/frontend/playground/webpack-memory-metrics/-/blob/master/doc/report.md).<br/>- A [legal department member](https://about.gitlab.com/handbook/legal/) if the license used by the new library hasn't been approved for use in GitLab.<br/><br/>More information about license compatibility can be found in our [GitLab Licensing and Compatibility documentation](licensing.md). |
+| A new dependency or a file system change | - [Distribution team member](https://about.gitlab.com/company/team/). See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/systems/distribution/#how-to-work-with-distribution) for more details.<br/>- For Rubygems, request an [AppSec review](gemfile.md#request-an-appsec-review). |
+| `~documentation` changes | [Technical writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages). |
+| Changes to development guidelines | Follow the [review process](development_processes.md#development-guidelines-review) and get the approvals accordingly. |
+| End-to-end **and** non-end-to-end changes (*4*) | [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors). |
+| Only End-to-end changes (*4*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors) | [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa). |
+| A new or updated [application limit](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits) | [Product manager](https://about.gitlab.com/company/team/). |
+| Product Intelligence (telemetry or analytics) changes | [Product Intelligence engineer](https://gitlab.com/gitlab-org/analytics-section/product-intelligence/engineers). |
+| An addition of, or changes to a [Feature spec](testing_guide/testing_levels.md#frontend-feature-tests) | [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa) or [Quality reviewer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_qa). |
+| A new service to GitLab (Puma, Sidekiq, Gitaly are examples) | [Product manager](https://about.gitlab.com/company/team/). See the [process for adding a service component to GitLab](adding_service_component.md) for details. |
+| Changes related to authentication or authorization | [Manage:Authentication and Authorization team member](https://about.gitlab.com/company/team/). Check the [code review section on the group page](https://about.gitlab.com/handbook/engineering/development/dev/manage/authentication-and-authorization/#additional-considerations) for more details. Patterns for files known to require review from the team are listed in the in the `Authentication and Authorization` section of the [`CODEOWNERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/CODEOWNERS) file, and the team will be listed in the approvers section of all merge requests that modify these files. |
+
+- (*1*): Specs other than JavaScript specs are considered `~backend` code. Haml markup is considered `~frontend` code. However, Ruby code within Haml templates is considered `~backend` code. When in doubt, request both a frontend and backend review.
- (*2*): We encourage you to seek guidance from a database maintainer if your merge
request is potentially introducing expensive queries. It is most efficient to comment
on the line of code in question with the SQL queries so they can give their advice.
@@ -333,33 +334,29 @@ Because a maintainer's job only depends on their knowledge of the overall GitLab
codebase, and not that of any specific domain, they can review, approve, and merge
merge requests from any team and in any product area.
+Maintainers are the DRI of assuring that the acceptance criteria of a merge request are reasonably met.
+In general, [quality is everyone’s responsibility](https://about.gitlab.com/handbook/engineering/quality/),
+but maintainers of an MR are held responsible for **ensuring** that an MR meets those general quality standards.
+
+If a maintainer feels that an MR is substantial enough, or requires a [domain expert](#domain-experts),
+maintainers have the discretion to request a review from another reviewer, or maintainer. Here are some
+examples of maintainers proactively doing this during review:
+
+- <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82708#note_872325561>
+- <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38003#note_387981596>
+- <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/14017#note_178828088>
+
Maintainers do their best to also review the specifics of the chosen solution
before merging, but as they are not necessarily [domain experts](#domain-experts), they may be poorly
placed to do so without an unreasonable investment of time. In those cases, they
defer to the judgment of the author and earlier reviewers, in favor of focusing on their primary responsibilities.
-If a maintainer feels that an MR is substantial enough that it warrants a review from a [domain expert](#domain-experts),
-and it is unclear whether a domain expert have been involved in the reviews to date,
-they may request a [domain expert's](#domain-experts) review before merging the MR.
-
If a developer who happens to also be a maintainer was involved in a merge request
as a reviewer, it is recommended that they are not also picked as the maintainer to ultimately approve and merge it.
Maintainers should check before merging if the merge request is approved by the
required approvers. If still awaiting further approvals from others, remove yourself as a reviewer then `@` mention the author and explain why in a comment. Stay as reviewer if you're merging the code.
-Maintainers must check before merging if the merge request is introducing new
-vulnerabilities, by inspecting the list in the merge request
-[Security Widget](../user/application_security/index.md).
-When in doubt, a [Security Engineer](https://about.gitlab.com/company/team/) can be involved. The list of detected
-vulnerabilities must be either empty or containing:
-
-- dismissed vulnerabilities in case of false positives
-- vulnerabilities converted to issues
-
-Maintainers should **never** dismiss vulnerabilities to "empty" the list,
-without duly verifying them.
-
Note that certain merge requests may target a stable branch. These are rare
events. These types of merge requests cannot be merged by the Maintainer.
Instead, these should be sent to the [Release Manager](https://about.gitlab.com/community/release-managers/).
@@ -418,7 +415,7 @@ first time.
codebase. Thorough descriptions help all reviewers understand your request
and test effectively.
- If you know your change depends on another being merged first, note it in the
- description and set a [merge request dependency](../user/project/merge_requests/merge_request_dependencies.md).
+ description and set a [merge request dependency](../user/project/merge_requests/dependencies.md).
- Be grateful for the reviewer's suggestions. ("Good call. I'll make that change.")
- Don't take it personally. The review is of the code, not of you.
- Explain why the code exists. ("It's like that because of these reasons. Would
@@ -489,7 +486,7 @@ experience, refactors the existing code). Then:
optionally resolve within the merge request or follow-up at a later stage.
- There's a [Chrome/Firefox add-on](https://gitlab.com/conventionalcomments/conventional-comments-button) which you can use to apply [Conventional Comment](https://conventionalcomments.org/) prefixes.
- Ensure there are no open dependencies. Check [linked issues](../user/project/issues/related_issues.md) for blockers. Clarify with the authors
-if necessary. If blocked by one or more open MRs, set an [MR dependency](../user/project/merge_requests/merge_request_dependencies.md).
+if necessary. If blocked by one or more open MRs, set an [MR dependency](../user/project/merge_requests/dependencies.md).
- After a round of line notes, it can be helpful to post a summary note such as
"Looks good to me", or "Just a couple things to address."
- Let the author know if changes are required following your review.
@@ -507,24 +504,25 @@ Before taking the decision to merge:
before merging. A comment must be posted if the MR is merged with any failed job.
- If the MR contains both Quality and non-Quality-related changes, the MR should be merged by the relevant maintainer for user-facing changes (backend, frontend, or database) after the Quality related changes are approved by a Software Engineer in Test.
-If a merge request is fundamentally ready, but needs only trivial fixes (such as
-typos), consider demonstrating a [bias for action](https://about.gitlab.com/handbook/values/#bias-for-action)
-by making those changes directly without going back to the author. You can do this by
-using the [suggest changes](../user/project/merge_requests/reviews/suggestions.md) feature to apply
-your own suggestions to the merge request. Note that:
-
-- If the changes are not straightforward, please prefer allowing the author to make the change.
-- **Before applying suggestions**, edit the merge request to make sure
- [squash and merge](../user/project/merge_requests/squash_and_merge.md#squash-and-merge)
- is enabled, otherwise, the pipeline's Danger job fails.
- - If a merge request does not have squash and merge enabled, and it
- has more than one commit, then see the note below about rewriting
- commit history.
-
-Authors are not authorized to merge their own merge requests and need to seek another maintainer to merge.
+At least one maintainer must approve an MR before it can be merged. MR authors and
+people who add commits to an MR are not authorized to approve the merge request,
+so they must seek a maintainer who has not contributed to the MR to approve the MR before it can be merged.
+
This policy is in place to satisfy the CHG-04 control of the GitLab
[Change Management Controls](https://about.gitlab.com/handbook/engineering/security/security-assurance/security-compliance/guidance/change-management.html).
+To implement this policy in `gitlab-org/gitlab`, we have enabled the following
+settings to ensure MRs get an approval from a top-level CODEOWNERS maintainer:
+
+- [Prevent approval by author](../user/project/merge_requests/approvals/settings.md#prevent-approval-by-author).
+- [Prevent approvals by users who add commits](../user/project/merge_requests/approvals/settings.md#prevent-approvals-by-users-who-add-commits).
+- [Prevent editing approval rules in merge requests](../user/project/merge_requests/approvals/settings.md#prevent-editing-approval-rules-in-merge-requests).
+- [Remove all approvals when commits are added to the source branch](../user/project/merge_requests/approvals/settings.md#remove-all-approvals-when-commits-are-added-to-the-source-branch)
+
+ There are scenarios such as rebasing locally or applying suggestions that are considered
+ the same as adding a commit and could reset existing approvals. Approvals are not removed
+ when rebasing from the UI or with the [`/rebase` quick action](../user/project/quick_actions.md).
+
When ready to merge:
WARNING:
diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md
index 6999ffe810e..2ff51e765a3 100644
--- a/doc/development/contributing/index.md
+++ b/doc/development/contributing/index.md
@@ -99,10 +99,14 @@ If you would like to contribute to GitLab:
could speed them up.
- Consult the [Contribution Flow](#contribution-flow) section to learn the process.
-If you have any questions or need help visit [Getting Help](https://about.gitlab.com/get-help/) to
-learn how to communicate with GitLab. We have a [Gitter channel for contributors](https://gitter.im/gitlab/contributors),
-however we favor
-[asynchronous communication](https://about.gitlab.com/handbook/communication/#internal-communication) over real time communication.
+### Communication channels
+
+If you have any questions or need help, visit [Getting Help](https://about.gitlab.com/get-help/) to learn how to
+communicate with the GitLab community. GitLab prefers [asynchronous communication](https://about.gitlab.com/handbook/communication/#internal-communication) over real-time communication.
+
+We do encourage you to connect and hang out with us. GitLab has a Gitter room dedicated for [contributors](https://gitter.im/gitlab/contributors), which is bridged with our
+internal Slack. We actively monitor this channel. There is also a community-run [Discord server](https://discord.gg/S4cwz9sR8u) where you can
+find other contributors in the `#contributors` channel.
Thanks for your contribution!
diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md
index faa1642d50a..0e9ac569558 100644
--- a/doc/development/contributing/merge_request_workflow.md
+++ b/doc/development/contributing/merge_request_workflow.md
@@ -35,12 +35,23 @@ and see the [Development section](../../index.md) for the required guidelines.
## Merge request guidelines for contributors
-If you find an issue, please submit a merge request with a fix or improvement, if
-you can, and include tests. If you don't know how to fix the issue but can write a test
-that exposes the issue, we will accept that as well. In general, bug fixes that
-include a regression test are merged quickly, while new features without proper
-tests might be slower to receive feedback. The workflow to make a merge
-request is as follows:
+If you find an issue, please submit a merge request with a fix or improvement,
+if you can, and include tests.
+
+If the change is non-trivial, we encourage you to
+start a discussion with [a product manager or a member of the team](https://about.gitlab.com/handbook/product/categories/).
+You can do
+this by tagging them in an MR before submitting the code for review. Talking
+to team members can be helpful when making design decisions. Communicating the
+intent behind your changes can also help expedite merge request reviews.
+
+If
+you don't know how to fix the issue but can write a test that exposes the
+issue, we will accept that as well. In general, bug fixes that include a
+regression test are merged quickly. New features without proper tests
+might be slower to receive feedback.
+
+To create a merge request:
1. [Fork](../../user/project/repository/forking_workflow.md) the project into
your personal namespace (or group) on GitLab.com.
@@ -199,48 +210,27 @@ To reach the definition of done, the merge request must create no regressions an
- Verified as working in production on GitLab.com.
- Verified as working for self-managed instances.
-If a regression occurs, we prefer you revert the change. We break the definition of done into two phases: [MR Merge](#mr-merge) and [Production use](#production-use).
+If a regression occurs, we prefer you revert the change.
Your contribution is *incomplete* until you have made sure it meets all of these
requirements.
-### MR Merge
+### Functionality
-1. Clear title and description explaining the relevancy of the contribution.
1. Working and clean code that is commented where needed.
1. The change is evaluated to [limit the impact of far-reaching work](https://about.gitlab.com/handbook/engineering/development/#reducing-the-impact-of-far-reaching-work).
-1. Testing:
-
- - [Unit, integration, and system tests](../testing_guide/index.md) that all pass
- on the CI server.
- - Peer member testing is optional but recommended when the risk of a change is high.
- This includes when the changes are [far-reaching](https://about.gitlab.com/handbook/engineering/development/#reducing-the-impact-of-far-reaching-work)
- or are for [components critical for security](../code_review.md#security).
- - Description includes any steps or setup required to ensure reviewers can view the changes you've made (for example, include any information about feature flags).
- - Regressions and bugs are covered with tests that reduce the risk of the issue happening
- again.
- - For tests that use Capybara, read
- [how to write reliable, asynchronous integration tests](https://thoughtbot.com/blog/write-reliable-asynchronous-integration-tests-with-capybara).
- - [Black-box tests/end-to-end tests](../testing_guide/testing_levels.md#black-box-tests-at-the-system-level-aka-end-to-end-tests)
- added if required. Please contact [the quality team](https://about.gitlab.com/handbook/engineering/quality/#teams)
- with any questions.
- - The change is tested in a review app where possible and if appropriate.
-1. In case of UI changes:
-
- - Use available components from the GitLab Design System,
- [Pajamas](https://design.gitlab.com/).
- - The MR must include *Before* and *After* screenshots if UI changes are made.
- - If the MR changes CSS classes, please include the list of affected pages, which
- can be found by running `grep css-class ./app -R`.
+1. [Performance guidelines](../merge_request_performance_guidelines.md) have been followed.
+1. [Secure coding guidelines](https://gitlab.com/gitlab-com/gl-security/security-guidelines) have been followed.
+1. [Application and rate limit guidelines](../merge_request_application_and_rate_limit_guidelines.md) have been followed.
+1. [Documented](../documentation/index.md) in the `/doc` directory.
1. If your MR touches code that executes shell commands, reads or opens files, or
handles paths to files on disk, make sure it adheres to the
[shell command guidelines](../shell_commands.md)
1. [Code changes should include observability instrumentation](../code_review.md#observability-instrumentation).
1. If your code needs to handle file storage, see the [uploads documentation](../uploads/index.md).
-1. If your merge request adds one or more migrations:
- - Make sure to execute all migrations on a fresh database before the MR is reviewed.
- If the review leads to large changes in the MR, execute the migrations again
- after the review is complete.
- - Write tests for more complex migrations.
+1. If your merge request adds one or more migrations, make sure to execute all migrations on a fresh database
+ before the MR is reviewed.
+ If the review leads to large changes in the MR, execute the migrations again
+ after the review is complete.
1. If your merge request adds new validations to existing models, to make sure the
data processing is backwards compatible:
@@ -259,12 +249,37 @@ requirements.
[self-managed instances](../../subscriptions/self_managed/index.md), so keep
that in mind for any data implications with your merge request.
+### Testing
+
+1. [Unit, integration, and system tests](../testing_guide/index.md) that all pass
+ on the CI server.
+1. Peer member testing is optional but recommended when the risk of a change is high.
+ This includes when the changes are [far-reaching](https://about.gitlab.com/handbook/engineering/development/#reducing-the-impact-of-far-reaching-work)
+ or are for [components critical for security](../code_review.md#security).
+1. Regressions and bugs are covered with tests that reduce the risk of the issue happening
+ again.
+1. For tests that use Capybara, read
+ [how to write reliable, asynchronous integration tests](https://thoughtbot.com/blog/write-reliable-asynchronous-integration-tests-with-capybara).
+1. [Black-box tests/end-to-end tests](../testing_guide/testing_levels.md#black-box-tests-at-the-system-level-aka-end-to-end-tests)
+ added if required. Please contact [the quality team](https://about.gitlab.com/handbook/engineering/quality/#teams)
+ with any questions.
+1. The change is tested in a review app where possible and if appropriate.
1. Code affected by a feature flag is covered by [automated tests with the feature flag enabled and disabled](../feature_flags/index.md#feature-flags-in-tests), or both
states are tested as part of peer member testing or as part of the rollout plan.
-1. [Performance guidelines](../merge_request_performance_guidelines.md) have been followed.
-1. [Secure coding guidelines](https://gitlab.com/gitlab-com/gl-security/security-guidelines) have been followed.
-1. [Application and rate limit guidelines](../merge_request_application_and_rate_limit_guidelines.md) have been followed.
-1. [Documented](../documentation/index.md) in the `/doc` directory.
+1. If your merge request adds one or more migrations, write tests for more complex migrations.
+
+### UI changes
+
+1. Use available components from the GitLab Design System,
+ [Pajamas](https://design.gitlab.com/).
+1. The MR must include *Before* and *After* screenshots if UI changes are made.
+1. If the MR changes CSS classes, please include the list of affected pages, which
+ can be found by running `grep css-class ./app -R`.
+
+### Description of changes
+
+1. Clear title and description explaining the relevancy of the contribution.
+1. Description includes any steps or setup required to ensure reviewers can view the changes you've made (for example, include any information about feature flags).
1. [Changelog entry added](../changelog.md), if necessary.
1. If your merge request introduces changes that require additional steps when
installing GitLab from source, add them to `doc/install/installation.md` in
@@ -274,10 +289,13 @@ requirements.
`doc/update/upgrading_from_source.md` in the same merge request. If these
instructions are specific to a version, add them to the "Version specific
upgrading instructions" section.
-1. Reviewed by relevant reviewers, and all concerns are addressed for Availability, Regressions, and Security. Documentation reviews should take place as soon as possible, but they should not block a merge request.
+
+### Approval
+
1. The [MR acceptance checklist](../code_review.md#acceptance-checklist) has been checked as confirmed in the MR.
1. Create an issue in the [infrastructure issue tracker](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues) to inform the Infrastructure department when your contribution is changing default settings or introduces a new setting, if relevant.
1. An agreed-upon [rollout plan](https://about.gitlab.com/handbook/engineering/development/processes/rollout-plans/).
+1. Reviewed by relevant reviewers, and all concerns are addressed for Availability, Regressions, and Security. Documentation reviews should take place as soon as possible, but they should not block a merge request.
1. Your merge request has at least 1 approval, but depending on your changes
you might need additional approvals. Refer to the [Approval guidelines](../code_review.md#approval-guidelines).
- You don't have to select any specific approvers, but you can if you really want
@@ -286,6 +304,8 @@ requirements.
### Production use
+The following items are checked after the merge request has been merged:
+
1. Confirmed to be working in staging before implementing the change in production, where possible.
1. Confirmed to be working in the production with no new [Sentry](https://about.gitlab.com/handbook/engineering/monitoring/#sentry) errors after the contribution is deployed.
1. Confirmed that the [rollout plan](https://about.gitlab.com/handbook/engineering/development/processes/rollout-plans/) has been completed.
diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md
index 7a4ebbdbadf..2e696cf517b 100644
--- a/doc/development/contributing/style_guides.md
+++ b/doc/development/contributing/style_guides.md
@@ -146,13 +146,20 @@ reduces the aforementioned [bike-shedding](https://en.wiktionary.org/wiki/bikesh
To that end, we encourage creation of new RuboCop rules in the codebase.
-We currently maintain Cops across several Ruby code bases, and not all of them are
+We maintain Cops across several Ruby code bases, and not all of them are
specific to the GitLab application.
When creating a new cop that could be applied to multiple applications, we encourage you
to add it to our [GitLab Styles](https://gitlab.com/gitlab-org/gitlab-styles) gem.
If the Cop targets rules that only apply to the main GitLab application,
it should be added to [GitLab](https://gitlab.com/gitlab-org/gitlab) instead.
+#### RuboCop node pattern
+
+When creating [node patterns](https://docs.rubocop.org/rubocop-ast/node_pattern.html) to match
+Ruby's AST, you can use [`scripts/rubocop-parse`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/rubocop-parse)
+to display the AST of a Ruby expression, in order to help you create the matcher.
+See also [!97024](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97024).
+
### Resolving RuboCop exceptions
When the number of RuboCop exceptions exceed the default [`exclude-limit` of 15](https://docs.rubocop.org/rubocop/1.2/usage/basic_usage.html#command-line-flags),
@@ -222,6 +229,34 @@ We're following [Ciro Santilli's Markdown Style Guide](https://cirosantilli.com/
See the dedicated [Documentation Style Guide](../documentation/styleguide/index.md).
+### Guidelines for good practices
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36576/) in GitLab 13.2 as GitLab Development documentation.
+
+*Good practice* examples demonstrate encouraged ways of writing code while
+comparing with examples of practices to avoid. These examples are labeled as
+*Bad* or *Good*. In GitLab development guidelines, when presenting the cases,
+it's recommended to follow a *first-bad-then-good* strategy. First demonstrate
+the *Bad* practice (how things *could* be done, which is often still working
+code), and then how things *should* be done better, using a *Good* example. This
+is typically an improved example of the same code.
+
+Consider the following guidelines when offering examples:
+
+- First, offer the *Bad* example, and then the *Good* one.
+- When only one bad case and one good case is given, use the same code block.
+- When more than one bad case or one good case is offered, use separated code
+ blocks for each. With many examples being presented, a clear separation helps
+ the reader to go directly to the good part. Consider offering an explanation
+ (for example, a comment, or a link to a resource) on why something is bad
+ practice.
+- Better and best cases can be considered part of the good cases' code block.
+ In the same code block, precede each with comments: `# Better` and `# Best`.
+
+Although the bad-then-good approach is acceptable for the GitLab development
+guidelines, do not use it for user documentation. For user documentation, use
+*Do* and *Don't*. For examples, see the [Pajamas Design System](https://design.gitlab.com/content/punctuation/).
+
## Python
See the dedicated [Python Development Guidelines](../python_guide/index.md).
diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md
index 8a8fe3c0a1e..4be3296b2bb 100644
--- a/doc/development/database/add_foreign_key_to_existing_column.md
+++ b/doc/development/database/add_foreign_key_to_existing_column.md
@@ -71,7 +71,7 @@ Migration file for adding `NOT VALID` foreign key:
```ruby
class AddNotValidForeignKeyToEmailsUser < Gitlab::Database::Migration[2.0]
def up
- add_concurrent_foreign_key :emails, :users, on_delete: :cascade, validate: false
+ add_concurrent_foreign_key :emails, :users, column: :user_id, on_delete: :cascade, validate: false
end
def down
diff --git a/doc/development/database/adding_database_indexes.md b/doc/development/database/adding_database_indexes.md
index 8abd7c8298e..774703bd54f 100644
--- a/doc/development/database/adding_database_indexes.md
+++ b/doc/development/database/adding_database_indexes.md
@@ -328,8 +328,8 @@ asynchronously during weekend hours. Due to generally lower traffic and fewer de
index destruction can proceed at a lower level of risk.
1. [Schedule the index to be removed](#schedule-the-index-to-be-removed).
-1. [Verify the MR was deployed and the index exists in production](#verify-the-mr-was-deployed-and-the-index-exists-in-production).
-1. [Add a migration to create the index synchronously](#add-a-migration-to-create-the-index-synchronously).
+1. [Verify the MR was deployed and the index exists in production](#verify-the-mr-was-deployed-and-the-index-no-longer-exists-in-production).
+1. [Add a migration to destroy the index synchronously](#add-a-migration-to-destroy-the-index-synchronously).
### Schedule the index to be removed
@@ -357,21 +357,21 @@ to remove them.
You must test the database index changes locally before creating a merge request.
-### Verify the MR was deployed and the index exists in production
+### Verify the MR was deployed and the index no longer exists in production
You can verify if the MR was deployed to GitLab.com with
`/chatops run auto_deploy status <merge_sha>`. To verify the existence of
the index, you can:
- Use a meta-command in `#database-lab`, for example: `\d <index_name>`.
- - Make sure the index is not [`invalid`](https://www.postgresql.org/docs/12/sql-createindex.html#:~:text=The%20psql%20%5Cd%20command%20will%20report%20such%20an%20index%20as%20INVALID).
+- Make sure the index no longer exists
- Ask someone in `#database` to check if the index exists.
- If you have access, you can verify directly on production or in a
production clone.
### Add a migration to destroy the index synchronously
-After you verify the index exists in the production database, create a second
+After you verify the index no longer exists in the production database, create a second
merge request that removes the index synchronously. The schema changes must be
updated and committed to `structure.sql` in this second merge request.
The synchronous migration results in a no-op on GitLab.com, but you should still add the
@@ -379,7 +379,7 @@ migration as expected for other installations. For example, to
create the second migration for the previous asynchronous example:
**WARNING:**
-Verify that the index no longer exist in production before merging a second migration with `remove_concurrent_index_by_name`.
+Verify that the index no longer exists in production before merging a second migration with `remove_concurrent_index_by_name`.
If the second migration is deployed before the index has been destroyed,
the index is destroyed synchronously when the second migration executes.
@@ -395,7 +395,7 @@ def up
end
def down
- add_concurrent_index :ci_builds, :some_column, INDEX_NAME
+ add_concurrent_index :ci_builds, :some_column, name: INDEX_NAME
end
```
@@ -403,7 +403,7 @@ end
To test changes for removing an index, use the asynchronous index helpers on your local environment:
-1. Enable the feature flags by running `Feature.enable(:database_async_index_destruction)` and `Feature.enable(:database_reindexing)` in the Rails console.
+1. Enable the feature flags by running `Feature.enable(:database_reindexing)` in the Rails console.
1. Run `bundle exec rails db:migrate` which should create an entry in the `postgres_async_indexes` table.
1. Run `bundle exec rails gitlab:db:reindex` destroy the index asynchronously.
1. To verify the index, open the PostgreSQL console by using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md)
diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md
index edb22fcf436..192cd0d3e49 100644
--- a/doc/development/database/batched_background_migrations.md
+++ b/doc/development/database/batched_background_migrations.md
@@ -233,7 +233,7 @@ class CopyColumnUsingBackgroundMigrationJob < BatchedMigrationJob
end
```
-### Additional filters
+## Additional filters
By default, when creating background jobs to perform the migration, batched background migrations
iterate over the full specified table. This iteration is done using the
@@ -276,6 +276,10 @@ In the second (filtered) example, we know exactly 100 will be updated with each
end
```
+ NOTE:
+ For EE migrations that define `scope_to`, ensure the module extends `ActiveSupport::Concern`.
+ Otherwise, records are processed without taking the scope into consideration.
+
1. In the post-deployment migration, enqueue the batched background migration:
```ruby
diff --git a/doc/development/database/ci_mirrored_tables.md b/doc/development/database/ci_mirrored_tables.md
index 06f0087fafe..1d285e607fa 100644
--- a/doc/development/database/ci_mirrored_tables.md
+++ b/doc/development/database/ci_mirrored_tables.md
@@ -10,9 +10,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w
As part of the database [decomposition work](https://gitlab.com/groups/gitlab-org/-/epics/6168),
which had the goal of splitting the single database GitLab is using, into two databases: `main` and
-`ci`, came the big challenge of
+`ci`, came the big challenge of
[removing all joins between the `main` and the `ci` tables](multiple_databases.md#removing-joins-between-ci-and-non-ci-tables).
-That is because PostgreSQL doesn't support joins between tables that belong to different databases.
+That is because PostgreSQL doesn't support joins between tables that belong to different databases.
However, some core application models in the main database are queried very often by the CI side.
For example:
diff --git a/doc/development/database/client_side_connection_pool.md b/doc/development/database/client_side_connection_pool.md
index 3cd0e836a8d..3143391a553 100644
--- a/doc/development/database/client_side_connection_pool.md
+++ b/doc/development/database/client_side_connection_pool.md
@@ -10,7 +10,7 @@ Ruby processes accessing the database through
ActiveRecord, automatically calculate the connection-pool size for the
process based on the concurrency.
-Because of the way [Ruby on Rails manages database connections](#connection-lifecycle),
+Because of the way [Ruby on Rails manages database connections](#connection-lifecycle),
it is important that we have at
least as many connections as we have threads. While there is a 'pool'
setting in [`database.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/database.yml.postgresql), it is not very practical because you need to
@@ -28,7 +28,7 @@ because connections are instantiated lazily.
## Troubleshooting connection-pool issues
-The connection-pool usage can be seen per environment in the
+The connection-pool usage can be seen per environment in the
[connection-pool saturation dashboard](https://dashboards.gitlab.net/d/alerts-sat_rails_db_connection_pool/alerts-rails_db_connection_pool-saturation-detail?orgId=1).
If the connection-pool is too small, this would manifest in
diff --git a/doc/development/database/database_debugging.md b/doc/development/database/database_debugging.md
index 5921dc942f2..591e526cc96 100644
--- a/doc/development/database/database_debugging.md
+++ b/doc/development/database/database_debugging.md
@@ -4,7 +4,7 @@ group: Database
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
-# Troubleshooting and Debugging Database
+# Troubleshooting and debugging the database
This section is to help give some copy-pasta you can use as a reference when you
run into some head-banging database problems.
diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md
index 8dbccf048d7..0af12939629 100644
--- a/doc/development/database/loose_foreign_keys.md
+++ b/doc/development/database/loose_foreign_keys.md
@@ -221,7 +221,7 @@ ON DELETE CASCADE;
```
The migration must run after the `DELETE` trigger is installed and the loose
-foreign key definition is deployed. As such, it must be a
+foreign key definition is deployed. As such, it must be a
[post-deployment migration](post_deployment_migrations.md) dated after the migration for the
trigger. If the foreign key is deleted earlier, there is a good chance of
introducing data inconsistency which needs manual cleanup:
diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md
index 31fc454f8a7..034a2c2e438 100644
--- a/doc/development/database/multiple_databases.md
+++ b/doc/development/database/multiple_databases.md
@@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Multiple Databases
To allow GitLab to scale further we
-[decomposed the GitLab application database into multiple databases](https://gitlab.com/groups/gitlab-org/-/epics/6168).
+[decomposed the GitLab application database into multiple databases](https://gitlab.com/groups/gitlab-org/-/epics/6168).
The two databases are `main` and `ci`. GitLab supports being run with either one database or two databases.
On GitLab.com we are using two separate databases.
diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md
index 9b3d017b09f..cd2adc3ca28 100644
--- a/doc/development/database/not_null_constraints.md
+++ b/doc/development/database/not_null_constraints.md
@@ -53,8 +53,13 @@ end
## Add a `NOT NULL` constraint to an existing column
-Adding `NOT NULL` to existing database columns requires multiple steps split into at least two
-different releases:
+Adding `NOT NULL` to existing database columns usually requires multiple steps split into at least two
+different releases. If your table is small enough that you don't need to
+use a background migration, you can include all these in the same merge
+request. We recommend to use separate migrations to reduce
+transaction durations.
+
+The steps required are:
1. Release `N.M` (current release)
diff --git a/doc/development/database/ordering_table_columns.md b/doc/development/database/ordering_table_columns.md
index 7cd3d4fb208..a16df6a4499 100644
--- a/doc/development/database/ordering_table_columns.md
+++ b/doc/development/database/ordering_table_columns.md
@@ -117,7 +117,7 @@ divided into fixed size chunks as follows:
This means that excluding the variable sized data and tuple header, we need at
least 8 * 6 = 48 bytes per row.
-We can optimise this by using the following column order instead:
+We can optimize this by using the following column order instead:
| Column | Type | Size |
|:--------------|:----------------------------|:---------|
diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md
index e2e1191018b..4b5d1fc8f72 100644
--- a/doc/development/database/strings_and_the_text_data_type.md
+++ b/doc/development/database/strings_and_the_text_data_type.md
@@ -148,7 +148,7 @@ to update the `title_html` with a title that has more than 1024 characters, the
a database error.
Adding or removing a constraint to an existing attribute requires that any application changes are
-deployed _first_,
+deployed _first_,
otherwise servers still in the old version of the application
[may try to update the attribute with invalid values](../multi_version_compatibility.md#ci-artifact-uploads-were-failing).
For these reasons, `add_text_limit` should run in a post-deployment migration.
diff --git a/doc/development/database/understanding_explain_plans.md b/doc/development/database/understanding_explain_plans.md
index 446a84d5232..c3cb408b35f 100644
--- a/doc/development/database/understanding_explain_plans.md
+++ b/doc/development/database/understanding_explain_plans.md
@@ -252,7 +252,7 @@ A scan on an index that required retrieving some data from the table.
Bitmap scans fall between sequential scans and index scans. These are typically
used when we would read too much data from an index scan, but too little to
-perform a sequential scan. A bitmap scan uses what is known as a
+perform a sequential scan. A bitmap scan uses what is known as a
[bitmap index](https://en.wikipedia.org/wiki/Bitmap_index) to perform its work.
The [source code of PostgreSQL](https://gitlab.com/postgres/postgres/blob/REL_11_STABLE/src/include/nodes/plannodes.h#L441)
@@ -295,9 +295,9 @@ because the previous node produced 36 rows.
This means that nested loops can quickly slow the query down if the various
child nodes keep producing many rows.
-## Optimising queries
+## Optimizing queries
-With that out of the way, let's see how we can optimise a query. Let's use the
+With that out of the way, let's see how we can optimize a query. Let's use the
following query as an example:
```sql
@@ -453,7 +453,7 @@ this works is that now PostgreSQL no longer needs to apply a `Filter`, as the
index only contains `twitter` values that are not empty.
Keep in mind that you shouldn't just add partial indexes every time you want to
-optimise a query. Every index has to be updated for every write, and they may
+optimize a query. Every index has to be updated for every write, and they may
require quite a bit of space, depending on the amount of indexed data. As a
result, first check if there are any existing indexes you may be able to reuse.
If there aren't any, check if you can perhaps slightly change an existing one to
@@ -471,10 +471,10 @@ buffer numbers. [Database Lab Engine](#database-lab-engine) guarantees that the
identical to production (and overall number of buffers is the same as on production),
but difference in cache state and I/O speed may lead to different timings.
-## Queries that can't be optimised
+## Queries that can't be optimized
-Now that we have seen how to optimise a query, let's look at another query that
-we might not be able to optimise:
+Now that we have seen how to optimize a query, let's look at another query that
+we might not be able to optimize:
```sql
EXPLAIN (ANALYZE, BUFFERS)
@@ -546,7 +546,7 @@ improve this query, other than _not_ running it at all.
What is important here is that while some may recommend to straight up add an
index the moment you see a sequential scan, it is _much more important_ to first
understand what your query does, how much data it retrieves, and so on. After
-all, you can not optimise something you do not understand.
+all, you can not optimize something you do not understand.
### Cardinality and selectivity
@@ -567,7 +567,7 @@ using an index is not worth it, because it would produce almost no unique rows.
## Rewriting queries
-So the above query can't really be optimised as-is, or at least not much. But
+So the above query can't really be optimized as-is, or at least not much. But
what if we slightly change the purpose of it? What if instead of retrieving all
projects with `visibility_level` 0 or 20, we retrieve those that a user
interacted with somehow?
diff --git a/doc/development/database_review.md b/doc/development/database_review.md
index 2decd304103..14d73437c36 100644
--- a/doc/development/database_review.md
+++ b/doc/development/database_review.md
@@ -116,9 +116,9 @@ the following preparations into account.
- Ensure that the Database Dictionary is updated as [documented](database/database_dictionary.md).
- Make migrations reversible by using the `change` method or include a `down` method when using `up`.
- Include either a rollback procedure or describe how to rollback changes.
-- Add the output of both migrating (`db:migrate`) and rolling back (`db:rollback`) for all migrations into the MR description.
- - Ensure the down method reverts the changes in `db/structure.sql`.
- - Update the migration output whenever you modify the migrations during the review process.
+- Check that the [`db:check-migrations`](database/dbcheck-migrations-job.md) pipeline job has run successfully and the migration rollback behaves as expected.
+ - Ensure the `db:check-schema` job has run successfully and no unexpected schema changes are introduced in a rollback. This job may only trigger a warning if the schema was changed.
+ - Verify that the previously mentioned jobs continue to succeed whenever you modify the migrations during the review process.
- Add tests for the migration in `spec/migrations` if necessary. See [Testing Rails migrations at GitLab](testing_guide/testing_migrations_guide.md) for more details.
- When [high-traffic](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3) tables are involved in the migration, use the [`enable_lock_retries`](migration_style_guide.md#retry-mechanism-when-acquiring-database-locks) method to enable lock-retries. Review the relevant [examples in our documentation](migration_style_guide.md#usage-with-transactional-migrations) for use cases and solutions.
- Ensure RuboCop checks are not disabled unless there's a valid reason to.
@@ -181,9 +181,11 @@ Include in the MR description:
- When providing query plans, make sure it hits enough data:
- You can use a GitLab production replica to test your queries on a large scale,
through the `#database-lab` Slack channel or through [ChatOps](database/understanding_explain_plans.md#chatops).
- - Usually, the `gitlab-org` namespace (`namespace_id = 9970`) and the
- `gitlab-org/gitlab-foss` (`project_id = 13083`) or the `gitlab-org/gitlab` (`project_id = 278964`)
- projects provide enough data to serve as a good example.
+ - To produce a query plan with enough data, you can use the IDs of:
+ - The `gitlab-org` namespace (`namespace_id = 9970`), for queries involving a group.
+ - The `gitlab-org/gitlab-foss` (`project_id = 13083`) or the `gitlab-org/gitlab` (`project_id = 278964`) projects, for queries involving a project.
+ - The `gitlab-qa` user (`user_id = 1614863`), for queries involving a user.
+ - Optionally, you can also use your own `user_id`, or the `user_id` of a user with a long history within the project or group being used to generate the query plan.
- That means that no query plan should return 0 records or less records than the provided limit (if a limit is included). If a query is used in batching, a proper example batch with adequate included results should be identified and provided.
- If your queries belong to a new feature in GitLab.com and thus they don't return data in production:
- You may analyze the query and to provide the plan from a local environment.
diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md
index f0364f60d38..1e9d3ebda77 100644
--- a/doc/development/deprecation_guidelines/index.md
+++ b/doc/development/deprecation_guidelines/index.md
@@ -11,16 +11,23 @@ to GitLab features.
## Terminology
+<!--
+If updating these definitions, be sure to update them in the handbook as well:
+https://about.gitlab.com/handbook/product/gitlab-the-product/#definitions
+-->
+
**Deprecation**:
+- Required before ending support for a feature or removing a feature.
- Feature not recommended for use.
- Development restricted to Priority 1 / Severity 1 bug fixes.
- Will be removed in a future major release.
-- Begins after a deprecation announcement outlining an end-of-support date.
+- Begins after a deprecation announcement outlining an end-of-support or removal date.
- Ends after the end-of-support date or removal date has passed.
**End of Support**:
+- Optional step before removal.
- Feature usage strongly discouraged.
- No support or fixes provided.
- No longer tested internally.
@@ -28,6 +35,10 @@ to GitLab features.
- Begins after an end-of-support date has passed.
- Ends after all relevant code has been removed.
+[Announcing an End of Support period](https://about.gitlab.com/handbook/marketing/blog/release-posts/#announcing-an-end-of-support-period)
+should only be used in special circumstances and is not recommended for general use.
+Most features should be deprecated and then removed.
+
**Removal**:
- Feature usage impossible.
diff --git a/doc/development/development_processes.md b/doc/development/development_processes.md
index e199aedd3f5..27ebe98bc63 100644
--- a/doc/development/development_processes.md
+++ b/doc/development/development_processes.md
@@ -67,7 +67,7 @@ Some changes affect more than one group. For example:
- Changes to [code review guidelines](code_review.md).
- Changes to [commit message guidelines](contributing/merge_request_workflow.md#commit-messages-guidelines).
-- Changes to guidelines in [feature flags in development of GitLab](feature_flags/).
+- Changes to guidelines in [feature flags in development of GitLab](feature_flags/index.md).
- Changes to [feature flags documentation guidelines](documentation/feature_flags.md).
In these cases, use the following workflow:
diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md
index f49d024095d..9d62f2061ca 100644
--- a/doc/development/distributed_tracing.md
+++ b/doc/development/distributed_tracing.md
@@ -73,13 +73,13 @@ In this example, we have the following hypothetical values:
- `driver`: the driver such a Jaeger.
- `param_name`, `param_value`: these are driver specific configuration values. Configuration
- parameters for Jaeger are documented [further on in this document](#2-configure-the-gitlab_tracing-environment-variable)
+ parameters for Jaeger are documented [further on in this document](#2-configure-the-gitlab_tracing-environment-variable)
they should be URL encoded.
Multiple values should be separated by `&` characters like a URL.
## Using Jaeger in the GitLab Development Kit
-The first tracing implementation that GitLab supports is Jaeger, and the
+The first tracing implementation that GitLab supports is Jaeger, and the
[GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit/) supports distributed tracing with
Jaeger out-of-the-box.
@@ -116,7 +116,7 @@ Jaeger has many configuration options, but is very easy to start in an "all-in-o
memory for trace storage (and is therefore non-persistent). The main advantage of "all-in-one" mode
being ease of use.
-For more detailed configuration options, refer to the
+For more detailed configuration options, refer to the
[Jaeger documentation](https://www.jaegertracing.io/docs/1.9/getting-started/).
#### Using Docker
@@ -201,7 +201,7 @@ If `GITLAB_TRACING` is not configured correctly, this issue is logged:
```
By default, GitLab ships with the Jaeger tracer, but other tracers can be included at compile time.
-Details of how this can be done are included in the
+Details of how this can be done are included in the
[LabKit tracing documentation](https://pkg.go.dev/gitlab.com/gitlab-org/labkit/tracing).
If no log messages about tracing are emitted, the `GITLAB_TRACING` environment variable is likely
diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md
index 89e54183e50..87d2930dcb5 100644
--- a/doc/development/documentation/feature_flags.md
+++ b/doc/development/documentation/feature_flags.md
@@ -24,7 +24,8 @@ When you document feature flags, you must:
## Add version history text
-When the state of a flag changes (for example, disabled by default to enabled by default), add the change to the version history.
+When the state of a flag changes (for example, disabled by default to enabled by default), add the change to the
+[version history](versions.md#add-a-version-history-item).
Possible version history entries are:
diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md
index ee439e93011..73c1874f09e 100644
--- a/doc/development/documentation/index.md
+++ b/doc/development/documentation/index.md
@@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
description: Learn how to contribute to GitLab Documentation.
---
-# GitLab Documentation guidelines
+# GitLab documentation
The GitLab documentation is [intended as the single source of truth (SSOT)](https://about.gitlab.com/handbook/documentation/) for information about how to configure, use, and troubleshoot GitLab. The documentation contains use cases and usage instructions for every GitLab feature, organized by product area and subject. This includes topics and workflows that span multiple GitLab features and the use of GitLab with other applications.
@@ -42,7 +42,7 @@ run only the jobs that match the type of contribution. If your contribution cont
**only** documentation changes, then only documentation-related jobs run, and
the pipeline completes much faster than a code contribution.
-If you are submitting documentation-only changes to Omnibus or Charts,
+If you are submitting documentation-only changes to Omnibus, Charts, or Operator,
the fast pipeline is not determined automatically. Instead, create branches for
docs-only merge requests using the following guide:
@@ -240,9 +240,9 @@ Every GitLab instance includes documentation at `/help` (`https://gitlab.example
that matches the version of the instance. For example, <https://gitlab.com/help>.
The documentation available online at <https://docs.gitlab.com> is deployed every
-four hours from the default branch of [GitLab, Omnibus, Runner, and Charts](#source-files-and-rendered-web-locations).
+hour from the default branch of [GitLab, Omnibus, Runner, and Charts](#source-files-and-rendered-web-locations).
After a merge request that updates documentation is merged, it is available online
-in 4 hours or less.
+in an hour or less.
However, it's only available at `/help` on self-managed instances in the next released
version. The date an update is merged can impact which self-managed release the update
@@ -380,6 +380,47 @@ is made, Danger Bot leaves a comment with further instructions about the documen
process. This is configured in the `Dangerfile` in the GitLab repository under
[/danger/documentation/](https://gitlab.com/gitlab-org/gitlab/-/tree/master/danger/documentation).
+## Help and feedback section
+
+This section ([introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/319) in GitLab 11.4)
+is displayed at the end of each document and can be omitted by adding a key into
+the front matter:
+
+```yaml
+---
+feedback: false
+---
+```
+
+The default is to leave it there. If you want to omit it from a document, you
+must check with a technical writer before doing so.
+
+## Disqus
+
+We have integrated the docs site with Disqus (introduced by
+[!151](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/151)),
+allowing our users to post comments.
+
+To omit only the comments from the feedback section, use the following key in
+the front matter:
+
+```yaml
+---
+comments: false
+---
+```
+
+We're hiding comments only in main index pages, such as [the main documentation index](../../index.md),
+since its content is too broad to comment on. Before omitting Disqus, you must
+check with a technical writer.
+
+Note that after adding `feedback: false` to the front matter, it will omit
+Disqus, therefore, don't add both keys to the same document.
+
+The click events in the feedback section are tracked with Google Tag Manager.
+The conversions can be viewed on Google Analytics by navigating to
+**Behavior > Events > Top events > docs**.
+
## Automatic screenshot generator
You can now set up an automatic screenshot generator to take and compress screenshots with the
diff --git a/doc/development/documentation/redirects.md b/doc/development/documentation/redirects.md
index 4c748924c67..1bc697f2878 100644
--- a/doc/development/documentation/redirects.md
+++ b/doc/development/documentation/redirects.md
@@ -16,12 +16,7 @@ description: Learn how to contribute to GitLab Documentation.
# Redirects in GitLab documentation
When you move, rename, or delete a page, you must add a redirect. Redirects reduce
-how often users get 404s when visiting the documentation site from out-of-date links, like:
-
-- Bookmarks
-- Links from external sites
-- Links from old blog posts
-- Links in the documentation site global navigation
+how often users get 404s when they visit the documentation site from out-of-date links.
Add a redirect to ensure:
@@ -36,9 +31,11 @@ Add a redirect to ensure:
Be sure to assign a technical writer to any merge request that moves, renames, or deletes a page.
Technical Writers can help with any questions and can review your change.
+## Types of redirects
+
There are two types of redirects:
-- [Redirect added into the documentation files themselves](#add-a-redirect), for users who
+- [Redirects added into the documentation files themselves](#redirect-to-a-page-that-already-exists), for users who
view the docs in `/help` on self-managed instances. For example,
[`/help` on GitLab.com](https://gitlab.com/help). These must be added in the same
MR that renames or moves a doc. Redirects to internal pages expire after three months
@@ -52,96 +49,109 @@ Expired redirect files are removed from the documentation projects by the
[`clean_redirects` Rake task](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/raketasks.md#clean-up-redirects),
as part of the Technical Writing team's [monthly tasks](https://gitlab.com/gitlab-org/technical-writing/-/blob/main/.gitlab/issue_templates/tw-monthly-tasks.md).
-## Add a redirect
+## Redirect to a page that already exists
+
+To redirect a page to another page in the same repository:
+
+1. In the Markdown file that you want to direct to a new location:
+
+ - Delete all of the content.
+ - Add this content:
+
+ ```markdown
+ ---
+ redirect_to: '../newpath/to/file/index.md'
+ remove_date: 'YYYY-MM-DD'
+ ---
+
+ This document was moved to [another location](../path/to/file/index.md).
+
+ <!-- This redirect file can be deleted after <YYYY-MM-DD>. -->
+ <!-- Redirects that point to other docs in the same project expire in three months. -->
+ <!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
+ <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
+ ```
+
+ - Replace both instances of `../newpath/to/file/index.md` with the new file path.
+ - Replace both instances of `YYYY-MM-DD` with the expiration date, as explained in the template.
+
+1. If the page has Disqus comments, follow [the steps for pages with Disqus comments](#redirections-for-pages-with-disqus-comments).
+1. If the page had images that aren't used on any other pages, delete them.
-NOTE:
-If the renamed page is new, you can sometimes skip the following steps and ask a
-Technical Writer to manually add the redirect to [`redirects.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/redirects.yaml).
-For example, if you add a new page and then rename it before it's added to a release
-on the 18th. The old page is not in any version's `/help` section, so a technical writer
-can jump straight to the [Pages redirect](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/maintenance.md#pages-redirects).
+After your changes are committed, search for and update all links that point to the old file:
-To add a redirect:
+- In <https://gitlab.com/gitlab-com/www-gitlab-com>, search for full URLs:
-1. In the repository (`gitlab`, `gitlab-runner`, `omnibus-gitlab`, or `charts`),
- create a new documentation file. Don't delete the old one. The easiest
- way is to copy it. For example:
+ ```shell
+ grep -r "docs.gitlab.com/ee/path/to/file.html" .
+ ```
- ```shell
- cp doc/user/search/old_file.md doc/api/new_file.md
+- In <https://gitlab.com/gitlab-org/gitlab-docs/-/tree/master/content/_data>,
+ search the navigation bar configuration files for the path with `.html`:
+
+ ```shell
+ grep -r "path/to/file.html" .
```
-1. Add the redirect code to the old documentation file by running the
- following Rake task. The first argument is the path of the old file,
- and the second argument is the path of the new file:
+- In any of the four internal projects, search for links in the docs
+ and codebase. Search for all variations, including full URL and just the path.
+ For example, go to the root directory of the `gitlab` project and run:
- - To redirect to a page in the same project, use relative paths and
- the `.md` extension. Both old and new paths start from the same location.
- In the following example, both paths are relative to `doc/`:
+ ```shell
+ grep -r "docs.gitlab.com/ee/path/to/file.html" .
+ grep -r "path/to/file.html" .
+ grep -r "path/to/file.md" .
+ grep -r "path/to/file" .
+ ```
- ```shell
- bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, doc/api/new_file.md]"
- ```
+ You might need to try variations of relative links, such as `../path/to/file` or
+ `../file` to find every case.
- - To redirect to a page in a different project or site, use the full URL (with `https://`) :
+### Move a file's location
- ```shell
- bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, https://example.com]"
- ```
+If you want to move a file from one location to another, you do not move it.
+Instead, you duplicate the file, and add the redirect code to the old file.
- - Alternatively, you can omit the arguments and be prompted to enter the values:
+1. Create the new file.
+1. Copy the contents of the old file to the new one.
+1. In the old file, delete all the content.
+1. In the old file, add the redirect code and follow the rest of the steps in
+ the [Redirect to a page that already exists](#redirect-to-a-page-that-already-exists) topic.
- ```shell
- bundle exec rake gitlab:docs:redirect
- ```
+## Use code to add a redirect
- If you don't want to use the Rake task, you can use the following template:
+If you prefer to use a script to create the redirect:
- ```markdown
- ---
- redirect_to: '../newpath/to/file/index.md'
- remove_date: 'YYYY-MM-DD'
- ---
+Add the redirect code to the old documentation file by running the
+following Rake task. The first argument is the path of the old file,
+and the second argument is the path of the new file:
- This document was moved to [another location](../path/to/file/index.md).
+- To redirect to a page in the same project, use relative paths and
+ the `.md` extension. Both old and new paths start from the same location.
+ In the following example, both paths are relative to `doc/`:
- <!-- This redirect file can be deleted after <YYYY-MM-DD>. -->
- <!-- Redirects that point to other docs in the same project expire in three months. -->
- <!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
- <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
- ```
+ ```shell
+ bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, doc/api/new_file.md]"
+ ```
- - Replace both instances of `../newpath/to/file/index.md` with the new file path.
- - Replace `YYYY-MM-DD` with the expiry date, as explained in the template.
-
-1. If the documentation page being moved has any Disqus comments, follow the steps
- described in [Redirections for pages with Disqus comments](#redirections-for-pages-with-disqus-comments).
-1. Open a merge request with your changes. If a documentation page
- you're removing includes images that aren't used
- with any other documentation pages, be sure to use your merge request to delete
- those images from the repository.
-1. Assign the merge request to a technical writer for review and merge.
-1. Search for links to the old documentation file. You must find and update all
- links that point to the old documentation file:
-
- - In <https://gitlab.com/gitlab-com/www-gitlab-com>, search for full URLs:
- `grep -r "docs.gitlab.com/ee/path/to/file.html" .`
- - In <https://gitlab.com/gitlab-org/gitlab-docs/-/tree/master/content/_data>,
- search the navigation bar configuration files for the path with `.html`:
- `grep -r "path/to/file.html" .`
- - In any of the four internal projects, search for links in the docs
- and codebase. Search for all variations, including full URL and just the path.
- For example, go to the root directory of the `gitlab` project and run:
-
- ```shell
- grep -r "docs.gitlab.com/ee/path/to/file.html" .
- grep -r "path/to/file.html" .
- grep -r "path/to/file.md" .
- grep -r "path/to/file" .
- ```
+- To redirect to a page in a different project or site, use the full URL (with `https://`) :
+
+ ```shell
+ bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, https://example.com]"
+ ```
+
+- Alternatively, you can omit the arguments and be prompted to enter the values:
+
+ ```shell
+ bundle exec rake gitlab:docs:redirect
+ ```
+
+## Redirecting a page created before the release
+
+If you create a new page and then rename it before it's added to a release on the 18th:
- You may need to try variations of relative links, such as `../path/to/file` or
- `../file` to find every case.
+Instead of following that procedure, ask a Technical Writer to manually add the redirect
+to [`redirects.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/redirects.yaml).
## Redirections for pages with Disqus comments
diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md
index bf1461a810d..2991c1b3224 100644
--- a/doc/development/documentation/restful_api_styleguide.md
+++ b/doc/development/documentation/restful_api_styleguide.md
@@ -59,12 +59,12 @@ METHOD /endpoint
Supported attributes:
-| Attribute | Type | Required | Description |
-|:-------------------------|:---------|:-----------------------|:----------------------|
-| `attribute` | datatype | **{check-circle}** Yes | Detailed description. |
-| `attribute` **(<tier>)** | datatype | **{dotted-circle}** No | Detailed description. |
-| `attribute` | datatype | **{dotted-circle}** No | Detailed description. |
-| `attribute` | datatype | **{dotted-circle}** No | Detailed description. |
+| Attribute | Type | Required | Description |
+|:-------------------------|:---------|:---------|:----------------------|
+| `attribute` | datatype | Yes | Detailed description. |
+| `attribute` **(<tier>)** | datatype | No | Detailed description. |
+| `attribute` | datatype | No | Detailed description. |
+| `attribute` | datatype | No | Detailed description. |
If successful, returns [`<status_code>`](../../api/index.md#status-codes) and the following
response attributes:
@@ -123,13 +123,13 @@ To deprecate an attribute:
1. Add inline deprecation text to the description.
```markdown
- | Attribute | Type | Required | Description |
- |:--------------|:-------|:-----------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------|
- | `widget_name` | string | **{dotted-circle}** No | [Deprecated](<link-to-issue>) in GitLab 14.7 and is planned for removal in 15.4. Use `widget_id` instead. The name of the widget. |
+ | Attribute | Type | Required | Description |
+ |:--------------|:-------|:---------|:----------------------------------------------------------------------------------------------------------------------------------------------------|
+ | `widget_name` | string | No | [Deprecated](<link-to-issue>) in GitLab 14.7 and is planned for removal in 15.4. Use `widget_id` instead. The name of the widget. |
```
To widely announce a deprecation, or if it's a breaking change,
-[update the deprecations and removals documentation](../deprecation_guidelines/#update-the-deprecations-and-removals-documentation).
+[update the deprecations and removals documentation](../deprecation_guidelines/index.md#update-the-deprecations-and-removals-documentation).
## Method description
@@ -139,20 +139,20 @@ always be in code blocks using backticks (`` ` ``).
Sort the table by required attributes first, then alphabetically.
```markdown
-| Attribute | Type | Required | Description |
-|:-----------------------------|:--------------|:-----------------------|:----------------------------------------------------|
-| `title` | string | **{check-circle}** Yes | Title of the issue. |
-| `assignee_ids` **(PREMIUM)** | integer array | **{dotted-circle}** No | IDs of the users to assign the issue to. |
-| `confidential` | boolean | **{dotted-circle}** No | Sets the issue to confidential. Default is `false`. |
+| Attribute | Type | Required | Description |
+|:-----------------------------|:--------------|:---------|:----------------------------------------------------|
+| `title` | string | Yes | Title of the issue. |
+| `assignee_ids` **(PREMIUM)** | integer array | No | IDs of the users to assign the issue to. |
+| `confidential` | boolean | No | Sets the issue to confidential. Default is `false`. |
```
Rendered example:
-| Attribute | Type | Required | Description |
-|:-----------------------------|:--------------|:-----------------------|:----------------------------------------------------|
-| `title` | string | **{check-circle}** Yes | Title of the issue. |
-| `assignee_ids` **(PREMIUM)** | integer array | **{dotted-circle}** No | IDs of the users to assign the issue to. |
-| `confidential` | boolean | **{dotted-circle}** No | Sets the issue to confidential. Default is `false`. |
+| Attribute | Type | Required | Description |
+|:-----------------------------|:--------------|:---------|:----------------------------------------------------|
+| `title` | string | Yes | Title of the issue. |
+| `assignee_ids` **(PREMIUM)** | integer array | No | IDs of the users to assign the issue to. |
+| `confidential` | boolean | No | Sets the issue to confidential. Default is `false`. |
For information about writing attribute descriptions, see the [GraphQL API description style guide](../api_graphql_styleguide.md#description-style-guide).
diff --git a/doc/development/documentation/review_apps.md b/doc/development/documentation/review_apps.md
index 8cb9e6437b8..a50efceb307 100644
--- a/doc/development/documentation/review_apps.md
+++ b/doc/development/documentation/review_apps.md
@@ -57,7 +57,7 @@ If you want to know the in-depth details, here's what's really happening:
The following GitLab features are used among others:
- [Manual jobs](../../ci/jobs/job_control.md#create-a-job-that-must-be-run-manually)
-- [Multi project pipelines](../../ci/pipelines/multi_project_pipelines.md)
+- [Multi project pipelines](../../ci/pipelines/downstream_pipelines.md#multi-project-pipelines)
- [Review Apps](../../ci/review_apps/index.md)
- [Artifacts](../../ci/yaml/index.md#artifacts)
- [Specific runner](../../ci/runners/runners_scope.md#prevent-a-specific-runner-from-being-enabled-for-other-projects)
diff --git a/doc/development/documentation/site_architecture/deployment_process.md b/doc/development/documentation/site_architecture/deployment_process.md
index 5f6076f3195..8a9c2e1e8d7 100644
--- a/doc/development/documentation/site_architecture/deployment_process.md
+++ b/doc/development/documentation/site_architecture/deployment_process.md
@@ -144,18 +144,21 @@ graph LR
### Manually deploy to production
-GitLab Docs is deployed to production whenever the `Build docs.gitlab.com every 4 hours` scheduled pipeline runs. By
-default, this pipeline runs every four hours.
+GitLab Docs is deployed to production whenever the `Build docs.gitlab.com every hour` scheduled pipeline runs. By
+default, this pipeline runs every hour.
Maintainers can [manually](../../../ci/pipelines/schedules.md#run-manually) run this pipeline to force a deployment to
production:
1. Go to the [scheduled pipelines](https://gitlab.com/gitlab-org/gitlab-docs/-/pipeline_schedules) for `gitlab-docs`.
-1. Next to `Build docs.gitlab.com every 4 hours`, select **Play** (**{play}**).
+1. Next to `Build docs.gitlab.com every hour`, select **Play** (**{play}**).
The updated documentation is available in production after the `pages` and `pages:deploy` jobs
complete in the new pipeline.
+If you do not have the Maintainer role to perform this task, ask for help in the
+`#docs` Slack channel.
+
## Docker files
The [`dockerfiles` directory](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/) contains all needed
diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md
index 2864bbe7404..44e4aac2756 100644
--- a/doc/development/documentation/site_architecture/index.md
+++ b/doc/development/documentation/site_architecture/index.md
@@ -33,7 +33,7 @@ Then you can use one of these approaches:
of the documentation in the external repository. The landing page is indexed and
searchable on <https://docs.gitlab.com>, but the rest of the documentation is not.
For example, the [GitLab Workflow extension for VS Code](../../../user/project/repository/vscode.md).
- We do not encourage the use of [pages with lists of links](../structure.md#topics-and-resources-pages),
+ We do not encourage the use of [pages with lists of links](../topic_types/index.md#topics-and-resources),
so only use this option if the recommended options are not feasible.
## Monthly release process (versions)
diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md
index a5d1290a17a..35a93f08f66 100644
--- a/doc/development/documentation/structure.md
+++ b/doc/development/documentation/structure.md
@@ -1,395 +1,11 @@
---
-stage: none
-group: Style Guide
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
-description: What to include in GitLab documentation pages.
+redirect_to: 'topic_types/index.md'
+remove_date: '2022-11-16'
---
-# Documentation topic types (CTRT)
+This document was moved to [another location](topic_types/index.md).
-At GitLab, we have not traditionally used types for our content. However, we are starting to
-move in this direction, and we now use four primary topic types (CTRT):
-
-- [Concept](#concept)
-- [Task](#task)
-- [Reference](#reference)
-- [Troubleshooting](#troubleshooting)
-
-In general, each page in our docset contains multiple topics. (Each heading indicates a new topic.)
-Each topic on a page should be a specific topic type. For example,
-a page with the title `Pipelines` can include topics that are concepts and tasks.
-
-A page might also contain only one type of information. These pages are generally one of our
-[other content types](#other-types-of-content).
-
-## Concept
-
-A concept introduces a single feature or concept.
-
-A concept should answer the questions:
-
-- What is this?
-- Why would I use it?
-
-Think of everything someone might want to know if they've never heard of this concept before.
-
-Don't tell them **how** to do this thing. Tell them **what it is**.
-
-If you start describing another concept, start a new concept and link to it.
-
-Concepts should be in this format:
-
-```markdown
-# Title (a noun, like "Widgets")
-
-A paragraph that explains what this thing is.
-
-Another paragraph that explains what this thing is.
-
-Remember, if you start to describe about another concept, stop yourself.
-Each concept should be about one concept only.
-```
-
-### Concept headings
-
-For the heading text, use a noun. For example, `Widgets` or `GDK dependency management`.
-
-If a noun is ambiguous, you can add a gerund. For example, `Documenting versions` instead of `Versions`.
-
-Avoid these heading titles:
-
-- `Overview` or `Introduction`. Instead, use a more specific
- noun or phrase that someone would search for.
-- `Use cases`. Instead, incorporate the information as part of the concept.
-- `How it works`. Instead, use a noun followed by `workflow`. For example, `Merge request workflow`.
-
-## Task
-
-A task gives instructions for how to complete a procedure.
-
-Tasks should be in this format:
-
-```markdown
-# Title (starts with an active verb, like "Create a widget" or "Delete a widget")
-
-Do this task when you want to...
-
-Prerequisites (optional):
-
-- Thing 1
-- Thing 2
-- Thing 3
-
-To do this task:
-
-1. Location then action. (Go to this menu, then select this item.)
-1. Another step.
-1. Another step.
-
-Task result (optional). Next steps (optional).
-```
-
-Here is an example.
-
-```markdown
-# Create an issue
-
-Create an issue when you want to track bugs or future work.
-
-Prerequisites:
-
-- You must have at least the Developer role for the project.
-
-To create an issue:
-
-1. On the top bar, select **Menu > Projects** and find your project.
-1. On the left sidebar, select **Issues > List**.
-1. In the top right corner, select **New issue**.
-1. Complete the fields. (If you have reference content that lists each field, link to it here.)
-1. Select **Create issue**.
-
-The issue is created. You can view it by going to **Issues > List**.
-```
-
-### Task headings
-
-For the heading text, use the structure `active verb` + `noun`.
-For example, `Create an issue`.
-
-If you have several tasks on a page that share prerequisites, you can use the title
-`Prerequisites` and link to it.
-
-## Reference
-
-Reference information should be in an easily-scannable format,
-like a table or list. It's similar to a dictionary or encyclopedia entry.
-
-```markdown
-# Title (a noun, like "Pipeline settings" or "Administrator options")
-
-Introductory sentence.
-
-| Setting | Description |
-|---------|-------------|
-| **Name** | Descriptive sentence about the setting. |
-```
-
-### Reference headings
-
-Reference headings are usually nouns.
-
-Avoid these heading titles:
-
-- `Important notes`. Instead, incorporate this information
- closer to where it belongs. For example, this information might be a prerequisite
- for a task, or information about a concept.
-- `Limitations`. Instead, move the content near other similar information.
- If you must, you can use the title `Known issues`.
-
-## Troubleshooting
-
-Troubleshooting topics should be the last topics on a page.
-
-Troubleshooting can be one of three categories:
-
-- **An introductory topic.** This topic introduces the troubleshooting section of a page.
- For example:
-
- ```markdown
- ## Troubleshooting
-
- When working with <x feature>, you might encounter the following issues.
- ```
-
-- **Troubleshooting task.** The title should be similar to a [standard task](#task).
- For example, "Run debug tools" or "Verify syntax."
-
-- **Troubleshooting reference.** This information includes the error message. For example:
-
- ```markdown
- ### The error message or a description of it
-
- You might get an error that states <error message>.
-
- This issue occurs when...
-
- The workaround is...
- ```
-
- If multiple causes or workarounds exist, consider putting them into a table format.
- If you use the exact error message, surround it in backticks so it's styled as code.
-
-If a page has more than five troubleshooting topics, put the content on a separate page that has troubleshooting information exclusively. Name the page `Troubleshooting <featurename>`.
-
-### Troubleshooting headings
-
-For the heading of a **Troubleshooting reference** topic:
-
-- Consider including at least a partial error message.
-- Use fewer than 70 characters.
-
-If you do not put the full error in the title, include it in the body text.
-
-### Related topics
-
-If inline links are not sufficient, you can create a topic called **Related topics**
-and include an unordered list of related topics. This topic should be above the Troubleshooting section.
-
-```markdown
-# Related topics
-
-- [Configure your pipeline](link-to-topic)
-- [Trigger a pipeline manually](link-to-topic)
-```
-
-## General heading text guidelines
-
-In general, for heading text:
-
-- Be clear and direct. Make every word count.
-- Use articles and prepositions.
-- Follow [capitalization](styleguide/index.md#capitalization) guidelines.
-- Do not repeat text from earlier headings. For example, if the page is about merge requests,
- instead of `Troubleshooting merge requests`, use only `Troubleshooting`.
-
-See also [guidelines for headings in Markdown](styleguide/index.md#headings-in-markdown).
-
-## Other types of content
-
-There are other types of content in the GitLab documentation that don't
-classify as one of the four primary [topic types](#documentation-topic-types-ctrt).
-These include:
-
-- [Tutorials](#tutorials)
-- [Get started pages](#get-started)
-- [Topics and resources pages](#topics-and-resources-pages)
-
-In most cases, these pages are standalone.
-
-### Tutorials
-
-A tutorial is an end-to-end walkthrough of a complex workflow or scenario.
-In general, you might consider using a tutorial when:
-
-- The workflow requires a number of sequential steps where each step consists
- of sub-steps.
-- The steps cover a variety of GitLab features or third-party tools.
-
-Tutorials are learning aids that complement our core documentation.
-They do not introduce new features.
-Always use the primary [topic types](#documentation-topic-types-ctrt) to document new features.
-
-Tutorials should be in this format:
-
-```markdown
-# Title (starts with "Tutorial:" followed by an active verb, like "Tutorial: Create a website")
-
-A paragraph that explains what the tutorial does, and the expected outcome.
-
-To create a website:
-
-1. [Do the first task](#do-the-first-task)
-1. [Do the second task](#do-the-second-task)
-
-Prerequisites (optional):
-
-- Thing 1
-- Thing 2
-- Thing 3
-
-## Do the first task
-
-To do step 1:
-
-1. First step.
-1. Another step.
-1. Another step.
-
-## Do the second task
-
-Before you begin, make sure you have [done the first task](#do-the-first-task).
-
-To do step 2:
-
-1. First step.
-1. Another step.
-1. Another step.
-```
-
-### Get started
-
-A get started page is a set of steps to help a user get set up
-quickly to use a single GitLab feature or tool.
-It might consist of more than one task.
-
-Get started pages should be in this format:
-
-```markdown
-# Title ("Get started with <feature>")
-
-Complete the following steps to ... .
-
-1. First step.
-1. Another step.
-1. Another step.
-
-If you need to add more than one task,
-consider using subsections for each distinct task.
-```
-
-### Topics and resources pages
-
-This page has a list of links that point to important sections
-of documentation for a specific GitLab feature or tool.
-
-We do not encourage the use of these types of pages.
-Lists like this can get out of date quickly and offer little value to users.
-We've included this type here because:
-
-- There are existing pages in the documentation that follow this format,
- and they should be standardized.
-- They can sometimes help navigate a complex section of the documentation.
-
-If you come across a page like this
-or you have to create one, use this format:
-
-```markdown
-# Title ("Topics and resources for <feature>")
-
-Brief sentence to describe the feature.
-
-Refer to these resources for more information about <this feature>:
-
-- Link 1
-- Link 2
-- Link 3
-```
-
-## Help and feedback section
-
-This section ([introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/319) in GitLab 11.4)
-is displayed at the end of each document and can be omitted by adding a key into
-the front matter:
-
-```yaml
----
-feedback: false
----
-```
-
-The default is to leave it there. If you want to omit it from a document, you
-must check with a technical writer before doing so.
-
-### Disqus
-
-We also have integrated the docs site with Disqus (introduced by
-[!151](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/151)),
-allowing our users to post comments.
-
-To omit only the comments from the feedback section, use the following key in
-the front matter:
-
-```yaml
----
-comments: false
----
-```
-
-We're hiding comments only in main index pages, such as [the main documentation index](../../index.md),
-since its content is too broad to comment on. Before omitting Disqus, you must
-check with a technical writer.
-
-Note that after adding `feedback: false` to the front matter, it will omit
-Disqus, therefore, don't add both keys to the same document.
-
-The click events in the feedback section are tracked with Google Tag Manager.
-The conversions can be viewed on Google Analytics by navigating to
-**Behavior > Events > Top events > docs**.
-
-## Guidelines for good practices
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36576/) in GitLab 13.2 as GitLab Development documentation.
-
-*Good practice* examples demonstrate encouraged ways of writing code while
-comparing with examples of practices to avoid. These examples are labeled as
-*Bad* or *Good*. In GitLab development guidelines, when presenting the cases,
-it's recommended to follow a *first-bad-then-good* strategy. First demonstrate
-the *Bad* practice (how things *could* be done, which is often still working
-code), and then how things *should* be done better, using a *Good* example. This
-is typically an improved example of the same code.
-
-Consider the following guidelines when offering examples:
-
-- First, offer the *Bad* example, and then the *Good* one.
-- When only one bad case and one good case is given, use the same code block.
-- When more than one bad case or one good case is offered, use separated code
- blocks for each. With many examples being presented, a clear separation helps
- the reader to go directly to the good part. Consider offering an explanation
- (for example, a comment, or a link to a resource) on why something is bad
- practice.
-- Better and best cases can be considered part of the good cases' code block.
- In the same code block, precede each with comments: `# Better` and `# Best`.
-
-Although the bad-then-good approach is acceptable for the GitLab development
-guidelines, do not use it for user documentation. For user documentation, use
-*Do* and *Don't*. For examples, see the [Pajamas Design System](https://design.gitlab.com/content/punctuation/).
+<!-- This redirect file can be deleted after <2022-11-16>. -->
+<!-- Redirects that point to other docs in the same project expire in three months. -->
+<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
+<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> \ No newline at end of file
diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md
index 709e6b2d0d9..bc79bf0fbe2 100644
--- a/doc/development/documentation/styleguide/index.md
+++ b/doc/development/documentation/styleguide/index.md
@@ -887,7 +887,7 @@ To be consistent, use these templates when you write navigation steps in a task
To open project settings:
```markdown
-1. On the top bar, select **Menu > Projects** and find your project.
+1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Settings > CI/CD**.
1. Expand **General pipelines**.
```
@@ -895,7 +895,7 @@ To open project settings:
To open group settings:
```markdown
-1. On the top bar, select **Menu > Groups** and find your group.
+1. On the top bar, select **Main menu > Groups** and find your group.
1. On the left sidebar, select **Settings > CI/CD**.
1. Expand **General pipelines**.
```
@@ -903,7 +903,7 @@ To open group settings:
To open the Admin Area:
```markdown
-1. On the top bar, select **Menu > Admin**.
+1. On the top bar, select **Main menu > Admin**.
```
To select your avatar:
@@ -950,7 +950,7 @@ Use the phrase **Complete the fields**.
For example:
-1. On the top bar, select **Menu > Projects** and find your project.
+1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Settings > Repository**.
1. Expand **Push rules**.
1. Complete the fields.
@@ -1361,6 +1361,48 @@ It renders on the GitLab documentation site as:
> - This is a list item
> - Second item in the list
+## Tabs
+
+On the docs site, you can format text so it's displayed as tabs.
+
+To create a set of tabs, follow this example:
+
+```plaintext
+::Tabs
+
+:::TabTitle Tab One
+
+Here's some content in tab one.
+
+:::TabTitle Tab Two
+
+Here's some other content in tab two.
+
+::EndTabs
+```
+
+This code renders on the GitLab documentation site as:
+
+::Tabs
+
+:::TabTitle Tab One
+
+Here's some content in tab one.
+
+:::TabTitle Tab Two
+
+Here's some other content in tab two.
+
+::EndTabs
+
+For tab titles, be brief and consistent. Ensure they are parallel, and start each with a capital letter.
+For example:
+
+- `Omnibus package`, `Helm chart`, `Source`
+- `15.1 and earlier`, `15.2 and later`
+
+See [Pajamas](https://design.gitlab.com/components/tabs/#guidelines) for details.
+
## Terms
To maintain consistency through GitLab documentation, use these styles and terms.
diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md
index 1976caefc8e..029c7389290 100644
--- a/doc/development/documentation/styleguide/word_list.md
+++ b/doc/development/documentation/styleguide/word_list.md
@@ -8,8 +8,12 @@ description: 'Writing styles, markup, formatting, and other standards for GitLab
# Recommended word list
To help ensure consistency in the documentation, the Technical Writing team
-recommends these wording choices. The GitLab handbook also maintains a list of
-[top misused terms](https://about.gitlab.com/handbook/communication/top-misused-terms/).
+recommends these word choices. In addition:
+
+- The GitLab handbook contains a list of
+ [top misused terms](https://about.gitlab.com/handbook/communication/top-misused-terms/).
+- The documentation [style guide](../styleguide#language) includes details
+ about language and capitalization.
For guidance not on this page, we defer to these style guides:
@@ -19,6 +23,10 @@ For guidance not on this page, we defer to these style guides:
<!-- vale off -->
<!-- markdownlint-disable -->
+## `&`
+
+Do not use Latin abbreviations. Use **and** instead, unless you are documenting a UI element that uses an `&`.
+
## `@mention`
Try to avoid **`@mention`**. Say **mention** instead, and consider linking to the
@@ -75,7 +83,7 @@ Instead of:
## Admin Area
-Use title case **Admin Area** to refer to the area of the UI that you access when you select **Menu > Admin**.
+Use title case **Admin Area** to refer to the area of the UI that you access when you select **Main menu > Admin**.
This area of the UI says **Admin Area** at the top of the page and on the menu.
## agent
@@ -139,6 +147,18 @@ Do not use **and so on**. Instead, be more specific. For details, see
Use [**section**](#section) instead of **area**. The only exception is [the Admin Area](#admin-area).
+## as
+
+Do not use **as** to mean **because**.
+
+Use:
+
+- Because none of the endpoints return an ID...
+
+Instead of:
+
+- As none of the endpoints return an ID...
+
## associate
Do not use **associate** when describing adding issues to epics, or users to issues, merge requests,
@@ -265,6 +285,13 @@ Use title case for the GitLab Container Registry.
Do not use **currently** when talking about the product or its features. The documentation describes the product as it is today.
([Vale](../testing.md#vale) rule: [`CurrentStatus.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/CurrentStatus.yml))
+## default branch
+
+Use **default branch** to refer generically to the primary branch in the repository.
+Users can set the default branch by using a UI setting.
+
+For examples that use the default branch, use `main` instead of [`master`](#master).
+
## Dependency Proxy
Use title case for the GitLab Dependency Proxy.
@@ -394,7 +421,7 @@ Information in FAQs belongs with other similar information, under an easily sear
## field
-Use **box** instead of **field** or **text box**.
+Use **text box** instead of **field** or **box**.
Use:
@@ -407,7 +434,7 @@ Instead of:
However, you can make an exception when you are writing a task and you need to refer to all
of the fields at once. For example:
-1. On the top bar, select **Menu > Projects** and find your project.
+1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Settings > CI/CD**.
1. Expand **General pipelines**.
1. Complete the fields.
@@ -604,6 +631,10 @@ Instead of:
- Buy a license.
- Purchase a license.
+## limitations
+
+Do not use **limitations**. Use **known issues** instead.
+
## log in, log on
Do not use **log in** or **log on**. Use [sign in](#sign-in) instead. If the user interface has **Log in**, you can use it.
@@ -644,7 +675,8 @@ Do not use **manpower**. Use words like **workforce** or **GitLab team members**
## master
-Do not use **master**. Options are **primary** or **main**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml))
+Do not use `master`. Use `main` when you need a sample [default branch name](#default-branch).
+([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml))
## may, might
diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md
index 428a57a11fb..59a078bdec0 100644
--- a/doc/development/documentation/testing.md
+++ b/doc/development/documentation/testing.md
@@ -190,7 +190,7 @@ To update the linting images:
1. In `gitlab-docs`, open a merge request to update `.gitlab-ci.yml` to use the new tooling
version. ([Example MR](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/2571))
-1. When merged, start a `Build docs.gitlab.com every 4 hours` [scheduled pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipeline_schedules).
+1. When merged, start a `Build docs.gitlab.com every hour` [scheduled pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipeline_schedules).
1. Go the pipeline you started, and manually run the relevant build-images job,
for example, `image:docs-lint-markdown`.
1. In the job output, get the name of the new image.
diff --git a/doc/development/documentation/topic_types/concept.md b/doc/development/documentation/topic_types/concept.md
new file mode 100644
index 00000000000..a20bb93a97f
--- /dev/null
+++ b/doc/development/documentation/topic_types/concept.md
@@ -0,0 +1,46 @@
+---
+stage: none
+group: Style Guide
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Concept topic type
+
+A concept introduces a single feature or concept.
+
+A concept should answer the questions:
+
+- What is this?
+- Why would I use it?
+
+Think of everything someone might want to know if they've never heard of this concept before.
+
+Don't tell them **how** to do this thing. Tell them **what it is**.
+
+If you start describing another concept, start a new concept and link to it.
+
+Concepts should be in this format:
+
+```markdown
+# Title (a noun, like "Widgets")
+
+A paragraph that explains what this thing is.
+
+Another paragraph that explains what this thing is.
+
+Remember, if you start to describe about another concept, stop yourself.
+Each concept should be about one concept only.
+```
+
+## Concept headings
+
+For the heading text, use a noun. For example, `Widgets` or `GDK dependency management`.
+
+If a noun is ambiguous, you can add a gerund. For example, `Documenting versions` instead of `Versions`.
+
+Avoid these heading titles:
+
+- `Overview` or `Introduction`. Instead, use a more specific
+ noun or phrase that someone would search for.
+- `Use cases`. Instead, incorporate the information as part of the concept.
+- `How it works`. Instead, use a noun followed by `workflow`. For example, `Merge request workflow`.
diff --git a/doc/development/documentation/topic_types/index.md b/doc/development/documentation/topic_types/index.md
new file mode 100644
index 00000000000..af3e66fe87a
--- /dev/null
+++ b/doc/development/documentation/topic_types/index.md
@@ -0,0 +1,130 @@
+---
+stage: none
+group: Style Guide
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Documentation topic types (CTRT)
+
+At GitLab, we have not traditionally used types for our content. However, we are starting to
+move in this direction, and we now use four primary topic types:
+
+- [Concept](concept.md)
+- [Task](task.md)
+- [Reference](reference.md)
+- [Troubleshooting](troubleshooting.md)
+
+The tech writing team sometimes uses the acronym `CTRT` to refer to our topic types.
+The acronym refers to the first letter of each topic type.
+
+In general, each page in the GitLab documentation contains multiple topics.
+Each topic on a page should be recognizable as a specific topic type.
+
+## Other topic types
+
+In addition to the four primary topic types, we have a few other types.
+
+### Related topics
+
+If inline links are not sufficient, you can create a topic called **Related topics**
+and include an unordered list of related topics. This topic should be above the Troubleshooting section.
+
+```markdown
+# Related topics
+
+- [Configure your pipeline](link-to-topic).
+- [Trigger a pipeline manually](link-to-topic).
+```
+
+### Tutorials
+
+A tutorial is page that contains an end-to-end walkthrough of a complex workflow or scenario.
+In general, you might consider using a tutorial when:
+
+- The workflow requires a number of sequential steps where each step consists
+ of sub-steps.
+- The steps cover a variety of GitLab features or third-party tools.
+
+Tutorials are learning aids that complement our core documentation.
+They do not introduce new features.
+Always use the primary [topic types](#documentation-topic-types-ctrt) to document new features.
+
+Tutorials should be in this format:
+
+```markdown
+# Title (starts with "Tutorial:" followed by an active verb, like "Tutorial: Create a website")
+
+A paragraph that explains what the tutorial does, and the expected outcome.
+
+To create a website:
+
+1. [Do the first task](#do-the-first-task)
+1. [Do the second task](#do-the-second-task)
+
+Prerequisites (optional):
+
+- Thing 1
+- Thing 2
+- Thing 3
+
+## Do the first task
+
+To do step 1:
+
+1. First step.
+1. Another step.
+1. Another step.
+
+## Do the second task
+
+Before you begin, make sure you have [done the first task](#do-the-first-task).
+
+To do step 2:
+
+1. First step.
+1. Another step.
+1. Another step.
+```
+
+### Get started
+
+A get started page is a set of steps to help a user get set up
+quickly to use a single GitLab feature or tool.
+It consists of more than one task.
+
+Get started pages should be in this format:
+
+```markdown
+# Title ("Get started with <feature>")
+
+Complete the following steps to ... .
+
+1. First step.
+1. Another step.
+1. Another step.
+
+If you need to add more than one task,
+consider using subsections for each distinct task.
+```
+
+In the left nav, use `Get started` as the text. On the page itself, spell out
+the full name. For example, `Get started with application security`.
+
+### Topics and resources
+
+Some pages are solely a list of links to other documentation.
+
+We do not encourage this page type. Lists of links can get out-of-date quickly
+and offer little value to users, who prefer to search to find information.
+
+## Heading text guidelines
+
+In general, for heading text:
+
+- Be clear and direct. Make every word count.
+- Use articles and prepositions.
+- Follow [capitalization](../styleguide/index.md#capitalization) guidelines.
+- Do not repeat text from earlier headings. For example, if the page is about merge requests,
+ instead of `Troubleshooting merge requests`, use only `Troubleshooting`.
+
+See also [guidelines for headings in Markdown](../styleguide/index.md#headings-in-markdown).
diff --git a/doc/development/documentation/topic_types/reference.md b/doc/development/documentation/topic_types/reference.md
new file mode 100644
index 00000000000..42f4f5f6f94
--- /dev/null
+++ b/doc/development/documentation/topic_types/reference.md
@@ -0,0 +1,32 @@
+---
+stage: none
+group: Style Guide
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Reference topic type
+
+Reference information should be in an easily-scannable format,
+like a table or list. It's similar to a dictionary or encyclopedia entry.
+
+```markdown
+# Title (a noun, like "Pipeline settings" or "Administrator options")
+
+Introductory sentence.
+
+| Setting | Description |
+|---------|-------------|
+| **Name** | Descriptive sentence about the setting. |
+```
+
+## Reference headings
+
+Reference headings are usually nouns.
+
+Avoid these heading titles:
+
+- `Important notes`. Instead, incorporate this information
+ closer to where it belongs. For example, this information might be a prerequisite
+ for a task, or information about a concept.
+- `Limitations`. Instead, move the content near other similar information.
+ If you must, you can use the title `Known issues`.
diff --git a/doc/development/documentation/topic_types/task.md b/doc/development/documentation/topic_types/task.md
new file mode 100644
index 00000000000..3488cb90cf9
--- /dev/null
+++ b/doc/development/documentation/topic_types/task.md
@@ -0,0 +1,65 @@
+---
+stage: none
+group: Style Guide
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Task topic type
+
+A task gives instructions for how to complete a procedure.
+
+Tasks should be in this format:
+
+```markdown
+# Title (starts with an active verb, like "Create a widget" or "Delete a widget")
+
+Do this task when you want to...
+
+Prerequisites (optional):
+
+- Thing 1
+- Thing 2
+- Thing 3
+
+To do this task:
+
+1. Location then action. (Go to this menu, then select this item.)
+1. Another step.
+1. Another step.
+
+Task result (optional). Next steps (optional).
+```
+
+Here is an example.
+
+```markdown
+# Create an issue
+
+Create an issue when you want to track bugs or future work.
+
+Prerequisites:
+
+- You must have at least the Developer role for the project.
+
+To create an issue:
+
+1. On the top bar, select **Main menu > Projects** and find your project.
+1. On the left sidebar, select **Issues > List**.
+1. In the top right corner, select **New issue**.
+1. Complete the fields. (If you have reference content that lists each field, link to it here.)
+1. Select **Create issue**.
+
+The issue is created. You can view it by going to **Issues > List**.
+```
+
+## Task headings
+
+For the heading text, use the structure `active verb` + `noun`.
+For example, `Create an issue`.
+
+If you have several tasks on a page that share prerequisites, you can use the title
+`Prerequisites` and link to it.
+
+## Related topics
+
+- [View the format for writing task steps](../styleguide/index.md#navigation).
diff --git a/doc/development/documentation/topic_types/troubleshooting.md b/doc/development/documentation/topic_types/troubleshooting.md
new file mode 100644
index 00000000000..35187bd892e
--- /dev/null
+++ b/doc/development/documentation/topic_types/troubleshooting.md
@@ -0,0 +1,56 @@
+---
+stage: none
+group: Style Guide
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Troubleshooting topic type
+
+Troubleshooting topics should be the last topics on a page.
+
+If a page has more than five troubleshooting topics, put the content on a separate page that has troubleshooting information exclusively. Name the page `Troubleshooting <feature>`
+and in the left nav, use the word `Troubleshoot` only.
+
+Troubleshooting can be one of three types.
+
+## An introductory topic
+
+This topic introduces the troubleshooting section of a page.
+For example:
+
+```markdown
+## Troubleshooting
+
+When working with <x feature>, you might encounter the following issues.
+```
+
+## Troubleshooting task
+
+The title should be similar to a [standard task](task.md).
+For example, "Run debug tools" or "Verify syntax."
+
+## Troubleshooting reference
+
+This topic includes the error message. For example:
+
+```markdown
+### The error message or a description of it
+
+You might get an error that states <error message>.
+
+This issue occurs when...
+
+The workaround is...
+```
+
+If multiple causes or workarounds exist, consider putting them into a table format.
+If you use the exact error message, surround it in backticks so it's styled as code.
+
+## Troubleshooting headings
+
+For the heading of a **Troubleshooting reference** topic:
+
+- Consider including at least a partial error message.
+- Use fewer than 70 characters.
+
+If you do not put the full error in the title, include it in the body text.
diff --git a/doc/development/documentation/versions.md b/doc/development/documentation/versions.md
index 3679c731a77..85733603cfe 100644
--- a/doc/development/documentation/versions.md
+++ b/doc/development/documentation/versions.md
@@ -69,6 +69,11 @@ If a feature is moved to another subscription tier, use `moved`:
> - [Moved](<link-to-issue>) from GitLab Premium to GitLab Free in 12.0.
```
+#### Features introduced behind feature flags
+
+When features are introduced behind feature flags, you must add details about the feature flag to the documentation.
+For more information, see [Document features deployed behind feature flags](feature_flags.md).
+
### Inline version text
If you're adding content to an existing topic, you can add version information
@@ -205,7 +210,7 @@ You can say that we plan to remove a feature.
### Legal disclaimer for future features
-If you **must** write about features we have not yet delivered, put this exact disclaimer near the content it applies to.
+If you **must** write about features we have not yet delivered, put this exact disclaimer about forward-looking statements near the content it applies to.
```markdown
DISCLAIMER:
@@ -227,6 +232,6 @@ As with all projects, the items mentioned on this page are subject to change or
The development, release, and timing of any products, features, or functionality remain at the
sole discretion of GitLab Inc.
-If all of the content on the page is not available, use the disclaimer once at the top of the page.
+If all of the content on the page is not available, use the disclaimer about forward-looking statements once at the top of the page.
If the content in a topic is not ready, use the disclaimer in the topic.
diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md
index 777bc77875e..869cb0bab0a 100644
--- a/doc/development/ee_features.md
+++ b/doc/development/ee_features.md
@@ -20,7 +20,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
## Implement a new EE feature
-If you're developing a GitLab Starter, GitLab Premium, or GitLab Ultimate licensed feature, use these steps to
+If you're developing a GitLab Premium or GitLab Ultimate licensed feature, use these steps to
add your new feature or extend it.
GitLab license features are added to [`ee/app/models/gitlab_subscriptions/features.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/gitlab_subscriptions/features.rb). To determine how
@@ -33,9 +33,9 @@ Use the following questions to guide you:
must locate the existing feature identifier to [guard it](#guard-your-ee-feature).
- If this is a new feature, decide on an identifier, such as `my_feature_name`, to add to the
`features.rb` file.
-1. Is this a **GitLab Starter**, **GitLab Premium**, or **GitLab Ultimate** feature?
- - Based on the plan you choose to use the feature in, add the feature identifier to `STARTER_FEATURES`,
- `PREMIUM_FEATURES`, or `ULTIMATE_FEATURES`.
+1. Is this a **GitLab Premium** or **GitLab Ultimate** feature?
+ - Based on the plan you choose to use the feature in, add the feature identifier to `PREMIUM_FEATURES`
+ or `ULTIMATE_FEATURES`.
1. Will this feature be available globally (system-wide at the GitLab instance level)?
- Features such as [Geo](../administration/geo/index.md) and
[Database Load Balancing](../administration/postgresql/database_load_balancing.md) are used by the entire instance
@@ -281,7 +281,7 @@ There are a few gotchas with it:
overriding the method, because we can't know when the overridden method
(that is, calling `super` in the overriding method) would want to stop early.
In this case, we shouldn't just override it, but update the original method
- to make it call the other method we want to extend, like a
+ to make it call the other method we want to extend, like a
[template method pattern](https://en.wikipedia.org/wiki/Template_method_pattern).
For example, given this base:
@@ -1128,7 +1128,7 @@ EE licensed features that enhance existing functionality in the UI add new
elements or interactions to your Vue application as components.
To separate template differences, use a child EE component to separate Vue template differences.
-You must import the EE component [asynchronously](https://vuejs.org/v2/guide/components-dynamic-async.html#Async-Components).
+You must import the EE component [asynchronously](https://v2.vuejs.org/v2/guide/components-dynamic-async.html#Async-Components).
This allows GitLab to load the correct component in EE, while in CE GitLab loads an empty component
that renders nothing. This code **must** exist in the CE repository, in addition to the EE repository.
diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md
index 47942817790..b3996e16fa1 100644
--- a/doc/development/elasticsearch.md
+++ b/doc/development/elasticsearch.md
@@ -277,7 +277,7 @@ These Advanced Search migrations, like any other GitLab changes, need to support
Depending on the order of deployment, it's possible that the migration
has started or finished and there's still a server running the application code from before the
-migration. We need to take this into consideration until we can
+migration. We need to take this into consideration until we can
[ensure all Advanced Search migrations start after the deployment has finished](https://gitlab.com/gitlab-org/gitlab/-/issues/321619).
### Reverting a migration
@@ -317,7 +317,7 @@ safely can.
We choose to use GitLab major version upgrades as a safe time to remove
backwards compatibility for indices that have not been fully migrated. We
-[document this in our upgrade documentation](../update/index.md#upgrading-to-a-new-major-version).
+[document this in our upgrade documentation](../update/index.md#upgrading-to-a-new-major-version).
We also choose to replace the migration code with the halted migration
and remove tests so that:
@@ -399,7 +399,7 @@ that may contain information to help diagnose performance issues.
### Performance Bar
-Elasticsearch requests will be displayed in the
+Elasticsearch requests will be displayed in the
[`Performance Bar`](../administration/monitoring/performance/performance_bar.md), which can
be used both locally in development and on any deployed GitLab instance to
diagnose poor search performance. This will show the exact queries being made,
@@ -495,7 +495,7 @@ theoretically be used to figure out what needs to be replayed are:
These updates can be replayed by triggering another
`ElasticDeleteProjectWorker`.
-With the above methods and taking regular
+With the above methods and taking regular
[Elasticsearch snapshots](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html)
we should be able to recover from different kinds of data loss issues in a
relatively short period of time compared to indexing everything from
diff --git a/doc/development/emails.md b/doc/development/emails.md
index 1b3c9226dd8..c997916aa21 100644
--- a/doc/development/emails.md
+++ b/doc/development/emails.md
@@ -160,9 +160,9 @@ and Helm Chart configuration (see [example merge request](https://gitlab.com/git
#### Rationale
This was done because to avoid [thread deadlocks](https://github.com/ruby/net-imap/issues/14), `MailRoom` needs
-an updated version of the `net-imap` gem. However, this
-[version of the net-imap cannot be installed by an unprivileged user](https://github.com/ruby/net-imap/issues/14) due to
-[an error installing the digest gem](https://github.com/ruby/digest/issues/14).
+an updated version of the `net-imap` gem. However, this
+[version of the net-imap cannot be installed by an unprivileged user](https://github.com/ruby/net-imap/issues/14) due to
+[an error installing the digest gem](https://github.com/ruby/digest/issues/14).
[This bug in the Ruby interpreter](https://bugs.ruby-lang.org/issues/17761) was fixed in Ruby
3.0.2.
diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md
index e11e516485a..500a19fe1ad 100644
--- a/doc/development/experiment_guide/index.md
+++ b/doc/development/experiment_guide/index.md
@@ -10,7 +10,7 @@ Experiments can be conducted by any GitLab team, most often the teams from the
[Growth Sub-department](https://about.gitlab.com/handbook/engineering/development/growth/).
Experiments are not tied to releases because they primarily target GitLab.com.
-Experiments are run as an A/B/n test, and are behind an [experiment feature flag](../feature_flags/#experiment-type)
+Experiments are run as an A/B/n test, and are behind an [experiment feature flag](../feature_flags/index.md#experiment-type)
to turn the test on or off. Based on the data the experiment generates, the team decides
if the experiment had a positive impact and should be made the new default, or rolled back.
diff --git a/doc/development/export_csv.md b/doc/development/export_csv.md
index 0f50d1438fc..29e80f676da 100644
--- a/doc/development/export_csv.md
+++ b/doc/development/export_csv.md
@@ -11,10 +11,10 @@ This document lists the different implementations of CSV export in GitLab codeba
| Export type | How it works | Advantages | Disadvantages | Existing examples |
|---|---|---|---|---|
| Streaming | - Query and yield data in batches to a response stream.<br>- Download starts immediately. | - Report available immediately. | - No progress indicator.<br>- Requires a reliable connection. | [Export Audit Event Log](../administration/audit_events.md#export-to-csv) |
-| Downloading | - Query and write data in batches to a temporary file.<br>- Loads the file into memory.<br>- Sends the file to the client. | - Report available immediately. | - Large amount of data might cause request timeout.<br>- Memory intensive.<br>- Request expires when user navigates to a different page. | - [Export Chain of Custody Report](../user/compliance/compliance_report/#chain-of-custody-report)<br>- [Export License Usage File](../subscriptions/self_managed/index.md#export-your-license-usage) |
+| Downloading | - Query and write data in batches to a temporary file.<br>- Loads the file into memory.<br>- Sends the file to the client. | - Report available immediately. | - Large amount of data might cause request timeout.<br>- Memory intensive.<br>- Request expires when user navigates to a different page. | - [Export Chain of Custody Report](../user/compliance/compliance_report/index.md#chain-of-custody-report)<br>- [Export License Usage File](../subscriptions/self_managed/index.md#export-your-license-usage) |
| As email attachment | - Asynchronously process the query with background job.<br>- Email uses the export as an attachment. | - Asynchronous processing. | - Requires users use a different app (email) to download the CSV.<br>- Email providers may limit attachment size. | - [Export issues](../user/project/issues/csv_export.md)<br>- [Export merge requests](../user/project/merge_requests/csv_export.md) |
| As downloadable link in email (*) | - Asynchronously process the query with background job.<br>- Email uses an export link. | - Asynchronous processing.<br>- Bypasses email provider attachment size limit. | - Requires users use a different app (email).<br>- Requires additional storage and cleanup. | [Export User Permissions](https://gitlab.com/gitlab-org/gitlab/-/issues/1772) |
-| Polling (non-persistent state) | - Asynchronously processes the query with the background job.<br>- Frontend(FE) polls every few seconds to check if CSV file is ready. | - Asynchronous processing.<br>- Automatically downloads to local machine on completion.<br>- In-app solution. | - Non-persistable request - request expires when user navigates to a different page.<br>- API is processed for each polling request. | [Export Vulnerabilities](../user/application_security/vulnerability_report/#export-vulnerability-details) |
+| Polling (non-persistent state) | - Asynchronously processes the query with the background job.<br>- Frontend(FE) polls every few seconds to check if CSV file is ready. | - Asynchronous processing.<br>- Automatically downloads to local machine on completion.<br>- In-app solution. | - Non-persistable request - request expires when user navigates to a different page.<br>- API is processed for each polling request. | [Export Vulnerabilities](../user/application_security/vulnerability_report/index.md#export-vulnerability-details) |
| Polling (persistent state) (*) | - Asynchronously processes the query with background job.<br>- Backend (BE) maintains the export state<br>- FE polls every few seconds to check status.<br>- FE shows 'Download link' when export is ready.<br>- User can download or regenerate a new report. | - Asynchronous processing.<br>- No database calls made during the polling requests (HTTP 304 status is returned until export status changes).<br>- Does not require user to stay on page until export is complete.<br>- In-app solution.<br>- Can be expanded into a generic CSV feature (such as dashboard / CSV API). | - Requires to maintain export states in DB.<br>- Does not automatically download the CSV export to local machine, requires users to select 'Download'. | [Export Merge Commits Report](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43055) |
NOTE:
diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md
index 442dda20d23..6dcc57b0ff5 100644
--- a/doc/development/fe_guide/graphql.md
+++ b/doc/development/fe_guide/graphql.md
@@ -729,8 +729,8 @@ In this case, we can either:
- Skip passing a cursor.
- Pass `null` explicitly to `after`.
-After data is fetched, we can use the `update`-hook as an opportunity
-[to customize the data that is set in the Vue component property](https://apollo.vuejs.org/api/smart-query.html#options).
+After data is fetched, we can use the `update`-hook as an opportunity
+[to customize the data that is set in the Vue component property](https://apollo.vuejs.org/api/smart-query.html#options).
This allows us to get a hold of the `pageInfo` object among other data.
In the `result`-hook, we can inspect the `pageInfo` object to see if we need to fetch
diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md
index 02086ec5f1b..d90c270153e 100644
--- a/doc/development/fe_guide/index.md
+++ b/doc/development/fe_guide/index.md
@@ -36,7 +36,8 @@ Sign in to BrowserStack with the credentials saved in the **Engineering** vault
## Initiatives
-Current high-level frontend goals are listed on [Frontend Epics](https://gitlab.com/groups/gitlab-org/-/epics?label_name%5B%5D=frontend).
+You can find current frontend initiatives with a cross-functional impact on epics
+with the label [frontend-initiative](https://gitlab.com/groups/gitlab-org/-/epics?state=opened&page=1&sort=UPDATED_AT_DESC&label_name[]=frontend-initiative).
## Principles
diff --git a/doc/development/fe_guide/storybook.md b/doc/development/fe_guide/storybook.md
index 45342eb6d72..a3a1fa2160f 100644
--- a/doc/development/fe_guide/storybook.md
+++ b/doc/development/fe_guide/storybook.md
@@ -46,10 +46,12 @@ To add a story:
1. Write the story as per the [official Storybook instructions](https://storybook.js.org/docs/vue/writing-stories/introduction/)
- Notes:
- - Specify the `title` field of the story as the component's file path from the `javascripts/` directory.
-
- For example, if the component is located at `app/assets/javascripts/vue_shared/components/sidebar/todo_toggle/todo_button.vue`, specify the story `title` as `vue_shared/components/sidebar/todo_toggle/todo_button`. This will ensure the Storybook navigation maps closely to our internal directory structure.
+ NOTE:
+ Specify the `title` field of the story as the component's file path from the `javascripts/` directory, without the `/components` part.
+ For example, if the component is located at `app/assets/javascripts/vue_shared/components/sidebar/todo_toggle/todo_button.vue`,
+ specify the story `title` as `vue_shared/sidebar/todo_toggle/todo_button`.
+ If the component is located in the `ee/` directory, make sure to prefix the story's title with `ee/` as well.
+ This will ensure the Storybook navigation maps closely to our internal directory structure.
## Mock backend APIs
diff --git a/doc/development/fe_guide/style/vue.md b/doc/development/fe_guide/style/vue.md
index c9bd0e1b35a..39dc9cc9c4e 100644
--- a/doc/development/fe_guide/style/vue.md
+++ b/doc/development/fe_guide/style/vue.md
@@ -121,7 +121,7 @@ Check the [rules](https://github.com/vuejs/eslint-plugin-vue#bulb-rules) for mor
1. **Extensions**: Use `.vue` extension for Vue components. Do not use `.js` as file extension
([#34371](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34371)).
-1. **Reference Naming**: Use PascalCase for their instances:
+1. **Reference Naming**: Use PascalCase for their default imports:
```javascript
// bad
@@ -431,7 +431,7 @@ must be unique. It's advised to use `kebab-case` namespaces.
Useful links:
-1. [`key`](https://vuejs.org/v2/guide/list.html#key)
+1. [Maintaining State](https://v2.vuejs.org/v2/guide/list.html#Maintaining-State)
1. [Vue Style Guide: Keyed v-for](https://vuejs.org/v2/style-guide/#Keyed-v-for-essential)
## Vue testing
@@ -448,10 +448,8 @@ Typically, when testing a Vue component, the component should be "re-mounted" in
To achieve this:
1. Create a mutable `wrapper` variable inside the top-level `describe` block.
-1. Mount the component using [`mount`](https://v1.test-utils.vuejs.org/api/#mount)/
-[`shallowMount`](https://v1.test-utils.vuejs.org/api/#shallowMount).
-1. Reassign the resulting [`Wrapper`](https://v1.test-utils.vuejs.org/api/wrapper/#wrapper)
-instance to our `wrapper` variable.
+1. Mount the component using [`mount`](https://v1.test-utils.vuejs.org/api/#mount) or [`shallowMount`](https://v1.test-utils.vuejs.org/api/#shallowMount).
+1. Reassign the resulting [`Wrapper`](https://v1.test-utils.vuejs.org/api/wrapper/#wrapper) instance to our `wrapper` variable.
Creating a global, mutable wrapper provides a number of advantages, including the ability to:
@@ -671,6 +669,6 @@ In the coming months you should fix that tech debt, with its priority to be dete
not be rewritten. For example, jQuery tests rewritten to Vue tests.
1. You may choose to use VueX as a centralized state management. If you choose not to use VueX, you
must use the *store pattern* which can be found in the
-[Vue.js documentation](https://vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch).
+[Vue.js documentation](https://v2.vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch).
1. Once you have chosen a centralized state-management solution you must use it for your entire
application. Don't mix and match your state-management solutions.
diff --git a/doc/development/fe_guide/tooling.md b/doc/development/fe_guide/tooling.md
index 2bb6cbfaf7a..4b55580c33c 100644
--- a/doc/development/fe_guide/tooling.md
+++ b/doc/development/fe_guide/tooling.md
@@ -63,8 +63,8 @@ disabled due to legacy compatibility reasons but they are in the process of bein
Do not disable specific ESLint rules. To avoid introducing technical debt, you may disable the following
rules only if you are invoking/instantiating existing code modules.
-- [`no-new`](https://eslint.org/docs/rules/no-new)
-- [`class-method-use-this`](https://eslint.org/docs/rules/class-methods-use-this)
+- [`no-new`](https://eslint.org/docs/latest/rules/no-new)
+- [`class-method-use-this`](https://eslint.org/docs/latest/rules/class-methods-use-this)
Disable these rules on a per-line basis. This makes it easier to refactor in the
future. For example, use `eslint-disable-next-line` or `eslint-disable-line`.
diff --git a/doc/development/fe_guide/troubleshooting.md b/doc/development/fe_guide/troubleshooting.md
index c0894621ed1..ab10c5bf988 100644
--- a/doc/development/fe_guide/troubleshooting.md
+++ b/doc/development/fe_guide/troubleshooting.md
@@ -4,7 +4,7 @@ group: unassigned
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
-# Troubleshooting
+# Troubleshooting frontend development issues
Running into a problem? Maybe this will help ¯\_(ツ)_/¯.
diff --git a/doc/development/fe_guide/view_component.md b/doc/development/fe_guide/view_component.md
index 2e373e6933b..7ddce609ee7 100644
--- a/doc/development/fe_guide/view_component.md
+++ b/doc/development/fe_guide/view_component.md
@@ -33,7 +33,7 @@ Consider this list a best effort. The full list can be found in [`app/components
#### Alert
-The `Pajamas::AlertComponent` follows the [Pajamas Alert](https://design.gitlab.com/components/alert) specification.
+The `Pajamas::AlertComponent` follows the [Pajamas Alert](https://design.gitlab.com/components/alert/) specification.
**Examples:**
@@ -57,7 +57,7 @@ For the full list of options, see its
#### Banner
-The `Pajamas::BannerComponent` follows the [Pajamas Banner](https://design.gitlab.com/components/banner) specification.
+The `Pajamas::BannerComponent` follows the [Pajamas Banner](https://design.gitlab.com/components/banner/) specification.
**Examples:**
@@ -88,7 +88,7 @@ For the full list of options, see its
#### Button
-The `Pajamas::ButtonComponent` follows the [Pajamas Button](https://design.gitlab.com/components/button) specification.
+The `Pajamas::ButtonComponent` follows the [Pajamas Button](https://design.gitlab.com/components/button/) specification.
**Examples:**
@@ -125,7 +125,7 @@ For the full list of options, see its
#### Card
-The `Pajamas::CardComponent` follows the [Pajamas Card](https://design.gitlab.com/components/card) specification.
+The `Pajamas::CardComponent` follows the [Pajamas Card](https://design.gitlab.com/components/card/) specification.
**Examples:**
@@ -188,7 +188,7 @@ For the full list of options, see its
#### Toggle
-The `Pajamas::ToggleComponent` follows the [Pajamas Toggle](https://design.gitlab.com/components/toggle) specification.
+The `Pajamas::ToggleComponent` follows the [Pajamas Toggle](https://design.gitlab.com/components/toggle/) specification.
```haml
= render Pajamas::ToggleComponent.new(classes: 'js-force-push-toggle',
@@ -205,7 +205,34 @@ To actually initialize this component, make sure to call the `initToggle` helper
For the full list of options, see its
[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/components/pajamas/toggle_component.rb).
-### Best practices
+## Layouts
+
+Layout components can be used to create common layout patterns used in GitLab.
+
+### Available components
+
+#### Horizontal section
+
+Many of the settings pages use a layout where the title and description are on the left and the settings fields are on the right. The `Layouts::HorizontalSectionComponent` can be used to create this layout.
+
+**Example:**
+
+```haml
+= render ::Layouts::HorizontalSectionComponent.new(options: { class: 'gl-mb-6' }) do |c|
+ = c.title { _('Naming, visibility') }
+ = c.description do
+ = _('Update your group name, description, avatar, and visibility.')
+ = link_to _('Learn more about groups.'), help_page_path('user/group/index')
+ = c.body do
+ .form-group.gl-form-group
+ = f.label :name, _('New group name')
+ = f.text_field :name
+```
+
+For the full list of options, see its
+[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/components/layouts/horizontal_section_component.rb).
+
+## Best practices
- If you are about to create a new view in Haml, use the available components
over creating plain Haml tags with CSS classes.
diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md
index 27660c0f5f7..5cb461c8ca0 100644
--- a/doc/development/fe_guide/vue.md
+++ b/doc/development/fe_guide/vue.md
@@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Vue
-To get started with Vue, read through [their documentation](https://vuejs.org/v2/guide/).
+To get started with Vue, read through [their documentation](https://v2.vuejs.org/v2/guide/index.html).
## Examples
@@ -23,8 +23,8 @@ The main goal we are trying to achieve is to have only one data flow, and only o
To achieve this goal we use [Vuex](#vuex).
You can also read about this architecture in Vue documentation about
-[state management](https://vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch)
-and about [one way data flow](https://vuejs.org/v2/guide/components.html#One-Way-Data-Flow).
+[state management](https://v2.vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch)
+and about [one way data flow](https://v2.vuejs.org/v2/guide/components-props.html#One-Way-Data-Flow).
### Components and Store
@@ -322,7 +322,7 @@ For example, tables are used in a quite amount of places across GitLab, a table
would be a good fit for a component. On the other hand, a table cell used only
in one table would not be a good use of this pattern.
-You can read more about components in Vue.js site, [Component System](https://vuejs.org/v2/guide/#Composing-with-Components).
+You can read more about components in Vue.js site, [Component System](https://v2.vuejs.org/v2/guide/#Composing-with-Components).
### A folder for the Store
@@ -348,8 +348,7 @@ recommended to observe objects with their own stateful behavior.
Based on the Vue guidance:
-- **Do not** use or create a JavaScript class in your [data function](https://v2.vuejs.org/v2/api/#data),
-such as `user: new User()`.
+- **Do not** use or create a JavaScript class in your [data function](https://v2.vuejs.org/v2/api/#data).
- **Do not** add new JavaScript class implementations.
- **Do** use [GraphQL](../api_graphql_styleguide.md), [Vuex](vuex.md) or a set of components if
cannot use primitives or objects.
@@ -531,8 +530,7 @@ Each Vue component has a unique output. This output is always present in the ren
Although each method of a Vue component can be tested individually, our goal is to test the output
of the render function, which represents the state at all times.
-Visit the [Vue testing guide](https://v2.vuejs.org/v2/guide/testing.html#Unit-Testing) for help
-testing the rendered output.
+Visit the [Vue testing guide](https://v2.vuejs.org/v2/guide/testing.html#Unit-Testing) for help.
Here's an example of a well structured unit test for [this Vue component](#appendix---vue-component-subject-under-test):
@@ -671,7 +669,7 @@ it('should fire the click event', () => {
})
```
-When firing a Vue event, use [`emit`](https://vuejs.org/v2/guide/components-custom-events.html).
+When firing a Vue event, use [`emit`](https://v2.vuejs.org/v2/guide/components-custom-events.html).
```javascript
wrapper = shallowMount(DropdownItem);
diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md
index 8bfb912161a..2d1569b7812 100644
--- a/doc/development/fe_guide/vuex.md
+++ b/doc/development/fe_guide/vuex.md
@@ -245,7 +245,7 @@ A mutation written like this is easier to maintain. In addition, we avoid errors
### `getters.js`
Sometimes we may need to get derived state based on store state, like filtering for a specific prop.
-Using a getter also caches the result based on dependencies due to [how computed props work](https://vuejs.org/v2/guide/computed.html#Computed-Caching-vs-Methods)
+Using a getter also caches the result based on dependencies due to [how computed props work](https://v2.vuejs.org/v2/guide/computed.html#Computed-Caching-vs-Methods)
This can be done through the `getters`:
```javascript
@@ -364,7 +364,7 @@ export default initialState => ({
We made the conscious decision to avoid this pattern to improve the ability to
discover and search our frontend codebase. The same applies
-when [providing data to a Vue app](vue.md#providing-data-from-haml-to-javascript). The reasoning for this is described in
+when [providing data to a Vue app](vue.md#providing-data-from-haml-to-javascript). The reasoning for this is described in
[this discussion](https://gitlab.com/gitlab-org/frontend/rfcs/-/issues/56#note_302514865):
> Consider a `someStateKey` is being used in the store state. You _may_ not be
diff --git a/doc/development/fe_guide/widgets.md b/doc/development/fe_guide/widgets.md
index b54f9add97d..c6bb89d1fe8 100644
--- a/doc/development/fe_guide/widgets.md
+++ b/doc/development/fe_guide/widgets.md
@@ -18,11 +18,11 @@ When building a widget, we should follow a few principles described below.
All widgets should use the same stack (Vue + Apollo Client).
To make it happen, we must add Vue Apollo to the application root (if we use a widget
as a component) or provide it directly to a widget. For sidebar widgets, use the
-[sidebar Apollo Client and Apollo Provider](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/sidebar/graphql.js):
+[issuable Apollo Client and Apollo Provider](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/graphql_shared/issuable_client.js):
```javascript
import SidebarConfidentialityWidget from '~/sidebar/components/confidential/sidebar_confidentiality_widget.vue';
-import { apolloProvider } from '~/sidebar/graphql';
+import { apolloProvider } from '~/graphql_shared/issuable_client';
function mountConfidentialComponent() {
new Vue({
@@ -118,7 +118,7 @@ In this case, we can use a renderless component that imports a client and listen
```javascript
import { fetchPolicies } from '~/lib/graphql';
import { confidentialityQueries } from '~/sidebar/constants';
-import { defaultClient as gqlClient } from '~/sidebar/graphql';
+import { defaultClient as gqlClient } from '~/graphql_shared/issuable_client';
created() {
if (this.issuableType !== IssuableType.Issue) {
diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md
index e50c1edd282..fd1c7f4afa5 100644
--- a/doc/development/feature_development.md
+++ b/doc/development/feature_development.md
@@ -174,6 +174,7 @@ See [database guidelines](database/index.md).
## Domain-specific guides
- [CI/CD development documentation](cicd/index.md)
+- [Sec Section development documentation](sec/index.md)
## Technical Reference by Group
diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md
index 8a862e5f7cd..63dad3070c7 100644
--- a/doc/development/feature_flags/controls.md
+++ b/doc/development/feature_flags/controls.md
@@ -112,6 +112,8 @@ incidents or in-progress change issues, for example:
2021-06-29 Canary deployment failing QA tests
```
+Before enabling a feature flag, verify that you are not violating any [Production Change Lock periods](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl) and are in compliance with the [Feature Flags and the Change Management Process](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process).
+
The following `/chatops` commands should be performed in the Slack
`#production` channel.
@@ -337,7 +339,7 @@ take one of the following actions:
To remove a feature flag, open **one merge request** to make the changes. In the MR:
-1. Add the ~"feature flag" label so release managers are aware the changes are hidden behind a feature flag.
+1. Add the ~"feature flag" label so release managers are aware of the removal.
1. If the merge request has to be picked into a stable branch, add the
appropriate `~"Pick into X.Y"` label, for example `~"Pick into 13.0"`.
See [the feature flag process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release)
diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md
index 502a028f089..444b53f9c8d 100644
--- a/doc/development/feature_flags/index.md
+++ b/doc/development/feature_flags/index.md
@@ -433,8 +433,8 @@ When using the percentage rollout of actors on multiple feature flags, the actor
For example, the following feature flags are enabled for a certain percentage of actors:
```plaintext
-/chatops run chatops feature set feature-set-1 25 --actors
-/chatops run chatops feature set feature-set-2 25 --actors
+/chatops run feature set feature-set-1 25 --actors
+/chatops run feature set feature-set-2 25 --actors
```
If a project A has `:feature-set-1` enabled, there is no guarantee that project A also has `:feature-set-2` enabled.
@@ -514,12 +514,12 @@ You can also enable a feature flag for a given gate:
Feature.enable(:feature_flag_name, Project.find_by_full_path("root/my-project"))
```
-### Removing a feature flag locally (in development)
+### Disabling a feature flag locally (in development)
When manually enabling or disabling a feature flag from the Rails console, its default value gets overwritten.
This can cause confusion when changing the flag's `default_enabled` attribute.
-To reset the feature flag to the default status, you can remove it in the rails console (`rails c`)
+To reset the feature flag to the default status, you can disable it in the rails console (`rails c`)
as follows:
```ruby
@@ -535,16 +535,18 @@ Feature.remove(:feature_flag_name)
```mermaid
graph LR
- A[flag: default off] -->|'added' / 'changed'| B(flag: default on)
+ A[flag: default off] -->|'added' / 'changed' / 'fixed' / '...'| B(flag: default on)
B -->|'other'| C(remove flag, keep new code)
B -->|'removed' / 'changed'| D(remove flag, keep old code)
- A -->|'added' / 'changed'| C
+ A -->|'added' / 'changed' / 'fixed' / '...'| C
A -->|no changelog| D
```
- Any change behind a feature flag that is **enabled** by default **should** have a changelog entry.
- The changelog for a feature flag should describe the feature and not the
flag, unless a default on feature flag is removed keeping the new code (`other` in the flowchart above).
+- A feature flag can also be used for rolling out a bug fix or a maintenance work. In this scenario, the changelog
+ must be related to it, for example; `fixed` or `other`.
## Feature flags in tests
diff --git a/doc/development/fips_compliance.md b/doc/development/fips_compliance.md
index c690408ee60..1029ed88eac 100644
--- a/doc/development/fips_compliance.md
+++ b/doc/development/fips_compliance.md
@@ -65,7 +65,7 @@ listed here that also do not work properly in FIPS mode:
- [Solutions for vulnerabilities](../user/application_security/vulnerabilities/index.md#resolve-a-vulnerability)
for yarn projects.
- [Static Application Security Testing (SAST)](../user/application_security/sast/index.md)
- supports a reduced set of [analyzers](../user/application_security/sast/#fips-enabled-images)
+ supports a reduced set of [analyzers](../user/application_security/sast/index.md#fips-enabled-images)
when operating in FIPS-compliant mode.
- Advanced Search is currently not included in FIPS mode. It must not be enabled in order to be FIPS-compliant.
- [Gravatar or Libravatar-based profile images](../administration/libravatar.md) are not FIPS-compliant.
@@ -578,6 +578,6 @@ Merge requests that can trigger Package and QA, can trigger a FIPS package and a
Reference Architecture test pipeline. The base image used for the trigger is
Ubuntu 20.04 FIPS:
-1. Trigger `package-and-qa`, if not already triggered.
+1. Trigger `e2e:package-and-test` job, if not already triggered.
1. On the `gitlab-omnibus-mirror` child pipeline, manually trigger `Trigger:package:fips`.
1. When the package job is complete, manually trigger the `RAT:FIPS` job.
diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md
index f9cf69020bb..87304a761ea 100644
--- a/doc/development/gemfile.md
+++ b/doc/development/gemfile.md
@@ -4,7 +4,7 @@ group: unassigned
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
-# `Gemfile` guidelines
+# Gemfile guidelines
When adding a new entry to `Gemfile` or upgrading an existing dependency pay
attention to the following rules.
@@ -15,6 +15,23 @@ We do not allow gems that are fetched from Git repositories. All gems have
to be available in the RubyGems index. We want to minimize external build
dependencies and build times.
+## Request an Appsec review
+
+When adding a new gem to our `Gemfile` or even changing versions in
+`Gemfile.lock` it is strongly recommended that you
+[request a Security review](https://about.gitlab.com/handbook/engineering/security/#how-to-request-a-security-review).
+New gems add an extra security risk for GitLab, and it is important to
+evaluate this risk before we ship this to production. Technically, just adding
+a new gem and pushing to a branch in our main `gitlab` project is a security
+risk as it will run in CI using your GitLab.com credentials. As such you should
+evaluate early on if you think this gem seems legitimate before you even
+install it.
+
+Reviewers should also be aware of our related
+[recommendations for reviewing community contributions](code_review.md#community-contributions)
+and take care before running a pipeline for community contributions that
+contains changes to `Gemfile` or `Gemfile.lock`.
+
## License compliance
Refer to [licensing guidelines](licensing.md) for ensuring license compliance.
@@ -73,7 +90,7 @@ to a gem, go through these steps:
apply if someone who currently works at GitLab wants to maintain
the gem beyond their time working at GitLab.
-When publishing a gem to RubyGems.org, also note the section on
+When publishing a gem to RubyGems.org, also note the section on
[gem owners](https://about.gitlab.com/handbook/developer-onboarding/#ruby-gems)
in the handbook.
@@ -132,7 +149,7 @@ that also relied on `thor` but had its version pinned to a vulnerable
one. These changes are easy to miss in the `Gemfile.lock`. Pinning the
version would result in a conflict that would need to be solved.
-To avoid upgrading indirect dependencies, we can use
+To avoid upgrading indirect dependencies, we can use
[`bundle update --conservative`](https://bundler.io/man/bundle-update.1.html#OPTIONS).
When submitting a merge request including a dependency update,
diff --git a/doc/development/geo.md b/doc/development/geo.md
index f042af42de5..1ae9d9ee32b 100644
--- a/doc/development/geo.md
+++ b/doc/development/geo.md
@@ -653,8 +653,7 @@ on, check out our [self-service framework](geo/framework.md).
### GET:Geo pipeline
-As part of the [package-and-qa](testing_guide/end_to_end/index.md#using-the-package-and-qa-job) pipeline, there is an option to manually trigger a job named `GET:Geo`. This
-pipeline uses [GET](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) to spin up a
+As part of the [e2e:package-and-test](testing_guide/end_to_end/index.md#using-the-package-and-test-job) pipeline, there is an option to manually trigger a job named `GET:Geo`. This pipeline uses [GET](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) to spin up a
[1k](../administration/reference_architectures/1k_users.md) Geo installation,
and run the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa) Geo scenario against the instance.
When working on Geo features, it is a good idea to ensure the `qa-geo` job passes in a triggered `GET:Geo pipeline`.
@@ -669,7 +668,7 @@ see the [QA documentation](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa
The pipeline involves the interaction of multiple different projects:
-- [GitLab](https://gitlab.com/gitlab-org/gitlab) - The [package-and-qa job](testing_guide/end_to_end/index.md#using-the-package-and-qa-job) is launched from merge requests in this project.
+- [GitLab](https://gitlab.com/gitlab-org/gitlab) - The [`e2e:package-and-test` job](testing_guide/end_to_end/index.md#using-the-package-and-test-job) is launched from merge requests in this project.
- [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) - Builds relevant artifacts containing the changes from the triggering merge request pipeline.
- [GET-Configs/Geo](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/Geo) - Coordinates the lifecycle of a short-lived Geo installation that can be evaluated.
- [GET](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) - Contains the necessary logic for creating and destroying Geo installations. Used by `GET-Configs/Geo`.
diff --git a/doc/development/geo/proxying.md b/doc/development/geo/proxying.md
index 2f0226c489c..d4cb611e965 100644
--- a/doc/development/geo/proxying.md
+++ b/doc/development/geo/proxying.md
@@ -128,7 +128,7 @@ Secondary-->>Client: admin/geo/replication/projects logged in response (session
## Git pull
-For historical reasons, the `push_from_secondary` path is used to forward a Git pull. There is
+For historical reasons, the `push_from_secondary` path is used to forward a Git pull. There is
[an issue proposing to rename this route](https://gitlab.com/gitlab-org/gitlab/-/issues/292690) to avoid confusion.
### Git pull over HTTP(s)
diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md
index a6b359769f8..a20bdf633cd 100644
--- a/doc/development/git_object_deduplication.md
+++ b/doc/development/git_object_deduplication.md
@@ -18,7 +18,7 @@ GitLab implements Git object deduplication.
### Understanding Git alternates
-At the Git level, we achieve deduplication by using
+At the Git level, we achieve deduplication by using
[Git alternates](https://git-scm.com/docs/gitrepository-layout#gitrepository-layout-objects).
Git alternates is a mechanism that lets a repository borrow objects from
another repository on the same machine.
@@ -99,7 +99,7 @@ are as follows:
### Assumptions
-- All repositories in a pool must use [hashed storage](../administration/repository_storage_types.md).
+- All repositories in a pool must use [hashed storage](../administration/repository_storage_types.md).
This is so that we don't have to ever worry about updating paths in
`object/info/alternates` files.
- All repositories in a pool must be on the same Gitaly storage shard.
diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md
index 0aa1bad711d..9740ae04553 100644
--- a/doc/development/github_importer.md
+++ b/doc/development/github_importer.md
@@ -71,7 +71,7 @@ This worker imports all pull requests. For every pull request a job for the
### 5. Stage::ImportPullRequestsMergedByWorker
-This worker imports the pull requests' _merged-by_ user information. The
+This worker imports the pull requests' _merged-by_ user information. The
[_List pull requests_](https://docs.github.com/en/rest/pulls#list-pull-requests)
API doesn't provide this information. Therefore, this stage must fetch each merged pull request
individually to import this information. A
@@ -101,7 +101,20 @@ label links in the same worker removes the need for performing a separate crawl
through the API data, reducing the number of API calls necessary to import a
project.
-### 8. Stage::ImportNotesWorker
+### 8. Stage::ImportIssueEventsWorker
+
+This worker imports all issues and pull request events. For every event, we
+schedule a job for the `Gitlab::GithubImport::ImportIssueEventWorker` worker.
+
+We can import both issues and pull request events by single stage because of a specific aspect of the GitHub API. It looks like that under the hood, issues and pull requests
+GitHub are stored in a single table. Therefore, they have globally-unique IDs and so:
+
+- Every pull request is an issue.
+- Issues aren't pull requests.
+
+Therefore, both issues and pull requests have a common API for most related things.
+
+### 9. Stage::ImportNotesWorker
This worker imports regular comments for both issues and pull requests. For
every comment, we schedule a job for the
@@ -112,7 +125,28 @@ returns comments for both issues and pull requests. This means we have to wait
for all issues and pull requests to be imported before we can import regular
comments.
-### 9. Stage::FinishImportWorker
+### 10. Stage::ImportAttachmentsWorker
+
+This worker imports release notes attachments that are linked inside Markdown.
+For every release of the project, we schedule a job of
+`Gitlab::GithubImport::ImportReleaseAttachmentsWorker` for every comment.
+
+Each job:
+
+1. Iterates over all attachment links inside of a specific release note.
+1. Downloads the attachment.
+1. Replaces the old link with a newly-generated link to GitLab.
+
+### 11. Stage::ImportProtectedBranchesWorker
+
+This worker imports protected branch rules.
+For every rule that exists on GitHub, we schedule a job of
+`Gitlab::GithubImport::ImportProtectedBranchWorker`.
+
+Each job compares the branch protection rules from GitHub and GitLab and applies
+the strictest of the rules to the branches in GitLab.
+
+### 12. Stage::FinishImportWorker
This worker completes the import process by performing some housekeeping
(such as flushing any caches) and by marking the import as completed.
diff --git a/doc/development/gitlab_flavored_markdown/specification_guide/index.md b/doc/development/gitlab_flavored_markdown/specification_guide/index.md
index 756b87cd407..c1227e5d33f 100644
--- a/doc/development/gitlab_flavored_markdown/specification_guide/index.md
+++ b/doc/development/gitlab_flavored_markdown/specification_guide/index.md
@@ -345,8 +345,50 @@ For the [Markdown snapshot testing](#markdown-snapshot-testing) to work
properly, you must account for these differences in a way that ensures the tests are reliable,
and always behave the same across different test runs or environments.
-To account for these differences, there is a process called **_normalization_**. Normalization
-allows custom regular expressions with
+To account for these differences, there is a process called **_normalization_**. Several ways to approach normalization exist:
+
+1. Fixture-based normalization
+1. Environment-variable-based normalization
+1. Regex-based normalization
+
+#### Fixture-based normalization
+
+Fixture-based normalization should be used whenever possible, because it is simpler and easier to
+understand than regex-based normalization.
+
+The [Markdown snapshot testing](#markdown-snapshot-testing) uses RSpec to generate the
+[example snapshot files](#example-snapshot-files). RSpec enables you to:
+
+- Use the same powerful fixture support and helpers as all the rest of the GitLab RSpec suite.
+- Use fixtures to control the state of the database when the example snapshots are generated.
+- Extract this fixture setup to an RSpec shared context. This shared context is used to ensure
+ the same database state exists wherever the snapshot tests are run, either by the CI suite, or
+ locally via [`run-snapshot-tests.sh`](#run-snapshot-testssh-script).
+
+You can see the RSpec shared context containing these fixtures in
+[`spec/support/shared_contexts/glfm/example_snapshot_fixtures.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/support/shared_contexts/glfm/example_snapshot_fixtures.rb).
+
+#### Environment-variable-based normalization
+
+In some cases, fixtures may not be usable, because they do not provide control over the varying
+values. In these cases, we can introduce support for a environment variable into the production
+code, which allows us to override the randommness in our test environment when we are
+generating the HTML for footnote examples. Even though it is in the production code path, it has
+no effect unless it is explicitly set, therefore it is innocuous. It allows us to avoid
+the more-complex regex-based normalization described below.
+
+The current example of this is when normally random footnote IDs are overridden to be deterministic
+by setting `GITLAB_TEST_FOOTNOTE_ID`. It is set along with the fixtures setup in the
+[`spec/support/shared_contexts/glfm/example_snapshot_fixtures.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/support/shared_contexts/glfm/example_snapshot_fixtures.rb)
+shared context.
+
+#### Regex-based normalization
+
+If neither fixture-based nor environment-variable-based normalization can be used, use regex-based
+normalization. It is powerful, but more complex, and requires more maintenance.
+It requires referring to specific examples by name, and crafting the proper regexes.
+
+Regex-based normalization allows custom regular expressions with
[_capturing groups_](https://ruby-doc.org/core-3.1.2/Regexp.html#class-Regexp-label-Capturing)
to be applied to two different versions of HTML or JSON for a given Markdown example,
and the contents of the captured groups can be replaced with the same fixed values.
@@ -653,10 +695,16 @@ is the manually updated canonical Markdown+HTML examples for GLFM extensions.
- It contains examples in the [standard backtick-delimited `spec.txt` format](#various-markdown-specifications),
each of which contain a Markdown example and the corresponding canonical HTML.
+- For all GitLab examples, the "extension" annotation after the backticks should consist of only
+ `example gitlab`. It does not currently include any additional extension annotations describing
+ the specific Markdown, unlike the GitHub Flavored Markdown examples, which do include
+ these additional annotations (such as `example strikethrough`).
- The `update-specification.rb` script inserts it as new sections before the appendix
of generated `spec.txt`.
-- It should consist of `H1` header sections, with all examples nested exactly 2 levels deep within `H2`
- header sections.
+- It should consist of `H1` header sections, with all examples nested either 2 or 3 levels deep
+ within `H2` or `H3` header sections.
+- `H3` header sections must be nested within `H2` header sections. They cannot be
+ nested directly within `H1` header sections.
`glfm_specification/input/gitlab_flavored_markdown/glfm_canonical_examples.txt` sample entries:
@@ -670,7 +718,7 @@ The actual file should not have these prefixed `|` characters.
|
|## Strong but with two asterisks
|
-|```````````````````````````````` example
+|```````````````````````````````` example gitlab
|**bold**
|.
|<p><strong>bold</strong></p>
@@ -680,7 +728,7 @@ The actual file should not have these prefixed `|` characters.
|
|## Strong but with HTML
|
-|```````````````````````````````` example
+|```````````````````````````````` example gitlab
|<strong>
|bold
|</strong>
@@ -738,7 +786,7 @@ The following optional entries are supported for each example. They all default
`glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml` sample entry:
```yaml
-07_99_an_example_with_incomplete_wysiwyg_implementation_1:
+07_99_00_an_example_with_incomplete_wysiwyg_implementation_1:
skip_update_example_snapshots: 'An explanation of the reason for skipping.'
skip_update_example_snapshot_html_static: 'An explanation of the reason for skipping.'
skip_update_example_snapshot_html_wysiwyg: 'An explanation of the reason for skipping.'
@@ -771,40 +819,73 @@ to be specified for a Markdown example.
00_uri: &00_uri
- regex: '(href|data-src)(=")(.*?)(test-file\.(png|zip)")'
replacement: '\1\2URI_PREFIX\4'
-01_01__section_one__example_containing_a_uri__001:
+01_01_00__section_one__example_containing_a_uri__001:
html:
static:
canonical:
- 01_01_uri: *00_uri
+ 01_01_00_uri: *00_uri
snapshot:
- 01_01_uri: *00_uri
+ 01_01_00_uri: *00_uri
wysiwyg:
- 01_01_uri: *00_uri
+ 01_01_00_uri: *00_uri
prosemirror_json:
- 01_01_uri: *00_uri
-07_01__gitlab_specific_markdown__footnotes__001:
+ 01_01_00_uri: *00_uri
+07_01_00__gitlab_specific_markdown__footnotes__001:
# YAML anchors which are only shared within a single example should be defined within the example
shared:
- 07_01_href: &07_01_href
+ 07_01_00_href: &07_01_00_href
- regex: '(href)(=")(.+?)(")'
replacement: '\1\2REF\4'
- 07_01_id: &07_01_id
+ 07_01_00_id: &07_01_00_id
- regex: '(id)(=")(.+?)(")'
replacement: '\1\2ID\4'
html:
static:
canonical:
- 07_01_href: *07_01_href
- 07_01_id: *07_01_id
+ 07_01_00_href: *07_01_00_href
+ 07_01_00_id: *07_01_00_id
snapshot:
- 07_01_href: *07_01_href
- 07_01_id: *07_01_id
+ 07_01_00_href: *07_01_00_href
+ 07_01_00_id: *07_01_00_id
wysiwyg:
- 07_01_href: *07_01_href
- 07_01_id: *07_01_id
+ 07_01_00_href: *07_01_00_href
+ 07_01_00_id: *07_01_00_id
prosemirror_json:
- 07_01_href: *07_01_href
- 07_01_id: *07_01_id
+ 07_01_00_href: *07_01_00_href
+ 07_01_00_id: *07_01_00_id
+```
+
+##### `glfm_example_metadata.yml`
+
+[`glfm_specification/input/gitlab_flavored_markdown/glfm_example_metadata.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_example_metadata.yml)
+allows control over other aspects of the snapshot example generation process.
+
+- It is manually updated.
+- The `ee` fields determine whether the example is an EE-only example. If the `ee` field is `true`,
+ the example will only be run by `ee/spec/requests/api/markdown_snapshot_spec.rb`, not by
+ `spec/requests/api/markdown_snapshot_spec.rb`.
+- The `api_request_override_path` field overrides the API endpoint path which is used to
+ generate the `static` HTML for the specifed example. Different endpoints can generate different
+ HTML in some cases, so we want to be able to exercise different API endpoints for the same
+ Markdown. By default, the `/markdown` endpoint is used.
+
+`glfm_specification/input/gitlab_flavored_markdown/glfm_example_metadata.yml` sample entries:
+
+```yaml
+---
+08_01_00__examples_using_internal_extensions__markdown_preview_api_request_overrides__001:
+ api_request_override_path: /groups/glfm_group/preview_markdown
+08_01_00__examples_using_internal_extensions__markdown_preview_api_request_overrides__002:
+ api_request_override_path: /glfm_group/glfm_project/preview_markdown
+08_01_00__examples_using_internal_extensions__markdown_preview_api_request_overrides__003:
+ api_request_override_path: /glfm_group/glfm_project/preview_markdown
+08_01_00__examples_using_internal_extensions__markdown_preview_api_request_overrides__004:
+ api_request_override_path: /-/snippets/preview_markdown
+08_01_00__examples_using_internal_extensions__markdown_preview_api_request_overrides__005:
+ api_request_override_path: /glfm_group/glfm_project/-/wikis/new_page/preview_markdown
+08_01_00__examples_using_internal_extensions__markdown_preview_api_request_overrides__006:
+ ee: true
+ api_request_override_path: /groups/glfm_group/-/wikis/new_page/preview_markdown
```
#### Output specification files
@@ -891,19 +972,19 @@ CommonMark, GFM, and GLFM example names, each with a unique canonical name.
`glfm_specification/example_snapshots/examples_index.yml` sample entries:
```yaml
-02_01_preliminaries_characters_and_lines_1:
+02_01_00_preliminaries_characters_and_lines_1:
spec_txt_example_position: 1
source_specification: commonmark
-03_01_blocks_and_inlines_precedence_1:
+03_01_00_blocks_and_inlines_precedence_1:
spec_txt_example_position: 12
source_specification: commonmark
-05_03_container_blocks_task_list_items_1:
+05_03_00_container_blocks_task_list_items_1:
spec_txt_example_position: 279
source_specification: github
-06_04_inlines_emphasis_and_strong_emphasis_1:
+06_04_00_inlines_emphasis_and_strong_emphasis_1:
spec_txt_example_position: 360
source_specification: github
-07_01_audio_link_1:
+07_01_00_audio_link_1:
spec_txt_example_position: 301
source_specification: gitlab
```
@@ -923,7 +1004,7 @@ for each entry in `glfm_specification/example_snapshots/examples_index.yml`
`glfm_specification/example_snapshots/markdown.yml` sample entry:
```yaml
-06_04_inlines_emphasis_and_strong_emphasis_1: |
+06_04_00_inlines_emphasis_and_strong_emphasis_1: |
*foo bar*
```
@@ -958,7 +1039,7 @@ Any exceptions or failures which occur when generating HTML are replaced with an
`glfm_specification/example_snapshots/html.yml` sample entry:
```yaml
-06_04_inlines_emphasis_and_strong_emphasis_1:
+06_04_00_inlines_emphasis_and_strong_emphasis_1:
canonical: |
<p><em>foo bar</em></p>
static: |
@@ -983,7 +1064,7 @@ contains the ProseMirror JSON for each entry in `glfm_specification/example_snap
`glfm_specification/example_snapshots/prosemirror_json.yml` sample entry:
```yaml
-06_04_inlines_emphasis_and_strong_emphasis_1: |-
+06_04_00_inlines_emphasis_and_strong_emphasis_1: |-
{
"type": "doc",
"content": [
diff --git a/doc/development/go_guide/dependencies.md b/doc/development/go_guide/dependencies.md
index 2a53fa590e3..7cad5bbf417 100644
--- a/doc/development/go_guide/dependencies.md
+++ b/doc/development/go_guide/dependencies.md
@@ -44,8 +44,8 @@ end with a timestamp and the first 12 characters of the commit identifier:
If a VCS tag matches one of these patterns, it is ignored.
-For a complete understanding of Go modules and versioning, see
-[this series of blog posts](https://go.dev/blog/using-go-modules)
+For a complete understanding of Go modules and versioning, see
+[this series of blog posts](https://go.dev/blog/using-go-modules)
on the official Go website.
## 'Module' vs 'Package'
diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md
index 711b0662a8c..3adafa8750f 100644
--- a/doc/development/go_guide/index.md
+++ b/doc/development/go_guide/index.md
@@ -145,7 +145,7 @@ Go GitLab linter plugins are maintained in the [`gitlab-org/language-tools/go/li
## Dependencies
Dependencies should be kept to the minimum. The introduction of a new
-dependency should be argued in the merge request, as per our [Approval Guidelines](../code_review.md#approval-guidelines).
+dependency should be argued in the merge request, as per our [Approval Guidelines](../code_review.md#approval-guidelines).
Both [License Scanning](../../user/compliance/license_compliance/index.md)
and [Dependency Scanning](../../user/application_security/dependency_scanning/index.md)
should be activated on all projects to ensure new dependencies
@@ -153,7 +153,7 @@ security status and license compatibility.
### Modules
-In Go 1.11 and later, a standard dependency system is available behind the name
+In Go 1.11 and later, a standard dependency system is available behind the name
[Go Modules](https://github.com/golang/go/wiki/Modules). It provides a way to
define and lock dependencies for reproducible builds. It should be used
whenever possible.
@@ -166,7 +166,7 @@ projects, and makes merge requests easier to review.
In some cases, such as building a Go project for it to act as a dependency of a
CI run for another project, removing the `vendor/` directory means the code must
be downloaded repeatedly, which can lead to intermittent problems due to rate
-limiting or network failures. In these circumstances, you should
+limiting or network failures. In these circumstances, you should
[cache the downloaded code between](../../ci/caching/index.md#cache-go-dependencies).
There was a
diff --git a/doc/development/import_export.md b/doc/development/import_export.md
index 6cbbb6bf716..c66ac0418ac 100644
--- a/doc/development/import_export.md
+++ b/doc/development/import_export.md
@@ -305,6 +305,29 @@ export_reorders:
nulls_position: :nulls_last
```
+### Conditional export
+
+When associated resources are from outside the project, you might need to
+validate that a user who is exporting the project or group can access these
+associations. `include_if_exportable` accepts an array of associations for a
+resource. During export, the `exportable_association?` method on the resource
+is called with the association's name and user to validate if associated
+resource can be included in the export.
+
+For example:
+
+```yaml
+include_if_exportable:
+ project:
+ issues:
+ - epic_issue
+```
+
+This definition:
+
+1. Calls the issue's `exportable_association?(:epic_issue, current_user: current_user)` method.
+1. If the method returns true, includes the issue's `epic_issue` association for the issue.
+
### Import
The import job status moves from `none` to `finished` or `failed` into different states:
diff --git a/doc/development/import_project.md b/doc/development/import_project.md
index 7c55d2e2668..1f3bf860257 100644
--- a/doc/development/import_project.md
+++ b/doc/development/import_project.md
@@ -149,26 +149,10 @@ You might see an error like `N is out of range for ActiveModel::Type::Integer wi
where `N` is the integer exceeding the 4-byte integer limit. If that's the case, you
are likely hitting the issue with rebalancing of `relative_position` field of the issues.
-The feature flag to enable the rebalance automatically was enabled on GitLab.com.
-We intend to enable it by default on self-managed instances when the issue
-[Rebalance issues FF rollout](https://gitlab.com/gitlab-org/gitlab/-/issues/343368)
-is implemented.
-
-If the feature is not enabled by default on your GitLab version, run the following
-commands in the [Rails console](../administration/operations/rails_console.md) as
-a workaround. Replace the ID with the ID of your project you were trying to import:
-
```ruby
-# Check if the feature is enabled on your instance. If it is, rebalance should work automatically on your instance
-Feature.enabled?(:rebalance_issues,Project.find(ID).root_namespace)
-
# Check the current maximum value of relative_position
Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position)
-# Enable `rebalance_issues` feauture and check that it was successfully enabled
-Feature.enable(:rebalance_issues,Project.find(ID).root_namespace)
-Feature.enabled?(:rebalance_issues,Project.find(ID).root_namespace)
-
# Run the rebalancing process and check if the maximum value of relative_position has changed
Issues::RelativePositionRebalancingService.new(Project.find(ID).root_namespace.all_projects).execute
Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position)
diff --git a/doc/development/integrations/jenkins.md b/doc/development/integrations/jenkins.md
index f430fc380b1..bb541d26df2 100644
--- a/doc/development/integrations/jenkins.md
+++ b/doc/development/integrations/jenkins.md
@@ -24,7 +24,7 @@ brew services start jenkins
GitLab does not allow requests to localhost or the local network by default. When running Jenkins on your local machine, you need to enable local access.
1. Log into your GitLab instance as an administrator.
-1. On the top bar, select **Menu > Admin**.
+1. On the top bar, select **Main menu > Admin**.
1. On the left sidebar, select **Settings > Network**.
1. Expand **Outbound requests** and check the following checkboxes:
diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md
index 55e57a3c2ee..2c5dd1c0500 100644
--- a/doc/development/integrations/secure.md
+++ b/doc/development/integrations/secure.md
@@ -1,6 +1,6 @@
---
-stage: Protect
-group: Container Security
+stage: Secure
+group: Static Analysis
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
@@ -254,7 +254,7 @@ Following the POSIX exit code standard, the scanner exits with 0 for success and
Success also includes the case when vulnerabilities are found.
When a CI job fails, security report results are not ingested by GitLab, even if the job
-[allows failure](../../ci/yaml/#allow_failure). The report artifacts are still uploaded to GitLab and available
+[allows failure](../../ci/yaml/index.md#allow_failure). The report artifacts are still uploaded to GitLab and available
for [download in the pipeline security tab](../../user/application_security/vulnerability_report/pipeline.md#download-security-scan-outputs).
When executing a scanning job using the [Docker-in-Docker privileged mode](../../user/application_security/sast/index.md#requirements),
@@ -488,8 +488,8 @@ the risk. End-users interact with this field, whereas GitLab automatically proce
##### Identifiers
The `identifiers` array describes the detected vulnerability. An identifier object's `type` and
-`value` fields are used to tell if two identifiers are the same. The user interface uses the
-object's `name` and `url` fields to display the identifier.
+`value` fields are used to [tell if two identifiers are the same](../../user/application_security/vulnerability_report/pipeline.md#deduplication-process).
+The user interface uses the object's `name` and `url` fields to display the identifier.
We recommend that you use the identifiers the GitLab scanners already define:
@@ -509,12 +509,10 @@ which is shared by some of the analyzers that GitLab maintains. You can [contrib
new generic identifiers to if needed. Analyzers may also produce vendor-specific or product-specific
identifiers, which don't belong in the [common library](https://gitlab.com/gitlab-org/security-products/analyzers/common).
-The first item of the `identifiers` array is called the
-[primary identifier](../../user/application_security/terminology/index.md#primary-identifier).
-The primary identifier is particularly important, because it is used to
+The first item of the `identifiers` array is called the
+[primary identifier](../../user/application_security/terminology/index.md#primary-identifier), and
+it is used to
[track vulnerabilities](#tracking-and-merging-vulnerabilities) as new commits are pushed to the repository.
-Identifiers are also used to [merge duplicate vulnerabilities](#tracking-and-merging-vulnerabilities)
-reported for the same commit, except for `CWE` and `WASC`.
Not all vulnerabilities have CVEs, and a CVE can be identified multiple times. As a result, a CVE
isn't a stable identifier and you shouldn't assume it as such when tracking vulnerabilities.
@@ -666,11 +664,14 @@ Users may give feedback on a vulnerability:
GitLab tracks vulnerabilities so that user feedback is not lost
when new Git commits are pushed to the repository.
-Vulnerabilities are tracked using a combination of three attributes:
+Vulnerabilities are tracked using a
+[`UUIDv5`](https://gitlab.com/gitlab-org/gitlab/-/blob/1272957c4a55e616569721febccb685c056ca1e4/ee/app/models/vulnerabilities/finding.rb#L364-368)
+digest, which is generated by a `SHA-1` hash of four attributes:
- [Report type](#category)
-- [Location fingerprint](#location)
- [Primary identifier](#identifiers)
+- [Location fingerprint](#location)
+- Project ID
Right now, GitLab cannot track a vulnerability if its location changes
as new Git commits are pushed, and this results in user feedback being lost.
@@ -678,12 +679,7 @@ For instance, user feedback on a SAST vulnerability is lost
if the affected file is renamed or the affected line moves down.
This is addressed in [issue #7586](https://gitlab.com/gitlab-org/gitlab/-/issues/7586).
-In some cases, the multiple scans executed in the same CI pipeline result in duplicates
-that are automatically merged using the vulnerability location and identifiers.
-Two vulnerabilities are considered to be the same if they share the same [location fingerprint](#location)
-and at least one [identifier](#identifiers). Two identifiers are the same if they share the same `type` and `id`.
-CWE and WASC identifiers are not considered because they describe categories of vulnerability flaws,
-but not specific security flaws.
+See also [deduplication process](../../user/application_security/vulnerability_report/pipeline.md#deduplication-process).
##### Severity and confidence
diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md
index 9b29af3e433..c35d40a7b7f 100644
--- a/doc/development/internal_api/index.md
+++ b/doc/development/internal_api/index.md
@@ -148,7 +148,7 @@ curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \
## Authorized Keys Check
This endpoint is called by the GitLab Shell authorized keys
-check. Which is called by OpenSSH for
+check. Which is called by OpenSSH for
[fast SSH key lookup](../../administration/operations/fast_ssh_key_lookup.md).
| Attribute | Type | Required | Description |
diff --git a/doc/development/internal_users.md b/doc/development/internal_users.md
index 95ca593e31e..24785c0cf48 100644
--- a/doc/development/internal_users.md
+++ b/doc/development/internal_users.md
@@ -8,6 +8,8 @@ info: "See the Technical Writers assigned to Development Guidelines: https://abo
# Internal users
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97584) in GitLab 15.4, bots are indicated with a badge in user listings.
+
GitLab uses internal users (sometimes referred to as "bots") to perform
actions or functions that cannot be attributed to a regular user.
diff --git a/doc/development/lfs.md b/doc/development/lfs.md
index 5900eb68294..20157e9e805 100644
--- a/doc/development/lfs.md
+++ b/doc/development/lfs.md
@@ -76,13 +76,13 @@ process, which writes the contents to the standard output.
1. The archive data is sent back to the client.
In step 7, the `gitaly-lfs-smudge` filter must talk to Workhorse, not to
-Rails, or an invalid LFS blob is saved. To support this, GitLab 13.5
+Rails, or an invalid LFS blob is saved. To support this, GitLab 13.5
[changed the default Omnibus configuration to have Gitaly talk to the Workhorse](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4592)
instead of Rails.
One side effect of this change: the correlation ID of the original
request is not preserved for the internal API requests made by Gitaly
(or `gitaly-lfs-smudge`), such as the one made in step 8. The
-correlation IDs for those API requests are random values until
+correlation IDs for those API requests are random values until
[this Workhorse issue](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/309) is
resolved.
diff --git a/doc/development/logging.md b/doc/development/logging.md
index f1fa7f4c8c9..467fb68f3ae 100644
--- a/doc/development/logging.md
+++ b/doc/development/logging.md
@@ -385,7 +385,7 @@ end
## Additional steps with new log files
1. Consider log retention settings. By default, Omnibus rotates any
- logs in `/var/log/gitlab/gitlab-rails/*.log` every hour and
+ logs in `/var/log/gitlab/gitlab-rails/*.log` every hour and
[keep at most 30 compressed files](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate).
On GitLab.com, that setting is only 6 compressed files. These settings should suffice
for most users, but you may need to tweak them in [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab).
@@ -395,7 +395,7 @@ end
a merge request to the [`gitlab_fluentd`](https://gitlab.com/gitlab-cookbooks/gitlab_fluentd)
project. See [this example](https://gitlab.com/gitlab-cookbooks/gitlab_fluentd/-/merge_requests/51/diffs).
-1. Be sure to update the [GitLab CE/EE documentation](../administration/logs/index.md) and the
+1. Be sure to update the [GitLab CE/EE documentation](../administration/logs/index.md) and the
[GitLab.com runbooks](https://gitlab.com/gitlab-com/runbooks/blob/master/docs/logging/README.md).
## Control logging visibility
diff --git a/doc/development/merge_request_concepts/index.md b/doc/development/merge_request_concepts/index.md
index 331f0e01579..d463f6ba290 100644
--- a/doc/development/merge_request_concepts/index.md
+++ b/doc/development/merge_request_concepts/index.md
@@ -10,7 +10,7 @@ info: "See the Technical Writers assigned to Development Guidelines: https://abo
NOTE:
The documentation below is the single source of truth for the merge request terminology and functionality.
-The merge request is made up of several different key components and ideas that encompass the overall merge request experience. These concepts sometimes have competing and confusing terminology or overlap with other concepts. The concepts this will cover are:
+The merge request is made up of several different key components and ideas that encompass the overall merge request experience. These concepts sometimes have competing and confusing terminology or overlap with other concepts. This page covers the following concepts:
1. Merge widget
1. Report widgets
@@ -40,15 +40,17 @@ Reports are widgets within the merge request that report information about chang
## Merge checks
-Merge checks are statuses that can either pass or fail and conditionally control the availability of the merge button being available within a merge request. The key distinguishing factor in a merge check is that users **do not** interact with the merge checks inside of the merge request, but are able to influence whether or not the check passes or fails. Results from the check are processed as true/false to determine whether or not a merge request can be merged. Examples include:
+Merge checks are statuses that can either pass or fail and conditionally control the availability of the merge button being available within a merge request. The key distinguishing factor in a merge check is that users **do not** interact with the merge checks inside of the merge request, but are able to influence whether or not the check passes or fails. Results from the check are processed as true/false to determine whether or not a merge request can be merged.
-- Merge conflicts.
-- Pipeline success.
-- Threads resolution.
-- [External status checks](../../user/project/merge_requests/status_checks.md).
-- Required approvals.
+Examples of merge checks include:
-When all of the required merge checks are satisfied a merge request becomes mergeable.
+- Merge conflicts
+- Pipeline success
+- Threads resolution
+- [External status checks](../../user/project/merge_requests/status_checks.md)
+- Required approvals
+
+A merge request can be merged only when all of the required merge checks are satisfied.
## Approvals
@@ -58,8 +60,8 @@ Additionally, approval settings provide configuration options to define how thos
Examples of approval rules and settings include:
-1. [merge request approval rules](../../user/project/merge_requests/approvals/rules.md)
-1. [code owner approvals](../../user/project/code_owners.md)
-1. [security approvals](../../user/application_security/index.md#security-approvals-in-merge-requests)
-1. [prevent editing approval rules](../../user/project/merge_requests/approvals/settings.md#prevent-editing-approval-rules-in-merge-requests)
-1. [remove all approvals when commits are added](../../user/project/merge_requests/approvals/settings.md#remove-all-approvals-when-commits-are-added-to-the-source-branch)
+- [Merge request approval rules](../../user/project/merge_requests/approvals/rules.md)
+- [Code owner approvals](../../user/project/code_owners.md)
+- [Security approvals](../../user/application_security/index.md#security-approvals-in-merge-requests)
+- [Prevent editing approval rules](../../user/project/merge_requests/approvals/settings.md#prevent-editing-approval-rules-in-merge-requests)
+- [Remove all approvals when commits are added](../../user/project/merge_requests/approvals/settings.md#remove-all-approvals-when-commits-are-added-to-the-source-branch)
diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md
index 7ff25705ae6..895fc6f92a4 100644
--- a/doc/development/merge_request_performance_guidelines.md
+++ b/doc/development/merge_request_performance_guidelines.md
@@ -394,7 +394,7 @@ query for every mention of `@alice`.
Caching data per transaction can be done using
[RequestStore](https://github.com/steveklabnik/request_store) (use
`Gitlab::SafeRequestStore` to avoid having to remember to check
-`RequestStore.active?`). Caching data in Redis can be done using
+`RequestStore.active?`). Caching data in Redis can be done using
[Rails' caching system](https://guides.rubyonrails.org/caching_with_rails.html).
## Pagination
diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md
index 64d8b22f1b8..4e569579f37 100644
--- a/doc/development/migration_style_guide.md
+++ b/doc/development/migration_style_guide.md
@@ -1228,7 +1228,7 @@ If using a model in the migrations, you should first
[clear the column cache](https://api.rubyonrails.org/classes/ActiveRecord/ModelSchema/ClassMethods.html#method-i-reset_column_information)
using `reset_column_information`.
-If using a model that leverages single table inheritance (STI), there are
+If using a model that leverages single table inheritance (STI), there are
[special considerations](database/single_table_inheritance.md#in-migrations).
This avoids problems where a column that you are using was altered and cached
diff --git a/doc/development/newlines_styleguide.md b/doc/development/newlines_styleguide.md
index 57962129b2f..014affa3e04 100644
--- a/doc/development/newlines_styleguide.md
+++ b/doc/development/newlines_styleguide.md
@@ -1,108 +1,11 @@
---
-stage: none
-group: unassigned
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+redirect_to: 'backend/ruby_style_guide.md#newlines-style-guide'
+remove_date: '2022-12-15'
---
-# Newlines style guide
+This document was moved to [another location](backend/ruby_style_guide.md#newlines-style-guide).
-This style guide recommends best practices for newlines in Ruby code.
-
-## Rule: separate code with newlines only to group together related logic
-
-```ruby
-# bad
-def method
- issue = Issue.new
-
- issue.save
-
- render json: issue
-end
-```
-
-```ruby
-# good
-def method
- issue = Issue.new
- issue.save
-
- render json: issue
-end
-```
-
-## Rule: separate code and block with newlines
-
-### Newline before block
-
-```ruby
-# bad
-def method
- issue = Issue.new
- if issue.save
- render json: issue
- end
-end
-```
-
-```ruby
-# good
-def method
- issue = Issue.new
-
- if issue.save
- render json: issue
- end
-end
-```
-
-## Newline after block
-
-```ruby
-# bad
-def method
- if issue.save
- issue.send_email
- end
- render json: issue
-end
-```
-
-```ruby
-# good
-def method
- if issue.save
- issue.send_email
- end
-
- render json: issue
-end
-```
-
-### Exception: no need for newline when code block starts or ends right inside another code block
-
-```ruby
-# bad
-def method
-
- if issue
-
- if issue.valid?
- issue.save
- end
-
- end
-
-end
-```
-
-```ruby
-# good
-def method
- if issue
- if issue.valid?
- issue.save
- end
- end
-end
-```
+<!-- This redirect file can be deleted after 2022-12-15. -->
+<!-- Redirects that point to other docs in the same project expire in three months. -->
+<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
+<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/development/packages/new_format_development.md b/doc/development/packages/new_format_development.md
index f7d02f9160b..73a2b2f1f81 100644
--- a/doc/development/packages/new_format_development.md
+++ b/doc/development/packages/new_format_development.md
@@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
This document guides you through adding support to GitLab for a new a [package management system](../../administration/packages/index.md).
-See the already supported formats in the [Packages & Registries documentation](../../user/packages/index.md)
+See the already supported formats in the [Packages and registries documentation](../../user/packages/index.md)
It is possible to add a new format with only backend changes.
This guide is superficial and does not cover the way the code should be written.
@@ -91,7 +91,7 @@ extended when possible to keep the common package logic grouped as much as possi
### Configuration
-GitLab has a `packages` section in its configuration file (`gitlab.rb`).
+GitLab has a `packages` section in its configuration file (`gitlab.rb` or `gitlab.yml`).
It applies to all package systems supported by GitLab. Usually you don't need
to add anything there.
diff --git a/doc/development/permissions.md b/doc/development/permissions.md
index 8e517b8577c..d348d659cad 100644
--- a/doc/development/permissions.md
+++ b/doc/development/permissions.md
@@ -147,15 +147,41 @@ into different features like Merge Requests and CI flow.
## Where should permissions be checked?
-By default, controllers, API endpoints, and GraphQL types/fields are responsible for authorization. See [Secure Coding Guidelines > Permissions](secure_coding_guidelines.md#permissions).
+We should typically apply defense-in-depth (implementing multiple checks at
+various layers) starting with low-level layers, such as finders and services,
+followed by high-level layers, such as GraphQL, public REST API, and controllers.
+
+See [Guidelines for reusing abstractions](reusing_abstractions.md).
+
+Protecting the same resources at many points means that if one layer of defense is compromised
+or missing, customer data is still protected by the additional layers.
+
+See the permissions section in the [Secure Coding Guidelines](secure_coding_guidelines.md#permissions).
### Considerations
-- Many actions are completely or partially extracted to services, finders, and other classes, so it is normal to do permission checks "downstream".
-- Often, authorization logic must be incorporated in DB queries to filter records.
+Services or finders are appropriate locations because:
+
+- Multiple endpoints share services or finders so downstream logic is more likely to be re-used.
+- Sometimes authorization logic must be incorporated in DB queries to filter records.
+- Permission checks at the display layer should be avoided except to provide better UX
+ and not as a security check. For example, showing and hiding non-data elements like buttons.
+
+The downsides to defense-in-depth are:
+
- `DeclarativePolicy` rules are relatively performant, but conditions may perform database calls.
-- Multiple permission checks across layers can be difficult to reason about, which is its own security risk. For example, duplicate authorization logic could diverge.
-- Should we apply defense-in-depth with permission checks? [Join the discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/324135)
+- Higher maintenance costs.
+
+### Exceptions
+
+Developers can choose to do authorization in only a single area after weighing
+the risks and drawbacks for their specific case.
+
+Prefer domain logic (services or finders) as the source of truth when making exceptions.
+
+Logic, like backend worker logic, might not need authorization based on the current user.
+If the service or finder's constructor does not expect `current_user`, then it typically won't
+check permissions.
### Tips
diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md
index d57e5bbeb26..648a2aac339 100644
--- a/doc/development/pipelines.md
+++ b/doc/development/pipelines.md
@@ -103,7 +103,7 @@ When you need to revert a merge request, to get accelerated feedback, you can ad
When this label is assigned, the following steps of the CI/CD pipeline are skipped:
-- The `package-and-qa` job.
+- The `e2e:package-and-test` job.
- The `rspec:undercoverage` job.
- The entire [Review Apps process](testing_guide/review_apps.md).
@@ -221,8 +221,8 @@ that includes `rspec-profile` in their name.
### Logging
-- Rails logging to `log/test.log` is disabled by default in CI
- [for performance reasons](https://jtway.co/speed-up-your-rails-test-suite-by-6-in-1-line-13fedb869ec4).
+- Rails logging to `log/test.log` is disabled by default in CI
+ [for performance reasons](https://jtway.co/speed-up-your-rails-test-suite-by-6-in-1-line-13fedb869ec4).
To override this setting, provide the
`RAILS_ENABLE_TEST_LOG` environment variable.
@@ -341,13 +341,20 @@ We also run our test suite against PG11 upon specific database library changes i
### Current versions testing
-| Where? | PostgreSQL version | Ruby version |
-| ------ | ------------------ | ------------ |
-| Merge requests | 12 (default version), 11 for DB library changes | 2.7 (default version) |
-| `master` branch commits | 12 (default version), 11 for DB library changes | 2.7 (default version) |
-| `maintenance` scheduled pipelines (every 2 hours at even hour) | 12 (default version), 11 for DB library changes | 2.7 (default version) |
-| `maintenance` scheduled pipelines (every 2 hours at odd hour) | 12 (default version), 11 for DB library changes | 3.0 (set in the schedule variables) |
-| `nightly` scheduled pipelines | 12 (default version), 11, 13 | 2.7 (default version) |
+| Where? | PostgreSQL version | Ruby version |
+|------------------------------------------------------------------------------------------------|-------------------------------------------------|--------------|
+| Merge requests | 12 (default version), 11 for DB library changes | 2.7 (default version) |
+| `master` branch commits | 12 (default version), 11 for DB library changes | 2.7 (default version) |
+| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour) | 12 (default version), 11 for DB library changes | 2.7 (default version) |
+| `maintenance` scheduled pipelines for the `ruby3` branch (every odd-numbered hour), see below. | 12 (default version), 11 for DB library changes | 3.0 (coded in the branch) |
+| `nightly` scheduled pipelines for the `master` branch | 12 (default version), 11, 13 | 2.7 (default version) |
+
+The pipeline configuration for the scheduled pipeline testing Ruby 3 is
+stored in the `ruby3-sync` branch. The pipeline updates the `ruby3` branch
+with latest `master`, and then it triggers a regular branch pipeline for
+`ruby3`. Any changes in `ruby3` are only for running the pipeline. It should
+never be merged back to `master`. Any other Ruby 3 changes should go into
+`master` directly, which should be compatible with Ruby 2.7.
### Long-term plan
@@ -489,7 +496,7 @@ graph RL;
class 2_3-1 criticalPath;
2_3-1 --> 1-5
- 2_4-1["package-and-qa (102 minutes)"];
+ 2_4-1["e2e:package-and-test (102 minutes)"];
class 2_4-1 criticalPath;
click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0"
2_4-1 --> 1-2 & 2_3-1 & 1-15;
diff --git a/doc/development/policies.md b/doc/development/policies.md
index f0c9d0ec5f9..ddd6b8e9bce 100644
--- a/doc/development/policies.md
+++ b/doc/development/policies.md
@@ -10,6 +10,8 @@ The DeclarativePolicy framework is designed to assist in performance of policy c
The policy used is based on the subject's class name - so `Ability.allowed?(user, :some_ability, project)` creates a `ProjectPolicy` and check permissions on that.
+The Ruby gem source is available in the [declarative-policy](https://gitlab.com/gitlab-org/ruby/gems/declarative-policy) GitLab project.
+
## Managing Permission Rules
Permissions are broken into two parts: `conditions` and `rules`. Conditions are boolean expressions that can access the database and the environment, while rules are statically configured combinations of expressions and other rules that enable or prevent certain abilities. For an ability to be allowed, it must be enabled by at least one rule, and not prevented by any.
diff --git a/doc/development/rails_update.md b/doc/development/rails_update.md
index 9907a78421f..bda21860eae 100644
--- a/doc/development/rails_update.md
+++ b/doc/development/rails_update.md
@@ -27,7 +27,7 @@ We strive to run GitLab using the latest Rails releases to benefit from performa
1. Run `yarn patch-package @rails/ujs` after updating this to ensure our local patch file version matches.
1. Create an MR with the `pipeline:run-all-rspec` label and see if pipeline breaks.
1. To resolve and debug spec failures use `git bisect` against the rails repository. See the [debugging section](#git-bisect-against-rails) below.
-1. Include links to the Gem diffs between the two versions in the merge request description. For example, this is the gem diff for
+1. Include links to the Gem diffs between the two versions in the merge request description. For example, this is the gem diff for
[`activesupport` 6.1.3.2 to 6.1.4.1](https://my.diffend.io/gems/activerecord/6.1.3.2/6.1.4.1).
### Prepare an MR for Gitaly
diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md
index c13f1195df3..f300904fc19 100644
--- a/doc/development/rake_tasks.md
+++ b/doc/development/rake_tasks.md
@@ -188,6 +188,8 @@ Alternatively you can use the following on each spec run,
bundle exec spring rspec some_spec.rb
```
+## RuboCop tasks
+
## Generate initial RuboCop TODO list
One way to generate the initial list is to run the Rake task `rubocop:todo:generate`:
@@ -209,6 +211,18 @@ Some shells require brackets to be escaped or quoted.
See [Resolving RuboCop exceptions](contributing/style_guides.md#resolving-rubocop-exceptions)
on how to proceed from here.
+### Run RuboCop in graceful mode
+
+You can run RuboCop in "graceful mode". This means all enabled cop rules are
+silenced which have "grace period" activated (via `Details: grace period`).
+
+Run:
+
+```shell
+bundle exec rake 'rubocop:check:graceful'
+bundle exec rake 'rubocop:check:graceful[Gitlab/NamespacedClass]'
+```
+
## Compile Frontend Assets
You shouldn't ever need to compile frontend assets manually in development, but
diff --git a/doc/development/real_time.md b/doc/development/real_time.md
index 21f3ee1f3b2..f113d4dd3b7 100644
--- a/doc/development/real_time.md
+++ b/doc/development/real_time.md
@@ -60,7 +60,7 @@ downstream services.
To mitigate this, ensure that the code establishing the new WebSocket connection
is feature flagged and defaulted to `off`. A careful, percentage-based roll-out
-of the feature flag ensures that effects can be observed on the
+of the feature flag ensures that effects can be observed on the
[WebSocket dashboard](https://dashboards.gitlab.net/d/websockets-main/websockets-overview?orgId=1)
1. Create a
diff --git a/doc/development/redis/new_redis_instance.md b/doc/development/redis/new_redis_instance.md
index efaf1e5a6d0..24885b40eb9 100644
--- a/doc/development/redis/new_redis_instance.md
+++ b/doc/development/redis/new_redis_instance.md
@@ -265,7 +265,7 @@ instances to cope without this functional partition.
If we decide to keep the migration code:
- We should document the migration steps.
-- If we used a feature flag, we should ensure it's an
+- If we used a feature flag, we should ensure it's an
[ops type feature flag](../feature_flags/index.md#ops-type), as these are long-lived flags.
Otherwise, we can remove the flags and conclude the project.
diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md
index ef4e8b0310f..826782d7036 100644
--- a/doc/development/reusing_abstractions.md
+++ b/doc/development/reusing_abstractions.md
@@ -206,6 +206,31 @@ response = ServiceResponse.success(payload: { issue: issue })
response.payload[:issue] # => issue
```
+Error responses can also specify the failure `reason` which can be used by the caller
+to understand the nature of the failure.
+The caller, if an HTTP endpoint, could translate the reason symbol into an HTTP status code:
+
+```ruby
+response = ServiceResponse.error(
+ message: 'Job is in a state that cannot be retried',
+ reason: :job_not_retrieable)
+
+if response.success?
+ head :ok
+if response.reason == :job_not_retriable
+ head :unprocessable_entity
+else
+ head :bad_request
+end
+```
+
+For common failures such as resource `:not_found` or operation `:forbidden`, we could
+leverage the Rails [HTTP status symbols](http://www.railsstatuscodes.com/) as long as
+they are sufficiently specific for the domain logic involved.
+For other failures use domain-specific reasons whenever possible.
+
+For example: `:job_not_retriable`, `:duplicate_package`, `:merge_request_not_mergeable`.
+
### Finders
Everything in `app/finders`, typically used for retrieving data from a database.
diff --git a/doc/development/routing.md b/doc/development/routing.md
index 3d5857b4237..54a531730f9 100644
--- a/doc/development/routing.md
+++ b/doc/development/routing.md
@@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Routing
-The GitLab backend is written primarily with Rails so it uses
+The GitLab backend is written primarily with Rails so it uses
[Rails routing](https://guides.rubyonrails.org/routing.html). Beside Rails best
practices, there are few rules unique to the GitLab application. To
support subgroups, GitLab project and group routes use the wildcard
diff --git a/doc/development/ruby3_gotchas.md b/doc/development/ruby3_gotchas.md
index dbe6fa13eee..db328b0b1a5 100644
--- a/doc/development/ruby3_gotchas.md
+++ b/doc/development/ruby3_gotchas.md
@@ -163,3 +163,40 @@ For Ruby 3 compliance, this should be changed to one of the following invocation
- `f(**{k: v})`
- `f(k: v)`
+
+## RSpec `with` argument matcher fails for shorthand Hash syntax
+
+Because keyword arguments ("kwargs") are a first-class concept in Ruby 3, keyword arguments are not
+converted into internal `Hash` instances anymore. This leads to RSpec method argument matchers failing
+when the receiver takes a positional options hash instead of kwargs:
+
+```ruby
+def m(options={}); end
+```
+
+```ruby
+expect(subject).to receive(:m).with(a: 42)
+```
+
+In Ruby 3 this expectations fails with the following error:
+
+```plaintext
+ Failure/Error:
+
+ #<subject> received :m with unexpected arguments
+ expected: ({:a=>42})
+ got: ({:a=>42})
+```
+
+This happens because RSpec uses a kwargs argument matcher here, but the method takes a hash.
+It works in Ruby 2, because `a: 42` is converted to a hash first and RSpec will use a hash argument matcher.
+
+A workaround is to not use the shorthand syntax and pass an actual `Hash` instead whenever we know a method
+to take an options hash:
+
+```ruby
+# Note the braces around the key-value pair.
+expect(subject).to receive(:m).with({ a: 42 })
+```
+
+For more information, see [the official issue report for RSpec](https://github.com/rspec/rspec-mocks/issues/1460).
diff --git a/doc/development/scalability.md b/doc/development/scalability.md
index b7ee0ca1167..66f436bd391 100644
--- a/doc/development/scalability.md
+++ b/doc/development/scalability.md
@@ -35,7 +35,7 @@ The application has a tight coupling to the database schema. When the
application starts, Rails queries the database schema, caching the tables and
column types for the data requested. Because of this schema cache, dropping a
column or table while the application is running can produce 500 errors to the
-user. This is why we have a
+user. This is why we have a
[process for dropping columns and other no-downtime changes](database/avoiding_downtime_in_migrations.md).
#### Multi-tenancy
@@ -61,10 +61,10 @@ There are two ways to deal with this:
- Sharding. Distribute data across multiple databases.
Partitioning is a built-in PostgreSQL feature and requires minimal changes
-in the application. However, it
+in the application. However, it
[requires PostgreSQL 11](https://www.2ndquadrant.com/en/blog/partitioning-evolution-postgresql-11/).
-For example, a natural way to partition is to
+For example, a natural way to partition is to
[partition tables by dates](https://gitlab.com/groups/gitlab-org/-/epics/2023). For example,
the `events` and `audit_events` table are natural candidates for this
kind of partitioning.
@@ -77,9 +77,9 @@ to abstract data access into API calls that abstract the database from
the application, but this is a significant amount of work.
There are solutions that may help abstract the sharding to some extent
-from the application. For example, we want to look at
+from the application. For example, we want to look at
[Citus Data](https://www.citusdata.com/product/community) closely. Citus Data
-provides a Rails plugin that adds a
+provides a Rails plugin that adds a
[tenant ID to ActiveRecord models](https://www.citusdata.com/blog/2017/01/05/easily-scale-out-multi-tenant-apps/).
Sharding can also be done based on feature verticals. This is the
@@ -97,11 +97,11 @@ systems.
#### Database size
-A recent
+A recent
[database checkup shows a breakdown of the table sizes on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/8022#master-1022016101-8).
Since `merge_request_diff_files` contains over 1 TB of data, we want to
-reduce/eliminate this table first. GitLab has support for
-[storing diffs in object storage](../administration/merge_request_diffs.md), which we
+reduce/eliminate this table first. GitLab has support for
+[storing diffs in object storage](../administration/merge_request_diffs.md), which we
[want to do on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/7356).
#### High availability
@@ -149,7 +149,7 @@ limitation:
- Use a multi-threaded connection pooler (for example,
[Odyssey](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/7776).
-On some Linux systems, it's possible to run
+On some Linux systems, it's possible to run
[multiple PgBouncer instances on the same port](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4796).
On GitLab.com, we run multiple PgBouncer instances on different ports to
diff --git a/doc/development/sec/index.md b/doc/development/sec/index.md
new file mode 100644
index 00000000000..06c20cee0bb
--- /dev/null
+++ b/doc/development/sec/index.md
@@ -0,0 +1,69 @@
+---
+stage: Secure
+group: Static Analysis
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+type: index, concepts, howto
+---
+
+# Sec section development **(FREE)**
+
+The Sec section is responsible for GitLab application security features, the "Sec" part of
+DevSecOps. Development guides that are specific to the Sec section are listed here.
+
+See [Terminology](../../user/application_security/terminology) for an overview of our shared terminology.
+
+## Architecture
+
+- [Overview](#overview)
+- [Scanning](#scanning)
+- [Processing, visualization, and management](#processing-visualization-and-management)
+- [Severity Levels](../../user/application_security/vulnerabilities/severities.md)
+
+## Overview
+
+The architecture supporting the Secure features is split into two main parts:
+
+- Scanning
+- Processing, visualization, and management
+
+```mermaid
+flowchart LR
+ subgraph G1[Scanning]
+ Scanner
+ Analyzer
+ CI[CI Jobs]
+ end
+ subgraph G2[Processing, visualization, and management]
+ Parsers
+ Database
+ Views
+ Interactions
+ end
+ G1 --Report Artifact--> G2
+```
+
+### Scanning
+
+The scanning part is responsible for finding vulnerabilities in given resources, and exporting results.
+The scans are executed in CI/CD jobs via several small projects called [Analyzers](../../user/application_security/terminology/index.md#analyzer), which can be found in our [Analyzers sub-group](https://gitlab.com/gitlab-org/security-products/analyzers).
+The Analyzers are wrappers around security tools called [Scanners](../../user/application_security/terminology/index.md#scanner), developed internally or externally, to integrate them into GitLab.
+The Analyzers are mainly written in Go.
+
+Some 3rd party integrators also make additional Scanners available by following our [integration documentation](../integrations/secure.md), which leverages the same architecture.
+
+The results of the scans are exported as JSON reports that must comply with the [Secure report format](../../user/application_security/terminology/index.md#secure-report-format) and are uploaded as [CI/CD Job Report artifacts](../../ci/pipelines/job_artifacts.md) to make them available for processing after the pipelines completes.
+
+### Processing, visualization, and management
+
+After the data is available as a Report Artifact it can be processed by the GitLab Rails application to enable our security features, including:
+
+- [Security Dashboards](../../user/application_security/security_dashboard/index.md), Merge Request widget, Pipeline view, and so on.
+- [Interactions with vulnerabilities](../../user/application_security/index.md#interact-with-findings-and-vulnerabilities).
+- [Approval rules](../../user/application_security/index.md#security-approvals-in-merge-requests).
+
+Depending on the context, the security reports may be stored either in the database or stay as Report Artifacts for on-demand access.
+
+## CI/CD template development
+
+While CI/CD templates are the responsibiility of the Verify section, many are critical to the Sec Section's feature usage.
+If you are working with CI/CD templates, please read the [development guide for GitLab CI/CD templates](../cicd/templates.md).
diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md
index 8053b4285e6..4c2f3118366 100644
--- a/doc/development/secure_coding_guidelines.md
+++ b/doc/development/secure_coding_guidelines.md
@@ -81,7 +81,7 @@ text = "foo\nbar"
p text.match /^bar$/
```
-The output of this example is `#<MatchData "bar">`, as Ruby treats the input `text` line by line. In order to match the whole __string__ the Regex anchors `\A` and `\z` should be used.
+The output of this example is `#<MatchData "bar">`, as Ruby treats the input `text` line by line. To match the whole **string**, the Regex anchors `\A` and `\z` should be used.
#### Impact
diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md
index 0ebc58dd669..5448bbb4293 100644
--- a/doc/development/service_ping/implement.md
+++ b/doc/development/service_ping/implement.md
@@ -127,7 +127,7 @@ Examples using `usage_data.rb` have been [deprecated](usage_data.md). We recomme
#### Grouping and batch operations
-The `count`, `distinct_count`, `sum`, and `average` batch counters can accept an `ActiveRecord::Relation`
+The `count`, `distinct_count` and `sum` batch counters can accept an `ActiveRecord::Relation`
object, which groups by a specified column. With a grouped relation, the methods do batch counting,
handle errors, and returns a hash table of key-value pairs.
@@ -142,9 +142,6 @@ distinct_count(Project.group(:visibility_level), :creator_id)
sum(Issue.group(:state_id), :weight))
# returns => {1=>3542, 2=>6820}
-
-average(Issue.group(:state_id), :weight))
-# returns => {1=>3.5, 2=>2.5}
```
#### Add operation
@@ -275,7 +272,7 @@ Events are handled by counter classes in the `Gitlab::UsageDataCounters` namespa
1. Listed in [`Gitlab::UsageDataCounters::COUNTERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters.rb#L5) to be then included in `Gitlab::UsageData`.
-1. Specified in the metric definition using the `RedisMetric` instrumentation class as a `counter_class` option to be picked up using the [metric instrumentation](metrics_instrumentation.md) framework. Refer to the [Redis metrics](metrics_instrumentation.md#redis-metrics) documentation for an example implementation.
+1. Specified in the metric definition using the `RedisMetric` instrumentation class by their `prefix` option to be picked up using the [metric instrumentation](metrics_instrumentation.md) framework. Refer to the [Redis metrics](metrics_instrumentation.md#redis-metrics) documentation for an example implementation.
Inheriting classes are expected to override `KNOWN_EVENTS` and `PREFIX` constants to build event names and associated metrics. For example, for prefix `issues` and events array `%w[create, update, delete]`, three metrics will be added to the Service Ping payload: `counts.issues_create`, `counts.issues_update` and `counts.issues_delete`.
@@ -385,7 +382,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd/) and [P
- `controller_actions`: the controller actions to track.
- `name`: the event name.
- `conditions`: optional custom conditions. Uses the same format as Rails callbacks.
- - `destinations`: optional list of destinations. Currently supports `:redis_hll` and `:snowplow`. Default: [:redis_hll].
+ - `destinations`: optional list of destinations. Currently supports `:redis_hll` and `:snowplow`. Default: `:redis_hll`.
- `&block`: optional block that computes and returns the `custom_id` that we want to track. This overrides the `visitor_id`.
Example:
@@ -623,7 +620,7 @@ alt_usage_data(999)
### Add counters to build new metrics
When adding the results of two counters, use the `add` Service Data method that
-handles fallback values and exceptions. It also generates a valid [SQL export](index.md#export-service-ping-sql-queries-and-definitions).
+handles fallback values and exceptions. It also generates a valid [SQL export](index.md#export-service-ping-data).
Example:
@@ -773,7 +770,7 @@ To set up Service Ping locally, you must:
1. Using the `gitlab` Rails console, manually trigger Service Ping:
```ruby
- ServicePing::SubmitService.new.execute
+ GitlabServicePingWorker.new.perform('triggered_from_cron' => false)
```
1. Use the `versions` Rails console to check the Service Ping was successfully received,
@@ -809,7 +806,7 @@ This is the recommended approach to test Prometheus-based Service Ping.
To verify your change, build a new Omnibus image from your code branch using CI/CD, download the image,
and run a local container instance:
-1. From your merge request, select the `qa` stage, then trigger the `package-and-qa` job. This job triggers an Omnibus
+1. From your merge request, select the `qa` stage, then trigger the `e2e:package-and-test` job. This job triggers an Omnibus
build in a [downstream pipeline of the `omnibus-gitlab-mirror` project](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/pipelines).
1. In the downstream pipeline, wait for the `gitlab-docker` job to finish.
1. Open the job logs and locate the full container name including the version. It takes the following form: `registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>`.
diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md
index 4481fe33bda..f252eb967aa 100644
--- a/doc/development/service_ping/index.md
+++ b/doc/development/service_ping/index.md
@@ -374,9 +374,9 @@ Possible values are "Amazon Aurora PostgreSQL", "PostgreSQL on Amazon RDS", "Clo
In GitLab 13.5, `pg_system_id` was added to send the [PostgreSQL system identifier](https://www.2ndquadrant.com/en/blog/support-for-postgresqls-system-identifier-in-barman/).
-## Export Service Ping SQL queries and definitions
+## Export Service Ping data
-Two Rake tasks exist to export Service Ping definitions.
+Rake tasks exist to export Service Ping data in different formats.
- The Rake tasks export the raw SQL queries for `count`, `distinct_count`, `sum`.
- The Rake tasks export the Redis counter class or the line of the Redis block for `redis_usage_data`.
@@ -385,12 +385,15 @@ Two Rake tasks exist to export Service Ping definitions.
In the home directory of your local GitLab installation run the following Rake tasks for the YAML and JSON versions respectively:
```shell
-# for YAML export
+# for YAML export of SQL queries
bin/rake gitlab:usage_data:dump_sql_in_yaml
-# for JSON export
+# for JSON export of SQL queries
bin/rake gitlab:usage_data:dump_sql_in_json
+# for JSON export of Non SQL data
+bin/rake gitlab:usage_data:dump_non_sql_in_json
+
# You may pipe the output into a file
bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml
```
@@ -405,7 +408,7 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta
1. Request temporary [access](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to the required environment.
1. After your approval is issued, [access the Rails console](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#access-approval).
-1. Run `ServicePing::SubmitService.new.execute`.
+1. Run `GitlabServicePingWorker.new.perform('triggered_from_cron' => false)`.
#### Trigger Service Ping with a detached screen session
@@ -430,7 +433,7 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta
1. Run:
```shell
- ServicePing::SubmitService.new.execute
+ GitlabServicePingWorker.new.perform('triggered_from_cron' => false)
```
1. To detach from screen, press `ctrl + A`, `ctrl + D`.
@@ -490,7 +493,7 @@ To skip database write operations, DevOps report creation, and storage of usage
```shell
skip_db_write:
-ServicePing::SubmitService.new(skip_db_write: true).execute
+GitlabServicePingWorker.new.perform('triggered_from_cron' => false, 'skip_db_write' => true)
```
## Monitoring
diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md
index 9dc37386111..debb3a68c04 100644
--- a/doc/development/service_ping/metrics_instrumentation.md
+++ b/doc/development/service_ping/metrics_instrumentation.md
@@ -154,22 +154,24 @@ end
You can use Redis metrics to track events not kept in the database, for example, a count of how many times the search bar has been used.
-[Example of a merge request that adds a `Redis` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66582).
+[Example of a merge request that adds a `Redis` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97009).
+
+Please note that `RedisMetric` class can only be used as the `instrumentation_class` for Redis metrics with simple counters classes (classes that only inherit `BaseCounter` and set `PREFIX` and `KNOWN_EVENTS` constants). In case the counter class has additional logic included in it, a new `instrumentation_class`, inheriting from `RedisMetric`, needs to be created. This new class needs to include the additional logic from the counter class.
Count unique values for `source_code_pushes` event.
Required options:
- `event`: the event name.
-- `counter_class`: one of the counter classes from the `Gitlab::UsageDataCounters` namespace; it should implement `read` method or inherit it from `BaseCounter`.
+- `prefix`: the value of the `PREFIX` constant used in the counter classes from the `Gitlab::UsageDataCounters` namespace.
```yaml
time_frame: all
data_source: redis
-instrumentation_class: 'RedisMetric'
+instrumentation_class: RedisMetric
options:
event: pushes
- counter_class: SourceCodeCounter
+ prefix: source_code
```
### Availability-restrained Redis metrics
@@ -197,14 +199,14 @@ You must also use the class's name in the YAML setup.
```yaml
time_frame: all
data_source: redis
-instrumentation_class: 'MergeUsageCountRedisMetric'
+instrumentation_class: MergeUsageCountRedisMetric
options:
event: pushes
- counter_class: SourceCodeCounter
+ prefix: source_code
```
## Redis HyperLogLog metrics
-
+
You can use Redis HyperLogLog metrics to track events not kept in the database and incremented for unique values such as unique users,
for example, a count of how many different users used the search bar.
@@ -215,7 +217,7 @@ Count unique values for `i_quickactions_approve` event.
```yaml
time_frame: 28d
data_source: redis_hll
-instrumentation_class: 'RedisHLLMetric'
+instrumentation_class: RedisHLLMetric
options:
events:
- i_quickactions_approve
@@ -246,7 +248,7 @@ You must also use the class's name in the YAML setup.
```yaml
time_frame: 28d
data_source: redis_hll
-instrumentation_class: 'MergeUsageCountRedisHLLMetric'
+instrumentation_class: MergeUsageCountRedisHLLMetric
options:
events:
- i_quickactions_approve
@@ -286,7 +288,7 @@ You must also include the instrumentation class name in the YAML setup.
```yaml
time_frame: 28d
-instrumentation_class: 'IssuesBoardsCountMetric'
+instrumentation_class: IssuesBoardsCountMetric
```
## Generic metrics
@@ -337,13 +339,13 @@ To create a stub instrumentation for a Service Ping metric, you can use a dedica
The generator takes the class name as an argument and the following options:
- `--type=TYPE` Required. Indicates the metric type. It must be one of: `database`, `generic`, `redis`, `numbers`.
-- `--operation` Required for `database` & `numebers` type.
+- `--operation` Required for `database` & `numbers` type.
- For `database` it must be one of: `count`, `distinct_count`, `estimate_batch_distinct_count`, `sum`, `average`.
- For `numbers` it must be: `add`.
- `--ee` Indicates if the metric is for EE.
```shell
-rails generate gitlab:usage_metric CountIssues --type database
+rails generate gitlab:usage_metric CountIssues --type database --operation distinct_count
create lib/gitlab/usage/metrics/instrumentations/count_issues_metric.rb
create spec/lib/gitlab/usage/metrics/instrumentations/count_issues_metric_spec.rb
```
diff --git a/doc/development/service_ping/troubleshooting.md b/doc/development/service_ping/troubleshooting.md
index 29ab334f867..4431d5df3ff 100644
--- a/doc/development/service_ping/troubleshooting.md
+++ b/doc/development/service_ping/troubleshooting.md
@@ -4,7 +4,7 @@ group: Product Intelligence
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
-# Troubleshooting
+# Troubleshooting Service Ping
## Service Ping Payload drop
@@ -58,7 +58,7 @@ checking the configuration file of your GitLab instance:
- Using the Admin Area:
- 1. On the top bar, select **Menu > Admin**.
+ 1. On the top bar, select **Main menu > Admin**.
1. On the left sidebar, select **Settings > Metrics and profiling**.
1. Expand **Usage Statistics**.
1. Are you able to check or uncheck the checkbox to disable Service Ping?
@@ -115,7 +115,7 @@ To work around this bug, you have two options:
sudo gitlab-ctl reconfigure
```
- 1. In GitLab, on the top bar, select **Menu > Admin**.
+ 1. In GitLab, on the top bar, select **Main menu > Admin**.
1. On the left sidebar, select **Settings > Metrics and profiling**.
1. Expand **Usage Statistics**.
1. Clear the **Enable Service Ping** checkbox.
diff --git a/doc/development/shell_scripting_guide/index.md b/doc/development/shell_scripting_guide/index.md
index 3d58fabad72..0591d2c64d0 100644
--- a/doc/development/shell_scripting_guide/index.md
+++ b/doc/development/shell_scripting_guide/index.md
@@ -38,7 +38,7 @@ in a particular case.
According to the [GitLab installation requirements](../../install/requirements.md),
this guide covers only those shells that are used by
-[supported Linux distributions](../../install/requirements.md#supported-linux-distributions),
+[supported Linux distributions](../../administration/package_information/supported_os.md#supported-operating-systems),
that is:
- [POSIX Shell](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html)
diff --git a/doc/development/sidekiq/compatibility_across_updates.md b/doc/development/sidekiq/compatibility_across_updates.md
index 1d369b5a970..ac34d099202 100644
--- a/doc/development/sidekiq/compatibility_across_updates.md
+++ b/doc/development/sidekiq/compatibility_across_updates.md
@@ -18,13 +18,13 @@ several possible situations:
## Adding new workers
-On GitLab.com, we
-[do not currently have a Sidekiq deployment in the canary stage](https://gitlab.com/gitlab-org/gitlab/-/issues/19239).
+On GitLab.com, we
+[do not currently have a Sidekiq deployment in the canary stage](https://gitlab.com/gitlab-org/gitlab/-/issues/19239).
This means that a new worker than can be scheduled from an HTTP endpoint may
be scheduled from canary but not run on Sidekiq until the full
production deployment is complete. This can be several hours later than
scheduling the job. For some workers, this will not be a problem. For
-others - particularly [latency-sensitive jobs](worker_attributes.md#latency-sensitive-jobs) -
+others - particularly [latency-sensitive jobs](worker_attributes.md#latency-sensitive-jobs) -
this will result in a poor user experience.
This only applies to new worker classes when they are first introduced.
diff --git a/doc/development/sidekiq/idempotent_jobs.md b/doc/development/sidekiq/idempotent_jobs.md
index 5d1ebce763e..da36cdc72aa 100644
--- a/doc/development/sidekiq/idempotent_jobs.md
+++ b/doc/development/sidekiq/idempotent_jobs.md
@@ -78,7 +78,7 @@ GitLab supports two deduplication strategies:
- `until_executing`, which is the default strategy
- `until_executed`
-More [deduplication strategies have been suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/195).
+More [deduplication strategies have been suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/195).
If you are implementing a worker that could benefit from a different
strategy, please comment in the issue.
diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md
index 24cd9093267..42a968b9df0 100644
--- a/doc/development/snowplow/index.md
+++ b/doc/development/snowplow/index.md
@@ -31,7 +31,7 @@ Snowplow tracking is enabled on GitLab.com, and we use it for most of our tracki
To enable Snowplow tracking on a self-managed instance:
-1. On the top bar, select **Menu > Admin**, then select **Settings > General**.
+1. On the top bar, select **Main menu > Admin**, then select **Settings > General**.
Alternatively, go to `admin/application_settings/general` in your browser.
1. Expand **Snowplow**.
diff --git a/doc/development/snowplow/troubleshooting.md b/doc/development/snowplow/troubleshooting.md
index 42a433e6a94..3ad4c6c9549 100644
--- a/doc/development/snowplow/troubleshooting.md
+++ b/doc/development/snowplow/troubleshooting.md
@@ -4,7 +4,13 @@ group: Product Intelligence
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
-# Troubleshooting
+# Troubleshooting Snowplow
+
+## Monitoring
+
+This page covers dashboards and alerts coming from a number of internal tools.
+
+For a brief video overview of the tools used to monitor Snowplow usage, please check out [this internal video](https://www.youtube.com/watch?v=NxPS0aKa_oU) (you must be logged into GitLab Unfiltered to view).
## Good events drop
diff --git a/doc/development/sql.md b/doc/development/sql.md
index 7101bf7fb4b..029874011c4 100644
--- a/doc/development/sql.md
+++ b/doc/development/sql.md
@@ -80,7 +80,7 @@ USING GIN(column_name gin_trgm_ops);
```
The key here is the `GIN(column_name gin_trgm_ops)` part. This creates a
-[GIN index](https://www.postgresql.org/docs/current/gin.html)
+[GIN index](https://www.postgresql.org/docs/current/gin.html)
with the operator class set to `gin_trgm_ops`. These indexes
_can_ be used by `ILIKE` / `LIKE` and can lead to greatly improved performance.
One downside of these indexes is that they can easily get quite large (depending
@@ -397,7 +397,7 @@ default.
While `WHERE IN` and `WHERE EXISTS` can be used to produce the same data it is
recommended to use `WHERE EXISTS` whenever possible. While in many cases
-PostgreSQL can optimise `WHERE IN` quite well there are also many cases where
+PostgreSQL can optimize `WHERE IN` quite well there are also many cases where
`WHERE EXISTS` performs (much) better.
In Rails you have to use this by creating SQL fragments:
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md
index 79a72981e3f..221d6b89b20 100644
--- a/doc/development/testing_guide/best_practices.md
+++ b/doc/development/testing_guide/best_practices.md
@@ -83,6 +83,31 @@ You can silence deprecation warnings by setting the environment variable
SILENCE_DEPRECATIONS=1 bin/rspec spec/models/project_spec.rb
```
+### Test order
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/93137) in GitLab 15.4.
+
+All new spec files are run in [random order](https://gitlab.com/gitlab-org/gitlab/-/issues/337399)
+to surface flaky tests that are dependent on test order.
+
+When randomized:
+
+- The string `# order random` is added below the example group description.
+- The used seed is shown in the spec output below the test suite summary. For example, `Randomized with seed 27443`.
+
+For a list of spec files which are still run in defined order, see [`rspec_order_todo.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/rspec_order_todo.yml).
+
+To make spec files run in random order, check their order dependency with:
+
+```shell
+scripts/rspec_check_order_dependence spec/models/project_spec.rb
+```
+
+If the specs pass the check the script removes them from
+[`rspec_order_todo.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/rspec_order_todo.yml) automatically.
+
+If the specs fail the check they must be fixed before than can run in random order.
+
### Test speed
GitLab has a massive test suite that, without [parallelization](../pipelines.md#test-suite-parallelization), can take hours
@@ -232,7 +257,7 @@ Here is an example of when `let_it_be` cannot be used, but `let_it_be_with_reloa
```ruby
let_it_be(:user) { create(:user) }
-let_it_be_with_reload(:project) { create(:project) } # The test will fail if `let_it_be` is used
+let_it_be_with_reload(:project) { create(:project) } # The test will fail if `let_it_be` is used
context 'with a developer' do
before do
@@ -422,7 +447,7 @@ Use the coverage reports to ensure your tests cover 100% of your code.
### System / Feature tests
NOTE:
-Before writing a new system test,
+Before writing a new system test,
[please consider **not** writing one](testing_levels.md#consider-not-writing-a-system-test)!
- Feature specs should be named `ROLE_ACTION_spec.rb`, such as
@@ -711,6 +736,7 @@ should either:
- Add `require_dependency 're2'` to files in your library that need `re2` gem,
to make this requirement explicit. This approach is preferred.
- Add it to the spec itself.
+- Use `rubocop_spec_helper` for RuboCop related specs.
It takes around one second to load tests that are using `fast_spec_helper`
instead of 30+ seconds in case of a regular `spec_helper`.
@@ -909,7 +935,7 @@ By default, Sidekiq jobs are enqueued into a jobs array and aren't processed.
If a test queues Sidekiq jobs and need them to be processed, the
`:sidekiq_inline` trait can be used.
-The `:sidekiq_might_not_need_inline` trait was added when
+The `:sidekiq_might_not_need_inline` trait was added when
[Sidekiq inline mode was changed to fake mode](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15479)
to all the tests that needed Sidekiq to actually process jobs. Tests with
this trait should be either fixed to not rely on Sidekiq processing jobs, or their
@@ -1048,7 +1074,7 @@ Most tests for Elasticsearch logic relate to:
There are some exceptions, such as checking for structural changes rather than individual records in an index.
-The `:elastic_with_delete_by_query` trait was added to reduce run time for pipelines by creating and deleting indices
+The `:elastic_delete_by_query` trait was added to reduce run time for pipelines by creating and deleting indices
at the start and end of each context only. The [Elasticsearch DeleteByQuery API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html)
is used to delete data in all indices in between examples to ensure a clean index.
@@ -1081,12 +1107,11 @@ Snowplow performs **runtime type checks** by using the [contracts gem](https://r
Because Snowplow is **by default disabled in tests and development**, it can be hard to
**catch exceptions** when mocking `Gitlab::Tracking`.
-To catch runtime errors due to type checks, you can enable Snowplow in tests. Mark the spec with
-`:snowplow` and use the `expect_snowplow_event` helper, which checks for
+To catch runtime errors due to type checks you can use `expect_snowplow_event`, which checks for
calls to `Gitlab::Tracking#event`.
```ruby
-describe '#show', :snowplow do
+describe '#show' do
it 'tracks snowplow events' do
get :show
@@ -1111,7 +1136,7 @@ end
When you want to ensure that no event got called, you can use `expect_no_snowplow_event`.
```ruby
- describe '#show', :snowplow do
+ describe '#show' do
it 'does not track any snowplow events' do
get :show
@@ -1357,6 +1382,47 @@ RSpec.configure do |config|
end
```
+### Testing Ruby constants
+
+When testing code that uses Ruby constants, focus the test on the behavior that depends on the constant,
+rather than testing the values of the constant.
+
+For example, the following is preferred because it tests the behavior of the class method `.categories`.
+
+```ruby
+ describe '.categories' do
+ it 'gets CE unique category names' do
+ expect(described_class.categories).to include(
+ 'deploy_token_packages',
+ 'user_packages',
+ # ...
+ 'kubernetes_agent'
+ )
+ end
+ end
+```
+
+On the other hand, testing the value of the constant itself, often only repeats the values
+in the code and the test, which provides little value.
+
+```ruby
+ describe CATEGORIES do
+ it 'has values' do
+ expect(CATEGORIES).to eq([
+ 'deploy_token_packages',
+ 'user_packages',
+ # ...
+ 'kubernetes_agent'
+ ])
+ end
+end
+```
+
+In critical cases where an error on a constant could have a catastrophic impact,
+testing the constant values might be useful as an added safeguard. For example,
+if it could bring down the entire GitLab service, cause a customer to be billed more than they should be,
+or [cause the universe to implode](../contributing/verify/index.md#do-not-cause-our-universe-to-implode).
+
### Factories
GitLab uses [factory_bot](https://github.com/thoughtbot/factory_bot) as a test fixture replacement.
diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md
index bfda94b1f1d..b17ca9e6f8f 100644
--- a/doc/development/testing_guide/end_to_end/best_practices.md
+++ b/doc/development/testing_guide/end_to_end/best_practices.md
@@ -62,13 +62,13 @@ In those and similar cases we need to include the test case link by other means.
To illustrate, there are two tests in the shared examples in [`qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb):
```ruby
-shared_examples 'unselected maintainer' do |testcase|
+RSpec.shared_examples 'unselected maintainer' do |testcase|
it 'user fails to push', testcase: testcase do
...
end
end
-shared_examples 'selected developer' do |testcase|
+RSpec.shared_examples 'selected developer' do |testcase|
it 'user pushes and merges', testcase: testcase do
...
end
@@ -415,7 +415,7 @@ except(page).to have_no_text('hidden')
Unfortunately, that's not automatically the case for the predicate methods that we add to our
[page objects](page_objects.md). We need to [create our own negatable matchers](https://relishapp.com/rspec/rspec-expectations/v/3-9/docs/custom-matchers/define-a-custom-matcher#matcher-with-separate-logic-for-expect().to-and-expect().not-to).
-The initial example uses the `have_job` matcher which is derived from the
+The initial example uses the `have_job` matcher which is derived from the
[`has_job?` predicate method of the `Page::Project::Pipeline::Show` page object](https://gitlab.com/gitlab-org/gitlab/-/blob/87864b3047c23b4308f59c27a3757045944af447/qa/qa/page/project/pipeline/show.rb#L53).
To create a negatable matcher, we use `has_no_job?` for the negative case:
diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md
index 33f73304a26..9a39161f1ad 100644
--- a/doc/development/testing_guide/end_to_end/feature_flags.md
+++ b/doc/development/testing_guide/end_to_end/feature_flags.md
@@ -195,12 +195,10 @@ End-to-end tests should pass with a feature flag enabled before it is enabled on
There are two ways to confirm that end-to-end tests pass:
- If a merge request adds or edits a [feature flag definition file](../../feature_flags/index.md#feature-flag-definition-and-validation),
- two `package-and-qa` jobs (`package-and-qa-ff-enabled` and `package-and-qa-ff-disabled`) are included automatically in the merge request
- pipeline. One job enables the defined feature flag and the other job disables it. The jobs execute the same suite of tests to confirm
- that they pass with the feature flag either enabled or disabled.
-- In some cases, if `package-and-qa` hasn't been triggered automatically, or if it has run the tests with the default feature flag values
- (which might not be desired), you can create a Draft MR that enables the feature flag to ensure that all E2E tests pass with the feature
- flag enabled.
+ two `e2e:package-and-test` jobs (`ee:instance-parallel` and `ee:instance-parallel-ff-inverse`) are included automatically in the merge request pipeline.
+ One job runs the application with default feature flag state and another sets it to inverse value. The jobs execute the same suite of tests to confirm that they pass with the feature flag either enabled or disabled.
+- In some cases, if end-to-end test jobs didn't trigger automatically, or if it has run the tests with the default feature flag values (which might not be desired),
+ you can create a Draft MR that enables the feature flag to ensure that all E2E tests pass with the feature flag enabled and disabled.
### Troubleshooting end-to-end test failures with feature flag enabled
@@ -216,8 +214,8 @@ If enabling the feature flag results in E2E test failures, you can browse the ar
### Test execution during feature development
If an end-to-end test enables a feature flag, the end-to-end test suite can be used to test changes in a merge request
-by running the `package-and-qa` job in the merge request pipeline. If the feature flag and relevant changes have already been merged, you can confirm that the tests
-pass on the default branch. The end-to-end tests run on the default branch every two hours, and the results are posted to a
+by running the `e2e:package-and-test` job in the merge request pipeline. If the feature flag and relevant changes have already been merged, you can confirm that the tests
+pass on the default branch. The end-to-end tests run on the default branch every two hours, and the results are posted to a
[Test Session Report, which is available in the testcase-sessions project](https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues?label_name%5B%5D=found%3Amain).
If the relevant tests do not enable the feature flag themselves, you can check if the tests will need to be updated by opening
diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md
index 989d090d581..cc9c02a65ce 100644
--- a/doc/development/testing_guide/end_to_end/index.md
+++ b/doc/development/testing_guide/end_to_end/index.md
@@ -8,16 +8,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w
## What is end-to-end testing?
-End-to-end testing is a strategy used to check whether your application works
+End-to-end (e2e) testing is a strategy used to check whether your application works
as expected across the entire software stack and architecture, including
integration of all micro-services and components that are supposed to work
together.
## How do we test GitLab?
-We use [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) to build GitLab packages and then we
-test these packages using the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) tool, which is
-a black-box testing framework for the API and the UI.
+We use [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) to build GitLab packages and then we test these packages
+using the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) tool to run the end-to-end tests located in the `qa` directory.
### Testing nightly builds
@@ -33,11 +32,9 @@ You can find these pipelines at <https://gitlab.com/gitlab-org/quality/staging/p
### Testing code in merge requests
-#### Using the `package-and-qa` job
+#### Using the package-and-test job
-It is possible to run end-to-end tests for a merge request, eventually being run in
-a pipeline in the [`gitlab-org/gitlab-qa-mirror`](https://gitlab.com/gitlab-org/gitlab-qa-mirror) project,
-by triggering the `package-and-qa` manual action in the `qa` stage (not
+It is possible to run end-to-end tests for a merge request by triggering the `e2e:package-and-test` manual action in the `qa` stage (not
available for forks).
**This runs end-to-end tests against a custom EE (with an Ultimate license)
@@ -59,7 +56,7 @@ graph TB
subgraph "`gitlab-org/gitlab` pipeline"
A1[`build-images` stage<br>`build-qa-image` and `build-assets-image` jobs]
- A2[`qa` stage<br>`package-and-qa` job]
+ A2[`qa` stage<br>`e2e:package-and-test` job]
end
subgraph "`gitlab-org/build/omnibus-gitlab-mirror` pipeline"
@@ -72,38 +69,30 @@ subgraph "`gitlab-org/gitlab-qa-mirror` pipeline"
```
1. In the [`gitlab-org/gitlab` pipeline](https://gitlab.com/gitlab-org/gitlab):
- 1. Developer triggers the `package-and-qa` manual action (available once the `build-qa-image` and
+ 1. Developer triggers the `e2e:package-and-test` manual action (available once the `build-qa-image` and
`build-assets-image` jobs are done), that can be found in GitLab merge
- requests. This starts a chain of pipelines in multiple projects.
- 1. The script being executed triggers a pipeline in
+ requests. This starts a e2e test child pipeline.
+ 1. E2E child pipeline triggers a downstream pipeline in
[`gitlab-org/build/omnibus-gitlab-mirror`](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror)
and polls for the resulting status. We call this a _status attribution_.
1. In the [`gitlab-org/build/omnibus-gitlab-mirror` pipeline](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror):
1. Docker image is being built and pushed to its Container Registry.
- 1. Finally, the `Trigger:qa-test` job triggers a new end-to-end pipeline in
- [`gitlab-org/gitlab-qa-mirror`](https://gitlab.com/gitlab-org/gitlab-qa-mirror/pipelines) and polls for the resulting status.
+ 1. Once Docker images are built and pushed jobs in `test` stage are started
-1. In the [`gitlab-org/gitlab-qa-mirror` pipeline](https://gitlab.com/gitlab-org/gitlab-qa-mirror):
+1. In the `Test` stage:
1. Container for the Docker image stored in the [`gitlab-org/build/omnibus-gitlab-mirror`](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) registry is spun-up.
1. End-to-end tests are run with the `gitlab-qa` executable, which spin up a container for the end-to-end image from the [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) registry.
-1. The result of the [`gitlab-org/gitlab-qa-mirror` pipeline](https://gitlab.com/gitlab-org/gitlab-qa-mirror) is being
- propagated upstream (through polling from upstream pipelines), through [`gitlab-org/build/omnibus-gitlab-mirror`](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror), back to the [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) merge request.
-
-We plan to [add more specific information](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/156)
-about the tests included in each job/scenario that runs in `gitlab-org/gitlab-qa-mirror`.
-
NOTE:
You may have noticed that we use `gitlab-org/build/omnibus-gitlab-mirror` instead of
-`gitlab-org/omnibus-gitlab`, and `gitlab-org/gitlab-qa-mirror` instead of `gitlab-org/gitlab-qa`.
+`gitlab-org/omnibus-gitlab`.
This is due to technical limitations in the GitLab permission model: the ability to run a pipeline
against a protected branch is controlled by the ability to push/merge to this branch.
This means that for developers to be able to trigger a pipeline for the default branch in
-`gitlab-org/omnibus-gitlab`/`gitlab-org/gitlab-qa`, they would need to have the
-Maintainer role for those projects.
+`gitlab-org/omnibus-gitlab`, they would need to have the Maintainer role for this project.
For security reasons and implications, we couldn't open up the default branch to all the Developers.
-Hence we created these mirrors where Developers and Maintainers are allowed to push/merge to the default branch.
+Hence we created this mirror where Developers and Maintainers are allowed to push/merge to the default branch.
This problem was discovered in <https://gitlab.com/gitlab-org/gitlab-qa/-/issues/63#note_107175160> and the "mirror"
work-around was suggested in <https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4717>.
A feature proposal to segregate access control regarding running pipelines from ability to push/merge was also created at <https://gitlab.com/gitlab-org/gitlab/-/issues/24585>.
@@ -111,22 +100,19 @@ A feature proposal to segregate access control regarding running pipelines from
#### With merged results pipelines
In a merged results pipeline, the pipeline runs on a new ref that contains the merge result of the source and target branch.
-However, this ref is not available to the `gitlab-qa-mirror` pipeline.
-For this reason, the end-to-end tests on a merged results pipeline would use the head of the merge request source branch.
+The end-to-end tests on a merged results pipeline would use the new ref instead of the head of the merge request source branch.
```mermaid
graph LR
-A["a1b1c1 - branch HEAD (CI_MERGE_REQUEST_SOURCE_BRANCH_SHA)"]
-B["x1y1z1 - master HEAD"]
-C["d1e1f1 - merged results (CI_COMMIT_SHA)"]
+A["x1y1z1 - master HEAD"]
+B["d1e1f1 - merged results (CI_COMMIT_SHA)"]
-A --> C
-B --> C
+A --> B
-A --> E["E2E tests"]
-C --> D["Merged results pipeline"]
+B --> C["Merged results pipeline"]
+C --> D["E2E tests"]
```
##### Running custom tests
@@ -140,7 +126,7 @@ a flaky test we first want to make sure that it's no longer flaky.
We can do that using the `ce:custom-parallel` and `ee:custom-parallel` jobs.
Both are manual jobs that you can configure using custom variables.
When clicking the name (not the play icon) of one of the parallel jobs,
-you are prompted to enter variables. You can use any of
+you are prompted to enter variables. You can use any of
[the variables that can be used with `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables)
as well as these:
@@ -150,8 +136,8 @@ as well as these:
| `QA_TESTS` | The tests to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, for example, `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. |
| `QA_RSPEC_TAGS` | The RSpec tags to add (no default) |
-For now,
-[manual jobs with custom variables don't use the same variable when retried](https://gitlab.com/gitlab-org/gitlab/-/issues/31367),
+For now,
+[manual jobs with custom variables don't use the same variable when retried](https://gitlab.com/gitlab-org/gitlab/-/issues/31367),
so if you want to run the same tests multiple times,
specify the same variables in each `custom-parallel` job (up to as
many of the 10 available jobs that you want to run).
@@ -165,7 +151,7 @@ automatically started: it runs the QA smoke suite against the
You can also manually start the `review-qa-all`: it runs the full QA suite
against the [Review App](../review_apps.md).
-**This runs end-to-end tests against a Review App based on
+**This runs end-to-end tests against a Review App based on
[the official GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/), itself deployed with custom
[Cloud Native components](https://gitlab.com/gitlab-org/build/CNG) built from your merge request's changes.**
@@ -197,7 +183,7 @@ Use these environment variables to configure metrics export:
| -------- | -------- | ----------- |
| `QA_INFLUXDB_URL` | `true` | Should be set to `https://influxdb.quality.gitlab.net`. No default value. |
| `QA_INFLUXDB_TOKEN` | `true` | InfluxDB write token that can be found under `Influxdb auth tokens` document in `Gitlab-QA` `1Password` vault. No default value. |
-| `QA_RUN_TYPE` | `false` | Arbitrary name for test execution, like `package-and-qa`. Automatically inferred from the project name for live environment test executions. No default value. |
+| `QA_RUN_TYPE` | `false` | Arbitrary name for test execution, like `package-and-test`. Automatically inferred from the project name for live environment test executions. No default value. |
| `QA_EXPORT_TEST_METRICS` | `false` | Flag to enable or disable metrics export. Defaults to `true`. |
## Test reports
@@ -231,7 +217,7 @@ a link to the current test report.
Each type of scheduled pipeline generates a static link for the latest test report according to its stage:
-- [`master`](https://storage.googleapis.com/gitlab-qa-allure-reports/package-and-qa/master/index.html)
+- [`master`](https://storage.googleapis.com/gitlab-qa-allure-reports/e2e-package-and-test/master/index.html)
- [`staging-full`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-full/master/index.html)
- [`staging-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity/master/index.html)
- [`staging-sanity-no-admin`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity-no-admin/master/index.html)
@@ -244,7 +230,7 @@ Each type of scheduled pipeline generates a static link for the latest test repo
If you are not [testing code in a merge request](#testing-code-in-merge-requests),
there are two main options for running the tests. If you want to run
the existing tests against a live GitLab instance or against a pre-built Docker image,
-use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also
+use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also
[examples of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples).
On the other hand, if you would like to run against a local development GitLab
@@ -263,7 +249,7 @@ architecture. See the [documentation about it](https://gitlab.com/gitlab-org/git
Once you decided where to put [test environment orchestration scenarios](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario) and
[instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features), take a look at the [GitLab QA README](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/README.md),
-the [GitLab QA orchestrator README](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md),
+the [GitLab QA orchestrator README](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md),
and [the already existing instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features).
### Consider **not** writing an end-to-end test
diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
index 322f2412e5b..1abaf3ef323 100644
--- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
+++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
@@ -15,27 +15,28 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. |
| `:except` | The test is to be run in their typical execution contexts _except_ as specified. See [test execution context selection](execution_context_selection.md) for more information. |
-| `:feature_flag` | The test uses a feature flag and therefore requires an administrator account to run. When `scope` is set to `:global`, the test will be skipped on all live .com environments. Otherwise, it will be skipped only on Canary, Production, and Preprod. See [testing with feature flags](../../../development/testing_guide/end_to_end/feature_flags.md) for more details. |
+| `:feature_flag` | The test uses a feature flag and therefore requires an administrator account to run. When `scope` is set to `:global`, the test will be skipped on all live .com environments. Otherwise, it will be skipped only on Canary, Production, and Preprod. See [testing with feature flags](../../../development/testing_guide/end_to_end/feature_flags.md) for more details. |
| `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. |
| `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. |
| `:github` | The test requires a GitHub personal access token. |
| `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. |
| `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. |
-| `:integrations` | This aims to test the available [integrations](../../../user/project/integrations/index.md#available-integrations). The test requires Docker to be installed in the run context. It will provision the containers and can be run against a local instance or using the `gitlab-qa` scenario `Test::Integration::Integrations` |
+| `:integrations` | This aims to test the available [integrations](../../../user/project/integrations/index.md#available-integrations). The test requires Docker to be installed in the run context. It will provision the containers and can be run against a local instance or using the `gitlab-qa` scenario `Test::Integration::Integrations` |
| `:issue`, `:issue_${num}` | Optional links to issues which might be related to the spec. Helps keep track of related issues and can also be used by tools that create test reports. Currently added automatically to `Allure` test report. Multiple tags can be used by adding an optional numeric suffix like `issue_1`, `issue_2` etc. |
-| `:service_ping_disabled` | The test interacts with the GitLab configuration service ping at the instance level to turn Admin Area setting service ping checkbox on or off. This tag will have the test run only in the `service_ping_disabled` job and must be paired with the `:orchestrated` and `:requires_admin` tags. |
+| `:service_ping_disabled` | The test interacts with the GitLab configuration service ping at the instance level to turn Admin Area setting service ping checkbox on or off. This tag will have the test run only in the `service_ping_disabled` job and must be paired with the `:orchestrated` and `:requires_admin` tags. |
| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. |
| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ |
| `:ldap_no_server` | The test requires a GitLab instance to be configured to use LDAP. To be used with the `:orchestrated` tag. It does not spin up an LDAP server at orchestration time. Instead, it creates the LDAP server at runtime. |
| `:ldap_no_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS not enabled. |
| `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. |
| `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. |
-| `:metrics` | The test requires a GitLab instance where [dedicated metrics exporters](../../../administration/monitoring/prometheus/web_exporter.md) are running alongside Puma and Sidekiq. |
+| `:metrics` | The test requires a GitLab instance where [dedicated metrics exporters](../../../administration/monitoring/prometheus/web_exporter.md) are running alongside Puma and Sidekiq. |
| `:mixed_env` | The test should only be executed in environments that have a paired canary version available through traffic routing based on the existence of the `gitlab_canary=true` cookie. Tests in this category are switching the cookie mid-test to validate mixed deployment environments. |
| `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. |
| `:only` | The test is only to be run in specific execution contexts. See [test execution context selection](execution_context_selection.md) for more information. |
| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). |
-| `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. |
+| `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/index.md#gitlab-package-registry-administration) enabled. |
+| `:product_group` | Specifies what product group the test belongs to. See [Product sections, stages, groups, and categories](https://about.gitlab.com/handbook/product/categories) for the comprehensive groups list. |
| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs in a specific context](execution_context_selection.md#quarantine-a-test-for-a-specific-environment). |
| `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). |
| `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |
@@ -44,7 +45,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| `:requires_git_protocol_v2` | The test requires that Git protocol version 2 is enabled on the server. It's assumed to be enabled by default but if not the test can be skipped by setting `QA_CAN_TEST_GIT_PROTOCOL_V2` to `false`. |
| `:requires_praefect` | The test requires that the GitLab instance uses [Gitaly Cluster](../../../administration/gitaly/praefect.md) (a.k.a. Praefect) as the repository storage . It's assumed to be used by default but if not the test can be skipped by setting `QA_CAN_TEST_PRAEFECT` to `false`. |
| `:runner` | The test depends on and sets up a GitLab Runner instance, typically to run a pipeline. |
-| `:sanity_feature_flags` | The test verifies the functioning of the feature flag handling part of the test framework |
+| `:sanity_feature_flags` | The test verifies the functioning of the feature flag handling part of the test framework |
| `:skip_live_env` | The test is excluded when run against live deployed environments such as Staging, Canary, and Production. |
| `:skip_fips_env` | The test is excluded when run against an environment in FIPS mode. |
| `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. |
diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md
index 3c8df7d3416..7409f15969d 100644
--- a/doc/development/testing_guide/flaky_tests.md
+++ b/doc/development/testing_guide/flaky_tests.md
@@ -123,7 +123,7 @@ reproduction.
### Hanging specs
-If a spec hangs, it might be caused by a [bug in Rails](https://github.com/rails/rails/issues/34310):
+If a spec hangs, it might be caused by a [bug in Rails](https://github.com/rails/rails/issues/45994):
- <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81112>
- <https://gitlab.com/gitlab-org/gitlab/-/issues/337039>
diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md
index 2845dde9a24..22a8792bac6 100644
--- a/doc/development/testing_guide/frontend_testing.md
+++ b/doc/development/testing_guide/frontend_testing.md
@@ -420,7 +420,11 @@ it('passes', () => {
### Waiting in tests
Sometimes a test needs to wait for something to happen in the application before it continues.
-Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/setTimeout) because it makes the reason for waiting unclear. Instead use one of the following approaches.
+
+You should try to avoid:
+
+- [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/setTimeout) because it makes the reason for waiting unclear. Additionally, it is faked in our tests so its usage is tricky.
+- [`setImmediate`](https://developer.mozilla.org/en-US/docs/Web/API/Window/setImmediate) because it is no longer supported in Jest 27 and later. See [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7002) for details.
#### Promises and Ajax calls
@@ -448,14 +452,15 @@ it('waits for an Ajax call', async () => {
});
```
-If you are not able to register handlers to the `Promise`, for example because it is executed in a synchronous Vue life cycle hook, take a look at the [waitFor](#wait-until-axios-requests-finish) helpers or you can flush all pending `Promise`s:
+If you cannot register handlers to the `Promise`, for example because it is executed in a synchronous Vue lifecycle hook, take a look at the [`waitFor`](#wait-until-axios-requests-finish) helpers or flush all pending `Promise`s with:
**in Jest:**
```javascript
-it('waits for an Ajax call', () => {
+it('waits for an Ajax call', async () => {
synchronousFunction();
- jest.runAllTicks();
+
+ await waitForPromises();
expect(something).toBe('done');
});
@@ -872,7 +877,7 @@ This will create a new fixture located at
`tmp/tests/frontend/fixtures-ee/graphql/releases/graphql/queries/all_releases.query.graphql.json`.
You can import the JSON fixture in a Jest test using the `test_fixtures` alias
-[as described below](#use-fixtures).
+[as described previously](#use-fixtures).
## Data-driven tests
@@ -1073,7 +1078,7 @@ testAction(
<!-- vale gitlab.Spelling = NO -->
-The Axios Utils mock module located in `spec/frontend/mocks/ce/lib/utils/axios_utils.js` contains two helper methods for Jest tests that spawn HTTP requests.
+The Axios Utils mock module located in `spec/frontend/__helpers__/mocks/axios_utils.js` contains two helper methods for Jest tests that spawn HTTP requests.
These are very useful if you don't have a handle to the request's Promise, for example when a Vue component does a request as part of its life cycle.
<!-- vale gitlab.Spelling = YES -->
diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md
index cd7c70e2eaa..e50902c4995 100644
--- a/doc/development/testing_guide/index.md
+++ b/doc/development/testing_guide/index.md
@@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
This document describes various guidelines and best practices for automated
testing of the GitLab project.
-It is meant to be an _extension_ of the
+It is meant to be an _extension_ of the
[Thoughtbot testing style guide](https://github.com/thoughtbot/guides/tree/master/testing-rspec). If
this guide defines a rule that contradicts the Thoughtbot guide, this guide
takes precedence. Some guidelines may be repeated verbatim to stress their
diff --git a/doc/development/testing_guide/testing_migrations_guide.md b/doc/development/testing_guide/testing_migrations_guide.md
index 261a4f4a27e..3006a2230ac 100644
--- a/doc/development/testing_guide/testing_migrations_guide.md
+++ b/doc/development/testing_guide/testing_migrations_guide.md
@@ -317,7 +317,7 @@ To test these you usually have to:
- Verify that the expected jobs were scheduled, with the correct set
of records, the correct batch size, interval, etc.
-The behavior of the background migration itself needs to be verified in a
+The behavior of the background migration itself needs to be verified in a
[separate test for the background migration class](#example-background-migration-test).
This spec tests the
diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md
index 0d545fa8e3f..bbea89d5645 100644
--- a/doc/development/value_stream_analytics.md
+++ b/doc/development/value_stream_analytics.md
@@ -6,33 +6,86 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Value stream analytics development guide
-Value stream analytics calculates the time between two arbitrary events recorded on domain objects and provides aggregated statistics about the duration.
+For information on how to configure value stream analytics (VSA) in GitLab, see our [analytics documentation](../user/analytics/value_stream_analytics.md).
-For information on how to configure value stream analytics in GitLab, see our [analytics documentation](../user/analytics/value_stream_analytics.md).
+## How does Value Stream Analytics work?
-## Stage
+Value Stream Analytics calculates the duration between two timestamp columns or timestamp
+expressions and runs various aggregations on the data.
-During development, events occur that move issues and merge requests through different stages of progress until they are considered finished. These stages can be expressed with the `Stage` model.
+For example:
-Example stage:
+- Duration between the Merge Request creation time and Merge Request merge time.
+- Duration between the Issue creation time and Issue close time.
-- Name: Development
-- Start event: Issue created
-- End event: Issue first mentioned in commit
-- Parent: `Group: gitlab-org`
+This duration is exposed in various ways:
+
+- Aggregation: median, average
+- Listing: list the duration for individual Merge Request and Issue records
+
+Apart from the durations, we expose the record count within a stage.
+
+## Feature availability
+
+- Group level (licensed): Requires Ultimate or Premium subscription. This version is the most
+feature-full.
+- Project level (licensed): We are continually adding features to project level VSA to bring it in line with group level VSA.
+- Project level (FOSS): Keep it as is.
+
+|Feature|Group level (licensed)|Project level (licensed)|Project level (FOSS)|
+|-|-|-|-|
+|Create custom value streams|Yes|No, only one value stream (default) is present with the default stages|no, only one value stream (default) is present with the default stages|
+|Create custom stages|Yes|No|No|
+|Filtering (author, label, milestone, etc.)|Yes|Yes|Yes|
+|Stage time chart|Yes|No|No|
+|Total time chart|Yes|No|No|
+|Task by type chart|Yes|No|No|
+|DORA Metrics|Yes|Yes|No|
+|Cycle time and lead time summary (Key metrics)|Yes|Yes|No|
+|New issues, commits and deploys (Key metrics)|Yes, excluding commits|Yes|Yes|
+|Uses aggregated backend|Yes|No|No|
+|Date filter behavior|Filters items [finished within the date range](https://gitlab.com/groups/gitlab-org/-/epics/6046)|Filters items by creation date.|Filters items by creation date.|
+|Authorization|At least reporter|At least reporter|Can be public.|
+
+## VSA core domain objects
+
+### Stages
+
+A stage represents an event pair (start and end events) with additional metadata, such as the name
+of the stage. Stages are configurable by the user within the pairing rules defined in the backend.
+
+**Example stage: Code Review**
+
+- Start event identifier: Merge request creation time.
+- Start event column: uses the `merge_requests.created_at` timestamp column.
+- End event identifier: Merge request merge time.
+- End event column: uses the `merge_request_metrics.merged_at` timestamp column.
+- Stage event hash ID: a calculated hash for the pair of start and end event identifiers.
+ - If two stages have the same configuration of start and end events, then their stage event hash.
+ IDs are identical.
+ - The stage event hash ID is later used to store the aggregated data in partitioned database tables.
+
+Historically, value stream analytics defined [7 stages](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/analytics/cycle_analytics/default_stages.rb)
+which are always available to the end-users regardless of the subscription.
+
+### Value streams
+
+Value streams are container objects for the stages. There can be multiple value streams per group
+focusing on different aspects of the Dev Ops lifecycle.
### Events
Events are the smallest building blocks of the value stream analytics feature. A stage consists of two events:
-- Start
-- End
+- Start event
+- End event
These events play a key role in the duration calculation.
Formula: `duration = end_event_time - start_event_time`
-To make the duration calculation flexible, each `Event` is implemented as a separate class. They're responsible for defining a timestamp expression that is used in the calculation query.
+To make the duration calculation flexible, each `Event` is implemented as a separate class.
+They're responsible for defining a timestamp expression that is used in the calculation query.
#### Implementing an `Event` class
@@ -81,7 +134,7 @@ def apply_query_customization(query)
end
```
-### Validating start and end events
+#### Validating start and end events
Some start/end event pairs are not "compatible" with each other. For example:
@@ -171,23 +224,7 @@ graph LR;
MergeRequestLabelRemoved --> MergeRequestLabelRemoved;
```
-### Parent
-
-Teams and organizations might define their own way of building software, thus stages can be completely different. For each stage, a parent object needs to be defined.
-
-Currently supported parents:
-
-- `Project`
-- `Group`
-
-#### How parent relationship it work
-
-1. User navigates to the value stream analytics page.
-1. User selects a group.
-1. Backend loads the defined stages for the selected group.
-1. Additions and modifications to the stages are persisted within the selected group only.
-
-### Default stages
+## Default stages
The [original implementation](https://gitlab.com/gitlab-org/gitlab/-/issues/847) of value stream analytics defined 7 stages. These stages are always available for each parent, however altering these stages is not possible.
@@ -209,31 +246,15 @@ The reason for this was that we'd like to add the abilities to hide and order st
For a new calculation or a query, implement it as a new method call in the `DataCollector` class.
-## Database query
-
-Structure of the database query:
-
-```sql
-SELECT (customized by: Median or RecordsFetcher or DataForDurationChart)
-FROM OBJECT_TYPE (Issue or MergeRequest)
-INNER JOIN (several JOIN statements, depending on the events)
-WHERE
- (Filter by the PARENT model, example: filter Issues from Project A)
- (Date range filter based on the OBJECT_TYPE.created_at)
- (Check if the START_EVENT is earlier than END_EVENT, preventing negative duration)
-```
-
-Structure of the `SELECT` statement for `Median`:
+To support the aggregated value stream analytics backend, these classes were reimplemented within [`Aggregated`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/analytics/cycle_analytics/aggregated) namespace.
-```sql
-SELECT (calculate median from START_EVENT_TIME-END_EVENT_TIME)
-```
+### Database query backend
-Structure of the `SELECT` statement for `DataForDurationChart`:
+VSA supports two backends: [aggregated](value_stream_analytics/value_stream_analytics_aggregated_backend.md) and "live". The live query backend can be
+considered legacy, which will be phased out at some point.
-```sql
-SELECT (START_EVENT_TIME-END_EVENT_TIME) as duration, END_EVENT.timestamp
-```
+- "live": uses the standard `IssuableFinders`.
+- aggregated: queries data from pre-aggregated database tables.
## High-level overview
@@ -244,6 +265,31 @@ SELECT (START_EVENT_TIME-END_EVENT_TIME) as duration, END_EVENT.timestamp
- Responsible for composing queries and define feature specific business logic.
- `DataCollector`, `Event`, `StageEvents`, etc.
+## Frontend
+
+[Project VSA](../user/analytics/value_stream_analytics.md) is available for all users and:
+
+- Includes a mixture of key and DORA metrics based on the tier.
+- Uses the set of [default stages](#default-stages).
+
+[Group VSA](../user/group/value_stream_analytics/index.md) is only available for licensed users and extends project VSA to include:
+
+- An [overview stage](https://gitlab.com/gitlab-org/gitlab/-/issues/321438).
+- The ability to create custom value streams.
+
+The group and project level VSA frontends are both built with Vue and Vuex and follow a similar pattern:
+
+- The `index.js` file extracts any URL query parameters, creates the Vue app and Vuex store, and dispatches an `initialize` Vuex action.
+- The `base.vue` file is used to render the main components for each page, metrics, filters, charts, and the stage table.
+
+The group VSA Vuex store makes use of [Vuex modules](https://vuex.vuejs.org/guide/modules.html) to separate some of the state and logic used for rendering the charts.
+
+### Shared components
+
+Parts of the UI are shared between project VSA and group VSA such as the stage table and path. These shared components live in the project VSA directory `app/assets/javascripts/cycle_analytics/components` and are included at the group level VSA where needed.
+
+All the frontend code for group-level features are located in `ee/app/assets/javascripts/analytics/cycle_analytics/components`.
+
## Testing
Since we have a lots of events and possible pairings, testing each pairing is not possible. The rule is to have at least one test case using an `Event` class.
@@ -252,3 +298,31 @@ Writing a test case for a stage using a new `Event` can be challenging since dat
- Different parents: `Group` or `Project`
- Different calculations: `Median`, `RecordsFetcher` or `DataForDurationChart`
+
+The VSA frontend is tested extensively on two different levels (integration, unit):
+
+- End-to-end integration tests using a real backend via Capybara and RSpec.
+- Jest frontend tests with pre-generated data fixtures.
+
+## Development setup and testing
+
+Running Value Stream Analytics can be done via the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit). By default, you'll be able to view the project-level (FOSS) version of the feature.
+
+If your GDK is up and running, you can run the seed script to generate some data:
+
+```shell
+SEED_CYCLE_ANALYTICS=true SEED_VSA=true FILTER=cycle_analytics rake db:seed_fu
+```
+
+The data generator script creates a new group and a new project with issue and merge request
+data (see the output of the script). To view the group-level version of the feature, you
+need to request a license for your GDK instance.
+
+After this step, you can access the group level value stream analytics page where you can create
+value streams and stages. The data aggregation might be delayed so you might not see the
+data right after the stage creation. To speed up this process, you can run the following command
+in your rails console (`rails c`):
+
+```ruby
+Analytics::CycleAnalytics::ReaggregationWorker.new.perform
+```
diff --git a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md
index 79262e2d0dc..6087d4bd8f7 100644
--- a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md
+++ b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md
@@ -21,8 +21,7 @@ Value Stream Analytics (VSA).
## Current Status
-As of 14.8 the aggregated VSA backend is used only in the `gitlab-org` group, for testing purposes
-. We plan to gradually roll it out in the next major release (15.0) for the rest of the groups.
+The aggregated backend is used by default since GitLab 15.0 on the group-level.
## Motivation
@@ -53,44 +52,6 @@ database with a minimal development effort.
used by the system.
- Example: `MIN(issues.created_at, issues.updated_at)`
-## How does Value Stream Analytics work?
-
-Value Stream Analytics calculates the duration between two timestamp columns or timestamp
-expressions and runs various aggregations on the data.
-
-Examples:
-
-- Duration between the Merge Request creation time and Merge Request merge time.
-- Duration between the Issue creation time and Issue close time.
-
-This duration is exposed in various ways:
-
-- Aggregation: median, average
-- Listing: list the duration for individual Merge Request and Issue records
-
-Apart from the durations, we expose the record count within a stage.
-
-### Stages
-
-A stage represents an event pair (start and end events) with additional metadata, such as the name
-of the stage. Stages are configurable by the user within the pairing rules defined in the backend.
-
-**Example stage: Code Review**
-
-- Start event identifier: Merge Request creation time
-- Start event column: uses the `merge_requests.created_at` timestamp column.
-- End event identifier: Merge Request merge time
-- End event column: uses the `merge_request_metrics.merged_at` timestamp column.
-- Stage event hash ID: a calculated hash for the pair of start and end event identifiers.
- - If two stages have the same configuration of start and end events, then their stage event hash
- IDs are identical.
- - The stage event hash ID is later used to store the aggregated data in partitioned database tables.
-
-### Value streams
-
-Value streams are container objects for the stages. There can be multiple value streams per group
-or project focusing on different aspects of the Dev Ops lifecycle.
-
### Example configuration
![vsa object hierarchy example](img/object_hierarchy_example_V14_10.png)
diff --git a/doc/development/work_items.md b/doc/development/work_items.md
index 3625f85eb82..f15c66ae847 100644
--- a/doc/development/work_items.md
+++ b/doc/development/work_items.md
@@ -36,7 +36,7 @@ Here are some problems with current issues usage and why we are looking into wor
differences in common interactions that the user needs to hold a complicated mental
model of how they each behave.
- Issues are not extensible enough to support all of the emerging jobs they need to facilitate.
-- Codebase maintainability and feature development becomes a bigger challenge as we grow the Issue type.
+- Codebase maintainability and feature development becomes a bigger challenge as we grow the Issue type
beyond its core role of issue tracking into supporting the different work item types and handling
logic and structure differences.
- New functionality is typically implemented with first class objects that import behavior from issues via
@@ -204,3 +204,33 @@ provide a smooth migration path of epics to WIT with minimal disruption to user
We will move towards work items, work item types, and custom widgets (CW) in an iterative process.
For a rough outline of the work ahead of us, see [epic 6033](https://gitlab.com/groups/gitlab-org/-/epics/6033).
+
+## Redis HLL Counter Schema
+
+We need a more scalable Redis counter schema for work items that is inclusive of Plan xMAU, Project Management xMAU, Certify xMAU, and
+Product Planning xMAU. We cannot aggregate and dedupe events across features within a group or at the stage level with
+our current Redis slot schema.
+
+All three Plan product groups will be using the same base object (`work item`). Each product group still needs to
+track MAU.
+
+### Proposed aggregate counter schema
+
+```mermaid
+graph TD
+ Event[Specific Interaction Counter] --> AC[Aggregate Counters]
+ AC --> Plan[Plan xMAU]
+ AC --> PM[Project Management xMAU]
+ AC --> PP[Product Planning xMAU]
+ AC --> Cer[Certify xMAU]
+ AC --> WI[Work Items Users]
+```
+
+### Implementation
+
+The new aggregate schema is already implemented and we are already tracking work item unique actions
+in [GitLab.com](https://gitlab.com).
+
+For implementation details, this [MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/93231) can be used
+as a reference. The MR covers the definition of new unique actions, event tracking in the code and also
+adding the new unique actions to the required aggregate counters.
diff --git a/doc/development/workhorse/index.md b/doc/development/workhorse/index.md
index 962124248ef..f210f511954 100644
--- a/doc/development/workhorse/index.md
+++ b/doc/development/workhorse/index.md
@@ -10,7 +10,7 @@ GitLab Workhorse is a smart reverse proxy for GitLab. It handles
"large" HTTP requests such as file downloads, file uploads, Git
push/pull and Git archive downloads.
-Workhorse itself is not a feature, but there are
+Workhorse itself is not a feature, but there are
[several features in GitLab](gitlab_features.md) that would not work efficiently without Workhorse.
The canonical source for Workhorse is