Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2023-02-20 16:49:51 +0300
committerGitLab Bot <gitlab-bot@gitlab.com>2023-02-20 16:49:51 +0300
commit71786ddc8e28fbd3cb3fcc4b3ff15e5962a1c82e (patch)
tree6a2d93ef3fb2d353bb7739e4b57e6541f51cdd71 /doc/development
parenta7253423e3403b8c08f8a161e5937e1488f5f407 (diff)
Add latest changes from gitlab-org/gitlab@15-9-stable-eev15.9.0-rc42
Diffstat (limited to 'doc/development')
-rw-r--r--doc/development/adding_service_component.md11
-rw-r--r--doc/development/api_graphql_styleguide.md98
-rw-r--r--doc/development/application_slis/rails_request_apdex.md19
-rw-r--r--doc/development/approval_rules.md289
-rw-r--r--doc/development/architecture.md10
-rw-r--r--doc/development/bulk_import.md14
-rw-r--r--doc/development/cached_queries.md12
-rw-r--r--doc/development/caching.md6
-rw-r--r--doc/development/chaos_endpoints.md22
-rw-r--r--doc/development/cicd/index.md4
-rw-r--r--doc/development/code_review.md28
-rw-r--r--doc/development/contributing/community_roles.md21
-rw-r--r--doc/development/contributing/index.md92
-rw-r--r--doc/development/contributing/issue_workflow.md18
-rw-r--r--doc/development/contributing/merge_request_workflow.md27
-rw-r--r--doc/development/contributing/style_guides.md21
-rw-r--r--doc/development/database/add_foreign_key_to_existing_column.md115
-rw-r--r--doc/development/database/adding_database_indexes.md63
-rw-r--r--doc/development/database/avoiding_downtime_in_migrations.md146
-rw-r--r--doc/development/database/batched_background_migrations.md77
-rw-r--r--doc/development/database/clickhouse/gitlab_activity_data.md482
-rw-r--r--doc/development/database/clickhouse/index.md85
-rw-r--r--doc/development/database/clickhouse/optimization.md60
-rw-r--r--doc/development/database/database_debugging.md18
-rw-r--r--doc/development/database/index.md17
-rw-r--r--doc/development/database/migrations_for_multiple_databases.md3
-rw-r--r--doc/development/database/new_database_migration_version.md64
-rw-r--r--doc/development/database/pagination_guidelines.md2
-rw-r--r--doc/development/database/query_recorder.md1
-rw-r--r--doc/development/database/table_partitioning.md103
-rw-r--r--doc/development/deprecation_guidelines/index.md2
-rw-r--r--doc/development/development_processes.md2
-rw-r--r--doc/development/documentation/feature_flags.md26
-rw-r--r--doc/development/documentation/index.md4
-rw-r--r--doc/development/documentation/redirects.md6
-rw-r--r--doc/development/documentation/restful_api_styleguide.md11
-rw-r--r--doc/development/documentation/review_apps.md2
-rw-r--r--doc/development/documentation/styleguide/img/admin_access_level.pngbin9821 -> 8291 bytes
-rw-r--r--doc/development/documentation/styleguide/index.md5
-rw-r--r--doc/development/documentation/styleguide/word_list.md55
-rw-r--r--doc/development/documentation/testing.md76
-rw-r--r--doc/development/documentation/topic_types/task.md68
-rw-r--r--doc/development/documentation/versions.md18
-rw-r--r--doc/development/documentation/workflow.md21
-rw-r--r--doc/development/elasticsearch.md44
-rw-r--r--doc/development/experiment_guide/index.md4
-rw-r--r--doc/development/fe_guide/customizable_dashboards.md34
-rw-r--r--doc/development/fe_guide/design_anti_patterns.md6
-rw-r--r--doc/development/fe_guide/graphql.md4
-rw-r--r--doc/development/fe_guide/style/javascript.md4
-rw-r--r--doc/development/fe_guide/style/scss.md10
-rw-r--r--doc/development/fe_guide/view_component.md4
-rw-r--r--doc/development/fe_guide/vue.md48
-rw-r--r--doc/development/feature_categorization/index.md22
-rw-r--r--doc/development/feature_development.md4
-rw-r--r--doc/development/feature_flags/controls.md17
-rw-r--r--doc/development/feature_flags/index.md45
-rw-r--r--doc/development/gitaly.md22
-rw-r--r--doc/development/gitlab_flavored_markdown/specification_guide/index.md106
-rw-r--r--doc/development/gitlab_shell/index.md2
-rw-r--r--doc/development/go_guide/index.md2
-rw-r--r--doc/development/i18n/translation.md1
-rw-r--r--doc/development/import_export.md5
-rw-r--r--doc/development/index.md43
-rw-r--r--doc/development/integrations/codesandbox.md15
-rw-r--r--doc/development/integrations/jira_connect.md2
-rw-r--r--doc/development/integrations/secure.md62
-rw-r--r--doc/development/internal_api/index.md5
-rw-r--r--doc/development/logging.md26
-rw-r--r--doc/development/merge_request_application_and_rate_limit_guidelines.md31
-rw-r--r--doc/development/merge_request_concepts/approval_rules.md286
-rw-r--r--doc/development/merge_request_concepts/diffs/frontend.md208
-rw-r--r--doc/development/merge_request_concepts/performance.md23
-rw-r--r--doc/development/merge_request_concepts/rate_limits.md28
-rw-r--r--doc/development/newlines_styleguide.md11
-rw-r--r--doc/development/packages/dependency_proxy.md5
-rw-r--r--doc/development/packages/harbor_registry_development.md153
-rw-r--r--doc/development/packages/index.md6
-rw-r--r--doc/development/pages/index.md6
-rw-r--r--doc/development/performance.md16
-rw-r--r--doc/development/pipelines.md11
-rw-r--r--doc/development/pipelines/index.md244
-rw-r--r--doc/development/product_qualified_lead_guide/index.md44
-rw-r--r--doc/development/project_templates.md4
-rw-r--r--doc/development/rake_tasks.md8
-rw-r--r--doc/development/redis.md58
-rw-r--r--doc/development/ruby3_gotchas.md4
-rw-r--r--doc/development/sec/index.md6
-rw-r--r--doc/development/sec/security_report_ingestion_overview.md2
-rw-r--r--doc/development/sec/token_revocation_api.md118
-rw-r--r--doc/development/secure_coding_guidelines.md7
-rw-r--r--doc/development/service_ping/implement.md5
-rw-r--r--doc/development/service_ping/index.md1
-rw-r--r--doc/development/service_ping/metrics_lifecycle.md14
-rw-r--r--doc/development/service_ping/review_guidelines.md2
-rw-r--r--doc/development/sidekiq/index.md34
-rw-r--r--doc/development/sidekiq/worker_attributes.md26
-rw-r--r--doc/development/spam_protection_and_captcha/exploratory_testing.md4
-rw-r--r--doc/development/spam_protection_and_captcha/index.md4
-rw-r--r--doc/development/spam_protection_and_captcha/rest_api.md2
-rw-r--r--doc/development/stage_group_observability/dashboards/stage_group_dashboard.md2
-rw-r--r--doc/development/stage_group_observability/index.md2
-rw-r--r--doc/development/testing_guide/best_practices.md21
-rw-r--r--doc/development/testing_guide/contract/consumer_tests.md4
-rw-r--r--doc/development/testing_guide/contract/provider_tests.md4
-rw-r--r--doc/development/testing_guide/end_to_end/index.md6
-rw-r--r--doc/development/testing_guide/end_to_end/package_and_test_pipeline.md134
-rw-r--r--doc/development/testing_guide/end_to_end/resources.md2
-rw-r--r--doc/development/testing_guide/end_to_end/style_guide.md2
-rw-r--r--doc/development/testing_guide/flaky_tests.md27
-rw-r--r--doc/development/testing_guide/review_apps.md50
-rw-r--r--doc/development/transient/prevention-patterns.md4
-rw-r--r--doc/development/uploads/working_with_uploads.md28
-rw-r--r--doc/development/value_stream_analytics.md2
-rw-r--r--doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md10
-rw-r--r--doc/development/wikis.md6
-rw-r--r--doc/development/work_items_widgets.md25
-rw-r--r--doc/development/workhorse/gitlab_features.md3
118 files changed, 3242 insertions, 1222 deletions
diff --git a/doc/development/adding_service_component.md b/doc/development/adding_service_component.md
index ee14d8e6414..314065ffc10 100644
--- a/doc/development/adding_service_component.md
+++ b/doc/development/adding_service_component.md
@@ -10,7 +10,7 @@ The GitLab product is made up of several service components that run as independ
## Integration phases
-The following outline re-uses the [maturity metric](https://about.gitlab.com/direction/maturity/) naming as an example of the various phases of integrating a component. These are only loosely coupled to a components actual maturity, and are intended as a guide for implementation order (for example, a component does not need to be enabled by default to be Lovable, and being enabled by default does not on its own cause a component to be Lovable).
+The following outline re-uses the [maturity metric](https://about.gitlab.com/direction/maturity/) naming as an example of the various phases of integrating a component. These phases are only loosely coupled to a components actual maturity, and are intended as a guide for implementation order. For example, a component does not need to be enabled by default to be Lovable. Being enabled by default does not on its own cause a component to be Lovable.
- Proposed
- [Proposing a new component](#proposing-a-new-component)
@@ -37,7 +37,8 @@ The general steps for getting any GitLab feature from proposal to release can be
## Integrating a new service with GitLab
-Adding a new service follows the same [merge request workflow](contributing/merge_request_workflow.md) as other contributions, and must meet the same [completion criteria](contributing/merge_request_workflow.md#definition-of-done) and in addition needs to cover the following:
+Adding a new service follows the same [merge request workflow](contributing/merge_request_workflow.md) as other contributions, and must meet the same [completion criteria](contributing/merge_request_workflow.md#definition-of-done).
+In addition, it needs to cover the following:
- The [architecture component list](architecture.md#component-list) has been updated to include the service.
- Features provided by the component have been accepted into the [GitLab Product Direction](https://about.gitlab.com/direction/).
@@ -87,10 +88,10 @@ In addition, any system dependencies used in Omnibus packages or the Cloud Nativ
## Release management
-If the service component needs to be updated or released with the monthly GitLab release, then the component should be added to the [release tools automation](https://gitlab.com/gitlab-org/release-tools). This project is maintained by the [Delivery group](https://about.gitlab.com/handbook/engineering/infrastructure/team/delivery/).
+If the service component needs to be updated or released with the monthly GitLab release, then it should be added to the [release tools automation](https://gitlab.com/gitlab-org/release-tools). This project is maintained by the [Delivery group](https://about.gitlab.com/handbook/engineering/infrastructure/team/delivery/).
-There are different levels of automation available to include a component in GitLab releases. The requirements and process for including a component in a release at these different levels are detailed in the [release documentation](https://gitlab.com/gitlab-org/release/docs/-/tree/master/components).
+Different levels of automation are available to include a component in GitLab monthly releases. The requirements and process for including a component in a release at these different levels are detailed in the [release documentation](https://gitlab.com/gitlab-org/release/docs/-/tree/master/components).
A list of the projects with releases managed by release tools can be found in the [release tools project directory](https://gitlab.com/gitlab-org/release-tools/-/tree/master/lib/release_tools/project).
-For example, during the monthly GitLab release, the desired version of Gitaly, GitLab Workhorse and GitLab Shell need to be synchronized through the various release pipelines.
+For example, the desired version of Gitaly, GitLab Workhorse, and GitLab Shell need to be synchronized through the various release pipelines.
diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md
index 396bba2623e..94abbda9671 100644
--- a/doc/development/api_graphql_styleguide.md
+++ b/doc/development/api_graphql_styleguide.md
@@ -32,7 +32,7 @@ with anyone who may work in this part of the codebase in the future. You can fin
[Google Slides](https://docs.google.com/presentation/d/1qOTxpkTdHIp1CRjuTvO-aXg0_rUtzE3ETfLUdnBB5uQ/edit)
and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/8e78ea7f326b2ef649e7d7d569c26d56/GraphQL_Deep_Dive__Create_.pdf).
Everything covered in this deep dive was accurate as of GitLab 11.9, and while specific
-details may have changed since then, it should still serve as a good introduction.
+details may have changed after that release, it should still serve as a good introduction.
## GraphiQL
@@ -210,8 +210,8 @@ The `iid`, `title` and `description` are _scalar_ GraphQL types.
`iid` is a `GraphQL::Types::ID`, a special string type that signifies a unique ID.
`title` and `description` are regular `GraphQL::Types::String` types.
-Note that the old scalar types `GraphQL:ID`, `GraphQL::INT_TYPE`, `GraphQL::STRING_TYPE`,
-`GraphQL:BOOLEAN_TYPE`, and `GraphQL::FLOAT_TYPE` are no longer allowed. Please use `GraphQL::Types::ID`,
+The old scalar types `GraphQL:ID`, `GraphQL::INT_TYPE`, `GraphQL::STRING_TYPE`,
+`GraphQL:BOOLEAN_TYPE`, and `GraphQL::FLOAT_TYPE` are no longer allowed. Use `GraphQL::Types::ID`,
`GraphQL::Types::Int`, `GraphQL::Types::String`, `GraphQL::Types::Boolean`, and `GraphQL::Types::Float`.
When exposing a model through the GraphQL API, we do so by creating a
@@ -250,7 +250,7 @@ the following reasons:
- Changing from a non-nullable field to a nullable field is difficult with a versionless schema
Non-nullable fields should only be used when a field is required, very unlikely
-to become optional in the future, and very easy to calculate. An example would
+to become optional in the future, and straightforward to calculate. An example would
be `id` fields.
A non-nullable GraphQL schema field is an object type followed by the exclamation point (bang) `!`. Here's an example from the `gitlab_schema.graphql` file:
@@ -388,12 +388,12 @@ query($project_path: ID!) {
```
To ensure that we get consistent ordering, we append an ordering on the primary
-key, in descending order. This is usually `id`, so we add `order(id: :desc)`
+key, in descending order. The primary key is usually `id`, so we add `order(id: :desc)`
to the end of the relation. A primary key _must_ be available on the underlying table.
#### Shortcut fields
-Sometimes it can seem easy to implement a "shortcut field", having the resolver return the first of a collection if no parameters are passed.
+Sometimes it can seem straightforward to implement a "shortcut field", having the resolver return the first of a collection if no parameters are passed.
These "shortcut fields" are discouraged because they create maintenance overhead.
They need to be kept in sync with their canonical field, and deprecated or modified if their canonical field changes.
Use the functionality the framework provides unless there is a compelling reason to do otherwise.
@@ -692,7 +692,7 @@ Global IDs, so as such they are coupled to model names. When we rename a
model, its Global ID changes.
If the Global ID is used as an _argument_ type anywhere in the schema, then the Global ID
-change would normally constitute a breaking change.
+change would typically constitute a breaking change.
To continue to support clients using the old Global ID argument, we add a deprecation
to `Gitlab::GlobalId::Deprecations`.
@@ -763,24 +763,24 @@ support for the former argument style, remove the `Deprecation`:
DEPRECATIONS = [].freeze
```
-During the deprecation period the API will accept either of these formats for the argument value:
+During the deprecation period, the API accepts either of these formats for the argument value:
- `"gid://gitlab/PrometheusService/1"`
- `"gid://gitlab/Integrations::Prometheus/1"`
-The API will also accept these types in the query signature for the argument:
+The API also accepts these types in the query signature for the argument:
- `PrometheusServiceID`
- `IntegrationsPrometheusID`
NOTE:
-Although queries that use the old type (`PrometheusServiceID` in this example) will be
-considered valid and executable by the API, validator tools will consider them to be invalid.
-This is because we are deprecating using a bespoke method outside of the
+Although queries that use the old type (`PrometheusServiceID` in this example) are
+considered valid and executable by the API, validator tools consider them to be invalid.
+They are considered invalid because we are deprecating using a bespoke method outside of the
[`@deprecated` directive](https://spec.graphql.org/June2018/#sec--deprecated), so validators are not
aware of the support.
-The documentation will mention that the old Global ID style is now deprecated.
+The documentation mentions that the old Global ID style is now deprecated.
## Mark schema items as Alpha
@@ -897,7 +897,7 @@ An example of the use of a union for this purpose is
Field names can be mapped to hash data keys using the `hash_key:` keyword if needed.
-For example, given the following simple JSON data:
+For example, given the following JSON data:
```json
{
@@ -951,13 +951,25 @@ You can view descriptions of fields and arguments in:
#### Language and punctuation
-Use `{x} of the {y}` where possible, where `{x}` is the item you're describing,
-and `{y}` is the resource it applies to. For example:
+To describe fields and arguments, use `{x} of the {y}` where possible,
+where `{x}` is the item you're describing, and `{y}` is the resource it applies to. For example:
```plaintext
ID of the issue.
```
+```plaintext
+Author of the epics.
+```
+
+For arguments that sort or search, start with the appropriate verb.
+To indicate the specified values, for conciseness, you can use `this` instead of
+`the given` or `the specified`. For example:
+
+```plaintext
+Sort issues by this criteria.
+```
+
Do not start descriptions with `The` or `A`, for consistency and conciseness.
End all descriptions with a period (`.`).
@@ -1042,7 +1054,7 @@ the objects in question.
To find objects to display in a field, we can add resolvers to
`app/graphql/resolvers`.
-Arguments can be defined within the resolver in the same way as in a mutation.
+Arguments can be defined in the resolver in the same way as in a mutation.
See the [Mutation arguments](#object-identifier-arguments) section.
To limit the amount of queries performed, we can use [BatchLoader](graphql_guide/batchloader.md).
@@ -1086,7 +1098,7 @@ application:
- Services in mutations to apply operations.
- Loaders (batch-aware finders) specific to queries.
-Note that there is never any reason to use batching in a mutation. Mutations are
+There is never any reason to use batching in a mutation. Mutations are
executed in series, so there are no batching opportunities. All values are
evaluated eagerly as soon as they are requested, so batching is unnecessary
overhead. If you are writing:
@@ -1122,7 +1134,7 @@ the entire field should resolve to `null`.
### Deriving resolvers (`BaseResolver.single` and `BaseResolver.last`)
-For some simple use cases, we can derive resolvers from others.
+For some use cases, we can derive resolvers from others.
The main use case for this is one resolver to find all items, and another to
find one specific one. For this, we supply convenience methods:
@@ -1167,7 +1179,7 @@ class JobsResolver < BaseResolver
end
```
-Here we have a simple resolver for getting pipeline jobs. The `name` argument is
+Here we have a resolver for getting pipeline jobs. The `name` argument is
optional when getting a list, but required when getting a single job.
If there are multiple arguments, and neither can be made required, we can use
@@ -1315,7 +1327,7 @@ class MyThingResolver < BaseResolver
end
```
-For an example of real world use, please
+For an example of real world use,
see [`ResolvesMergeRequests`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/resolvers/concerns/resolves_merge_requests.rb).
### Negated arguments
@@ -1438,7 +1450,7 @@ It's acceptable to have both fine-grained mutations and coarse-grained mutations
that too many fine-grained mutations can lead to organizational challenges in maintainability, code
comprehensibility, and testing.
Each mutation requires a new class, which can lead to technical debt.
-It also means the schema becomes very big, and we want users to easily navigate our schema.
+It also means the schema becomes very big, which can make it difficult for users to navigate our schema.
As each new mutation also needs tests (including slower request integration tests), adding mutations
slows down the test suite.
@@ -1718,7 +1730,7 @@ two fields: `errors: [String]`, and `thing: ThingType`. The specific nature of
the `thing` itself is irrelevant to these examples, as we are considering the
errors.
-There are three states a mutation response can be in:
+The three states a mutation response can be in are:
- [Success](#success)
- [Failure (relevant to the user)](#failure-relevant-to-the-user)
@@ -1764,11 +1776,11 @@ Examples of this include:
- Model validation errors: the user may need to change the inputs.
- Permission errors: the user needs to know they cannot do this, they may need to request permission or sign in.
-- Problems with application state that prevent the user's action, for example: merge conflicts, the resource was locked, and so on.
+- Problems with the application state that prevent the user's action (for example, merge conflicts or a locked resource).
Ideally, we should prevent the user from getting this far, but if they do, they
need to be told what is wrong, so they understand the reason for the failure and
-what they can do to achieve their intent, even if that is as simple as retrying the
+what they can do to achieve their intent. For example, they might only need to retry the
request.
It is possible to return *recoverable* errors alongside mutation data. For example, if
@@ -1791,7 +1803,7 @@ In this case there is no `data`:
}
```
-This is the result of raising an error during the mutation. In our implementation,
+This results from raising an error during the mutation. In our implementation,
the messages of argument errors and validation errors are returned to the client, and all other
`StandardError` instances are caught, logged and presented to the client with the message set to `"Internal server error"`.
See [`GraphqlController`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/graphql_controller.rb) for details.
@@ -1820,7 +1832,7 @@ needs of the _user_ from the needs of the _client_.
> _Never catch an error unless the user needs to know about it._
If the user does need to know about it, communicate with frontend developers
-to make sure the error information we are passing back is useful.
+to make sure the error information we are passing back is relevant and serves a purpose.
See also the [frontend GraphQL guide](../development/fe_guide/graphql.md#handling-errors).
@@ -1863,7 +1875,7 @@ process, read [merge request !42588](https://gitlab.com/gitlab-org/gitlab/-/merg
We use subscriptions to push updates to clients. We use the [Action Cable implementation](https://graphql-ruby.org/subscriptions/action_cable_implementation)
to deliver the messages over websockets.
-When a client subscribes to a subscription, we store their query in-memory within Puma workers. Then when the subscription is triggered,
+When a client subscribes to a subscription, we store their query in-memory in Puma workers. Then when the subscription is triggered,
the Puma workers execute the stored GraphQL queries and push the results to the clients.
NOTE:
@@ -1885,7 +1897,7 @@ This class runs during the initial subscription request and subsequent updates.
You should implement the `#authorized?` method of the subscription class so that the initial subscription and subsequent updates are authorized.
When a user is not authorized, you should call the `unauthorized!` helper so that execution is halted and the user is unsubscribed. Returning `false`
-results in redaction of the response but we leak information that some updates are happening. This is due to a
+results in redaction of the response, but we leak information that some updates are happening. This leakage is due to a
[bug in the GraphQL gem](https://github.com/rmosolgo/graphql-ruby/issues/3390).
### Triggering subscriptions
@@ -1895,13 +1907,13 @@ code so that we have a single source of truth and we do not trigger a subscripti
## Pagination implementation
-To learn more, visit [GraphQL pagination](graphql_guide/pagination.md).
+For more information, see [GraphQL pagination](graphql_guide/pagination.md).
## Validating arguments
For validations of single arguments, use the
[`prepare` option](https://github.com/rmosolgo/graphql-ruby/blob/master/guides/fields/arguments.md)
-as normal.
+as usual.
Sometimes a mutation or resolver may accept a number of optional
arguments, but we still want to validate that at least one of the optional
@@ -1957,8 +1969,8 @@ field :created_at, Types::TimeType, null: true, description: 'Timestamp of when
## Testing
For testing mutations and resolvers, consider the unit of
-test a full GraphQL request, not a call to a resolver. The reasons for this are
-that we want to avoid lots of coupling to the framework, since this makes
+test a full GraphQL request, not a call to a resolver. This allows us to
+avoid tight coupling to the framework because such coupling makes
upgrades to dependencies much more difficult.
You should:
@@ -2022,7 +2034,7 @@ When adding a query, you can use the `a working graphql query` shared example to
renders valid results.
You can construct a query including all available fields using the `GraphqlHelpers#all_graphql_fields_for`
-helper. This makes it easy to add a test rendering all possible fields for a query.
+helper. This makes it more straightforward to add a test rendering all possible fields for a query.
If you're adding a field to a query that supports pagination and sorting,
visit [Testing](graphql_guide/pagination.md#testing) for details.
@@ -2164,11 +2176,11 @@ end
`spec/requests/api/graphql/ci/pipeline_spec.rb` regardless of the query being
used to fetch the pipeline data.
-- There can be possible cyclic dependencies within our GraphQL types.
+- There can be possible cyclic dependencies in our GraphQL types.
See [Adding field with resolver on a Type causes "Can't determine the return type " error on a different Type](https://github.com/rmosolgo/graphql-ruby/issues/3974#issuecomment-1084444214)
and [Fix unresolved name due to cyclic definition](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84202/diffs#diff-content-32d14251082fd45412e1fdbf5820e62d157e70d2).
- In particular, this can happen with `connection_type`. Normally we might use the following in a resolver:
+ In particular, this can happen with `connection_type`. Typically we might use the following in a resolver:
```ruby
type Types::IssueType.connection_type, null: true
@@ -2214,8 +2226,8 @@ end
type "Types::IssueConnection", null: true
```
- Only use this style if you are having spec failures. This is not intended to be a new
- pattern that we use. This issue should disappear after we've upgraded to `2.x`.
+ Only use this style if you are having spec failures. We should not typically
+ use this pattern. This issue should disappear after we've upgraded to `2.x`.
- There can be instances where a spec fails because the class is not loaded correctly.
It relates to the
@@ -2256,8 +2268,8 @@ end
See [this merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87295#note_946174036) for some discussion.
- Only use this style if you are having spec failures. This is not intended to be a new
- pattern that we use. This issue may disappear after we've upgraded to `2.x`.
+ Only use this style if you are having spec failures. We should not typically use this pattern.
+ This issue may disappear after we've upgraded to `2.x`.
- When testing resolvers using `GraphqlHelpers#resolve`, arguments for the resolver can be handled two ways.
@@ -2287,8 +2299,8 @@ end
```
The use of `:internal_prepared` was added as a bridge for the
- [GraphQL gem](https://graphql-ruby.org) upgrade. Testing resolvers directly will be
- [removed eventually](https://gitlab.com/gitlab-org/gitlab/-/issues/363121),
+ [GraphQL gem](https://graphql-ruby.org) upgrade. Testing resolvers directly will
+ [eventually be removed](https://gitlab.com/gitlab-org/gitlab/-/issues/363121),
and writing unit tests for resolvers/mutations is
[already deprecated](#writing-unit-tests-deprecated)
@@ -2329,7 +2341,7 @@ Queries and mutations are limited by depth, complexity, and recursion
to protect server resources from overly ambitious or malicious queries.
These values can be set as defaults and overridden in specific queries as needed.
The complexity values can be set per object as well, and the final query complexity is
-evaluated based on how many objects are being returned. This is useful
+evaluated based on how many objects are being returned. This can be used
for objects that are expensive (such as requiring Gitaly calls).
For example, a conditional complexity method in a resolver:
diff --git a/doc/development/application_slis/rails_request_apdex.md b/doc/development/application_slis/rails_request_apdex.md
index 8fcd725f74d..dc9d67b0a2b 100644
--- a/doc/development/application_slis/rails_request_apdex.md
+++ b/doc/development/application_slis/rails_request_apdex.md
@@ -228,18 +228,13 @@ get 'client/features', urgency: :low do
end
```
+WARNING:
+We can't specify the urgency at the namespace level. The directive is ignored when doing so.
+
### Error budget attribution and ownership
This SLI is used for service level monitoring. It feeds into the
-[error budget for stage groups](../stage_group_observability/index.md#error-budget). For this
-particular SLI, we have opted everyone out by default to give time to
-set the correct urgencies on endpoints before it affects a group's
-error budget.
-
-To include this SLI in the error budget, remove the `rails_requests`
-from the `ignored_components` array in the entry for your group. Read
-more about what is configurable in the
-[runbooks documentation](https://gitlab.com/gitlab-com/runbooks/-/tree/master/services#teamsyml).
+[error budget for stage groups](../stage_group_observability/index.md#error-budget).
For more information, read the epic for
[defining custom SLIs and incorporating them into error budgets](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525)).
@@ -252,7 +247,7 @@ request rates on the
In the **Budget Attribution** row, the **Puma Apdex** log link shows you
how many requests are not meeting a 1s or 5s target.
-Learn more about the content of the dashboard in the documentation for
+For more information about the content of the dashboard, see
[Dashboards for stage groups](../stage_group_observability/index.md). For more information
-on our exploration of the error budget itself, read the infrastructure issue
-[Stage group error budget exploration dashboard](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1365).
+about our exploration of the error budget itself, see
+[issue 1365](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1365).
diff --git a/doc/development/approval_rules.md b/doc/development/approval_rules.md
index f75cf35b32a..2e36be1231d 100644
--- a/doc/development/approval_rules.md
+++ b/doc/development/approval_rules.md
@@ -1,286 +1,11 @@
---
-stage: Create
-group: Code Review
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+redirect_to: 'merge_request_concepts/approval_rules.md'
+remove_date: '2023-04-23'
---
-# Approval Rules development guide
+This document was moved to [another location](merge_request_concepts/approval_rules.md).
-This document explains the backend design and flow of all related functionality
-about [merge request approval rules](../user/project/merge_requests/approvals/index.md).
-
-This should help contributors to understand the code design easier and to also
-help see if there are parts to improve as the feature and its implementation
-evolves.
-
-It's intentional that it doesn't contain too much implementation detail as they
-can change often. The code should explain those things better. The components
-mentioned here are the major parts of the application for the approval rules
-feature to work.
-
-NOTE:
-This is a living document and should be updated accordingly when parts
-of the codebase touched in this document are changed or removed, or when new components
-are added.
-
-## Data Model
-
-```mermaid
-erDiagram
- Project ||--o{ MergeRequest: " "
- Project ||--o{ ApprovalProjectRule: " "
- ApprovalProjectRule }o--o{ User: " "
- ApprovalProjectRule }o--o{ Group: " "
- ApprovalProjectRule }o--o{ ProtectedBranch: " "
- MergeRequest ||--|| ApprovalState: " "
- ApprovalState ||--o{ ApprovalWrappedRule: " "
- MergeRequest ||--o{ Approval: " "
- MergeRequest ||--o{ ApprovalMergeRequestRule: " "
- ApprovalMergeRequestRule }o--o{ User: " "
- ApprovalMergeRequestRule }o--o{ Group: " "
- ApprovalMergeRequestRule ||--o| ApprovalProjectRule: " "
-```
-
-### `Project` and `MergeRequest`
-
-`Project` and `MergeRequest` models are defined in `ee/app/models/ee/project.rb`
-and `ee/app/models/ee/merge_request.rb`. They extend the non-EE versions, because
-approval rules are an EE-only feature. Associations and other related stuff to
-merge request approvals are defined here.
-
-### `ApprovalState`
-
-```mermaid
-erDiagram
- MergeRequest ||--|| ApprovalState: " "
-```
-
-`ApprovalState` class is defined in `ee/app/models/approval_state.rb`. It's not
-an actual `ActiveRecord` model. This class encapsulates all logic related to the
-state of the approvals for a certain merge request like:
-
-- Knowing the approval rules that are applicable to the merge request based on
- its target branch.
-- Knowing the approval rules that are applicable to a certain target branch.
-- Checking if all rules were approved.
-- Checking if approval is required.
-- Knowing how many approvals were given or still required.
-
-It gets the approval rules data from the project (`ApprovalProjectRule`) or the
-merge request (`ApprovalMergeRequestRule`) and wrap it as `ApprovalWrappedRule`.
-
-### `ApprovalProjectRule`
-
-```mermaid
-erDiagram
- Project ||--o{ ApprovalProjectRule: " "
- ApprovalProjectRule }o--o{ User: " "
- ApprovalProjectRule }o--o{ Group: " "
- ApprovalProjectRule }o--o{ ProtectedBranch: " "
-```
-
-`ApprovalProjectRule` model is defined in `ee/app/models/approval_project_rule.rb`.
-
-A record is created/updated/deleted when an approval rule is added/edited/removed
-via project settings or the [project level approvals API](../api/merge_request_approvals.md#project-level-mr-approvals).
-The `ApprovalState` model get these records when approval rules are not
-overwritten.
-
-The `protected_branches` attribute is set and used when a rule is scoped to
-protected branches. See [Approvals for protected branches](../user/project/merge_requests/approvals/rules.md#approvals-for-protected-branches)
-for more information about the feature.
-
-### `ApprovalMergeRequestRule`
-
-```mermaid
-erDiagram
- MergeRequest ||--o{ ApprovalMergeRequestRule: " "
- ApprovalMergeRequestRule }o--o{ User: " "
- ApprovalMergeRequestRule }o--o{ Group: " "
- ApprovalMergeRequestRule ||--o| ApprovalProjectRule: " "
-```
-
-`ApprovalMergeRequestRule` model is defined in `ee/app/models/approval_merge_request_rule.rb`.
-
-A record is created/updated/deleted when a rule is added/edited/removed via merge
-request create/edit form or the [merge request level approvals API](../api/merge_request_approvals.md#merge-request-level-mr-approvals).
-
-The `approval_project_rule` is set when it is based from an existing `ApprovalProjectRule`.
-
-An `ApprovalMergeRequestRule` doesn't have `protected_branches` as it inherits
-them from the `approval_project_rule` if not overridden.
-
-### `ApprovalWrappedRule`
-
-```mermaid
-erDiagram
- ApprovalState ||--o{ ApprovalWrappedRule: " "
-```
-
-`ApprovalWrappedRule` is defined in `ee/app/modes/approval_wrapped_rule.rb` and
-is not an `ActiveRecord` model. It's used to wrap an `ApprovalProjectRule` or
-`ApprovalMergeRequestRule` for common interface. It also has the following sub
-types:
-
-- `ApprovalWrappedAnyApprovalRule` - for wrapping an `any_approver` rule.
-- `ApprovalWrappedCodeOwnerRule` - for wrapping a `code_owner` rule.
-
-This class delegates most of the responsibilities to the approval rule it wraps
-but it's also responsible for:
-
-- Checking if the approval rule is approved.
-- Knowing how many approvals were given or still required for the approval rule.
-
-It gets this information from the approval rule and the `Approval` records from
-the merge request.
-
-### `Approval`
-
-```mermaid
-erDiagram
- MergeRequest ||--o{ Approval: " "
-```
-
-`Approval` model is defined in `ee/app/models/approval.rb`. This model is
-responsible for storing information about an approval made on a merge request.
-Whenever an approval is given/revoked, a record is created/deleted.
-
-## Controllers and Services
-
-The following controllers and services below are being used for the approval
-rules feature to work.
-
-### `API::ProjectApprovalSettings`
-
-This private API is defined in `ee/lib/api/project_approval_settings.rb`.
-
-This is used for the following:
-
-- Listing the approval rules in project settings.
-- Creating/updating/deleting rules in project settings.
-- Listing the approval rules on create merge request form.
-
-### `Projects::MergeRequests::CreationsController`
-
-This controller is defined in `app/controllers/projects/merge_requests/creations_controller.rb`.
-
-The `create` action of this controller is used when create merge request form is
-submitted. It accepts the `approval_rules_attributes` parameter for creating/updating/deleting
-`ApprovalMergeRequestRule` records. It passes the parameter along when it executes
-`MergeRequests::CreateService`.
-
-### `Projects::MergeRequestsController`
-
-This controller is defined in `app/controllers/projects/merge_requests_controller.rb`.
-
-The `update` action of this controller is used when edit merge request form is
-submitted. It's like `Projects::MergeRequests::CreationsController` but it executes
-`MergeRequests::UpdateService` instead.
-
-### `API::MergeRequestApprovals`
-
-This API is defined in `ee/lib/api/merge_request_approvals.rb`.
-
-The [Approvals API endpoint](../api/merge_request_approvals.md#get-configuration-1)
-is requested when merge request page loads.
-
-The `/projects/:id/merge_requests/:merge_request_iid/approval_settings` is a
-private API endpoint used for the following:
-
-- Listing the approval rules on edit merge request form.
-- Listing the approval rules on the merge request page.
-
-When approving/unapproving MR via UI and API, the [Approve Merge Request](../api/merge_request_approvals.md#approve-merge-request)
-API endpoint and the [Unapprove Merge Request](../api/merge_request_approvals.md#unapprove-merge-request)
-API endpoint are requested. They execute `MergeRequests::ApprovalService` and
-`MergeRequests::RemoveApprovalService` accordingly.
-
-### `API::ProjectApprovalRules` and `API::MergeRequestApprovalRules`
-
-These APIs are defined in `ee/lib/api/project_approval_rules.rb` and
-`ee/lib/api/merge_request_approval_rules.rb`.
-
-Used to list/create/update/delete project and merge request level rules via
-[Merge request approvals API](../api/merge_request_approvals.md).
-
-Executes `ApprovalRules::CreateService`, `ApprovalRules::UpdateService`,
-`ApprovalRules::ProjectRuleDestroyService`, and `ApprovalRules::MergeRequestRuleDestroyService`
-accordingly.
-
-### `ApprovalRules::ParamsFilteringService`
-
-This service is defined in `ee/app/services/approval_rules/params_filtering_service.rb`.
-
-It is called only when `MergeRequests::CreateService` and
-`MergeRequests::UpdateService` are executed.
-
-It is responsible for parsing `approval_rules_attributes` parameter to:
-
-- Remove it when user can't update approval rules.
-- Filter the user IDs whether they are members of the project or not.
-- Filter the group IDs whether they are visible to user.
-- Identify the `any_approver` rule.
-- Append hidden groups to it when specified.
-- Append user defined inapplicable (rules that do not apply to the merge request's target
- branch) approval rules.
-
-## Flow
-
-These flowcharts should help explain the flow from the controllers down to the
-models for different functionalities.
-
-Some CRUD API endpoints are intentionally skipped because they are pretty
-straightforward.
-
-### Creating a merge request with approval rules via web UI
-
-```mermaid
-graph LR
- Projects::MergeRequests::CreationsController --> MergeRequests::CreateService
- MergeRequests::CreateService --> ApprovalRules::ParamsFilteringService
- ApprovalRules::ParamsFilteringService --> MergeRequests::CreateService
- MergeRequests::CreateService --> MergeRequest
- MergeRequest --> db[(Database)]
- MergeRequest --> User
- MergeRequest --> Group
- MergeRequest --> ApprovalProjectRule
- User --> db[(Database)]
- Group --> db[(Database)]
- ApprovalProjectRule --> db[(Database)]
-```
-
-When updating, same flow is followed but it starts at `Projects::MergeRequestsController`
-and executes `MergeRequests::UpdateService` instead.
-
-### Viewing the merge request approval rules on an MR page
-
-```mermaid
-graph LR
- API::MergeRequestApprovals --> MergeRequest
- MergeRequest --> ApprovalState
- ApprovalState --> id1{approval rules are overridden}
- id1{approval rules are overridden} --> |No| ApprovalProjectRule & ApprovalMergeRequestRule
- id1{approval rules are overridden} --> |Yes| ApprovalMergeRequestRule
- ApprovalState --> ApprovalWrappedRule
- ApprovalWrappedRule --> Approval
-```
-
-This flow gets initiated by the frontend component. The data returned is
-used to display information on the MR widget.
-
-### Approving a merge request
-
-```mermaid
-graph LR
- API::MergeRequestApprovals --> MergeRequests::ApprovalService
- MergeRequests::ApprovalService --> Approval
- Approval --> db[(Database)]
-```
-
-When unapproving, same flow is followed but the `MergeRequests::RemoveApprovalService`
-is executed instead.
-
-## TODO
-
-1. Add information related to other rule types, such as `code_owner` and `report_approver`.
-1. Add information about side effects of approving/unapproving merge request.
+<!-- This redirect file can be deleted after <2023-04-23>. -->
+<!-- Redirects that point to other docs in the same project expire in three months. -->
+<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
+<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/development/architecture.md b/doc/development/architecture.md
index cba868d3fed..96b70e2fbd8 100644
--- a/doc/development/architecture.md
+++ b/doc/development/architecture.md
@@ -18,7 +18,7 @@ GitLab is available under [different subscriptions](https://about.gitlab.com/pri
New versions of GitLab are released from stable branches, and the `main` branch is used for
bleeding-edge development.
-For more information, visit the [GitLab Release Process](https://about.gitlab.com/handbook/engineering/releases/).
+For more information, see the [GitLab release process](https://about.gitlab.com/handbook/engineering/releases/).
Both distributions require additional components. These components are described in the
[Component details](#components) section, and all have their own repositories.
@@ -34,8 +34,8 @@ Kubernetes platform. The largest known GitLab instance is on GitLab.com, which i
[official GitLab Helm chart](https://docs.gitlab.com/charts/) and the [official Linux package](https://about.gitlab.com/install/).
A typical installation uses NGINX or Apache as a web server to proxy through
-[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) and into the [Puma](https://puma.io)
-application server. GitLab serves web pages and the [GitLab API](../api/index.md) using the Puma
+[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse) and into the [Puma](https://puma.io)
+application server. GitLab serves web pages and the [GitLab API](../api/rest/index.md) using the Puma
application server. It uses Sidekiq as a job queue which, in turn, uses Redis as a non-persistent
database backend for job information, metadata, and incoming jobs.
@@ -378,6 +378,7 @@ Component statuses are linked to configuration documentation for each component.
| [Runner](#gitlab-runner) | Executes GitLab CI/CD jobs | ⤓ | ⤓ | ✅ | ⚙ | ✅ | ⚙ | ⚙ | CE & EE |
| [Sentry integration](#sentry) | Error tracking for deployed apps | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | CE & EE |
| [Sidekiq](#sidekiq) | Background jobs processor | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | CE & EE |
+| [Token Revocation API](sec/token_revocation_api.md) | Receives and revokes leaked secrets | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | EE Only |
### Component details
@@ -770,7 +771,8 @@ Whenever a client requests to pull or push an image from the registry, it
returns a `401` response along with a header detailing where to get an
authentication token, in this case the GitLab instance. The client then
requests a pull or push auth token from GitLab and retries the original request
-to the registry. Learn more about [token authentication](https://docs.docker.com/registry/spec/auth/token/).
+to the registry. For more information, see
+[token authentication](https://docs.docker.com/registry/spec/auth/token/).
An external registry can also be configured to use GitLab as an auth endpoint.
diff --git a/doc/development/bulk_import.md b/doc/development/bulk_import.md
index 92236594f8e..0d9c7348915 100644
--- a/doc/development/bulk_import.md
+++ b/doc/development/bulk_import.md
@@ -4,16 +4,16 @@ group: Import
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
-# GitLab Group Migration
+# Group migration by direct transfer
[Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2771) in GitLab 13.7.
WARNING:
This feature is [under construction](https://gitlab.com/groups/gitlab-org/-/epics/2771) and its API/Architecture might change in the future.
-GitLab Group Migration is the evolution of Project and Group Import functionality. The
-goal is to have an easier way to the user migrate a whole Group, including
-Projects, from one GitLab instance to another.
+[Group migration by direct transfer](../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended) is the
+evolution of migrating groups and projects using file exports. The goal is to have an easier way for the user to migrate a whole group,
+including projects, from one GitLab instance to another.
## Design decisions
@@ -24,7 +24,7 @@ works with a set of [ETL](#etl) Pipelines leveraging from the current [GitLab AP
![Simplified Component Overview](img/bulk_imports_overview_v13_7.png)
-### [ETL](https://www.ibm.com/cloud/learn/etl)
+### [ETL](https://www.ibm.com/topics/etl)
<!-- Direct quote from the IBM URL link -->
@@ -40,9 +40,9 @@ idea is to have one ETL pipeline for each relation to be imported.
The current [Project](../user/project/settings/import_export.md) and [Group](../user/group/settings/import_export.md) Import are file based, so they require an export
step to generate the file to be imported.
-GitLab Group migration leverages on [GitLab API](../api/index.md) to speed the migration.
+GitLab Group migration leverages on [GitLab API](../api/rest/index.md) to speed the migration.
-And, because we're on the road to [GraphQL](../api/index.md#graphql-api),
+And, because we're on the road to [GraphQL](../api/graphql/index.md),
GitLab Group Migration will be contributing towards to expand the GraphQL API coverage, which benefits both GitLab
and its users.
diff --git a/doc/development/cached_queries.md b/doc/development/cached_queries.md
index 1b590d68d18..0525603893f 100644
--- a/doc/development/cached_queries.md
+++ b/doc/development/cached_queries.md
@@ -65,8 +65,8 @@ to view the list of database queries, including cached queries. The
performance bar shows a warning when the number of total executed and cached queries is
greater than 100.
-To learn more about the statistics available to you, read the
-[Performance Bar documentation](../administration/monitoring/performance/performance_bar.md).
+For more information about the statistics available to you, see
+[Performance bar](../administration/monitoring/performance/performance_bar.md).
## What to look for
@@ -149,16 +149,16 @@ the following statistics:
- Total retained: 757595 bytes (6070 objects)
- `db_count`: 144
- `db_cached_count`: 55
-- `db_duration`: 303ms
+- `db_duration`: 303 ms
The fix reduced the allocated memory, and the number of cached queries. These
factors help improve the overall execution time:
-- Total allocated: 5313899 bytes (65290 objects), 1810KB (25%) less
-- Total retained: 685593 bytes (5278 objects), 72KB (9%) less
+- Total allocated: 5313899 bytes (65290 objects), 1810 KB (25%) less
+- Total retained: 685593 bytes (5278 objects), 72 KB (9%) less
- `db_count`: 95 (34% less)
- `db_cached_count`: 6 (89% less)
-- `db_duration`: 162ms (87% faster)
+- `db_duration`: 162 ms (87% faster)
## For more information
diff --git a/doc/development/caching.md b/doc/development/caching.md
index 58ec7a77591..9b3f9a4215e 100644
--- a/doc/development/caching.md
+++ b/doc/development/caching.md
@@ -22,11 +22,11 @@ A faster store for data, which is:
## What is fast?
-The goal for every web page should be to return in under 100ms:
+The goal for every web page should be to return in under 100 ms:
- This is achievable, but you need caching on a modern application.
- Larger responses take longer to build, and caching becomes critical to maintaining a constant speed.
-- Cache reads are typically sub-1ms. There is very little that this doesn't improve.
+- Cache reads are typically sub-1 ms. There is very little that this doesn't improve.
- It's no good only being fast on subsequent page loads, as the initial experience
is important too, so this isn't a complete solution.
- User-specific data makes this challenging, and presents the biggest challenge
@@ -219,7 +219,7 @@ Use conditional GET caching when the entire response is cacheable:
- Users and API libraries can ignore the cache.
- Sometimes Chrome does weird things with caches.
-- You will forget it exists in development mode and get angry when your changes aren't appearing.
+- You forget it exists in development mode and get angry when your changes aren't appearing.
- In theory using conditional GET caching makes sense everywhere, but in practice it can
sometimes cause odd issues.
diff --git a/doc/development/chaos_endpoints.md b/doc/development/chaos_endpoints.md
index 008c3700253..196ec14bffa 100644
--- a/doc/development/chaos_endpoints.md
+++ b/doc/development/chaos_endpoints.md
@@ -12,7 +12,7 @@ As [Werner Vogels](https://twitter.com/Werner), the CTO at Amazon Web Services,
<!-- vale gitlab.Spelling = NO -->
-As a developer, it's as important to consider the failure modes in which your software may operate as much as normal operation. Doing so can mean the difference between a minor hiccup leading to a scattering of `500` errors experienced by a tiny fraction of users, and a full site outage that affects all users for an extended period.
+As a developer, it's as important to consider the failure modes in which your software may operate as much as typical operation. Doing so can mean the difference between a minor hiccup leading to a scattering of `500` errors experienced by a tiny fraction of users, and a full site outage that affects all users for an extended period.
To paraphrase [Tolstoy](https://en.wikipedia.org/wiki/Anna_Karenina_principle), _all happy servers are alike, but all failing servers are failing in their own way_. Luckily, there are ways we can attempt to simulate these failure modes, and the chaos endpoints are tools for assisting in this process.
@@ -65,8 +65,8 @@ GET /-/chaos/leakmem?memory_mb=1024&duration_s=50&async=true
| Attribute | Type | Required | Description |
| ------------ | ------- | -------- | ------------------------------------------------------------------------------------ |
-| `memory_mb` | integer | no | How much memory, in MB, should be leaked. Defaults to 100MB. |
-| `duration_s` | integer | no | Minimum duration_s, in seconds, that the memory should be retained. Defaults to 30s. |
+| `memory_mb` | integer | no | How much memory, in MB, should be leaked. Defaults to 100 MB. |
+| `duration_s` | integer | no | Minimum duration_s, in seconds, that the memory should be retained. Defaults to 30 s. |
| `async` | boolean | no | Set to true to leak memory in a Sidekiq background worker process |
```shell
@@ -79,7 +79,7 @@ curl "http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10&token=s
This endpoint attempts to fully use a single core, at 100%, for the given period.
-Depending on your rack server setup, your request may timeout after a predetermined period (normally 60 seconds).
+Depending on your rack server setup, your request may timeout after a predetermined period (typically 60 seconds).
```plaintext
GET /-/chaos/cpu_spin
@@ -89,7 +89,7 @@ GET /-/chaos/cpu_spin?duration_s=50&async=true
| Attribute | Type | Required | Description |
| ------------ | ------- | -------- | --------------------------------------------------------------------- |
-| `duration_s` | integer | no | Duration, in seconds, that the core is used. Defaults to 30s |
+| `duration_s` | integer | no | Duration, in seconds, that the core is used. Defaults to 30 s |
| `async` | boolean | no | Set to true to consume CPU in a Sidekiq background worker process |
```shell
@@ -103,7 +103,7 @@ curl "http://localhost:3000/-/chaos/cpu_spin?duration_s=60&token=secret"
This endpoint attempts to fully use a single core, and interleave it with DB request, for the given period.
This endpoint can be used to model yielding execution to another threads when running concurrently.
-Depending on your rack server setup, your request may timeout after a predetermined period (normally 60 seconds).
+Depending on your rack server setup, your request may timeout after a predetermined period (typically 60 seconds).
```plaintext
GET /-/chaos/db_spin
@@ -113,8 +113,8 @@ GET /-/chaos/db_spin?duration_s=50&async=true
| Attribute | Type | Required | Description |
| ------------ | ------- | -------- | --------------------------------------------------------------------------- |
-| `interval_s` | float | no | Interval, in seconds, for every DB request. Defaults to 1s |
-| `duration_s` | integer | no | Duration, in seconds, that the core is used. Defaults to 30s |
+| `interval_s` | float | no | Interval, in seconds, for every DB request. Defaults to 1 s |
+| `duration_s` | integer | no | Duration, in seconds, that the core is used. Defaults to 30 s |
| `async` | boolean | no | Set to true to perform the operation in a Sidekiq background worker process |
```shell
@@ -137,7 +137,7 @@ GET /-/chaos/sleep?duration_s=50&async=true
| Attribute | Type | Required | Description |
| ------------ | ------- | -------- | ---------------------------------------------------------------------- |
-| `duration_s` | integer | no | Duration, in seconds, that the request sleeps for. Defaults to 30s |
+| `duration_s` | integer | no | Duration, in seconds, that the request sleeps for. Defaults to 30 s |
| `async` | boolean | no | Set to true to sleep in a Sidekiq background worker process |
```shell
@@ -170,7 +170,7 @@ curl "http://localhost:3000/-/chaos/kill?token=secret"
## Quit
This endpoint simulates the unexpected death of a worker process using the `QUIT` signal.
-Unlike `KILL`, the `QUIT` signal will also attempt to write a core dump.
+Unlike `KILL`, the `QUIT` signal also attempts to write a core dump.
See [core(5)](https://man7.org/linux/man-pages/man5/core.5.html) for more information.
```plaintext
@@ -191,7 +191,7 @@ curl "http://localhost:3000/-/chaos/quit?token=secret"
This endpoint triggers a GC run on the worker handling the request and returns its worker ID
plus GC stats as JSON. This is mostly useful when running Puma in standalone mode, since
-otherwise the worker handling the request will not be known upfront.
+otherwise the worker handling the request cannot be known upfront.
Endpoint:
diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md
index 2a60ca18169..641d0ea4f6f 100644
--- a/doc/development/cicd/index.md
+++ b/doc/development/cicd/index.md
@@ -69,7 +69,7 @@ looks for the next jobs to be transitioned towards completion. While doing that,
updates the status of jobs, stages and the overall pipeline.
On the right side of the diagram we have a list of [runners](../../ci/runners/index.md)
-connected to the GitLab instance. These can be shared runners, group runners, or project-specific runners.
+connected to the GitLab instance. These can be shared runners, group runners, or project runners.
The communication between runners and the Rails server occurs through a set of API endpoints, grouped as
the `Runner API Gateway`.
@@ -131,7 +131,7 @@ After the runner is [registered](https://docs.gitlab.com/runner/register/) using
- The type of runner it is registered as:
- a shared runner
- a group runner
- - a project specific runner
+ - a project runner
- Any associated tags.
The runner initiates the communication by requesting jobs to execute with `POST /api/v4/jobs/request`. Although polling happens every few seconds, we leverage caching through HTTP headers to reduce the server-side work load if the job queue doesn't change.
diff --git a/doc/development/code_review.md b/doc/development/code_review.md
index e194453565a..245fb2152cd 100644
--- a/doc/development/code_review.md
+++ b/doc/development/code_review.md
@@ -37,7 +37,7 @@ also to spread the workload.
For assistance with security scans or comments, include the Application Security Team (`@gitlab-com/gl-security/appsec`).
-The reviewers use the [reviewer functionality](../user/project/merge_requests/getting_started.md#reviewer) in the sidebar.
+The reviewers use the [reviewer functionality](../user/project/merge_requests/reviews/index.md) in the sidebar.
Reviewers can add their approval by [approving additionally](../user/project/merge_requests/approvals/index.md#approve-a-merge-request).
Depending on the areas your merge request touches, it must be **approved** by one
@@ -290,6 +290,23 @@ warrant a comment could be:
If there are any projects, snippets, or other assets that are required for a reviewer to validate the solution, ensure they have access to those assets before requesting review.
+When assigning reviewers, it can be helpful to:
+
+- Add a comment to the MR indicating which *type* of review you are looking for
+ from that reviewer.
+ - For example, if an MR changes a database query and updates
+ backend code, the MR author first needs a `~backend` review and a `~database`
+ review. While assigning the reviewers, the author adds a comment to the MR
+ letting each reviewer know which domain they should review.
+ - Many GitLab team members are domain experts in more than one area,
+ so without this type of comment it is sometimes ambiguous what type
+ of review they are being asked to provide.
+ - Explicitness around MR review types is efficient for the MR author because
+ they receive the type of review that they are looking for and it is
+ efficient for the MR reviewers because they immediately know which type of review to provide.
+ - [Example 1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75921#note_758161716)
+ - [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/109500#note_1253955051)
+
Avoid:
- Adding TODO comments (referenced above) directly to the source code unless the reviewer requires
@@ -459,7 +476,7 @@ first time.
### Requesting a review
When you are ready to have your merge request reviewed,
-you should [request an initial review](../user/project/merge_requests/getting_started.md#reviewer) by selecting a reviewer based on the [approval guidelines](#approval-guidelines).
+you should [request an initial review](../user/project/merge_requests/reviews/index.md) by selecting a reviewer based on the [approval guidelines](#approval-guidelines).
When a merge request has multiple areas for review, it is recommended you specify which area a reviewer should be reviewing, and at which stage (first or second).
This will help team members who qualify as a reviewer for multiple areas to know which area they're being requested to review.
@@ -522,8 +539,8 @@ Before taking the decision to merge:
- If the MR contains both Quality and non-Quality-related changes, the MR should be merged by the relevant maintainer for user-facing changes (backend, frontend, or database) after the Quality related changes are approved by a Software Engineer in Test.
At least one maintainer must approve an MR before it can be merged. MR authors and
-people who add commits to an MR are not authorized to approve the merge request,
-so they must seek a maintainer who has not contributed to the MR to approve the MR before it can be merged.
+people who add commits to an MR are not authorized to approve or merge the MR and
+must seek a maintainer who has not contributed to the MR to approve and merge it.
This policy is in place to satisfy the CHG-04 control of the GitLab
[Change Management Controls](https://about.gitlab.com/handbook/security/security-assurance/security-compliance/guidance/change-management.html).
@@ -701,11 +718,12 @@ Enterprise Edition instance. This has some implications:
cached value returns (say, from a string or nil to an array), change the
cache key at the same time.
1. **Settings** should be added as a
- [last resort](https://about.gitlab.com/handbook/product/#convention-over-configuration).
+ [last resort](https://about.gitlab.com/handbook/product/product-principles/#convention-over-configuration).
If you're adding a new setting in `gitlab.yml`:
1. Try to avoid that, and add to `ApplicationSetting` instead.
1. Ensure that it is also
[added to Omnibus](https://docs.gitlab.com/omnibus/settings/gitlab.yml#adding-a-new-setting-to-gitlabyml).
+ 1. Ensure that it is also [added to Charts](https://docs.gitlab.com/charts/development/style_guide.html), if needed.
1. **File system access** is not possible in a [cloud-native architecture](architecture.md#adapting-existing-and-introducing-new-components).
Ensure that we support object storage for any file storage we need to perform. For more
information, see the [uploads documentation](uploads/index.md).
diff --git a/doc/development/contributing/community_roles.md b/doc/development/contributing/community_roles.md
index 8aa219d72a1..3c9362138c2 100644
--- a/doc/development/contributing/community_roles.md
+++ b/doc/development/contributing/community_roles.md
@@ -1,18 +1,11 @@
---
-stage: none
-group: Development
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+redirect_to: 'index.md'
+remove_date: '2023-05-08'
---
-# Community members & roles
+This document was moved to [another location](index.md).
-GitLab community members and their privileges/responsibilities.
-
-| Roles | Responsibilities | Requirements |
-|-------|------------------|--------------|
-| Maintainer | Accepts merge requests on several GitLab projects | Added to the [team page](https://about.gitlab.com/company/team/). An expert on code reviews and knows the product/codebase |
-| Reviewer | Performs code reviews on MRs | Added to the [team page](https://about.gitlab.com/company/team/) |
-| Developer | Has access to GitLab internal infrastructure & issues (for example, HR-related) | GitLab employee or a Core Team member (with an NDA) |
-| Contributor | Can make contributions to all GitLab public projects | Have a GitLab.com account |
-
-[List of current reviewers/maintainers](https://about.gitlab.com/handbook/engineering/projects/#gitlab).
+<!-- This redirect file can be deleted after <2023-05-08>. -->
+<!-- Redirects that point to other docs in the same project expire in three months. -->
+<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
+<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md
index 1a5b801a95a..55827e00e43 100644
--- a/doc/development/contributing/index.md
+++ b/doc/development/contributing/index.md
@@ -23,63 +23,14 @@ GitLab comes in two flavors:
Throughout this guide you will see references to CE and EE for abbreviation.
-To get an overview of GitLab community membership, including those that would review or merge
-your contributions, visit [the community roles page](community_roles.md).
-
-## Security vulnerability disclosure
-
-Report suspected security vulnerabilities by following the
-[disclosure process on the GitLab.com website](https://about.gitlab.com/security/disclosure/).
-
-WARNING:
-Do **not** create publicly viewable issues for suspected security vulnerabilities.
-
## Code of conduct
We want to create a welcoming environment for everyone who is interested in contributing.
-Visit our [Code of Conduct page](https://about.gitlab.com/community/contribute/code-of-conduct/) to learn more about our commitment to an open and welcoming environment.
-
-## Closing policy for issues and merge requests
-
-GitLab is a popular open source project and the capacity to deal with issues
-and merge requests is limited. Out of respect for our volunteers, issues and
-merge requests not in line with the guidelines listed in this document may be
-closed without notice.
-
-Treat our volunteers with courtesy and respect, it will go a long way
-towards getting your issue resolved.
+For more information about our commitment to an open and welcoming environment, see our [Code of Conduct page](https://about.gitlab.com/community/contribute/code-of-conduct/).
Issues and merge requests should be in English and contain appropriate language
for audiences of all ages.
-If a contributor is no longer actively working on a submitted merge request,
-we can:
-
-- Decide that the merge request will be finished by one of our
- [Merge request coaches](https://about.gitlab.com/company/team/).
-- Close the merge request.
-
-We make this decision based on how important the change is for our product vision. If a merge
-request coach is going to finish the merge request, we assign the
-`~coach will finish` label.
-
-When a team member picks up a community contribution,
-we credit the original author by adding a changelog entry crediting the author
-and optionally include the original author on at least one of the commits
-within the MR.
-
-## Closing policy for inactive bugs
-
-GitLab values the time spent by contributors on reporting bugs. However, if a bug remains inactive for a very long period,
-it will qualify for auto-closure. Please refer to the [auto-close inactive bugs](https://about.gitlab.com/handbook/engineering/quality/triage-operations/#auto-close-inactive-bugs) section in our handbook to understand the complete workflow.
-
-## Helping others
-
-Help other GitLab users when you can.
-The methods people use to seek help can be found on the [getting help page](https://about.gitlab.com/get-help/).
-
-Sign up for the mailing list, answer GitLab questions on StackOverflow or respond in the IRC channel.
-
## How to contribute
If you would like to contribute to GitLab:
@@ -94,26 +45,6 @@ If you would like to contribute to GitLab:
could speed them up.
- Consult the [Contribution Flow](#contribution-flow) section to learn the process.
-### Communication channels
-
-If you have any questions or need help, visit [Getting Help](https://about.gitlab.com/get-help/) to learn how to
-communicate with the GitLab community. GitLab prefers [asynchronous communication](https://about.gitlab.com/handbook/communication/#internal-communication) over real-time communication.
-
-We do encourage you to connect and hang out with us. GitLab has a Gitter room dedicated for [contributors](https://gitter.im/gitlab/contributors), which is bridged with our
-internal Slack. We actively monitor this channel. There is also a community-run [Discord server](https://discord.gg/gitlab) where you can
-find other contributors in the `#contributors` channel.
-
-Thanks for your contribution!
-
-### GitLab Development Kit
-
-The GitLab Development Kit (GDK) helps contributors run a local GitLab instance with all the
-required dependencies. It can be used to test changes to GitLab and related projects before raising
-a Merge Request.
-
-For more information, see the [`gitlab-development-kit`](https://gitlab.com/gitlab-org/gitlab-development-kit)
-project.
-
### Contribution flow
The general flow of contributing to GitLab is:
@@ -194,7 +125,6 @@ If you are not sure who to mention, the reviewer will do this for you early in t
This [documentation](issue_workflow.md) outlines the current issue workflow:
-- [Issue tracker guidelines](issue_workflow.md#issue-tracker-guidelines)
- [Issue triaging](issue_workflow.md#issue-triaging)
- [Labels](issue_workflow.md#labels)
- [Feature proposals](issue_workflow.md#feature-proposals)
@@ -212,21 +142,17 @@ This [documentation](merge_request_workflow.md) outlines the current merge reque
- [Definition of done](merge_request_workflow.md#definition-of-done)
- [Dependencies](merge_request_workflow.md#dependencies)
-## Style guides
-
-This [documentation](style_guides.md) outlines the current style guidelines.
-
-## Implement design & UI elements
-
-This [design documentation](design.md) outlines the current process for implementing design and UI
-elements.
-
-## Contribute documentation
+## Closing policy for issues and merge requests
-For information on how to contribute documentation, see GitLab
-[documentation guidelines](../documentation/index.md).
+- For the criteria for closing issues, see [the Issue Triage handbook page](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#outdated-issues).
+- For the criteria for closing merge requests, see [the Merge Request Workflow](merge_request_workflow.md).
## Getting an Enterprise Edition License
If you need a license for contributing to an EE-feature, see
[relevant information](https://about.gitlab.com/handbook/marketing/community-relations/code-contributor-program/operations/#contributing-to-the-gitlab-enterprise-edition-ee).
+
+## Finding help
+
+- [Get help](https://about.gitlab.com/get-help/).
+- Join the community-run [Discord server](https://discord.com/invite/gitlab) and find other contributors in the `#contribute` channel.
diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md
index 9058eded2c7..b55fef25302 100644
--- a/doc/development/contributing/issue_workflow.md
+++ b/doc/development/contributing/issue_workflow.md
@@ -7,15 +7,19 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Issues workflow
-## Issue tracker guidelines
+**Before you submit an issue, [search the issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues)**
+for similar entries. Someone else might have already had the same bug or feature proposal.
+If you find an existing issue, show your support with an award emoji and add your notes to the discussion.
-**[Search the issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues)** for similar entries before
-submitting your own, there's a good chance somebody else had the same issue or
-feature proposal. Show your support with an award emoji and/or join the
-discussion.
+To submit a bug:
-Please submit bugs using the ['Bug' issue template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Bug.md) provided on the issue tracker.
-The text in the comments (`<!-- ... -->`) is there to help you with what to include.
+- Use the ['Bug' issue template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Bug.md).
+ The text in the comments (`<!-- ... -->`) should help you with which information to include.
+- To report a suspected security vulnerability, follow the
+ [disclosure process on the GitLab.com website](https://about.gitlab.com/security/disclosure/).
+
+WARNING:
+Do **not** create publicly viewable issues for suspected security vulnerabilities.
## Issue triaging
diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md
index ac3afa14b81..01bfdae5999 100644
--- a/doc/development/contributing/merge_request_workflow.md
+++ b/doc/development/contributing/merge_request_workflow.md
@@ -13,8 +13,23 @@ for community contributions have the [`Seeking community contributions`](issue_w
label, but you are free to contribute to any issue you want.
If an issue is marked for the current milestone at any time, even
-when you are working on it, a GitLab Inc. team member may take over the merge request
-to ensure the work is finished before the release date.
+when you are working on it, a GitLab team member may take over the merge request to ensure the work is finished before the release date.
+
+If a contributor is no longer actively working on a submitted merge request,
+we can:
+
+- Decide that the merge request will be finished by one of our
+ [Merge request coaches](https://about.gitlab.com/company/team/).
+- Close the merge request.
+
+We make this decision based on how important the change is for our product vision. If a merge
+request coach is going to finish the merge request, we assign the
+`~coach will finish` label.
+
+When a team member picks up a community contribution,
+we credit the original author by adding a changelog entry crediting the author
+and optionally include the original author on at least one of the commits
+within the MR.
If you want to add a new feature that is not labeled, it is best to first create
an issue (if there isn't one already) and leave a comment asking for it
@@ -117,7 +132,7 @@ Commit messages should follow the guidelines below, for reasons explained by Chr
- The commit subject or body must not contain Emojis.
- Commits that change 30 or more lines across at least 3 files should
describe these changes in the commit body.
-- Use issues and merge requests' full URLs instead of short references,
+- Use issues, milestones, and merge requests' full URLs instead of short references,
as they are displayed as plain text outside of GitLab.
- The merge request should not contain more than 10 commit messages.
- The commit subject should contain at least 3 words.
@@ -213,7 +228,7 @@ To reach the definition of done, the merge request must create no regressions an
- Verified as working in production on GitLab.com.
- Verified as working for self-managed instances.
-- Verified as supporting [Geo](../../administration/geo/index.md) via the [self-service framework](../geo/framework.md). To learn more see [here](../geo/framework.md#geo-is-a-requirement-in-the-definition-of-done).
+- Verified as supporting [Geo](../../administration/geo/index.md) through the [self-service framework](../geo/framework.md). For more information, see [Geo is a requirement in the definition of done](../geo/framework.md#geo-is-a-requirement-in-the-definition-of-done).
If a regression occurs, we prefer you revert the change.
Your contribution is *incomplete* until you have made sure it meets all of these
@@ -224,8 +239,8 @@ requirements.
1. Working and clean code that is commented where needed.
1. The change is evaluated to [limit the impact of far-reaching work](https://about.gitlab.com/handbook/engineering/development/#reducing-the-impact-of-far-reaching-work).
1. [Performance guidelines](../merge_request_concepts/performance.md) have been followed.
-1. [Secure coding guidelines](https://gitlab.com/gitlab-com/gl-security/security-guidelines) have been followed.
-1. [Application and rate limit guidelines](../merge_request_application_and_rate_limit_guidelines.md) have been followed.
+1. [Secure coding guidelines](../secure_coding_guidelines.md) have been followed.
+1. [Application and rate limit guidelines](../merge_request_concepts/rate_limits.md) have been followed.
1. [Documented](../documentation/index.md) in the `/doc` directory.
1. If your MR touches code that executes shell commands, reads or opens files, or
handles paths to files on disk, make sure it adheres to the
diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md
index 0a030d1f3bc..28ce8e6ff4b 100644
--- a/doc/development/contributing/style_guides.md
+++ b/doc/development/contributing/style_guides.md
@@ -9,15 +9,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w
## Editor/IDE styling standardization
-We use [EditorConfig](https://editorconfig.org/) to automatically apply certain styling
-standards before files are saved locally. Most editors/IDEs will honor the `.editorconfig`
-settings automatically by default. If your editor/IDE does not automatically support `.editorconfig`,
-we suggest investigating to see if a plugin exists. For instance here is the
+We use [EditorConfig](https://editorconfig.org/) to automatically apply certain styling standards before files are saved
+locally. Some editors and IDEs honor the `.editorconfig` settings [automatically by default](https://editorconfig.org/#pre-installed).
+
+If your editor or IDE does not automatically support `.editorconfig`, we suggest investigating to
+[see if a plugin exists](https://editorconfig.org/#download). For example, a
[plugin for vim](https://github.com/editorconfig/editorconfig-vim).
## Pre-push static analysis with Lefthook
-[Lefthook](https://github.com/Arkweid/lefthook) is a Git hooks manager that allows
+[Lefthook](https://github.com/evilmartians/lefthook) is a Git hooks manager that allows
custom logic to be executed prior to Git committing or pushing. GitLab comes with
Lefthook configuration (`lefthook.yml`), but it must be installed.
@@ -54,7 +55,7 @@ This should return the Lefthook version and the list of executable commands with
Lefthook is configured with a combination of:
- Project configuration in [`lefthook.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lefthook.yml).
-- Any [local configuration](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#local-config).
+- Any [local configuration](https://github.com/evilmartians/lefthook/blob/master/README.md#local-config).
### Disable Lefthook temporarily
@@ -72,7 +73,7 @@ To run the `pre-push` Git hook, run:
bundle exec lefthook run pre-push
```
-For more information, check out [Lefthook documentation](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#run-githook-group-directly).
+For more information, check out [Lefthook documentation](https://github.com/evilmartians/lefthook/blob/master/README.md#direct-control).
### Skip Lefthook checks per tag
@@ -91,7 +92,7 @@ pre-push:
- documentation
```
-For more information, check out [Lefthook documentation](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#skip-some-tags-on-the-fly).
+For more information, check out [Lefthook documentation](https://github.com/evilmartians/lefthook/blob/master/docs/configuration.md#exclude_tags).
### Skip or enable a specific Lefthook check
@@ -106,7 +107,7 @@ pre-push:
skip: false
```
-For more information, check out [Lefthook documentation Skipping commands section](https://github.com/evilmartians/lefthook/blob/master/docs/full_guide.md#skipping-commands).
+For more information, check out [Lefthook documentation Skipping commands section](https://github.com/evilmartians/lefthook/blob/master/docs/configuration.md#skip).
## Database migrations
@@ -150,8 +151,6 @@ See the dedicated [Documentation Style Guide](../documentation/styleguide/index.
### Guidelines for good practices
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36576/) in GitLab 13.2 as GitLab Development documentation.
-
*Good practice* examples demonstrate encouraged ways of writing code while
comparing with examples of practices to avoid. These examples are labeled as
*Bad* or *Good*. In GitLab development guidelines, when presenting the cases,
diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md
index 07fa8133496..2c2999e69d6 100644
--- a/doc/development/database/add_foreign_key_to_existing_column.md
+++ b/doc/development/database/add_foreign_key_to_existing_column.md
@@ -143,3 +143,118 @@ class ValidateForeignKeyOnEmailUsers < Gitlab::Database::Migration[2.1]
end
end
```
+
+### Validate the foreign key asynchronously
+
+For very large tables, foreign key validation can be a challenge to manage when
+it runs for many hours. Necessary database operations like `autovacuum` cannot
+run, and on GitLab.com, the deployment process is blocked waiting for the
+migrations to finish.
+
+To limit impact on GitLab.com, a process exists to validate them asynchronously
+during weekend hours. Due to generally lower traffic and fewer deployments,
+FK validation can proceed at a lower level of risk.
+
+### Schedule foreign key validation for a low-impact time
+
+1. [Schedule the FK to be validated](#schedule-the-fk-to-be-validated).
+1. [Verify the MR was deployed and the FK is valid in production](#verify-the-mr-was-deployed-and-the-fk-is-valid-in-production).
+1. [Add a migration to validate the FK synchronously](#add-a-migration-to-validate-the-fk-synchronously).
+
+### Schedule the FK to be validated
+
+1. Create a merge request containing a post-deployment migration, which prepares
+ the foreign key for asynchronous validation.
+1. Create a follow-up issue to add a migration that validates the foreign key
+ synchronously.
+1. In the merge request that prepares the asynchronous foreign key, add a
+ comment mentioning the follow-up issue.
+
+An example of validating the foreign key using the asynchronous helpers can be
+seen in the block below. This migration enters the foreign key name into the
+`postgres_async_foreign_key_validations` table. The process that runs on
+weekends pulls foreign keys from this table and attempts to validate them.
+
+```ruby
+# in db/post_migrate/
+
+FK_NAME = :fk_be5624bf37
+
+# TODO: FK to be validated synchronously in issue or merge request
+def up
+ # `some_column` can be an array of columns, and is not mandatory if `name` is supplied.
+ # `name` takes precedence over other arguments.
+ prepare_async_foreign_key_validation :ci_builds, :some_column, name: FK_NAME
+
+ # Or in case of partitioned tables, use:
+ prepare_partitioned_async_foreign_key_validation :p_ci_builds, :some_column, name: FK_NAME
+end
+
+def down
+ unprepare_async_foreign_key_validation :ci_builds, :some_column, name: FK_NAME
+
+ # Or in case of partitioned tables, use:
+ unprepare_partitioned_async_foreign_key_validation :p_ci_builds, :some_column, name: FK_NAME
+end
+```
+
+### Verify the MR was deployed and the FK is valid in production
+
+1. Verify that the post-deploy migration was executed on GitLab.com using ChatOps with
+ `/chatops run auto_deploy status <merge_sha>`. If the output returns `db/gprd`,
+ the post-deploy migration has been executed in the production database. For more information, see
+ [How to determine if a post-deploy migration has been executed on GitLab.com](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/post_deploy_migration/readme.md#how-to-determine-if-a-post-deploy-migration-has-been-executed-on-gitlabcom).
+1. Wait until the next week so that the FK can be validated over a weekend.
+1. Use [Database Lab](database_lab.md) to check if validation was successful.
+ Ensure the output does not indicate the foreign key is `NOT VALID`.
+
+### Add a migration to validate the FK synchronously
+
+After the foreign key is valid on the production database, create a second
+merge request that validates the foreign key synchronously. The schema changes
+must be updated and committed to `structure.sql` in this second merge request.
+The synchronous migration results in a no-op on GitLab.com, but you should still
+add the migration as expected for other installations. The below block
+demonstrates how to create the second migration for the previous
+asynchronous example.
+
+WARNING:
+Verify that the foreign key is valid in production before merging a second
+migration with `validate_foreign_key`. If the second migration is deployed
+before the validation has been executed, the foreign key is validated
+synchronously when the second migration executes.
+
+```ruby
+# in db/post_migrate/
+
+ FK_NAME = :fk_be5624bf37
+
+ def up
+ validate_foreign_key :ci_builds, :some_column, name: FK_NAME
+ end
+
+ def down
+ # Can be safely a no-op if we don't roll back the inconsistent data.
+ end
+end
+
+```
+
+## Test database FK changes locally
+
+You must test the database foreign key changes locally before creating a merge request.
+
+### Verify the foreign keys validated asynchronously
+
+Use the asynchronous helpers on your local environment to test changes for
+validating a foreign key:
+
+1. Enable the feature flags by running `Feature.enable(:database_async_foreign_key_validation)`
+ and `Feature.enable(:database_reindexing)` in the Rails console.
+1. Run `bundle exec rails db:migrate` so that it creates an entry in the async validation table.
+1. Run `bundle exec rails gitlab:db:reindex` so that the FK is validated asynchronously.
+1. To verify the foreign key, open the PostgreSQL console using the
+ [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md)
+ command `gdk psql` and run the command `\d+ table_name` to check that your
+ foreign key is valid. A successful validation removes `NOT VALID` from
+ the foreign key definition.
diff --git a/doc/development/database/adding_database_indexes.md b/doc/development/database/adding_database_indexes.md
index e1d5a7af6d9..1e3a1de9b69 100644
--- a/doc/development/database/adding_database_indexes.md
+++ b/doc/development/database/adding_database_indexes.md
@@ -214,6 +214,45 @@ def down
end
```
+## Analyzing a new index before a batched background migration
+
+Sometimes it is necessary to add an index to support a [batched background migration](batched_background_migrations.md).
+It is commonly done by creating two [post deployment migrations](post_deployment_migrations.md):
+
+1. Add the new index, often a [temporary index](#temporary-indexes).
+1. [Queue the batched background migration](batched_background_migrations.md#queueing).
+
+In most cases, no additional work is needed. The new index is created and is used
+as expected when queuing and executing the batched background migration.
+
+[Expression indexes](https://www.postgresql.org/docs/current/indexes-expressional.html),
+however, do not generate statistics for the new index on creation. Autovacuum
+eventually runs `ANALYZE`, and updates the statistics so the new index is used.
+Run `ANALYZE` explicitly only if it is needed right after the index
+is created, such as in the background migration scenario described above.
+
+To trigger `ANALYZE` after the index is created, update the index creation migration
+to analyze the table:
+
+```ruby
+# in db/post_migrate/
+
+INDEX_NAME = 'tmp_index_projects_on_owner_and_lower_name_where_emails_disabled'
+TABLE = :projects
+
+disable_ddl_transaction!
+
+def up
+ add_concurrent_index TABLE, '(creator_id, lower(name))', where: 'emails_disabled = false', name: INDEX_NAME
+
+ connection.execute("ANALYZE #{TABLE}")
+end
+```
+
+`ANALYZE` should only be run in post deployment migrations and should not target
+[large tables](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3).
+If this behavior is needed on a larger table, ask for assistance in the `#database` Slack channel.
+
## Indexes for partitioned tables
Indexes [cannot be created](https://www.postgresql.org/docs/15/ddl-partitioning.html#DDL-PARTITIONING-DECLARATIVE-MAINTENANCE)
@@ -254,7 +293,7 @@ end
For very large tables, index creation can be a challenge to manage.
While `add_concurrent_index` creates indexes in a way that does not block
-normal traffic, it can still be problematic when index creation runs for
+ordinary traffic, it can still be problematic when index creation runs for
many hours. Necessary database operations like `autovacuum` cannot run, and
on GitLab.com, the deployment process is blocked waiting for index
creation to finish.
@@ -271,8 +310,13 @@ index creation can proceed at a lower level of risk.
### Schedule the index to be created
-Create an MR with a post-deployment migration which prepares the index
-for asynchronous creation. An example of creating an index using
+1. Create a merge request containing a post-deployment migration, which prepares
+ the index for asynchronous creation.
+1. [Create a follow-up issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Synchronous%20Database%20Index)
+ to add a migration that creates the index synchronously.
+1. In the merge request that prepares the asynchronous index, add a comment mentioning the follow-up issue.
+
+An example of creating an index using
the asynchronous index helpers can be seen in the block below. This migration
enters the index name and definition into the `postgres_async_indexes`
table. The process that runs on weekends pulls indexes from this
@@ -283,6 +327,7 @@ table and attempt to create them.
INDEX_NAME = 'index_ci_builds_on_some_column'
+# TODO: Index to be created synchronously in https://gitlab.com/gitlab-org/gitlab/-/issues/XXXXX
def up
prepare_async_index :ci_builds, :some_column, name: INDEX_NAME
end
@@ -351,7 +396,7 @@ Use the asynchronous index helpers on your local environment to test changes for
For very large tables, index destruction can be a challenge to manage.
While `remove_concurrent_index` removes indexes in a way that does not block
-normal traffic, it can still be problematic if index destruction runs for
+ordinary traffic, it can still be problematic if index destruction runs for
during `autovacuum`. Necessary database operations like `autovacuum` cannot run, and
the deployment process on GitLab.com is blocked while waiting for index
destruction to finish.
@@ -366,8 +411,13 @@ index destruction can proceed at a lower level of risk.
### Schedule the index to be removed
-Create an MR with a post-deployment migration which prepares the index
-for asynchronous destruction. For example. to destroy an index using
+1. Create a merge request containing a post-deployment migration, which prepares
+ the index for asynchronous destruction.
+1. [Create a follow-up issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Synchronous%20Database%20Index)
+ to add a migration that destroys the index synchronously.
+1. In the merge request that prepares the asynchronous index removal, add a comment mentioning the follow-up issue.
+
+For example, to destroy an index using
the asynchronous index helpers:
```ruby
@@ -375,6 +425,7 @@ the asynchronous index helpers:
INDEX_NAME = 'index_ci_builds_on_some_column'
+# TODO: Index to be destroyed synchronously in https://gitlab.com/gitlab-org/gitlab/-/issues/XXXXX
def up
prepare_async_index_removal :ci_builds, :some_column, name: INDEX_NAME
end
diff --git a/doc/development/database/avoiding_downtime_in_migrations.md b/doc/development/database/avoiding_downtime_in_migrations.md
index bb6e13eff53..8e1eeee7a42 100644
--- a/doc/development/database/avoiding_downtime_in_migrations.md
+++ b/doc/development/database/avoiding_downtime_in_migrations.md
@@ -6,20 +6,20 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Avoiding downtime in migrations
-When working with a database certain operations may require downtime. Since we
+When working with a database certain operations may require downtime. As we
cannot have downtime in migrations we need to use a set of steps to get the
same end result without downtime. This guide describes various operations that
may appear to need downtime, their impact, and how to perform them without
requiring downtime.
-## Dropping Columns
+## Dropping columns
Removing columns is tricky because running GitLab processes may still be using
the columns. To work around this safely, you need three steps in three releases:
-1. Ignoring the column (release M)
-1. Dropping the column (release M+1)
-1. Removing the ignore rule (release M+2)
+1. [Ignoring the column](#ignoring-the-column-release-m) (release M)
+1. [Dropping the column](#dropping-the-column-release-m1) (release M+1)
+1. [Removing the ignore rule](#removing-the-ignore-rule-release-m2) (release M+2)
The reason we spread this out across three releases is that dropping a column is
a destructive operation that can't be rolled back easily.
@@ -27,9 +27,9 @@ a destructive operation that can't be rolled back easily.
Following this procedure helps us to make sure there are no deployments to GitLab.com
and upgrade processes for self-managed installations that lump together any of these steps.
-### Step 1: Ignoring the column (release M)
+### Ignoring the column (release M)
-The first step is to ignore the column in the application code. This is
+The first step is to ignore the column in the application code. This step is
necessary because Rails caches the columns and re-uses this cache in various
places. This can be done by defining the columns to ignore. For example, to ignore
`updated_at` in the User model you'd use the following:
@@ -50,7 +50,7 @@ ignore_columns %i[updated_at created_at], remove_with: '12.7', remove_after: '20
If the model exists in CE and EE, the column has to be ignored in the CE model. If the
model only exists in EE, then it has to be added there.
-We require indication of when it is safe to remove the column ignore with:
+We require indication of when it is safe to remove the column ignore rule with:
- `remove_with`: set to a GitLab release typically two releases (M+2) after adding the
column ignore.
@@ -64,7 +64,7 @@ to ignore the column and subsequently remove the column ignore (which would resu
In this example, the change to ignore the column went into release 12.5.
-### Step 2: Dropping the column (release M+1)
+### Dropping the column (release M+1)
Continuing our example, dropping the column goes into a _post-deployment_ migration in release 12.6:
@@ -74,12 +74,14 @@ Start by creating the **post-deployment migration**:
bundle exec rails g post_deployment_migration remove_users_updated_at_column
```
-There are two scenarios that you need to consider
-to write a migration that removes a column:
+You must consider these scenarios when you write a migration that removes a column:
-#### A. The removed column has no indexes or constraints that belong to it
+- [The removed column has no indexes or constraints that belong to it](#the-removed-column-has-no-indexes-or-constraints-that-belong-to-it)
+- [The removed column has an index or constraint that belongs to it](#the-removed-column-has-an-index-or-constraint-that-belongs-to-it)
-In this case, a **transactional migration** can be used. Something as simple as:
+#### The removed column has no indexes or constraints that belong to it
+
+In this case, a **transactional migration** can be used:
```ruby
class RemoveUsersUpdatedAtColumn < Gitlab::Database::Migration[2.1]
@@ -97,10 +99,10 @@ You can consider [enabling lock retries](../migration_style_guide.md#usage-with-
when you run a migration on big tables, because it might take some time to
acquire a lock on this table.
-#### B. The removed column has an index or constraint that belongs to it
+#### The removed column has an index or constraint that belongs to it
If the `down` method requires adding back any dropped indexes or constraints, that cannot
-be done within a transactional migration, then the migration would look like this:
+be done in a transactional migration. The migration would look like this:
```ruby
class RemoveUsersUpdatedAtColumn < Gitlab::Database::Migration[2.1]
@@ -131,7 +133,7 @@ is used to disable the transaction that wraps the whole migration.
You can refer to the page [Migration Style Guide](../migration_style_guide.md)
for more information about database migrations.
-### Step 3: Removing the ignore rule (release M+2)
+### Removing the ignore rule (release M+2)
With the next release, in this example 12.7, we set up another merge request to remove the ignore rule.
This removes the `ignore_column` line and - if not needed anymore - also the inclusion of `IgnoreableColumns`.
@@ -139,18 +141,24 @@ This removes the `ignore_column` line and - if not needed anymore - also the inc
This should only get merged with the release indicated with `remove_with` and once
the `remove_after` date has passed.
-## Renaming Columns
+## Renaming columns
-Renaming columns the normal way requires downtime as an application may continue
+Renaming columns the standard way requires downtime as an application may continue
to use the old column names during or after a database migration. To rename a column
without requiring downtime, we need two migrations: a regular migration and a
post-deployment migration. Both these migrations can go in the same release.
+The steps:
+
+1. [Add the regular migration](#add-the-regular-migration-release-m) (release M)
+1. [Ignore the column](#ignore-the-column-release-m) (release M)
+1. [Add a post-deployment migration](#add-a-post-deployment-migration-release-m) (release M)
+1. [Remove the ignore rule](#remove-the-ignore-rule-release-m1) (release M+1)
NOTE:
It's not possible to rename columns with default values. For more details, see
[this merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52032#default-values).
-### Step 1: Add The Regular Migration
+### Add the regular migration (release M)
First we need to create the regular migration. This migration should use
`Gitlab::Database::MigrationHelpers#rename_column_concurrently` to perform the
@@ -178,7 +186,20 @@ If a column contains one or more indexes that don't contain the name of the
original column, the previously described procedure fails. In that case,
you need to rename these indexes.
-### Step 2: Add A Post-Deployment Migration
+### Ignore the column (release M)
+
+The next step is to ignore the column in the application code, and make sure it is not used. This step is
+necessary because Rails caches the columns and re-uses this cache in various places.
+This step is similar to [the first step when column is dropped](#ignoring-the-column-release-m), and the same requirements apply.
+
+```ruby
+class User < ApplicationRecord
+ include IgnorableColumns
+ ignore_column :updated_at, remove_with: '12.7', remove_after: '2020-01-22'
+end
+```
+
+### Add a post-deployment migration (release M)
The renaming procedure requires some cleaning up in a post-deployment migration.
We can perform this cleanup using
@@ -202,7 +223,11 @@ end
If you're renaming a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3), carefully consider the state when the first migration has run but the second cleanup migration hasn't been run yet.
With [Canary](https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/library/canary/) it is possible that the system runs in this state for a significant amount of time.
-## Changing Column Constraints
+### Remove the ignore rule (release M+1)
+
+Same as when column is dropped, after the rename is completed, we need to [remove the ignore rule](#removing-the-ignore-rule-release-m2) in a subsequent release.
+
+## Changing column constraints
Adding or removing a `NOT NULL` clause (or another constraint) can typically be
done without requiring downtime. However, this does require that any application
@@ -218,14 +243,18 @@ You can check the following guides for each specific use case:
- [Adding `NOT NULL` constraints](not_null_constraints.md)
- [Adding limits to text columns](strings_and_the_text_data_type.md)
-## Changing Column Types
+## Changing column types
Changing the type of a column can be done using
`Gitlab::Database::MigrationHelpers#change_column_type_concurrently`. This
method works similarly to `rename_column_concurrently`. For example, let's say
-we want to change the type of `users.username` from `string` to `text`.
+we want to change the type of `users.username` from `string` to `text`:
+
+1. [Create a regular migration](#create-a-regular-migration)
+1. [Create a post-deployment migration](#create-a-post-deployment-migration)
+1. [Casting data to a new type](#casting-data-to-a-new-type)
-### Step 1: Create A Regular Migration
+### Create a regular migration
A regular migration is used to create a new column with a temporary name along
with setting up some triggers to keep data in sync. Such a migration would look
@@ -246,7 +275,7 @@ class ChangeUsersUsernameStringToText < Gitlab::Database::Migration[2.1]
end
```
-### Step 2: Create A Post Deployment Migration
+### Create a post-deployment migration
Next we need to clean up our changes using a post-deployment migration:
@@ -293,13 +322,13 @@ specify the old default.
Doing this requires steps in two minor releases:
-1. Add the `SafelyChangeColumnDefault` concern to the model and change the default in a post-migration.
-1. Clean up the `SafelyChangeColumnDefault` concern in the next minor release.
+1. [Add the `SafelyChangeColumnDefault` concern to the model](#add-the-safelychangecolumndefault-concern-to-the-model-and-change-the-default-in-a-post-migration) and change the default in a post-migration.
+1. [Clean up the `SafelyChangeColumnDefault` concern](#clean-up-the-safelychangecolumndefault-concern-in-the-next-minor-release) in the next minor release.
We must wait a minor release before cleaning up the `SafelyChangeColumnDefault` because self-managed
releases bundle an entire minor release into a single zero-downtime deployment.
-### Step 1: Add the `SafelyChangeColumnDefault` concern to the model and change the default in a post-migration
+### Add the `SafelyChangeColumnDefault` concern to the model and change the default in a post-migration
The first step is to mark the column as safe to change in application code.
@@ -333,12 +362,12 @@ You can consider [enabling lock retries](../migration_style_guide.md#usage-with-
when you run a migration on big tables, because it might take some time to
acquire a lock on this table.
-### Step 2: Clean up the `SafelyChangeColumnDefault` concern in the next minor release
+### Clean up the `SafelyChangeColumnDefault` concern in the next minor release
In the next minor release, create a new merge request to remove the `columns_changing_default` call. Also remove the `SafelyChangeColumnDefault` include
if it is not needed for a different column.
-## Changing The Schema For Large Tables
+## Changing the schema for large tables
While `change_column_type_concurrently` and `rename_column_concurrently` can be
used for changing the schema of a table without downtime, it doesn't work very
@@ -354,7 +383,7 @@ down deployments.
For more information, see [the documentation on cleaning up batched background migrations](batched_background_migrations.md#cleaning-up).
-## Adding Indexes
+## Adding indexes
Adding indexes does not require downtime when `add_concurrent_index`
is used.
@@ -362,15 +391,15 @@ is used.
See also [Migration Style Guide](../migration_style_guide.md#adding-indexes)
for more information.
-## Dropping Indexes
+## Dropping indexes
Dropping an index does not require downtime.
-## Adding Tables
+## Adding tables
This operation is safe as there's no code using the table just yet.
-## Dropping Tables
+## Dropping tables
Dropping tables can be done safely using a post-deployment migration, but only
if the application no longer uses the table.
@@ -378,7 +407,7 @@ if the application no longer uses the table.
Add the table to [`db/docs/deleted_tables`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/db/docs/deleted_tables) using the process described in [database dictionary](database_dictionary.md#dropping-tables).
Even though the table is deleted, it is still referenced in database migrations.
-## Renaming Tables
+## Renaming tables
Renaming tables requires downtime as an application may continue
using the old table name during/after a database migration.
@@ -389,7 +418,7 @@ table and creating a new one is the preferred way to "rename" the table.
Renaming a table is possible without downtime by following our multi-release
[rename table process](rename_database_tables.md#rename-table-without-downtime).
-## Adding Foreign Keys
+## Adding foreign keys
Adding foreign keys usually works in 3 steps:
@@ -404,7 +433,7 @@ GitLab allows you to work around this by using
`Gitlab::Database::MigrationHelpers#add_concurrent_foreign_key`. This method
ensures that no downtime is needed.
-## Removing Foreign Keys
+## Removing foreign keys
This operation does not require downtime.
@@ -418,14 +447,16 @@ without downtime and causing too much load on the database is described below.
To start the process, add a regular migration to create the new `bigint` columns. Use the provided
`initialize_conversion_of_integer_to_bigint` helper. The helper also creates a database trigger
-to keep in sync both columns for any new records ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/migrate/20210608072312_initialize_conversion_of_ci_stages_to_bigint.rb)):
+to keep in sync both columns for any new records ([code](https://gitlab.com/gitlab-org/gitlab/-/blob/97aee76c4bfc2043dc0a1ef9ffbb71c58e0e2857/db/migrate/20230127093353_initialize_conversion_of_merge_request_metrics_to_bigint.rb)):
```ruby
-class InitializeConversionOfCiStagesToBigint < ActiveRecord::Migration[6.1]
- include Gitlab::Database::MigrationHelpers
+# frozen_string_literal: true
+
+class InitializeConversionOfMergeRequestMetricsToBigint < Gitlab::Database::Migration[2.1]
+ disable_ddl_transaction!
- TABLE = :ci_stages
- COLUMNS = %i(id)
+ TABLE = :merge_request_metrics
+ COLUMNS = %i[id]
def up
initialize_conversion_of_integer_to_bigint(TABLE, COLUMNS)
@@ -440,29 +471,28 @@ end
Ignore the new `bigint` columns:
```ruby
-module Ci
- class Stage < Ci::ApplicationRecord
- include IgnorableColumns
- ignore_column :id_convert_to_bigint, remove_with: '14.2', remove_after: '2021-08-22'
- end
+# frozen_string_literal: true
+
+class MergeRequest::Metrics < ApplicationRecord
+ include IgnorableColumns
+ ignore_column :id_convert_to_bigint, remove_with: '16.0', remove_after: '2023-05-22'
+end
```
-To migrate existing data, we introduced new type of _batched background migrations_.
-Unlike the classic background migrations, built on top of Sidekiq, batched background migrations
-don't have to enqueue and schedule all the background jobs at the beginning.
-They also have other advantages, like automatic tuning of the batch size, better progress visibility,
-and collecting metrics. To start the process, use the provided `backfill_conversion_of_integer_to_bigint`
-helper ([example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/migrate/20210608072346_backfill_ci_stages_for_bigint_conversion.rb)):
+Enqueue batched background migration ([code](https://gitlab.com/gitlab-org/gitlab/-/blob/97aee76c4bfc2043dc0a1ef9ffbb71c58e0e2857/db/post_migrate/20230127101834_backfill_merge_request_metrics_for_bigint_conversion.rb))
+to migrate the existing data:
```ruby
-class BackfillCiStagesForBigintConversion < ActiveRecord::Migration[6.1]
- include Gitlab::Database::MigrationHelpers
+# frozen_string_literal: true
+
+class BackfillMergeRequestMetricsForBigintConversion < Gitlab::Database::Migration[2.1]
+ restrict_gitlab_migration gitlab_schema: :gitlab_main
- TABLE = :ci_stages
- COLUMNS = %i(id)
+ TABLE = :merge_request_metrics
+ COLUMNS = %i[id]
def up
- backfill_conversion_of_integer_to_bigint(TABLE, COLUMNS)
+ backfill_conversion_of_integer_to_bigint(TABLE, COLUMNS, sub_batch_size: 200)
end
def down
diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md
index 88fdfab9828..c6fe6d16faf 100644
--- a/doc/development/database/batched_background_migrations.md
+++ b/doc/development/database/batched_background_migrations.md
@@ -309,7 +309,7 @@ In the second (filtered) example, we know exactly 100 will be updated with each
NOTE:
When applying additional filters, it is important to ensure they are properly covered by an index to optimize `EachBatch` performance.
-In the example above we need an index on `(type, id)` to support the filters. See [the `EachBatch` docs for more information](iterating_tables_in_batches.md).
+In the example above we need an index on `(type, id)` to support the filters. See [the `EachBatch` documentation for more information](iterating_tables_in_batches.md).
## Example
@@ -349,20 +349,35 @@ background migration.
`BatchedMigrationJob` is initialized with necessary arguments to
execute the batch, as well as a connection to the tracking database.
-1. Add a new trigger to the database to update newly created and updated routes,
- similar to this example:
+1. Create a database migration that adds a new trigger to the database. Example:
```ruby
- execute(<<~SQL)
- CREATE OR REPLACE FUNCTION example() RETURNS trigger
- LANGUAGE plpgsql
- AS $$
- BEGIN
- NEW."namespace_id" = NEW."source_id"
- RETURN NEW;
- END;
- $$;
- SQL
+ class AddTriggerToRoutesToCopySourceIdToNamespaceId < Gitlab::Database::Migration[2.1]
+ FUNCTION_NAME = 'example_function'
+ TRIGGER_NAME = 'example_trigger'
+
+ def up
+ execute(<<~SQL)
+ CREATE OR REPLACE FUNCTION #{FUNCTION_NAME}() RETURNS trigger
+ LANGUAGE plpgsql
+ AS $$
+ BEGIN
+ NEW."namespace_id" = NEW."source_id"
+ RETURN NEW;
+ END;
+ $$;
+
+ CREATE TRIGGER #{TRIGGER_NAME}() AFTER INSERT OR UPDATE
+ ON routes
+ FOR EACH ROW EXECUTE FUNCTION #{FUNCTION_NAME}();
+ SQL
+ end
+
+ def down
+ drop_trigger(TRIGGER_NAME, :routes)
+ drop_function(FUNCTION_NAME)
+ end
+ end
```
1. Create a post-deployment migration that queues the migration for existing data:
@@ -398,10 +413,28 @@ background migration.
`restrict_gitlab_migration gitlab_schema: :gitlab_ci`.
After deployment, our application:
- - Continues using the data as before.
- - Ensures that both existing and new data are migrated.
+ - Continues using the data as before.
+ - Ensures that both existing and new data are migrated.
+
+1. In the next release, add a database migration to remove the trigger.
-1. In the next release, remove the trigger. We must also add a new post-deployment migration
+ ```ruby
+ class RemoveNamepaceIdTriggerFromRoutes < Gitlab::Database::Migration[2.1]
+ FUNCTION_NAME = 'example_function'
+ TRIGGER_NAME = 'example_trigger'
+
+ def up
+ drop_trigger(TRIGGER_NAME, :routes)
+ drop_function(FUNCTION_NAME)
+ end
+
+ def down
+ # Should reverse the trigger and the function in the up method of the migration that added it
+ end
+ end
+ ```
+
+1. Add a new post-deployment migration
that checks that the batched background migration is completed. For example:
```ruby
@@ -502,7 +535,7 @@ end
```
NOTE:
-[Additional filters](#additional-filters) defined with `scope_to` will be ignored by `LooseIndexScanBatchingStrategy` and `distinct_each_batch`.
+[Additional filters](#additional-filters) defined with `scope_to` are ignored by `LooseIndexScanBatchingStrategy` and `distinct_each_batch`.
## Testing
@@ -686,6 +719,16 @@ You can view failures in two ways:
WHERE transition_logs.next_status = '2' AND migration.job_class_name = "CLASS_NAME";
```
+### Adding indexes to support batched background migrations
+
+Sometimes it is necessary to add a new or temporary index to support a batched background migration.
+To do this, create the index in a post-deployment migration that precedes the post-deployment
+migration that queues the background migration.
+
+See the documentation for [adding database indexes](adding_database_indexes.md#analyzing-a-new-index-before-a-batched-background-migration)
+for additional information about some cases that require special attention to allow the index to be used directly after
+creation.
+
## Legacy background migrations
Batched background migrations replaced the [legacy background migrations framework](background_migrations.md).
diff --git a/doc/development/database/clickhouse/gitlab_activity_data.md b/doc/development/database/clickhouse/gitlab_activity_data.md
new file mode 100644
index 00000000000..6ba11b8afaf
--- /dev/null
+++ b/doc/development/database/clickhouse/gitlab_activity_data.md
@@ -0,0 +1,482 @@
+---
+stage: Data Stores
+group: Database
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Store GitLab activity data in ClickHouse
+
+## Overview of the existing implementation
+
+### What is GitLab activity data
+
+GitLab records activity data during its operation as users interact with the application. Most of these interactions revolve around the projects, issues, and merge requests domain objects. Users can perform several different actions and some of these actions are recorded in a separate PostgreSQL database table called `events`.
+
+Example events:
+
+- Issue opened
+- Issue reopened
+- User joined a project
+- Merge Request merged
+- Repository pushed
+- Snippet created
+
+### Where is the activity data used
+
+Several features use activity data:
+
+- The user's [contribution calendar](../../../user/profile/contributions_calendar.md) on the profile page.
+- Paginated list of the user's contributions.
+- Paginated list of user activity for a Project and a Group.
+- [Contribution analytics](../../../user/group/contribution_analytics/index.md).
+
+### How is the activity data created
+
+The activity data is usually generated on the service layer when a specific operation is executed by the user. The persistence characteristics of an `events` record depend on the implementation of the service. Two main approaches exist:
+
+1. In the database transaction where the actual event occurs.
+1. After the database transaction (which could be delayed).
+
+The above-mentioned mechanics provide a "mostly" consistent stream of `events`.
+
+For example, consistently recording an `events` record:
+
+```ruby
+ApplicationRecord.transaction do
+ issue.closed!
+ Event.create!(action: :closed, target: issue)
+end
+```
+
+Example, unsafe recording of an `events` record:
+
+```ruby
+ApplicationRecord.transaction do
+ issue.closed!
+end
+
+# If a crash happens here, the event will not be recorded.
+Event.create!(action: :closed, target: issue)
+```
+
+### Database table structure
+
+The `events` table uses [polymorphic association](https://guides.rubyonrails.org/association_basics.html#polymorphic-associations) to allow associating different database tables (issues, merge requests, etc.) with a record. A simplified database structure:
+
+```sql
+ Column | Type | Nullable | Default | Storage |
+-------------+--------------------------+-----------+----------+------------------------------------+
+ project_id | integer | | | plain |
+ author_id | integer | not null | | plain |
+ target_id | integer | | | plain |
+ created_at | timestamp with time zone | not null | | plain |
+ updated_at | timestamp with time zone | not null | | plain |
+ action | smallint | not null | | plain |
+ target_type | character varying | | | extended |
+ group_id | bigint | | | plain |
+ fingerprint | bytea | | | extended |
+ id | bigint | not null | nextval('events_id_seq'::regclass) | plain |
+```
+
+Some unexpected characteristics due to the evolving database design:
+
+- The `project_id` and the `group_id` columns are mutually exclusive, internally we call them resource parent.
+ - Example 1: for an issue opened event, the `project_id` field is populated.
+ - Example 2: for an epic-related event, the `group_id` field is populated (epic is always part of a group).
+- The `target_id` and `target_type` column pair identifies the target record.
+ - Example: `target_id=1` and `target_type=Issue`.
+ - When the columns are `null`, we refer to an event which has no representation in the database. For example a repository `push` action.
+- Fingerprint is used in some cases to later alter the event based on some metadata change. This approach is mostly used for Wiki pages.
+
+### Database record modifications
+
+Most of the data is written once however, we cannot say that the table is append-only. A few use cases where actual row updates and deletions happen:
+
+- Fingerprint-based update for certain Wiki page records.
+- When user or an associated resource is deleted, the event rows are also deleted.
+ - The deletion of the associated `events` records happens in batches.
+
+### Current performance problems
+
+- The table uses significant disk space.
+- Adding new events may significantly increase the database record count.
+- Implementing data pruning logic is difficult.
+- Time-range-based aggregations are not performant enough, some features may break due to slow database queries.
+
+### Example queries
+
+NOTE:
+These queries have been significantly simplified from the actual queries from production.
+
+Database query for the user's contribution graph:
+
+```sql
+SELECT DATE(events.created_at), COUNT(*)
+FROM events
+WHERE events.author_id = 1
+AND events.created_at BETWEEN '2022-01-17 23:00:00' AND '2023-01-18 22:59:59.999999'
+AND (
+ (
+ events.action = 5
+ ) OR
+ (
+ events.action IN (1, 3) -- Enum values are documented in the Event model, see the ACTIONS constant in app/models/event.rb
+ AND events.target_type IN ('Issue', 'WorkItem')
+ ) OR
+ (
+ events.action IN (7, 1, 3)
+ AND events.target_type = 'MergeRequest'
+ ) OR
+ (
+ events.action = 6
+ )
+)
+GROUP BY DATE(events.created_at)
+```
+
+Query for group contributions for each user:
+
+```sql
+SELECT events.author_id, events.target_type, events.action, COUNT(*)
+FROM events
+WHERE events.created_at BETWEEN '2022-01-17 23:00:00' AND '2023-03-18 22:59:59.999999'
+AND events.project_id IN (1, 2, 3) -- list of project ids in the group
+GROUP BY events.author_id, events.target_type, events.action
+```
+
+## Storing activity data in ClickHouse
+
+### Data persistence
+
+At the moment, there is no consensus about the way we would replicate data from the PostgreSQL database to ClickHouse. A few ideas that might work for the `events` table:
+
+#### Record data immediately
+
+This approach provides a simple way to keep the existing `events` table working while we're also sending data to the ClickHouse database. When an event record is created, ensure that it's created outside of the transaction. After persisting the data in PostgreSQL, persist it in ClickHouse.
+
+```ruby
+ApplicationRecord.transaction do
+ issue.update!(state: :closed)
+end
+
+# could be a method to hide complexity
+Event.create!(action: :closed, target: issue)
+ClickHouse::Event.create(action: :closed, target: issue)
+```
+
+What's behind the implementation of `ClickHouse::Event` is not decided yet, it could be one of the following:
+
+- ActiveRecord model directly connecting the ClickHouse database.
+- REST API call to an intermediate service.
+- Enqueueing an event to an event-streaming tool (like Kafka).
+
+#### Replication of `events` rows
+
+Assuming that the creation of `events` record is an integral part of the system, introducing another storage call might cause performance degradation in various code paths, or it could introduce significant complexity.
+
+Rather than sending data to ClickHouse on event creation time, we would move this processing in the background by iterating over the `events` table and sending the newly created database rows.
+
+By keeping track of which records have been sent over ClickHouse, we could incrementally send data.
+
+```ruby
+last_updated_at = SyncProcess.last_updated_at
+
+# oversimplified loop, we would probably batch this...
+Event.where(updated_at > last_updated_at).each do |row|
+ last_row = ClickHouse::Event.create(row)
+end
+
+SyncProcess.last_updated_at = last_row.updated_at
+```
+
+### ClickHouse database table structure
+
+When coming up with the initial database structure, we must look at the way the data is queried.
+
+We have two main use cases:
+
+- Query data for a certain user, within a time range.
+ - `WHERE author_id = 1 AND created_at BETWEEN '2021-01-01' AND '2021-12-31'`
+ - Additionally, there might be extra `project_id` condition due to the access control check.
+- Query data for a project or group, within a time range.
+ - `WHERE project_id IN (1, 2) AND created_at BETWEEN '2021-01-01' AND '2021-12-31'`
+
+The `author_id` and `project_id` columns are considered high-selectivity columns. By this we mean that optimizing the filtering of the `author_id` and the `project_id` columns is desirable for having performant database queries.
+
+The most recent activity data is queried more often. At some point, we might just drop or relocate older data. Most of the features look back only a year.
+
+For these reasons, we could start with a database table storing low-level `events` data:
+
+```plantuml
+hide circle
+
+entity "events" as events {
+ id : UInt64 ("primary key")
+--
+ project_id : UInt64
+ group_id : UInt64
+ target_id : UInt64
+ target_type : String
+ action : UInt8
+ fingerprint : UInt64
+ created_at : DateTime
+ updated_at : DateTime
+}
+```
+
+The SQL statement for creating the table:
+
+```sql
+CREATE TABLE events
+(
+ `id` UInt64,
+ `project_id` UInt64 DEFAULT 0 NOT NULL,
+ `group_id` UInt64 DEFAULT 0 NOT NULL,
+ `author_id` UInt64 DEFAULT 0 NOT NULL,
+ `target_id` UInt64 DEFAULT 0 NOT NULL,
+ `target_type` LowCardinality(String) DEFAULT '' NOT NULL,
+ `action` UInt8 DEFAULT 0 NOT NULL,
+ `fingerprint` UInt64 DEFAULT 0 NOT NULL,
+ `created_at` DateTime64(6, 'UTC') DEFAULT now() NOT NULL,
+ `updated_at` DateTime64(6, 'UTC') DEFAULT now() NOT NULL
+)
+ENGINE = ReplacingMergeTree(updated_at)
+ORDER BY id;
+```
+
+A few changes compared to the PostgreSQL version:
+
+- `target_type` uses [an optimization](https://clickhouse.com/docs/en/sql-reference/data-types/lowcardinality/) for low-cardinality column values.
+- `fingerprint` becomes an integer and leverages a performant integer-based hashing function such as xxHash64.
+- All columns get a default value, the 0 default value for the integer columns means no value. See the related [best practices](https://clickhouse.com/docs/en/cloud/bestpractices/avoid-nullable-columns/).
+- `NOT NULL` to ensure that we always use the default values when data is missing (different behavior compared to PostgreSQL).
+- The "primary" key automatically becomes the `id` column due to the `ORDER BY` clause.
+
+Let's insert the same primary key value twice:
+
+```sql
+INSERT INTO events (id, project_id, target_id, author_id, target_type, action) VALUES (1, 2, 3, 4, 'Issue', null);
+INSERT INTO events (id, project_id, target_id, author_id, target_type, action) VALUES (1, 20, 30, 5, 'Issue', null);
+```
+
+Let's inspect the results:
+
+```sql
+SELECT * FROM events
+```
+
+- We have two rows with the same `id` value (primary key).
+- The `null` `action` becomes `0`.
+- The non-specified fingerprint column becomes `0`.
+- The `DateTime` columns have the insert timestamp.
+
+ClickHouse will eventually "replace" the rows with the same primary key in the background. When running this operation, the higher `updated_at` value takes precedence. The same behavior can be simulated with the `final` keyword:
+
+```sql
+SELECT * FROM events FINAL
+```
+
+Adding `FINAL` to a query can have significant performance consequences, some of the issues are documented in the [ClickHouse documentation](https://clickhouse.com/docs/en/sql-reference/statements/select/from/#final-modifier).
+
+We should always expect duplicated values in the table, so we must take care of the deduplication in query time.
+
+### ClickHouse database queries
+
+ClickHouse uses SQL for querying the data, in some cases, a PostgreSQL query can be used in ClickHouse without major modifications assuming that the underlying database structure is very similar.
+
+Query for group contributions for each user (PostgreSQL):
+
+```sql
+SELECT events.author_id, events.target_type, events.action, COUNT(*)
+FROM events
+WHERE events.created_at BETWEEN '2022-01-17 23:00:00' AND '2023-03-18 22:59:59.999999'
+AND events.project_id IN (1, 2, 3) -- list of project ids in the group
+GROUP BY events.author_id, events.target_type, events.action
+```
+
+The same query would work in PostgreSQL however, we might see duplicated values in ClickHouse due to the way the table engine works. The deduplication can be achieved by using a nested `FROM` statement.
+
+```sql
+SELECT author_id, target_type, action, count(*)
+FROM (
+ SELECT
+ id,
+ argMax(events.project_id, events.updated_at) AS project_id,
+ argMax(events.group_id, events.updated_at) AS group_id,
+ argMax(events.author_id, events.updated_at) AS author_id,
+ argMax(events.target_type, events.updated_at) AS target_type,
+ argMax(events.target_id, events.updated_at) AS target_id,
+ argMax(events.action, events.updated_at) AS action,
+ argMax(events.fingerprint, events.updated_at) AS fingerprint,
+ FIRST_VALUE(events.created_at) AS created_at,
+ MAX(events.updated_at) AS updated_at
+ FROM events
+ WHERE events.created_at BETWEEN '2022-01-17 23:00:00' AND '2023-03-18 22:59:59.999999'
+ AND events.project_id IN (1, 2, 3) -- list of project ids in the group
+ GROUP BY id
+) AS events
+GROUP BY author_id, target_type, action
+```
+
+- Take the most recent column values based on the `updated_at` column.
+- Take the first value for `created_at`, assuming that the first `INSERT` contains the correct value. An issue only when we don't sync `created_at` at all and the default value (`NOW()`) is used.
+- Take the most recent `updated_at` value.
+
+The query looks more complicated now because of the deduplication logic. The complexity can be hidden behind a database view.
+
+### Optimizing the performance
+
+The aggregation query in the previous section might not be performant enough for production use due to the large volume of data.
+
+Let's add 1 million extra rows to the `events` table:
+
+```sql
+INSERT INTO events (id, project_id, author_id, target_id, target_type, action) SELECT id, project_id, author_id, target_id, 'Issue' AS target_type, action FROM generateRandom('id UInt64, project_id UInt64, author_id UInt64, target_id UInt64, action UInt64') LIMIT 1000000;
+```
+
+Running the previous aggregation query in the console prints out some performance data:
+
+```plaintext
+1 row in set. Elapsed: 0.122 sec. Processed 1.00 million rows, 42.00 MB (8.21 million rows/s., 344.96 MB/s.)
+```
+
+The query returned 1 row (correctly) however, it had to process 1 million rows (full table). We can optimize the query with an index on the `project_id` column:
+
+```sql
+ALTER TABLE events ADD INDEX project_id_index project_id TYPE minmax GRANULARITY 10;
+ALTER TABLE events MATERIALIZE INDEX project_id_index;
+```
+
+Executing the query returns much better figures:
+
+```plaintext
+Read 2 rows, 107.00 B in 0.005616811 sec., 356 rows/sec., 18.60 KiB/sec.
+```
+
+To optimize the date range filter on the `created_at` column, we could try adding another index on the `created_at` column.
+
+#### Query for the contribution graph
+
+Just to recap, this is the PostgreSQL query:
+
+```sql
+SELECT DATE(events.created_at), COUNT(*)
+FROM events
+WHERE events.author_id = 1
+AND events.created_at BETWEEN '2022-01-17 23:00:00' AND '2023-01-18 22:59:59.999999'
+AND (
+ (
+ events.action = 5
+ ) OR
+ (
+ events.action IN (1, 3) -- Enum values are documented in the Event model, see the ACTIONS constant in app/models/event.rb
+ AND events.target_type IN ('Issue', 'WorkItem')
+ ) OR
+ (
+ events.action IN (7, 1, 3)
+ AND events.target_type = 'MergeRequest'
+ ) OR
+ (
+ events.action = 6
+ )
+)
+GROUP BY DATE(events.created_at)
+```
+
+The filtering and the count aggregation is mainly done on the `author_id` and the `created_at` columns. Grouping the data by these two columns would probably give an adequate performance.
+
+We could attempt adding an index on the `author_id` column however, we still need an additional index on the `created_at` column to properly cover this query. Besides, under the contribution graph, GitLab shows the list of ordered contributions of the user which would be great to get it efficiently via a different query with the `ORDER BY` clause.
+
+For these reasons, it's probably better to use a ClickHouse projection which stores the events rows redundantly but we can specify a different sort order.
+
+The ClickHouse query would be the following (with a slightly adjusted date range):
+
+```sql
+SELECT DATE(events.created_at) AS date, COUNT(*) AS count
+FROM (
+ SELECT
+ id,
+ argMax(events.created_at, events.updated_at) AS created_at
+ FROM events
+ WHERE events.author_id = 4
+ AND events.created_at BETWEEN '2023-01-01 23:00:00' AND '2024-01-01 22:59:59.999999'
+ AND (
+ (
+ events.action = 5
+ ) OR
+ (
+ events.action IN (1, 3) -- Enum values are documented in the Event model, see the ACTIONS constant in app/models/event.rb
+ AND events.target_type IN ('Issue', 'WorkItem')
+ ) OR
+ (
+ events.action IN (7, 1, 3)
+ AND events.target_type = 'MergeRequest'
+ ) OR
+ (
+ events.action = 6
+ )
+ )
+ GROUP BY id
+) AS events
+GROUP BY DATE(events.created_at)
+```
+
+The query does a full table scan, let's optimize it:
+
+```sql
+ALTER TABLE events ADD PROJECTION events_by_authors (
+ SELECT * ORDER BY author_id, created_at -- different sort order for the table
+);
+
+ALTER TABLE events MATERIALIZE PROJECTION events_by_authors;
+```
+
+#### Pagination of contributions
+
+Listing the contributions of a user can be queried in the following way:
+
+```sql
+SELECT events.*
+FROM (
+ SELECT
+ id,
+ argMax(events.project_id, events.updated_at) AS project_id,
+ argMax(events.group_id, events.updated_at) AS group_id,
+ argMax(events.author_id, events.updated_at) AS author_id,
+ argMax(events.target_type, events.updated_at) AS target_type,
+ argMax(events.target_id, events.updated_at) AS target_id,
+ argMax(events.action, events.updated_at) AS action,
+ argMax(events.fingerprint, events.updated_at) AS fingerprint,
+ FIRST_VALUE(events.created_at) AS created_at,
+ MAX(events.updated_at) AS updated_at
+ FROM events
+ WHERE events.author_id = 4
+ GROUP BY id
+ ORDER BY created_at DESC, id DESC
+) AS events
+LIMIT 20
+```
+
+ClickHouse supports the standard `LIMIT N OFFSET M` clauses, so we can request the next page:
+
+```sql
+SELECT events.*
+FROM (
+ SELECT
+ id,
+ argMax(events.project_id, events.updated_at) AS project_id,
+ argMax(events.group_id, events.updated_at) AS group_id,
+ argMax(events.author_id, events.updated_at) AS author_id,
+ argMax(events.target_type, events.updated_at) AS target_type,
+ argMax(events.target_id, events.updated_at) AS target_id,
+ argMax(events.action, events.updated_at) AS action,
+ argMax(events.fingerprint, events.updated_at) AS fingerprint,
+ FIRST_VALUE(events.created_at) AS created_at,
+ MAX(events.updated_at) AS updated_at
+ FROM events
+ WHERE events.author_id = 4
+ GROUP BY id
+ ORDER BY created_at DESC, id DESC
+) AS events
+LIMIT 20 OFFSET 20
+```
diff --git a/doc/development/database/clickhouse/index.md b/doc/development/database/clickhouse/index.md
new file mode 100644
index 00000000000..a26bac261fd
--- /dev/null
+++ b/doc/development/database/clickhouse/index.md
@@ -0,0 +1,85 @@
+---
+stage: Data Stores
+group: Database
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Introduction to ClickHouse use and table design
+
+## How it differs from PostgreSQL
+
+The [intro](https://clickhouse.com/docs/en/intro/) page is quite good to give an overview of ClickHouse.
+
+ClickHouse has a lot of differences from traditional OLTP (online transaction processing) databases like PostgreSQL. The underlying architecture is a bit different, and the processing is a lot more CPU-bound than in traditional databases.
+ClickHouse is a log-centric database where immutability is a key component. The advantages of such approaches are well documented [[1]](https://www.odbms.org/2015/10/the-rise-of-immutable-data-stores/) however it also makes updates much harder. See ClickHouse [documentation](https://clickhouse.com/docs/en/guides/developer/mutations/) for operations that provide UPDATE/DELETE support. It is noticeable that these operations are supposed to be non-frequent.
+
+This distinction is important while designing tables. Either:
+
+- The updates are not required (best case)
+- If they are needed, they aren't to be run during query execution.
+
+## ACID compatibility
+
+ClickHouse has a slightly different overview of Transactional support, where the guarantees are applicable only up to a block of inserted data to a specific table. See [the Transactional (ACID) support](https://clickhouse.com/docs/en/guides/developer/transactional/) documentation for details.
+
+Multiple insertions in a single write should be avoided as transactional support across multiple tables is only covered in materialized views.
+
+ClickHouse is heavily geared towards providing the best-in-class support for analytical queries. Operations like aggregation are very fast and there are several features to augment these capabilities.
+ClickHouse has some good blog posts covering [details of aggregations](https://altinity.com/blog/clickhouse-aggregation-fun-part-1-internals-and-handy-tools).
+
+## Primary indexes, sorting index and dictionaries
+
+It is highly recommended to read ["A practical introduction to primary indexes in ClickHouse""](https://clickhouse.com/docs/en/guides/improving-query-performance/sparse-primary-indexes/sparse-primary-indexes-intro) to get an understanding of indexes in ClickHouse.
+
+Particularly how database index design in ClickHouse [differs](https://clickhouse.com/docs/en/guides/improving-query-performance/sparse-primary-indexes/sparse-primary-indexes-design/#an-index-design-for-massive-data-scales) from those in transactional databases like PostgreSQL.
+
+Primary index design plays a very important role in query performance and should be stated carefully. Almost all of the queries should rely on the primary index as full data scans are bound to take longer.
+
+Read the documentation for [primary keys and indexes in queries](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#primary-keys-and-indexes-in-queries) to learn how indexes can affect query performance in MergeTree Table engines (default table engine in ClickHouse).
+
+Secondary indexes in ClickHouse are different from what is available in other systems. They are also called data-skipping indexes as they are used to skip over a block of data. See the documentation for [data-skipping indexes](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-data_skipping-indexes).
+
+ClickHouse also offers ["Dictionaries"](https://clickhouse.com/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts/) which can be used as external indexes. Dictionaries are loaded from memory and can be used to look up values on query runtime.
+
+## Data types & Partitioning
+
+ClickHouse offers SQL-compatible [data types](https://clickhouse.com/docs/en/sql-reference/data-types/) and few specialized data types like:
+
+- [`LowCardinality`](https://clickhouse.com/docs/en/sql-reference/data-types/lowcardinality)
+- [UUID](https://clickhouse.com/docs/en/sql-reference/data-types/uuid)
+- [Maps](https://clickhouse.com/docs/en/sql-reference/data-types/map)
+- [Nested](https://clickhouse.com/docs/en/sql-reference/data-types/nested-data-structures/nested) which is interesting, because it simulates a table inside a column.
+
+One key design aspect that comes up front while designing a table is the partitioning key. Partitions can be any arbitrary expression but usually, these are time duration like months, days, or weeks. ClickHouse takes a best-effort approach to minimize the data read by using the smallest set of partitions.
+
+Suggested reads:
+
+- [Choose a low cardinality partitioning key](https://clickhouse.com/docs/en/optimize/partitioning-key)
+- [Custom partitioning key](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/custom-partitioning-key).
+
+## Sharding and replication
+
+Sharding is a feature that allows splitting the data into multiple ClickHouse nodes to increase throughput and decrease latency. The sharding feature uses a distributed engine that is backed by local tables. The distributed engine is a "virtual" table that does not store any data. It is used as an interface to insert and query data.
+
+See [the ClickHouse documentation](https://clickhouse.com/docs/en/engines/table-engines/special/distributed/) and this section on [replication and sharding](https://clickhouse.com/docs/en/manage/replication-and-sharding/). ClickHouse can use either Zookeeper or its own compatible API via a component called [ClickHouse Keeper](https://clickhouse.com/docs/en/operations/clickhouse-keeper) to maintain consensus.
+
+After nodes are set up, they can become invisible from the Clients and both write and read queries can be issued to any node.
+
+In most cases, clusters usually start with a fixed number of nodes(~ shards). [Rebalancing shards](https://clickhouse.com/docs/en/guides/sre/scaling-clusters) is operationally heavy and requires rigorous testing.
+
+Replication is supported by MergeTree Table engine, see the [replication section](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/) in documentation for details on how to define them.
+ClickHouse relies on a distributed coordination component (either Zookeeper or ClickHouse Keeper) to track the participating nodes in the quorum. Replication is asynchronous and multi-leader. Inserts can be issued to any node and they can appear on other nodes with some latency. If desired, stickiness to a specific node can be used to make sure that reads observe the latest written data.
+
+## Materialized views
+
+One of the defining features of ClickHouse is materialized views. Functionally they resemble insert triggers for ClickHouse.
+Materialized views can be used for a variety of use cases which are well [documented](https://www.polyscale.ai/blog/polyscale-metrics-materialized-views/) on the web.
+
+We recommended reading the [views](https://clickhouse.com/docs/en/sql-reference/statements/create/view#materialized-view) section from the official documentation to get a better understanding of how they work.
+
+Quoting the [documentation](https://clickhouse.com/docs/en/sql-reference/statements/create/view#materialized-view):
+
+> Materialized views in ClickHouse are implemented more like insert triggers.
+> If there's some aggregation in the view query, it's applied only to the batch
+> of freshly inserted data. Any changes to existing data of the source table
+> (like update, delete, drop a partition, etc.) do not change the materialized view.
diff --git a/doc/development/database/clickhouse/optimization.md b/doc/development/database/clickhouse/optimization.md
new file mode 100644
index 00000000000..166bbf7d2b1
--- /dev/null
+++ b/doc/development/database/clickhouse/optimization.md
@@ -0,0 +1,60 @@
+---
+stage: Data Stores
+group: Database
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Optimizing query execution
+
+ClickHouse Inc has listed a [variety of optimization strategies](https://clickhouse.com/docs/en/optimize/).
+
+ClickHouse relies heavily on the structure of the primary index. However, in some cases, it's possible that queries rely on a column that's part of the primary index, but isn't the first column. See [Using multiple primary indexes](https://clickhouse.com/docs/en/guides/improving-query-performance/sparse-primary-indexes/sparse-primary-indexes-multiple) which offers several options in such cases. For example: using a data skipping index as a secondary index.
+
+In cases of compound primary indexes, it's helpful to understand the data characteristics of key columns is also very helpful. They can allow the index to be more efficient. [Ordering key columns efficiently](https://clickhouse.com/docs/en/guides/improving-query-performance/sparse-primary-indexes/sparse-primary-indexes-cardinality) goes into details on these concepts.
+
+ClickHouse blog also has a very good post, [Super charging your ClickHouse queries](https://clickhouse.com/blog/clickhouse-faster-queries-with-projections-and-primary-indexes), that outlines almost all of the approaches listed above.
+
+It is possible to use [`EXPLAIN`](https://clickhouse.com/docs/en/sql-reference/statements/explain/) statements with queries to get visible steps of the query pipeline. Note the different [types](https://clickhouse.com/docs/en/sql-reference/statements/explain/#explain-types) of `EXPLAIN`.
+
+Also, to get detailed query execution pipeline, you can toggle the logs level to `trace` via `clickhouse-client` and then execute the query.
+
+For example:
+
+```plaintext
+$ clickhouse-client :) SET send_logs_level = 'trace'
+$ clickhouse-client :) select count(traceID) from jaeger_index WHERE tenant = '12' AND service != 'jaeger-query' FORMAT Vertical ;
+
+SELECT count(traceID)
+FROM jaeger_index
+WHERE (tenant = '12') AND (service != 'jaeger-query')
+FORMAT Vertical
+
+Query id: 6ce40daf-e1b1-4714-ab02-268246f3c5c9
+
+[cluster-0-0-0] 2023.01.30 06:31:32.240819 [ 4991 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Debug> executeQuery: (from 127.0.0.1:53654) select count(traceID) from jaeger_index WHERE tenant = '12' AND service != 'jaeger-query' FORMAT Vertical ; (stage: Complete)
+....
+[cluster-0-0-0] 2023.01.30 06:31:32.244071 [ 4991 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Debug> InterpreterSelectQuery: MergeTreeWhereOptimizer: condition "service != 'jaeger-query'" moved to PREWHERE
+[cluster-0-0-0] 2023.01.30 06:31:32.244420 [ 4991 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Debug> InterpreterSelectQuery: MergeTreeWhereOptimizer: condition "service != 'jaeger-query'" moved to PREWHERE
+....
+[cluster-0-0-0] 2023.01.30 06:31:32.245153 [ 4991 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
+[cluster-0-0-0] 2023.01.30 06:31:32.245255 [ 4991 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Trace> InterpreterSelectQuery: Complete -> Complete
+[cluster-0-0-0] 2023.01.30 06:31:32.245590 [ 4991 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Debug> tracing_gcs.jaeger_index_local (66c6ca81-e20d-44dc-8101-92678fc24d99) (SelectExecutor): Key condition: (column 1 not in ['jaeger-query', 'jaeger-query']), unknown, (column 0 in ['12', '12']), and, and
+[cluster-0-0-0] 2023.01.30 06:31:32.245784 [ 4991 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Debug> tracing_gcs.jaeger_index_local (66c6ca81-e20d-44dc-8101-92678fc24d99) (SelectExecutor): MinMax index condition: unknown, unknown, and, unknown, and
+[cluster-0-0-0] 2023.01.30 06:31:32.246239 [ 1503 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Trace> tracing_gcs.jaeger_index_local (66c6ca81-e20d-44dc-8101-92678fc24d99) (SelectExecutor): Used generic exclusion search over index for part 202301_1512_21497_9164 with 4 steps
+[cluster-0-0-0] 2023.01.30 06:31:32.246293 [ 1503 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Trace> tracing_gcs.jaeger_index_local (66c6ca81-e20d-44dc-8101-92678fc24d99) (SelectExecutor): Used generic exclusion search over index for part 202301_21498_24220_677 with 1 steps
+[cluster-0-0-0] 2023.01.30 06:31:32.246488 [ 4991 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Debug> tracing_gcs.jaeger_index_local (66c6ca81-e20d-44dc-8101-92678fc24d99) (SelectExecutor): Selected 2/2 parts by partition key, 1 parts by primary key, 2/4 marks by primary key, 2 marks to read from 1 ranges
+[cluster-0-0-0] 2023.01.30 06:31:32.246591 [ 4991 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Trace> MergeTreeInOrderSelectProcessor: Reading 1 ranges in order from part 202301_1512_21497_9164, approx. 16384 rows starting from 0
+[cluster-0-0-0] 2023.01.30 06:31:32.642095 [ 348 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Trace> AggregatingTransform: Aggregating
+[cluster-0-0-0] 2023.01.30 06:31:32.642193 [ 348 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Trace> Aggregator: An entry for key=16426982211452591884 found in cache: sum_of_sizes=2, median_size=1
+[cluster-0-0-0] 2023.01.30 06:31:32.642210 [ 348 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Trace> Aggregator: Aggregation method: without_key
+[cluster-0-0-0] 2023.01.30 06:31:32.642330 [ 348 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Debug> AggregatingTransform: Aggregated. 3211 to 1 rows (from 50.18 KiB) in 0.395452983 sec. (8119.802 rows/sec., 126.89 KiB/sec.)
+[cluster-0-0-0] 2023.01.30 06:31:32.642343 [ 348 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Trace> Aggregator: Merging aggregated data
+Row 1:
+──────
+count(traceID): 3211
+[cluster-0-0-0] 2023.01.30 06:31:32.642887 [ 4991 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Information> executeQuery: Read 16384 rows, 620.52 KiB in 0.401978272 sec., 40758 rows/sec., 1.51 MiB/sec.
+[cluster-0-0-0] 2023.01.30 06:31:32.645232 [ 4991 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Debug> MemoryTracker: Peak memory usage (for query): 831.98 KiB.
+[cluster-0-0-0] 2023.01.30 06:31:32.645251 [ 4991 ] {6ce40daf-e1b1-4714-ab02-268246f3c5c9} <Debug> TCPHandler: Processed in 0.404908496 sec.
+
+1 row in set. Elapsed: 0.402 sec. Processed 16.38 thousand rows, 635.41 KB (40.71 thousand rows/s., 1.58 MB/s.)
+```
diff --git a/doc/development/database/database_debugging.md b/doc/development/database/database_debugging.md
index edc35dd95e8..9cc85610e98 100644
--- a/doc/development/database/database_debugging.md
+++ b/doc/development/database/database_debugging.md
@@ -177,3 +177,21 @@ you should set the `SKIP_SCHEMA_VERSION_CHECK` environment variable.
```shell
bundle exec rake db:migrate SKIP_SCHEMA_VERSION_CHECK=true
```
+
+## Performance issues
+
+### Reduce connection overhead with connection pooling
+
+Creating new database connections is not free, and in PostgreSQL specifically, it requires
+forking an entire process to handle each new one. In case a connection lives for a very long time,
+this is no problem. However, forking a process for several small queries can turn out to be costly.
+If left unattended, peaks of new database connections can cause performance degradation,
+or even lead to a complete outage.
+
+A proven solution for instances that deal with surges of small, short-lived database connections
+is to implement [PgBouncer](../../administration/postgresql/pgbouncer.md#pgbouncer-as-part-of-a-fault-tolerant-gitlab-installation) as a connection pooler.
+This pool can be used to hold thousands of connections for almost no overhead. The drawback is the addition of
+a small amount of latency, in exchange for up to more than 90% performance improvement, depending on the usage patterns.
+
+PgBouncer can be fine-tuned to fit different installations. See our documentation on
+[fine-tuning PgBouncer](../../administration/postgresql/pgbouncer.md#fine-tuning) for more information.
diff --git a/doc/development/database/index.md b/doc/development/database/index.md
index 5abc7cd3ffa..8f22eaac496 100644
--- a/doc/development/database/index.md
+++ b/doc/development/database/index.md
@@ -20,6 +20,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
- [explain.depesz.com](https://explain.depesz.com/) or [explain.dalibo.com](https://explain.dalibo.com/) for visualizing the output of `EXPLAIN`
- [pgFormatter](https://sqlformat.darold.net/) a PostgreSQL SQL syntax beautifier
- [db:check-migrations job](dbcheck-migrations-job.md)
+- [Database migration pipeline](database_migration_pipeline.md)
## Migrations
@@ -93,6 +94,22 @@ info: To determine the technical writer assigned to the Stage/Group associated w
- [Troubleshooting PostgreSQL](../../administration/troubleshooting/postgresql.md)
- [Working with the bundled PgBouncer service](../../administration/postgresql/pgbouncer.md)
+## User information for scaling
+
+For GitLab administrators, information about
+[configuring PostgreSQL for scaling](../../administration/postgresql/index.md) is available,
+including the major methods:
+
+- [Standalone PostgreSQL](../../administration/postgresql/standalone.md)
+- [External PostgreSQL instances](../../administration/postgresql/external.md)
+- [Replication and failover](../../administration/postgresql/replication_and_failover.md)
+
+## ClickHouse
+
+- [Introduction](clickhouse/index.md)
+- [Optimizing query execution](clickhouse/optimization.md)
+- [Rebuild GitLab features using ClickHouse 1: Activity data](clickhouse/gitlab_activity_data.md)
+
## Miscellaneous
- [Maintenance operations](maintenance_operations.md)
diff --git a/doc/development/database/migrations_for_multiple_databases.md b/doc/development/database/migrations_for_multiple_databases.md
index bc0ef654336..b903c56651d 100644
--- a/doc/development/database/migrations_for_multiple_databases.md
+++ b/doc/development/database/migrations_for_multiple_databases.md
@@ -10,8 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
This document describes how to properly write database migrations
for [the decomposed GitLab application using multiple databases](https://gitlab.com/groups/gitlab-org/-/epics/6168).
-
-Learn more about general multiple databases support in a [separate document](multiple_databases.md).
+For more information, see [Multiple databases](multiple_databases.md).
The design for multiple databases (except for the Geo database) assumes
that all decomposed databases have **the same structure** (for example, schema), but **the data is different** in each database. This means that some tables do not contain data on each database.
diff --git a/doc/development/database/new_database_migration_version.md b/doc/development/database/new_database_migration_version.md
new file mode 100644
index 00000000000..b97ecd83f37
--- /dev/null
+++ b/doc/development/database/new_database_migration_version.md
@@ -0,0 +1,64 @@
+---
+stage: Data Stores
+group: Database
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Introducing a new database migration version
+
+At GitLab we've added many helpers for the database migrations to help developers manipulate
+the schema and data of tables on a large scale like on GitLab.com. To avoid the repetitive task
+of including the helpers into each database migration, we use a subclass of the Rails `ActiveRecord::Migration`
+class that we use for all of our database migrations. This subclass is `Gitlab::Database::Migration`, and it already
+includes all the helpers that developers can use. You can see many use cases of helpers built
+in-house in [Avoiding downtime in migrations](avoiding_downtime_in_migrations.md).
+
+Sometimes, we need to add or modify existing an helper's functionality without having a reverse effect on all the
+previous database migrations. That's why we introduced versioning to `Gitlab::Database::Migration`. Now,
+each database migration can inherit the latest version of this class at the time of the writing the database migration.
+After we add a new feature, those old database migrations are no longer affected. We usually
+refer to the version using `Gitlab::Database::Migration[2.1]`, where `2.1` is the current version.
+
+Because we are chasing a moving target, adding a new migration and deprecating older versions
+can be challenging. Database migrations are introduced every day, and this can break the pipeline.
+In this document, we explain a two-step method to add a new database migration version.
+
+1. [Introduce a new version, and keep the older version allowed](#introduce-a-new-version-and-keep-the-older-version-allowed)
+1. [Prevent the usage of the older database migration version](#prevent-the-usage-of-the-older-database-migration-version)
+
+## Introduce a new version, and keep the older version allowed
+
+1. The new version can be added to the
+ [`migration.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/migration.rb)
+ class, along with any new helpers that should be included in the new version.
+ Make sure that `current_version` refers to this new version. For example:
+
+ ```ruby
+ class V2_2 < V2_1 # rubocop:disable Naming/ClassAndModuleCamelCase
+ include Gitlab::Database::MigrationHelpers::ANY_NEW_HELPER
+ end
+
+ def self.current_version
+ 2.2
+ end
+ ```
+
+1. Update all the examples in the documentation to refer to the new database
+ migration version `Gitlab::Database::Migration[2.2]`.
+1. Make sure that [`migration_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/db/migration_spec.rb)
+ doesn't fail for the new database migrations by adding an open date rate for
+ the **new database version**.
+
+## Prevent the usage of the older database migration version
+
+After some time passes, and ensuring all developers are using the
+new database migration version in their merge requests, prevent the older
+version from being used:
+
+1. Close the date range in [`migration_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/db/migration_spec.rb)
+ for the older database version.
+1. Modify the
+ [`RuboCop::Cop::Migration::VersionedMigrationClass`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/cop/migration/versioned_migration_class.rb)
+ and [its owned tests](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/rubocop/cop/migration/versioned_migration_class_spec.rb).
+1. Communicate this change on our Slack `#backend` and `#database` channels and
+ [Engineering Week-in-Review document](https://about.gitlab.com/handbook/engineering/#communication).
diff --git a/doc/development/database/pagination_guidelines.md b/doc/development/database/pagination_guidelines.md
index aeab45e2158..04a2a8cdf9c 100644
--- a/doc/development/database/pagination_guidelines.md
+++ b/doc/development/database/pagination_guidelines.md
@@ -66,7 +66,7 @@ Offset-based pagination is the easiest way to paginate over records, however, it
- Avoid using page numbers, use next and previous page buttons.
- Keyset pagination doesn't support page numbers.
- For APIs, advise against building URLs for the next page by "hand".
- - Promote the usage of the [`Link` header](../../api/index.md#pagination-link-header) where the URLs for the next and previous page are provided by the backend.
+ - Promote the usage of the [`Link` header](../../api/rest/index.md#pagination-link-header) where the URLs for the next and previous page are provided by the backend.
- This way changing the URL structure is possible without breaking backward compatibility.
NOTE:
diff --git a/doc/development/database/query_recorder.md b/doc/development/database/query_recorder.md
index dfaaf8afcde..bae211c1618 100644
--- a/doc/development/database/query_recorder.md
+++ b/doc/development/database/query_recorder.md
@@ -148,3 +148,4 @@ There are multiple ways to find the source of queries.
- [Performance guidelines](../performance.md)
- [Merge request performance guidelines - Query counts](../merge_request_concepts/performance.md#query-counts)
- [Merge request performance guidelines - Cached queries](../merge_request_concepts/performance.md#cached-queries)
+- [RedisCommands::Recorder](../redis.md#n1-calls-problem) For testing `N+1` calls in Redis
diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md
index 30131fc0347..0d5e3c233f6 100644
--- a/doc/development/database/table_partitioning.md
+++ b/doc/development/database/table_partitioning.md
@@ -307,44 +307,12 @@ class PrepareIndexesForPartitioning < Gitlab::Database::Migration[2.1]
end
```
-### Step 3 - Swap primary key
+### Step 3 - Enforce unique constraint
-Swap the primary key including the partitioning key column. For example, in a rails migration:
-
-```ruby
-class PreparePrimaryKeyForPartitioning < Gitlab::Database::Migration[2.1]
- disable_ddl_transaction!
-
- TABLE_NAME = :table_name
- PRIMARY_KEY = :primary_key
- OLD_INDEX_NAME = :old_index_name
- NEW_INDEX_NAME = :new_index_name
-
- def up
- swap_primary_key(TABLE_NAME, PRIMARY_KEY, NEW_INDEX_NAME)
- end
-
- def down
- add_concurrent_index(TABLE_NAME, :id, unique: true, name: OLD_INDEX_NAME)
- add_concurrent_index(TABLE_NAME, [:id, :partition_id], unique: true, name: NEW_INDEX_NAME)
-
- unswap_primary_key(TABLE_NAME, PRIMARY_KEY, OLD_INDEX_NAME)
- end
-end
-```
-
-NOTE:
-Do not forget to set the primary key explicitly in your model as `ActiveRecord` does not support composite primary keys.
-
-```ruby
-class Model < ApplicationRecord
- self.primary_key = :id
-end
-```
-
-### Step 4 - Enforce unique constraint
-
-Enforce unique indexes including the partitioning key column. For example, in a rails migration:
+Change all unique indexes to include the partitioning key column,
+including the primary key index. You can start by adding an unique
+index on `[primary_key_column, :partition_id]`, which will be
+required for the next two steps. For example, in a rails migration:
```ruby
class PrepareUniqueContraintForPartitioning < Gitlab::Database::Migration[2.1]
@@ -355,20 +323,20 @@ class PrepareUniqueContraintForPartitioning < Gitlab::Database::Migration[2.1]
NEW_UNIQUE_INDEX_NAME = :new_index_name
def up
- add_concurrent_index(TABLE_NAME, [:some_column, :partition_id], unique: true, name: NEW_UNIQUE_INDEX_NAME)
+ add_concurrent_index(TABLE_NAME, [:id, :partition_id], unique: true, name: NEW_UNIQUE_INDEX_NAME)
remove_concurrent_index_by_name(TABLE_NAME, OLD_UNIQUE_INDEX_NAME)
end
def down
- add_concurrent_index(TABLE_NAME, :some_column, unique: true, name: OLD_UNIQUE_INDEX_NAME)
+ add_concurrent_index(TABLE_NAME, :id, unique: true, name: OLD_UNIQUE_INDEX_NAME)
remove_concurrent_index_by_name(TABLE_NAME, NEW_UNIQUE_INDEX_NAME)
end
end
```
-### Step 5 - Enforce foreign key constraint
+### Step 4 - Enforce foreign key constraint
Enforce foreign keys including the partitioning key column. For example, in a rails migration:
@@ -380,7 +348,7 @@ class PrepareForeignKeyForPartitioning < Gitlab::Database::Migration[2.1]
TARGET_TABLE_NAME = :target_table_name
COLUMN = :foreign_key_id
TARGET_COLUMN = :id
- CONSTRAINT_NAME = :fk_365d1db505_p
+ FK_NAME = :fk_365d1db505_p
PARTITION_COLUMN = :partition_id
def up
@@ -391,14 +359,17 @@ class PrepareForeignKeyForPartitioning < Gitlab::Database::Migration[2.1]
target_column: [PARTITION_COLUMN, TARGET_COLUMN],
validate: false,
on_update: :cascade,
- name: CONSTRAINT_NAME
+ name: FK_NAME
)
- validate_foreign_key(TARGET_TABLE_NAME, CONSTRAINT_NAME)
+ # This should be done in a separate post migration when dealing with a high traffic table
+ validate_foreign_key(TABLE_NAME, [PARTITION_COLUMN, COLUMN], name: FK_NAME)
end
def down
- drop_constraint(TARGET_TABLE_NAME, CONSTRAINT_NAME)
+ with_lock_retries do
+ remove_foreign_key_if_exists(SOURCE_TABLE_NAME, name: FK_NAME)
+ end
end
end
```
@@ -410,6 +381,48 @@ result in a `Key is still referenced from table ...` error and updating the
partition column on the source table would raise a
`Key is not present in table ...` error.
+This migration can be automatically generated using:
+
+```shell
+./scripts/partitioning/generate-fk --source source_table_name --target target_table_name
+```
+
+### Step 5 - Swap primary key
+
+Swap the primary key including the partitioning key column. This can be done only after
+including the partition key for all references foreign keys. For example, in a rails migration:
+
+```ruby
+class PreparePrimaryKeyForPartitioning < Gitlab::Database::Migration[2.1]
+ disable_ddl_transaction!
+
+ TABLE_NAME = :table_name
+ PRIMARY_KEY = :primary_key
+ OLD_INDEX_NAME = :old_index_name
+ NEW_INDEX_NAME = :new_index_name
+
+ def up
+ swap_primary_key(TABLE_NAME, PRIMARY_KEY, NEW_INDEX_NAME)
+ end
+
+ def down
+ add_concurrent_index(TABLE_NAME, :id, unique: true, name: OLD_INDEX_NAME)
+ add_concurrent_index(TABLE_NAME, [:id, :partition_id], unique: true, name: NEW_INDEX_NAME)
+
+ unswap_primary_key(TABLE_NAME, PRIMARY_KEY, OLD_INDEX_NAME)
+ end
+end
+```
+
+NOTE:
+Do not forget to set the primary key explicitly in your model as `ActiveRecord` does not support composite primary keys.
+
+```ruby
+class Model < ApplicationRecord
+ self.primary_key = :id
+end
+```
+
### Step 6 - Create parent table and attach existing table as the initial partition
You can now create the parent table attaching the existing table as the initial
@@ -465,7 +478,7 @@ class ConvertTableToListPartitioning < Gitlab::Database::Migration[2.1]
table_name: TABLE_NAME,
partitioning_column: PARTITION_COLUMN,
parent_table_name: PARENT_TABLE_NAME,
- initial_partitioning_value: FIRST_PARTITION
+ initial_partitioning_value: FIRST_PARTITION,
lock_tables: [TABLE_FK, TABLE_NAME]
)
end
diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md
index f4af005b849..e532fa82011 100644
--- a/doc/development/deprecation_guidelines/index.md
+++ b/doc/development/deprecation_guidelines/index.md
@@ -77,7 +77,7 @@ However, at GitLab, we [give agency](https://about.gitlab.com/handbook/values/#g
Generally, feature or configuration can be removed/changed only on major release.
It also should be [deprecated in advance](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations).
-For API removals, see the [GraphQL](../../api/graphql/index.md#deprecation-and-removal-process) and [GitLab API](../../api/index.md#compatibility-guidelines) guidelines.
+For API removals, see the [GraphQL](../../api/graphql/index.md#deprecation-and-removal-process) and [GitLab API](../../api/rest/index.md#compatibility-guidelines) guidelines.
For configuration removals, see the [Omnibus deprecation policy](../../administration/package_information/deprecation_policy.md).
diff --git a/doc/development/development_processes.md b/doc/development/development_processes.md
index e1df3b55d06..c5d60d18419 100644
--- a/doc/development/development_processes.md
+++ b/doc/development/development_processes.md
@@ -18,7 +18,7 @@ Must-reads:
- [Database review guidelines](database_review.md) for reviewing
database-related changes and complex SQL queries, and having them reviewed
- [Secure coding guidelines](secure_coding_guidelines.md)
-- [Pipelines for the GitLab project](pipelines.md)
+- [Pipelines for the GitLab project](pipelines/index.md)
Complementary reads:
diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md
index 0fab693fdee..37be2178592 100644
--- a/doc/development/documentation/feature_flags.md
+++ b/doc/development/documentation/feature_flags.md
@@ -13,9 +13,29 @@ feature flag depends on its state (enabled or disabled). When the state
changes, the developer who made the change **must update the documentation**
accordingly.
-Every feature introduced to the codebase, even if it's behind a feature flag,
-must be documented. For context, see the
-[latest merge request that updated this guideline](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47917#note_459984428).
+## When to document features behind a feature flag
+
+Every feature introduced to the codebase, even if it's behind a disabled feature flag,
+must be documented. For more information, see
+[the discussion that led to this decision](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47917#note_459984428).
+
+When the feature is [implemented over multiple merge requests](../feature_flags/index.md#feature-flags-in-gitlab-development),
+discuss the exact documentation plan with your technical writer. Consider
+creating a dedicated documentation issue if the feature:
+
+- Is far-reaching (makes changes across many areas of GitLab), like navigation changes.
+- Includes many MRs.
+- Affects more than a few documentation pages.
+- Is not fully functional if the feature flag is enabled for testing.
+
+If you and the technical writer agree to delay the product documentation (for example, until after testing),
+collaborate with the TW to create a documentation issue detailing the plan for adding the content.
+The PM and EM should be included in the discussions to make sure the task of adding the documentation is assigned and scheduled.
+Despite any planned delays, every feature flag in the codebase is automatically listed at
+<https://docs.gitlab.com/ee/user/feature_flags.html#available-feature-flags>,
+even when the feature is not fully functional.
+
+## How to add feature flag documentation
When you document feature flags, you must:
diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md
index 8a5a993d913..fd9e885e097 100644
--- a/doc/development/documentation/index.md
+++ b/doc/development/documentation/index.md
@@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
description: Learn how to contribute to GitLab Documentation.
---
-# GitLab documentation
+# Contribute to the GitLab documentation
The GitLab documentation is [intended as the single source of truth (SSOT)](https://about.gitlab.com/handbook/documentation/) for information about how to configure, use, and troubleshoot GitLab. The documentation contains use cases and usage instructions for every GitLab feature, organized by product area and subject. This includes topics and workflows that span multiple GitLab features and the use of GitLab with other applications.
@@ -37,7 +37,7 @@ Documentation issues and merge requests are part of their respective repositorie
### Branch naming
-The [CI pipeline for the main GitLab project](../pipelines.md) is configured to automatically
+The [CI pipeline for the main GitLab project](../pipelines/index.md) is configured to automatically
run only the jobs that match the type of contribution. If your contribution contains
**only** documentation changes, then only documentation-related jobs run, and
the pipeline completes much faster than a code contribution.
diff --git a/doc/development/documentation/redirects.md b/doc/development/documentation/redirects.md
index 1118dc97926..4cfe8be713a 100644
--- a/doc/development/documentation/redirects.md
+++ b/doc/development/documentation/redirects.md
@@ -31,6 +31,12 @@ Add a redirect to ensure:
Be sure to assign a technical writer to any merge request that moves, renames, or deletes a page.
Technical Writers can help with any questions and can review your change.
+NOTE:
+When you change the filename of a page, the Google Analytics are removed
+from the content audit and the page views start from scratch.
+If you want to change the filename, edit the page first,
+so you can ensure the new page name is as accurate as possible.
+
## Types of redirects
There are two types of redirects:
diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md
index 21c8c8543ab..a92d58ead96 100644
--- a/doc/development/documentation/restful_api_styleguide.md
+++ b/doc/development/documentation/restful_api_styleguide.md
@@ -9,7 +9,8 @@ description: 'Writing styles, markup, formatting, and other standards for the Gi
REST API resources are documented in Markdown under
[`/doc/api`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api). Each
-resource has its own Markdown file, which is linked from `api_resources.md`.
+resource has its own Markdown file, which is linked from
+[`api_resources.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/api/api_resources.md).
When modifying the Markdown, also update the corresponding
[OpenAPI definition](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api/openapi)
@@ -35,6 +36,8 @@ In the Markdown doc for a resource (AKA endpoint):
Put the badge in the **Attribute** column, like the
`**(<tier>)**` code in the following template.
+After a new API documentation page is added, [add an entry in the global navigation](site_architecture/global_nav.md#add-a-navigation-entry). [Example](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/3497).
+
## API topic template
Use the following template to help you get started. Be sure to list any
@@ -66,7 +69,7 @@ Supported attributes:
| `attribute` | datatype | No | Detailed description. |
| `attribute` | datatype | No | Detailed description. |
-If successful, returns [`<status_code>`](../../api/index.md#status-codes) and the following
+If successful, returns [`<status_code>`](rest/index.md#status-codes) and the following
response attributes:
| Attribute | Type | Description |
@@ -159,10 +162,10 @@ For information about writing attribute descriptions, see the [GraphQL API descr
## Response body description
Start the description with the following sentence, replacing `status code` with the
-relevant [HTTP status code](../../api/index.md#status-codes), for example:
+relevant [HTTP status code](../../api/rest/index.md#status-codes), for example:
```markdown
-If successful, returns [`200 OK`](../../api/index.md#status-codes) and the
+If successful, returns [`200 OK`](../../api/rest/index.md#status-codes) and the
following response attributes:
```
diff --git a/doc/development/documentation/review_apps.md b/doc/development/documentation/review_apps.md
index 3cf77fda22b..adc9d727844 100644
--- a/doc/development/documentation/review_apps.md
+++ b/doc/development/documentation/review_apps.md
@@ -32,6 +32,8 @@ The `review-docs-deploy*` job triggers a cross project pipeline and builds the
docs site with your changes. When the pipeline finishes, the review app URL
appears in the merge request widget. Use it to navigate to your changes.
+The `review-docs-cleanup` job is triggered automatically on merge. This job deletes the review app.
+
You must have the Developer role in the project. Users without the Developer role, such
as external contributors, cannot run the manual job. In that case, ask someone from
the GitLab team to run the job.
diff --git a/doc/development/documentation/styleguide/img/admin_access_level.png b/doc/development/documentation/styleguide/img/admin_access_level.png
index 191ba78cd6c..f31e4c65fe5 100644
--- a/doc/development/documentation/styleguide/img/admin_access_level.png
+++ b/doc/development/documentation/styleguide/img/admin_access_level.png
Binary files differ
diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md
index 921bb13721b..74437ea46c9 100644
--- a/doc/development/documentation/styleguide/index.md
+++ b/doc/development/documentation/styleguide/index.md
@@ -182,6 +182,7 @@ the page is rendered to HTML. There can be only **one** level 1 heading per page
Heading levels greater than `H5` do not display in the right sidebar navigation.
- Do not skip a level. For example: `##` > `####`.
- Leave one blank line before and after the topic title.
+- If you use code in topic titles, ensure the code is in backticks.
### Backticks in Markdown
@@ -320,7 +321,7 @@ You can use these fake tokens as examples:
| Application ID | `2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6` |
| Application secret | `04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df` |
| CI/CD variable | `Li8j-mLUVA3eZYjPfd_H` |
-| Specific runner token | `yrnZW46BrtBFqM7xDzE7dddd` |
+| Project runner token | `yrnZW46BrtBFqM7xDzE7dddd` |
| Shared runner token | `6Vk7ZsosqQyfreAxXTZr` |
| Trigger token | `be20d8dcc028677c931e04f3871a9b` |
| Webhook secret token | `6XhDroRcYPM5by_h-HLY` |
@@ -943,7 +944,7 @@ To open the Admin Area:
To select your avatar:
```markdown
-1. On the top bar, in the top right corner, select your avatar.
+1. On the top bar, in the upper-right corner, select your avatar.
```
To save the selection in some dropdown lists:
diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md
index fcebe3b3649..e64fd4df7ff 100644
--- a/doc/development/documentation/styleguide/word_list.md
+++ b/doc/development/documentation/styleguide/word_list.md
@@ -14,6 +14,7 @@ recommends these word choices. In addition:
[top misused terms](https://about.gitlab.com/handbook/communication/top-misused-terms/).
- The documentation [style guide](../styleguide#language) includes details
about language and capitalization.
+- The GitLab handbook provides guidance on the [use of third-party trademarks](https://about.gitlab.com/handbook/legal/policies/product-third-party-trademarks-guidelines/#process-for-adding-third-party-trademarks-to-gitlab).
For guidance not on this page, we defer to these style guides:
@@ -60,7 +61,7 @@ Instead of:
## access level
Access levels are different than [roles](#roles) or [permissions](#permissions).
-When you create a user, you choose an access level: **Regular**, **Auditor**, or **Admin**.
+When you create a user, you choose an access level: **Regular**, **Auditor**, or **Administrator**.
Capitalize these words when you refer to the UI. Otherwise use lowercase.
@@ -151,6 +152,15 @@ Use uppercase for **Alpha**. For example: **The XYZ feature is in Alpha.** or **
You might also want to link to [this section](../../../policy/alpha-beta-support.md#alpha-features)
in the handbook when writing about Alpha features.
+## analytics
+
+Use lowercase for **analytics** and its variations, like **contribution analytics** and **issue analytics**.
+However, if the UI has different capitalization, make the documentation match the UI.
+
+For example:
+
+- You can view merge request analytics for a project. They are displayed on the Merge Request Analytics dashboard.
+
## and/or
Instead of **and/or**, use **or** or rewrite the sentence to spell out both options.
@@ -306,6 +316,20 @@ Use title case for the GitLab Container Registry.
Do not use **currently** when talking about the product or its features. The documentation describes the product as it is today.
([Vale](../testing.md#vale) rule: [`CurrentStatus.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/CurrentStatus.yml))
+## data
+
+Use **data** as a singular noun.
+
+Use:
+
+- Data is collected.
+- The data shows a performance increase.
+
+Instead of:
+
+- Data are collected.
+- The data show a performance increase.
+
## default branch
Use **default branch** to refer generically to the primary branch in the repository.
@@ -351,6 +375,12 @@ Use **inactive** or **off** instead. ([Vale](../testing.md#vale) rule: [`Inclusi
Use **prevent** instead of **disallow**. ([Vale](../testing.md#vale) rule: [`Substitutions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Substitutions.yml))
+## Docker-in-Docker, `dind`
+
+Use **Docker-in-Docker** when you are describing running a Docker container by using the Docker executor.
+
+Use `dind` in backticks to describe the container name: `docker:dind`. Otherwise, spell it out.
+
## downgrade
To be more upbeat and precise, do not use **downgrade**. Focus instead on the action the user is taking.
@@ -1046,6 +1076,14 @@ The search results are displayed on a search page.
Searching is different from [filtering](#filter).
+## seats
+
+When referring to the subscription billing model:
+
+- For GitLab SaaS, use **seats**. Customers purchase seats. Users occupy seats when they are invited
+to a group, with some [exceptions](../../../subscriptions/gitlab_com/index.md#how-seat-usage-is-determined).
+- For GitLab self-managed, use **users**. Customers purchase subscriptions for a specified number of **users**.
+
## section
Use **section** to describe an area on a page. For example, if a page has lines that separate the UI
@@ -1260,7 +1298,7 @@ Do not use **update** for any other case. Instead, use **upgrade**.
Use **upgrade** for:
- Choosing a higher subscription tier (Premium or Ultimate).
-- Installing a newer **major** (13.0, 14.0) or **minor** (13.8, 14.5) version of GitLab.
+- Installing a newer **major** (13.0) or **minor** (13.2) version of GitLab.
For example:
@@ -1274,6 +1312,12 @@ you're talking about the product version or the subscription tier.
See also [downgrade](#downgrade) and [roll back](#roll-back).
+## upper left, upper right
+
+Use **upper left** and **upper right** instead of **top left** and **top right**. Hyphenate as adjectives (for example, **upper-left corner**).
+
+For details, see the [Microsoft style guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/u/upper-left-upper-right).
+
## useful
Do not use **useful**. If the user doesn't find the process to be useful, we lose their trust. ([Vale](../testing.md#vale) rule: [`Simplicity.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Simplicity.yml))
@@ -1342,16 +1386,19 @@ in present tense, active voice.
## you, your, yours
-Use **you**, **your**, and **yours** instead of **the user** and **the user's**.
-Documentation should be from the [point of view](https://design.gitlab.com/content/voice-and-tone#point-of-view) of the reader.
+Use **you** instead of **the user**, **the administrator** or **the customer**.
+Documentation should speak directly to the user, whether that user is someone installing the product,
+configuring it, administering it, or using it.
Use:
- You can configure a pipeline.
+- You can reset a user's password. (In content for an administrator)
Instead of:
- Users can configure a pipeline.
+- Administrators can reset a user's password.
## you can
diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md
index 58584b5168b..4f61b2424eb 100644
--- a/doc/development/documentation/testing.md
+++ b/doc/development/documentation/testing.md
@@ -13,7 +13,7 @@ processes similar to those used for code to maintain standards and quality of do
We have tests:
- To lint the words and structure of the documentation.
-- To check the validity of internal links within the documentation suite.
+- To check the validity of internal links in the documentation suite.
- To check the validity of links from UI elements, such as files in `app/views` files.
For the specifics of each test run in our CI/CD pipelines, see the configuration for those tests
@@ -51,7 +51,7 @@ To run tests locally, it's important to:
Lint checks are performed by the [`lint-doc.sh`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/lint-doc.sh)
script and can be executed as follows:
-1. Navigate to the `gitlab` directory.
+1. Go to the `gitlab` directory.
1. Run:
```shell
@@ -113,7 +113,7 @@ Even if a specific broken link appears multiple times on a page, the test report
To execute documentation link tests locally:
-1. Navigate to the [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) directory.
+1. Go to the [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) directory.
1. Run the following commands:
```shell
@@ -243,7 +243,9 @@ This configuration is also used in build pipelines.
You can use markdownlint:
-- [On the command line](https://github.com/igorshubovych/markdownlint-cli#markdownlint-cli--).
+- On the command line, with either:
+ - [`markdownlint-cli`](https://github.com/igorshubovych/markdownlint-cli#markdownlint-cli).
+ - [`markdownlint-cli2`](https://github.com/DavidAnson/markdownlint-cli2#markdownlint-cli2).
- [In a code editor](#configure-editors).
- [In a `pre-push` hook](#configure-pre-push-hooks).
@@ -281,7 +283,7 @@ You can use Vale:
Vale returns three types of results:
- **Error** - For branding guidelines, trademark guidelines, and anything that causes content on
- the docs site to render incorrectly.
+ the documentation site to render incorrectly.
- **Warning** - For general style guide rules, tenets, and best practices.
- **Suggestion** - For technical writing style preferences that may require refactoring of documentation or updates to an exceptions list.
@@ -366,23 +368,34 @@ In general, follow these guidelines:
### Install linters
-At a minimum, install [markdownlint](#markdownlint) and [Vale](#vale) to match the checks run in
-build pipelines:
+At a minimum, install [markdownlint](#markdownlint) and [Vale](#vale) to match the checks run in build pipelines.
-1. Install `markdownlint-cli`:
+#### Install markdownlint
- ```shell
- yarn global add markdownlint-cli
- ```
+You can install either `markdownlint-cli` or `markdownlint-cli2` to run `markdownlint`.
+
+To install `markdownlint-cli`, run:
+
+```shell
+yarn global add markdownlint-cli
+```
+
+To install `markdownlint-cli2`, run:
- We recommend installing the version of `markdownlint-cli`
- [used (see `variables:` section)](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/.gitlab-ci.yml) when building
- the `image:docs-lint-markdown`.
+```shell
+yarn global add markdownlint-cli2
+```
+
+You should install the version of `markdownlint-cli` or `markdownlint-cli2`
+[used (see `variables:` section)](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/.gitlab-ci.yml) when building
+the `image:docs-lint-markdown`.
+
+#### Install Vale
-1. Install [`vale`](https://github.com/errata-ai/vale/releases). To install for:
+To install Install [`vale`](https://github.com/errata-ai/vale/releases) for:
- - macOS using `brew`, run: `brew install vale`.
- - Linux, use your distribution's package manager or a [released binary](https://github.com/errata-ai/vale/releases).
+- macOS using `brew`, run: `brew install vale`.
+- Linux, use your distribution's package manager or a [released binary](https://github.com/errata-ai/vale/releases).
These tools can be [integrated with your code editor](#configure-editors).
@@ -391,16 +404,18 @@ These tools can be [integrated with your code editor](#configure-editors).
It's important to use linter versions that are the same or newer than those run in
CI/CD. This provides access to new features and possible bug fixes.
-To match the versions of `markdownlint-cli` and `vale` used in the GitLab projects, refer to the
+To match the versions of `markdownlint-cli` (or `markdownlint-cli2`) and `vale` used in the GitLab projects, refer to the
[versions used (see `variables:` section)](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/.gitlab-ci.yml)
when building the `image:docs-lint-markdown` Docker image containing these tools for CI/CD.
-| Tool | Version | Command | Additional information |
-|--------------------|-----------|--------------------------------------------------------|------------------------|
-| `markdownlint-cli` | Latest | `yarn global add markdownlint-cli` | None. |
-| `markdownlint-cli` | Specific | `yarn global add markdownlint-cli@0.23.2` | The `@` indicates a specific version, and this example updates the tool to version `0.23.2`. |
-| Vale | Latest | `brew update && brew upgrade vale` | This command is for macOS only. |
-| Vale | Specific | Not applicable. | Binaries can be [directly downloaded](https://github.com/errata-ai/vale/releases). |
+| Tool | Version | Command | Additional information |
+|:--------------------|:---------|:------------------------------------------|:---------------------------------------------------------------------------------------------|
+| `markdownlint-cli` | Latest | `yarn global add markdownlint-cli` | None. |
+| `markdownlint-cli2` | Latest | `yarn global add markdownlint-cli2` | None. |
+| `markdownlint-cli` | Specific | `yarn global add markdownlint-cli@0.23.2` | The `@` indicates a specific version, and this example updates the tool to version `0.23.2`. |
+| `markdownlint-cli` | Specific | `yarn global add markdownlint-cli@0.6.0` | The `@` indicates a specific version, and this example updates the tool to version `0.6.0`. |
+| Vale | Latest | `brew update && brew upgrade vale` | This command is for macOS only. |
+| Vale | Specific | Not applicable. | Binaries can be [directly downloaded](https://github.com/errata-ai/vale/releases). |
### Configure editors
@@ -410,6 +425,15 @@ command line.
To configure markdownlint in your editor, install one of the following as appropriate:
- Sublime Text [`SublimeLinter-contrib-markdownlint` package](https://packagecontrol.io/packages/SublimeLinter-contrib-markdownlint).
+ This package uses `markdownlint-cli` by default, but can be configured to use `markdownlint-cli2` with this
+ SublimeLinter configuration:
+
+ ```json
+ "markdownlint": {
+ "executable": [ "markdownlint-cli2" ]
+ }
+ ```
+
- Visual Studio Code [`DavidAnson.vscode-markdownlint` extension](https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint).
- Vim [ALE plugin](https://github.com/dense-analysis/ale).
@@ -463,7 +487,7 @@ Git [pre-push hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) a
- Avoid pushing a branch if failures occur with these tests.
[`lefthook`](https://github.com/Arkweid/lefthook) is a Git hooks manager, making configuring,
-installing, and removing Git hooks easy.
+installing, and removing Git hooks simpler.
Configuration for `lefthook` is available in the [`lefthook.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lefthook.yml)
file for the [`gitlab`](https://gitlab.com/gitlab-org/gitlab) project.
@@ -509,7 +533,7 @@ You can set Visual Studio Code to display only a subset of Vale alerts when view
To display only a subset of Vale alerts when running Vale from the command line, use
the `--minAlertLevel` flag, which accepts `error`, `warning`, or `suggestion`. Combine it with `--config`
-to point to the configuration file within the project, if needed:
+to point to the configuration file in the project, if needed:
```shell
vale --config .vale.ini --minAlertLevel error doc/**/*.md
diff --git a/doc/development/documentation/topic_types/task.md b/doc/development/documentation/topic_types/task.md
index 0dba3e079b6..8d23a5f322e 100644
--- a/doc/development/documentation/topic_types/task.md
+++ b/doc/development/documentation/topic_types/task.md
@@ -45,7 +45,7 @@ To create an issue:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Issues > List**.
-1. In the top right corner, select **New issue**.
+1. In the upper-right corner, select **New issue**.
1. Complete the fields. (If you have reference content that lists each field, link to it here.)
1. Select **Create issue**.
@@ -57,8 +57,70 @@ The issue is created. You can view it by going to **Issues > List**.
For the title text, use the structure `active verb` + `noun`.
For example, `Create an issue`.
-If you have several tasks on a page that share prerequisites, you can use the title
-`Prerequisites` and link to it.
+If several tasks on a page share prerequisites, you can create a separate
+topic with the title `Prerequisites`.
+
+### When more than one way exists to perform a task
+
+If more than one way exists to perform a task in the UI, you should
+document the primary way only.
+
+However, sometimes you must document multiple ways to perform a task.
+When this situation occurs:
+
+- Introduce the task as usual. Then, for each way of performing the task, add a topic title.
+- Nest the topic titles one level below the task topic title.
+- List the tasks in descending order, with the most likely method first.
+- Make the task titles as brief as possible. When possible,
+ use `infinitive` + `noun`.
+
+Here is an example.
+
+```markdown
+# Change the default branch name
+
+You can change the default branch name for the instance or group.
+If the name is set for the instance, you can override it for a group.
+
+## For the instance
+
+Prerequisites:
+
+- You must have at least the Maintainer role for the instance.
+
+To change the default branch name for an instance:
+
+1. Step.
+1. Step.
+
+## For the group
+
+Prerequisites:
+
+- You must have at least the Developer role for the group.
+
+To change the default branch name for a group:
+
+1. Step.
+1. Step.
+```
+
+### To perform the task in the UI and API
+
+Usually an API exists to perform the same task that you perform in the UI.
+
+When this situation occurs:
+
+- Do not use a separate heading for a one-sentence link to the API.
+- Do not include API examples in the **Use GitLab** documentation. API examples
+ belong in the API documentation. If you have GraphQL examples, put them on
+ their own page, because the API documentation might move some day.
+- Do not mention the API if you do not need to. Users can search for
+ the API documentation, and extra linking adds clutter.
+- If someone feels strongly that you mention the API, at the end
+ of the UI task, add this sentence:
+
+ `To create an issue, you can also [use the API](link).`
## Task introductions
diff --git a/doc/development/documentation/versions.md b/doc/development/documentation/versions.md
index 334dcd73ea5..30a0d4d4cf4 100644
--- a/doc/development/documentation/versions.md
+++ b/doc/development/documentation/versions.md
@@ -13,7 +13,7 @@ including when features were introduced and when they were updated or removed.
## View older documentation versions
Previous versions of the documentation are available on `docs.gitlab.com`.
-To view a previous version, select the **Versions** button in the top right.
+To view a previous version, select the **Versions** button in the upper right.
To view versions that are not available on `docs.gitlab.com`:
@@ -118,6 +118,22 @@ To deprecate a page or topic:
You can add any additional context-specific details that might help users.
+1. Add the following HTML comments above and below the content.
+ For the `remove_date`, set a date three months after the release where it
+ was deprecated.
+
+ ```markdown
+ <!--- start_remove The following content will be removed on remove_date: 'YYYY-MM-DD' -->
+
+ ## Title (deprecated) **(ULTIMATE SELF)**
+
+ WARNING:
+ This feature was [deprecated](<link-to-issue>) in GitLab 14.8
+ and is planned for removal in 15.4. Use [feature X](<link-to-issue>) instead.
+
+ <!--- end_remove -->
+ ```
+
1. Open a merge request to add the word `(deprecated)` to the left nav, after the page title.
### Remove a page
diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md
index 3c73030aceb..fe5453a4a58 100644
--- a/doc/development/documentation/workflow.md
+++ b/doc/development/documentation/workflow.md
@@ -15,21 +15,36 @@ Anyone can contribute to the GitLab documentation! You can create a merge reques
If you are working on a feature or enhancement, use the
[feature workflow process described in the GitLab Handbook](https://about.gitlab.com/handbook/product/ux/technical-writing/workflow/#documentation-for-a-product-change).
+## Do not use ChatGPT or AI-generated content for the docs
+
+GitLab documentation is distributed under the [CC BY-SA 4.0 license](https://creativecommons.org/licenses/by-sa/4.0/), which presupposes that GitLab owns the documentation.
+
+Under current law in the US and the EU, it’s possible that AI-generated works might either:
+
+- not be owned by anyone because they weren't created by a human, or
+- belong to the AI training data’s creator, if the AI verbatim reproduces content that it trained on
+
+If the documentation contains AI-generated content, GitLab probably wouldn't own this content, which would risk invalidating the CC BY-SA 4.0 license.
+
+Contributions to GitLab documentation are made under either our [DCO or our CLA terms](https://about.gitlab.com/community/contribute/dco-cla/). In both, contributors have to make certain certifications about the authorship of their work that they can't validly make for AI-generated text.
+
+For these reasons, do not add AI-generated content to the documentation.
+
## How to update the docs
If you are not a GitLab team member, or do not have the Developer role for the GitLab repository, to update GitLab documentation:
1. Select an [issue](https://about.gitlab.com/handbook/product/ux/technical-writing/#community-contribution-opportunities) you'd like to work on.
- - You don't need an issue to open a merge request.
- For a Hackathon, mention `@docs-hackathon` in a comment and ask for the issue to be assigned to you.
To be fair to other contributors, if you see someone has already asked to work on the issue, choose another issue.
+ - If you're not taking part in a Hackathon, you don't need an issue to open a merge request.
If you are looking for issues to work on and don't see any that suit you, you can always fix [Vale](testing.md#vale) issues.
1. Go to the [GitLab repository](https://gitlab.com/gitlab-org/gitlab).
-1. In the top right, select **Fork**. Forking makes a copy of the repository on GitLab.com.
+1. In the upper right, select **Fork**. Forking makes a copy of the repository on GitLab.com.
1. In your fork, find the documentation page in the `\doc` directory.
1. If you know Git, make your changes and open a merge request.
If not, follow these steps:
- 1. On the top right, select **Edit** if it is visible. If it is not, select the down arrow (**{chevron-lg-down}**) next to **Open in Web IDE** or **Gitpod**, and select **Edit**.
+ 1. In the upper right, select **Edit** if it is visible. If it is not, select the down arrow (**{chevron-lg-down}**) next to **Open in Web IDE** or **Gitpod**, and select **Edit**.
1. In the **Commit message** text box, enter a commit message. Use 3-5 words, start with a capital letter, and do not end with a period.
1. Select **Commit changes**.
1. On the left sidebar, select **Merge requests**.
diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md
index 961a47e005b..935964a9a90 100644
--- a/doc/development/elasticsearch.md
+++ b/doc/development/elasticsearch.md
@@ -276,6 +276,50 @@ class MigrationName < Elastic::Migration
end
```
+#### `Elastic::MigrationRemoveFieldsHelper`
+
+Removes specified fields from an index.
+
+Requires the `index_name`, `document_type` methods. If there is one field to remove, add the `field_to_remove` method, otherwise add `fields_to_remove` with an array of fields.
+
+Checks in batches if any documents that match `document_type` have the fields specified in Elasticsearch. If documents exist, uses a Painless script to perform `update_by_query`.
+
+```ruby
+class MigrationName < Elastic::Migration
+ include Elastic::MigrationRemoveFieldsHelper
+
+ batched!
+ throttle_delay 1.minute
+
+ private
+
+ def index_name
+ User.__elasticsearch__.index_name
+ end
+
+ def document_type
+ 'user'
+ end
+
+ def fields_to_remove
+ %w[two_factor_enabled has_projects]
+ end
+end
+```
+
+The default batch size is `10_000`. You can override this value by specifying `BATCH_SIZE`:
+
+```ruby
+class MigrationName < Elastic::Migration
+ include Elastic::MigrationRemoveFieldsHelper
+
+ batched!
+ BATCH_SIZE = 100
+
+ ...
+end
+```
+
#### `Elastic::MigrationObsolete`
Marks a migration as obsolete when it's no longer required.
diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md
index e185cd702a4..fd5b0ea15e6 100644
--- a/doc/development/experiment_guide/index.md
+++ b/doc/development/experiment_guide/index.md
@@ -63,3 +63,7 @@ We recommend the following workflow:
1. **If the experiment is a success**, designers add the new icon or illustration to the Pajamas UI kit as part of the cleanup process.
Engineers can then add it to the [SVG library](https://gitlab-org.gitlab.io/gitlab-svgs/) and modify the implementation based on the
[Frontend Development Guidelines](../fe_guide/icons.md#usage-in-hamlrails-2).
+
+## Related topics
+
+- [Experiments API](../../api/experiments.md)
diff --git a/doc/development/fe_guide/customizable_dashboards.md b/doc/development/fe_guide/customizable_dashboards.md
index 7e7718f8e60..26c73b26126 100644
--- a/doc/development/fe_guide/customizable_dashboards.md
+++ b/doc/development/fe_guide/customizable_dashboards.md
@@ -19,7 +19,7 @@ To use customizable dashboards:
1. Create your dashboard component.
1. Render an instance of `CustomizableDashboard`.
-1. Pass a list of widgets to render.
+1. Pass a list of panels to render.
For example, a customizable dashboard for users over time:
@@ -35,10 +35,10 @@ export default {
},
data() {
return {
- widgets: [
+ panels: [
{
- component: 'CubeLineChart', // The name of the widget component.
- title: s__('ProductAnalytics|Users / Time'), // The title shown on the widget component.
+ component: 'CubeLineChart', // The name of the panel component.
+ title: s__('ProductAnalytics|Users / Time'), // The title shown on the panel component.
// Gridstack settings based upon https://github.com/gridstack/gridstack.js/tree/master/doc#item-options.
// All values are grid row/column numbers up to 12.
// We use the default 12 column grid https://github.com/gridstack/gridstack.js#change-grid-columns.
@@ -50,22 +50,22 @@ export default {
xPos: 0,
yPos: 0,
},
- // Options that are used to set bespoke values for each widget.
- // Available customizations are determined by the widget itself.
+ // Options that are used to set bespoke values for each panel.
+ // Available customizations are determined by the panel itself.
customizations: {},
- // Chart options defined by the charting library being used by the widget.
+ // Chart options defined by the charting library being used by the panel.
chartOptions: {
xAxis: { name: __('Time'), type: 'time' },
yAxis: { name: __('Counts') },
},
- // The data for the widget.
- // This could be imported or in this case, a query passed to be used by the widgets API.
- // Each widget type determines how it handles this property.
+ // The data for the panel.
+ // This could be imported or in this case, a query passed to be used by the panels API.
+ // Each panel type determines how it handles this property.
data: {
query: {
users: {
- measures: ['Jitsu.count'],
- dimensions: ['Jitsu.eventType'],
+ measures: ['TrackedEvents.count'],
+ dimensions: ['TrackedEvents.eventType'],
},
},
},
@@ -78,20 +78,20 @@ export default {
<template>
<h1>{{ s__('ProductAnalytics|Analytics dashboard') }}</h1>
- <customizable-dashboard :widgets="widgets" />
+ <customizable-dashboard :panels="panels" />
</template>
```
-The widgets data can be retrieved from a file or API request, or imported through HTML data attributes.
+The panels data can be retrieved from a file or API request, or imported through HTML data attributes.
-For each widget, a `component` is defined. Each `component` is a component declaration and should be included in
-[`vue_shared/components/customizable_dashboard/widgets_base.vue`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/assets/javascripts/vue_shared/components/customizable_dashboard/widgets_base.vue)
+For each panel, a `component` is defined. Each `component` is a component declaration and should be included in
+[`vue_shared/components/customizable_dashboard/panels_base.vue`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/assets/javascripts/vue_shared/components/customizable_dashboard/panels_base.vue)
as a dynamic import, to keep the memory usage down until it is used.
For example:
```javascript
components: {
- CubeLineChart: () => import('ee/product_analytics/dashboards/components/widgets/cube_line_chart.vue')
+ CubeLineChart: () => import('ee/product_analytics/dashboards/components/panels/cube_line_chart.vue')
}
```
diff --git a/doc/development/fe_guide/design_anti_patterns.md b/doc/development/fe_guide/design_anti_patterns.md
index 07de86f5810..f087fbd8235 100644
--- a/doc/development/fe_guide/design_anti_patterns.md
+++ b/doc/development/fe_guide/design_anti_patterns.md
@@ -67,9 +67,7 @@ side-effects can be notoriously difficult to reason with.
### References
-To read more on this topic, check out the following references:
-
-- [GlobalVariablesAreBad from C2 wiki](https://wiki.c2.com/?GlobalVariablesAreBad)
+For more information, see [Global Variables Are Bad on the C2 wiki](https://wiki.c2.com/?GlobalVariablesAreBad).
## Singleton (Anti-pattern)
@@ -175,7 +173,7 @@ export const fuzzify = (id) => { /* ... */ };
#### Dependency Injection
[Dependency Injection](https://en.wikipedia.org/wiki/Dependency_injection) is an approach which breaks
-coupling by declaring a module's dependencies to be injected from outside the module (for example, through constructor parameters, a bona-fide Dependency Injection framework, and even Vue's `provide/inject`).
+coupling by declaring a module's dependencies to be injected from outside the module (for example, through constructor parameters, a bona-fide Dependency Injection framework, and even in Vue `provide/inject`).
```javascript
// bad - Vue component coupled to Singleton
diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md
index 0fec38b1200..8ae0458e47c 100644
--- a/doc/development/fe_guide/graphql.md
+++ b/doc/development/fe_guide/graphql.md
@@ -14,7 +14,7 @@ info: "See the Technical Writers assigned to Development Guidelines: https://abo
**General resources**:
- [📚 Official Introduction to GraphQL](https://graphql.org/learn/)
-- [📚 Official Introduction to Apollo](https://www.apollographql.com/tutorials/fullstack-quickstart/introduction)
+- [📚 Official Introduction to Apollo](https://www.apollographql.com/tutorials/fullstack-quickstart/01-introduction)
**GraphQL at GitLab**:
@@ -87,7 +87,7 @@ where needed.
You can check all existing queries and mutations on the right side
of GraphiQL in its **Documentation explorer**. You can also
write queries and mutations directly on the left tab and check
-their execution by clicking **Execute query** button on the top left:
+their execution by selecting **Execute query** in the upper left:
![GraphiQL interface](img/graphiql_explorer_v12_4.png)
diff --git a/doc/development/fe_guide/style/javascript.md b/doc/development/fe_guide/style/javascript.md
index 0536d1c5c77..3e3a79dd7bb 100644
--- a/doc/development/fe_guide/style/javascript.md
+++ b/doc/development/fe_guide/style/javascript.md
@@ -41,12 +41,12 @@ instead.
```javascript
// bad
-function a(p1, p2, p3) {
+function a(p1, p2, p3, p4) {
// ...
};
// good
-function a(p) {
+function a({ p1, p2, p3, p4 }) {
// ...
};
```
diff --git a/doc/development/fe_guide/style/scss.md b/doc/development/fe_guide/style/scss.md
index b84f41311b6..aed7310e95d 100644
--- a/doc/development/fe_guide/style/scss.md
+++ b/doc/development/fe_guide/style/scss.md
@@ -212,13 +212,3 @@ catch any warnings.
If the Rake task is throwing warnings you don't understand, SCSS Lint's
documentation includes [a full list of their rules](https://stylelint.io/user-guide/rules/).
-
-### Fixing issues
-
-If you want to automate changing a large portion of the codebase to conform to
-the SCSS style guide, you can use [CSSComb](https://github.com/csscomb/csscomb.js). First install
-[Node](https://github.com/nodejs/node) and [npm](https://www.npmjs.com/), then run `npm install csscomb -g` to install
-CSSComb globally (system-wide). Run it in the GitLab directory with
-`csscomb app/assets/stylesheets` to automatically fix issues with CSS/SCSS.
-
-Note that this doesn't fix every problem, but it should fix a majority.
diff --git a/doc/development/fe_guide/view_component.md b/doc/development/fe_guide/view_component.md
index f9d148d8b82..0245110ec75 100644
--- a/doc/development/fe_guide/view_component.md
+++ b/doc/development/fe_guide/view_component.md
@@ -10,8 +10,8 @@ ViewComponent is a framework for creating reusable, testable & encapsulated view
components with Ruby on Rails, without the need for a JavaScript framework like Vue.
They are rendered server-side and can be seamlessly used with template languages like [Haml](haml.md).
-Refer to the official [documentation](https://viewcomponent.org/) to learn more or
-watch this [introduction video](https://youtu.be/akRhUbvtnmo).
+For more information, see the [official documentation](https://viewcomponent.org/) or
+[this introduction video](https://youtu.be/akRhUbvtnmo).
## Browse components with Lookbook
diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md
index 779010b8aa1..cea47bc0e4c 100644
--- a/doc/development/fe_guide/vue.md
+++ b/doc/development/fe_guide/vue.md
@@ -72,6 +72,54 @@ The advantage of providing data from the DOM to the Vue instance through `props`
`provide` in the `render` function, instead of querying the DOM inside the main Vue
component, is that you avoid creating a fixture or an HTML element in the unit test.
+##### The `initSimpleApp` helper
+
+`initSimpleApp` is a helper function that streamlines the process of mounting a component in Vue.js. It accepts two arguments: a selector string representing the mount point in the HTML, and a Vue component.
+
+To use `initSimpleApp`, include the HTML element in the page with the appropriate selector and add a data-view-model attribute containing a JSON object. Then, import the desired Vue component and pass it along with the selector to `initSimpleApp`. This mounts the component at the specified location.
+
+`initSimpleApp` automatically retrieves the content of the data-view-model attribute as a JSON object and passes it as props to the mounted Vue component. This can be used to pre-populate the component with data.
+
+Example:
+
+```vue
+//my_component.vue
+<template>
+ <div>
+ <p>Prop1: {{ prop1 }}</p>
+ <p>Prop2: {{ prop2 }}</p>
+ </div>
+</template>
+
+<script>
+export default {
+ name: 'MyComponent',
+ props: {
+ prop1: {
+ type: String,
+ required: true
+ },
+ prop2: {
+ type: Number,
+ required: true
+ }
+ }
+}
+</script>
+```
+
+```html
+<div id="js-my-element" data-view-model='{"prop1": "my object", "prop2": 42 }'></div>
+```
+
+```javascript
+//index.js
+import MyComponent from './my_component.vue'
+import initSimpleApp from '~/helpers/init_simple_app_helper'
+
+initSimpleApp('#js-my-element', MyComponent)
+```
+
##### `provide` and `inject`
Vue supports dependency injection through [`provide` and `inject`](https://v2.vuejs.org/v2/api/#provide-inject).
diff --git a/doc/development/feature_categorization/index.md b/doc/development/feature_categorization/index.md
index 7f275f25c3d..dad94a2aae2 100644
--- a/doc/development/feature_categorization/index.md
+++ b/doc/development/feature_categorization/index.md
@@ -166,7 +166,7 @@ specific routes:
```ruby
module API
class Users < ::API::Base
- feature_category :users, ['/users/:id/custom_attributes', '/users/:id/custom_attributes/:key']
+ feature_category :user_profile, ['/users/:id/custom_attributes', '/users/:id/custom_attributes/:key']
end
end
```
@@ -176,7 +176,7 @@ Or the feature category can be specified in the action itself:
```ruby
module API
class Users < ::API::Base
- get ':id', feature_category: :users do
+ get ':id', feature_category: :user_profile do
end
end
end
@@ -191,7 +191,7 @@ within that class.
You must set feature category metadata for each RSpec example. This information is used for flaky test
issues to identify the group that owns the feature.
-The `feature_category` should be a value from [`categories.json`](https://about.gitlab.com/categories.json).
+The `feature_category` should be a value from [`config/feature_categories.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/feature_categories.yml).
The `feature_category` metadata can be set:
@@ -208,23 +208,21 @@ Example:
For examples that don't have a `feature_category` set we add a warning when running them in local environment.
-In order to disable the warning use `RSPEC_WARN_MISSING_FEATURE_CATEGORY=false` when running RSpec tests:
+To disable the warning use `RSPEC_WARN_MISSING_FEATURE_CATEGORY=false` when running RSpec tests:
```shell
RSPEC_WARN_MISSING_FEATURE_CATEGORY=false bin/rspec spec/<test_file>
```
-### Excluding specs from feature categorization
-
-In the rare case an action cannot be tied to a feature category this
-can be done using the `not_owned` feature category.
-
-```ruby
-RSpec.describe Utils, feature_category: :not_owned do
-```
+Additionally, we flag the offenses via `RSpec/MissingFeatureCategory` RuboCop rule.
### Tooling feature category
For Engineering Productivity internal tooling we use `feature_category: :tooling`.
For example in [`spec/tooling/danger/specs_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/tooling/danger/specs_spec.rb#L12).
+
+### Shared feature category
+
+For features that support developers and they are not specific to a product group we use `feature_category: :shared`
+For example [`spec/lib/gitlab/job_waiter_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/job_waiter_spec.rb)
diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md
index 874a56555fb..141b24161b4 100644
--- a/doc/development/feature_development.md
+++ b/doc/development/feature_development.md
@@ -22,7 +22,7 @@ Consult these topics for information on contributing to specific GitLab features
- [Software design guides](software_design.md)
- [GitLab EventStore](event_store.md) to publish/subscribe to domain events
- [GitLab utilities](utilities.md)
-- [Newlines style guide](newlines_styleguide.md)
+- [Newlines style guide](backend/ruby_style_guide.md#newlines-style-guide)
- [Logging](logging.md)
- [Dealing with email/mailers](emails.md)
- [Kubernetes integration guidelines](kubernetes.md)
@@ -81,7 +81,7 @@ Consult these topics for information on contributing to specific GitLab features
- [Working with Gitaly](gitaly.md)
- [Elasticsearch integration docs](elasticsearch.md)
- [Working with merge request diffs](diffs.md)
-- [Approval Rules](approval_rules.md)
+- [Approval Rules](merge_request_concepts/approval_rules.md)
- [Repository mirroring](repository_mirroring.md)
- [Uploads development guide](uploads/index.md)
- [Auto DevOps development guide](auto_devops.md)
diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md
index f1eafc2a95a..3adf5248b8d 100644
--- a/doc/development/feature_flags/controls.md
+++ b/doc/development/feature_flags/controls.md
@@ -196,7 +196,22 @@ enabled for only the `gitlab` project. The project is passed by supplying a
/chatops run feature set --project=gitlab-org/gitlab some_feature true
```
-For groups the `--group` flag is available:
+You can use the `--user` option to enable a feature flag for a specific user:
+
+```shell
+/chatops run feature set --user=myusername some_feature true
+```
+
+If you would like to gather feedback internally first,
+feature flags scoped to a user can also be enabled
+for GitLab team members with the `gitlab_team_members`
+[feature group](index.md#feature-groups):
+
+```shell
+/chatops run feature set --feature-group=gitlab_team_members some_feature true
+```
+
+You can use the `--group` flag to enable a feature flag for a specific group:
```shell
/chatops run feature set --group=gitlab-org some_feature true
diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md
index 5ff4292dfb6..7370697b082 100644
--- a/doc/development/feature_flags/index.md
+++ b/doc/development/feature_flags/index.md
@@ -39,20 +39,25 @@ should be leveraged:
existing feature flag because a feature is deemed stable must have the
~"feature flag" label assigned.
-When the feature implementation is delivered among multiple merge requests:
-
- 1. [Create a new feature flag](#create-a-new-feature-flag)
- which is **off** by default, in the first merge request which uses the flag.
- Flags [should not be added separately](#risk-of-a-broken-main-branch).
- 1. Submit incremental changes via one or more merge requests, ensuring that any
- new code added can only be reached if the feature flag is **on**.
- You can keep the feature flag enabled on your local GDK during development.
- 1. When the feature is ready to be tested, enable the feature flag for
- a specific project and ensure that there are no issues with the implementation.
- 1. When the feature is ready to be announced, create a merge request that adds
- documentation about the feature, including [documentation for the feature flag itself](../documentation/feature_flags.md),
- and a [changelog entry](#changelog). In the same merge request either flip the feature flag to
- be **on by default** or remove it entirely to enable the new behavior.
+When the feature implementation is delivered over multiple merge requests:
+
+1. [Create a new feature flag](#create-a-new-feature-flag)
+ which is **off** by default, in the first merge request which uses the flag.
+ Flags [should not be added separately](#risk-of-a-broken-main-branch).
+1. Submit incremental changes via one or more merge requests, ensuring that any
+ new code added can only be reached if the feature flag is **on**.
+ You can keep the feature flag enabled on your local GDK during development.
+1. When the feature is ready to be tested by other team members, [create the initial documentation](../documentation/feature_flags.md#when-to-document-features-behind-a-feature-flag).
+ Include details about the status of the [feature flag](../documentation/feature_flags.md#how-to-add-feature-flag-documentation).
+1. Enable the feature flag for a specific project and ensure that there are no issues
+ with the implementation. Do not enable the feature flag for a public project
+ like `gitlab` if there is no documentation. Team members and contributors might search for
+ documentation on how to use the feature if they see it enabled in a public project.
+1. When the feature is ready for production use, open a merge request to:
+ - Update the documentation to describe the latest flag status.
+ - Add a [changelog entry](#changelog).
+ - Flip the feature flag to be **on by default** or remove it entirely
+ to enable the new behavior.
One might be tempted to think that feature flags will delay the release of a
feature by at least one month (= one release). This is not the case. A feature
@@ -454,6 +459,18 @@ dynamic (querying the DB, for example).
Once defined in `lib/feature.rb`, you can to activate a
feature for a given feature group via the [`feature_group` parameter of the features API](../../api/features.md#set-or-create-a-feature)
+The available feature groups are:
+
+| Group name | Scoped to | Description |
+| --------------------- | --------- | ----------- |
+| `gitlab_team_members` | Users | Enables the feature for users who are members of [`gitlab-com`](https://gitlab.com/gitlab-com) |
+
+Feature groups can be enabled via the group name:
+
+```ruby
+Feature.enable(:feature_flag_name, :gitlab_team_members)
+```
+
### Enabling a feature flag locally (in development)
In the rails console (`rails c`), enter the following command to enable a feature flag:
diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md
index a23f1dd2d80..b4f5501ccac 100644
--- a/doc/development/gitaly.md
+++ b/doc/development/gitaly.md
@@ -219,19 +219,31 @@ as a [CI/CD variable](../ci/variables/index.md).
If you are making changes to the RPC client, such as adding a new endpoint or adding a new
parameter to an existing endpoint, follow the guide for
-[Gitaly protobuf specifications](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/protobuf.md). After pushing
-the branch with the changes (`new-feature-branch`, for example):
+[Gitaly protobuf specifications](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/protobuf.md). Then:
+
+1. Run `bundle install` in the `tools/protogem` directory of Gitaly.
+1. Build the RPC client gem from the root directory of Gitaly:
+
+ ```shell
+ BUILD_GEM_OPTIONS=--skip-verify-tag make build-proto-gem
+ ```
+
+1. In the `_build` directory of Gitaly, unpack the newly created `.gem` file and create a `gemspec`:
+
+ ```shell
+ gem unpack gitaly.gem &&
+ gem spec gitaly.gem > gitaly/gitaly.gemspec
+ ```
1. Change the `gitaly` line in the Rails' `Gemfile` to:
```ruby
- gem 'gitaly', git: 'https://gitlab.com/gitlab-org/gitaly.git', branch: 'new-feature-branch'
+ gem 'gitaly', path: '../gitaly/_build'
```
1. Run `bundle install` to use the modified RPC client.
-Re-run `bundle install` in the `gitlab` project each time the Gitaly branch
-changes to embed a new SHA in the `Gemfile.lock` file.
+Re-run steps 2-5 each time you want to try out new changes.
---
diff --git a/doc/development/gitlab_flavored_markdown/specification_guide/index.md b/doc/development/gitlab_flavored_markdown/specification_guide/index.md
index 79a4ac9b49a..806ac3837bf 100644
--- a/doc/development/gitlab_flavored_markdown/specification_guide/index.md
+++ b/doc/development/gitlab_flavored_markdown/specification_guide/index.md
@@ -514,29 +514,30 @@ of how the normalizations are specified.
Given all the constraints above, we can summarize the various goals related to the GLFM
specification and testing infrastructure:
-1. A canonical `spec.txt` exists, and represents the official specification for
- GLFM, which meets these requirements:
- 1. The spec is a strict superset of the GitHub Flavored Markdown
+1. There is an official specification and single source of truth for how GLFM should render Markdown to HTML.
+ This source of truth is represented by three Markdown files:
+ 1. [`ghfm_spec_v_?.??.md`](#github-flavored-markdown-specification) for the CommonMark + GFM examples.
+ 1. [`glfm_official_specification.md`](#glfm_official_specificationmd) for the GLFM official examples.
+ 1. [`glfm_internal_extensions.md`](#glfm_internal_extensionsmd) for the GLFM internal extensions.
+1. This official specification meets these requirements:
+ 1. The specification is a strict superset of the GitHub Flavored Markdown
(GFM) specification, just as
<abbr title="GitHub Flavored Markdown">GFM</abbr> is a strict superset
[of the CommonMark specification](https://github.github.com/gfm/#what-is-github-flavored-markdown-).
1. Therefore, it contains the superset of all [Markdown examples](#markdown-examples)
for CommonMark and GFM, as well as the GLFM
[official specification](#official-specifications) and [internal extensions](#internal-extensions).
- 1. It contains a prose introduction section which is specific to GitLab and GLFM.
- 1. It contains all other non-introduction sections verbatim from the
- [GFM specification](#github-flavored-markdown-specification).
- 1. It contains new, extra sections for all the additional Markdown contained in the GLFM
+ 1. It contains sections and examples for all the additional Markdown contained in the GLFM
[official specification](#official-specifications) and [internal extensions](#internal-extensions),
- with [Markdown examples](#markdown-examples) and accompanying prose, just like the CommonMark and GFM examples.
- 1. All its headers and [Markdown examples](#markdown-examples) should be in the standard format which can be processed by the standard
+ with [Markdown examples](#markdown-examples) and any accompanying prose, just like the CommonMark and GFM examples.
+ 1. All headers and [Markdown examples](#markdown-examples) should be in the standard format, which can be processed by the standard
CommonMark tool [`spec_tests.py`](https://github.com/github/cmark-gfm/blob/master/test/spec_tests.py) used to perform
[Markdown conformance testing](#markdown-conformance-testing)
against all examples contained in a `spec.txt`.
1. The GLFM parsers and HTML renderers for
both the static backend (Ruby) and WYSIWYG frontend (JavaScript) implementations
support _consistent_ rendering of all canonical Markdown + HTML examples in the
- GLFM `spec.txt` specification, as verified by `spec_tests.py`.
+ specification, as verified by [`run-spec-tests.sh`](#run-spec-testssh-script).
NOTE:
Consistent does not mean that both of these implementations render
@@ -616,26 +617,41 @@ them from the corresponding implementation class entry point files under
#### `update-specification.rb` script
The `scripts/glfm/update-specification.rb` script uses [input specification files](#input-specification-files) to
-generate and update `spec.txt` (Markdown) and `spec.html` (HTML). The `spec.html` is
-generated by passing the generated (or updated) `spec.txt` Markdown to the backend API
-for rendering to static HTML:
+generate and update Markdown and HTML output files for the
+[`spec.txt`](#spectxt) and [`spec.html`](#spechtml)
+[output specification files](#output-specification-files) as well as the
+[`snapshot_spec.md`](#snapshot_specmd) and [`snapshot_spec.html`](#snapshot_spechtml)
+[output example snapshot files](#output-example-snapshot-files).
+
+The HTML files are created by passing the generated (or updated) Markdown to the backend API
+for rendering to HTML.
+
+```mermaid
+graph LR
+subgraph script:
+ S{update-specification.rb}
+end
+subgraph input - markdown files
+ I1[glfm_official_specification.md - GLFM official specification examples] --> S
+end
+subgraph output - specification files
+ S --> O1[spec.txt - GLFM official specification examples]
+ S --> O2[spec.html - GLFM official specification examples]
+end
+```
```mermaid
graph LR
subgraph script:
- A{update-specification.rb}
- A --> B{Backend Markdown API}
+ S{update-specification.rb}
end
-subgraph input:<br/>input specification files
- C[ghfm_spec_v_0.29.md] --> A
- D[glfm_intro.md] --> A
- E[glfm_official_specification.md] --> A
- F[glfm_internal_extensions.md] --> A
+subgraph input - markdown files
+ I1[ghfm_spec_v_0.29.md - CommonMark and GHFM specification examples] --> S
+ I2[glfm_internal_extensions.md - GLFM internal extension examples] --> S
end
-subgraph output:<br/>GLFM specification files
- A --> G[spec.txt]
- G --> B
- B --> H[spec.html]
+subgraph output - example snapshot files
+ S --> O1[snapshot_spec.md - CommonMark, GHFM, GLFM internal extension examples]
+ S --> O2[snapshot_spec.html - CommonMark, GHFM, GLFM internal extension examples]
end
```
@@ -655,7 +671,8 @@ script, which expects canonical HTML, against the GitLab renderer implementation
`scripts/glfm/run-spec-tests.sh` is a convenience shell script which runs
conformance specs via the CommonMark standard `spec_tests.py` script,
-which uses the `glfm_specification/output_spec/spec.txt` file and `scripts/glfm/canonicalize-html.rb`
+which uses the `ghfm_spec_v_0.29.md` and `glfm_specification/output_spec/spec.txt` files
+with the `scripts/glfm/canonicalize-html.rb`
helper script to test the GLFM renderer implementations' support for rendering Markdown
specification examples to canonical HTML.
@@ -669,7 +686,8 @@ subgraph scripts:
end
end
subgraph input
- D[spec.txt GLFM specification] --> C
+ D1[ghfm_spec_v_0.29.md GLFM specification] --> C
+ D2[spec.txt GLFM specification] --> C
E((GLFM static<br/>renderer implementation)) --> B
F((GLFM WYSIWYG<br/>renderer implementation)) --> B
end
@@ -680,22 +698,28 @@ end
#### `update-example-snapshots.rb` script
-The `scripts/glfm/update-example-snapshots.rb` script uses the GLFM
-`glfm_specification/output_spec/spec.txt` specification file and the
-`glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml`
-file to create and update the [example snapshot](#output-example-snapshot-files)
-YAML files:
+The `scripts/glfm/update-example-snapshots.rb` script creates and updates the
+[example snapshot](#output-example-snapshot-files) YAML files.
+
+Its inputs are:
+
+- The `glfm_specification/output_spec/snapshot_spec.md` file, which contains the
+ superset of all CommonMark, GFM, and GLFM official and internal examples.
+- The `glfm_specification/input/gitlab_flavored_markdown/glfm_example_*.yml` YAML files, which
+ contain metadata to control how to generate the example snapshot files.
```mermaid
graph LR
subgraph script:
A{update-example-snapshots.rb}
end
-subgraph input:<br/>input specification file
- B[spec.txt] --> A
- C[glfm_example_status.yml] --> A
+subgraph input: markdown input specification files
+ B1[snapshot_spec.md] --> A
+ C1[glfm_example_status.yml] --> A
+ C2[glfm_example_normalizations.yml] --> A
+ C3[glfm_example_metadata.yml] --> A
end
-subgraph output:<br/>example snapshot files
+subgraph output: YAML example snapshot files
A --> E[examples_index.yml]
A --> F[markdown.yml]
A --> G[html.yml]
@@ -736,15 +760,15 @@ end
subgraph script:
A{run-snapshopt-tests.sh} -->|invokes| B
end
+subgraph output:<br/>test results/output
+ B --> H[rspec+jest output]
+end
subgraph input:<br/>YAML
C[examples_index.yml] --> B
D[markdown.yml] --> B
E[html.yml] --> B
F[prosemirror_json.yml] --> B
end
-subgraph output:<br/>test results/output
- B --> H[rspec+jest output]
-end
```
#### `verify-all-generated-files-are-up-to-date.rb` script
@@ -1111,8 +1135,12 @@ move or copy a hosted version of the rendered HTML `spec.html` version to anothe
is a Markdown specification file, in the standard format
with prose and Markdown + canonical HTML examples.
+In the GLFM specification, `spex.txt` only contains the official specifiaction examples from
+[`glfm_official_specification.md`](#glfm_official_specificationmd). It does not contain
+the internal extension examples from [`glfm_internal_extensions.md`](#glfm_internal_extensionsmd).
+
It also serves as input for other scripts such as
-`run-spec-tests.sh`.
+[`run-spec-tests.sh`](#run-spec-testssh-script).
It is generated or updated by the `update-specification.rb` script, using the
[input specification files](#input-specification-files) as input.
diff --git a/doc/development/gitlab_shell/index.md b/doc/development/gitlab_shell/index.md
index 7f2c113fa0c..7097fd48cea 100644
--- a/doc/development/gitlab_shell/index.md
+++ b/doc/development/gitlab_shell/index.md
@@ -21,7 +21,7 @@ Ruby to build and test, but not to run.
GitLab Shell runs on `port 22` on an Omnibus installation. To use a regular SSH
service, configure it on an alternative port.
-Download and install the current version of Go from [golang.org](https://golang.org/dl/).
+Download and install the current version of Go from [golang.org](https://go.dev/dl/).
We follow the [Golang Release Policy](https://golang.org/doc/devel/release.html#policy)
and support:
diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md
index e7e628f5293..508219cee43 100644
--- a/doc/development/go_guide/index.md
+++ b/doc/development/go_guide/index.md
@@ -134,7 +134,7 @@ lint:
Including a `.golangci.yml` in the root directory of the project allows for
configuration of `golangci-lint`. All options for `golangci-lint` are listed in
-this [example](https://github.com/golangci/golangci-lint/blob/master/.golangci.example.yml).
+this [example](https://github.com/golangci/golangci-lint/blob/master/.golangci.yml).
Once [recursive includes](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/56836)
become available, you can share job templates like this
diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md
index 2c8d7eac2a6..bfc4d817c73 100644
--- a/doc/development/i18n/translation.md
+++ b/doc/development/i18n/translation.md
@@ -75,6 +75,7 @@ The level of formality used in software varies by language:
| -------- | --------- | ------- |
| French | formal | `vous` for `you` |
| German | informal | `du` for `you` |
+| Spanish | informal | `tú` for `you` |
Refer to other translated strings and notes in the glossary to assist you in determining a suitable
level of formality.
diff --git a/doc/development/import_export.md b/doc/development/import_export.md
index 9aab7f38dbb..17e733e8904 100644
--- a/doc/development/import_export.md
+++ b/doc/development/import_export.md
@@ -105,10 +105,9 @@ module Gitlab
VERSION = '0.2.4'
```
-## Version history
+## Compatibility
-Check the [version history](../user/project/settings/import_export.md#version-history)
-for compatibility when importing and exporting projects.
+Check for [compatibility](../user/project/settings/import_export.md#compatibility) when importing and exporting projects.
### When to bump the version up
diff --git a/doc/development/index.md b/doc/development/index.md
index d4a34051bc8..a39a83b257a 100644
--- a/doc/development/index.md
+++ b/doc/development/index.md
@@ -11,38 +11,11 @@ description: "Development Guidelines: learn how to contribute to GitLab."
Learn how to contribute to the development of the GitLab product.
-This content is intended for members of the GitLab Team as well as community
-contributors. Content specific to the GitLab Team should instead be included in
-the [Handbook](https://about.gitlab.com/handbook/).
-
-For information on using GitLab to work on your own software projects, see the
-[GitLab user documentation](../user/index.md).
-
-For information on working with the GitLab APIs, see the [API documentation](../api/index.md).
-
-For information about how to install, configure, update, and upgrade your own
-GitLab instance, see the [Administrator documentation](../administration/index.md).
-
-## Get started
-
-- Set up the GitLab development environment with the
- [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/README.md)
-- [GitLab contributing guide](contributing/index.md)
- - [Issues workflow](contributing/issue_workflow.md) for more information about:
- - Issue tracker guidelines.
- - Triaging.
- - Labels.
- - Feature proposals.
- - Issue weight.
- - Regression issues.
- - Technical or UX debt.
- - [Merge requests workflow](contributing/merge_request_workflow.md) for more
- information about:
- - Merge request guidelines.
- - Contribution acceptance criteria.
- - Definition of done.
- - Dependencies.
- - [Style guides](contributing/style_guides.md)
- - [Implement design & UI elements](contributing/design.md)
-- [GitLab Architecture Overview](architecture.md)
-- [Rake tasks](rake_tasks.md) for development
+This content is intended for GitLab team members as well as members of the wider community.
+
+- [Contribute to GitLab development](contributing/index.md)
+- [Contribute to Omnibus development](https://docs.gitlab.com/omnibus/development/)
+- [Contribute to GitLab Pages development](pages/index.md)
+- [Contribute to GitLab Runner development](https://docs.gitlab.com/runner/development/)
+- [Contribute to Helm chart development](https://docs.gitlab.com/charts/development/)
+- [Contribute to the GitLab documentation](documentation/index.md)
diff --git a/doc/development/integrations/codesandbox.md b/doc/development/integrations/codesandbox.md
index b7fe3fbd1c4..4553ed2966f 100644
--- a/doc/development/integrations/codesandbox.md
+++ b/doc/development/integrations/codesandbox.md
@@ -2,12 +2,17 @@
stage: none
group: Development
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+remove_date: '2023-02-01'
---
-# Set up local CodeSandbox development environment
+# Set up local CodeSandbox development environment (removed)
+
+WARNING:
+This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108627) in GitLab 15.8
+and is planned for removal in 15.9. This change is a breaking change.
This guide walks through setting up a local [CodeSandbox repository](https://github.com/codesandbox/codesandbox-client) and integrating it with a local GitLab instance. CodeSandbox
-is used to power the Web IDE [Live Preview feature](../../user/project/web_ide/index.md#live-preview). Having a local CodeSandbox setup is useful for debugging upstream issues or
+is used to power the Web IDE [Live Preview feature](../../user/project/web_ide/index.md#live-preview-removed). Having a local CodeSandbox setup is useful for debugging upstream issues or
creating upstream contributions like [this one](https://github.com/codesandbox/codesandbox-client/pull/5137).
## Initial setup
@@ -114,7 +119,11 @@ out of the box:
npx http-server --proxy http://localhost:3000 -S -C $PATH_TO_CERT_PEM -K $PATH_TO_KEY_PEM -p 8044 -d false
```
-### Update `bundler_url` setting in GitLab
+### Update `bundler_url` setting in GitLab (removed)
+
+WARNING:
+This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108627) in GitLab 15.8
+and is planned for removal in 15.9. This change is a breaking change.
We need to update our `application_setting_implementation.rb` to point to the server that hosts the
CodeSandbox `sandpack` assets. For instance, if these assets are hosted by a server at `https://sandpack.local:8044`:
diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md
index 5b460f8723a..eca4d9775c5 100644
--- a/doc/development/integrations/jira_connect.md
+++ b/doc/development/integrations/jira_connect.md
@@ -54,7 +54,7 @@ To install the app in Jira:
1. Select **Upload**.
- If the install was successful, you should see the **GitLab.com for Jira Cloud** app under **Manage apps**.
+ If the install was successful, you should see the **GitLab for Jira Cloud** app under **Manage apps**.
You can also select **Getting Started** to open the configuration page rendered from your GitLab instance.
_Note that any changes to the app descriptor requires you to uninstall then reinstall the app._
diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md
index f5bb2df2494..002579d9b83 100644
--- a/doc/development/integrations/secure.md
+++ b/doc/development/integrations/secure.md
@@ -31,7 +31,6 @@ For consistency, scanning jobs should be named after the scanner, in lower case.
The job name is suffixed after the type of scanning:
- `_dependency_scanning`
-- `_cluster_image_scanning`
- `_container_scanning`
- `_dast`
- `_sast`
@@ -79,7 +78,6 @@ Valid reports are:
- `dependency_scanning`
- `container_scanning`
-- `cluster_image_scanning`
- `dast`
- `api_fuzzing`
- `coverage_fuzzing`
@@ -96,7 +94,7 @@ mysec_sast:
sast: gl-sast-report.json
```
-Note that `gl-sast-report.json` is an example file path but any other filename can be used. See
+`gl-sast-report.json` is an example file path but any other filename can be used. See
[the Output file section](#output-file) for more details. It's processed as a SAST report because
it's declared under the `reports:sast` key in the job definition, not because of the filename.
@@ -108,7 +106,6 @@ for variables such as:
- `DEPENDENCY_SCANNING_DISABLED`
- `CONTAINER_SCANNING_DISABLED`
-- `CLUSTER_IMAGE_SCANNING_DISABLED`
- `SAST_DISABLED`
- `DAST_DISABLED`
@@ -150,7 +147,7 @@ regardless of the individual machine the scanner runs on.
Depending on the CI infrastructure,
the CI may have to fetch the Docker image every time the job runs.
For the scanning job to run fast and avoid wasting bandwidth, Docker images should be as small as
-possible. You should aim for 50MB or smaller. If that isn't possible, try to keep it below 1.46 GB,
+possible. You should aim for 50 MB or smaller. If that isn't possible, try to keep it below 1.46 GB,
which is the size of a DVD-ROM.
If the scanner requires a fully functional Linux environment,
@@ -199,7 +196,7 @@ SAST and Dependency Scanning scanners must scan the files in the project directo
#### Container Scanning
-In order to be consistent with the official Container Scanning for GitLab,
+To be consistent with the official Container Scanning for GitLab,
scanners must scan the Docker image whose name and tag are given by
`CI_APPLICATION_REPOSITORY` and `CI_APPLICATION_TAG`, respectively. If the `DOCKER_IMAGE`
CI/CD variable is provided, then the `CI_APPLICATION_REPOSITORY` and `CI_APPLICATION_TAG` variables
@@ -214,19 +211,6 @@ using the variables `DOCKER_USER` and `DOCKER_PASSWORD`.
If these are not defined, then the scanner should use
`CI_REGISTRY_USER` and `CI_REGISTRY_PASSWORD` as default values.
-#### Cluster Image Scanning
-
-To be consistent with the official `cluster_image_scanning` for GitLab, scanners must scan the
-Kubernetes cluster whose configuration is given by `KUBECONFIG`.
-
-If you use the `CIS_KUBECONFIG` CI/CD variable, then the
-`KUBECONFIG` variable is ignored and the cluster specified in the
-`CIS_KUBECONFIG` variable is scanned instead. If you don't provide
-the `CIS_KUBECONFIG` CI/CD variable, the value defaults to the value of
-`$KUBECONFIG`. `$KUBECONFIG` is a predefined CI/CD variable configured when the project is assigned to a
-Kubernetes cluster. When multiple contexts are provided in the `KUBECONFIG` variable, the context
-selected as `current-context` will be used to fetch vulnerabilities.
-
#### Configuration files
While scanners may use `CI_PROJECT_DIR` to load specific configuration files,
@@ -320,7 +304,6 @@ and [Container Scanning](../../user/application_security/container_scanning/inde
You can find the schemas for these scanners here:
-- [Cluster Image Scanning](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/cluster-image-scanning-report-format.json)
- [Container Scanning](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/container-scanning-report-format.json)
- [Coverage Fuzzing](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/coverage-fuzzing-report-format.json)
- [DAST](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dast-report-format.json)
@@ -414,7 +397,6 @@ We recommend that you generate a UUID and use it as the `id` field's value.
The value of the `category` field matches the report type:
- `dependency_scanning`
-- `cluster_image_scanning`
- `container_scanning`
- `sast`
- `dast`
@@ -440,7 +422,7 @@ Even when the [`Vulnerabilities`](#vulnerabilities) array for a given scan may b
should contain the complete list of potential identifiers to inform the Rails application of which
rules were executed.
-When populated, the Rails application will automatically resolve previously detected vulnerabilities as no
+When populated, the Rails application automatically resolves previously detected vulnerabilities as no
longer relevant when their primary identifier is not included.
##### Name, message, and description
@@ -526,7 +508,7 @@ Not all vulnerabilities have CVEs, and a CVE can be identified multiple times. A
isn't a stable identifier and you shouldn't assume it as such when tracking vulnerabilities.
The maximum number of identifiers for a vulnerability is set as 20. If a vulnerability has more than 20 identifiers,
-the system saves only the first 20 of them. Note that vulnerabilities in the [Pipeline Security](../../user/application_security/vulnerability_report/pipeline.md#view-vulnerabilities-in-a-pipeline)
+the system saves only the first 20 of them. The vulnerabilities in the [Pipeline Security](../../user/application_security/vulnerability_report/pipeline.md#view-vulnerabilities-in-a-pipeline)
tab do not enforce this limit and all identifiers present in the report artifact are displayed.
#### Details
@@ -604,40 +586,6 @@ so these attributes are mandatory.
The `image` is also mandatory.
All other attributes are optional.
-##### Cluster Image Scanning
-
-The `location` of a `cluster_image_scanning` vulnerability has a `dependency` field. It also has
-an `operating_system` field. For example, here is the `location` object for a vulnerability
-affecting version `2.50.3-2+deb9u1` of Debian package `glib2.0`:
-
-```json
-{
- "dependency": {
- "package": {
- "name": "glib2.0"
- },
- },
- "version": "2.50.3-2+deb9u1",
- "operating_system": "debian:9",
- "image": "index.docker.io/library/nginx:1.18",
- "kubernetes_resource": {
- "namespace": "production",
- "kind": "Deployment",
- "name": "nginx-ingress",
- "container_name": "nginx",
- "agent_id": "1"
- }
-}
-```
-
-The affected package is found when scanning a deployment using the `index.docker.io/library/nginx:1.18` image.
-
-The location fingerprint of a Cluster Image Scanning vulnerability combines the
-`namespace`, `kind`, `name`, and `container_name` fields from the `kubernetes_resource`,
-as well as the package `name`, so these fields are required. The `image` field is also mandatory.
-The `cluster_id` and `agent_id` are mutually exclusive, and one of them must be present.
-All other fields are optional.
-
##### SAST
The `location` of a SAST vulnerability must have a `file` and a `start_line` field,
diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md
index f0fdedd801f..b19e431ebc6 100644
--- a/doc/development/internal_api/index.md
+++ b/doc/development/internal_api/index.md
@@ -476,7 +476,7 @@ agent to be authorized is [not yet implemented](https://gitlab.com/gitlab-org/gi
| Attribute | Type | Required | Description |
|:----------|:-------|:---------|:------------|
-| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../api/index.md#namespaced-path-encoding) |
+| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../api/rest/index.md#namespaced-path-encoding) |
```plaintext
GET /internal/kubernetes/project_info
@@ -1377,7 +1377,8 @@ Returns an empty response with a `204` status code if successful.
### Remove a single SCIM provisioned user
-Removes the user's SSO identity.
+The user is placed in an `ldap_blocked` status and signed out. This means
+the user cannot sign in or push or pull code.
```plaintext
DELETE /api/scim/v2/application/Users/:id
diff --git a/doc/development/logging.md b/doc/development/logging.md
index 6282f0f6677..538fc7ccfe1 100644
--- a/doc/development/logging.md
+++ b/doc/development/logging.md
@@ -390,14 +390,32 @@ end
On GitLab.com, that setting is only 6 compressed files. These settings should suffice
for most users, but you may need to tweak them in [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab).
-1. If you add a new file, submit an issue to the
- [production tracker](https://gitlab.com/gitlab-com/gl-infra/production/-/issues) or
- a merge request to the [`gitlab_fluentd`](https://gitlab.com/gitlab-cookbooks/gitlab_fluentd)
- project. See [this example](https://gitlab.com/gitlab-cookbooks/gitlab_fluentd/-/merge_requests/51/diffs).
+1. On GitLab.com all new JSON log files generated by GitLab Rails are
+ automatically shipped to Elasticsearch (and available in Kibana) on GitLab
+ Rails Kubernetes pods. If you need the file forwarded from Gitaly nodes then
+ submit an issue to the
+ [production tracker](https://gitlab.com/gitlab-com/gl-infra/production/-/issues)
+ or a merge request to the
+ [`gitlab_fluentd`](https://gitlab.com/gitlab-cookbooks/gitlab_fluentd)
+ project. See
+ [this example](https://gitlab.com/gitlab-cookbooks/gitlab_fluentd/-/merge_requests/51/diffs).
1. Be sure to update the [GitLab CE/EE documentation](../administration/logs/index.md) and the
[GitLab.com runbooks](https://gitlab.com/gitlab-com/runbooks/blob/master/docs/logging/README.md).
+## Finding new log files in Kibana (GitLab.com only)
+
+On GitLab.com all new JSON log files generated by GitLab Rails are
+automatically shipped to Elasticsearch (and available in Kibana) on GitLab
+Rails Kubernetes pods. The `json.subcomponent` field in Kibana will allow you
+to filter by the different kinds of log files. For example the
+`json.subcomponent` will be `production_json` for entries forwarded from
+`production_json.log`.
+
+It's also worth noting that log files from Web/API pods go to a different
+index than log files from Sidekiq pods. Depending on where you log from you
+will find the logs in a different index pattern.
+
## Control logging visibility
An increase in the logs can cause a growing backlog of unacknowledged messages. When adding new log messages, make sure they don't increase the overall volume of logging by more than 10%.
diff --git a/doc/development/merge_request_application_and_rate_limit_guidelines.md b/doc/development/merge_request_application_and_rate_limit_guidelines.md
index 07a48ad7723..07788400adf 100644
--- a/doc/development/merge_request_application_and_rate_limit_guidelines.md
+++ b/doc/development/merge_request_application_and_rate_limit_guidelines.md
@@ -1,28 +1,11 @@
---
-stage: none
-group: unassigned
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+redirect_to: 'merge_request_concepts/rate_limits.md'
+remove_date: '2023-04-23'
---
-# Application and rate limit guidelines
+This document was moved to [another location](merge_request_concepts/rate_limits.md).
-GitLab, like most large applications, enforces limits within certain features.
-The absences of limits can affect security, performance, data, or could even
-exhaust the allocated resources for the application.
-
-Every new feature should have safe usage limits included in its implementation.
-Limits are applicable for:
-
-- System-level resource pools such as API requests, SSHD connections, database connections, storage, and so on.
-- Domain-level objects such as CI/CD minutes, groups, sign-in attempts, and so on.
-
-## When limits are required
-
-1. Limits are required if the absence of the limit matches severity 1 - 3 in the severity definitions for [limit-related bugs](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#limit-related-bugs).
-1. [GitLab application limits](../administration/instance_limits.md) documentation must be updated anytime limits are added, removed, or updated.
-
-## Additional reading
-
-- Existing [GitLab application limits](../administration/instance_limits.md)
-- Product processes: [introducing application limits](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits)
-- Development docs: [guide for adding application limits](application_limits.md)
+<!-- This redirect file can be deleted after <2023-04-23>. -->
+<!-- Redirects that point to other docs in the same project expire in three months. -->
+<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
+<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/development/merge_request_concepts/approval_rules.md b/doc/development/merge_request_concepts/approval_rules.md
new file mode 100644
index 00000000000..d119644cd7c
--- /dev/null
+++ b/doc/development/merge_request_concepts/approval_rules.md
@@ -0,0 +1,286 @@
+---
+stage: Create
+group: Code Review
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Approval Rules development guide
+
+This document explains the backend design and flow of all related functionality
+about [merge request approval rules](../../user/project/merge_requests/approvals/index.md).
+
+This should help contributors to understand the code design easier and to also
+help see if there are parts to improve as the feature and its implementation
+evolves.
+
+It's intentional that it doesn't contain too much implementation detail as they
+can change often. The code should explain those things better. The components
+mentioned here are the major parts of the application for the approval rules
+feature to work.
+
+NOTE:
+This is a living document and should be updated accordingly when parts
+of the codebase touched in this document are changed or removed, or when new components
+are added.
+
+## Data Model
+
+```mermaid
+erDiagram
+ Project ||--o{ MergeRequest: " "
+ Project ||--o{ ApprovalProjectRule: " "
+ ApprovalProjectRule }o--o{ User: " "
+ ApprovalProjectRule }o--o{ Group: " "
+ ApprovalProjectRule }o--o{ ProtectedBranch: " "
+ MergeRequest ||--|| ApprovalState: " "
+ ApprovalState ||--o{ ApprovalWrappedRule: " "
+ MergeRequest ||--o{ Approval: " "
+ MergeRequest ||--o{ ApprovalMergeRequestRule: " "
+ ApprovalMergeRequestRule }o--o{ User: " "
+ ApprovalMergeRequestRule }o--o{ Group: " "
+ ApprovalMergeRequestRule ||--o| ApprovalProjectRule: " "
+```
+
+### `Project` and `MergeRequest`
+
+`Project` and `MergeRequest` models are defined in `ee/app/models/ee/project.rb`
+and `ee/app/models/ee/merge_request.rb`. They extend the non-EE versions, because
+approval rules are an EE-only feature. Associations and other related stuff to
+merge request approvals are defined here.
+
+### `ApprovalState`
+
+```mermaid
+erDiagram
+ MergeRequest ||--|| ApprovalState: " "
+```
+
+`ApprovalState` class is defined in `ee/app/models/approval_state.rb`. It's not
+an actual `ActiveRecord` model. This class encapsulates all logic related to the
+state of the approvals for a certain merge request like:
+
+- Knowing the approval rules that are applicable to the merge request based on
+ its target branch.
+- Knowing the approval rules that are applicable to a certain target branch.
+- Checking if all rules were approved.
+- Checking if approval is required.
+- Knowing how many approvals were given or still required.
+
+It gets the approval rules data from the project (`ApprovalProjectRule`) or the
+merge request (`ApprovalMergeRequestRule`) and wrap it as `ApprovalWrappedRule`.
+
+### `ApprovalProjectRule`
+
+```mermaid
+erDiagram
+ Project ||--o{ ApprovalProjectRule: " "
+ ApprovalProjectRule }o--o{ User: " "
+ ApprovalProjectRule }o--o{ Group: " "
+ ApprovalProjectRule }o--o{ ProtectedBranch: " "
+```
+
+`ApprovalProjectRule` model is defined in `ee/app/models/approval_project_rule.rb`.
+
+A record is created/updated/deleted when an approval rule is added/edited/removed
+via project settings or the [project level approvals API](../../api/merge_request_approvals.md#project-level-mr-approvals).
+The `ApprovalState` model get these records when approval rules are not
+overwritten.
+
+The `protected_branches` attribute is set and used when a rule is scoped to
+protected branches. See [Approvals for protected branches](../../user/project/merge_requests/approvals/rules.md#approvals-for-protected-branches)
+for more information about the feature.
+
+### `ApprovalMergeRequestRule`
+
+```mermaid
+erDiagram
+ MergeRequest ||--o{ ApprovalMergeRequestRule: " "
+ ApprovalMergeRequestRule }o--o{ User: " "
+ ApprovalMergeRequestRule }o--o{ Group: " "
+ ApprovalMergeRequestRule ||--o| ApprovalProjectRule: " "
+```
+
+`ApprovalMergeRequestRule` model is defined in `ee/app/models/approval_merge_request_rule.rb`.
+
+A record is created/updated/deleted when a rule is added/edited/removed via merge
+request create/edit form or the [merge request level approvals API](../../api/merge_request_approvals.md#merge-request-level-mr-approvals).
+
+The `approval_project_rule` is set when it is based from an existing `ApprovalProjectRule`.
+
+An `ApprovalMergeRequestRule` doesn't have `protected_branches` as it inherits
+them from the `approval_project_rule` if not overridden.
+
+### `ApprovalWrappedRule`
+
+```mermaid
+erDiagram
+ ApprovalState ||--o{ ApprovalWrappedRule: " "
+```
+
+`ApprovalWrappedRule` is defined in `ee/app/modes/approval_wrapped_rule.rb` and
+is not an `ActiveRecord` model. It's used to wrap an `ApprovalProjectRule` or
+`ApprovalMergeRequestRule` for common interface. It also has the following sub
+types:
+
+- `ApprovalWrappedAnyApprovalRule` - for wrapping an `any_approver` rule.
+- `ApprovalWrappedCodeOwnerRule` - for wrapping a `code_owner` rule.
+
+This class delegates most of the responsibilities to the approval rule it wraps
+but it's also responsible for:
+
+- Checking if the approval rule is approved.
+- Knowing how many approvals were given or still required for the approval rule.
+
+It gets this information from the approval rule and the `Approval` records from
+the merge request.
+
+### `Approval`
+
+```mermaid
+erDiagram
+ MergeRequest ||--o{ Approval: " "
+```
+
+`Approval` model is defined in `ee/app/models/approval.rb`. This model is
+responsible for storing information about an approval made on a merge request.
+Whenever an approval is given/revoked, a record is created/deleted.
+
+## Controllers and Services
+
+The following controllers and services below are being used for the approval
+rules feature to work.
+
+### `API::ProjectApprovalSettings`
+
+This private API is defined in `ee/lib/api/project_approval_settings.rb`.
+
+This is used for the following:
+
+- Listing the approval rules in project settings.
+- Creating/updating/deleting rules in project settings.
+- Listing the approval rules on create merge request form.
+
+### `Projects::MergeRequests::CreationsController`
+
+This controller is defined in `app/controllers/projects/merge_requests/creations_controller.rb`.
+
+The `create` action of this controller is used when create merge request form is
+submitted. It accepts the `approval_rules_attributes` parameter for creating/updating/deleting
+`ApprovalMergeRequestRule` records. It passes the parameter along when it executes
+`MergeRequests::CreateService`.
+
+### `Projects::MergeRequestsController`
+
+This controller is defined in `app/controllers/projects/merge_requests_controller.rb`.
+
+The `update` action of this controller is used when edit merge request form is
+submitted. It's like `Projects::MergeRequests::CreationsController` but it executes
+`MergeRequests::UpdateService` instead.
+
+### `API::MergeRequestApprovals`
+
+This API is defined in `ee/lib/api/merge_request_approvals.rb`.
+
+The [Approvals API endpoint](../../api/merge_request_approvals.md#get-configuration-1)
+is requested when merge request page loads.
+
+The `/projects/:id/merge_requests/:merge_request_iid/approval_settings` is a
+private API endpoint used for the following:
+
+- Listing the approval rules on edit merge request form.
+- Listing the approval rules on the merge request page.
+
+When approving/unapproving MR via UI and API, the [Approve Merge Request](../../api/merge_request_approvals.md#approve-merge-request)
+API endpoint and the [Unapprove Merge Request](../../api/merge_request_approvals.md#unapprove-merge-request)
+API endpoint are requested. They execute `MergeRequests::ApprovalService` and
+`MergeRequests::RemoveApprovalService` accordingly.
+
+### `API::ProjectApprovalRules` and `API::MergeRequestApprovalRules`
+
+These APIs are defined in `ee/lib/api/project_approval_rules.rb` and
+`ee/lib/api/merge_request_approval_rules.rb`.
+
+Used to list/create/update/delete project and merge request level rules via
+[Merge request approvals API](../../api/merge_request_approvals.md).
+
+Executes `ApprovalRules::CreateService`, `ApprovalRules::UpdateService`,
+`ApprovalRules::ProjectRuleDestroyService`, and `ApprovalRules::MergeRequestRuleDestroyService`
+accordingly.
+
+### `ApprovalRules::ParamsFilteringService`
+
+This service is defined in `ee/app/services/approval_rules/params_filtering_service.rb`.
+
+It is called only when `MergeRequests::CreateService` and
+`MergeRequests::UpdateService` are executed.
+
+It is responsible for parsing `approval_rules_attributes` parameter to:
+
+- Remove it when user can't update approval rules.
+- Filter the user IDs whether they are members of the project or not.
+- Filter the group IDs whether they are visible to user.
+- Identify the `any_approver` rule.
+- Append hidden groups to it when specified.
+- Append user defined inapplicable (rules that do not apply to the merge request's target
+ branch) approval rules.
+
+## Flow
+
+These flowcharts should help explain the flow from the controllers down to the
+models for different functionalities.
+
+Some CRUD API endpoints are intentionally skipped because they are pretty
+straightforward.
+
+### Creating a merge request with approval rules via web UI
+
+```mermaid
+graph LR
+ Projects::MergeRequests::CreationsController --> MergeRequests::CreateService
+ MergeRequests::CreateService --> ApprovalRules::ParamsFilteringService
+ ApprovalRules::ParamsFilteringService --> MergeRequests::CreateService
+ MergeRequests::CreateService --> MergeRequest
+ MergeRequest --> db[(Database)]
+ MergeRequest --> User
+ MergeRequest --> Group
+ MergeRequest --> ApprovalProjectRule
+ User --> db[(Database)]
+ Group --> db[(Database)]
+ ApprovalProjectRule --> db[(Database)]
+```
+
+When updating, same flow is followed but it starts at `Projects::MergeRequestsController`
+and executes `MergeRequests::UpdateService` instead.
+
+### Viewing the merge request approval rules on an MR page
+
+```mermaid
+graph LR
+ API::MergeRequestApprovals --> MergeRequest
+ MergeRequest --> ApprovalState
+ ApprovalState --> id1{approval rules are overridden}
+ id1{approval rules are overridden} --> |No| ApprovalProjectRule & ApprovalMergeRequestRule
+ id1{approval rules are overridden} --> |Yes| ApprovalMergeRequestRule
+ ApprovalState --> ApprovalWrappedRule
+ ApprovalWrappedRule --> Approval
+```
+
+This flow gets initiated by the frontend component. The data returned is
+used to display information on the MR widget.
+
+### Approving a merge request
+
+```mermaid
+graph LR
+ API::MergeRequestApprovals --> MergeRequests::ApprovalService
+ MergeRequests::ApprovalService --> Approval
+ Approval --> db[(Database)]
+```
+
+When unapproving, same flow is followed but the `MergeRequests::RemoveApprovalService`
+is executed instead.
+
+## TODO
+
+1. Add information related to other rule types, such as `code_owner` and `report_approver`.
+1. Add information about side effects of approving/unapproving merge request.
diff --git a/doc/development/merge_request_concepts/diffs/frontend.md b/doc/development/merge_request_concepts/diffs/frontend.md
new file mode 100644
index 00000000000..6bd6d80af94
--- /dev/null
+++ b/doc/development/merge_request_concepts/diffs/frontend.md
@@ -0,0 +1,208 @@
+---
+stage: Create
+group: Code Review
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Merge request diffs frontend overview
+
+This document provides an overview on how the frontend diffs Vue application works, and
+the various different parts that exist. It should help contributors:
+
+- Understand how the diffs Vue app is set up.
+- Identify any areas that need improvement.
+
+This document is a living document. Update it whenever anything significant changes in
+the diffs application.
+
+## Diffs Vue app
+
+### Components
+
+The Vue app for rendering diffs uses many different Vue components, some of which get shared
+with other areas of the GitLab app. The below chart shows the direction for which components
+get rendered.
+
+NOTE:
+[Issue #388843](https://gitlab.com/gitlab-org/gitlab/-/issues/388843) is open to
+generate a Mermaid graph of the components diagram. An image version of the
+diagram is available in the issue.
+
+Some of the components are rendered more than others, but the main component is `diff_row.vue`.
+This component renders every diff line in a diff file. For performance reasons, this
+component is a functional component. However, when we upgrade to Vue 3, this is no longer
+required.
+
+The main diff app component is the main entry point to the diffs app. One of the most important parts
+of this component is to dispatch the action that assigns discussions to diff lines. This action
+gets dispatched after the metadata request is completed, and after the batch diffs requests are
+finished. There is also a watcher set up to watches for changes in both the diff files array and the notes
+array. Whenever a change happens here, the set discussion action gets dispatched.
+
+The DiffRow component is set up in a way that allows for us to store the diff line data in one format.
+Previously, we had to request two different formats for inline and side-by-side. The DiffRow component
+then uses this standard format to render the diff line data. With this standard format, the user
+can then switch between inline and side-by-side without the need to re-fetch any data.
+
+NOTE:
+For this component, a lot of the data used and rendered gets memoized and cached, based on
+various conditions. It is possible that data sometimes gets cached between each different
+component render.
+
+### Vuex store
+
+The Vuex store for the diffs app consists of 3 different modules:
+
+- Notes
+- Diffs
+- Batch comments
+
+The notes module is responsible for the discussions, including diff discussions. In this module,
+the discussions get fetched, and the polling for new discussions is setup. This module gets shared
+with the issue app as well, so changes here need to be tested in both issues and merge requests.
+
+The diffs module is responsible for the everything related to diffs. This includes, but is not limited
+to, fetching diffs, assigning diff discussions to lines, and creating diff discussions.
+
+Finally, the batch comments module is not complex, and is responsible only for the draft comments feature.
+However, this module does dispatch actions in the notes and diff modules whenever draft comments
+are published.
+
+### API Requests
+
+#### Metadata
+
+The diffs metadata endpoint exists to fetch the base data the diffs app requires quickly, without
+the need to fetch all the diff files. This includes, but is not limited to:
+
+- Diff file names, including some extra meta data for diff files
+- Added and removed line numbers
+- Branch names
+- Diff versions
+
+The most important part of the metadata response is the diff file names. This data allows the diffs
+app to render the file browser inside of the diffs app, without waiting for all batch diffs
+requests to complete.
+
+When the metadata response is received, the diff file data gets sent to a web worker. The web worker
+exists to allow for this data, which for larger merge requests could be huge, to be processed off
+the main thread. Processing this data involves getting the data into the correct structure
+that the frontend requires to render the file browser in either tree view or list view.
+
+```mermaid
+graph TD
+ A[fetchDiffFilesMeta]
+ B[Create web worker]
+ C[Fetch data]
+ D[SET_DIFF_METADATA]
+ E[Post worker data]
+ F[Worker message event listener]
+ K[SET_TREE_DATA]
+
+ G[TreeWorker]
+ H[onMessage]
+ I[generateTreeList]
+ J[postMessage sortTree]
+
+ A -->B
+ E -->F
+ B -->C
+ C -->D
+ D -->E
+
+ G -->H
+ H -->I
+ I -->J
+ J -->F
+
+ F -->K
+```
+
+The structure for this file object is:
+
+```javascript
+{
+ "key": "",
+ "path": "",
+ "name": "",
+ "type": "",
+ "tree": [],
+ "changed": true,
+ "tempFile": false,
+ "deleted": false,
+ "fileHash": "",
+ "addedLines": 1,
+ "removedLines": 1,
+ "parentPath": "/",
+ "submodule": false
+}
+```
+
+#### Batch diffs
+
+To reduce the response size for the diffs endpoint, we are splitting this response up into different
+requests, to:
+
+- Reduces the response size of each request.
+- Allows the diffs app to start rendering diffs as quickly as the first request finishes.
+
+To make the first request quicker, the request gets sent asking for a small amount of
+diffs. The number of diffs requested then increases, until the maximum number of diffs per request is 30.
+
+When the request finishes, the diffs app formats the data received into a format that makes
+it easier for the diffs app to render the diffs lines.
+
+```mermaid
+graph TD
+ A[fetchDiffFilesBatch] -->
+ B[commit SET_DIFF_DATA_BATCH] -->
+ C[prepareDiffData] -->
+ D[prepareRawDiffFile] -->
+ E[ensureBasicDiffFileLines] -->
+ F[prepareDiffFileLines] -->
+ G[finalizeDiffFile] -->
+ H[deduplicateFilesList]
+```
+
+After this has been completed, the diffs app can now begin to render the diff lines. However, before
+anything can be rendered the diffs app does one more format. It takes the diff line data, and maps
+the data into a format for easier switching between inline and side-by-side modes. This
+formatting happens in a computed property inside the `diff_content.vue` component.
+
+### Render queue
+
+NOTE:
+This _might_ not be required any more. Some investigation work is required to decide
+the future of the render queue. The virtual scroll bar we created has probably removed
+any performance benefit we got from this approach.
+
+To render diffs quickly, we have a render queue that allows the diffs to render only if the
+browser is idle. This saves the browser getting frozen when rendering a lot of large diffs at once,
+and allows us to reduce the total blocking time.
+
+This pipeline of rendering files happens only if all the below conditions are `true` for every
+diff file. If any of these are `false`, then this render queue does not happen and the diffs get
+rendered as expected.
+
+- Are the diffs in this file already rendered?
+- Does this diff have a viewer? (Meaning, is it not a download?)
+- Is the diff expanded?
+
+This chart gives a brief overview of the pipeline that happens:
+
+```mermaid
+graph TD
+ A[startRenderDiffsQueue] -->B
+ B[commit RENDER_FILE current file index] -->C
+ C[canRenderNextFile?]
+ C -->|Yes| D[Render file] -->B
+ C -->|No| E[Re-run requestIdleCallback] -->C
+```
+
+The checks that happen:
+
+- Is the idle time remaining less than 5 ms?
+- Have we already tried to render this file 4 times?
+
+After these checks happen, the file is marked in Vuex as `renderable`, which allows the diffs
+app to start rendering the diff lines and discussions.
diff --git a/doc/development/merge_request_concepts/performance.md b/doc/development/merge_request_concepts/performance.md
index c1bdd45891d..740b8f1607b 100644
--- a/doc/development/merge_request_concepts/performance.md
+++ b/doc/development/merge_request_concepts/performance.md
@@ -15,7 +15,7 @@ with and agreed upon by backend maintainers and performance specialists.
It's also highly recommended that you read the following guides:
-- [Performance Guidelines../performance.md)
+- [Performance Guidelines](../performance.md)
- [Avoiding downtime in migrations](../database/avoiding_downtime_in_migrations.md)
## Definition
@@ -59,8 +59,8 @@ section below for more information.
about the impact.
Sometimes it's hard to assess the impact of a merge request. In this case you
-should ask one of the merge request reviewers to review your changes. You can
-find a list of these reviewers at <https://about.gitlab.com/company/team/>. A reviewer
+should ask one of the merge request reviewers to review your changes.
+([A list of reviewers](https://about.gitlab.com/company/team/) is available.) A reviewer
in turn can request a performance specialist to review the changes.
## Think outside of the box
@@ -119,7 +119,7 @@ data migration. Migrating millions of rows is always troublesome and
can have a negative impact on the application.
To better understand how to get help with the query plan reviews
-read this section on [how to prepare the merge request for a database review../database_review.md#how-to-prepare-the-merge-request-for-a-database-review).
+read this section on [how to prepare the merge request for a database review](../database_review.md#how-to-prepare-the-merge-request-for-a-database-review).
## Query Counts
@@ -193,7 +193,12 @@ costly, time-consuming query to the replicas.
## Use CTEs wisely
-Read about [complex queries on the relation object../database/iterating_tables_in_batches.md#complex-queries-on-the-relation-object) for considerations on how to use CTEs. We have found in some situations that CTEs can become problematic in use (similar to the n+1 problem above). In particular, hierarchical recursive CTE queries such as the CTE in [AuthorizedProjectsWorker](https://gitlab.com/gitlab-org/gitlab/-/issues/325688) are very difficult to optimize and don't scale. We should avoid them when implementing new features that require any kind of hierarchical structure.
+Read about [complex queries on the relation object](../database/iterating_tables_in_batches.md#complex-queries-on-the-relation-object)
+for considerations on how to use CTEs. We have found in some situations that CTEs can become
+problematic in use (similar to the n+1 problem above). In particular, hierarchical recursive
+CTE queries such as the CTE in [AuthorizedProjectsWorker](https://gitlab.com/gitlab-org/gitlab/-/issues/325688)
+are very difficult to optimize and don't scale. We should avoid them when implementing new features
+that require any kind of hierarchical structure.
CTEs have been effectively used as an optimization fence in many simpler cases,
such as this [example](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/43242#note_61416277).
@@ -224,7 +229,7 @@ The total number of the queries (including cached ones) executed by the code mod
should not increase unless absolutely necessary.
The number of executed queries (including cached queries) should not depend on
collection size.
-You can write a test by passing the `skip_cached` variable to [QueryRecorder../database/query_recorder.md) to detect this and prevent regressions.
+You can write a test by passing the `skip_cached` variable to [QueryRecorder](../database/query_recorder.md) to detect this and prevent regressions.
As an example, say you have a CI pipeline. All pipeline builds belong to the same pipeline,
thus they also belong to the same project (`pipeline.project`):
@@ -312,7 +317,7 @@ This could result in Puma timeout and should be avoided at all cost.
You should set a reasonable timeout, gracefully handle exceptions and surface the
errors in UI or logging internally.
-Using [`ReactiveCaching`../utilities.md#reactivecaching) is one of the best solutions to fetch external data.
+Using [`ReactiveCaching`](../utilities.md#reactivecaching) is one of the best solutions to fetch external data.
## Keep database transaction minimal
@@ -424,7 +429,7 @@ Take into consideration the following when choosing a pagination strategy:
The database has to sort and iterate all previous items, and this operation usually
can result in substantial load put on database.
-You can find useful tips related to pagination in the [pagination guidelines../database/pagination_guidelines.md).
+You can find useful tips related to pagination in the [pagination guidelines](../database/pagination_guidelines.md).
## Badge counters
@@ -561,5 +566,5 @@ can time out, which is especially problematic for slow clients. If clients take
to upload/download the processing slot might be killed due to request processing
timeout (usually between 30s-60s).
-For the above reasons it is required that [Workhorse direct upload../uploads/index.md#direct-upload) is implemented
+For the above reasons it is required that [Workhorse direct upload](../uploads/index.md#direct-upload) is implemented
for all file uploads and downloads.
diff --git a/doc/development/merge_request_concepts/rate_limits.md b/doc/development/merge_request_concepts/rate_limits.md
new file mode 100644
index 00000000000..97d20b57eb4
--- /dev/null
+++ b/doc/development/merge_request_concepts/rate_limits.md
@@ -0,0 +1,28 @@
+---
+stage: Create
+group: Source Code
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Application and rate limit guidelines
+
+GitLab, like most large applications, enforces limits within certain features.
+The absences of limits can affect security, performance, data, or could even
+exhaust the allocated resources for the application.
+
+Every new feature should have safe usage limits included in its implementation.
+Limits are applicable for:
+
+- System-level resource pools such as API requests, SSHD connections, database connections, storage, and so on.
+- Domain-level objects such as CI/CD minutes, groups, sign-in attempts, and so on.
+
+## When limits are required
+
+1. Limits are required if the absence of the limit matches severity 1 - 3 in the severity definitions for [limit-related bugs](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#limit-related-bugs).
+1. [GitLab application limits](../../administration/instance_limits.md) documentation must be updated anytime limits are added, removed, or updated.
+
+## Additional reading
+
+- Existing [GitLab application limits](../../administration/instance_limits.md)
+- Product processes: [introducing application limits](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits)
+- Development docs: [guide for adding application limits](../application_limits.md)
diff --git a/doc/development/newlines_styleguide.md b/doc/development/newlines_styleguide.md
deleted file mode 100644
index 014affa3e04..00000000000
--- a/doc/development/newlines_styleguide.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-redirect_to: 'backend/ruby_style_guide.md#newlines-style-guide'
-remove_date: '2022-12-15'
----
-
-This document was moved to [another location](backend/ruby_style_guide.md#newlines-style-guide).
-
-<!-- This redirect file can be deleted after 2022-12-15. -->
-<!-- Redirects that point to other docs in the same project expire in three months. -->
-<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
-<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/development/packages/dependency_proxy.md b/doc/development/packages/dependency_proxy.md
index cc8b202e556..e9568699c7e 100644
--- a/doc/development/packages/dependency_proxy.md
+++ b/doc/development/packages/dependency_proxy.md
@@ -6,9 +6,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Dependency Proxy
-The Dependency Proxy is a pull-through-cache for registry images from DockerHub. This document describes how this
+The Dependency Proxy is a pull-through-cache for public registry images from DockerHub. This document describes how this
feature is constructed in GitLab.
+NOTE:
+Support for private registry images is proposed in [issue 331741](https://gitlab.com/gitlab-org/gitlab/-/issues/331741).
+
## Container registry
The Dependency Proxy for the container registry acts a stand-in for a remote container registry. In our case,
diff --git a/doc/development/packages/harbor_registry_development.md b/doc/development/packages/harbor_registry_development.md
new file mode 100644
index 00000000000..dc97ecfa7b2
--- /dev/null
+++ b/doc/development/packages/harbor_registry_development.md
@@ -0,0 +1,153 @@
+---
+stage: Package
+group: Harbor Registry
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+# Harbor Registry
+
+## Enable Harbor Registry
+
+To enable the Harbor Registry, you must configure the Harbor integration for your group or project.
+The Harbor configuration requires four fields: `url`, `project_name`, `username` and `password`.
+
+| Field | Description |
+| --- | --- |
+| `url` | The URL of the Harbor instance. |
+| `project_name` | The project name of the Harbor instance. |
+| `username` | The username used to log in to the Harbor instance. |
+| `password` | The password used to log in to the Harbor instance. |
+
+You can use [GitLab CI/CD predefined variables](../../ci/variables/index.md) along with the following Harbor Registry variables to request data from the Harbor instance.
+
+| Variable | Description |
+| --- | --- |
+| `HARBOR_URL` | The URL of the Harbor instance. |
+| `HARBOR_HOST` | The host of the Harbor instance URL. |
+| `HARBOR_OCI` | The OCI URL of the Harbor instance URL. |
+| `HARBOR_PROJECT` | The project name of the Harbor instance. |
+| `HARBOR_USERNAME` | The username used to log in to the Harbor instance. |
+| `HARBOR_PASSWORD` | The password used to log in to the Harbor instance. |
+
+### Test settings
+
+When testing the settings, a request is sent to `/api/v2.0/ping` of the Harbor instance. A successful test returns status code `200`. This test is primarily to verify that the Harbor instance is configured correctly. It doesn't verify that the `username` and `password` are correct.
+
+## Code structure
+
+```shell
+app/controllers/concerns/harbor
+├── access.rb
+├── artifact.rb
+├── repository.rb
+└── tag.rb
+
+app/controllers/projects/harbor
+├── application_controller.rb
+├── artifacts_controller.rb
+├── repositories_controller.rb
+└── tags_controller.rb
+
+app/controllers/groups/harbor
+├── application_controller.rb
+├── artifacts_controller.rb
+├── repositories_controller.rb
+└── tags_controller.rb
+
+app/models/integrations/harbor.rb
+
+app/serializers/integrations/harbor_serializers
+├── artifact_entity.rb
+├── artifact_serializer.rb
+├── repository_entity.rb
+├── repository_serializer.rb
+├── tag_entity.rb
+└── tag_serializer.rb
+
+lib/gitlab/harbor
+├── client.rb
+└── query.rb
+```
+
+The controllers under `app/controllers/projects/harbor` and `app/controllers/groups/harbor` provide the API interface for front-end calls.
+
+The modules under `app/controllers/concerns/harbor` provide some common methods used by controllers.
+
+The Harbor integration model is under `app/models/integrations`, and it contains some configuration information for Harbor integration.
+
+The serializers under `app/serializers/integrations/harbor_serializers` are used by the controllers under `app/controllers/projects/harbor` and `app/controllers/groups/harbor`, and they help controllers to serialize the JSON data in the response.
+
+The `lib/gitlab/harbor` directory contains the Harbor client, which sends API requests to the Harbor instances to retrieve data.
+
+## Sequence diagram
+
+```mermaid
+sequenceDiagram
+ Client->>+GitLab: Request Harbor Registry
+ GitLab->>+Harbor instance: Request repositories data via API
+ Harbor instance->>+GitLab: Repositories data
+ GitLab->>+Client: Return repositories data
+ Client->>+GitLab: Request Harbor Registry artifacts
+ GitLab->>+Harbor instance: Request artifacts data via API
+ Harbor instance->>+GitLab: Artifacts data
+ GitLab->>+Client: Return artifacts data
+ Client->>+GitLab: Request Harbor Registry tags
+ GitLab->>+Harbor instance: Request tags data via API
+ Harbor instance->>+GitLab: Tags data
+ GitLab->>+Client: Return tags data
+```
+
+## Policy
+
+The`read_harbor_registry` policy for groups and projects is used to control whether users have access to Harbor Registry.
+This policy is enabled for every user with the Reporter role and above.
+
+## Frontend Development
+
+The relevant front-end code is located in the `app/assets/javascripts/packages_and_registries/harbor_registry/` directory. The file structure is as follows:
+
+```shell
+├── components
+│ ├── details
+│ │ ├── artifacts_list_row.vue
+│ │ ├── artifacts_list.vue
+│ │ └── details_header.vue
+│ ├── list
+│ │ ├── harbor_list_header.vue
+│ │ ├── harbor_list_row.vue
+│ │ └── harbor_list.vue
+│ ├── tags
+│ │ ├── tags_header.vue
+│ │ ├── tags_list_row.vue
+│ │ └── tags_list.vue
+│ └── harbor_registry_breadcrumb.vue
+├── constants
+│ ├── common.js
+│ ├── details.js
+│ ├── index.js
+│ └── list.js
+├── pages
+│ ├── details.vue
+│ ├── harbor_tags.vue
+│ ├── index.vue
+│ └── list.vue
+├── index.js
+├── router.js
+└── utils.js
+```
+
+NOTE:
+You can check out this [discussion](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82777#note_1017875324) to see why we use the REST API instead of GraphQL.
+
+The file `harbor_registry/pages/index.vue` only contains a single Vue router-view component, which navigates to the `images list`, `image detail`, and `tags list` pages via `router.js`.
+
+Because `registry_breadcrumb.vue` component does not support multi-level paths, we have reimplemented the `harbor_registry/components/harbor_registry_breadcrumb.vue` component.
+
+A multi-level breadcrumb component can be generated by passing a path array to `harbor_registry_breadcrumb.vue`.
+
+```javascript
+const routeNameList = [];
+const hrefList = [];
+
+this.breadCrumbState.updateName(nameList);
+this.breadCrumbState.updateHref(hrefList);
+```
diff --git a/doc/development/packages/index.md b/doc/development/packages/index.md
index e6ec7e9654a..fa0e9f5d926 100644
--- a/doc/development/packages/index.md
+++ b/doc/development/packages/index.md
@@ -37,3 +37,9 @@ Development and architectural documentation for the container registry
- [Settings](settings.md)
- [Structure / Schema](structure.md)
- [Cleanup policies](cleanup_policies.md)
+
+## Harbor registry development
+
+Development and architectural documentation for the harbor registry
+
+- [Development documentation](harbor_registry_development.md)
diff --git a/doc/development/pages/index.md b/doc/development/pages/index.md
index 05eeb1965d1..e71d7df642c 100644
--- a/doc/development/pages/index.md
+++ b/doc/development/pages/index.md
@@ -6,11 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w
description: "GitLab's development guidelines for GitLab Pages"
---
-# Getting started with development
+# Contribute to GitLab Pages development
+
+Learn how to configure GitLab Pages so you can help develop the feature.
## Configuring GitLab Pages hostname
-GitLab Pages need a hostname or domain, as each different GitLab Pages site is accessed via a
+GitLab Pages needs a hostname or domain, as each different GitLab Pages site is accessed through a
subdomain. You can set the GitLab Pages hostname:
- [Without wildcard, editing your hosts file](#without-wildcard-editing-your-hosts-file).
diff --git a/doc/development/performance.md b/doc/development/performance.md
index 21f80364358..346f70e04b0 100644
--- a/doc/development/performance.md
+++ b/doc/development/performance.md
@@ -154,7 +154,7 @@ allowing you to profile which code is running on CPU in detail.
It's important to note that profiling an application *alters its performance*.
Different profiling strategies have different overheads. Stackprof is a sampling
profiler. It samples stack traces from running threads at a configurable
-frequency (for example, 100hz, that is 100 stacks per second). This type of profiling
+frequency (for example, 100 hz, that is 100 stacks per second). This type of profiling
has quite a low (albeit non-zero) overhead and is generally considered to be
safe for production.
@@ -234,7 +234,7 @@ The following configuration options can be configured:
- `STACKPROF_INTERVAL`: Sampling interval. Unit semantics depend on `STACKPROF_MODE`.
For `object` mode this is a per-event interval (every `nth` event is sampled)
and defaults to `100`.
- For other modes such as `cpu` this is a frequency interval and defaults to `10100` μs (99hz).
+ For other modes such as `cpu` this is a frequency interval and defaults to `10100` μs (99 hz).
- `STACKPROF_FILE_PREFIX`: File path prefix where profiles are stored. Defaults
to `$TMPDIR` (often corresponds to `/tmp`).
- `STACKPROF_TIMEOUT_S`: Profiling timeout in seconds. Profiling will
@@ -477,7 +477,7 @@ The `mem_*` values represent different aspects of how objects and memory are all
=> {:mem_objects=>1002, :mem_bytes=>32000, :mem_mallocs=>1000}
```
-- The following example will allocate over 40kB of data, and perform only a single memory allocation.
+- The following example will allocate over 40 kB of data, and perform only a single memory allocation.
The existing object will be reallocated/resized on subsequent iterations:
```ruby
@@ -583,8 +583,8 @@ You may find the results:
**Metrics Reports** [dropdown list](../ci/testing/metrics_reports.md).
- In the `memory-on-boot` artifacts for a full report and a dependency breakdown.
-`derailed_benchmarks` also provides other methods to investigate memory. To learn more,
-refer to the [gem documentation](https://github.com/zombocom/derailed_benchmarks#running-derailed-exec).
+`derailed_benchmarks` also provides other methods to investigate memory. For more information, see
+the [gem documentation](https://github.com/zombocom/derailed_benchmarks#running-derailed-exec).
Most of the methods (`derailed exec perf:*`) attempt to boot your Rails app in a
`production` environment and run benchmarks against it.
It is possible both in GDK and GCK:
@@ -730,7 +730,7 @@ test = +"hello"
test += " world"
```
-When adding new Ruby files, please check that you can add the above header,
+When adding new Ruby files, check that you can add the above header,
as omitting it may lead to style check failures.
## Banzai pipelines and filters
@@ -825,7 +825,7 @@ source into memory, memory use grows by _at least_ the size of the data source.
In the case of `readlines`, it grows even further, due to extra bookkeeping
the Ruby VM has to perform to represent each line.
-Consider the following program, which reads a text file that is 750MB on disk:
+Consider the following program, which reads a text file that is 750 MB on disk:
```ruby
File.readlines('large_file.txt').each do |line|
@@ -859,7 +859,7 @@ We can see that `heap_live_slots` (the number of reachable objects) jumped to ~2
which is roughly two orders of magnitude more compared to reading the file line by
line instead. It was not just the raw memory usage that increased, but also how the garbage collector (GC)
responded to this change in anticipation of future memory use. We can see that `malloc_increase_bytes` jumped
-to ~30MB, which compares to just ~4kB for a "fresh" Ruby program. This figure specifies how
+to ~30 MB, which compares to just ~4 kB for a "fresh" Ruby program. This figure specifies how
much additional heap space the Ruby GC claims from the operating system next time it runs out of memory.
Not only did we occupy more memory, we also changed the behavior of the application
to increase memory use at a faster rate.
diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md
deleted file mode 100644
index a4e06e98d14..00000000000
--- a/doc/development/pipelines.md
+++ /dev/null
@@ -1,11 +0,0 @@
----
-redirect_to: 'pipelines/index.md'
-remove_date: '2023-01-20'
----
-
-This document was moved to [another location](pipelines/index.md).
-
-<!-- This redirect file can be deleted after <2023-01-20>. -->
-<!-- Redirects that point to other docs in the same project expire in three months. -->
-<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. -->
-<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/development/pipelines/index.md b/doc/development/pipelines/index.md
index 1797e082aea..240d98a855f 100644
--- a/doc/development/pipelines/index.md
+++ b/doc/development/pipelines/index.md
@@ -188,7 +188,7 @@ Note that the merge request also needs to have the `master:broken` or `master:fo
To make your Revert MRs faster, use the [revert MR template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/merge_request_templates/Revert%20To%20Resolve%20Incident.md) **before** you create your merge request. It will apply the `pipeline:expedite` label and others that will expedite the pipelines that run on the merge request.
-### The `~pipeline:expedite` label
+### The `pipeline:expedite` label
When this label is assigned, the following steps of the CI/CD pipeline are skipped:
@@ -207,83 +207,33 @@ If you want to force all the RSpec jobs to run regardless of your changes, you c
WARNING:
Forcing all jobs on docs only related MRs would not have the prerequisite jobs and would lead to errors
-### Test suite parallelization
+### End-to-end jobs
-Our current RSpec tests parallelization setup is as follows:
-
-1. The `retrieve-tests-metadata` job in the `prepare` stage ensures we have a
- `knapsack/report-master.json` file:
- - The `knapsack/report-master.json` file is fetched from the latest `main` pipeline which runs `update-tests-metadata`
- (for now it's the 2-hourly `maintenance` scheduled master pipeline), if it's not here we initialize the file with `{}`.
-1. Each `[rspec|rspec-ee] [migration|unit|integration|system|geo] n m` job are run with
- `knapsack rspec` and should have an evenly distributed share of tests:
- - It works because the jobs have access to the `knapsack/report-master.json`
- since the "artifacts from all previous stages are passed by default".
- - the jobs set their own report path to
- `"knapsack/${TEST_TOOL}_${TEST_LEVEL}_${DATABASE}_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json"`.
- - if knapsack is doing its job, test files that are run should be listed under
- `Report specs`, not under `Leftover specs`.
-1. The `update-tests-metadata` job (which only runs on scheduled pipelines for
- [the canonical project](https://gitlab.com/gitlab-org/gitlab) takes all the
- `knapsack/rspec*.json` files and merge them all together into a single
- `knapsack/report-master.json` file that is saved as artifact.
-
-After that, the next pipeline uses the up-to-date `knapsack/report-master.json` file.
-
-### Flaky tests
-
-#### Automatic skipping of flaky tests
-
-Tests that are [known to be flaky](../testing_guide/flaky_tests.md#automatic-retries-and-flaky-tests-detection) are
-skipped unless the `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable is set to `false` or if the `~"pipeline:run-flaky-tests"`
-label is set on the MR.
-
-See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1069).
-
-#### Automatic retry of failing tests in a separate process
-
-Unless `$RETRY_FAILED_TESTS_IN_NEW_PROCESS` variable is set to `false` (`true` by default), RSpec tests that failed are automatically retried once in a separate
-RSpec process. The goal is to get rid of most side-effects from previous tests that may lead to a subsequent test failure.
-
-We keep track of retried tests in the `$RETRIED_TESTS_REPORT_FILE` file saved as artifact by the `rspec:flaky-tests-report` job.
-
-See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1148).
-
-### Compatibility testing
-
-By default, we run all tests with the versions that runs on GitLab.com.
-
-Other versions (usually one back-compatible version, and one forward-compatible version) should be running in nightly scheduled pipelines.
-
-Exceptions to this general guideline should be motivated and documented.
-
-#### Single database testing
-
-By default, all tests run with [multiple databases](../database/multiple_databases.md).
-
-We also run tests with a single database in nightly scheduled pipelines, and in merge requests that touch database-related files.
-
-If you want to force tests to run with a single database, you can add the `pipeline:run-single-db` label to the merge request.
+The [`e2e:package-and-test`](../testing_guide/end_to_end/index.md#using-the-package-and-test-job) child pipeline
+runs end-to-end jobs automatically depending on changes, and is manual in other cases.
+See `.qa:rules:package-and-test` in
+[`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml) for
+the specific list of rules.
-### Monitoring
+If you want to force `e2e:package-and-test` to run regardless of your changes, you can add the
+`pipeline:run-all-e2e` label to the merge request.
-The GitLab test suite is [monitored](../performance.md#rspec-profiling) for the `main` branch, and any branch
-that includes `rspec-profile` in their name.
+Consult the [End-to-end Testing](../testing_guide/end_to_end/index.md) dedicated page for more information.
-### Logging
+### Review app jobs
-- Rails logging to `log/test.log` is disabled by default in CI
- [for performance reasons](https://jtway.co/speed-up-your-rails-test-suite-by-6-in-1-line-13fedb869ec4).
- To override this setting, provide the
- `RAILS_ENABLE_TEST_LOG` environment variable.
+The [`start-review-app-pipeline`](../testing_guide/review_apps.md) child pipeline deploys a Review App and runs
+end-to-end tests against it automatically depending on changes, and is manual in other cases.
+See `.review:rules:start-review-app-pipeline` in
+[`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml) for
+the specific list of rules.
-## Review app jobs
+If you want to force a Review App to be deployed regardless of your changes, you can add the
+`pipeline:run-review-app` label to the merge request.
Consult the [Review Apps](../testing_guide/review_apps.md) dedicated page for more information.
-If you want to force a Review App to be deployed regardless of your changes, you can add the `pipeline:run-review-app` label to the merge request.
-
-## As-if-FOSS jobs
+### As-if-FOSS jobs
The `* as-if-foss` jobs run the GitLab test suite "as if FOSS", meaning as if the jobs would run in the context
of `gitlab-org/gitlab-foss`. These jobs are only created in the following cases:
@@ -297,7 +247,7 @@ set and get the `ee/` folder removed before the tests start running.
The intent is to ensure that a change doesn't introduce a failure after `gitlab-org/gitlab` is synced to `gitlab-org/gitlab-foss`.
-## As-if-JH cross project downstream pipeline
+### As-if-JH cross project downstream pipeline
The `start-as-if-jh` job triggers a cross project downstream pipeline which
runs the GitLab test suite "as if JiHu", meaning as if the pipeline would run
@@ -321,13 +271,13 @@ The intent is to ensure that a change doesn't introduce a failure after
[GitLab](https://gitlab.com/gitlab-org/gitlab) is synchronized to
[GitLab JH](https://jihulab.com/gitlab-cn/gitlab).
-### When to consider applying `pipeline:run-as-if-jh` label
+#### When to consider applying `pipeline:run-as-if-jh` label
If a Ruby file is renamed and there's a corresponding [`prepend_mod` line](../jh_features_review.md#jh-features-based-on-ce-or-ee-features),
it's likely that GitLab JH is relying on it and requires a corresponding
change to rename the module or class it's prepending.
-### Corresponding JH branch
+#### Corresponding JH branch
You can create a corresponding JH branch on [GitLab JH](https://jihulab.com/gitlab-cn/gitlab) by
appending `-jh` to the branch name. If a corresponding JH branch is found,
@@ -344,7 +294,7 @@ it does not include any corresponding JH branch beside the default `main-jh`.
This is why when we want to fetch corresponding JH branch we should fetch it
from the main mirror, rather than the validation project.
-### How as-if-JH pipeline was configured
+#### How as-if-JH pipeline was configured
The whole process looks like this:
@@ -373,14 +323,14 @@ flowchart TD
JH --"pull mirror with corresponding JH branches"--> Mirror
```
-#### Tokens set in the project variables
+##### Tokens set in the project variables
- `ADD_JH_FILES_TOKEN`: This is a [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab)
project token with `read_api` permission, to be able to download JiHu files.
- `AS_IF_JH_TOKEN`: This is a [GitLab JH validation](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation)
project token with `write_repository` permission, to push generated `as-if-jh/*` branch.
-#### How we generate the as-if-JH branch
+##### How we generate the as-if-JH branch
First `add-jh-files` job will download the required JiHu files from the
corresponding JH branch, saving in artifacts. Next `prepare-as-if-jh-branch`
@@ -388,13 +338,13 @@ job will create a new branch from the merge request branch, commit the
changes, and finally push the branch to the
[validation project](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation).
-#### How we trigger and run the as-if-JH pipeline
+##### How we trigger and run the as-if-JH pipeline
After having the `as-if-jh/*` branch, `start-as-if-jh` job will trigger a pipeline
in the [validation project](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation)
to run the cross-project downstream pipeline.
-#### How the GitLab JH mirror project is set up
+##### How the GitLab JH mirror project is set up
The [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab) project is private and CI is disabled.
@@ -408,7 +358,7 @@ engineering vault.
No password is used from mirroring because GitLab JH is a public project.
-#### How the GitLab JH validation project is set up
+##### How the GitLab JH validation project is set up
This [GitLab JH validation](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation) project is public and CI is enabled, without any project variables.
@@ -432,24 +382,7 @@ running every day, updating cache.
The default CI/CD configuration file is also set at `jh/.gitlab-ci.yml` so it
runs exactly like [GitLab JH](https://jihulab.com/gitlab-cn/gitlab/-/blob/main-jh/jh/.gitlab-ci.yml).
-## Ruby 2.7 jobs
-
-We're running Ruby 3.0 for the merge requests and the default branch. However,
-we're still running Ruby 2.7 for GitLab.com and there are older versions that
-we need to maintain. We need a way to still try out Ruby 2.7 in merge requests.
-
-You can add the `pipeline:run-in-ruby2` label to the merge request to switch
-the Ruby version used for running the whole test suite to 2.7. When you do
-this, the test suite will no longer run in Ruby 3.0 (default), and an
-additional job `verify-ruby-3.0` will also run and always fail to remind us to
-remove the label and run in Ruby 3.0 before merging the merge request.
-
-This should let us:
-
-- Test changes for Ruby 2.7
-- Make sure it will not break anything when it's merged into the default branch
-
-## `undercover` RSpec test
+### `rspec:undercoverage` job
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74859) in GitLab 14.6.
@@ -463,25 +396,90 @@ In the event of an emergency, or false positive from this job, add the
`pipeline:skip-undercoverage` label to the merge request to allow this job to
fail.
-### Troubleshooting `rspec:undercoverage` failures
+#### Troubleshooting `rspec:undercoverage` failures
The `rspec:undercoverage` job has [known bugs](https://gitlab.com/groups/gitlab-org/-/epics/8254)
that can cause false positive failures. You can test coverage locally to determine if it's
-safe to apply `~"pipeline:skip-undercoverage"`. For example, using `<spec>` as the name of the
+safe to apply `pipeline:skip-undercoverage`. For example, using `<spec>` as the name of the
test causing the failure:
1. Run `SIMPLECOV=1 bundle exec rspec <spec>`.
1. Run `scripts/undercoverage`.
-If these commands return `undercover: ✅ No coverage is missing in latest changes` then you can apply `~"pipeline:skip-undercoverage"` to bypass pipeline failures.
+If these commands return `undercover: ✅ No coverage is missing in latest changes` then you can apply `pipeline:skip-undercoverage` to bypass pipeline failures.
-## Ruby versions testing
+## Test suite parallelization
-Our test suite runs against Ruby 3 in merge requests and default branch pipelines.
+Our current RSpec tests parallelization setup is as follows:
-We also run our test suite against Ruby 2.7 on another 2-hourly scheduled pipelines, as GitLab.com still runs on Ruby 2.7.
+1. The `retrieve-tests-metadata` job in the `prepare` stage ensures we have a
+ `knapsack/report-master.json` file:
+ - The `knapsack/report-master.json` file is fetched from the latest `main` pipeline which runs `update-tests-metadata`
+ (for now it's the 2-hourly `maintenance` scheduled master pipeline), if it's not here we initialize the file with `{}`.
+1. Each `[rspec|rspec-ee] [migration|unit|integration|system|geo] n m` job are run with
+ `knapsack rspec` and should have an evenly distributed share of tests:
+ - It works because the jobs have access to the `knapsack/report-master.json`
+ since the "artifacts from all previous stages are passed by default".
+ - the jobs set their own report path to
+ `"knapsack/${TEST_TOOL}_${TEST_LEVEL}_${DATABASE}_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json"`.
+ - if knapsack is doing its job, test files that are run should be listed under
+ `Report specs`, not under `Leftover specs`.
+1. The `update-tests-metadata` job (which only runs on scheduled pipelines for
+ [the canonical project](https://gitlab.com/gitlab-org/gitlab) takes all the
+ `knapsack/rspec*.json` files and merge them all together into a single
+ `knapsack/report-master.json` file that is saved as artifact.
+
+After that, the next pipeline uses the up-to-date `knapsack/report-master.json` file.
+
+## Flaky tests
-## PostgreSQL versions testing
+### Automatic skipping of flaky tests
+
+We used to skip tests that are [known to be flaky](../testing_guide/flaky_tests.md#automatic-retries-and-flaky-tests-detection),
+but we stopped doing so since that could actually lead to actual broken `master`.
+Instead, we proactively quarantine any flaky test reported in `#master-broken` incidents
+so that they're ultimately fixed by their respective group.
+
+The automatic skipping of flaky tests can still be enabled by setting the `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable to `true`.
+
+See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1069).
+
+### Automatic retry of failing tests in a separate process
+
+Unless `$RETRY_FAILED_TESTS_IN_NEW_PROCESS` variable is set to `false` (`true` by default), RSpec tests that failed are automatically retried once in a separate
+RSpec process. The goal is to get rid of most side-effects from previous tests that may lead to a subsequent test failure.
+
+We keep track of retried tests in the `$RETRIED_TESTS_REPORT_FILE` file saved as artifact by the `rspec:flaky-tests-report` job.
+
+See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1148).
+
+## Compatibility testing
+
+By default, we run all tests with the versions that runs on GitLab.com.
+
+Other versions (usually one back-compatible version, and one forward-compatible version) should be running in nightly scheduled pipelines.
+
+Exceptions to this general guideline should be motivated and documented.
+
+### Ruby versions testing
+
+We're running Ruby 3.0 for the merge requests and the default branch. However,
+we're still running Ruby 2.7 for GitLab.com and there are older versions that
+we need to maintain, so we also run our test suite against Ruby 2.7 on a
+dedicated 2-hourly scheduled pipelines.
+
+For merge requests, you can add the `pipeline:run-in-ruby2` label to switch
+the Ruby version used for running the whole test suite to 2.7. When you do
+this, the test suite will no longer run in Ruby 3.0 (default), and an
+additional job `verify-ruby-3.0` will also run and always fail to remind us to
+remove the label and run in Ruby 3.0 before merging the merge request.
+
+This should let us:
+
+- Test changes for Ruby 2.7
+- Make sure it will not break anything when it's merged into the default branch
+
+### PostgreSQL versions testing
Our test suite runs against PG12 as GitLab.com runs on PG12 and
[Omnibus defaults to PG12 for new installs and upgrades](../../administration/package_information/postgresql_versions.md).
@@ -490,7 +488,7 @@ We do run our test suite against PG11 and PG13 on nightly scheduled pipelines.
We also run our test suite against PG11 upon specific database library changes in MRs and `main` pipelines (with the `rspec db-library-code pg11` job).
-### Current versions testing
+#### Current versions testing
| Where? | PostgreSQL version | Ruby version |
|------------------------------------------------------------------------------------------------|-------------------------------------------------|--------------|
@@ -506,16 +504,14 @@ pipeline in `ruby2-sync` branch, which updates the `ruby2` branch with latest
is triggering a pipeline in `ruby2` 5 minutes after it, which is considered
the maintenance schedule to run test suites and update cache.
-Any changes in `ruby2` are only for running the pipeline. It should
-never be merged back to `master`. Any other Ruby 2.7 changes should go into
-`master` directly, which should be compatible with Ruby 3.
+The `ruby2` branch must not have any changes. The branch is only there to set
+`RUBY_VERSION` to `2.7` in the maintenance pipeline schedule.
-Previously, `ruby2-sync` was using a project token stored in `RUBY2_SYNC_TOKEN`
-(now backed up in `RUBY2_SYNC_TOKEN_NOT_USED`), however due to various
-permissions issues, we ended up using an access token from `gitlab-bot` so now
-`RUBY2_SYNC_TOKEN` is actually an access token from `gitlab-bot`.
+The `gitlab` job in the `ruby2-sync` branch uses a `gitlab-org/gitlab` project
+token with `write_repository` scope and `Maintainer` role with no expiration.
+The token is stored in the `RUBY2_SYNC_TOKEN` variable in `gitlab-org/gitlab`.
-### Long-term plan
+#### Long-term plan
We follow the [PostgreSQL versions shipped with Omnibus GitLab](../../administration/package_information/postgresql_versions.md):
@@ -525,14 +521,14 @@ We follow the [PostgreSQL versions shipped with Omnibus GitLab](../../administra
| PG11 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` |
| PG13 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` |
-## Redis versions testing
+### Redis versions testing
Our test suite runs against Redis 6 as GitLab.com runs on Redis 6 and
[Omnibus defaults to Redis 6 for new installs and upgrades](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/config/software/redis.rb).
We do run our test suite against Redis 5 on `nightly` scheduled pipelines, specifically when running backward-compatible and forward-compatible PostgreSQL jobs.
-### Current versions testing
+#### Current versions testing
| Where? | Redis version |
| ------ | ------------------ |
@@ -540,6 +536,26 @@ We do run our test suite against Redis 5 on `nightly` scheduled pipelines, speci
| `default branch` (non-scheduled pipelines) | 6 |
| `nightly` scheduled pipelines | 5 |
+### Single database testing
+
+By default, all tests run with [multiple databases](../database/multiple_databases.md).
+
+We also run tests with a single database in nightly scheduled pipelines, and in merge requests that touch database-related files.
+
+If you want to force tests to run with a single database, you can add the `pipeline:run-single-db` label to the merge request.
+
+## Monitoring
+
+The GitLab test suite is [monitored](../performance.md#rspec-profiling) for the `main` branch, and any branch
+that includes `rspec-profile` in their name.
+
+## Logging
+
+- Rails logging to `log/test.log` is disabled by default in CI
+ [for performance reasons](https://jtway.co/speed-up-your-rails-test-suite-by-6-in-1-line-13fedb869ec4).
+ To override this setting, provide the
+ `RAILS_ENABLE_TEST_LOG` environment variable.
+
## Pipelines types for merge requests
In general, pipelines for an MR fall into one of the following types (from shorter to longer), depending on the changes made in the MR:
diff --git a/doc/development/product_qualified_lead_guide/index.md b/doc/development/product_qualified_lead_guide/index.md
index 1ad4f622829..c72110bc253 100644
--- a/doc/development/product_qualified_lead_guide/index.md
+++ b/doc/development/product_qualified_lead_guide/index.md
@@ -13,36 +13,34 @@ A hand-raise PQL is a user who requests to speak to sales from within the produc
## Set up your development environment
1. Set up GDK with a connection to your local CustomersDot instance.
-1. Set up CustomersDot to talk to a staging instance of Platypus.
+1. Set up CustomersDot to talk to a staging instance of Workato.
1. Set up CustomersDot using the [normal install instructions](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/staging/doc/setup/installation_steps.md).
1. Set the `CUSTOMER_PORTAL_URL` environment variable to your local (or ngrok) URL of your CustomersDot instance.
1. Place `export CUSTOMER_PORTAL_URL='https://XXX.ngrok.io/'` in your shell `rc` script (`~/.zshrc` or `~/.bash_profile` or `~/.bashrc`) and restart GDK.
-1. Enter the credentials on CustomersDot development to Platypus in your `/config/secrets.yml` and restart. Credentials for the Platypus Staging are in the 1Password Growth vault. The URL for staging is `https://staging.ci.nexus.gitlabenvironment.cloud`.
+1. Enter the credentials on CustomersDot development to Workato in your `/config/secrets.yml` and restart. Credentials for the Workato Staging are in the 1Password Subscription portal vault. The URL for staging is `https://apim.workato.com/gitlab-dev/services/marketo/lead`.
```yaml
- platypus_url: "<%= ENV['PLATYPUS_URL'] %>"
- platypus_client_id: "<%= ENV['PLATYPUS_CLIENT_ID'] %>"
- platypus_client_secret: "<%= ENV['PLATYPUS_CLIENT_SECRET'] %>"
+ workato_url: "<%= ENV['WORKATO_URL'] %>"
+ workato_client_id: "<%= ENV['WORKATO_CLIENT_ID'] %>"
+ workato_client_secret: "<%= ENV['WORKATO_CLIENT_SECRET'] %>"
```
### Set up lead monitoring
-1. Set up access for Platypus Staging `https://staging.ci.nexus.gitlabenvironment.cloud` using the Platypus Staging credentials in the 1Password Growth vault.
1. Set up access for the Marketo sandbox, similar [to this example request](https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/13162).
### Manually test leads
1. Register a new user with a unique email on your local GitLab instance.
1. Send the PQL lead by submitting your new form or creating a new trial or a new hand raise lead.
-1. Use easily identifiable values that can be easily seen in Platypus staging.
-1. Observe the entry in the staging instance of Platypus and paste in the merge request comment and mention.
+1. Use easily identifiable values that can be easily seen in Workato staging.
+1. Observe the entry in the staging instance of Workato and paste in the merge request comment and mention.
## Troubleshooting
- Check the application and Sidekiq logs on `gitlab.com` and CustomersDot to monitor leads.
- Check the `leads` table in CustomersDot.
-- Set up staging credentials for Platypus, and track the leads on the Platypus Dashboard: `https://staging.ci.nexus.gitlabenvironment.cloud/admin/queues/queue/new-lead-queue`.
- Ask for access to the Marketo Sandbox and validate the leads there, [to this example request](https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/13162).
## Embed a hand-raise lead form
@@ -104,7 +102,7 @@ We currently use the following `glm_content` values:
| `discover-project-security` | This value is used in the project security feature discovery page. |
| `discover-project-security-pqltest` | This value is used in the project security feature discovery page [experiment with 3 CTAs](https://gitlab.com/gitlab-org/gitlab/-/issues/349799). |
| `group-billing` | This value is used in the group billing page. |
-| `trial-status-show-group` | This value is used in the top left nav when a namespace has an active trial. |
+| `trial-status-show-group` | This value is used in the upper-left nav when a namespace has an active trial. |
### Test the component
@@ -121,8 +119,7 @@ The flow of a PQL lead is as follows:
1. A user triggers a [`HandRaiseLeadButton` component](#embed-a-hand-raise-lead-form) on `gitlab.com`.
1. The `HandRaiseLeadButton` submits any information to the following API endpoint: `/-/trials/create_hand_raise_lead`.
1. That endpoint reposts the form to the CustomersDot `trials/create_hand_raise_lead` endpoint.
-1. CustomersDot records the form data to the `leads` table and posts the form to [Platypus](https://gitlab.com/gitlab-com/business-technology/enterprise-apps/integrations/platypus).
-1. Platypus posts the form to Workato (which is under the responsibility of the Business Operations team).
+1. CustomersDot records the form data to the `leads` table and posts the form to [Workato](https://about.gitlab.com/handbook/marketing/marketing-operations/workato/).
1. Workato sends the form to Marketo.
1. Marketo does scoring and sends the form to Salesforce.
1. Our Sales team uses Salesforce to connect to the leads.
@@ -150,11 +147,11 @@ sequenceDiagram
sequenceDiagram
CustomersDot|TrialsController#create->>HostedPlans|CreateTrialService#execute: Save [lead] to leads table for monitoring purposes
HostedPlans|CreateTrialService#execute->>BaseTrialService#create_account: Creates a customer record in customers table
- HostedPlans|CreateTrialService#create_platypus_lead->>PlatypusLogLeadService: Creates a platypus lead
- HostedPlans|CreateTrialService#create_platypus_lead->>Platypus|CreateLeadWorker: Async worker to submit [lead] to Platypus
- Platypus|CreateLeadWorker->>Platypus|CreateLeadService: [lead]
- Platypus|CreateLeadService->>PlatypusApp#post: [lead]
- PlatypusApp#post->>Platypus: [lead] is sent to Platypus
+ HostedPlans|CreateTrialService#create_lead->>CreateLeadService: Creates a lead record in customers table
+ HostedPlans|CreateTrialService#create_lead->>Workato|CreateLeadWorker: Async worker to submit [lead] to Workato
+ Workato|CreateLeadWorker->>Workato|CreateLeadService: [lead]
+ Workato|CreateLeadService->>WorkatoApp#create_lead: [lead]
+ WorkatoApp#create_lead->>Workato: [lead] is sent to Workato
```
#### Applying the trial to a namespace on CustomersDot
@@ -182,18 +179,17 @@ sequenceDiagram
```mermaid
sequenceDiagram
- CustomersDot|TrialsController#create_hand_raise_lead->>PlatypusLogLeadService: Save [lead] to leads table for monitoring purposes
- CustomersDot|TrialsController#create_hand_raise_lead->>Platypus|CreateLeadWorker: Async worker to submit [lead] to Platypus
- Platypus|CreateLeadWorker->>Platypus|CreateLeadService: [lead]
- Platypus|CreateLeadService->>PlatypusApp#post: [lead]
- PlatypusApp#post->>Platypus: [lead] is sent to Platypus
+ CustomersDot|TrialsController#create_hand_raise_lead->>CreateLeadService: Save [lead] to leads table for monitoring purposes
+ CustomersDot|TrialsController#create_hand_raise_lead->>Workato|CreateLeadWorker: Async worker to submit [lead] to Workato
+ Workato|CreateLeadWorker->>Workato|CreateLeadService: [lead]
+ Workato|CreateLeadService->>WorkatoApp#create_lead: [lead]
+ WorkatoApp#create_lead->>Workato: [lead] is sent to Workato
```
-### PQL flow after Platypus for all lead types
+### PQL flow after Workato for all lead types
```mermaid
sequenceDiagram
- Platypus->>Workato: [lead]
Workato->>Marketo: [lead]
Marketo->>Salesforce(SFDC): [lead]
```
diff --git a/doc/development/project_templates.md b/doc/development/project_templates.md
index 55a63d41425..3320f3134fb 100644
--- a/doc/development/project_templates.md
+++ b/doc/development/project_templates.md
@@ -1,6 +1,6 @@
---
-stage: Manage
-group: Organization
+stage: Create
+group: Source Code
info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments"
---
diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md
index caea2cecf57..82e96befd11 100644
--- a/doc/development/rake_tasks.md
+++ b/doc/development/rake_tasks.md
@@ -10,7 +10,7 @@ Rake tasks are available for developers and others contributing to GitLab.
## Set up database with developer seeds
-Note that if your database user does not have advanced privileges, you must create the database manually before running this command.
+If your database user does not have advanced privileges, you must create the database manually before running this command.
```shell
bundle exec rake setup
@@ -154,7 +154,7 @@ seeds, you can set the `FORCE` environment variable to `yes`:
FORCE=yes bundle exec rake setup
```
-This will skip the action confirmation/safety check, saving you from answering
+This skips the action confirmation/safety check, saving you from answering
`yes` manually.
### Discard `stdout`
@@ -168,7 +168,7 @@ it to a file. If we don't care about the output, we could just redirect it to
echo 'yes' | bundle exec rake setup > /dev/null
```
-Note that since you can't see the questions from `stdout`, you might just want
+Because you can't see the questions from `stdout`, you might just want
to `echo 'yes'` to keep it running. It would still print the errors on `stderr`
so no worries about missing errors.
@@ -182,7 +182,7 @@ There are a few environment flags you can pass to change how projects are seeded
## Run tests
-In order to run the test you can use the following commands:
+To run the test you can use the following commands:
- `bin/rake spec` to run the RSpec suite
- `bin/rake spec:unit` to run only the unit tests
diff --git a/doc/development/redis.md b/doc/development/redis.md
index 75c7df0737b..68cab9ac38d 100644
--- a/doc/development/redis.md
+++ b/doc/development/redis.md
@@ -142,6 +142,59 @@ those that use the most memory.
Currently this is not run automatically for the GitLab.com Redis instances, but
is run manually on an as-needed basis.
+## N+1 calls problem
+
+> Introduced in [`spec/support/helpers/redis_commands/recorder.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/helpers/redis_commands/recorder.rb) via [`f696f670`](https://gitlab.com/gitlab-org/gitlab/-/commit/f696f670005435472354a3dc0c01aa271aef9e32)
+
+`RedisCommands::Recorder` is a tool for detecting Redis N+1 calls problem from tests.
+
+Redis is often used for caching purposes. Usually, cache calls are lightweight and
+cannot generate enough load to affect the Redis instance. However, it is still
+possible to trigger expensive cache recalculations without knowing that. Use this
+tool to analyze Redis calls, and define expected limits for them.
+
+### Create a test
+
+It is implemented as a [`ActiveSupport::Notifications`](https://api.rubyonrails.org/classes/ActiveSupport/Notifications.html) instrumenter.
+
+You can create a test that verifies that a testable code only makes
+a single Redis call:
+
+```ruby
+it 'avoids N+1 Redis calls' do
+ control = RedisCommands::Recorder.new { visit_page }
+
+ expect(control.count).to eq(1)
+end
+```
+
+or a test that verifies the number of specific Redis calls:
+
+```ruby
+it 'avoids N+1 sadd Redis calls' do
+ control = RedisCommands::Recorder.new { visit_page }
+
+ expect(control.by_command(:sadd).count).to eq(1)
+end
+```
+
+You can also provide a pattern to capture only specific Redis calls:
+
+```ruby
+it 'avoids N+1 Redis calls to forks_count key' do
+ control = RedisCommands::Recorder.new(pattern: 'forks_count') { visit_page }
+
+ expect(control.count).to eq(1)
+end
+```
+
+These tests can help to identify N+1 problems related to Redis calls,
+and make sure that the fix for them works as expected.
+
+### See also
+
+- [Database query recorder](database/query_recorder.md)
+
## Utility classes
We have some extra classes to help with specific use cases. These are
@@ -189,9 +242,8 @@ The Redis [`PFCOUNT`](https://redis.io/commands/pfcount/),
[`PFADD`](https://redis.io/commands/pfadd/), and
[`PFMERGE`](https://redis.io/commands/pfmerge/) commands operate on
HyperLogLogs, a data structure that allows estimating the number of unique
-elements with low memory usage. (In addition to the `PFCOUNT` documentation,
-Thoughtbot's article on [HyperLogLogs in Redis](https://thoughtbot.com/blog/hyperloglogs-in-redis)
-provides a good background here.)
+elements with low memory usage. For more information,
+see [HyperLogLogs in Redis](https://thoughtbot.com/blog/hyperloglogs-in-redis).
[`Gitlab::Redis::HLL`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/redis/hll.rb)
provides a convenient interface for adding and counting values in HyperLogLogs.
diff --git a/doc/development/ruby3_gotchas.md b/doc/development/ruby3_gotchas.md
index 4768d6e1182..0d000bc68df 100644
--- a/doc/development/ruby3_gotchas.md
+++ b/doc/development/ruby3_gotchas.md
@@ -57,7 +57,7 @@ To write code that works under both 2.7 and 3.0, consider the following options:
We recommend always passing the block explicitly, and prefer two required arguments as block parameters.
-To learn more, see [Ruby issue 12706](https://bugs.ruby-lang.org/issues/12706).
+For more information, see [Ruby issue 12706](https://bugs.ruby-lang.org/issues/12706).
## `Symbol#to_proc` returns signature metadata consistent with lambdas
@@ -92,7 +92,7 @@ called without a receiver.
Ruby 3 corrects this: the code that tests `Proc` object arity or parameter lists might now break and
has to be updated.
-To learn more, see [Ruby issue 16260](https://bugs.ruby-lang.org/issues/16260).
+For more information, see [Ruby issue 16260](https://bugs.ruby-lang.org/issues/16260).
## `OpenStruct` does not evaluate fields lazily
diff --git a/doc/development/sec/index.md b/doc/development/sec/index.md
index 4ed0eadd92f..5ac5118aae8 100644
--- a/doc/development/sec/index.md
+++ b/doc/development/sec/index.md
@@ -85,7 +85,7 @@ a critical component to both describing and tracking vulnerabilities.
In most other cases, the `identifiers` collection is unordered, where the remaining secondary identifiers act as metadata for grouping vulnerabilities
(see [Analyzer vulnerability translation](#analyzer-vulnerability-translation) below for the exception).
-Any time the primary identifier changes and a project pipeline is re-run, ingestion of the new report will “orphan” the previous DB record.
+Any time the primary identifier changes and a project pipeline is re-run, ingestion of the new report will "orphan" the previous DB record.
Because our processing logic relies on generating a delta of two different vulnerabilities, it can end up looking rather confusing. For example:
[!Screenshot of primary identifier mismatch in MR widget](img/primary_identifier_changed_v15_6.png)
@@ -95,14 +95,14 @@ After being [merged](../integrations/secure.md#tracking-and-merging-vulnerabilit
### Guiding principles for ensuring primary identifier stability
- A primary identifier should never change unless we have a compelling reason.
-- Analyzer supporting vulnerability translation must include the legacy primary identifiers in a secondary position to prevent “orphaning” of results.
+- Analyzer supporting vulnerability translation must include the legacy primary identifiers in a secondary position to prevent "orphaning" of results.
- Beyond the primary identifier, the order of secondary identifiers does not matter.
- The identifier is unique based on a combination of the `Type` and `Value` fields (see [identifier fingerprint](https://gitlab.com/gitlab-org/gitlab/-/blob/v15.5.1-ee/lib/gitlab/ci/reports/security/identifier.rb#L63)).
- If we change the primary identifier, rolling back analyzers to previous versions will not fix the orphaned results. The data previously ingested into our database is an artifact of previous jobs with few ways of automating data migrations.
### Analyzer vulnerability translation
-In the case of the SAST Semgrep analyzer, there is a secondary identifier of particular importance: the identifier linking the report’s vulnerability
+In the case of the SAST Semgrep analyzer, there is a secondary identifier of particular importance: the identifier linking the report's vulnerability
to the legacy analyzer (that is, bandit or ESLint).
To [enable vulnerability translation](../../user/application_security/sast/analyzers.md#vulnerability-translation)
diff --git a/doc/development/sec/security_report_ingestion_overview.md b/doc/development/sec/security_report_ingestion_overview.md
index c1d977c2a17..688986e0eb1 100644
--- a/doc/development/sec/security_report_ingestion_overview.md
+++ b/doc/development/sec/security_report_ingestion_overview.md
@@ -56,7 +56,7 @@ If there is only a Security Finding, a Vulnerability Finding and a Vulnerability
### Issue creation
-If you select `Create issue`, a Vulnerabilities::Feedback record is created as well. The Feedback has a different `feedback_type` and an `issue_id` that’s not `NULL`.
+If you select `Create issue`, a Vulnerabilities::Feedback record is created as well. The Feedback has a different `feedback_type` and an `issue_id` that's not `NULL`.
NOTE:
Vulnerabilities::Feedback are in the process of being [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/5629). This will later create a `Vulnerabilities::IssueLink` record.
diff --git a/doc/development/sec/token_revocation_api.md b/doc/development/sec/token_revocation_api.md
new file mode 100644
index 00000000000..15d1d2d0ef3
--- /dev/null
+++ b/doc/development/sec/token_revocation_api.md
@@ -0,0 +1,118 @@
+---
+stage: Secure
+group: Static Analysis
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Token Revocation API
+
+The Token Revocation API is an externally-deployed HTTP API that interfaces with GitLab
+to receive and revoke API tokens and other secrets detected by GitLab Secret Detection.
+See the [high-level architecture](../../user/application_security/secret_detection/post_processing.md)
+to understand the Secret Detection post-processing and revocation flow.
+
+GitLab.com uses the internally-maintained [Secret Revocation Service](https://gitlab.com/gitlab-com/gl-security/engineering-and-research/automation-team/secret-revocation-service)
+(team-members only) as its Token Revocation API. For GitLab self-managed, you can create
+your own API and configure GitLab to use it.
+
+## Implement a Token Revocation API for self-managed
+
+GitLab self-managed instances interested in using the revocation capabilities must:
+
+- Implement and deploy your own Token Revocation API.
+- Configure the GitLab instance to use the Token Revocation API.
+
+Your service must:
+
+- Match the API specification below.
+- Provide two endpoints:
+ - Fetching revocable token types.
+ - Revoking leaked tokens.
+- Be rate-limited and idempotent.
+
+Requests to the documented endpoints are authenticated using API tokens passed in
+the `Authorization` header. Request and response bodies, if present, are
+expected to have the content type `application/json`.
+
+All endpoints may return these responses:
+
+- `401 Unauthorized`
+- `405 Method Not Allowed`
+- `500 Internal Server Error`
+
+### `GET /v1/revocable_token_types`
+
+Returns the valid `type` values for use in the `revoke_tokens` endpoint.
+
+NOTE:
+These values match the concatenation of [the `secrets` analyzer's](../../user/application_security/secret_detection/index.md)
+[primary identifier](../integrations/secure.md#identifiers) by means
+of concatenating the `primary_identifier.type` and `primary_identifier.value`.
+For example, the value `gitleaks_rule_id_gitlab_personal_access_token` matches the following finding identifier:
+
+```json
+{"type": "gitleaks_rule_id", "name": "Gitleaks rule ID GitLab Personal Access Token", "value": "GitLab Personal Access Token"}
+```
+
+| Status Code | Description |
+| ----- | --- |
+| `200` | The response body contains the valid token `type` values. |
+
+Example response body:
+
+```json
+{
+ "types": ["gitleaks_rule_id_gitlab_personal_access_token"]
+}
+```
+
+### `POST /v1/revoke_tokens`
+
+Accepts a list of tokens to be revoked by the appropriate provider. Your service is responsible for communicating
+with each provider to revoke the token.
+
+| Status Code | Description |
+| ----- | --- |
+| `204` | All submitted tokens have been accepted for eventual revocation. |
+| `400` | The request body is invalid or one of the submitted token types is not supported. The request should not be retried. |
+| `429` | The provider has received too many requests. The request should be retried later. |
+
+Example request body:
+
+```json
+[{
+ "type": "gitleaks_rule_id_gitlab_personal_access_token",
+ "token": "glpat--8GMtG8Mf4EnMJzmAWDU",
+ "location": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile1.java"
+},
+{
+ "type": "gitleaks_rule_id_gitlab_personal_access_token",
+ "token": "glpat--tG84EGK33nMLLDE70zU",
+ "location": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile2.java"
+}]
+```
+
+### Configure GitLab to interface with the Token Revocation API
+
+You must configure the following database settings in the GitLab instance:
+
+| Setting | Type | Description |
+| ------- | ---- | ----------- |
+| `secret_detection_token_revocation_enabled` | Boolean | Whether automatic token revocation is enabled |
+| `secret_detection_token_revocation_url` | String | A fully-qualified URL to the `/v1/revoke_tokens` endpoint of the Token Revocation API |
+| `secret_detection_revocation_token_types_url` | String | A fully-qualified URL to the `/v1/revocable_token_types` endpoint of the Token Revocation API |
+| `secret_detection_token_revocation_token` | String | A pre-shared token to authenticate requests to the Token Revocation API |
+
+For example, to configure these values in the
+[Rails console](../../administration/operations/rails_console.md#starting-a-rails-console-session):
+
+```ruby
+::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_token: 'MYSECRETTOKEN')
+::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_url: 'https://example.gitlab.com/revocation_service/v1/revoke_tokens')
+::Gitlab::CurrentSettings.update!(secret_detection_revocation_token_types_url: 'https://example.gitlab.com/revocation_service/v1/revocable_token_types')
+::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_enabled: true)
+```
+
+After you configure these values, the Token Revocation API will be called according to the
+[high-level architecture](../../user/application_security/secret_detection/post_processing.md#high-level-architecture)
+diagram.
diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md
index 3791cd4861e..6c64e3b2acc 100644
--- a/doc/development/secure_coding_guidelines.md
+++ b/doc/development/secure_coding_guidelines.md
@@ -266,8 +266,7 @@ value, we **will not** be protected against DNS rebinding.
This is the case with validators such as the `AddressableUrlValidator` (called with `validates :url, addressable_url: {opts}` or `public_url: {opts}`).
Validation errors are only raised when validations are called, for example when a record is created or saved. If we ignore the value returned by the validation
-when persisting the record, **we need to recheck** its validity before using it. You can learn more about [Time of Check to Time of Use bugs](#time-of-check-to-time-of-use-bugs) in a later section
-of these guidelines.
+when persisting the record, **we need to recheck** its validity before using it. For more information, see [Time of check to time of use bugs](#time-of-check-to-time-of-use-bugs).
#### Feature-specific mitigations
@@ -291,7 +290,7 @@ implement.
- For HTTP connections: Disable redirects or validate the redirect destination
- To mitigate DNS rebinding attacks, validate and use the first IP address received.
-See [`url_blocker_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/url_blocker_spec.rb) for examples of SSRF payloads. See [time of check to time of use bugs](#time-of-check-to-time-of-use-bugs) to learn more about DNS rebinding's class of bug.
+See [`url_blocker_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/url_blocker_spec.rb) for examples of SSRF payloads. For more information about the DNS-rebinding class of bugs, see [Time of check to time of use bugs](#time-of-check-to-time-of-use-bugs).
Don't rely on methods like `.start_with?` when validating a URL, or make assumptions about which
part of a string maps to which part of a URL. Use the `URI` class to parse the string, and validate
@@ -1270,7 +1269,7 @@ This sensitive data must be handled carefully to avoid leaks which could lead to
- The [Gitleaks Git hook](https://gitlab.com/gitlab-com/gl-security/security-research/gitleaks-endpoint-installer) is recommended for preventing credentials from being committed.
- Never log credentials under any circumstance. Issue [#353857](https://gitlab.com/gitlab-org/gitlab/-/issues/353857) is an example of credential leaks through log file.
- When credentials are required in a CI/CD job, use [masked variables](../ci/variables/index.md#mask-a-cicd-variable) to help prevent accidental exposure in the job logs. Be aware that when [debug logging](../ci/variables/index.md#enable-debug-logging) is enabled, all masked CI/CD variables are visible in job logs. Also consider using [protected variables](../ci/variables/index.md#protect-a-cicd-variable) when possible so that sensitive CI/CD variables are only available to pipelines running on protected branches or protected tags.
-- Proper scanners must be enabled depending on what data those credentials are protecting. See the [Application Security Inventory Policy](https://about.gitlab.com/handbook/security/security-engineering-and-research/application-security/inventory.html#policies) and our [Data Classification Standards](https://about.gitlab.com/handbook/security/data-classification-standard.html#data-classification-standards).
+- Proper scanners must be enabled depending on what data those credentials are protecting. See the [Application Security Inventory Policy](https://about.gitlab.com/handbook/security/security-engineering/application-security/inventory.html#policies) and our [Data Classification Standards](https://about.gitlab.com/handbook/security/data-classification-standard.html#data-classification-standards).
- To store and/or share credentials between teams, refer to [1Password for Teams](https://about.gitlab.com/handbook/security/#1password-for-teams) and follow [the 1Password Guidelines](https://about.gitlab.com/handbook/security/#1password-guidelines).
- If you need to share a secret with a team member, use 1Password. Do not share a secret over email, Slack, or other service on the Internet.
diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md
index 1e5f4191375..ef2e7e6edf5 100644
--- a/doc/development/service_ping/implement.md
+++ b/doc/development/service_ping/implement.md
@@ -356,8 +356,6 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd/) and [P
`{users}_creating_epics-2020-34`. If `redis_slot` is not defined the Redis key will
be `{users_creating_epics}-2020-34`.
Recommended slots to use are: `users`, `projects`. This is the value we count.
- - `expiry`: expiry time in days. Default: 29 days for daily aggregation and 6 weeks for weekly
- aggregation.
- `aggregation`: may be set to a `:daily` or `:weekly` key. Defines how counting data is stored in Redis.
Aggregation on a `daily` basis does not pull more fine grained data.
- `feature_flag`: if no feature flag is set then the tracking is enabled. One feature flag can be used for multiple events. For details, see our [GitLab internal Feature flags](../feature_flags/index.md) documentation. The feature flags are owned by the group adding the event tracking.
@@ -498,9 +496,6 @@ Next, get the unique events for the current week.
We have the following recommendations for [adding new events](#add-new-events):
- Event aggregation: weekly.
-- Key expiry time:
- - Daily: 29 days.
- - Weekly: 42 days.
- When adding new metrics, use a [feature flag](../../operations/feature_flags.md) to control the impact.
- For feature flags triggered by another service, set `default_enabled: false`,
- Events can be triggered using the `UsageData` API, which helps when there are > 10 events per change
diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md
index 40bb03cb5b4..14c9cb33446 100644
--- a/doc/development/service_ping/index.md
+++ b/doc/development/service_ping/index.md
@@ -210,7 +210,6 @@ The following is example content of the Service Ping payload.
"prometheus_metrics_enabled": false,
"reply_by_email_enabled": "incoming+%{key}@incoming.gitlab.com",
"signup_enabled": true,
- "web_ide_clientside_preview_enabled": true,
"projects_with_expiration_policy_disabled": 999,
"projects_with_expiration_policy_enabled": 999,
...
diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md
index 92c5d2d317d..8a8ceae1f4c 100644
--- a/doc/development/service_ping/metrics_lifecycle.md
+++ b/doc/development/service_ping/metrics_lifecycle.md
@@ -72,14 +72,16 @@ WARNING:
If a metric is not used in Sisense or any other system after 6 months, the
Product Intelligence team marks it as inactive and assigns it to the group owner for review.
-We are working on automating this process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338466) for details.
+We are working on automating this process. See [this epic](https://gitlab.com/groups/gitlab-org/-/epics/8988) for details.
Product Intelligence removes metrics from Service Ping if they are not used in any Sisense dashboard.
-For an example of the metric removal process, see this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297029).
+For an example of the metric removal process, see this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/388236).
To remove a metric:
+1. Create an issue for removing the metric if none exists yet. The issue needs to outline why the metric should be deleted. You can use this issue to document the removal process.
+
1. Check the following YAML files and verify the metric is not used in an aggregate:
- [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/)
- [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/)
@@ -111,7 +113,10 @@ To remove a metric:
[GitLab Data Team project](https://gitlab.com/gitlab-data/analytics/-/issues).
Ask for confirmation that the metric is not referred to in any SiSense dashboards and
can be safely removed from Service Ping. Use this
- [example issue](https://gitlab.com/gitlab-data/analytics/-/issues/7539) for guidance.
+ [example issue](https://gitlab.com/gitlab-data/analytics/-/issues/15266) for guidance.
+
+1. Notify the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) by `@` mentioning those groups in a comment in the issue regarding the deletion of the metric.
+ Many Service Ping metrics are relied upon for health score and XMAU reporting and unexpected changes to those metrics could break reporting.
1. After you verify the metric can be safely removed,
update the attributes of the metric's YAML definition:
@@ -139,6 +144,3 @@ To remove a metric:
1. Remove any other records related to the metric:
- The feature flag YAML file at [`config/feature_flags/*/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/feature_flags).
- The entry in the known events YAML file at [`lib/gitlab/usage_data_counters/known_events/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/usage_data_counters/known_events).
-
-1. Notify the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) by `@` mentioning those groups in a comment on the MR.
- Many Service Ping metrics are relied upon for health score and XMAU reporting and unexpected changes to those metrics could break reporting.
diff --git a/doc/development/service_ping/review_guidelines.md b/doc/development/service_ping/review_guidelines.md
index 70f7f3dca54..9813c9e0b12 100644
--- a/doc/development/service_ping/review_guidelines.md
+++ b/doc/development/service_ping/review_guidelines.md
@@ -68,7 +68,7 @@ are regular backend changes.
Read the [stages file](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml).
- Check the file location. Consider the time frame, and if the file should be under `ee`.
- Check the tiers.
-- If a metric was changed or removed: Make sure the MR author notified the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) by `@` mentioning those groups in a comment on the MR.
+- If a metric was changed or removed: Make sure the MR author notified the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) by `@` mentioning those groups in a comment on the issue for the MR and all of these groups have acknowledged the removal.
- Metrics instrumentations
- Recommend using metrics instrumentation for new metrics, [if possible](metrics_instrumentation.md#support-for-instrumentation-classes).
- Approve the MR, and relabel the MR with `~"product intelligence::approved"`.
diff --git a/doc/development/sidekiq/index.md b/doc/development/sidekiq/index.md
index f4f98641d39..355f5a3b753 100644
--- a/doc/development/sidekiq/index.md
+++ b/doc/development/sidekiq/index.md
@@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Sidekiq guides
We use [Sidekiq](https://github.com/mperham/sidekiq) as our background
-job processor. These guides are for writing jobs that will work well on
+job processor. These guides are for writing jobs that works well on
GitLab.com and be consistent with our existing worker classes. For
information on administering GitLab, see [configuring Sidekiq](../../administration/sidekiq/index.md).
@@ -74,7 +74,7 @@ A lower retry count may be applicable if any of the below apply:
1. The worker is not idempotent and running it multiple times could
leave the system in an inconsistent state. For example, a worker that
posts a system note and then performs an action: if the second step
- fails and the worker retries, the system note will be posted again.
+ fails and the worker retries, the system note is posted again.
1. The worker is a cronjob that runs frequently. For example, if a cron
job runs every hour, then we don't need to retry beyond an hour
because we don't need two of the same job running at once.
@@ -96,6 +96,24 @@ def perform
end
```
+## Failure handling
+
+Failures are typically handled by Sidekiq itself, which takes advantage of the inbuilt retry mechanism mentioned above. You should allow exceptions to be raised so that Sidekiq can reschedule the job.
+
+If you need to perform an action when a job fails after all of its retry attempts, add it to the `sidekiq_retries_exhausted` method.
+
+```ruby
+sidekiq_retries_exhausted do |msg, ex|
+ project = Project.find(msg['args'].first)
+ project.perform_a_rollback # handle the permanent failure
+end
+
+def perform(project_id)
+ project = Project.find(project_id)
+ project.some_action # throws an exception
+end
+```
+
## Sidekiq Queues
Previously, each worker had its own queue, which was automatically set based on the
@@ -113,8 +131,8 @@ gitlab:sidekiq:all_queues_yml:generate` to regenerate
`app/workers/all_queues.yml` or `ee/app/workers/all_queues.yml` so that
it can be picked up by
[`sidekiq-cluster`](../../administration/sidekiq/extra_sidekiq_processes.md)
-in installations that don't use routing rules. To learn more about potential changes,
-read [Use routing rules by default and deprecate queue selectors for self-managed](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/596).
+in installations that don't use routing rules. For more information about potential changes,
+see [epic 596](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/596).
Additionally, run
`bin/rake gitlab:sidekiq:sidekiq_queues_yml:generate` to regenerate
@@ -156,7 +174,7 @@ queues in a namespace (technically: all queues prefixed with the namespace name)
when a namespace is provided instead of a simple queue name in the `--queue`
(`-q`) option, or in the `:queues:` section in `config/sidekiq_queues.yml`.
-Note that adding a worker to an existing namespace should be done with care, as
+Adding a worker to an existing namespace should be done with care, as
the extra jobs take resources away from jobs from workers that were already
there, if the resources available to the Sidekiq process handling the namespace
are not adjusted appropriately.
@@ -195,9 +213,9 @@ can read the number or type of provided arguments.
GitLab stores Sidekiq jobs and their arguments in Redis. To avoid
excessive memory usage, we compress the arguments of Sidekiq jobs
-if their original size is bigger than 100KB.
+if their original size is bigger than 100 KB.
-After compression, if their size still exceeds 5MB, it raises an
+After compression, if their size still exceeds 5 MB, it raises an
[`ExceedLimitError`](https://gitlab.com/gitlab-org/gitlab/-/blob/f3dd89e5e510ea04b43ffdcb58587d8f78a8d77c/lib/gitlab/sidekiq_middleware/size_limiter/exceed_limit_error.rb#L8)
error when scheduling the job.
@@ -227,6 +245,6 @@ tests should be placed in `spec/workers`.
## Interacting with Sidekiq Redis and APIs
-The application should minimise interaction with of any `Sidekiq.redis` and Sidekiq [APIs](https://github.com/mperham/sidekiq/blob/main/lib/sidekiq/api.rb). Such interactions in generic application logic should be abstracted to a [Sidekiq middleware](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/sidekiq_middleware) for re-use across teams. By decoupling application logic from Sidekiq's datastore, it allows for greater freedom when horizontally scaling the GitLab background processing setup.
+The application should minimise interaction with of any `Sidekiq.redis` and Sidekiq [APIs](https://github.com/mperham/sidekiq/blob/main/lib/sidekiq/api.rb). Such interactions in generic application logic should be abstracted to a [Sidekiq middleware](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/sidekiq_middleware) for re-use across teams. By decoupling application logic from Sidekiq datastore, it allows for greater freedom when horizontally scaling the GitLab background processing setup.
Some exceptions to this rule would be migration-related logic or administration operations.
diff --git a/doc/development/sidekiq/worker_attributes.md b/doc/development/sidekiq/worker_attributes.md
index 4fcd8e33d5c..a3bfe5f27cc 100644
--- a/doc/development/sidekiq/worker_attributes.md
+++ b/doc/development/sidekiq/worker_attributes.md
@@ -37,7 +37,7 @@ end
### Latency sensitive jobs
If a large number of background jobs get scheduled at once, queueing of jobs may
-occur while jobs wait for a worker node to be become available. This is normal
+occur while jobs wait for a worker node to be become available. This is standard
and gives the system resilience by allowing it to gracefully handle spikes in
traffic. Some jobs, however, are more sensitive to latency than others.
@@ -79,7 +79,7 @@ On GitLab.com, we run Sidekiq in several
each of which represents a particular type of workload.
When changing a queue's urgency, or adding a new queue, we need to take
-into account the expected workload on the new shard. Note that, if we're
+into account the expected workload on the new shard. If we're
changing an existing queue, there is also an effect on the old shard,
but that always reduces work.
@@ -108,7 +108,7 @@ shard_consumption = shard_rps * shard_duration_avg
If we expect an increase of **less than 5%**, then no further action is needed.
-Otherwise, please ping `@gitlab-org/scalability` on the merge request and ask
+Otherwise, ping `@gitlab-org/scalability` on the merge request and ask
for a review.
## Jobs with External Dependencies
@@ -121,7 +121,7 @@ However, some jobs are dependent on external services to complete
successfully. Some examples include:
1. Jobs which call web-hooks configured by a user.
-1. Jobs which deploy an application to a k8s cluster configured by a user.
+1. Jobs which deploy an application to a Kubernetes cluster configured by a user.
These jobs have "external dependencies". This is important for the operation of
the background processing cluster in several ways:
@@ -179,8 +179,8 @@ performance.
Likewise, if a worker uses large amounts of memory, we can run these on a
bespoke low concurrency, high memory fleet.
-Note that memory-bound workers create heavy GC workloads, with pauses of
-10-50ms. This has an impact on the latency requirements for the
+Memory-bound workers create heavy GC workloads, with pauses of
+10-50 ms. This has an impact on the latency requirements for the
worker. For this reason, `memory` bound, `urgency :high` jobs are not
permitted and fail CI. In general, `memory` bound workers are
discouraged, and alternative approaches to processing the work should be
@@ -219,7 +219,7 @@ We use the following approach to determine whether a worker is CPU-bound:
- Divide `cpu_s` by `duration` to get the percentage time spend on-CPU.
- If this ratio exceeds 33%, the worker is considered CPU-bound and should be
annotated as such.
-- Note that these values should not be used over small sample sizes, but
+- These values should not be used over small sample sizes, but
rather over fairly large aggregates.
## Feature category
@@ -254,7 +254,7 @@ When setting this field, consider the following trade-off:
- Prefer read replicas to add relief to the primary, but increase the likelihood of stale reads that have to be retried.
To maintain the same behavior compared to before this field was introduced, set it to `:always`, so
-database operations will only target the primary. Reasons for having to do so include workers
+database operations only target the primary. Reasons for having to do so include workers
that mostly or exclusively perform writes, or workers that read their own writes and who might run
into data consistency issues should a stale record be read back from a replica. **Try to avoid
these scenarios, since `:always` should be considered the exception, not the rule.**
@@ -270,10 +270,10 @@ The difference is in what happens when there is still replication lag after the
switch over to the primary right away, whereas `delayed` workers fail fast and are retried once.
If they still encounter replication lag, they also switch to the primary instead.
**If your worker never performs any writes, it is strongly advised to apply one of these consistency settings,
-since it will never need to rely on the primary database node.**
+since it never needs to rely on the primary database node.**
The table below shows the `data_consistency` attribute and its values, ordered by the degree to which
-they prefer read replicas and will wait for replicas to catch up:
+they prefer read replicas and wait for replicas to catch up:
| **Data Consistency** | **Description** |
|--------------|-----------------------------|
@@ -300,14 +300,14 @@ end
The `feature_flag` property allows you to toggle a job's `data_consistency`,
which permits you to safely toggle load balancing capabilities for a specific job.
-When `feature_flag` is disabled, the job defaults to `:always`, which means that the job will always use the primary database.
+When `feature_flag` is disabled, the job defaults to `:always`, which means that the job always uses the primary database.
The `feature_flag` property does not allow the use of
[feature gates based on actors](../feature_flags/index.md).
This means that the feature flag cannot be toggled only for particular
projects, groups, or users, but instead, you can safely use [percentage of time rollout](../feature_flags/index.md).
-Note that since we check the feature flag on both Sidekiq client and server, rolling out a 10% of the time,
-will likely results in 1% (`0.1` `[from client]*0.1` `[from server]`) of effective jobs using replicas.
+Since we check the feature flag on both Sidekiq client and server, rolling out a 10% of the time,
+likely results in 1% (`0.1` `[from client]*0.1` `[from server]`) of effective jobs using replicas.
Example:
diff --git a/doc/development/spam_protection_and_captcha/exploratory_testing.md b/doc/development/spam_protection_and_captcha/exploratory_testing.md
index fe5c1c3db56..b2d780b1563 100644
--- a/doc/development/spam_protection_and_captcha/exploratory_testing.md
+++ b/doc/development/spam_protection_and_captcha/exploratory_testing.md
@@ -353,8 +353,8 @@ GraphQL response:
}
```
-### Scenario: `allow_possible_spam` feature flag enabled
+### Scenario: `allow_possible_spam` application setting enabled
-With the `allow_possible_spam` feature flag enabled, the API returns a 200 response. Any
+With the `allow_possible_spam` application setting enabled, the API returns a 200 response. Any
valid request is successful and no CAPTCHA is presented, even if the request is considered
spam.
diff --git a/doc/development/spam_protection_and_captcha/index.md b/doc/development/spam_protection_and_captcha/index.md
index 254c3401f81..8d73c12e30a 100644
--- a/doc/development/spam_protection_and_captcha/index.md
+++ b/doc/development/spam_protection_and_captcha/index.md
@@ -18,7 +18,7 @@ To add this support, you must implement the following areas as applicable:
for a feature which does not yet have support.
1. [REST API](rest_api.md): The changes needed to add
spam or CAPTCHA support to Grape REST API endpoints. Refer to the related
- [REST API documentation](../../api/index.md#resolve-requests-detected-as-spam).
+ [REST API documentation](../../api/rest/index.md#resolve-requests-detected-as-spam).
1. [GraphQL API](graphql_api.md): The changes needed to add spam or CAPTCHA support to GraphQL
mutations. Refer to the related
[GraphQL API documentation](../../api/graphql/index.md#resolve-mutations-detected-as-spam).
@@ -48,6 +48,6 @@ The possible values include:
## Related topics
- [Spam and CAPTCHA support in the GraphQL API](../../api/graphql/index.md#resolve-mutations-detected-as-spam)
-- [Spam and CAPTCHA support in the REST API](../../api/index.md#resolve-requests-detected-as-spam)
+- [Spam and CAPTCHA support in the REST API](../../api/rest/index.md#resolve-requests-detected-as-spam)
- [reCAPTCHA Spam and Anti-bot Protection](../../integration/recaptcha.md)
- [Akismet and Spam Logs](../../integration/akismet.md)
diff --git a/doc/development/spam_protection_and_captcha/rest_api.md b/doc/development/spam_protection_and_captcha/rest_api.md
index 7d749944163..d7012ffb418 100644
--- a/doc/development/spam_protection_and_captcha/rest_api.md
+++ b/doc/development/spam_protection_and_captcha/rest_api.md
@@ -32,7 +32,7 @@ The main steps are:
- Raise a Grape `#error!` exception with a descriptive spam-specific error message.
- Include the relevant information added as error fields to the response.
For more details on these fields, refer to the section in the REST API documentation on
- [Resolve requests detected as spam](../../api/index.md#resolve-requests-detected-as-spam).
+ [Resolve requests detected as spam](../../api/rest/index.md#resolve-requests-detected-as-spam).
NOTE:
If you use the standard ApolloLink or Axios interceptor CAPTCHA support described
diff --git a/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md b/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md
index abc46c52407..7c596e544b5 100644
--- a/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md
+++ b/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md
@@ -168,7 +168,7 @@ stageGroupDashboards.dashboard('source_code')
If you want to see the workflow in action, we've recorded a pairing session on customizing a dashboard,
available on [GitLab Unfiltered](https://youtu.be/shEd_eiUjdI).
-For deeper customization and more complicated metrics, visit the
+For deeper customization and more complicated metrics, see the
[Grafonnet lib](https://github.com/grafana/grafonnet-lib) project and the
[GitLab Prometheus Metrics](../../../administration/monitoring/prometheus/gitlab_metrics.md#gitlab-prometheus-metrics)
documentation.
diff --git a/doc/development/stage_group_observability/index.md b/doc/development/stage_group_observability/index.md
index 08f751c508e..b275b0bfec2 100644
--- a/doc/development/stage_group_observability/index.md
+++ b/doc/development/stage_group_observability/index.md
@@ -35,7 +35,7 @@ stage group is comparable to the
[monthly availability](https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators/#gitlabcom-availability)
we calculate for GitLab.com, except it's scoped to the features of a group.
-To learn more about how we use error budgets, see the
+For more information about how we use error budgets, see the
[Engineering Error Budgets](https://about.gitlab.com/handbook/engineering/error-budgets/) handbook page.
By default, the first row of panels on both dashboards shows the
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md
index e27d4911158..54d3f368bbd 100644
--- a/doc/development/testing_guide/best_practices.md
+++ b/doc/development/testing_guide/best_practices.md
@@ -358,6 +358,19 @@ a place to start. The most expensive examples here are in
shared examples; any reductions generally have a larger impact as
they are called in multiple places.
+#### Top slow tests
+
+We collect information about tests duration in [`rspec_profiling_stats`](https://gitlab.com/gitlab-org/rspec_profiling_stats) project. The data is showed using GitLab Pages in this
+[UI](https://gitlab-org.gitlab.io/rspec_profiling_stats/)
+
+With [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/375983) we defined thresholds for tests duration that can act a guide.
+
+For tests that are not meeting the thresholds it is recommended to create issues and improve the tests duration.
+
+| Date | Feature tests | Controllers and Requests tests | Other |
+| --- | --- | --- | --- |
+| 2023-02-15 | 67.42 seconds | 44.66 seconds | 76.86 seconds |
+
#### Avoid repeating expensive actions
While isolated examples are very clear, and help serve the purpose of specs as
@@ -415,7 +428,7 @@ results are available, and not just the first failure.
- Don't assert against the absolute value of a sequence-generated attribute (see
[Gotchas](../gotchas.md#do-not-assert-against-the-absolute-value-of-a-sequence-generated-attribute)).
- Avoid using `expect_any_instance_of` or `allow_any_instance_of` (see
- [Gotchas](../gotchas.md#do-not-assert-against-the-absolute-value-of-a-sequence-generated-attribute)).
+ [Gotchas](../gotchas.md#avoid-using-expect_any_instance_of-or-allow_any_instance_of-in-rspec)).
- Don't supply the `:each` argument to hooks because it's the default.
- On `before` and `after` hooks, prefer it scoped to `:context` over `:all`
- When using `evaluate_script("$('.js-foo').testSomething()")` (or `execute_script`) which acts on a given element,
@@ -432,6 +445,12 @@ results are available, and not just the first failure.
You must [set feature category metadata for each RSpec example](../feature_categorization/index.md#rspec-examples).
+### Tests depending on EE license
+
+You can use `if: Gitlab.ee?` or `unless: Gitlab.ee?` on context/spec blocks to execute tests depending on whether running with `FOSS_ONLY=1`.
+
+Example: [SchemaValidator reads a different path depending on the license](https://gitlab.com/gitlab-org/gitlab/-/blob/7cdcf9819cfa02c701d6fa9f18c1e7a8972884ed/spec/lib/gitlab/ci/parsers/security/validators/schema_validator_spec.rb#L571)
+
### Coverage
[`simplecov`](https://github.com/colszowka/simplecov) is used to generate code test coverage reports.
diff --git a/doc/development/testing_guide/contract/consumer_tests.md b/doc/development/testing_guide/contract/consumer_tests.md
index 39cc34d6153..60ce71db79d 100644
--- a/doc/development/testing_guide/contract/consumer_tests.md
+++ b/doc/development/testing_guide/contract/consumer_tests.md
@@ -17,7 +17,7 @@ Then, populate it with the following function and parameters:
- [`PactOptions`](#the-pactoptions-parameter)
- [`PactFn`](#the-pactfn-parameter)
-To learn more about how the contract test directory is structured, see the contract testing [test suite folder structure](index.md#test-suite-folder-structure).
+For more information about how the contract test directory is structured, see [Test suite folder structure](index.md#test-suite-folder-structure).
### The `pactWith` function
@@ -47,7 +47,7 @@ pactWith(
);
```
-To learn more about how to name the consumers and providers, see contract testing [naming conventions](index.md#naming-conventions).
+For more information about how to name consumers and providers, see [Naming conventions](index.md#naming-conventions).
### The `PactFn` parameter
diff --git a/doc/development/testing_guide/contract/provider_tests.md b/doc/development/testing_guide/contract/provider_tests.md
index 3ce9f91a307..cb3aeae529d 100644
--- a/doc/development/testing_guide/contract/provider_tests.md
+++ b/doc/development/testing_guide/contract/provider_tests.md
@@ -12,7 +12,7 @@ This tutorial guides you through writing a provider test from scratch. It is a c
Provider tests are quite simple. The goal is to set up the test data and then link that with the corresponding contract. Start by creating a file called `get_discussions_helper.rb` under `spec/contracts/provider/pact_helpers/project/merge_request`. Note that the files are called `helpers` to match how they are called by Pact in the Rake tasks, which are set up at the end of this tutorial.
-To learn more about how the contract test directory is structured, see the contract testing [test suite folder structure](index.md#test-suite-folder-structure).
+For more information about how the contract test directory is structured, see [Test suite folder structure](index.md#test-suite-folder-structure).
### The `service_provider` block
@@ -48,7 +48,7 @@ module Provider
end
```
-To learn more about how to name the consumers and providers, see contract testing [naming conventions](index.md#naming-conventions).
+For more information about how to name consumers and providers, see [Naming conventions](index.md#naming-conventions).
## Configure the test app
diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md
index 8369dcb0740..77ceb9fa546 100644
--- a/doc/development/testing_guide/end_to_end/index.md
+++ b/doc/development/testing_guide/end_to_end/index.md
@@ -97,6 +97,9 @@ This problem was discovered in <https://gitlab.com/gitlab-org/gitlab-qa/-/issues
work-around was suggested in <https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4717>.
A feature proposal to segregate access control regarding running pipelines from ability to push/merge was also created at <https://gitlab.com/gitlab-org/gitlab/-/issues/24585>.
+For more technical details on CI/CD setup and documentation on adding new test jobs to `package-and-test` pipeline, see
+[`package_and_test` setup documentation](package_and_test_pipeline.md).
+
#### With merged results pipelines
In a merged results pipeline, the pipeline runs on a new ref that contains the merge result of the source and target branch.
@@ -256,8 +259,7 @@ Learn how to perform [tests that require special setup or consideration to run o
## How do you write tests?
-In order to write new tests, you first need to learn more about GitLab QA
-architecture. See the [documentation about it](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md).
+Before you write new tests, review the [GitLab QA architecture](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md).
After you've decided where to put [test environment orchestration scenarios](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario) and
[instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features), take a look at the [GitLab QA README](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/README.md),
diff --git a/doc/development/testing_guide/end_to_end/package_and_test_pipeline.md b/doc/development/testing_guide/end_to_end/package_and_test_pipeline.md
new file mode 100644
index 00000000000..b0257e7b02c
--- /dev/null
+++ b/doc/development/testing_guide/end_to_end/package_and_test_pipeline.md
@@ -0,0 +1,134 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# e2e:package-and-test
+
+The `e2e:package-and-test` child pipeline is the main executor of E2E testing for the GitLab platform. The pipeline definition has several dynamic
+components to reduce the number of tests being executed in merge request pipelines.
+
+## Setup
+
+Pipeline setup consists of:
+
+- The `e2e-test-pipeline-generate` job in the `prepare` stage of the main GitLab pipeline.
+- The `e2e:package-and-test` job in the `qa` stage, which triggers the child pipeline that is responsible for building the `omnibus` package and
+ running E2E tests.
+
+### e2e-test-pipeline-generate
+
+This job consists of two components that implement selective test execution:
+
+- The [`detect_changes`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/tasks/ci.rake) Rake task determines which e2e specs should be executed
+ in a particular merge request pipeline. This task analyzes changes in a particular merge request and determines which specs must be executed.
+ Based on that, a `dry-run` of every [scenario](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test) executes and determines if a
+ scenario contains any executable tests. Selective test execution uses [these criteria](index.md#selective-test-execution) to determine which specific
+ tests to execute.
+- [`generate-e2e-pipeline`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/generate-e2e-pipeline) is executed, which generates a child
+ pipeline YAML definition file with appropriate environment variables.
+
+### e2e:package-and-test
+
+E2E test execution pipeline consists of several stages which all support execution of E2E tests.
+
+#### .pre
+
+This stage is responsible for the following tasks:
+
+- Fetching `knapsack` reports that support [parallel test execution](index.md#run-tests-in-parallel).
+- Triggering downstream pipeline which builds the [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) Docker image.
+
+#### test
+
+This stage runs e2e tests against different types of GitLab configurations. The number of jobs executed is determined dynamically by
+[`e2e-test-pipeline-generate`](package_and_test_pipeline.md#e2e-test-pipeline-generate) job.
+
+#### report
+
+This stage is responsible for [allure test report](index.md#allure-report) generation.
+
+## Adding new jobs
+
+Selective test execution depends on a set of rules present in every job definition. A typical job contains the following attributes:
+
+```yaml
+variables:
+ QA_SCENARIO: Test::Integration::MyNewJob
+rules:
+ - !reference [.rules:test:qa, rules]
+ - if: $QA_SUITES =~ /Test::Integration::MyNewJob/
+ - !reference [.rules:test:manual, rules]
+```
+
+In this example:
+
+- `QA_SCENARIO: Test::Integration::MyNewJob`: name of the scenario class that is passed to the
+ [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md) executor.
+- `!reference [.rules:test:qa, rules]`: main rule definition that is matched for pipelines that should execute all tests. For example, when changes to
+ `qa` framework are present.
+- `if: $QA_SUITES =~ /Test::Integration::MyNewJob/`: main rule responsible for selective test execution. `QA_SUITE` is the name of the scenario
+ abstraction located in [`qa framework`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test).
+
+ `QA_SUITE` is not the same as `QA_SCENARIO`, which is passed to the `gitlab-qa` executor. For consistency, it usually has the same name. `QA_SUITE`
+ abstraction class usually contains information on what tags to run and optionally some additional setup steps.
+- `!reference [.rules:test:manual, rules]`: final rule that is always matched and sets the job to `manual` so it can still be executed on demand,
+ even if not set to execute by selective test execution.
+
+Considering example above, perform the following steps to create a new job:
+
+1. Create new scenario type `my_new_job.rb` in the [`integration`](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/master/lib/gitlab/qa/scenario/test/integration) directory
+ of the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa) project and release new version so it's generally available.
+1. Create new scenario `my_new_job.rb` in [`integration`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test/integration) directory of the
+ [`qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa) framework. In the most simple case, this scenario would define RSpec tags that should be executed:
+
+ ```ruby
+ module QA
+ module Scenario
+ module Test
+ module Integration
+ class MyNewJob < Test::Instance::All
+ tags :some_special_tag
+ end
+ end
+ end
+ end
+ end
+ ```
+
+1. Add new job definition in the [`main.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/package-and-test/main.gitlab-ci.yml) pipeline definition:
+
+ ```yaml
+ ee:my-new-job:
+ extends: .qa
+ variables:
+ QA_SCENARIO: Test::Integration::MyNewJob
+ rules:
+ - !reference [.rules:test:qa, rules]
+ - if: $QA_SUITES =~ /Test::Integration::MyNewJob/
+ - !reference [.rules:test:manual, rules]
+ ```
+
+### Parallel jobs
+
+For selective execution to work correctly with job types that require running multiple parallel jobs,
+a job definition typically must be split into parallel and selective variants. Splitting is necessary so that when selective execution
+executes only a single spec, multiple unnecessary jobs are not spawned. For example:
+
+```yaml
+ee:my-new-job-selective:
+ extends: .qa
+ variables:
+ QA_SCENARIO: Test::Integration::MyNewJob
+ rules:
+ - !reference [.rules:test:qa-selective, rules]
+ - if: $QA_SUITES =~ /Test::Integration::MyNewJob/
+ee:my-new-job:
+ extends:
+ - .parallel
+ - ee:my-new-job-selective
+ rules:
+ - !reference [.rules:test:qa-parallel, rules]
+ - if: $QA_SUITES =~ /Test::Integration::MyNewJob/
+```
diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md
index 37d354bf60c..f3e072ffa21 100644
--- a/doc/development/testing_guide/end_to_end/resources.md
+++ b/doc/development/testing_guide/end_to_end/resources.md
@@ -49,7 +49,7 @@ create the resource via the public GitLab API:
- `#api_post_path`: The `POST` path to create a new resource.
- `#api_post_body`: The `POST` body (as a Ruby hash) to create a new resource.
-> Be aware that many API resources are [paginated](../../../api/index.md#pagination).
+> Be aware that many API resources are [paginated](../../../api/rest/index.md#pagination).
> If you don't find the results you expect, check if there is more that one page of results.
Let's take the `Shirt` resource class, and add these three API methods:
diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md
index 419942d6b8f..32d8bf339ed 100644
--- a/doc/development/testing_guide/end_to_end/style_guide.md
+++ b/doc/development/testing_guide/end_to_end/style_guide.md
@@ -8,6 +8,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w
This document describes the conventions used at GitLab for writing End-to-end (E2E) tests using the GitLab QA project.
+Please note that this guide is an extension of the primary [testing standards and style guidelines](../index.md). If this guide defines a rule that contradicts the primary guide, this guide takes precedence.
+
## `click_` versus `go_to_`
### When to use `click_`?
diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md
index ef5b75d166f..f476815bf32 100644
--- a/doc/development/testing_guide/flaky_tests.md
+++ b/doc/development/testing_guide/flaky_tests.md
@@ -154,8 +154,11 @@ When a test frequently fails in `master`,
create [a ~"failure::flaky-test" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master).
If the test cannot be fixed in a timely fashion, there is an impact on the
-productivity of all the developers, so it should be quarantined by
-assigning the `:quarantine` metadata with the issue URL, and add the `~"quarantined test"` label to the issue.
+productivity of all the developers, so it should be quarantined. There are two ways to quarantine tests, depending on the test framework being used: RSpec and Jest.
+
+### RSpec
+
+For RSpec tests, you can use the `:quarantine` metadata with the issue URL.
```ruby
it 'succeeds', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/12345' do
@@ -169,6 +172,26 @@ This means it is skipped unless run with `--tag quarantine`:
bin/rspec --tag quarantine
```
+### Jest
+
+For Jest specs, you can use the `.skip` method along with the `eslint-disable-next-line` comment to disable the `jest/no-disabled-tests` ESLint rule and include the issue URL. Here's an example:
+
+```javascript
+// https://gitlab.com/gitlab-org/gitlab/-/issues/56789
+// eslint-disable-next-line jest/no-disabled-tests
+it.skip('should throw an error', () => {
+ expect(response).toThrowError(expected_error)
+});
+```
+
+This means it is skipped unless the test suit is run with `--runInBand` Jest command line option:
+
+```shell
+jest --runInBand
+```
+
+For both test frameworks, make sure to add the `~"quarantined test"` label to the issue.
+
Once a test is in quarantine, there are 3 choices:
- Fix the test (that is, get rid of its flakiness).
diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md
index 5edf18725c0..b074a9e34f3 100644
--- a/doc/development/testing_guide/review_apps.md
+++ b/doc/development/testing_guide/review_apps.md
@@ -21,18 +21,23 @@ For any of the following scenarios, the `start-review-app-pipeline` job would be
- for scheduled pipelines
- the MR has the `pipeline:run-review-app` label set
-## QA runs on review apps
+## E2E test runs on review apps
-On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage (which comes after the
-`review` stage), the `review-qa-smoke` and `review-qa-reliable` jobs are automatically started. The `review-qa-smoke` runs
-the QA smoke suite and the `review-qa-reliable` executes E2E tests identified as [reliable](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/).
+On every pipeline in the `qa` stage (which comes after the `review` stage), the `review-qa-smoke` and `review-qa-blocking` jobs are automatically started.
-`review-qa-*` jobs ensure that end-to-end tests for the changes in the merge request pass in a live environment. This shifts the identification of e2e failures from an environment on the path to production to the merge request, to prevent breaking features on GitLab.com or costly GitLab.com deployment blockers. `review-qa-*` failures should be investigated with counterpart SET involvement if needed to help determine the root cause of the error.
+`qa` stage consists of following jobs:
-You can also manually start the `review-qa-all`: it runs the full QA suite.
+- `review-qa-smoke`: small and fast subset of tests to validate core functionality of GitLab.
+- `review-qa-blocking`: subset of tests identified as [reliable](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/). These tests are
+ considered stable and are not allowed to fail.
+- `review-qa-non-blocking`: rest of the e2e tests that can be triggered manually.
+
+`review-qa-*` jobs ensure that end-to-end tests for the changes in the merge request pass in a live environment. This shifts the identification of e2e failures from an environment
+on the path to production to the merge request to prevent breaking features on GitLab.com or costly GitLab.com deployment blockers. If needed, `review-qa-*` failures should be
+investigated with an SET (software engineer in test) counterpart to help determine the root cause of the error.
After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by
-the `allure-report-qa-smoke`, `allure-report-qa-reliable`, and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request.
+the `e2e-test-report` job. A comment with links to the reports is added to the merge request.
Errors can be found in the `gitlab-review-apps` Sentry project and [filterable by review app URL](https://sentry.gitlab.net/gitlab/gitlab-review-apps/?query=url%3A%22https%3A%2F%2Fgitlab-review-require-ve-u92nn2.gitlab-review.app%2F%22) or [commit SHA](https://sentry.gitlab.net/gitlab/gitlab-review-apps/releases/6095b501da7/all-events/).
@@ -129,31 +134,28 @@ resource.labels.pod_name:"review-qa-raise-e-12chm0-migrations"
```mermaid
graph TD
A["build-qa-image, compile-production-assets<br/>(canonical default refs only)"];
+ B1[start-review-app-pipeline];
B[review-build-cng];
- C[review-deploy];
+ C["review-deploy<br><br>Helm deploys the review app using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br><br>Cloud Native images are deployed to the `review-apps`<br>Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."];
D[CNG-mirror];
- E[review-qa-smoke, review-qa-reliable];
+ E[review-qa-smoke, review-qa-blocking, review-qa-non-blocking<br><br>gitlab-qa runs the e2e tests against the review app.];
- A -->|once the `prepare` stage is done| B
- B -.->|triggers a CNG-mirror pipeline and wait for it to be done| D
- D -.->|polls until completed| B
- B -->|once the `review-build-cng` job is done| C
- C -->|once the `review-deploy` job is done| E
+ A --> B1
+ B1 --> B
+ B -.->|triggers a CNG-mirror pipeline| D
+ D -.->|depends on the multi-project pipeline| B
+ B --> C
+ C --> E
-subgraph "1. gitlab `prepare` stage"
+subgraph "1. gitlab-org/gitlab parent pipeline"
A
+ B1
end
-subgraph "2. gitlab `review-prepare` stage"
+subgraph "2. gitlab-org/gitlab child pipeline"
B
- end
-
-subgraph "3. gitlab `review` stage"
- C["review-deploy<br><br>Helm deploys the review app using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br><br>Cloud Native images are deployed to the `review-apps`<br>Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."]
- end
-
-subgraph "4. gitlab `qa` stage"
- E[review-qa-smoke, review-qa-reliable<br><br>gitlab-qa runs the smoke and reliable suites against the review app.]
+ C
+ E
end
subgraph "CNG-mirror pipeline"
diff --git a/doc/development/transient/prevention-patterns.md b/doc/development/transient/prevention-patterns.md
index 98634df22e9..730cc757396 100644
--- a/doc/development/transient/prevention-patterns.md
+++ b/doc/development/transient/prevention-patterns.md
@@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Preventing Transient Bugs
-This page will cover architectural patterns and tips for developers to follow to prevent [transient bugs.](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#transient-bugs)
+This page covers architectural patterns and tips for developers to follow to prevent [transient bugs.](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#transient-bugs)
## Common root causes
@@ -21,7 +21,7 @@ We've noticed a few root causes that come up frequently when addressing transien
### Don't rely on response order
-When working with multiple requests, it's easy to assume the order of the responses will match the order in which they are triggered.
+When working with multiple requests, it's easy to assume the order of the responses matches the order in which they are triggered.
That's not always the case and can cause bugs that only happen if the order is switched.
diff --git a/doc/development/uploads/working_with_uploads.md b/doc/development/uploads/working_with_uploads.md
index a3951fb4c7e..6955f9c31cd 100644
--- a/doc/development/uploads/working_with_uploads.md
+++ b/doc/development/uploads/working_with_uploads.md
@@ -38,7 +38,7 @@ is:
File.join(model.class.underscore, mounted_as.to_s, model.id.to_s)
```
-If you look around in the GitLab code base you will find quite a few
+If you look around in the GitLab code base you find quite a few
Uploaders that have their own storage location. For object storage,
this means Uploaders have their own buckets. We now **discourage**
adding new buckets for the following reasons:
@@ -53,7 +53,7 @@ and friction. The `Gitlab.config.uploads` storage location, which is what
## Implementing Direct Upload support
-Below we will outline how to implement [direct upload](#direct-upload-via-workhorse) support.
+Below we outline how to implement [direct upload](#direct-upload-via-workhorse) support.
Using direct upload is not always necessary but it is usually a good
idea. Unless the uploads handled by your feature are both infrequent
@@ -107,7 +107,7 @@ You should also manually verify that when you perform an upload
request for your new feature, Workhorse makes a pre-authorization
request. You can check this by looking at the Rails access logs. This
is necessary because if you make a mistake in your routing rule you
-won't get a hard failure: you just end up using the less efficient
+don't get a hard failure: you just end up using the less efficient
default path.
### Adding a pre-authorization endpoint
@@ -123,8 +123,8 @@ Consider accepting your file upload via Grape instead.
For Grape pre-authorization endpoints, look for existing examples that
implement `/authorize` routes. One example is the
[POST `:id/uploads/authorize` endpoint](https://gitlab.com/gitlab-org/gitlab/-/blob/9ad53d623eecebb799ce89eada951e4f4a59c116/lib/api/projects.rb#L642-651).
-Note that this particular example is using FileUploader, which means
-that the upload will be stored in the storage location (bucket) of
+This particular example is using FileUploader, which means
+that the upload is stored in the storage location (bucket) of
that Uploader class.
For Rails endpoints you can use the
@@ -154,7 +154,7 @@ scale.
GitLab uses a modified version of
[CarrierWave](https://github.com/carrierwaveuploader/carrierwave) to
-manage uploads. Below we will describe how we use CarrierWave and how
+manage uploads. Below we describe how we use CarrierWave and how
we modified it.
The central concept of CarrierWave is the **Uploader** class. The
@@ -197,15 +197,15 @@ particular, you currently cannot use the `version` mechanism of
CarrierWave. Things you can do include:
- Filename validation
-- **Incompatible with direct upload:** One time pre-processing of file contents, e.g. image resizing
+- **Incompatible with direct upload:** One time pre-processing of file contents, for example, image resizing
- **Incompatible with direct upload:** Encryption at rest
-Note that CarrierWave pre-processing behaviors such as image resizing
+CarrierWave pre-processing behaviors such as image resizing
or encryption require local access to the uploaded file. This forces
you to upload the processed file from Ruby. This flies against direct
upload, which is all about _not_ doing the upload in Ruby. If you use
direct upload with an Uploader with pre-processing behaviors then the
-pre-processing behaviors will be skipped silently.
+pre-processing behaviors are skipped silently.
### CarrierWave Storage engines
@@ -218,7 +218,7 @@ CarrierWave has 2 storage engines:
GitLab uses both of these engines, depending on configuration.
-The normal way to choose a storage engine in CarrierWave is to use the
+The typical way to choose a storage engine in CarrierWave is to use the
`Uploader.storage` class method. In GitLab we do not do this; we have
overridden `Uploader#storage` instead. This allows us to vary the
storage engine file by file.
@@ -228,13 +228,13 @@ storage engine file by file.
An Uploader is associated with two storage areas: regular storage and
cache storage. Each has its own storage engine. If you assign a file
to a mount point setter (`project.avatar = File.open('/tmp/tanuki.png')`)
-you will copy/move the file to cache
+you have to copy/move the file to cache
storage as a side effect via the `cache!` method. To persist the file
you must somehow call the `store!` method. This either happens via
[ActiveRecord callbacks](https://github.com/carrierwaveuploader/carrierwave/blob/v1.3.2/lib/carrierwave/orm/activerecord.rb#L55)
or by calling `store!` on an Uploader instance.
-Normally you do not need to interact with `cache!` and `store!` but if
+Typically you do not need to interact with `cache!` and `store!` but if
you need to debug GitLab CarrierWave modifications it is useful to
know that they are there and that they always get called.
Specifically, it is good to know that CarrierWave pre-processing
@@ -278,7 +278,7 @@ Direct upload works as follows.
1. Rails deletes the temporary upload
1. Workhorse deletes the temporary upload a second time in case Rails timed out
-Normally, `cache!` returns an instance of
+Typically, `cache!` returns an instance of
`CarrierWave::SanitizedFile`, and `store!` then
[uploads that file using Fog](https://github.com/carrierwaveuploader/carrierwave/blob/v1.3.2/lib/carrierwave/storage/fog.rb#L327-L335).
@@ -294,7 +294,7 @@ this file to its intended location.
## Tables
-The Scalability::Frameworks team is going to make object storage and uploads more easy to use and more robust. If you add or change uploaders, it helps us if you update this table too. This helps us keep an overview of where and how uploaders are used.
+The Scalability::Frameworks team is making object storage and uploads more easy to use and more robust. If you add or change uploaders, it helps us if you update this table too. This helps us keep an overview of where and how uploaders are used.
### Feature bucket details
diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md
index 33a6744d5cd..77a32b62e32 100644
--- a/doc/development/value_stream_analytics.md
+++ b/doc/development/value_stream_analytics.md
@@ -261,7 +261,7 @@ considered legacy, which will be phased out at some point.
- Rails Controller (`Analytics::CycleAnalytics` module): Value stream analytics exposes its data via JSON endpoints, implemented within the `analytics` workspace. Configuring the stages are also implements JSON endpoints (CRUD).
- Services (`Analytics::CycleAnalytics` module): All `Stage` related actions are delegated to respective service objects.
-- Models (`Analytics::CycleAnalytics` module): Models are used to persist the `Stage` objects `ProjectStage` and `Stage`.
+- Models (`Analytics::CycleAnalytics` module): Models are used to persist the `Stage` objects.
- Feature classes (`Gitlab::Analytics::CycleAnalytics` module):
- Responsible for composing queries and define feature specific business logic.
- `DataCollector`, `Event`, `StageEvents`, etc.
diff --git a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md
index 3d1286a5809..4d2a751e6c6 100644
--- a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md
+++ b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md
@@ -31,7 +31,7 @@ for long-term growth.
Our main database is not prepared for analytical workloads. Executing long-running queries can
affect the reliability of the application. For large groups, the current
implementation (old backend) is slow and, in some cases, doesn't even load due to the configured
-statement timeout (15s).
+statement timeout (15 s).
The database queries in the old backend use the core domain models directly through
`IssuableFinders` classes: ([MergeRequestsFinder](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/finders/merge_requests_finder.rb) and [IssuesFinder](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/finders/issues_finder.rb)).
@@ -62,7 +62,7 @@ different development workflows within the `Test Group` (top-level namespace).
The first value stream uses standard timestamp-based events for defining the stages. The second
value stream uses label events.
-Each value stream and stage item from the example will be persisted in the database. Notice that
+Each value stream and stage item from the example is persisted in the database. Notice that
the `Deployment` stage is identical for both value streams; that means that the underlying
`stage_event_hash_id` is the same for both stages. The `stage_event_hash_id` reduces
the amount of data the backend collects and plays a vital role in database partitioning.
@@ -111,8 +111,8 @@ the service performs operations in batches and enforces strict application limit
later.
- Continue processing data from a specific point.
-As of GitLab 14.7, the data loading is done manually. Once the feature is ready, the service will
-be invoked periodically by the system via a cron job (this part is not implemented yet).
+As of GitLab 14.7, the data loading is done manually. Once the feature is ready, the service is
+invoked periodically by the system via a cron job (this part is not implemented yet).
#### Record iteration
@@ -193,7 +193,7 @@ aggregated data separated, we use two additional database tables:
- `analytics_cycle_analytics_merge_request_stage_events`
Both tables are hash partitioned by the `stage_event_hash_id`. Each table uses 32 partitions. It's
-an arbitrary number and it could be changed. Important is to keep the partitions under 100GB in
+an arbitrary number and it could be changed. Important is to keep the partitions under 100 GB in
size (which gives the feature a lot of headroom).
|Column|Description|
diff --git a/doc/development/wikis.md b/doc/development/wikis.md
index 4541b6cca66..acb7ed335d3 100644
--- a/doc/development/wikis.md
+++ b/doc/development/wikis.md
@@ -12,7 +12,7 @@ description: "GitLab's development guidelines for Wikis"
## Overview
The wiki functionality in GitLab is based on [Gollum 4.x](https://github.com/gollum/gollum/).
-It's used in [Gitaly's](gitaly.md) Ruby service, and accessed from the Rails app through Gitaly RPC calls.
+It's used in the [Gitaly](gitaly.md) Ruby service, and accessed from the Rails app through Gitaly RPC calls.
Wikis use Git repositories as storage backend, and can be accessed through:
@@ -40,9 +40,9 @@ We only use Gollum as a storage abstraction layer, to handle the mapping between
When rendering wiki pages, we don't use Gollum at all and instead go through a
[custom Banzai pipeline](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/banzai/pipeline/wiki_pipeline.rb).
-This adds some [wiki-specific markup](../user/markdown.md#wiki-specific-markdown), such as Gollum's `[[link]]` syntax.
+This adds some [wiki-specific markup](../user/markdown.md#wiki-specific-markdown), such as the Gollum `[[link]]` syntax.
-Because we do not make use of most of Gollum's features, we plan to move away from it entirely at some point.
+Because we do not make use of most of the Gollum features, we plan to move away from it entirely at some point.
[See this epic](https://gitlab.com/groups/gitlab-org/-/epics/2381) for reference.
## Model classes
diff --git a/doc/development/work_items_widgets.md b/doc/development/work_items_widgets.md
index ba15a3f0163..5b9602595bb 100644
--- a/doc/development/work_items_widgets.md
+++ b/doc/development/work_items_widgets.md
@@ -112,3 +112,28 @@ for work items widgets development:
- `ee/app/assets/javascripts/boards/components/assignee_select.vue`
- `ee/app/assets/javascripts/boards/components/milestone_select.vue`
+
+## Mapping widgets to work item types
+
+All Work Item types share the same pool of predefined widgets and are customized by which widgets are active on a specific type. Because we plan to allow users to create new Work Item types and define a set of widgets for them, mapping of widgets for each Work Item type is stored in database. Mapping of widgets is stored in widget_definitions table and it can be used for defining widgets both for default Work Item types and also in future for custom types. More details about expected database table structure can be found in [this issue description](https://gitlab.com/gitlab-org/gitlab/-/issues/374092).
+
+### Adding new widget to a work item type
+
+Because information about what widgets are assigned to each work item type is stored in database, adding new widget to a work item type needs to be done through a database migration. Also widgets importer (`lib/gitlab/database_importers/work_items/widgets_importer.rb`) should be updated.
+
+### Structure of widget definitions table
+
+Each record in the table defines mapping of a widget to a work item type. Currently only "global" definitions (definitions with NULL namespace_id) are used. In next iterations we plan to allow customization of these mappings. For example table below defines that:
+
+- Weight widget is enabled for work item types 0 and 1
+- in namespace 1 Weight widget is renamed to MyWeight. When user renames widget's name, it makes sense to rename all widget mappings in the namespace - because `name` attribute is denormalized, we have to create namespaced mappings for all work item types for this widget type.
+- Weight widget can be disabled for specific work item types (in namespace 3 it's disabled for work item type 0, while still left enabled for work item type 1)
+
+| ID | Namespace_id | Work_item_type_id | Widget_type_enum | Position | Name | Disabled |
+|----| ------------ | ----------------- |----------------- |--------- |---------- |-------|
+| 1 | | 0 | 1 | 1 | Weight | false |
+| 2 | | 1 | 1 | 1 | Weight | false |
+| 3 | 1 | 0 | 1 | 0 | MyWeight | false |
+| 4 | 1 | 1 | 1 | 0 | MyWeight | false |
+| 5 | 2 | 0 | 1 | 1 | Other Weight | false |
+| 6 | 3 | 0 | 1 | 1 | Weight | true |
diff --git a/doc/development/workhorse/gitlab_features.md b/doc/development/workhorse/gitlab_features.md
index 8b899242935..b4146e8b62c 100644
--- a/doc/development/workhorse/gitlab_features.md
+++ b/doc/development/workhorse/gitlab_features.md
@@ -12,7 +12,7 @@ GitLab that would not work efficiently without Workhorse.
To put the efficiency benefit in context, consider that in 2020Q3 on
GitLab.com [we see](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1)
Rails application threads using on average
-about 200MB of RSS vs about 200KB for Workhorse goroutines.
+about 200 MB of RSS vs about 200 KB for Workhorse goroutines.
Examples of features that rely on Workhorse:
@@ -66,7 +66,6 @@ memory than it costs to have Workhorse look after it.
- Workhorse does not connect to PostgreSQL, only to Rails and (optionally) Redis.
- We assume that all requests that reach Workhorse pass through an
upstream proxy such as NGINX or Apache first.
-- Workhorse does not accept HTTPS connections.
- Workhorse does not clean up idle client connections.
- We assume that all requests to Rails pass through Workhorse.