diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2024-01-16 13:42:19 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2024-01-16 13:42:19 +0300 |
| commit | 84d1bd786125c1c14a3ba5f63e38a4cc736a9027 (patch) | |
| tree | f550fa965f507077e20dbb6d61a8269a99ef7107 /doc/development | |
| parent | 3a105e36e689f7b75482236712f1a47fd5a76814 (diff) | |
Add latest changes from gitlab-org/gitlab@16-8-stable-eev16.8.0-rc42
Diffstat (limited to 'doc/development')
115 files changed, 1580 insertions, 891 deletions
diff --git a/doc/development/activitypub/actor.md b/doc/development/activitypub/actor.md deleted file mode 100644 index 1d10e421df7..00000000000 --- a/doc/development/activitypub/actor.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'actors/index.md' -remove_date: '2023-12-08' ---- - -This document was moved to [another location](actors/index.md). - -<!-- This redirect file can be deleted after <2023-12-08>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/activitypub/actors/index.md b/doc/development/activitypub/actors/index.md index 6d82e79b9a0..e0367d71871 100644 --- a/doc/development/activitypub/actors/index.md +++ b/doc/development/activitypub/actors/index.md @@ -72,7 +72,7 @@ render json: ActivityPub::ReleasesActorSerializer.new.represent(project, opts) ``` - `outbox` is the endpoint where to find the activities feed for this -actor. + actor. - `inbox` is where to POST to subscribe to the feed. Not yet implemented, so pass `nil`. ## Outbox page diff --git a/doc/development/ai_features.md b/doc/development/ai_features.md deleted file mode 100644 index a952d8f2804..00000000000 --- a/doc/development/ai_features.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'ai_features/index.md' -remove_date: '2023-12-01' ---- - -This document was moved to [another location](ai_features/index.md). - -<!-- This redirect file can be deleted after <2023-12-01>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/ai_features/index.md b/doc/development/ai_features/index.md index e99a49f3bb9..f8680ef91c9 100644 --- a/doc/development/ai_features/index.md +++ b/doc/development/ai_features/index.md @@ -48,7 +48,7 @@ To implement a new AI action, connect to the preferred AI provider. You can conn All AI features are experimental. -## Test AI features locally +## Test SaaS-only AI features locally **One-line setup** @@ -97,11 +97,11 @@ In order to obtain a GCP service key for local development, follow the steps bel - Create a sandbox GCP project by visiting [this page](https://about.gitlab.com/handbook/infrastructure-standards/#individual-environment) and following the instructions, or by requesting access to our existing group GCP project by using [this template](https://gitlab.com/gitlab-com/it/infra/issue-tracker/-/issues/new?issuable_template=gcp_group_account_iam_update_request). - If you are using an individual GCP project, you may also need to enable the Vertex AI API: - 1. Visit [welcome page](https://console.cloud.google.com/welcome), choose your project (e.g. jdoe-5d23dpe). - 1. Go to **APIs & Services > Enabled APIs & services**. - 1. Select **+ Enable APIs and Services**. - 1. Search for `Vertex AI API`. - 1. Select **Vertex AI API**, then select **Enable**. + 1. Visit [welcome page](https://console.cloud.google.com/welcome), choose your project (e.g. jdoe-5d23dpe). + 1. Go to **APIs & Services > Enabled APIs & services**. + 1. Select **+ Enable APIs and Services**. + 1. Search for `Vertex AI API`. + 1. Select **Vertex AI API**, then select **Enable**. - Install the [`gcloud` CLI](https://cloud.google.com/sdk/docs/install) - Authenticate locally with GCP using the [`gcloud auth application-default login`](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) command. - Open the Rails console. Update the settings to: @@ -166,6 +166,132 @@ end View [guidelines](duo_chat.md) for working with GitLab Duo Chat. +## Test AI features with AI Gateway locally + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/11251) in GitLab 16.8. + +In order to develop an AI feature that is compatible with both SaaS and Self-managed GitLab instances, +the feature must request to the [AI Gateway](../../architecture/blueprints/ai_gateway/index.md) instead of directly requesting to the 3rd party model providers. +Therefore, a different setup is required from the [SaaS-only AI features](#test-saas-only-ai-features-locally). + +### Setup + +1. Set up AI Gateway: + 1. [Install it](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#how-to-run-the-server-locally). + 1. Ensure that the following environment variables are set in the `.env` file: + + ```shell + AIGW_AUTH__BYPASS_EXTERNAL=true + ANTHROPIC_API_KEY="[REDACTED]" # IMPORTANT: Ensure you use Corp account. See https://gitlab.com/gitlab-org/gitlab/-/issues/435911#note_1701762954. + AIGW_VERTEX_TEXT_MODEL__PROJECT="[REDACTED]" + ``` + + 1. Run `poetry run ai_gateway`. + 1. Visit OpenAPI playground (`http://0.0.0.0:5052/docs`), try an endpoint (e.g. `/v1/chat/agent`) and make sure you get a successful response. + If something went wrong, check `modelgateway_debug.log` if it contains error information. +1. Setup GitLab Development Kit (GDK): + 1. [Install it](https://gitlab.com/gitlab-org/gitlab-development-kit#installation). + 1. [Set up `gdk.test` hostname](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/index.md#set-up-gdktest-hostname). + 1. [Activate GitLab Enterprise license](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/index.md#use-gitlab-enterprise-features) (e.g. Ultimate). + 1. Export these environment variables in the same terminal session with `gdk start`: + + ```shell + export CODE_SUGGESTIONS_BASE_URL=http://0.0.0.0:5052 # URL to the local AI Gateway instance + export LLM_DEBUG=1 # Enable debug logging + ``` + + Alternatively, you can create an `env.runit` file in the root of your GDK with the above snippet. + 1. Enable the following feature flags via `gdk rails console`: + + ```ruby + # NOTE: This feature flag name might be changed. See https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140352. + ::Feature.enable(:ai_global_switch) + + # This is to request to AI Gateway instead of built-in Anthropic client. See https://gitlab.com/gitlab-org/gitlab/-/issues/433213 for more info. + ::Feature.enable(:gitlab_duo_chat_requests_to_ai_gateway) + ``` + + 1. Create a dummy access token via `gdk rails console` OR skip this step and setup GitLab or Customer Dot as OIDC provider (See the following section): + + ```ruby + # Creating dummy token, and this will work as long as `AIGW_AUTH__BYPASS_EXTERNAL=true` in AI Gateway. + ::Ai::ServiceAccessToken.create!(token: 'dummy', expires_at: 1.month.from_now) + ``` + + 1. Ensure GitLab-Rails can talk to the AI Gateway. Run `gdk rails console` and execute: + + ```ruby + user = User.first + Gitlab::Llm::AiGateway::Client.new(user).stream(prompt: "\n\nHuman: Hi, how are you?\n\nAssistant:") + ``` + +#### Verify the setup with GraphQL + +1. Visit [GraphQL explorer](../../api/graphql/index.md#interactive-graphql-explorer). +1. Execute the `aiAction` mutation. Here is an example: + + ```graphql + mutation { + aiAction( + input: { + chat: { + resourceId: "gid://gitlab/User/1", + content: "Hello" + } + } + ){ + requestId + errors + } + } + ``` + +1. (GitLab Duo Chat only) Execute the following query to fetch the response: + + ```graphql + query { + aiMessages { + nodes { + requestId + content + role + timestamp + chunkId + errors + } + } + } + ``` + + If you can't fetch the response, check `graphql_json.log`, `sidekiq_json.log`, `llm.log` or `modelgateway_debug.log` if it contains error information. + +### Use GitLab as OIDC provider in AI Gateway + +1. Reconfigure AI Gateway: + 1. Additionally, ensure that the following environment variables are set in the `.env` file: + + ```shell + AIGW_GITLAB_URL="http://gdk.test:3000/" + AIGW_GITLAB_API_URL="http://gdk.test:3000/api/v4/" + AIGW_AUTH__BYPASS_EXTERNAL=False + ``` + + 1. Restart AI Gateway. +1. Reconfigure GitLab Development Kit (GDK): + 1. Additionally, export the following environment variables: + + ```shell + export GITLAB_SIMULATE_SAAS=1 # Simulate a SaaS instance. See https://docs.gitlab.com/ee/development/ee_features.html#simulate-a-saas-instance. + ``` + + 1. Restart GDK. + +### Use Customer Dot as OIDC provider in AI Gateway + +1. AI Gateway: + 1. Ensure `AIGW_CUSTOMER_PORTAL_BASE_URL` in the `.env` file points to your Customer Dot URL. + 1. Restart + ## Experimental REST API Use the [experimental REST API endpoints](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/ai/experimentation) to quickly experiment and prototype AI features. @@ -210,7 +336,7 @@ As an example, assume we want to build an "explain code" action. To do this, we ```graphql mutation { - aiAction(input: {explainCode: {resourceId: "gid://gitlab/MergeRequest/52", code: "foo() { console.log()" }}) { + aiAction(input: {explainCode: {resourceId: "gid://gitlab/MergeRequest/52", code: "foo() { console.log() }" }}) { clientMutationId } } diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index cfe82fe9b81..606d8c77432 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -178,7 +178,7 @@ Breaking changes are: allowed so long as all scalar type fields of the object continue to serialize in the same way. - Raising the [complexity](#max-complexity) of a field or complexity multipliers in a resolver. - Changing a field from being _not_ nullable (`null: false`) to nullable (`null: true`), as -discussed in [Nullable fields](#nullable-fields). + discussed in [Nullable fields](#nullable-fields). - Changing an argument from being optional (`required: false`) to being required (`required: true`). - Changing the [max page size](#page-size-limit) of a connection. - Lowering the global limits for query complexity and depth. @@ -1515,8 +1515,8 @@ To find the parent object in your `Presenter` class: ``` 1. Declare your field's method in your Presenter class and have it accept the `parent` keyword argument. -This argument contains the parent **GraphQL context**, so you have to access the parent object with -`parent[:parent_object]` or whatever key you used in your `Resolver`: + This argument contains the parent **GraphQL context**, so you have to access the parent object with + `parent[:parent_object]` or whatever key you used in your `Resolver`: ```ruby # in ChildPresenter diff --git a/doc/development/auto_devops.md b/doc/development/auto_devops.md index d5a011e7e55..94dbcea514b 100644 --- a/doc/development/auto_devops.md +++ b/doc/development/auto_devops.md @@ -18,7 +18,7 @@ is also available on YouTube. Auto DevOps builds on top of GitLab CI/CD to create an automatic pipeline based on your project contents. When Auto DevOps is enabled for a project, the user does not need to explicitly include any pipeline configuration -through a [`.gitlab-ci.yml` file](../ci/yaml/index.md). +through a [`.gitlab-ci.yml` file](../ci/index.md#the-gitlab-ciyml-file). In the absence of a `.gitlab-ci.yml` file, the [Auto DevOps CI/CD template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) diff --git a/doc/development/bulk_import.md b/doc/development/bulk_import.md index 6c0bed8e204..752bcbbac19 100644 --- a/doc/development/bulk_import.md +++ b/doc/development/bulk_import.md @@ -12,7 +12,7 @@ NOTE: To use direct transfer, ensure your GitLab installation is accessible from [GitLab IP addresses](../user/gitlab_com/index.md#ip-range) and has a public DNS entry. -[Group migration by direct transfer](../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended) is the +[Group migration by direct transfer](../user/group/import/index.md) is the evolution of migrating groups and projects using file exports. The goal is to have an easier way for the user to migrate a whole group, including projects, from one GitLab instance to another. @@ -36,8 +36,8 @@ idea is to have one ETL pipeline for each relation to be imported. ### API -The current [project](../user/project/settings/import_export.md) and -[group](../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) imports are file based, so +The current [project](../user/project/settings/import_export.md#migrate-projects-by-uploading-an-export-file) and +[group](../user/project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated) imports are file based, so they require an export step to generate the file to be imported. Group migration by direct transfer leverages the [GitLab API](../api/rest/index.md) to speed the migration. diff --git a/doc/development/cicd/cicd_reference_documentation_guide.md b/doc/development/cicd/cicd_reference_documentation_guide.md index ccd952f586c..3832ba182b6 100644 --- a/doc/development/cicd/cicd_reference_documentation_guide.md +++ b/doc/development/cicd/cicd_reference_documentation_guide.md @@ -6,7 +6,7 @@ info: Any user with at least the Maintainer role can merge updates to this conte # Documenting the `.gitlab-ci.yml` keywords -The [CI/CD YAML reference](../../ci/yaml/index.md) uses a standard style to make it easier to use and update. +The [CI/CD YAML syntax reference](../../ci/yaml/index.md) uses a standard style to make it easier to use and update. The reference information should be kept as simple as possible, and expanded details and examples should be documented on other pages. diff --git a/doc/development/cicd/components.md b/doc/development/cicd/components.md new file mode 100644 index 00000000000..56ab5a24bd1 --- /dev/null +++ b/doc/development/cicd/components.md @@ -0,0 +1,80 @@ +--- +stage: Verify +group: Pipeline Authoring +info: Any user with at least the Maintainer role can merge updates to this content. For details, see https://docs.gitlab.com/ee/development/development_processes.html#development-guidelines-review. +--- + +# Development guide for GitLab CI/CD components + +This document explains how to develop [CI/CD components](../../ci/components/index.md) that are maintained by GitLab. + +The official location for all GitLab-maintained component projects is the [`gitlab.com/components`](https://gitlab.com/components) group. +This group contains all components that are designed to be generic, served to all GitLab users, and maintained by GitLab. + +A component project can initially be created under a different group (for example `gitlab-org`) +but they need to be moved into the `components` group before the first version gets published to the catalog. + +Components that are for GitLab internal use only, for example specific to `gitlab-org/gitlab` project, should be +implemented under `gitlab-org` group. + +Component projects that are expected to be published in the [CI/CD catalog](../../ci/components/index.md#cicd-catalog) +should first be dogfooded it to ensure we stay on top of the project quality and have first-hand +experience with it. + +## Define ownership + +GitLab-maintained components are trusted by the community and require a high degree of quality and timely maintenance. +Components must be kept up to date, monitored for security vulnerabilities, and bugs fixed. + +Each component project must have a set of owners and maintainers that are also domain experts. +Experts can be from any department in GitLab, from Engineering to Support, Customer Success, and Developer Relations. + +If a component is related to a GitLab feature (for example Secret Detection), the team that owns the +feature category or is most closely related to it should maintain the project. + +The component project can be created by a separate team or individual initially but it must be transitioned +to a set of owners before the first version gets published to the catalog. + +The `README.md` file in the project repository must indicate the main owners of the project so that +they can be contacted by the wider community if needed. + +NOTE: +If a set of project owners cannot be guaranteed or the components cannot be dogfooded, we strongly recommend +not creating a GitLab-maintained component project and instead let the wider community fulfill the demand +in the catalog. + +## Development process + +1. Create a project under [`gitlab.com/components`](https://gitlab.com/components) + or ask one of the group owners to create an empty project for you. +1. Follow the [standard guide for creating components](../../ci/components/index.md). +1. Add a concise project description that clearly describes the capabilities offered by the component project. +1. Ensure that the [best practices](../../ci/components/index.md#best-practices) are followed. +1. Use [semantic versioning](https://semver.org) in the form `MAJOR.MINOR` or `MAJOR.MINOR.PATCH`. +1. Add a `LICENSE.md` file with the MIT license. +1. The project must have a `.gitlab-ci.yml` file that: + - Validates all the components in the project correctly. + - Contains a `release` job to publish newly released tags to the catalog. +1. Ensure that the `README.md` contains at least the sections below (for example, see the [Code quality component](https://gitlab.com/components/code-quality)): + - **Overview**: The capabilities offered by the component project. + - **Components**: Sub-sections for each component, each with: + - **Usage**: Examples with and without inputs (when optional). + - **Inputs**: A table showing the input names, types, default values (if any) and descriptions. + - **Variables** (when applicable): The variable names, possible values, and descriptions. + - **Contribute**: Notes and how to get in touch with the maintainers. + Usually the contribution process should follow the [official guide](../../ci/components/index.md). +1. Upload the [official avatar image](img/avatar_component_project.png) to the component project. + +## Review and contribution process + +It's possible that components in the project have a related [CI/CD template](templates.md) in the GitLab codebase. +In that case we need to cross link the component project and CI/CD template: + +- Add a comment in the CI/CD template with the location of the related component project. +- Add a section in the `README.md` of the component project with the location of the existing CI/CD template. + +When changes are applied to these components, check whether we can integrate the changes in the CI/CD template too. +This might not be possible due to the rigidity of versioning in CI/CD templates. + +Ping [`@gitlab-org/maintainers/ci-components`](https://gitlab.com/groups/gitlab-org/maintainers/ci-components/-/group_members?with_inherited_permissions=exclude) +for reviews to ensure that the components are written in consistent style and follow the best practices. diff --git a/doc/development/cicd/configuration.md b/doc/development/cicd/configuration.md new file mode 100644 index 00000000000..60f55174651 --- /dev/null +++ b/doc/development/cicd/configuration.md @@ -0,0 +1,100 @@ +--- +stage: Verify +group: Pipeline Authoring +info: Any user with at least the Maintainer role can merge updates to this content. For details, see https://docs.gitlab.com/ee/development/development_processes.html#development-guidelines-review. +--- + +# Contribute to the CI/CD configuration + +## Glossary + +- **CI/CD configuration**: The YAML file that defines the CI/CD configuration for a project. +- **keyword**: Each keyword in the CI/CD configuration. +- **entry**: An `Entry` class that represents a keyword in the CI/CD configuration. + +Not every keyword in the CI/CD configuration is represented by an `Entry` class. +We create `Entry` classes for keywords that have a complex structure or reusable parts. + +For example; + +- The `image` keyword is represented by the [`Entry::Image`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/config/entry/image.rb) class. +- The `name` subkeyword of the `image` keyword is not represented by an `Entry` class. +- The `pull_policy` subkeyword of the `image` keyword is represented by the [`Entry::PullPolicy`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/config/entry/pull_policy.rb) class. + +## Adding New Keywords + +CI config keywords are added in the [`lib/gitlab/ci/config/entry`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/config/entry) directory. +For EE-specific changes, use the [`ee/lib/gitlab/ci/config/entry`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/ci/config/entry) +or [`ee/lib/ee/gitlab/ci/config/entry`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/ee/gitlab/ci/config/entry) directory. + +### Inheritance + +An entry is represented by a class that inherits from; + +- `Entry::Node`: for simple keywords. + (e.g. [`Entry::Stage`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/config/entry/stage.rb)) +- `Entry::Simplifiable`: for keywords that have multiple structures. + For example, [`Entry::Retry`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/config/entry/retry.rb) can be a simple number or a hash configuration. +- `Entry::ComposableArray`: for keywords that have a list of single-type sub-elements. + For example, [`Entry::Includes`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/config/entry/includes.rb) has a list of `Entry::Include` elements. +- `Entry::ComposableHash`: for keywords that have single-type sub-elements with user-defined keys. + For example, [`Entry::Variables`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/config/entry/variables.rb) has a list of `Entry::Variable` elements with user-defined keys. + +### Helper Classes + +The following helper classes are available for use in entries: + +- `Entry::Validatable`: Enables the `validations` block in an entry class and provides validations. +- `Entry::Attributable`: Enables the `attributes` method in an entry class. It creates these methods for each attribute; `xxx`, `has_xxx?`, `has_xxx_value?`. +- `Entry::Configurable`: Enables the `entry` method in an entry class. It creates these methods for each entry; `xxx_defined?`, `xxx_entry`, `xxx_value`. + +### The `value` Method + +The `value` method is the main method of an entry class. It returns the actual value of the entry. +By default, from the `Entry::Node` class, the `value` method returns the hash configuration of the entry unless it has nested entries. +It can be useful for simple entries. For example, `Entry::Paths` has an array of strings as its value. So, it can return the array of strings directly. + +In some keywords, we override the `value` method. In this method, we return what and how we want to return from the entry. +The usage of `Entry::Attributable` and `Entry::Configurable` may have a significant role here. For example, +in `Entry::Secret`, we have this; + +```ruby +attributes %i[vault file token].freeze + +entry :vault, Entry::Vault::Secret +entry :file, ::Gitlab::Config::Entry::Boolean + +def value + { + vault: vault_value, + file: file_value, + token: token + }.compact +end +``` + +- `vault_value` is the value of the nested `vault` entry. +- `file_value` is the value of the nested `file` entry. +- `token` is the value of the basic `token` attribute. + +**It is important** that we should always use the `xxx_value` method to get the value of a nested entry. + +## Feature Flag Usage + +When adding new CI/CD configuration keywords, it is important to use feature flags to control the rollout of the change. +This allows us to test the change in production without affecting all users. For more information, see the [feature flags documentation](../feature_flags/index.md). + +### Feature Flag Actor + +In entry classes, we have no access to the current project or user. However, it's discouraged to use feature flags without [an actor](../feature_flags/index.md#feature-actors). +To solve this problem, we have three options; + +1. Use `Feature.enabled?(:feature_flag, Feature.current_request)`. +1. Use `YamlProcessor::FeatureFlags.enabled?(:feature_flag)` +1. Do not use feature flags in entry classes and use them in other parts of the code. + +## Testing and Validation + +When adding or modifying an entry, the corresponding spec file must be either added or updated. +Besides, to have a fully integrated test, it's also important to add/modify tests in the `spec/lib/gitlab/ci/yaml_processor_spec.rb` file or +the files in `spec/lib/gitlab/ci/yaml_processor/test_cases/*` directory. diff --git a/doc/development/cicd/img/avatar_component_project.png b/doc/development/cicd/img/avatar_component_project.png Binary files differnew file mode 100644 index 00000000000..e5c20d108fa --- /dev/null +++ b/doc/development/cicd/img/avatar_component_project.png diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index 18781f9315a..a8dfdeb30a3 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -9,10 +9,12 @@ info: Any user with at least the Maintainer role can merge updates to this conte Development guides that are specific to CI/CD are listed here: - If you are creating new CI/CD templates, read [the development guide for GitLab CI/CD templates](templates.md). -- If you are adding a new keyword or changing the CI schema, check the [CI schema guide](schema.md) +- If you are adding a new keyword or changing the CI schema, refer to the following guides: + - [The CI configuration guide](configuration.md) + - [The CI schema guide](schema.md) See the [CI/CD YAML reference documentation guide](cicd_reference_documentation_guide.md) -to learn how to update the [reference page](../../ci/yaml/index.md). +to learn how to update the [CI/CD YAML syntax reference page](../../ci/yaml/index.md). ## Examples of CI/CD usage diff --git a/doc/development/cicd/pipeline_wizard.md b/doc/development/cicd/pipeline_wizard.md index 0c0c0f3cc45..b534fede6e9 100644 --- a/doc/development/cicd/pipeline_wizard.md +++ b/doc/development/cicd/pipeline_wizard.md @@ -138,7 +138,7 @@ is planned to add the ability to create a MR from here. ### Events - `done` - Emitted after the file has been committed. Use this to redirect the -user to the pipeline, for example. + user to the pipeline, for example. ### Template file location diff --git a/doc/development/cicd/schema.md b/doc/development/cicd/schema.md index e9a0b93b5f3..c24b7d21286 100644 --- a/doc/development/cicd/schema.md +++ b/doc/development/cicd/schema.md @@ -30,7 +30,7 @@ a step-by-step introduction on how to work with JSON schemas. The CI/CD schema is at [`app/assets/javascripts/editor/schema/ci.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci.json). It contains all the keywords available for authoring CI/CD configuration files. -Check the [keyword reference](../../ci/yaml/index.md) for a comprehensive list of +Check the [CI/CD YAML syntax reference](../../ci/yaml/index.md) for a comprehensive list of all available keywords. All keywords are defined under `definitions`. We use these definitions as diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index a2b490b9106..bd3023ebf8d 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -13,7 +13,7 @@ we encourage team members to create [CI/CD components](../../ci/components/index for the catalog. This transition enhances the modularity and maintainability of our shared CI/CD resources, and avoids the complexities of contributing new CI/CD templates. If you need to update an existing template, you must also update the matching CI/CD component. -If no component exists that matches the CI/CD template yet, consider creating the matching component. +If no component exists that matches the CI/CD template yet, consider [creating the matching component](components.md). This ensures that template and component functionality remain in sync, aligning with our new development practices. diff --git a/doc/development/code_owners/index.md b/doc/development/code_owners/index.md index 45c632d5adc..b8e99475dd3 100644 --- a/doc/development/code_owners/index.md +++ b/doc/development/code_owners/index.md @@ -53,12 +53,27 @@ namespace. Code Owners is an EE-only feature, so the files only exist in the `./ have been changed when a user pushes to a protected branch with `require_code_owner_approval` enabled. - Defined in `./ee/lib/gitlab/code_owners/validator.rb`. +## Where Code Owners sits in the Git access check execution order + +`Gitlab::Checks::DiffCheck#file_paths_validations` returns either an empty array, or an array with a single member of the results of `#lfs_file_locks_validation` if LFS is enabled and file locks are present. The return result of `#validate_code_owners` in the EE version of this file is inserted at the end of this list in the `EE::Gitlab::Checks::DiffCheck#file_paths_validations`. LFS checks are performed before Code Owners checks. + +These checks are executed after those listed in `#validations_for_path`, which exists only in the EE version, and include `#path_locks_validation` and `#file_name_validation`. This means that checks for Path Locks precede checks for Code Owners in the flow. + +The check order is as follows in `EE` (only LFS exists as a non-EE feature): + +- Path Locks +- File Names + - Blocks files containing secrets for example `id_rsa` + - Blocks files matching the `PushRule#file_name_regex` +- LFS File Locks +- Code Owners + ## Related models ### `ProtectedBranch` The `ProtectedBranch` model is defined in `app/models/protected_branch.rb` and -extended in `ee/app/ee/models/protected_branch.rb`. The EE version includes a column +extended in `ee/app/models/concerns/ee/protected_branch.rb`. The EE version includes a column named `require_code_owner_approval` which prevents changes from being pushed directly to the branch being protected if the file is listed in `CODEOWNERS`. @@ -108,7 +123,9 @@ This service is defined in `services/merge_requests/sync_code_owner_approval_rul These flowcharts should help explain the flow from the controllers down to the models for different features. -### Push changes to a protected branch with `require_code_owner_approval` enabled +Note that many of the Code Owners implementations exist in the `EE` variants of the classes. + +### Push changes to a protected branch with `require_code_owner_approval` enabled, over SSH ```mermaid graph TD @@ -120,6 +137,19 @@ graph TD Gitlab::CodeOwners::Loader --> Gitlab::CodeOwners::Entry ``` +### Push changes to a protected branch with `require_code_owner_approval` enabled, over HTTPS + +```mermaid +graph TD + Repositories::GitHttpController --> Gitlab::GlRepository + Gitlab::GlRepository --> Gitlab::GitAccessProject + Gitlab::GitAccessProject --> Gitlab::Checks::DiffCheck + Gitlab::Checks::DiffCheck --> Gitlab::CodeOwners::Validator + Gitlab::CodeOwners::Validator --> ProtectedBranch + Gitlab::CodeOwners::Validator --> Gitlab::CodeOwners::Loader + Gitlab::CodeOwners::Loader --> Gitlab::CodeOwners::Entry +``` + ### Sync code owner rules to merge request approval rules ```mermaid diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 84d2537d058..cad71d4b843 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -376,7 +376,7 @@ Avoid: [_explain why, not what_](https://blog.codinghorror.com/code-tells-you-how-comments-tell-you-why/). - Requesting maintainer reviews of merge requests with failed tests. If the tests are failing and you have to request a review, ensure you leave a comment with an explanation. - Excessively mentioning maintainers through email or Slack (if the maintainer is reachable -through Slack). If you can't add a reviewer for a merge request, `@` mentioning a maintainer in a comment is acceptable and in all other cases adding a reviewer is sufficient. + through Slack). If you can't add a reviewer for a merge request, `@` mentioning a maintainer in a comment is acceptable and in all other cases adding a reviewer is sufficient. This saves reviewers time and helps authors catch mistakes earlier. @@ -412,7 +412,7 @@ that it meets all requirements, you should: - Select **Approve**. - `@` mention the author to generate a to-do notification, and advise them that their merge request has been reviewed and approved. - Request a review from a maintainer. Default to requests for a maintainer with [domain expertise](#domain-experts), -however, if one isn't available or you think the merge request doesn't need a review by a [domain expert](#domain-experts), feel free to follow the [Reviewer roulette](#reviewer-roulette) suggestion. + however, if one isn't available or you think the merge request doesn't need a review by a [domain expert](#domain-experts), feel free to follow the [Reviewer roulette](#reviewer-roulette) suggestion. - Remove yourself as a reviewer. ### The responsibility of the maintainer @@ -580,7 +580,7 @@ experience, refactors the existing code). Then: optionally resolve within the merge request or follow-up at a later stage. - There's a [Chrome/Firefox add-on](https://gitlab.com/conventionalcomments/conventional-comments-button) which you can use to apply [Conventional Comment](https://conventionalcomments.org/) prefixes. - Ensure there are no open dependencies. Check [linked issues](../user/project/issues/related_issues.md) for blockers. Clarify with the authors -if necessary. If blocked by one or more open MRs, set an [MR dependency](../user/project/merge_requests/dependencies.md). + if necessary. If blocked by one or more open MRs, set an [MR dependency](../user/project/merge_requests/dependencies.md). - After a round of line notes, it can be helpful to post a summary note such as "Looks good to me", or "Just a couple things to address." - Let the author know if changes are required following your review. diff --git a/doc/development/code_suggestions/index.md b/doc/development/code_suggestions/index.md index bdf3bcdd520..2bf36664437 100644 --- a/doc/development/code_suggestions/index.md +++ b/doc/development/code_suggestions/index.md @@ -12,7 +12,7 @@ The recommended setup for locally developing and debugging Code Suggestions is t - IDE Extension (e.g. VS Code Extension) - Main application configured correctly -- [Model gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist) +- [AI Gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist) This should enable everyone to see locally any change in an IDE being sent to the main application transformed to a prompt which is then sent to the respective model. @@ -24,26 +24,26 @@ This should enable everyone to see locally any change in an IDE being sent to th 1. Open the extension settings by clicking a small cog icon and select "Extension Settings" option 1. Check a "GitLab: Debug" checkbox. 1. Main Application - 1. Enable Feature Flags ```code_suggestions_completion_api``` and ```code_suggestions_tokens_api``` + 1. Enable Feature Flag ```code_suggestions_tokens_api``` 1. In your terminal, navigate to a `gitlab` inside your `gitlab-development-kit` directory 1. Run `bundle exec rails c` to start a Rails console - 1. Call `Feature.enable(:code_suggestions_completion_api)` and `Feature.enable(:code_suggestions_tokens_api)` from the console + 1. Call `Feature.enable(:code_suggestions_tokens_api)` from the console 1. Run the GDK with ```export CODE_SUGGESTIONS_BASE_URL=http://localhost:5052``` -1. [Setup Model Gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#how-to-run-the-server-locally) +1. [Setup AI Gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#how-to-run-the-server-locally) 1. Build tree sitter libraries ```poetry run scripts/build-tree-sitter-lib.py``` - 1. Extra .env Changes for all debugging insights - 1. LOG_LEVEL=DEBUG - 1. LOG_FORMAT_JSON=false - 1. LOG_TO_FILE=true + 1. Extra .env changes for all debugging insights + 1. `AIGW_LOGGING__LEVEL=DEBUG` + 1. `AIGW_LOGGING__FORMAT_JSON=false` + 1. `AIGW_LOGGING__TO_FILE=true` 1. Watch the new log file ```modelgateway_debug.log``` , e.g. ```tail -f modelgateway_debug.log | fblog -a prefix -a suffix -a current_file_name -a suggestion -a language -a input -a parameters -a score -a exception``` -### Setup instructions to use staging Model Gateway +### Setup instructions to use staging AI Gateway -When testing interactions with the Model Gateway, you might want to integrate your local GDK -with the deployed staging Model Gateway. To do this: +When testing interactions with the AI Gateway, you might want to integrate your local GDK +with the deployed staging AI Gateway. To do this: 1. You need a [cloud staging license](../../user/project/repository/code_suggestions/self_managed.md#upgrade-gitlab) that has the Code Suggestions add-on, because add-ons are enabled on staging. Drop a note in the `#s_fulfillment` internal Slack channel to request an add-on to your license. See this [handbook page](https://about.gitlab.com/handbook/developer-onboarding/#working-on-gitlab-ee-developer-licenses) for how to request a license for local development. -1. Set environment variables to point customers-dot to staging, and the Model Gateway to staging: +1. Set environment variables to point customers-dot to staging, and the AI Gateway to staging: ```shell export GITLAB_LICENSE_MODE=test @@ -52,5 +52,5 @@ with the deployed staging Model Gateway. To do this: ``` 1. Restart the GDK. -1. Ensure you followed the necessary [steps to enable the Code Suggestions feature](../../user/project/repository/code_suggestions/self_managed.md#gitlab-163-and-later). +1. Ensure you followed the necessary [steps to enable the Code Suggestions feature](../../user/project/repository/code_suggestions/self_managed.md). 1. Test out the Code Suggestions feature by opening the Web IDE for a project. diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index 0e922550856..c6027e27310 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -19,12 +19,12 @@ with additions and improvements. As a merge request (MR) author, you must: - Include _Before_ and _After_ -screenshots (or videos) of your changes in the description, as explained in our -[MR workflow](merge_request_workflow.md). These screenshots/videos are very helpful -for all reviewers and can speed up the review process, especially if the changes -are small. + screenshots (or videos) of your changes in the description, as explained in our + [MR workflow](merge_request_workflow.md). These screenshots/videos are very helpful + for all reviewers and can speed up the review process, especially if the changes + are small. - Attach the ~UX label to any merge request that has any user facing changes. This will trigger our -Reviewer Roulette to suggest a UX [reviewer](https://about.gitlab.com/handbook/product/ux/product-designer/mr-reviews/#stage-group-mrs). + Reviewer Roulette to suggest a UX [reviewer](https://about.gitlab.com/handbook/product/ux/product-designer/mr-reviews/#stage-group-mrs). If you are a **team member**: We recommend assigning the Product Designer suggested by the [Reviewer Roulette](../code_review.md#reviewer-roulette) as reviewer. [This helps us](https://about.gitlab.com/handbook/product/ux/product-designer/mr-reviews/#benefits) spread work evenly, improve communication, and make our UI more @@ -41,10 +41,10 @@ Check these aspects both when _designing_ and _reviewing_ UI changes. ### Writing -- Follow [Pajamas](https://design.gitlab.com/content/punctuation/) as the primary +- Follow [Pajamas](https://design.gitlab.com/content/ui-text/) as the primary guidelines for UI text and [documentation style guide](../documentation/styleguide/index.md) as the secondary. -- Use clear and consistent [terminology](https://design.gitlab.com/content/terminology/). +- Use clear and consistent terminology. - Check grammar and spelling. - Consider help content and follow its [guidelines](https://design.gitlab.com/usability/contextual-help). - Request review from the [appropriate Technical Writer](https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments), diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 2c8b5b2af20..d4541558d23 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -134,6 +134,14 @@ Lastly, keep the following in mind when submitting merge requests: be merged, as well as some guidance. The maintainers will be open to discussion about how to change the code so it can be approved and merged in the future. +## Tips + +- [Small MRs are the main key to a great review](https://about.gitlab.com/blog/2021/03/18/iteration-and-code-review/). +- Make sure to read our [merge request guidelines for contributors before you start for the first time](merge_request_workflow.md#merge-request-guidelines-for-contributors). +- Automated testing is required. Take your time to understand the different [testing levels](../testing_guide/testing_levels.md#how-to-test-at-the-correct-level) and apply them accordingly. +- Make sure to have a great description that includes steps to reproduce your implementation. +- [Make sure to follow our commit message guidelines](merge_request_workflow.md#commit-messages-guidelines). + ## Closing policy for issues and merge requests - For the criteria for closing issues, see [the Issue Triage handbook page](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#outdated-issues). diff --git a/doc/development/contributing/verify/index.md b/doc/development/contributing/verify/index.md index dbc48121dff..5f9e2e3acb3 100644 --- a/doc/development/contributing/verify/index.md +++ b/doc/development/contributing/verify/index.md @@ -53,7 +53,7 @@ and they serve us and our users well. Some examples of these principles are that If a job fails and we notify a user that it was successful, it can have severe negative consequences. - Feedback needs to be available when a user needs it and data cannot disappear unexpectedly when engineers need it. - It all doesn't matter if the platform is not secure and we -are leaking credentials or secrets. + are leaking credentials or secrets. - When a user provides a set of preconditions in a form of CI/CD configuration, the result should be deterministic each time a pipeline runs, because otherwise the platform might not be trustworthy. - If it is fast, simple to use and has a great UX it will serve our users well. diff --git a/doc/development/database/avoiding_downtime_in_migrations.md b/doc/development/database/avoiding_downtime_in_migrations.md index bf75b7c2ab2..57c71cbaec6 100644 --- a/doc/development/database/avoiding_downtime_in_migrations.md +++ b/doc/development/database/avoiding_downtime_in_migrations.md @@ -588,27 +588,27 @@ swap the columns. Swapping is done with post-deployment migration. The exact pro table being converted, but in general it's done in the following steps: 1. Using the provided `ensure_batched_background_migration_is_finished` helper, make sure the batched -migration has finished ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L13-18)). -If the migration has not completed, the subsequent steps fail anyway. By checking in advance we -aim to have more helpful error message. + migration has finished ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L13-18)). + If the migration has not completed, the subsequent steps fail anyway. By checking in advance we + aim to have more helpful error message. 1. Use the `add_bigint_column_indexes` helper method from `Gitlab::Database::MigrationHelpers::ConvertToBigint` module to create indexes with the `bigint` columns that match the existing indexes using the `integer` column. - The helper method is expected to create all required `bigint` indexes, but it's advised to recheck to make sure we are not missing any of the existing indexes. More information about the helper can be found in merge request [135781](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135781). 1. Create foreign keys (FK) using the `bigint` columns that match the existing FK using the -`integer` column. Do this both for FK referencing other tables, and FK that reference the table -that is being migrated ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L36-43)). + `integer` column. Do this both for FK referencing other tables, and FK that reference the table + that is being migrated ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L36-43)). 1. Inside a transaction, swap the columns: - 1. Lock the tables involved. To reduce the chance of hitting a deadlock, we recommended to do this in parent to child order ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L47)). - 1. Rename the columns to swap names ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L49-54)) - 1. Reset the trigger function ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L56-57)). - 1. Swap the defaults ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L59-62)). - 1. Swap the PK constraint (if any) ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L64-68)). - 1. Remove old indexes and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L70-72)). - - Names of the `bigint` indexes created using `add_bigint_column_indexes` helper can be retrieved by calling - `bigint_index_name` from `Gitlab::Database::MigrationHelpers::ConvertToBigint` module. - 1. Remove old foreign keys (if still present) and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L74)). + 1. Lock the tables involved. To reduce the chance of hitting a deadlock, we recommended to do this in parent to child order ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L47)). + 1. Rename the columns to swap names ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L49-54)) + 1. Reset the trigger function ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L56-57)). + 1. Swap the defaults ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L59-62)). + 1. Swap the PK constraint (if any) ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L64-68)). + 1. Remove old indexes and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L70-72)). + - Names of the `bigint` indexes created using `add_bigint_column_indexes` helper can be retrieved by calling + `bigint_index_name` from `Gitlab::Database::MigrationHelpers::ConvertToBigint` module. + 1. Remove old foreign keys (if still present) and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L74)). See example [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66088), and [migration](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb). diff --git a/doc/development/database/clickhouse/clickhouse_within_gitlab.md b/doc/development/database/clickhouse/clickhouse_within_gitlab.md index a459f89b185..8988ae182f7 100644 --- a/doc/development/database/clickhouse/clickhouse_within_gitlab.md +++ b/doc/development/database/clickhouse/clickhouse_within_gitlab.md @@ -272,3 +272,7 @@ end ``` Additionally, to view the executed ClickHouse queries in web interactions, on the performance bar, next to the `ch` label select the count. + +### Getting help + +For additional information or specific questions, please reach out to the ClickHouse Datastore working group in the `#f_clickhouse` Slack channel, or mention `@gitlab-org/maintainers/clickhouse` in a comment on GitLab.com. diff --git a/doc/development/database/clickhouse/merge_request_analytics.md b/doc/development/database/clickhouse/merge_request_analytics.md index db441ca9051..aa04d9a4fe1 100644 --- a/doc/development/database/clickhouse/merge_request_analytics.md +++ b/doc/development/database/clickhouse/merge_request_analytics.md @@ -100,7 +100,7 @@ LIMIT 1 ## Store merge request data in ClickHouse Several other use cases exist for storing and querying merge request data in -ClickHouse. In this document, we focus on this particular feature. +[ClickHouse](../../../integration/clickhouse.md). In this document, we focus on this particular feature. The core data exists in the `merge_request_metrics` and in the `merge_requests` database tables. Some filters require extra tables to be joined: diff --git a/doc/development/database/efficient_in_operator_queries.md b/doc/development/database/efficient_in_operator_queries.md index 8e334e5bb5c..e1a7b56e35e 100644 --- a/doc/development/database/efficient_in_operator_queries.md +++ b/doc/development/database/efficient_in_operator_queries.md @@ -916,16 +916,16 @@ Here's an outline of the steps we take in the recursive CTE query 1. Sort the initial `resultset` according to the `ORDER BY` clause. 1. Pick the top cursor to fetch the record, this is our first record. In the example, -this cursor would be (`2020-01-05`, `3`) for `project_id=9`. + this cursor would be (`2020-01-05`, `3`) for `project_id=9`. 1. We can use (`2020-01-05`, `3`) to fetch the next issue respecting the `ORDER BY` clause -`project_id=9` filter. This produces an updated `resultset`. + `project_id=9` filter. This produces an updated `resultset`. - | `project_ids` | `created_at_values` | `id_values` | - | ------------- | ------------------- | ----------- | - | 2 | 2020-01-10 | 5 | - | 5 | 2020-01-05 | 4 | - | 10 | 2020-01-15 | 7 | - | **9** | **2020-01-06** | **6** | + | `project_ids` | `created_at_values` | `id_values` | + | ------------- | ------------------- | ----------- | + | 2 | 2020-01-10 | 5 | + | 5 | 2020-01-05 | 4 | + | 10 | 2020-01-15 | 7 | + | **9** | **2020-01-06** | **6** | 1. Repeat 1 to 3 with the updated `resultset` until we have fetched `N=20` records. @@ -944,9 +944,9 @@ Example initializer row: | `NULL::issues` | `[9, 2, 5, 10]` | `[...]` | `[...]` | `0` | `NULL` | - The `records` column contains our sorted database records, and the initializer query sets the -first value to `NULL`, which is filtered out later. + first value to `NULL`, which is filtered out later. - The `count` column tracks the number of records found. We use this column to filter out the -initializer row from the result set. + initializer row from the result set. ### Recursive portion of the CTE query @@ -1016,7 +1016,7 @@ As the final step, we need to produce a new row by manipulating the initializer (`data_collector_query` method). Two things happen here: - Read the full row from the DB and return it in the `records` columns. (`result_collector_columns` -method) + method) - Replace the cursor values at the current position with the results from the keyset query. Reading the full row from the database is only one index scan by the primary key. We use the @@ -1091,7 +1091,7 @@ in the PostgreSQL's buffer cache. The optimized `IN` query reads maximum 519 entries (cursor values) from the index: - 500 index-only scans for populating the arrays for each project. The cursor values of the first -record is here. + record is here. - Maximum 19 additional index-only scans for the consecutive records. The optimized `IN` query sorts the array (cursor values per project array) 20 times, which diff --git a/doc/development/database/iterating_tables_in_batches.md b/doc/development/database/iterating_tables_in_batches.md index 802b7622592..7192a84508e 100644 --- a/doc/development/database/iterating_tables_in_batches.md +++ b/doc/development/database/iterating_tables_in_batches.md @@ -414,12 +414,12 @@ the queries, where the `issues` table is batched first. When to use `JOINS`: - When there's a 1:1 or 1:N relationship between the tables where we know that the joined record -(almost) always exists. This works well for "extension-like" tables: + (almost) always exists. This works well for "extension-like" tables: - `projects` - `project_settings` - `users` - `user_details` - `users` - `user_statuses` - `LEFT JOIN` works well in this case. Conditions on the joined table need to go to the yielded -relation so the iteration is not affected by the data distribution in the joined table. + relation so the iteration is not affected by the data distribution in the joined table. Example: @@ -555,7 +555,7 @@ table or a range of rows. The scaling and performance characteristics are very s Examples: - Iterate over the table in a specific order (timestamp columns) in combination with a tie-breaker -if column user to sort by does not contain unique values. + if column user to sort by does not contain unique values. - Iterate over the table with composite primary keys. ### Iterate over the issues in a project by creation date @@ -619,8 +619,8 @@ To keep the iteration stable and predictable, avoid updating the columns in the ### Iterate over the `merge_request_diff_commits` table -The `merge_request_diff_commits` table uses a composite primary key (`merge_request_diff_id, -relative_order`), which makes `EachBatch` impossible to use efficiently. +The `merge_request_diff_commits` table uses a composite primary key (`merge_request_diff_id, relative_order`), +which makes `EachBatch` impossible to use efficiently. To paginate over the `merge_request_diff_commits` table, you can use the following snippet: @@ -654,7 +654,7 @@ end ### Order object configuration Keyset pagination works well with simple `ActiveRecord` `order` scopes -([first example](#iterate-over-the-issues-in-a-project-by-creation-date). +([first example](#iterate-over-the-issues-in-a-project-by-creation-date)). However, in special cases, you need to describe the columns in the `ORDER BY` clause (second example) for the underlying keyset pagination library. When the `ORDER BY` configuration cannot be automatically determined by the keyset pagination library, an error is raised. diff --git a/doc/development/database/keyset_pagination.md b/doc/development/database/keyset_pagination.md index 9bb44d2ed71..6b2c4f9d146 100644 --- a/doc/development/database/keyset_pagination.md +++ b/doc/development/database/keyset_pagination.md @@ -123,7 +123,7 @@ For the REST API, the `paginate_with_strategies` helper can be used on a relatio In order for keyset pagination to be used, the following conditions must be met: -1. `params[:keyset]` must return `'keyset'` +1. `params[:pagination]` must return `'keyset'` 1. `params[:order_by]` and `params[:sort]` must both appear in the object returned by the `supported_keyset_orderings` class method on the model. In the following example, `Thing` supports keyset pagination when ordering by ID in either ascending or descending order. diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index 3003ee970ce..9ecc70c06bc 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -499,7 +499,7 @@ cron job where the schedule depends on the configuration of the GitLab instance. - Non-decomposed GitLab (1 database): invoked every minute. - Decomposed GitLab (2 databases, CI and Main): invoked every minute, cleaning up one database -at a time. For example, the cleanup worker for the main database runs every two minutes. + at a time. For example, the cleanup worker for the main database runs every two minutes. To avoid lock contention and the processing of the same database rows, the worker does not run parallel. This behavior is ensured with a Redis lock. @@ -509,13 +509,13 @@ parallel. This behavior is ensured with a Redis lock. 1. Acquire the Redis lock. 1. Determine which database to clean up. 1. Collect all database tables where the deletions are tracked (parent tables). - - This is achieved by reading the `config/gitlab_loose_foreign_keys.yml` file. - - A table is considered "tracked" when a loose foreign key definition exists for the table and - the `DELETE` trigger is installed. + - This is achieved by reading the `config/gitlab_loose_foreign_keys.yml` file. + - A table is considered "tracked" when a loose foreign key definition exists for the table and + the `DELETE` trigger is installed. 1. Cycle through the tables with an infinite loop. 1. For each table, load a batch of deleted parent records to clean up. 1. Depending on the YAML configuration, build `DELETE` or `UPDATE` (nullify) queries for the -referenced child tables. + referenced child tables. 1. Invoke the queries. 1. Repeat until all child records are cleaned up or the maximum limit is reached. 1. Remove the deleted parent records when all child records are cleaned up. @@ -530,13 +530,13 @@ The inserted record stores the following information about the deleted record: - `fully_qualified_table_name`: name of the database table where the record was located. - `primary_key_value`: the ID of the record, the value is present in the child tables as -the foreign key value. At the moment, composite primary keys are not supported, the parent table -must have an `id` column. + the foreign key value. At the moment, composite primary keys are not supported, the parent table + must have an `id` column. - `status`: defaults to pending, represents the status of the cleanup process. - `consume_after`: defaults to the current time. - `cleanup_attempts`: defaults to 0. The number of times the worker tried to clean up this record. -A non-zero number would mean that this record has many child records and cleaning it up requires -several runs. + A non-zero number would mean that this record has many child records and cleaning it up requires + several runs. #### Database decomposition @@ -692,7 +692,7 @@ The loop-based batch processing is preferred over `EachBatch` for the following - The records in the batch are modified, so the next batch contains different records. - There is always an index on the foreign key column however, the column is usually not unique. -`EachBatch` requires a unique column for the iteration. + `EachBatch` requires a unique column for the iteration. - The record order doesn't matter for the cleanup. Notice that we have two loops. The initial loop processes records with the `SKIP LOCKED` clause. @@ -747,23 +747,23 @@ For example, a project with millions of `ci_builds` records is deleted. The `ci_ is deleted by the loose foreign keys feature. 1. The cleanup worker is scheduled and picks up a batch of deleted `projects` records. The large -project is part of the batch. + project is part of the batch. 1. Deletion of the orphaned `ci_builds` rows has started. 1. The time limit is reached, but the cleanup is not complete. 1. The `cleanup_attempts` column is incremented for the deleted records. 1. Go to step 1. The next cleanup worker continues the cleanup. 1. When the `cleanup_attempts` reaches 3, the batch is re-scheduled 10 minutes later by updating -the `consume_after` column. + the `consume_after` column. 1. The next cleanup worker processes a different batch. We have Prometheus metrics in place to monitor the deleted record cleanup: - `loose_foreign_key_processed_deleted_records`: Number of processed deleted records. When large -cleanup happens, this number would decrease. + cleanup happens, this number would decrease. - `loose_foreign_key_incremented_deleted_records`: Number of deleted records which were not -finished processing. The `cleanup_attempts` column was incremented. + finished processing. The `cleanup_attempts` column was incremented. - `loose_foreign_key_rescheduled_deleted_records`: Number of deleted records that had to be -rescheduled at a later time after 3 cleanup attempts. + rescheduled at a later time after 3 cleanup attempts. Example Thanos query: diff --git a/doc/development/database/maintenance_operations.md b/doc/development/database/maintenance_operations.md index 2701c228b74..fcdf132aa09 100644 --- a/doc/development/database/maintenance_operations.md +++ b/doc/development/database/maintenance_operations.md @@ -18,7 +18,7 @@ There are certain situations in which you might want to disable an index before To disable an index before removing it: 1. Open a [production infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new) -and use the "Production Change" template. + and use the "Production Change" template. 1. Inform the database team in the issue `@gl-database` or in Slack `#database`. 1. Add a step to verify the index is used (this would likely be an `EXPLAIN` command known to use the index). 1. Add the step to disable the index: diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index e453a7f27ea..a8d088a372e 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -936,7 +936,7 @@ I, [2023-07-14T17:08:06.761709 #92505] INFO -- : SELECT set_config('lock_writes I, [2023-07-14T17:08:06.765272 #92505] INFO -- : SELECT set_config('lock_writes.ci_build_pending_states', 'false', false) I, [2023-07-14T17:08:06.768220 #92505] INFO -- : SELECT set_config('lock_writes.ci_build_report_results', 'false', false) [...] -I, [2023-07-14T17:08:06.957294 #92505] INFO -- : TRUNCATE TABLE ci_build_needs, ci_build_pending_states, ci_build_report_results, ci_build_trace_chunks, ci_build_trace_metadata, ci_builds, ci_builds_metadata, ci_builds_runner_session, ci_cost_settings, ci_daily_build_group_report_results, ci_deleted_objects, ci_editor_ai_conversation_messages, ci_freeze_periods, ci_group_variables, ci_instance_variables, ci_job_artifact_states, ci_job_artifacts, ci_job_token_project_scope_links, ci_job_variables, ci_minutes_additional_packs, ci_namespace_mirrors, ci_namespace_monthly_usages, ci_partitions, ci_pending_builds, ci_pipeline_artifacts, ci_pipeline_chat_data, ci_pipeline_messages, ci_pipeline_metadata, ci_pipeline_schedule_variables, ci_pipeline_schedules, ci_pipeline_variables, ci_pipelines, ci_pipelines_config, ci_platform_metrics, ci_project_mirrors, ci_project_monthly_usages, ci_refs, ci_resource_groups, ci_resources, ci_runner_machines, ci_runner_namespaces, ci_runner_projects, ci_runner_versions, ci_runners, ci_running_builds, ci_secure_file_states, ci_secure_files, ci_sources_pipelines, ci_sources_projects, ci_stages, ci_subscriptions_projects, ci_trigger_requests, ci_triggers, ci_unit_test_failures, ci_unit_tests, ci_variables, external_pull_requests, p_ci_builds, p_ci_builds_metadata, p_ci_job_annotations, p_ci_runner_machine_builds, taggings, tags RESTRICT +I, [2023-07-14T17:08:06.957294 #92505] INFO -- : TRUNCATE TABLE ci_build_needs, ci_build_pending_states, ci_build_report_results, ci_build_trace_chunks, ci_build_trace_metadata, ci_builds, ci_builds_metadata, ci_builds_runner_session, ci_cost_settings, ci_daily_build_group_report_results, ci_deleted_objects, ci_freeze_periods, ci_group_variables, ci_instance_variables, ci_job_artifact_states, ci_job_artifacts, ci_job_token_project_scope_links, ci_job_variables, ci_minutes_additional_packs, ci_namespace_mirrors, ci_namespace_monthly_usages, ci_partitions, ci_pending_builds, ci_pipeline_artifacts, ci_pipeline_chat_data, ci_pipeline_messages, ci_pipeline_metadata, ci_pipeline_schedule_variables, ci_pipeline_schedules, ci_pipeline_variables, ci_pipelines, ci_pipelines_config, ci_platform_metrics, ci_project_mirrors, ci_project_monthly_usages, ci_refs, ci_resource_groups, ci_resources, ci_runner_machines, ci_runner_namespaces, ci_runner_projects, ci_runner_versions, ci_runners, ci_running_builds, ci_secure_file_states, ci_secure_files, ci_sources_pipelines, ci_sources_projects, ci_stages, ci_subscriptions_projects, ci_trigger_requests, ci_triggers, ci_unit_test_failures, ci_unit_tests, ci_variables, external_pull_requests, p_ci_builds, p_ci_builds_metadata, p_ci_job_annotations, p_ci_runner_machine_builds, taggings, tags RESTRICT ``` The tasks will first find out the tables that need to be truncated. Truncation will diff --git a/doc/development/database/rename_database_tables.md b/doc/development/database/rename_database_tables.md index ad641797496..babd51f91a0 100644 --- a/doc/development/database/rename_database_tables.md +++ b/doc/development/database/rename_database_tables.md @@ -78,18 +78,18 @@ Execute a standard migration (not a post-migration): - Add a note in the Engineering Week-in-Review document: `table_name` is going to be renamed in N.M. Modifications to this table are not allowed in release N.M and N.M+1. - The helper method uses the standard `rename_table` helper from Rails for renaming the table. - The helper renames the sequence and the indexes. Sometimes it diverges from the standard Rails convention -when naming indexes, so there is a possibility that not all indexes are properly renamed. After running -the migration locally, check if there are inconsistently named indexes (`db/structure.sql`). Those can be -renamed manually in a separate migration, which can be also part of the release M.N+1. + when naming indexes, so there is a possibility that not all indexes are properly renamed. After running + the migration locally, check if there are inconsistently named indexes (`db/structure.sql`). Those can be + renamed manually in a separate migration, which can be also part of the release M.N+1. - Foreign key columns might still contain the old table name. For smaller tables, follow our [standard column rename process](avoiding_downtime_in_migrations.md#renaming-columns) - Avoid renaming database tables which are using with triggers. - Table modifications (add or remove columns) are not allowed during the rename process. Make sure that all changes to the table happen before the rename migration is started (or in the next release). - As the index names might change, verify that the model does not use bulk insert -(for example, `insert_all` and `upsert_all`) with the `unique_by: index_name` option. -Renaming an index while using these methods may break functionality. + (for example, `insert_all` and `upsert_all`) with the `unique_by: index_name` option. + Renaming an index while using these methods may break functionality. - Modify the model code to point to the new database table. Do this by -renaming the model directly or setting the `self.table_name` variable. + renaming the model directly or setting the `self.table_name` variable. At this point, we don't have applications using the old database table name in their queries. @@ -107,7 +107,7 @@ At this point, we don't have applications using the old database table name in t 1. Additionally the table definition from `TABLES_TO_BE_RENAMED` **must** be removed. -To do so, edit the `TABLES_TO_BE_RENAMED` constant in `lib/gitlab/database.rb`: + To do so, edit the `TABLES_TO_BE_RENAMED` constant in `lib/gitlab/database.rb`: From: diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index 70d59603e0c..50188dba14e 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -12,8 +12,8 @@ When adding new columns to store strings or other textual information: 1. We always use the `text` data type instead of the `string` data type. 1. `text` columns should always have a limit set, either by using the `create_table` with -the `#text ... limit: 100` helper (see below) when creating a table, or by using the `add_text_limit` -when altering an existing table. + the `#text ... limit: 100` helper (see below) when creating a table, or by using the `add_text_limit` + when altering an existing table. The standard Rails `text` column type cannot be defined with a limit, but we extend `create_table` to add a `limit: 255` option. Outside of `create_table`, `add_text_limit` can be used to add a [check constraint](https://www.postgresql.org/docs/11/ddl-constraints.html) diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md index 4d3101b40fb..cb159a404fd 100644 --- a/doc/development/database/table_partitioning.md +++ b/doc/development/database/table_partitioning.md @@ -19,15 +19,15 @@ table. If the application is designed to work with partitioning in mind, there can be multiple benefits, such as: - Query performance can be improved greatly, because the database can -cheaply eliminate much of the data from the search space, while still -providing full SQL capabilities. + cheaply eliminate much of the data from the search space, while still + providing full SQL capabilities. - Bulk deletes can be achieved with minimal impact on the database by -dropping entire partitions. This is a natural fit for features that need -to periodically delete data that falls outside the retention window. + dropping entire partitions. This is a natural fit for features that need + to periodically delete data that falls outside the retention window. - Administrative tasks like `VACUUM` and index rebuilds can operate on -individual partitions, rather than across a single massive table. + individual partitions, rather than across a single massive table. Unfortunately, not all models fit a partitioning scheme, and there are significant drawbacks if implemented incorrectly. Additionally, tables @@ -632,3 +632,73 @@ class AsyncPrepareTableConstraintsForListPartitioning < Gitlab::Database::Migrat end end ``` + +### Step 7 - Re-point foreign keys to parent table + +The tables that reference the initial partition must be updated to point to the +parent table now. Without this change, the records from those tables +will not be able to locate the rows in the next partitions because they will look +for them in the initial partition. + +Steps: + +- Add the foreign key to the partitioned table and validate it asynchronously, + [for example](https://gitlab.com/gitlab-org/gitlab/-/blob/65d63f6a00196c3a7d59f15191920f271ab2b145/db/post_migrate/20230524135543_replace_ci_build_pending_states_foreign_key.rb). +- Validate it synchronously after the asynchronously validation was completed on GitLab.com, + [for example](https://gitlab.com/gitlab-org/gitlab/-/blob/65d63f6a00196c3a7d59f15191920f271ab2b145/db/post_migrate/20230530140456_validate_fk_ci_build_pending_states_p_ci_builds.rb). +- Remove the old foreign key and rename the new one to the old name, + [for example](https://gitlab.com/gitlab-org/gitlab/-/blob/65d63f6a00196c3a7d59f15191920f271ab2b145/db/post_migrate/20230615083713_replace_old_fk_ci_build_pending_states_to_builds.rb#L9). + +### Step 8 - Ensure ID uniqueness across partitions + +All uniqueness constraints must include the partitioning key, so we can have +duplicate IDs across partitions. To solve this we enforce that only the database +can set the ID values and use a sequence to generate them because sequences are +guaranteed to generate unique values. + +For example: + +```ruby +class EnsureIdUniquenessForPCiBuilds < Gitlab::Database::Migration[2.1] + include Gitlab::Database::PartitioningMigrationHelpers::UniquenessHelpers + + enable_lock_retries! + + TABLE_NAME = :p_ci_builds + FUNCTION_NAME = :assign_p_ci_builds_id_value + + def up + ensure_unique_id(TABLE_NAME) + end + + def down + execute(<<~SQL.squish) + ALTER TABLE #{TABLE_NAME} + ALTER COLUMN id SET DEFAULT nextval('ci_builds_id_seq'::regclass); + + DROP FUNCTION IF EXISTS #{FUNCTION_NAME} CASCADE; + SQL + end +``` + +### Step 9 - Analyze the partitioned table and create new partitions + +The autovacuum daemon does not process partitioned tables. It is necessary to +periodically run a manual `ANALYZE` to keep the statistics of the table hierarchy +up to date. + +Models that implement `Ci::Partitionable` with `partitioned: true` option are +analyzed by default on a weekly basis. To enable this and create new partitions +you need to register the model in the [PostgreSQL initializer](https://gitlab.com/gitlab-org/gitlab/-/blob/b7f0e3f1bcd2ffc220768bbc373364151775ca8e/config/initializers/postgres_partitioning.rb). + +### Step 10 - Update the application to use the partitioned table + +Now that the parent table is ready, we can update the application to use it: + +```ruby +class Model < ApplicationRecord + self.table_name = :partitioned_table +end +``` + +Depending on the model, it might be safer to use a [change management issue](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/16387). diff --git a/doc/development/database_review.md b/doc/development/database_review.md index ec4c4f9f2c5..2bb2a6fc267 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -29,7 +29,7 @@ A database review is required for: These metrics could have complex queries over large tables. See the [Analytics Instrumentation Guide](https://about.gitlab.com/handbook/product/analytics-instrumentation-guide/) for implementation details. -- Changes that use [`update`, `delete`, `update_all`, `delete_all` or `destroy_all`](#preparation-when-using-update-delete-update_all-delete_all-or-destroy_all) +- Changes that use [`update`, `upsert`, `delete`, `update_all`, `upsert_all`, `delete_all` or `destroy_all`](#preparation-when-using-bulk-update-operations) methods on an ActiveRecord object. A database reviewer is expected to look out for overly complex @@ -113,7 +113,7 @@ the following preparations into account. #### Preparation when adding migrations - Ensure `db/structure.sql` is updated as [documented](migration_style_guide.md#schema-changes), and additionally ensure that the relevant version files under -`db/schema_migrations` were added or removed. + `db/schema_migrations` were added or removed. - Ensure that the Database Dictionary is updated as [documented](database/database_dictionary.md). - Make migrations reversible by using the `change` method or include a `down` method when using `up`. - Include either a rollback procedure or describe how to rollback changes. @@ -153,7 +153,7 @@ Include in the MR description: - Write the raw SQL in the MR description. Preferably formatted nicely with [pgFormatter](https://sqlformat.darold.net) or <https://paste.depesz.com> and using regular quotes - (for example, `"projects"."id"`) and avoiding smart quotes (for example, `"projects"."id"`). + (for example, `"projects"."id"`) and avoiding smart quotes (for example, `“projects”.“id”`). - In case of queries generated dynamically by using parameters, there should be one raw SQL query for each variation. For example, a finder for issues that may take as a parameter an optional filter on projects, @@ -227,9 +227,10 @@ Include in the MR description: - If you're adding a composite index, another index might become redundant, so remove that in the same migration. For example adding `index(column_A, column_B, column_C)` makes the indexes `index(column_A, column_B)` and `index(column_A)` redundant. -#### Preparation when using `update`, `delete`, `update_all`, `delete_all` or `destroy_all` +#### Preparation when using bulk update operations -Using these ActiveRecord methods requires extra care because they modify data and can perform poorly, or they +Using `update`, `upsert`, `delete`, `update_all`, `upsert_all`, `delete_all` or `destroy_all` +ActiveRecord methods requires extra care because they modify data and can perform poorly, or they can destroy data if improperly scoped. These methods are also [incompatible with Common Table Expression (CTE) statements](sql.md#when-to-use-common-table-expressions). Danger will comment on a Merge Request Diff when these methods are used. diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index 86732c3a8ac..ad41d21a9e7 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -93,9 +93,9 @@ concern, some instrumentations are disabled by default. To enable those disabled instrumentations, set the following environment variables: - `GITLAB_TRACING_TRACK_CACHES`: enable tracking cache operations, such as cache -read, write, or delete. + read, write, or delete. - `GITLAB_TRACING_TRACK_REDIS`: enable tracking Redis operations. Most Redis -operations are for caching, though. + operations are for caching, though. ## Using Jaeger in the GitLab Development Kit diff --git a/doc/development/documentation/alpha_beta.md b/doc/development/documentation/alpha_beta.md deleted file mode 100644 index 4579c57b448..00000000000 --- a/doc/development/documentation/alpha_beta.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'experiment_beta.md' -remove_date: '2023-11-29' ---- - -This document was moved to [another location](experiment_beta.md). - -<!-- This redirect file can be deleted after <2023-11-29>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/documentation/experiment_beta.md b/doc/development/documentation/experiment_beta.md index 85f6dc38621..96e8d6fb638 100644 --- a/doc/development/documentation/experiment_beta.md +++ b/doc/development/documentation/experiment_beta.md @@ -32,7 +32,7 @@ On self-managed GitLab, by default this feature is not available. To make it available, an administrator can enable the feature flag named `example_flag`. On GitLab.com, this feature is not available. This feature is not ready for production use. -Use this great new feature when you need to do this new thing. +Use this new feature when you need to do this new thing. This feature is an [Experiment](<link_to>/policy/experiment-beta-support.md). To join the list of users testing this feature, do this thing. If you find a bug, diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 617691e8628..637b2d163e7 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -17,7 +17,7 @@ When the state of a feature flag changes, the developer who made the change Every feature introduced to the codebase, even if it's behind a disabled feature flag, must be documented. For more information, see -[the discussion that led to this decision](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47917#note_459984428). [Experiment or Beta](../../policy/experiment-beta-support.md) features are usually behind a feature flag, and must also be documented. For more information, see [Document Experiment or Beta features](alpha_beta.md). +[the discussion that led to this decision](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47917#note_459984428). [Experiment or Beta](../../policy/experiment-beta-support.md) features are usually behind a feature flag, and must also be documented. For more information, see [Document Experiment or Beta features](experiment_beta.md). When the feature is [implemented in multiple merge requests](../feature_flags/index.md#feature-flags-in-gitlab-development), discuss the plan with your technical writer. diff --git a/doc/development/documentation/metadata.md b/doc/development/documentation/metadata.md index 4e80ea154b3..e6eff56d66d 100644 --- a/doc/development/documentation/metadata.md +++ b/doc/development/documentation/metadata.md @@ -32,11 +32,48 @@ To populate the metadata, include this information: - `info`: How to find the Technical Writer associated with the page's stage and group. +### Exceptions + +Documents in the `/development` directory get this metadata: + +```yaml +--- +stage: Example Stage +group: Example Group +info: Any user with at least the Maintainer role can merge updates to this content. For details, see https://docs.gitlab.com/ee/development/development_processes.html#development-guidelines-review. +--- +``` + +Documents in the `/solutions` directory get this metadata: + +```yaml +--- +stage: Solutions Architecture +group: Solutions Architecture +info: This page is owned by the Solutions Architecture team. +--- +``` + +## Description metadata + +The `description` tag: + +- Is used to populate text on the docs home page. +- Is shown in social media previews. +- Can be used in search result snippets. + +For the top-level pages, like **Use GitLab** and one level underneath, +the descriptions are lists of nouns. For example, for **Set up your organization**, +the description is `Users, groups, namespaces, SSH keys.` + +For other pages, descriptions are not actively maintained. However, if you want to add one, +use a short description of what the page is about. +See the Google [Best practices for creating quality meta descriptions](https://developers.google.com/search/docs/appearance/snippet#meta-descriptions) for tips. + ## Additional metadata The following metadata is optional and is not actively maintained. -- `description`: A short description of what the page is about. See the Google [Best practices for creating quality meta descriptions](https://developers.google.com/search/docs/appearance/snippet#meta-descriptions) for writing tips. This content can be used in search result snippets and is shown in social media previews. - `feedback`: Set to `false` to not include the "Help & Feedback" footer. - `noindex`: Set to `false` to prevent the page from being indexed by search engines. - `redirect_to`: Used to control redirects. For more information, see [Redirects in GitLab documentation](redirects.md). diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index 0dfe538cfbd..be10688766f 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -26,6 +26,11 @@ For example: While some older sections of the nav are alphabetical, the nav should primarily be workflow-based. +Without a navigation entry: + +- The navigation closes when the page is opened, and the reader loses their place. +- The page isn't visible in a group with other pages. + ## Choose the right words for your navigation entry Before you add an item to the left nav, choose the parts of speech you want to use. @@ -41,35 +46,14 @@ as helpful as **Get started with runners**. ## Add a navigation entry -To add a topic to the global nav, edit -[`navigation.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/content/_data/navigation.yaml) -and add your item. - -Without a navigation entry: - -- The navigation closes when the page is opened, and the reader loses their place. -- The page isn't visible in a group with other pages. - -### Pages you don't need to add - -Exclude these pages from the global nav: - -- Legal notices. -- Pages in the `architecture/blueprints` directory. -- Pages in the `user/application_security/dast/checks/` directory. - -The following pages should probably be in the global nav, but the technical writers -do not actively work to add them: - -- Pages in the `/development` directory. -- Pages authored by the support team, which are under the `doc/administration/troubleshooting` directory. +**Do not** add items to the global nav without +the consent of one of the technical writers. -Sometimes pages for deprecated features are not in the global nav, depending on how long ago the feature was deprecated. +To add a topic to the global navigation: -All other pages should be in the global nav. - -The technical writing team runs a report to determine which pages are not in the nav. -The team reviews this list each month. +1. In the [`navigation.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/content/_data/navigation.yaml) + file, add the item. +1. Assign the MR to a technical writer for review and merge. ### Where to add @@ -110,7 +94,28 @@ mechanics of what is required is [documented below](#data-file) but, in principl substitution for **Continuous Integration**. - Navigation links must follow the rules documented in the [data file](#data-file). -## How it works +### Pages you don't need to add + +Exclude these pages from the global nav: + +- Legal notices. +- Pages in the `architecture/blueprints` directory. +- Pages in the `user/application_security/dast/checks/` directory. + +The following pages should probably be in the global nav, but the technical writers +do not actively work to add them: + +- Pages in the `/development` directory. +- Pages authored by the support team, which are under the `doc/administration/troubleshooting` directory. + +Sometimes pages for deprecated features are not in the global nav, depending on how long ago the feature was deprecated. + +All other pages should be in the global nav. + +The technical writing team runs a report to determine which pages are not in the nav. +The team reviews this list each month. + +## Navigation structure The global nav has five levels: @@ -122,9 +127,6 @@ The global nav has five levels: You can view this structure in [the `navigation.yml` file](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/navigation.yaml). -**Do not** [add items](#add-a-navigation-entry) to the global nav without -the consent of one of the technical writers. - ## Composition The global nav is built from two files: diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 26660d2eba1..a18b376a1cc 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -243,7 +243,7 @@ by default. Capitalize names of: - GitLab [product tiers](https://about.gitlab.com/pricing/). For example, - GitLab Free and GitLab Ultimate. (Tested in [`BadgeCapitalization.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/BadgeCapitalization.yml).) + GitLab Free and GitLab Ultimate. - Third-party organizations, software, and products. For example, Prometheus, Kubernetes, Git, and The Linux Foundation. - Methods or methodologies. For example, Continuous Integration, @@ -351,10 +351,10 @@ For numbers in text, spell out zero through nine and use numbers for 10 and grea - [Write in Markdown](#markdown). - Insert an empty line for new paragraphs. - Insert an empty line between different markups (for example, after every - paragraph, header, list, and so on). Example: + paragraph, heading, list, and so on). Example: ```markdown - ## Header + ## Heading Paragraph. @@ -692,9 +692,9 @@ Markdown tables naturally fall out of alignment over time, but still render corr on `docs.gitlab.com`. The technical writing team can realign cells the next time the page is refactored. -### Table headings +### Table headers -Use sentence case for table headings. For example, `Keyword value` or `Project name`. +Use sentence case for table headers. For example, `Keyword value` or `Project name`. ### Feature tables @@ -1140,7 +1140,7 @@ When you take screenshots: Reduce the size of your browser window as much as possible to keep elements close together and reduce empty space. Try to keep the screenshot dimensions as small as possible. - **Review how the image renders on the page.** Preview the image locally or use the -review app in the merge request. Make sure the image isn't blurry or overwhelming. + review app in the merge request. Make sure the image isn't blurry or overwhelming. - **Be consistent.** Coordinate screenshots with the other screenshots already on a documentation page for a consistent reading experience. Ensure your navigation theme is **Indigo** and the syntax highlighting theme is **Light**. These are the default preferences. @@ -1237,8 +1237,7 @@ and annoying for users. If you're describing a complicated interaction in the user interface and want to include a visual representation to help readers understand it, you can: -- Use a static image (screenshot) and if necessary, add callouts to emphasize an - an area of the screen. +- Use a static image (screenshot) and if necessary, add callouts to emphasize an area of the screen. - Create a short video of the interaction and link to it. ### Automatic screenshot generator @@ -1349,12 +1348,14 @@ Do not upload videos to the product repositories. [Link](#link-to-video) or ### Link to video -To link out to a video, include a YouTube icon so that readers can scan the page -for videos before reading: +To link to a video, include a YouTube icon so that readers can scan the page +for videos before reading. Include the video's publication date after the link, to help identify +videos that might be out-of-date. ```markdown <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Video Title](link-to-video). +<!-- Video published on YYYY-MM-DD --> ``` You can link any up-to-date video that's useful to the GitLab user. @@ -1384,6 +1385,8 @@ To embed a video: (`https://www.youtube-nocookie.com/embed/VIDEO-ID`), and paste it, replacing the content of the `src` field in the `iframe` tag. +1. Include the video's publication date below the link, to help identify + videos that might be out-of-date. ```html leave a blank line here @@ -1393,6 +1396,7 @@ leave a blank line here <figure class="video-container"> <iframe src="https://www.youtube-nocookie.com/embed/MqL6BMOySIQ" frameborder="0" allowfullscreen> </iframe> </figure> +<!-- Video published on YYYY-MM-DD --> leave a blank line here ``` @@ -1434,7 +1438,7 @@ NOTE: This is something to note. ``` -To display an alert box for multiple paragraphs, lists, or headers, use +To display an alert box for multiple paragraphs, lists, or headings, use [blockquotes](#blockquotes) instead. Alert boxes render only on the GitLab documentation site (<https://docs.gitlab.com>). @@ -1636,13 +1640,37 @@ When names change, it is more complicated to search or grep text that has line b Tier badges provide information about a feature and are displayed next to the topic title. -You should assign a tier badge: +#### When to add tier badges -- To all H1 topic titles, except the pages under `doc/development/*`. -- To topic titles that don't apply to the same tier as the H1. +Assign tier badges to: + +- All H1 topic titles, except the pages under `doc/development/*` and `doc/solutions/*`. +- Topic titles that don't apply to the same tier as the H1. The H1 tier badge should be the badge that applies to the lowest tier for the features on the page. +#### When not to add tier badges + +Do not assign tier badges: + +- When a feature does not have one obvious subscription tier or offering. + For example, if a feature applies to one tier for SaaS and a different tier for self-managed. + +In this case, do any or all of the following: + +- Use a `NOTE` in an alert box to describe the tiers. +- Add tier badges to other topic titles where this information makes more sense. +- Do not add tier badges to the H1. + +##### Pages that don't need a tier badge + +Some pages won't have a tier badge, because no obvious tier badge applies. For example: + +- Tutorials. +- Pages that compare features from different tiers. +- Pages in the `/development` folder. These pages are automatically assigned a `Contribute` badge. +- Pages in the `/solutions` folder. These pages are automatically assigned a `Solutions` badge. + #### Available product tier badges Tier badges should include two components, in this order: a subscription tier and an offering. @@ -1667,7 +1695,9 @@ You can also add a third component for the feature's status: For example, `**(FREE ALL EXPERIMENT)**`. -A tier or status can stand alone. An offering should always have a tier. +- A tier or status can stand alone. +- An offering should always have a tier. +- Do not add more than one offering, tier, or status. Multiples do not render properly in the documentation. #### Add a tier badge @@ -1697,15 +1727,6 @@ Do not add tier badges inline with other text, except for [API attributes](../re The single source of truth for a feature should be the topic where the functionality is described. -##### Pages that don't need a tier badge - -Some pages won't have a tier badge, because no obvious tier badge applies. For example: - -- Tutorials. -- Pages that compare features from different tiers. -- Pages in the `/development` folder. These pages are automatically assigned a `Contribute` badge. -- Pages in the `/solutions` folder. These pages are automatically assigned a `Solutions` badge. - ##### Administrator documentation tier badges Topics that are only for instance administrators should be badged `<TIER> SELF`. Instance diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index 9d0d59e27c5..fed295b8ec9 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -130,6 +130,10 @@ The token generated when you create an agent for Kubernetes. Use **agent access Use **AI**. Do not spell out **artificial intelligence**. +## AI-powered DevSecOps platform + +If preceded by GitLab, capitalize **Platform**. For example, the GitLab AI-powered DevSecOps Platform. + ## air gap, air-gapped Use **offline environment** to describe installations that have physical barriers or security policies that prevent or limit internet access. Do not use **air gap**, **air gapped**, or **air-gapped**. For example: @@ -239,6 +243,22 @@ Use **text box** to refer to the UI field. Do not use **field** or **box**. For - In the **Variable name** text box, enter a value. +## branch + +Use **branch** by itself to describe a branch. For specific branches, use these terms only: + +- **default branch**: The primary branch in the repository. Users can use the UI to set the default + branch. For examples that use the default branch, use `main` instead of [`master`](#master). +- **source branch**: The branch you're merging from. +- **target branch**: The branch you're merging to. +- **current branch**: The branch you have checked out. + This branch might be the default branch, a branch you've created, a source branch, or some other branch. + +Do not use the terms **feature branch** or **merge request branch**. Be as specific as possible. For example: + +- The branch you have checked out... +- The branch you added commits to... + ## bullet Don't refer to individual items in an ordered or unordered list as **bullets**. Use **list item** instead. If you need to be less ambiguous, you can use: @@ -428,13 +448,6 @@ Instead of: - Data are collected. - The data show a performance increase. -## default branch - -Use **default branch** to refer generically to the primary branch in the repository. -Users can set the default branch by using a UI setting. - -For examples that use the default branch, use `main` instead of [`master`](#master). - ## delete Use **delete** when an object is completely deleted. **Delete** is the opposite of **create**. @@ -464,6 +477,10 @@ When writing about the Developer role: Do not use **Developer permissions**. A user who is assigned the Developer role has a set of associated permissions. +## DevSecOps platform + +If preceded by GitLab, capitalize **Platform**. For example, the GitLab DevSecOps Platform. + ## dialog Use **dialog** rather than any of these alternatives: @@ -661,6 +678,10 @@ Instead of: - Use the merge request feature to incorporate changes into the target branch. +## feature branch + +Do not use **feature branch**. See [branch](#branch). + ## field Use **text box** instead of **field** or **box**. @@ -1083,7 +1104,7 @@ Do not use **manpower**. Use words like **workforce** or **GitLab team members** ## master -Do not use `master`. Use `main` when you need a sample [default branch name](#default-branch). +Do not use **master**. Use **main** when you need a sample [default branch name](#branch). ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) ## may, might @@ -1115,6 +1136,10 @@ For **MB** and **GB**, follow the [Microsoft guidance](https://learn.microsoft.c When you add a [user account](#user-account) to a group or project, the user account becomes a **member**. +## merge request branch + +Do not use **merge request branch**. See [branch](#branch). + ## merge requests Use lowercase for **merge requests**. If you use **MR** as the acronym, spell it out on first use. @@ -1570,7 +1595,7 @@ Searching is different from [filtering](#filter). When referring to the subscription billing model: - For GitLab SaaS, use **seats**. Customers purchase seats. Users occupy seats when they are invited -to a group, with some [exceptions](../../../subscriptions/gitlab_com/index.md#how-seat-usage-is-determined). + to a group, with some [exceptions](../../../subscriptions/gitlab_com/index.md#how-seat-usage-is-determined). - For GitLab self-managed, use **users**. Customers purchase subscriptions for a specified number of **users**. ## section diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index 62df2395fbb..8a38b0ed0e4 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -492,6 +492,7 @@ command line. To configure markdownlint in your editor, install one of the following as appropriate: +- Visual Studio Code [`DavidAnson.vscode-markdownlint` extension](https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint). - Sublime Text [`SublimeLinter-contrib-markdownlint` package](https://packagecontrol.io/packages/SublimeLinter-contrib-markdownlint). This package uses `markdownlint-cli` by default, but can be configured to use `markdownlint-cli2` with this SublimeLinter configuration: @@ -502,11 +503,20 @@ To configure markdownlint in your editor, install one of the following as approp } ``` -- Visual Studio Code [`DavidAnson.vscode-markdownlint` extension](https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint). - Vim [ALE plugin](https://github.com/dense-analysis/ale). +- Emacs [Flycheck extension](https://github.com/flycheck/flycheck). `Flycheck` supports + `markdownlint-cli` out of the box, but you must add a `.dir-locals.el` file to + point it to the `.markdownlint.yml` at the base of the project directory: + + ```lisp + ;; Place this code in a file called `.dir-locals.el` at the root of the gitlab project. + ((markdown-mode . ((flycheck-markdown-markdownlint-cli-config . ".markdownlint.yml")))) + ``` To configure Vale in your editor, install one of the following as appropriate: +- Visual Studio Code [`ChrisChinchilla.vale-vscode` extension](https://marketplace.visualstudio.com/items?itemName=ChrisChinchilla.vale-vscode). + You can configure the plugin to [display only a subset of alerts](#show-subset-of-vale-alerts). - Sublime Text [`SublimeLinter-vale` package](https://packagecontrol.io/packages/SublimeLinter-vale). To have Vale suggestions appears as blue instead of red (which is how errors appear), add `vale` configuration to your [SublimeLinter](http://sublimelinter.readthedocs.org) configuration: @@ -522,26 +532,12 @@ To configure Vale in your editor, install one of the following as appropriate: ``` - [LSP for Sublime Text](https://lsp.sublimetext.io) package [`LSP-vale-ls`](https://packagecontrol.io/packages/LSP-vale-ls). -- Visual Studio Code [`ChrisChinchilla.vale-vscode` extension](https://marketplace.visualstudio.com/items?itemName=ChrisChinchilla.vale-vscode). - You can configure the plugin to [display only a subset of alerts](#show-subset-of-vale-alerts). - Vim [ALE plugin](https://github.com/dense-analysis/ale). - JetBrains IDEs - No plugin exists, but [this issue comment](https://github.com/errata-ai/vale-server/issues/39#issuecomment-751714451) contains tips for configuring an external tool. -- Emacs [Flycheck extension](https://github.com/flycheck/flycheck). - This requires some configuration: - - - `Flycheck` supports `markdownlint-cli` out of the box, but you must point it - to the `.markdownlint.yml` at the base of the project directory. A `.dir-locals.el` - file can accomplish this: - - ```lisp - ;; Place this code in a file called `.dir-locals.el` at the root of the gitlab project. - ((markdown-mode . ((flycheck-markdown-markdownlint-cli-config . ".markdownlint.yml")))) - - ``` - - - A minimal configuration for Flycheck to work with Vale could look like this: +- Emacs [Flycheck extension](https://github.com/flycheck/flycheck). A minimal configuration + for Flycheck to work with Vale could look like: ```lisp (flycheck-define-checker vale diff --git a/doc/development/documentation/topic_types/troubleshooting.md b/doc/development/documentation/topic_types/troubleshooting.md index aee5bd1377c..f970b58e4fc 100644 --- a/doc/development/documentation/topic_types/troubleshooting.md +++ b/doc/development/documentation/topic_types/troubleshooting.md @@ -64,7 +64,7 @@ The workaround is... If multiple causes or solutions exist, consider putting them into a table format. If you use the exact error message, surround it in backticks so it's styled as code. -For more guidance on solution types, see [workaround](../../documentation/styleguide/word_list.md#workaround) and [resolution, resolve](../../documentation/styleguide/word_list.md#resolution-resolve). +For more guidance on solution types, see [workaround](../../documentation/styleguide/word_list.md#workaround) and [resolution, resolve](../../documentation/styleguide/word_list.md#resolution-resolve). ## Troubleshooting topic titles diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 08045675295..78177612aa9 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -200,7 +200,7 @@ To guard your licensed feature: ``` 1. Optional. If your global feature is also available to namespaces with a paid plan, combine two -feature identifiers to allow both administrators and group users. For example: + feature identifiers to allow both administrators and group users. For example: ```ruby License.feature_available?(:my_feature_name) || group.licensed_feature_available?(:my_feature_name_for_namespace) # Both admins and group members can see this EE feature diff --git a/doc/development/fe_guide/architecture.md b/doc/development/fe_guide/architecture.md index 971c527ef68..75a815925ab 100644 --- a/doc/development/fe_guide/architecture.md +++ b/doc/development/fe_guide/architecture.md @@ -24,7 +24,7 @@ This structure allows the group to think through a proposed change, gather feedb ### Design Documents -When the work ahead may affect more than a single group, stage or potentially an entirement department (for example, all of the Frontend team) then it is likely that there is need for a [Design Document](https://about.gitlab.com/handbook/engineering/architecture/workflow/). +When the work ahead may affect more than a single group, stage or potentially an entire department (for example, all of the Frontend team) then it is likely that there is need for a [Design Document](https://about.gitlab.com/handbook/engineering/architecture/workflow/). This is well documented in the handbook, but to touch on it shortly, it is **the best way** to propose large changes and gather the required feedback and support to move forward. These documents are version controlled, keep evolving with time and are a great way to share a complex understanding across the entire organization. They also require a coach, which is a great way to involve someone with a lot of experience with larger changes. This process is shared across all engineering departments and is owned by the CTO. @@ -57,9 +57,9 @@ Very small changes may have a very broad impact. For example, a change to any ES For recommending certain code patterns in our documentation, you can write the MR that apply your proposed change, share it broadly with the department and if no strong objections are raised, merge your change. This is more efficient than RFCs because of the bias for action, while also gathering all the feedback necessary for everyone to feel included. -If you'd like to propose a major change to the technological stack (Vue to React, JavaScript to TypeScript, etc.), start by reaching out on Slack to gauge interest. Always ask yourself whether or not the problems that you see can be fixed from our current tech stack, as we should always try to fix our problems with the tools we already have. Other departments, such as Backend and QA, do not have a clear process to propose technological changes either. That is because these changes would require huge investements from the company and probably cannot be decided without involving high-ranking executives from engineering. +If you'd like to propose a major change to the technological stack (Vue to React, JavaScript to TypeScript, etc.), start by reaching out on Slack to gauge interest. Always ask yourself whether or not the problems that you see can be fixed from our current tech stack, as we should always try to fix our problems with the tools we already have. Other departments, such as Backend and QA, do not have a clear process to propose technological changes either. That is because these changes would require huge investments from the company and probably cannot be decided without involving high-ranking executives from engineering. -Instead, consider starting a Design Document that explains the problem and try to solve it with our current tools. Invite contribution from the department and research this thoroughly as there can only be two outcomes. Either the problem **can** be solved with our current tools or it cannot. If it can, this is a huge with for our teams since we've fixed and issue without the need to completly change our stack, and if it cannot, then the Design Document can be the start of the larger conversation around the technological change. +Instead, consider starting a Design Document that explains the problem and try to solve it with our current tools. Invite contribution from the department and research this thoroughly as there can only be two outcomes. Either the problem **can** be solved with our current tools or it cannot. If it can, this is a huge win for our teams since we've fixed an issue without the need to completely change our stack, and if it cannot, then the Design Document can be the start of the larger conversation around the technological change. ## Widget Architecture diff --git a/doc/development/fe_guide/content_editor.md b/doc/development/fe_guide/content_editor.md index 7a76b39edbc..64cf7de75ab 100644 --- a/doc/development/fe_guide/content_editor.md +++ b/doc/development/fe_guide/content_editor.md @@ -46,7 +46,7 @@ export default { The rich text editor requires two properties: - `renderMarkdown` is an asynchronous function that returns the response (String) of invoking the -[Markdown API](../../api/markdown.md). + [Markdown API](../../api/markdown.md). - `uploadsPath` is a URL that points to a [GitLab upload service](../uploads/index.md) with `multipart/form-data` support. diff --git a/doc/development/fe_guide/customizable_dashboards.md b/doc/development/fe_guide/customizable_dashboards.md index 93b5b20b047..9ba0d4bb40c 100644 --- a/doc/development/fe_guide/customizable_dashboards.md +++ b/doc/development/fe_guide/customizable_dashboards.md @@ -98,7 +98,7 @@ export const pageViewsOverTime = { To add a new visualization render type: 1. Create a new Vue component that accepts `data` and `options` properties. -See [`line_chart.vue`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/assets/javascripts/analytics/analytics_dashboards/components/visualizations/line_chart.vue) as an example. + See [`line_chart.vue`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/assets/javascripts/analytics/analytics_dashboards/components/visualizations/line_chart.vue) as an example. 1. Add your component to the list of conditional imports in [`panel_base.vue`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/assets/javascripts/vue_shared/components/customizable_dashboard/panels_base.vue#L13). 1. Add your component to the schema's list of `AnalyticsVisualization` types in [`analytics_visualizations.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/validators/json_schemas/analytics_visualization.json). diff --git a/doc/development/fe_guide/design_anti_patterns.md b/doc/development/fe_guide/design_anti_patterns.md deleted file mode 100644 index e0d3a80d9c8..00000000000 --- a/doc/development/fe_guide/design_anti_patterns.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -redirect_to: 'design_patterns.md' -remove_date: '2023-12-07' ---- - -This document was moved to [another location](design_patterns.md). - -<!-- This redirect file can be deleted after <2023-12-07>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/fe_guide/frontend_faq.md b/doc/development/fe_guide/frontend_faq.md index b8e98b47cac..0ab64841365 100644 --- a/doc/development/fe_guide/frontend_faq.md +++ b/doc/development/fe_guide/frontend_faq.md @@ -202,3 +202,10 @@ To see what polyfills are being used: ### 9. Why is my page broken in dark mode? See [dark mode docs](dark_mode.md) + +### 10. How to render GitLab-flavored Markdown? + +If you need to render [GitLab-flavored Markdown](../gitlab_flavored_markdown/index.md), then there are two things that you require: + +- Pass the GLFM content with the `v-safe-html` directive to a `div` HTML element inside your Vue component +- Add the `md` class to the root div, which will apply the appropriate CSS styling diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 22b977519be..bd35d8f237f 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -1671,7 +1671,7 @@ When [using Vuex](#using-with-vuex), disable the cache when: - The data is being cached elsewhere - The use case does not need caching -if the data is being cached elsewhere, or if there is no need for it for the given use case. + if the data is being cached elsewhere, or if there is no need for it for the given use case. ```javascript import createDefaultClient from '~/lib/graphql'; @@ -1767,12 +1767,12 @@ To improve performance, sometimes we want to make initial GraphQL queries early. ``` - Add startup calls with correct variables to the HAML file that serves as a view -for your application. To add GraphQL startup calls, we use -`add_page_startup_graphql_call` helper where the first parameter is a path to the -query, the second one is an object containing query variables. Path to the query is -relative to `app/graphql/queries` folder: for example, if we need a -`app/graphql/queries/repository/files.query.graphql` query, the path is -`repository/files`. + for your application. To add GraphQL startup calls, we use + `add_page_startup_graphql_call` helper where the first parameter is a path to the + query, the second one is an object containing query variables. Path to the query is + relative to `app/graphql/queries` folder: for example, if we need a + `app/graphql/queries/repository/files.query.graphql` query, the path is + `repository/files`. ## Troubleshooting diff --git a/doc/development/fe_guide/haml.md b/doc/development/fe_guide/haml.md index 88f4a785a70..4a6b349bbc8 100644 --- a/doc/development/fe_guide/haml.md +++ b/doc/development/fe_guide/haml.md @@ -20,8 +20,8 @@ used in HAML: - Some of the Pajamas components are available as a [ViewComponent](view_component.md#pajamas-components). Use these when possible. - If no ViewComponent exists, why not go ahead and create it? Talk to the Foundations team if you need help. - As a fallback, this can be done by applying the correct CSS classes to the elements. -- A custom -[Ruby on Rails form builder](https://gitlab.com/gitlab-org/gitlab/-/blob/7c108df101e86d8a27d69df2b5b1ff1fc24133c5/lib/gitlab/form_builders/gitlab_ui_form_builder.rb) exists to help use GitLab UI components in HAML forms. +- A custom [Ruby on Rails form builder](https://gitlab.com/gitlab-org/gitlab/-/blob/7c108df101e86d8a27d69df2b5b1ff1fc24133c5/lib/gitlab/form_builders/gitlab_ui_form_builder.rb) + exists to help use GitLab UI components in HAML forms. ### Use the GitLab UI form builder diff --git a/doc/development/fe_guide/storybook.md b/doc/development/fe_guide/storybook.md index 05d4397933f..f0233f71ad2 100644 --- a/doc/development/fe_guide/storybook.md +++ b/doc/development/fe_guide/storybook.md @@ -68,7 +68,7 @@ To add a story with API access: If you test against `gitlab.com`, make sure to use a token with `read_api` if possible and to make the token short-lived. 1. Create an `.env` file in the `storybook` directory. Use the `storybook/.env.template` file as -a starting point. + a starting point. 1. Set the `API_ACCESS_TOKEN` variable to the access token that you created. diff --git a/doc/development/fe_guide/style/vue.md b/doc/development/fe_guide/style/vue.md index 83d725d453b..870a4c51915 100644 --- a/doc/development/fe_guide/style/vue.md +++ b/doc/development/fe_guide/style/vue.md @@ -123,7 +123,7 @@ Check the [rules](https://github.com/vuejs/eslint-plugin-vue#bulb-rules) for mor ## Naming 1. **Extensions**: Use `.vue` extension for Vue components. Do not use `.js` as file extension -([#34371](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34371)). + ([#34371](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34371)). 1. **Reference Naming**: Use PascalCase for their default imports: ```javascript @@ -142,7 +142,7 @@ Check the [rules](https://github.com/vuejs/eslint-plugin-vue#bulb-rules) for mor }; ``` -1. **Props Naming:** Avoid using DOM component prop names. +1. **Props Naming:** Avoid using DOM component prop names. 1. **Props Naming:** Use kebab-case instead of camelCase to provide props in templates. ```html @@ -582,8 +582,8 @@ describe('MyComponent', () => { ### Setting component state 1. Avoid using [`setProps`](https://v1.test-utils.vuejs.org/api/wrapper/#setprops) to set -component state wherever possible. Instead, set the component's -[`propsData`](https://v1.test-utils.vuejs.org/api/options.html#propsdata) when mounting the component: + component state wherever possible. Instead, set the component's + [`propsData`](https://v1.test-utils.vuejs.org/api/options.html#propsdata) when mounting the component: ```javascript // bad @@ -603,7 +603,7 @@ component state wherever possible. Instead, set the component's ### Accessing component state 1. When accessing props or attributes, prefer the `wrapper.props('myProp')` syntax over -`wrapper.props().myProp` or `wrapper.vm.myProp`: + `wrapper.props().myProp` or `wrapper.vm.myProp`: ```javascript // good @@ -616,7 +616,7 @@ component state wherever possible. Instead, set the component's ``` 1. When asserting multiple props, check the deep equality of the `props()` object with -[`toEqual`](https://jestjs.io/docs/expect#toequalvalue): + [`toEqual`](https://jestjs.io/docs/expect#toequalvalue): ```javascript // good @@ -633,8 +633,8 @@ component state wherever possible. Instead, set the component's ``` 1. If you are only interested in some of the props, you can use -[`toMatchObject`](https://jestjs.io/docs/expect#tomatchobjectobject). Prefer `toMatchObject` -over [`expect.objectContaining`](https://jestjs.io/docs/expect#expectobjectcontainingobject): + [`toMatchObject`](https://jestjs.io/docs/expect#tomatchobjectobject). Prefer `toMatchObject` + over [`expect.objectContaining`](https://jestjs.io/docs/expect#expectobjectcontainingobject): ```javascript // good @@ -652,7 +652,7 @@ over [`expect.objectContaining`](https://jestjs.io/docs/expect#expectobjectconta ### Testing props validation -1. When checking component props use `assertProps` helper. Props validation failures will be thrown as errors +When checking component props use `assertProps` helper. Props validation failures will be thrown as errors: ```javascript import { assertProps } from 'helpers/assert_props' @@ -668,12 +668,12 @@ The goal of this accord is to make sure we are all on the same page. 1. When writing Vue, you may not use jQuery in your application. 1. If you need to grab data from the DOM, you may query the DOM 1 time while bootstrapping your - application to grab data attributes using `dataset`. You can do this without jQuery. + application to grab data attributes using `dataset`. You can do this without jQuery. 1. You may use a jQuery dependency in Vue.js following [this example from the docs](https://vuejs.org/v2/examples/select2.html). 1. If an outside jQuery Event needs to be listen to inside the Vue application, you may use - jQuery event listeners. + jQuery event listeners. 1. We avoid adding new jQuery events when they are not required. Instead of adding new jQuery - events take a look at [different methods to do the same task](https://v2.vuejs.org/v2/api/#vm-emit). + events take a look at [different methods to do the same task](https://v2.vuejs.org/v2/api/#vm-emit). 1. You may query the `window` object one time, while bootstrapping your application for application specific data (for example, `scrollTo` is ok to access anytime). Do this access during the bootstrapping of your application. diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 69967a5a2be..cf7659d0fc0 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -543,7 +543,7 @@ Based on the Vue guidance: - **Do not** use or create a JavaScript class in your [data function](https://v2.vuejs.org/v2/api/#data). - **Do not** add new JavaScript class implementations. - **Do** use [GraphQL](../api_graphql_styleguide.md), [Vuex](vuex.md) or a set of components if -cannot use primitives or objects. + cannot use primitives or objects. - **Do** maintain existing implementations using such approaches. - **Do** Migrate components to a pure object model when there are substantial changes to it. - **Do** add business logic to helpers or utilities, so you can test them separately from your component. @@ -555,7 +555,7 @@ Additional reasons why having a JavaScript class presents maintainability issues - After a class is created, it can be extended in a way that can infringe Vue reactivity and best practices. - A class adds a layer of abstraction, which makes the component API and its inner workings less clear. - It makes it harder to test. Because the class is instantiated by the component data function, it is -harder to 'manage' component and class separately. + harder to 'manage' component and class separately. - Adding Object Oriented Principles (OOP) to a functional codebase adds another way of writing code, reducing consistency and clarity. ## Style guide @@ -831,7 +831,7 @@ describe('~/todos/app.vue', () => { 1. Test any directive that defines if/how child component is rendered (for example, `v-if` and `v-for`). 1. Test any props we are passing to child components (especially if the prop is calculated in the -component under test, with the `computed` property, for example). Remember to use `.props()` and not `.vm.someProp`. + component under test, with the `computed` property, for example). Remember to use `.props()` and not `.vm.someProp`. 1. Test we react correctly to any events emitted from child components: ```javascript diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md index a5a6439b420..59785554ce6 100644 --- a/doc/development/feature_development.md +++ b/doc/development/feature_development.md @@ -97,6 +97,7 @@ Consult these topics for information on contributing to specific GitLab features - [Application limits](application_limits.md) - [AI features](ai_features/index.md) - [Application settings](application_settings.md) +- [Remote Development](remote_development/index.md) ### Import and Export diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 17dd8432f7b..bea42d6340d 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -7,7 +7,7 @@ info: "See the Technical Writers assigned to Development Guidelines: https://han # Feature flags in the development of GitLab NOTE: -This document explains how to contribute to the development of the GitLab product. +This document explains how to contribute to the development and operations of the GitLab product. If you want to use feature flags to show and hide functionality in your own applications, view [this feature flags information](../../operations/feature_flags.md) instead. @@ -17,6 +17,11 @@ All newly-introduced feature flags should be [disabled by default](https://about WARNING: All newly-introduced feature flags should be [used with an actor](controls.md#percentage-based-actor-selection). +Blueprints: + +- (Latest) [Feature Flags usage in GitLab development and operations](../../architecture/blueprints/feature_flags_usage_in_dev_and_ops/index.md) +- [Development Feature Flags Architecture](../../architecture/blueprints/feature_flags_development/index.md) + This document is the subject of continued work as part of an epic to [improve internal usage of feature flags](https://gitlab.com/groups/gitlab-org/-/epics/3551). Raise any suggestions as new issues and attach them to the epic. For an [overview of the feature flag lifecycle](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle), or if you need help deciding [if you should use a feature flag](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags) or not, see the [feature flag lifecycle](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) handbook page. @@ -44,77 +49,147 @@ should be leveraged: When the feature implementation is delivered over multiple merge requests: 1. [Create a new feature flag](#create-a-new-feature-flag) - which is **off** by default, in the first merge request which uses the flag. - Flags [should not be added separately](#risk-of-a-broken-main-branch). + which is **disabled** by default, in the first merge request which uses the flag. + Flags [should not be added separately](#risk-of-a-broken-default-branch). 1. Submit incremental changes via one or more merge requests, ensuring that any - new code added can only be reached if the feature flag is **on**. + new code added can only be reached if the feature flag is **enabled**. You can keep the feature flag enabled on your local GDK during development. 1. When the feature is ready to be tested by other team members, [create the initial documentation](../documentation/feature_flags.md#when-to-document-features-behind-a-feature-flag). Include details about the status of the [feature flag](../documentation/feature_flags.md#how-to-add-feature-flag-documentation). -1. Enable the feature flag for a specific project and ensure that there are no issues +1. Enable the feature flag for a specific group/project/user and ensure that there are no issues with the implementation. Do not enable the feature flag for a public project - like `gitlab` if there is no documentation. Team members and contributors might search for + like `gitlab-org/gitlab` if there is no documentation. Team members and contributors might search for documentation on how to use the feature if they see it enabled in a public project. 1. When the feature is ready for production use, open a merge request to: - Update the documentation to describe the latest flag status. - Add a [changelog entry](#changelog). - - Flip the feature flag to be **on by default** or remove it entirely - to enable the new behavior. + - Remove the feature flag to enable the new behavior, or flip the feature flag to be **enabled by default** (only for `ops` and `beta` feature flags). One might be tempted to think that feature flags will delay the release of a feature by at least one month (= one release). This is not the case. A feature flag does not have to stick around for a specific amount of time (for example, at least one release), instead they should stick around until the feature -is deemed stable. Stable means it works on GitLab.com without causing any -problems, such as outages. +is deemed stable. **Stable means it works on GitLab.com without causing any +problems, such as outages.** -## Risk of a broken main branch +## Risk of a broken default branch Feature flags must be used in the MR that introduces them. Not doing so causes a -[broken main branch](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) scenario due -to the `rspec:feature-flags` job that only runs on the `main` branch. +[broken default branch](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) scenario due +to the `rspec:feature-flags` job that only runs on the default branch. ## Types of feature flags Choose a feature flag type that matches the expected usage. -### `development` type +### `gitlab_com_derisk` type + +`gitlab_com_derisk` feature flags are short-lived feature flags, +used to de-risk GitLab.com deployments. Most feature flags used at +GitLab are of the `gitlab_com_derisk` type. -`development` feature flags are short-lived feature flags, -used for deploying unfinished code to production. Most feature flags used at -GitLab are the `development` type. +#### Constraints -A `development` feature flag must have a rollout issue -created from the [Feature flag Roll Out template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md). +- `default_enabled`: **Must not** be set to true. This kind of feature flag is meant to lower the risk on GitLab.com, thus there's no need to keep the flag in the codebase after it's been enabled on GitLab.com. `default_enabled: true` will not have any effect for this type of feature flag. +- Maximum Lifespan: 2 months after it's merged into the default branch +- Documentation: This type of feature flag don't need to be documented in the + [All feature flags in GitLab](../../user/feature_flags.md) page given they're short-lived and deployment-related +- Rollout issue: **Must** have a rollout issue created from the + [Feature flag Roll Out template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md) + +#### Usage + +The format for `gitlab_com_derisk` feature flags is `Feature.<state>(:<dev_flag_name>)`. -The format for `development` feature flags is `Feature.<state>(:<dev_flag_name>)`. To enable and disable them, run on the GitLab Rails console: ```ruby # To enable it for the instance: -Feature.enable(:<dev_flag_name>) +Feature.enable(:<dev_flag_name>, type: :gitlab_com_derisk) # To disable it for the instance: -Feature.disable(:<dev_flag_name>) +Feature.disable(:<dev_flag_name>, type: :gitlab_com_derisk) # To enable for a specific project: -Feature.enable(:<dev_flag_name>, Project.find(<project id>)) +Feature.enable(:<dev_flag_name>, Project.find(<project id>), type: :gitlab_com_derisk) # To disable for a specific project: -Feature.disable(:<dev_flag_name>, Project.find(<project id>)) +Feature.disable(:<dev_flag_name>, Project.find(<project id>), type: :gitlab_com_derisk) ``` -To check a `development` feature flag's state: +To check a `gitlab_com_derisk` feature flag's state: ```ruby # Check if the feature flag is enabled -Feature.enabled?(:dev_flag_name) +Feature.enabled?(:dev_flag_name, type: :gitlab_com_derisk) # Check if the feature flag is disabled -Feature.disabled?(:dev_flag_name) +Feature.disabled?(:dev_flag_name, type: :gitlab_com_derisk) +``` + +### `wip` type + +Some features are complex and need to be implemented through several MRs. Until they're fully implemented, +it needs to be hidden from anyone. In that case, the `wip` (for "Work In Progress") feature flag allows +to merge all the changes to the main branch without actually using the feature yet. + +Once the feature is complete, the feature flag type can be changed to the `gitlab_com_derisk` or +`beta` type depending on how the feature will be presented/documented to customers. + +#### Constraints + +- `default_enabled`: **Must not** be set to true. If needed, this type can be changed to beta once the feature is complete. +- Maximum Lifespan: 4 months after it's merged into the default branch +- Documentation: This type of feature flag don't need to be documented in the + [All feature flags in GitLab](../../user/feature_flags.md) page given they're mostly hiding unfinished code +- Rollout issue: Likely no need for a rollout issues, as `wip` feature flags should be transitioned to + another type before being enabled + +#### Usage + +```ruby +# Check if feature flag is enabled +Feature.enabled?(:my_wip_flag, project, type: :wip) + +# Check if feature flag is disabled +Feature.disabled?(:my_wip_flag, project, type: :wip) + +# Push feature flag to Frontend +push_frontend_feature_flag(:my_wip_flag, project, type: :wip) ``` -For `development` feature flags, the type doesn't need to be specified (they're the default type). +### `beta` type + +We might +[not be confident we'll be able to scale, support, and maintain a feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#experiment-beta-ga) +in its current form for every designed use case ([example](https://gitlab.com/gitlab-org/gitlab/-/issues/336070#note_1523983444)). +There are also scenarios where a feature is not complete enough to be considered an MVC. +Providing a flag in this case allows engineers and customers to disable the new feature until it's performant enough. + +#### Constraints + +- `default_enabled`: Can be set to `true` so that a feature can be "released" to everyone in Beta with the + possibility to disable it in the case of scalability issues (ideally it should only be disabled for this + reason on specific on-premise installations) +- Maximum Lifespan: 6 months after it's merged into the default branch +- Documentation: This type of feature flag **must** be documented in the + [All feature flags in GitLab](../../user/feature_flags.md) page +- Rollout issue: **Must** have a rollout issue + created from the + [Feature flag Roll Out template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md) + +#### Usage + +```ruby +# Check if feature flag is enabled +Feature.enabled?(:my_beta_flag, project, type: :beta) + +# Check if feature flag is disabled +Feature.disabled?(:my_beta_flag, project, type: :beta) + +# Push feature flag to Frontend +push_frontend_feature_flag(:my_beta_flag, project, type: :beta) +``` ### `ops` type @@ -122,10 +197,20 @@ For `development` feature flags, the type doesn't need to be specified (they're of GitLab product behavior. For example, feature flags that disable features that might have a performance impact such as Sidekiq worker behavior. -`ops` feature flags likely do not have rollout issues, as it is hard to -predict when they are enabled or disabled. +Remember that using this type should follow a conscious decision not to introduce an +instance/group/project/user setting. + +#### Constraints + +- `default_enabled`: Can be set to `true` so that a feature can be "released" to everyone in Beta with the + possibility to disable it in the case of scalability issues (ideally it should only be disabled for this + reason on specific on-premise installations) +- Maximum Lifespan: Unlimited +- Documentation: This type of feature flag **must** be documented in the + [All feature flags in GitLab](../../user/feature_flags.md) page +- Rollout issue: Likely no need for a rollout issues, as it is hard to predict when they are enabled or disabled -To invoke `ops` feature flags, you must append `type: :ops`: +#### Usage ```ruby # Check if feature flag is enabled @@ -142,18 +227,27 @@ push_frontend_feature_flag(:my_ops_flag, project, type: :ops) `experiment` feature flags are used for A/B testing on GitLab.com. -An `experiment` feature flag should conform to the same standards as a `development` feature flag, +An `experiment` feature flag should conform to the same standards as a `beta` feature flag, although the interface has some differences. An experiment feature flag should have a rollout issue, created using the [Experiment Tracking template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Experiment%20Rollout.md). More information can be found in the [experiment guide](../experiment_guide/index.md). +#### Constraints + +- `default_enabled`: **Must not** be set to `true`. +- Maximum Lifespan: 6 months after it's merged into the default branch + ### `worker` type -`worker` feature flags are used for controlling Sidekiq workers behavior, such as deferring Sidekiq jobs. +`worker` feature flags are special `ops` flags that allow to control Sidekiq workers behavior, such as deferring Sidekiq jobs. `worker` feature flags likely do not have any YAML definition as the name could be dynamically generated using the worker name itself, for example, `run_sidekiq_jobs_AuthorizedProjectsWorker`. Some examples for using `worker` type feature flags can be found in [deferring Sidekiq jobs](#deferring-sidekiq-jobs). +### (Deprecated) `development` type + +The `development` type is deprecated in favor of the `gitlab_com_derisk`, `wip`, and `beta` feature flag types. + ## Feature flag definition and validation > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229161) in GitLab 13.3. @@ -179,10 +273,12 @@ Each feature flag is defined in a separate YAML file consisting of a number of f | `name` | yes | Name of the feature flag. | | `type` | yes | Type of feature flag. | | `default_enabled` | yes | The default state of the feature flag. | -| `introduced_by_url` | no | The URL to the merge request that introduced the feature flag. | +| `introduced_by_url` | yes | The URL to the merge request that introduced the feature flag. | +| `milestone` | yes | Milestone in which the feature flag was created. | +| `group` | yes | The [group](https://about.gitlab.com/handbook/product/categories/#devops-stages) that owns the feature flag. | +| `feature_issue_url` | no | The URL to the original feature issue. | | `rollout_issue_url` | no | The URL to the Issue covering the feature flag rollout. | -| `milestone` | no | Milestone in which the feature flag was created. | -| `group` | no | The [group](https://about.gitlab.com/handbook/product/categories/#devops-stages) that owns the feature flag. | +| `log_state_changes` | no | Used to log the state of the feature flag | NOTE: All validations are skipped when running in `RAILS_ENV=production`. @@ -571,6 +667,19 @@ as follows: Feature.remove(:feature_flag_name) ``` +### Logging + +Usage and state of the feature flag is logged if either: + +- `log_state_changes` is set to `true` in the feature flag definition. +- `milestone` refers to a milestone that is greater than or equal to the current GitLab version. + +When the state of a feature flag is logged, it can be identified by using the `"json.feature_flag_states": "feature_flag_name:1"` or `"json.feature_flag_states": "feature_flag_name:0"` condition in Kibana. +You can see an example in [this](https://log.gprd.gitlab.net/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d%2Fd,to:now))&_a=(columns:!(json.feature_flag_states),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:json.feature_flag_states,index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',key:json.feature_flag_states,negate:!f,params:(query:'optimize_where_full_path_in:1'),type:phrase),query:(match_phrase:(json.feature_flag_states:'optimize_where_full_path_in:1')))),hideChart:!f,index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',interval:auto,query:(language:kuery,query:''),sort:!(!(json.time,desc)))) link. + +NOTE: +Only 20% of the requests log the state of the feature flags. This is controlled with the [`feature_flag_state_logs`](https://gitlab.com/gitlab-org/gitlab/-/blob/6deb6ecbc69f05a80d920a295dfc1a6a303fc7a0/config/feature_flags/ops/feature_flag_state_logs.yml) feature flag. + ## Changelog We want to avoid introducing a changelog when features are not accessible by an end-user either directly (example: ability to use the feature) or indirectly (examples: ability to take advantage of background jobs, performance improvements, or database migration updates). diff --git a/doc/development/gems.md b/doc/development/gems.md index 476ed8f916b..cd1a57fe136 100644 --- a/doc/development/gems.md +++ b/doc/development/gems.md @@ -150,6 +150,25 @@ You can see example adding a new gem: [!121676](https://gitlab.com/gitlab-org/gi gem '<name-of-gem>', path: 'gems/<name-of-gem>' ``` +### Specifying dependencies for the Gem + +It is important to note that while the gem has its own `Gemfile`, in the +actual application the top-level `Gemfile` for the monolith GitLab will be +used instead of the individual `Gemfile` sitting in the directory of the gem. + +This means we should be aware that the `Gemfile` for the gem should not use +any versions of dependencies which might be conflicting with the top-level +`Gemfile`, and we should try to use the same dependencies if possible. + +An example of this is [Rack](https://rubygems.org/gems/rack). If the monolith +is using Rack 2 and we're in the process of +[upgrading to Rack 3](https://gitlab.com/gitlab-org/gitlab/-/issues/396273), +all gems we develop should also be tested against Rack 2, optionally also with +Rack 3 if a separate `Gemfile` is used in CI. See an +[example here](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140463). + +Note that this does not limit to just Rack, but any dependencies. + ### Examples of Gem extractions The `gitlab-utils` is a Gem containing as of set of class that implement common intrinsic functions @@ -254,12 +273,12 @@ The project for a new Gem should always be created in [`gitlab-org/ruby/gems` na 1. Create a project in the [`gitlab-org/ruby/gems` group](https://gitlab.com/gitlab-org/ruby/gems/) (or in a subgroup of it): 1. Follow the [instructions for new projects](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#creating-a-new-project). 1. Follow the instructions for setting up a [CI/CD configuration](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#cicd-configuration). - 1. Use the [gem-release CI component](https://gitlab.com/gitlab-org/quality/pipeline-common/-/tree/master/gem-release) + 1. Use the [`gem-release` CI component](https://gitlab.com/gitlab-org/components/gem-release) to release and publish new gem versions by adding the following to their `.gitlab-ci.yml`: ```yaml include: - - component: gitlab.com/gitlab-org/quality/pipeline-common/gem-release@<REPLACE WITH LATEST TAG FROM https://gitlab.com/gitlab-org/quality/pipeline-common/-/releases> + - component: gitlab.com/gitlab-org/components/gem-release/gem-release@~latest ``` This job will handle building and publishing the gem (it uses a `gitlab_rubygems` Rubygems.org diff --git a/doc/development/geo.md b/doc/development/geo.md index d8b000c090a..365893b3be7 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -472,6 +472,10 @@ and the administrator can see this in the [Geo Admin Area](../administration/geo Geo secondaries can proxy web requests to the primary. Read more on the [Geo proxying (development) page](geo/proxying.md). +## Geo API + +Geo uses the external [API](geo/api.md) to facilitate communication between various components. + ## Glossary ### Primary site diff --git a/doc/development/geo/api.md b/doc/development/geo/api.md new file mode 100644 index 00000000000..b9f3f7ce5c9 --- /dev/null +++ b/doc/development/geo/api.md @@ -0,0 +1,32 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Geo API **(PREMIUM SELF)** + +The Geo API is used internally by GitLab components to assist in coordinating Geo actions. It is inaccessible to admins or users. + +## Fetch pipeline refs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/415179) in GitLab 16.7. + +This method returns a list of branches matching `pipeline/refs/X` that exist on the repository for `gl_repository` on the current Geo node. This endpoint is used by runners registered with a secondary Geo instance to check if a repository is up to date. + +```plaintext +GET /geo/repositories/:gl_repository/pipeline_refs +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +|--------------------------|----------|----------|-----------------------| +| `gl_repository` | string | Yes | The `gl_repository` ID of the repository to query | + +If successful, returns [`200`](../../api/rest/index.md#status-codes) and the following +response attributes: + +| Attribute | Type | Description | +|--------------------------|----------|-----------------------| +| `attribute` | 'array' | An array of ids matching `refs/pipeline/X` created for running pipelines. | diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index f3e35f9cdf8..5bfa77e2aa3 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -42,7 +42,7 @@ repositories that depend on the object pool. The danger lies in `git prune`, and `git gc` calls `git prune`. The problem is that `git prune`, when running in a pool repository, cannot -reliable decide if an object is no longer needed. +reliably decide if an object is no longer needed. ### Git alternates in GitLab: pool repositories @@ -51,7 +51,7 @@ which are hidden from the user. We then use Git alternates to let a collection of project repositories borrow from a single pool repository. We call such a collection of project repositories a pool. Pools form star-shaped networks of repositories -that borrow from a single pool, which resemble (but not be +that borrow from a single pool, which resemble (but are not identical to) the fork networks that get formed when users fork projects. diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index 56a4ea898df..9e9216c4f66 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -347,6 +347,23 @@ execute services that: - [Continue their loop](https://gitlab.com/gitlab-org/gitlab/-/blob/487521cc26c1e2bdba4fc67c14478d2b2a5f2bfa/lib/gitlab/github_import/importer/attachments/issues_importer.rb#L27) from where it left off. +## `sidekiq_options dead: false` + +Typically when a worker's retries are exhausted they go to the Sidekiq dead set +and can be retried by an instance admin. + +`GithubImport::Queue` sets the Sidekiq worker option `dead: false` to prevent +this from happening to GitHub importer workers. + +The reason is: + +- The dead set has a max limit and if object importer workers (ones that include + `ObjectImporter`) fail en masse they can spam the dead set and push other workers out. +- Stage workers (ones that include `StageMethods`) + [fail the import](https://gitlab.com/gitlab-org/gitlab/-/blob/dd7cde8d6a28254b9c7aff27f9bf6b7be1ac7532/app/workers/concerns/gitlab/github_import/stage_methods.rb#L23) + when their retries are exhausted, so a retry would be guaranteed to + [be a no-op](https://gitlab.com/gitlab-org/gitlab/-/blob/dd7cde8d6a28254b9c7aff27f9bf6b7be1ac7532/app/workers/concerns/gitlab/github_import/stage_methods.rb#L55-63). + ## Mapping labels and milestones To reduce pressure on the database we do not query it when setting labels and diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index a9f4b22c778..c6471d9720c 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -26,7 +26,7 @@ can still have specifics. They are described in their respective The Go upgrade documentation [provides an overview](go_upgrade.md#overview) of how GitLab manages and ships Go binary support. -If a GitLab component requires a newer version of Go, +If a GitLab component requires a newer version of Go, follow the [upgrade process](go_upgrade.md#updating-go-version) to ensure no customer, team, or component is adversely impacted. Sometimes, individual projects must also [manage builds with multiple versions of Go](go_upgrade.md#supporting-multiple-go-versions). diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 35e423b28e9..16f96687c4f 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -16,7 +16,7 @@ are very appreciative of the work done by translators and proofreaders! - Albanian - Proofreaders needed. - Amharic - - Tsegaselassie Tadesse - [GitLab](https://gitlab.com/tsega), [Crowdin](https://crowdin.com/profile/tsegaselassi) + - Proofreaders needed. - Arabic - Proofreaders needed. - Basque @@ -24,51 +24,38 @@ are very appreciative of the work done by translators and proofreaders! - Belarusian - Anton Katsuba - [GitLab](https://gitlab.com/coinvariant), [Crowdin](https://crowdin.com/profile/aerialfiddle) - Bosnian - - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) + - Proofreaders needed. - Bulgarian - - Lyubomir Vasilev - [Crowdin](https://crowdin.com/profile/lyubomirv) + - Proofreaders needed. - Catalan - - David Planella - [GitLab](https://gitlab.com/dplanella), [Crowdin](https://crowdin.com/profile/dplanella) + - Proofreaders needed. - Chinese Simplified 简体中文 - - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) - - Victor Wu - [GitLab](https://gitlab.com/_victorwu_), [Crowdin](https://crowdin.com/profile/victorwu) - - Xiaogang Wen - [GitLab](https://gitlab.com/xiaogang_cn), [Crowdin](https://crowdin.com/profile/xiaogang_gitlab) - Zhiyuan Lu - [GitLab](https://gitlab.com/luzhiyuan.deer), [Crowdin](https://crowdin.com/profile/luzhiyuan.deer) - Chinese Traditional 繁體中文 - - Weizhe Ding - [GitLab](https://gitlab.com/d.weizhe), [Crowdin](https://crowdin.com/profile/d.weizhe) - - Yi-Jyun Pan - [GitLab](https://gitlab.com/pan93412), [Crowdin](https://crowdin.com/profile/pan93412) - - Victor Wu - [GitLab](https://gitlab.com/_victorwu_), [Crowdin](https://crowdin.com/profile/victorwu) - Hansel Wang - [GitLab](https://gitlab.com/airness), [Crowdin](https://crowdin.com/profile/airness) - Chinese Traditional, Hong Kong 繁體中文 (香港) - - Victor Wu - [GitLab](https://gitlab.com/_victorwu_), [Crowdin](https://crowdin.com/profile/victorwu) - - Ivan Ip - [GitLab](https://gitlab.com/lifehome), [Crowdin](https://crowdin.com/profile/lifehome) + - Proofreaders needed. - Croatian - - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) + - Proofreaders needed. - Czech - - Jan Urbanec - [GitLab](https://gitlab.com/TatranskyMedved), [Crowdin](https://crowdin.com/profile/Tatranskymedved) + - Proofreaders needed. - Danish - - Saederup92 - [GitLab](https://gitlab.com/Saederup92), [Crowdin](https://crowdin.com/profile/Saederup92) - scootergrisen - [GitLab](https://gitlab.com/scootergrisen), [Crowdin](https://crowdin.com/profile/scootergrisen) - Dutch - - Emily Hendle - [GitLab](https://gitlab.com/pundachan), [Crowdin](https://crowdin.com/profile/pandachan) + - Proofreaders needed. - English (UK) - - Andrew Smith - [GitLab](https://gitlab.com/espadav8), [Crowdin](https://crowdin.com/profile/espadav8) + - Proofreaders needed. - Esperanto - - Lyubomir Vasilev - [Crowdin](https://crowdin.com/profile/lyubomirv) + - Proofreaders needed. - Estonian - Proofreaders needed. - Filipino - - Andrei Jiroh Halili - [GitLab](https://gitlab.com/ajhalili2006), [Crowdin](https://crowdin.com/profile/AndreiJirohHaliliDev2006) + - Proofreaders needed. - French - - Davy Defaud - [GitLab](https://gitlab.com/DevDef), [Crowdin](https://crowdin.com/profile/DevDef) - - Germain Gorisse - [GitLab](https://gitlab.com/ggorisse), [Crowdin](https://crowdin.com/profile/germaingorisse) - Xavier Delatour - [GitLab](https://gitlab.com/xdelatour), [Crowdin](https://crowdin.com/profile/xdelatour) - Galician - - Antón Méixome - [Crowdin](https://crowdin.com/profile/meixome) - - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) + - Proofreaders needed. - German - - Michael Hahnle - [GitLab](https://gitlab.com/mhah), [Crowdin](https://crowdin.com/profile/mhah) - - Katrin Leinweber - [GitLab](https://gitlab.com/katrinleinweber), [Crowdin](https://crowdin.com/profile/katrinleinweber) - Vladislav Wanner - [GitLab](https://gitlab.com/RumBugen), [Crowdin](https://crowdin.com/profile/RumBugen) - Daniel Ziegenberg - [GitLab](https://gitlab.com/ziegenberg), [Crowdin](https://crowdin.com/profile/ziegenberg) - Greek @@ -80,69 +67,47 @@ are very appreciative of the work done by translators and proofreaders! - Hungarian - Proofreaders needed. - Indonesian - - Adi Ferdian - [GitLab](https://gitlab.com/adiferd), [Crowdin](https://crowdin.com/profile/adiferd) - - Ahmad Naufal Mukhtar - [GitLab](https://gitlab.com/anaufalm), [Crowdin](https://crowdin.com/profile/anaufalm) + - Proofreaders needed. - Italian - - Massimiliano Cuttini - [GitLab](https://gitlab.com/maxcuttins), [Crowdin](https://crowdin.com/profile/maxcuttins) - - Paolo Falomo - [GitLab](https://gitlab.com/paolofalomo), [Crowdin](https://crowdin.com/profile/paolo.falomo) + - Proofreaders needed. - Japanese - - Hiroyuki Sato - [GitLab](https://gitlab.com/hiroponz), [Crowdin](https://crowdin.com/profile/hiroponz) - Tomo Dote - [GitLab](https://gitlab.com/fu7mu4), [Crowdin](https://crowdin.com/profile/fu7mu4) - - Hiromi Nozawa - [GitLab](https://gitlab.com/hir0mi), [Crowdin](https://crowdin.com/profile/hir0mi) - - Takuya Noguchi - [GitLab](https://gitlab.com/tnir), [Crowdin](https://crowdin.com/profile/tnir) - Tsukasa Komatsubara - [GitLab](https://gitlab.com/tkomatsubara), [Crowdin](https://crowdin.com/profile/tkomatsubara) - Korean - - Chang-Ho Cha - [GitLab](https://gitlab.com/changho-cha), [Crowdin](https://crowdin.com/profile/zzazang) - - Ji Hun Oh - [GitLab](https://gitlab.com/Baw-Appie), [Crowdin](https://crowdin.com/profile/BawAppie) - - Jeongwhan Choi - [GitLab](https://gitlab.com/jeongwhanchoi), [Crowdin](https://crowdin.com/profile/jeongwhanchoi) - Sunjung Park - [GitLab](https://gitlab.com/sunjungp), [Crowdin](https://crowdin.com/profile/sunjungp) + - Hwanyong Lee - [GitLab](https://gitlab.com/hwan_ajou), [Crowdin](https://crowdin.com/profile/grbear) - Mongolian - Proofreaders needed. - Norwegian Bokmal - Imre Kristoffer Eilertsen - [GitLab](https://gitlab.com/DandelionSprout), [Crowdin](https://crowdin.com/profile/DandelionSprout) - Polish - - Filip Mech - [GitLab](https://gitlab.com/mehenz), [Crowdin](https://crowdin.com/profile/mehenz) - - Maksymilian Roman - [GitLab](https://gitlab.com/villaincandle), [Crowdin](https://crowdin.com/profile/villaincandle) - - Jakub Gładykowski - [GitLab](https://gitlab.com/gladykov), [Crowdin](https://crowdin.com/profile/gladykov) + - Proofreaders needed. - Portuguese - - Diogo Trindade - [GitLab](https://gitlab.com/luisdiogo2071317), [Crowdin](https://crowdin.com/profile/ldiogotrindade) + - Proofreaders needed. - Portuguese, Brazilian - - Paulo George Gomes Bezerra - [GitLab](https://gitlab.com/paulobezerra) - - André Gama - [GitLab](https://gitlab.com/andregamma), [Crowdin](https://crowdin.com/profile/ToeOficial) - Eduardo Addad de Oliveira - [GitLab](https://gitlab.com/eduardoaddad), [Crowdin](https://crowdin.com/profile/eduardoaddad) - - Horberlan Brito - [GitLab](https://gitlab.com/horberlan), [Crowdin](https://crowdin.com/profile/horberlan) - Romanian - - Mircea Pop - [GitLab](https://gitlab.com/eeex), [Crowdin](https://crowdin.com/profile/eex) - - Rareș Pița - [GitLab](https://gitlab.com/dlphin) - - Nicolae Liviu - [GitLab](https://gitlab.com/nicklcanada), [Crowdin](https://crowdin.com/profile/nicklcanada) + - Proofreaders needed. - Russian - - Nikita Grylov - [GitLab](https://gitlab.com/nixel2007), [Crowdin](https://crowdin.com/profile/nixel2007) - - Alexy Lustin - [GitLab](https://gitlab.com/allustin), [Crowdin](https://crowdin.com/profile/lustin) - Mark Minakou - [GitLab](https://gitlab.com/sandzhaj), [Crowdin](https://crowdin.com/profile/sandzhaj) - - NickVolynkin - [Crowdin](https://crowdin.com/profile/NickVolynkin) - Andrey Komarov - [GitLab](https://gitlab.com/elkamarado), [Crowdin](https://crowdin.com/profile/kamarado) - - Iaroslav Postovalov - [GitLab](https://gitlab.com/CMDR_Tvis), [Crowdin](https://crowdin.com/profile/CMDR_Tvis) - Serbian (Latin and Cyrillic) - - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) + - Proofreaders needed. - Sinhalese/Sinhala සිංහල - හෙළබස (HelaBasa) - [GitLab](https://gitlab.com/helabasa), [Crowdin](https://crowdin.com/profile/helabasa) - Slovak - Proofreaders needed. - Spanish - - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) - David Elizondo - [GitLab](https://gitlab.com/daelmo), [Crowdin](https://crowdin.com/profile/daelmo) - Pablo Reyes - [GitLab](https://gitlab.com/pabloryst9n), [Crowdin](https://crowdin.com/profile/pabloryst9n) - - Swedish - Johannes Nilsson - [GitLab](https://gitlab.com/nlssn), [Crowdin](https://crowdin.com/profile/nlssn) - Turkish - - Ali Demirtaş - [GitLab](https://gitlab.com/alidemirtas), [Crowdin](https://crowdin.com/profile/alidemirtas) - - Rıfat Ünalmış (Rifat Unalmis) - [GitLab](https://gitlab.com/runalmis), [Crowdin](https://crowdin.com/profile/runalmis) - - İsmail Arılık - [GitLab](https://gitlab.com/ismailarilik), [Crowdin](https://crowdin.com/profile/ismailarilik) + - Proofreaders needed. - Ukrainian - Andrew Vityuk - [GitLab](https://gitlab.com/3_1_3_u), [Crowdin](https://crowdin.com/profile/andruwa13) - Welsh - - Delyth Prys - [GitLab](https://gitlab.com/Delyth), [Crowdin](https://crowdin.com/profile/DelythPrys) + - Proofreaders needed. <!-- vale gitlab.Spelling = YES --> ## Become a proofreader diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index 7149d431c30..ed3afb8efa6 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -96,7 +96,7 @@ For example, in German, the word _user_ can be translated into _Benutzer_ (male) ### Updating the glossary -To propose additions to the glossary, +To propose additions to the glossary, [open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=✓&state=all&label_name[]=Category%3AInternationalization). ## French translation guidelines diff --git a/doc/development/img/runner_fleet_dashboard.png b/doc/development/img/runner_fleet_dashboard.png Binary files differdeleted file mode 100644 index 242ebf4aea9..00000000000 --- a/doc/development/img/runner_fleet_dashboard.png +++ /dev/null diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 3fb89605bdd..6709c748994 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -7,16 +7,16 @@ info: Any user with at least the Maintainer role can merge updates to this conte # Security scanner integration Integrating a security scanner into GitLab consists of providing end users -with a [CI job definition](../../ci/yaml/index.md) -they can add to their CI configuration files to scan their GitLab projects. -This CI job should then output its results in a GitLab-specified format. These results are then +with a [CI/CD job definition](../../ci/jobs/index.md) +they can add to their CI/CD configuration files to scan their GitLab projects. +This job should then output its results in a GitLab-specified format. These results are then automatically presented in various places in GitLab, such as the Pipeline view, merge request widget, and Security Dashboard. The scanning job is usually based on a [Docker image](https://docs.docker.com/) that contains the scanner and all its dependencies in a self-contained environment. -This page documents requirements and guidelines for writing CI jobs that implement a security +This page documents requirements and guidelines for writing CI/CD jobs that implement a security scanner, as well as requirements and guidelines for the Docker image. ## Job definition @@ -285,12 +285,12 @@ See the [`logutil` README](https://gitlab.com/gitlab-org/security-products/analy The report is a JSON document that combines vulnerabilities with possible remediations. -This documentation gives an overview of the report JSON format, -as well as recommendations and examples to help integrators set its fields. +This documentation gives an overview of the report JSON format, recommendations, and examples to +help integrators set its fields. The format is extensively described in the documentation of -[SAST](../../user/application_security/sast/index.md#reports-json-format), +[SAST](../../user/application_security/sast/index.md#output), [DAST](../../user/application_security/dast/proxy-based.md#reports), -[Dependency Scanning](../../user/application_security/dependency_scanning/index.md#reports-json-format), +[Dependency Scanning](../../user/application_security/dependency_scanning/index.md#output), and [Container Scanning](../../user/application_security/container_scanning/index.md#reports-json-format) You can find the schemas for these scanners here: diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index 53c333a6f13..d0a2c8b828f 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -87,8 +87,8 @@ and complete an integration with the Secure stage. - Read about [job artifacts](../../ci/jobs/job_artifacts.md). - Your report artifact must be in one of our currently supported formats. For more information, see the [documentation on reports](secure.md#report). - - Documentation for [SAST reports](../../user/application_security/sast/index.md#reports-json-format). - - Documentation for [Dependency Scanning reports](../../user/application_security/dependency_scanning/index.md#reports-json-format). + - Documentation for [SAST output](../../user/application_security/sast/index.md#output). + - Documentation for [Dependency Scanning reports](../../user/application_security/dependency_scanning/index.md#output). - Documentation for [Container Scanning reports](../../user/application_security/container_scanning/index.md#reports-json-format). - See this [example secure job definition that also defines the artifact created](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Container-Scanning.gitlab-ci.yml). - If you need a new kind of scan or report, [create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new#) diff --git a/doc/development/internal_analytics/index.md b/doc/development/internal_analytics/index.md index b5403f56600..29d1a46c3cf 100644 --- a/doc/development/internal_analytics/index.md +++ b/doc/development/internal_analytics/index.md @@ -17,8 +17,8 @@ when developing new features or instrumenting existing ones. <div class="video-fallback"> See the video about <a href="https://www.youtube.com/watch?v=GtFNXbjygWo">the concepts of events and metrics.</a> </div> -<figure class="video_container"> - <iframe src="https://www.youtube-nocookie.com/embed/GtFNXbjygWo" frameborder="0" allowfullscreen="true"> </iframe> +<figure class="video-container"> + <iframe src="https://www.youtube-nocookie.com/embed/GtFNXbjygWo" frameborder="0" allowfullscreen> </iframe> </figure> Events and metrics are the foundation of the internal analytics system. @@ -95,7 +95,6 @@ SELECT COUNT(*) as event_occurences FROM common_mart.mart_behavior_structured_event WHERE event_action = 'feature_used' -AND event_category = 'InternalEventTracking' AND behavior_date > '2023-08-01' --restricted minimum date for performance AND app_id='gitlab' -- use gitlab for production events and gitlab-staging for events from staging GROUP BY 1 ORDER BY 1 desc diff --git a/doc/development/internal_analytics/internal_event_instrumentation/event_definition_guide.md b/doc/development/internal_analytics/internal_event_instrumentation/event_definition_guide.md index 2c53185a5db..9417861eee9 100644 --- a/doc/development/internal_analytics/internal_event_instrumentation/event_definition_guide.md +++ b/doc/development/internal_analytics/internal_event_instrumentation/event_definition_guide.md @@ -26,7 +26,7 @@ Each event is defined in a separate YAML file consisting of the following fields |------------------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `description` | yes | A description of the event. | | `category` | yes | Always InternalEventTracking (only different for legacy events). | -| `action` | yes | A unique name for the event. | +| `action` | yes | A unique name for the event. Use the format `<operation>_<target_of_operation>_<where/when>`. <br/><br/> Ex: `publish_go_module_to_the_registry_from_pipeline` <br/>`<operation> = publish`<br/>`<target> = go_module`<br/>`<when/where> = to_the_registry_from_pipeline`. | | `identifiers` | no | A list of identifiers sent with the event. Can be set to one or more of `project`, `user`, or `namespace`. | | `product_section` | yes | The [section](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/sections.yml). | | `product_stage` | no | The [stage](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) for the event. | @@ -43,7 +43,7 @@ This is an example YAML file for an internal event: ```yaml description: A user visited a product analytics dashboard category: InternalEventTracking -action: user_visited_dashboard +action: visit_product_analytics_dashboard identifiers: - project - user diff --git a/doc/development/internal_analytics/internal_event_instrumentation/local_setup_and_debugging.md b/doc/development/internal_analytics/internal_event_instrumentation/local_setup_and_debugging.md index 56e83184060..244482a4c2e 100644 --- a/doc/development/internal_analytics/internal_event_instrumentation/local_setup_and_debugging.md +++ b/doc/development/internal_analytics/internal_event_instrumentation/local_setup_and_debugging.md @@ -48,6 +48,15 @@ By default, self-managed instances do not collect event data via Snowplow. We ca 1. You can now see all events being sent by your local instance in the [Snowplow Micro UI](http://localhost:9091/micro/ui) and can filter for specific events. +### Introduction to Snowplow Micro UI and API + +<div class="video-fallback"> + Watch the video about <a href="https://www.youtube.com/watch?v=netZ0TogNcA">Snowplow Micro</a> +</div> +<figure class="video-container"> + <iframe src="https://www.youtube-nocookie.com/embed/netZ0TogNcA" frameborder="0" allowfullscreen> </iframe> +</figure> + ## Configure a remote event collector On GitLab.com events are sent to a collector configured by GitLab. By default, self-managed instances do not have a collector configured and do not collect data with Snowplow. @@ -72,8 +81,8 @@ You can configure your self-managed GitLab instance to use a custom Snowplow col <div class="video-fallback"> Watch the demo video about the <a href="https://www.youtube.com/watch?v=R7vT-VEzZOI">Internal Events Tracking Monitor</a> </div> -<figure class="video_container"> - <iframe src="https://www.youtube-nocookie.com/embed/R7vT-VEzZOI" frameborder="0" allowfullscreen="true"> </iframe> +<figure class="video-container"> + <iframe src="https://www.youtube-nocookie.com/embed/R7vT-VEzZOI" frameborder="0" allowfullscreen> </iframe> </figure> To understand how events are triggered and metrics are updated while you use the Rails app locally or `rails console`, diff --git a/doc/development/internal_analytics/internal_event_instrumentation/migration.md b/doc/development/internal_analytics/internal_event_instrumentation/migration.md index 2ef439e21e9..79ca45ed84c 100644 --- a/doc/development/internal_analytics/internal_event_instrumentation/migration.md +++ b/doc/development/internal_analytics/internal_event_instrumentation/migration.md @@ -125,6 +125,7 @@ To start using Internal Events Tracking, follow these steps: 1. Create an event definition that describes `git_write_action` ([guide](event_definition_guide.md)). 1. Find metric definitions that list `git_write_action` in the events section (`20210216182041_action_monthly_active_users_git_write.yml` and `20210216184045_git_write_action_weekly.yml`). 1. Change the `data_source` from `redis_hll` to `internal_events` in the metric definition files. +1. Remove the `instrumentation_class` property. It's not used for Internal Events metrics. 1. Add an `events` section to both metric definition files. ```yaml diff --git a/doc/development/internal_analytics/internal_event_instrumentation/quick_start.md b/doc/development/internal_analytics/internal_event_instrumentation/quick_start.md index 6f48f83e7ca..c70f7debe41 100644 --- a/doc/development/internal_analytics/internal_event_instrumentation/quick_start.md +++ b/doc/development/internal_analytics/internal_event_instrumentation/quick_start.md @@ -31,6 +31,13 @@ Triggering an event and thereby updating a metric is slightly different on backe ### Backend tracking +<div class="video-fallback"> + Watch the video about <a href="https://www.youtube.com/watch?v=Teid7o_2Mmg">Backend instrumentation using Internal Events</a> +</div> +<figure class="video-container"> + <iframe src="https://www.youtube-nocookie.com/embed/Teid7o_2Mmg" frameborder="0" allowfullscreen> </iframe> +</figure> + To trigger an event, call the `Gitlab::InternalEvents.track_event` method with the desired arguments: ```ruby @@ -50,6 +57,53 @@ It is encouraged to fill out as many of `user`, `namespace` and `project` as pos If a `project` but no `namespace` is provided, the `project.namespace` is used as the `namespace` for the event. +#### Controller and API helpers + +There is a helper module `ProductAnalyticsTracking` for controllers you can use to track internal events for particular controller actions by calling `#track_internal_event`: + +```ruby +class Projects::PipelinesController < Projects::ApplicationController + include ProductAnalyticsTracking + + track_internal_event :charts, name: 'p_analytics_ci_cd_pipelines', conditions: -> { should_track_ci_cd_pipelines? } + + def charts + ... + end + + private + + def should_track_ci_cd_pipelines? + params[:chart].blank? || params[:chart] == 'pipelines' + end +end +``` + +You need to add these two methods to the controller body, so that the helper can get the current project and namespace for the event: + +```ruby + private + + def tracking_namespace_source + project.namespace + end + + def tracking_project_source + project + end +``` + +Also, there is an API helper: + +```ruby +track_event( + event_name, + user: current_user, + namespace_id: namespace_id, + project_id: project_id +) +``` + ### Frontend tracking Any frontend tracking call automatically passes the values `user.id`, `namespace.id`, and `project.id` from the current context of the page. diff --git a/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md b/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md deleted file mode 100644 index 8793e317cdc..00000000000 --- a/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_event_instrumentation/event_definition_guide.md' -remove_date: '2023-12-27' ---- - -This document was moved to [another location](../internal_event_instrumentation/event_definition_guide.md). - -<!-- This redirect file can be deleted after <2023-12-27>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/internal_event_tracking/index.md b/doc/development/internal_analytics/internal_event_tracking/index.md deleted file mode 100644 index 03b539d2a03..00000000000 --- a/doc/development/internal_analytics/internal_event_tracking/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_event_instrumentation/index.md' -remove_date: '2023-12-27' ---- - -This document was moved to [another location](../internal_event_instrumentation/index.md). - -<!-- This redirect file can be deleted after <2023-12-27>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/internal_event_tracking/introduction.md b/doc/development/internal_analytics/internal_event_tracking/introduction.md deleted file mode 100644 index 3f769e7935e..00000000000 --- a/doc/development/internal_analytics/internal_event_tracking/introduction.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_event_instrumentation/introduction.md' -remove_date: '2023-12-27' ---- - -This document was moved to [another location](../internal_event_instrumentation/introduction.md). - -<!-- This redirect file can be deleted after <2023-12-27>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/internal_event_tracking/migration.md b/doc/development/internal_analytics/internal_event_tracking/migration.md deleted file mode 100644 index 9d78f544f4c..00000000000 --- a/doc/development/internal_analytics/internal_event_tracking/migration.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_event_instrumentation/migration.md' -remove_date: '2023-12-27' ---- - -This document was moved to [another location](../internal_event_instrumentation/migration.md). - -<!-- This redirect file can be deleted after <2023-12-27>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/internal_event_tracking/quick_start.md b/doc/development/internal_analytics/internal_event_tracking/quick_start.md deleted file mode 100644 index 4c378c4d7bb..00000000000 --- a/doc/development/internal_analytics/internal_event_tracking/quick_start.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_event_instrumentation/quick_start.md' -remove_date: '2023-12-27' ---- - -This document was moved to [another location](../internal_event_instrumentation/quick_start.md). - -<!-- This redirect file can be deleted after <2023-12-27>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/metrics/metrics_dictionary.md b/doc/development/internal_analytics/metrics/metrics_dictionary.md index 463d5be9100..ba3fb28743a 100644 --- a/doc/development/internal_analytics/metrics/metrics_dictionary.md +++ b/doc/development/internal_analytics/metrics/metrics_dictionary.md @@ -81,9 +81,9 @@ Metric definitions can have one of the following values for `value_type`: - `number` - `string` - `object`: A metric with `value_type: object` must have `value_json_schema` with a link to the JSON schema for the object. -In general, we avoid complex objects and prefer one of the `boolean`, `number`, or `string` value types. -An example of a metric that uses `value_type: object` is `topology` (`/config/metrics/settings/20210323120839_topology.yml`), -which has a related schema in `/config/metrics/objects_schemas/topology_schema.json`. + In general, we avoid complex objects and prefer one of the `boolean`, `number`, or `string` value types. + An example of a metric that uses `value_type: object` is `topology` (`/config/metrics/settings/20210323120839_topology.yml`), + which has a related schema in `/config/metrics/objects_schemas/topology_schema.json`. ### Metric `time_frame` diff --git a/doc/development/internal_analytics/review_guidelines.md b/doc/development/internal_analytics/review_guidelines.md index eb59b834cbc..802a6f410ed 100644 --- a/doc/development/internal_analytics/review_guidelines.md +++ b/doc/development/internal_analytics/review_guidelines.md @@ -29,7 +29,7 @@ In most cases, an Analytics Instrumentation review is automatically added, but i #### The merge request **author** should - Decide whether a Analytics Instrumentation review is needed. You can skip the Analytics Instrumentation -review and remove the labels if the changes are not related to the Analytics Instrumentation domain. + review and remove the labels if the changes are not related to the Analytics Instrumentation domain. - If an Analytics Instrumentation review is needed and was not assigned automatically, add the labels `~analytics instrumentation` and `~analytics instrumentation::review pending`. - Use reviewer roulette to assign an [Analytics Instrumentation reviewer](https://gitlab-org.gitlab.io/gitlab-roulette/?hourFormat24=true&visible=reviewer%7Canalytics+instrumentation) who is not the author. diff --git a/doc/development/internal_analytics/service_ping/review_guidelines.md b/doc/development/internal_analytics/service_ping/review_guidelines.md deleted file mode 100644 index 0ca7b084fc4..00000000000 --- a/doc/development/internal_analytics/service_ping/review_guidelines.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../review_guidelines.md' -remove_date: '2023-12-29' ---- - -This document was moved to [another location](../review_guidelines.md). - -<!-- This redirect file can be deleted after <2023-12-29>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/service_ping/troubleshooting.md b/doc/development/internal_analytics/service_ping/troubleshooting.md index 1b74921fb2f..637aec384c0 100644 --- a/doc/development/internal_analytics/service_ping/troubleshooting.md +++ b/doc/development/internal_analytics/service_ping/troubleshooting.md @@ -90,39 +90,7 @@ However, it has the following limitations: always runs as a process alongside other GitLab components on any given node. For Service Ping, none of the node data would therefore appear to be associated to any of the services running, because they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics appears in Service Ping. -## Service Ping Payload drop - -### Symptoms - -You will be alerted by the [Data team](https://about.gitlab.com/handbook/business-technology/data-team/) and their [Monte Carlo alerting](https://about.gitlab.com/handbook/business-technology/data-team/platform/monte-carlo/). - -### Locating the problem - -First you need to identify at which stage in Service Ping data pipeline the drop is occurring. - -Start at [Service Ping Health Dashboard](https://app.periscopedata.com/app/gitlab/968489) on Sisense. - -You can use [this query](https://gitlab.com/gitlab-org/gitlab/-/issues/347298#note_836685350) as an example, to start detecting when the drop started. - -### Troubleshoot the GitLab application layer - -We conducted an investigation into an unexpected drop in Service ping Payload events volume. -GitLab team members can view more information in this confidential issue: -`https://gitlab.com/gitlab-data/analytics/-/issues/11071` - -### Troubleshoot VersionApp layer - -Check if the [export jobs](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com/-/tree/main/#data-export-using-pipeline-schedules) are successful. - -Check [Service Ping errors](https://app.periscopedata.com/app/gitlab/968489?widget=14609989&udv=0) in the [Service Ping Health Dashboard](https://app.periscopedata.com/app/gitlab/968489). - -### Troubleshoot Google Storage layer - -Check if the files are present in [Google Storage](https://console.cloud.google.com/storage/browser/cloudsql-gs-production-efd5e8-cloudsql-exports;tab=objects?project=gs-production-efd5e8&prefix=&forceOnObjectsSortingFiltering=false). - -### Troubleshoot the data warehouse layer - -Reach out to the [Data team](https://about.gitlab.com/handbook/business-technology/data-team/) to ask about current state of data warehouse. On their handbook page there is a [section with contact details](https://about.gitlab.com/handbook/business-technology/data-team/#how-to-connect-with-us). +## Troubleshooting ### Cannot disable Service Ping with the configuration file diff --git a/doc/development/internal_analytics/snowplow/review_guidelines.md b/doc/development/internal_analytics/snowplow/review_guidelines.md deleted file mode 100644 index 0ca7b084fc4..00000000000 --- a/doc/development/internal_analytics/snowplow/review_guidelines.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../review_guidelines.md' -remove_date: '2023-12-29' ---- - -This document was moved to [another location](../review_guidelines.md). - -<!-- This redirect file can be deleted after <2023-12-29>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index 84f5d09418f..fc73fcb6c2c 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -534,7 +534,6 @@ metric counters. | `counters["k8s_api_proxy_requests_via_user_access"]` | integer | no | The number to increase the `k8s_api_proxy_requests_via_user_access` counter by | | `counters["k8s_api_proxy_requests_via_pat_access"]` | integer | no | The number to increase the `k8s_api_proxy_requests_via_pat_access` counter by | | `unique_counters` | hash | no | Array of unique numbers | -| `unique_counters["agent_users_using_ci_tunnel"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel to track the `agent_users_using_ci_tunnel` metric event | | `unique_counters["k8s_api_proxy_requests_unique_users_via_ci_access"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel via `ci_access` to track the `k8s_api_proxy_requests_unique_users_via_ci_access` metric event | | `unique_counters["k8s_api_proxy_requests_unique_agents_via_ci_access"]` | integer array | no | The set of unique agent ids that have interacted a CI Tunnel via `ci_access` to track the `k8s_api_proxy_requests_unique_agents_via_ci_access` metric event | | `unique_counters["k8s_api_proxy_requests_unique_users_via_user_access"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel via `user_access` to track the `k8s_api_proxy_requests_unique_users_via_user_access` metric event | diff --git a/doc/development/lfs.md b/doc/development/lfs.md index 289c258fafe..2fb7d20cabc 100644 --- a/doc/development/lfs.md +++ b/doc/development/lfs.md @@ -180,32 +180,32 @@ sequenceDiagram 1. The user requests the project archive from the UI. 1. Workhorse forwards this request to Rails. 1. If the user is authorized to download the archive, Rails replies with -an HTTP header of `Gitlab-Workhorse-Send-Data` with a base64-encoded -JSON payload prefaced with `git-archive`. This payload includes the -`SendArchiveRequest` binary message, which is encoded again in base64. + an HTTP header of `Gitlab-Workhorse-Send-Data` with a base64-encoded + JSON payload prefaced with `git-archive`. This payload includes the + `SendArchiveRequest` binary message, which is encoded again in base64. 1. Workhorse decodes the `Gitlab-Workhorse-Send-Data` payload. If the -archive already exists in the archive cache, Workhorse sends that -file. Otherwise, Workhorse sends the `SendArchiveRequest` to the -appropriate Gitaly server. + archive already exists in the archive cache, Workhorse sends that + file. Otherwise, Workhorse sends the `SendArchiveRequest` to the + appropriate Gitaly server. 1. The Gitaly server calls `git archive <ref>` to begin generating -the Git archive on-the-fly. If the `include_lfs_blobs` flag is enabled, -Gitaly enables a custom LFS smudge filter via the `-c -filter.lfs.smudge=/path/to/gitaly-lfs-smudge` Git option. + the Git archive on-the-fly. If the `include_lfs_blobs` flag is enabled, + Gitaly enables a custom LFS smudge filter via the `-c + filter.lfs.smudge=/path/to/gitaly-lfs-smudge` Git option. 1. When `git` identifies a possible LFS pointer using the -`.gitattributes` file, `git` calls `gitaly-lfs-smudge` and provides the -LFS pointer via the standard input. Gitaly provides `GL_PROJECT_PATH` -and `GL_INTERNAL_CONFIG` as environment variables to enable lookup of -the LFS object. + `.gitattributes` file, `git` calls `gitaly-lfs-smudge` and provides the + LFS pointer via the standard input. Gitaly provides `GL_PROJECT_PATH` + and `GL_INTERNAL_CONFIG` as environment variables to enable lookup of + the LFS object. 1. If a valid LFS pointer is decoded, `gitaly-lfs-smudge` makes an -internal API call to Workhorse to download the LFS object from GitLab. + internal API call to Workhorse to download the LFS object from GitLab. 1. Workhorse forwards this request to Rails. If the LFS object exists -and is associated with the project, Rails sends `ArchivePath` either -with a path where the LFS object resides (for local disk) or a -pre-signed URL (when object storage is enabled) via the -`Gitlab-Workhorse-Send-Data` HTTP header with a payload prefaced with -`send-url`. + and is associated with the project, Rails sends `ArchivePath` either + with a path where the LFS object resides (for local disk) or a + pre-signed URL (when object storage is enabled) via the + `Gitlab-Workhorse-Send-Data` HTTP header with a payload prefaced with + `send-url`. 1. Workhorse retrieves the file and send it to the `gitaly-lfs-smudge` -process, which writes the contents to the standard output. + process, which writes the contents to the standard output. 1. `git` reads this output and sends it back to the Gitaly process. 1. Gitaly sends the data back to Rails. 1. The archive data is sent back to the client. diff --git a/doc/development/logging.md b/doc/development/logging.md index 2af914d76ef..1cfcc6530c5 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -330,7 +330,7 @@ Entry points can be seen at: When adding new attributes, make sure they're exposed within the context of the entry points above and: - Pass them within the hash to the `with_context` (or `push`) method (make sure to pass a Proc if the -method or variable shouldn't be evaluated right away) + method or variable shouldn't be evaluated right away) - Change `Gitlab::ApplicationContext` to accept these new values - Make sure the new attributes are accepted at [`Labkit::Context`](https://gitlab.com/gitlab-org/labkit-ruby/blob/master/lib/labkit/context.rb) diff --git a/doc/development/merge_request_concepts/diffs/index.md b/doc/development/merge_request_concepts/diffs/index.md index ad0e8603983..eb7c836e610 100644 --- a/doc/development/merge_request_concepts/diffs/index.md +++ b/doc/development/merge_request_concepts/diffs/index.md @@ -28,6 +28,7 @@ codebase in the future: - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Recording on YouTube](https://www.youtube.com/watch?v=K6G3gMcFyek) + <!-- Video published on 2019-01-29 --> - Slides on [Google Slides](https://docs.google.com/presentation/d/1bGutFH2AT3bxOPZuLMGl1ANWHqFnrxwQwjiwAZkF-TU/edit) - [PDF slides](https://gitlab.com/gitlab-org/create-stage/uploads/b5ad2f336e0afcfe0f99db0af0ccc71a/) @@ -192,7 +193,7 @@ has been introduced. One of the key challenges to deal with when working on merge ref diffs are merge conflicts. If the target and source branch contains a merge conflict, the branches cannot be automatically merged. The -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=GFXIFA4ZuZw&feature=youtu.be&ab_channel=GitLabUnfiltered) +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=GFXIFA4ZuZw&feature=youtu.be&ab_channel=GitLabUnfiltered) <!-- Video published on 2020-07-24 --> is a quick introduction to the problem and the motivation behind the [epic](https://gitlab.com/groups/gitlab-org/-/epics/854). In 13.5 a solution for both-modified merge diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 30f598ef736..be76680c44c 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -723,7 +723,7 @@ The `with_lock_retries` method **cannot** be used within the `change` method, yo 1. For each iteration, set a pre-configured `lock_timeout`. 1. Try to execute the given block. (`remove_column`). 1. If `LockWaitTimeout` error is raised, sleep for the pre-configured `sleep_time` -and retry the block. + and retry the block. 1. If no error is raised, the current iteration has successfully executed the block. For more information check the [`Gitlab::Database::WithLockRetries`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/with_lock_retries.rb) class. The `with_lock_retries` helper method is implemented in the [`Gitlab::Database::MigrationHelpers`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/migration_helpers.rb) module. @@ -733,7 +733,7 @@ In a worst-case scenario, the method: - Executes the block for a maximum of 50 times over 40 minutes. - Most of the time is spent in a pre-configured sleep period after each iteration. - After the 50th retry, the block is executed without `lock_timeout`, just -like a standard migration invocation. + like a standard migration invocation. - If a lock cannot be acquired, the migration fails with `statement timeout` error. The migration might fail if there is a very long running transaction (40+ minutes) diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index 40a30aa4926..716e42655d0 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -234,7 +234,7 @@ And these deployments align perfectly with application changes. 1. At the beginning we have `Version N` on `Schema A`. 1. Then we have a _long_ transition period with both `Version N` and `Version N+1` on `Schema B`. 1. When we only have `Version N+1` on `Schema B` the schema changes again. -1. Finally we have `Version N+1` on `Schema C`. +1. Finally we have `Version N+1` on `Schema C`. With all those details in mind, let's imagine we need to replace a query, and this query has an index to support it. @@ -310,10 +310,10 @@ variable `CI_NODE_TOTAL` being an integer failed. This was caused because after 1. New code: Sidekiq created a new pipeline and new build. `build.options[:parallel]` is a `Hash`. 1. Old code: Runners requested a job from an API node that is running the previous version. 1. As a result, the [new code](https://gitlab.com/gitlab-org/gitlab/-/blob/42b82a9a3ac5a96f9152aad6cbc583c42b9fb082/app/models/concerns/ci/contextable.rb#L104) -was not run on the API server. The runner's request failed because the -older API server tried return the `CI_NODE_TOTAL` CI/CD variable, but -instead of sending an integer value (for example, 9), it sent a serialized -`Hash` value (`{:number=>9, :total=>9}`). + was not run on the API server. The runner's request failed because the + older API server tried return the `CI_NODE_TOTAL` CI/CD variable, but + instead of sending an integer value (for example, 9), it sent a serialized + `Hash` value (`{:number=>9, :total=>9}`). If you look at the [deployment pipeline](https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/202212), you see all nodes were updated in parallel: @@ -322,11 +322,11 @@ you see all nodes were updated in parallel: However, even though the updated started around the same time, the completion time varied significantly: -|Node type|Duration (min)| -|---------|--------------| -|API |54 | -|Sidekiq |21 | -|K8S |8 | +| Node type | Duration (min) | +|-----------|----------------| +| API | 54 | +| Sidekiq | 21 | +| K8S | 8 | Builds that used the `parallel` keyword and depended on `CI_NODE_TOTAL` and `CI_NODE_INDEX` would fail during the time after Sidekiq was diff --git a/doc/development/packages/cleanup_policies.md b/doc/development/packages/cleanup_policies.md index bbec6ce0623..789bb408dd7 100644 --- a/doc/development/packages/cleanup_policies.md +++ b/doc/development/packages/cleanup_policies.md @@ -27,7 +27,7 @@ The parameters are split into two groups: - The parameters that define tags to destroy: - `older_than`. Destroy tags older than this timestamp. - `name_regex`. Destroy tags matching this regular expression. - + The remaining parameters impact when the policy is executed: - `enabled`. Defines if the policy is enabled or not. @@ -41,8 +41,7 @@ follows this design. - Policy executions are limited in time. - Policy executions are either complete or partial. -- The background jobs will consider the next job to be executed based on two -priorities: +- The background jobs will consider the next job to be executed based on two priorities: - Policy with a `next_run_at` in the past. - Partially executed policies. @@ -54,7 +53,7 @@ Background jobs for this execution are organized on: - A cron background job that runs every hour. - A set of background jobs that will loop on container repositories that need -a policy execution. + a policy execution. #### The cron background job @@ -63,7 +62,7 @@ is quite simple. Its main tasks are: 1. Check if there are any container repositories in need of a cleanup. If any, -enqueue as many limited capacity jobs as necessary, up to a limit. + enqueue as many limited capacity jobs as necessary, up to a limit. 1. Compute metrics for cleanup policies and log them. #### The limited capacity job @@ -97,14 +96,14 @@ flowchart TD ``` - [`ContainerExpirationPolicies::CleanupService`](https://gitlab.com/gitlab-org/gitlab/-/blob/6546ffc6fe4e9b447a1b7f050edddb8926fe4a3d/app/services/container_expiration_policies/cleanup_service.rb). -This service mainly deals with container repository `expiration_policy_cleanup_status` -updates and will call the cleanup tags service. + This service mainly deals with container repository `expiration_policy_cleanup_status` + updates and will call the cleanup tags service. - [`Projects::ContainerRepository::CleanupTagsService`](https://gitlab.com/gitlab-org/gitlab/-/blob/f23d70b7d638c38d71af102cfd32a3f6751596f9/app/services/projects/container_repository/cleanup_tags_service.rb). -This service receives the policy parameters and builds the list of tags to -destroy on the container registry. + This service receives the policy parameters and builds the list of tags to + destroy on the container registry. - [`Projects::ContainerRepository::DeleteTagsService`](https://gitlab.com/gitlab-org/gitlab/-/blob/f23d70b7d638c38d71af102cfd32a3f6751596f9/app/services/projects/container_repository/delete_tags_service.rb). -This service receives a list of tags and loops on that list. For each tag, -the service will call the container registry API endpoint to destroy the target tag. + This service receives a list of tags and loops on that list. For each tag, + the service will call the container registry API endpoint to destroy the target tag. The cleanup tags service uses a very specific [execution order](../../user/packages/container_registry/reduce_container_registry_storage.md#how-the-cleanup-policy-works) to build the list of tags to destroy. diff --git a/doc/development/packages/settings.md b/doc/development/packages/settings.md index 89f91f41f4c..690f9ccae93 100644 --- a/doc/development/packages/settings.md +++ b/doc/development/packages/settings.md @@ -70,6 +70,8 @@ Setting | Table | Description `nuget_duplicates_allowed` | `namespace_package_settings` | Allow or prevent duplicate NuGet packages. `nuget_duplicate_exception_regex` | `namespace_package_settings` | Regex defining NuGet packages that are allowed to be duplicate when duplicates are not allowed. `nuget_symbol_server_enabled` | `namespace_package_settings` | Enable or disable the NuGet symbol server. +`terraform_module_duplicates_allowed` | `namespace_package_settings` | Allow or prevent duplicate Terraform module packages. +`terraform_module_duplicate_exception_regex` | `namespace_package_settings` | Regex defining Terraform module packages that are allowed to be duplicate when duplicates are not allowed. Dependency Proxy Cleanup Policies - `ttl` | `dependency_proxy_image_ttl_group_policies` | Number of days to retain an unused Dependency Proxy file before it is removed. Dependency Proxy - `enabled` | `dependency_proxy_image_ttl_group_policies` | Enable or disable the Dependency Proxy cleanup policy. diff --git a/doc/development/pages/index.md b/doc/development/pages/index.md index f4710c951ed..7bf89789ee0 100644 --- a/doc/development/pages/index.md +++ b/doc/development/pages/index.md @@ -22,14 +22,14 @@ subdomain. You can set the GitLab Pages hostname: As `/etc/hosts` don't support wildcard hostnames, you must configure one entry for GitLab Pages, and then one entry for each page site: - ```plaintext - 127.0.0.1 gdk.test # If you're using GDK - 127.0.0.1 pages.gdk.test # Pages host - # Any namespace/group/user needs to be added - # as a subdomain to the pages host. This is because - # /etc/hosts doesn't accept wildcards - 127.0.0.1 root.pages.gdk.test # for the root pages - ``` +```plaintext +127.0.0.1 gdk.test # If you're using GDK +127.0.0.1 pages.gdk.test # Pages host +# Any namespace/group/user needs to be added +# as a subdomain to the pages host. This is because +# /etc/hosts doesn't accept wildcards +127.0.0.1 root.pages.gdk.test # for the root pages +``` ### With DNS wildcard alternatives @@ -151,8 +151,8 @@ GitLab Pages access control is disabled by default. To enable it: 1. Create an [Instance-wide OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) with the `api` scope. 1. Set the value of your `redirect-uri` to the `pages-domain` authorization endpoint -(for example, `http://pages.gdk.test:3010/auth`). -The `redirect-uri` must not contain any GitLab Pages site domain. + (for example, `http://pages.gdk.test:3010/auth`). + The `redirect-uri` must not contain any GitLab Pages site domain. 1. Add the auth client configuration: diff --git a/doc/development/permissions/custom_roles.md b/doc/development/permissions/custom_roles.md index 9d02c0c6f39..53589658f56 100644 --- a/doc/development/permissions/custom_roles.md +++ b/doc/development/permissions/custom_roles.md @@ -34,7 +34,7 @@ Like static roles, custom roles are [inherited](../../user/project/members/index - A Group or project membership can be associated with any custom role that is defined on the root-level group of the group or project. - The `member_roles` table includes individual permissions and a `base_access_level` value. - The `base_access_level` must be a [valid access level](../../api/access_requests.md#valid-access-levels). -The `base_access_level` determines which abilities are included in the custom role. For example, if the `base_access_level` is `10`, the custom role will include any abilities that a static Guest role would receive, plus any additional abilities that are enabled by the `member_roles` record by setting an attribute, such as `read_code`, to true. + The `base_access_level` determines which abilities are included in the custom role. For example, if the `base_access_level` is `10`, the custom role will include any abilities that a static Guest role would receive, plus any additional abilities that are enabled by the `member_roles` record by setting an attribute, such as `read_code`, to true. - A custom role can enable additional abilities for a `base_access_level` but it cannot disable a permission. As a result, custom roles are "additive only". The rationale for this choice is [in this comment](https://gitlab.com/gitlab-org/gitlab/-/issues/352891#note_1059561579). - Custom role abilities are supported at project level and group level. @@ -169,7 +169,7 @@ For example, you see in `GroupPolicy` that there is an ability called than adding a row to the `member_roles` table for each ability, consider renaming them to `read_security_dashboard` and adding `read_security_dashboard` to the `member_roles` table. This is more expected because it means that -enabling `read_security_dashboard` on the parent group will enable the custom +enabling `read_security_dashboard` on the parent group will enable the custom role. For example, `GroupPolicy` has an ability called `read_group_security_dashboard` and `ProjectPolicy` has an ability called `read_project_security_dashboard`. If you would like to make both customizable, rather than adding a row to the `member_roles` table for each ability, consider renaming them to `read_security_dashboard` and adding @@ -183,23 +183,116 @@ security dashboard. To add a new ability to a custom role: - Generate YAML file by running `./ee/bin/custom-ability` generator -- Add a new column to `member_roles` table, for example in [this change in merge request 114734](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114734/diffs#diff-content-5c53d6f1c29a272a87eecea3f62d017ab6635275). -- Add the ability to the `MemberRole` model, `ALL_CUSTOMIZABLE_PERMISSIONS` hash, for example in [this change in merge request 121534](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121534/diffs#ce5ec769500a53ce2b603467d9984fc2b33ca71d_8_8). There are following possible keys in the `ALL_CUSTOMIZABLE_PERMISSIONS` hash: - - - `description` - description of the ability. - - `minimal_level` - minimal level a user has to have in order to be able to be assigned to the ability. - - `requirement` - required ability for the ability defined in the hash, in case the requirement is `false`, the ability can not be `true`. - +- Add a new column to `member_roles` table, either manually or by running `custom_roles:code` generator, eg. by running `rails generate gitlab:custom_roles:code --ability new_ability_name`. The ability parameter is case sensitive and has to exactly match the permission name from the YAML file. - Add the ability to the respective Policy for example in [this change in merge request 114734](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114734/diffs#diff-content-edcbe28bdecbd848d4d9efdc5b5e9bddd2a7299e). -- Update the specs. +- Update the specs. Don't forget to add a spec to `ee/spec/requests/custom_roles` - the spec template file was pre-generated if you used the code generator +- Compile the documentation by running `bundle exec rake gitlab:custom_roles:compile_docs` +- Update the GraphQL documentation by running `bundle exec rake gitlab:graphql:compile_docs` Examples of merge requests adding new abilities to custom roles: - [Read code](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106256) - [Read vulnerability](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114734) -- [Admin vulnerability](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121534) - this is the newest MR implementing a new custom role ability. Some changes from the previous MRs are not necessary anymore (such as a change of the Preloader query or adding a method to `User` model). +- [Admin vulnerability](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121534) + +The above merge requests don't use YAML files and code generators. Some of the changes are not needed anymore. We will update the documentation once we have a permission implemented using the generators. + +If you have any concerns, put the new ability behind a feature flag. + +#### Documenting handling the feature flag + +- When you introduce a new custom ability under a feature flag, add the `feature_flag` attribute to the appropriate ability YAML file. +- When you enable the ability by default, add the `feature_flag_enabled_milestone` and `feature_flag_enabled_mr` attributes to the appropriate ability YAML file and regenerate the documentation. +- You do not have to include these attributes in the YAML file if the feature flag is enabled by default in the same release as the ability is introduced. + +#### Testing + +Unit tests are preferred to test out changes to any policies affected by the +addition of new custom permissions. Custom Roles is an Ultimate tier feature so +these tests can be found in the `ee/spec/policies` directory. The [spec file](https://gitlab.com/gitlab-org/gitlab/-/blob/13baa4e8c92a56260591a5bf0a58d3339890ee10/ee/spec/policies/project_policy_spec.rb#L2726-2740) for +the `ProjectPolicy` contains shared examples that can be used to test out the +following conditions: + +- when the `custom_roles` licensed feature is not enabled +- when the `custom_roles` licensed feature is enabled + - when a user is a member of a custom role via an inherited group member + - when a user is a member of a custom role via a direct group member + - when a user is a member of a custom role via a direct project membership + +Below is an example for testing out `ProjectPolicy` related changes. + +```ruby + context 'for a role with `custom_permission` enabled' do + let(:member_role_abilities) { { custom_permission: true } } + let(:allowed_abilities) { [:custom_permission] } + + it_behaves_like 'custom roles abilities' + end +``` + +Request specs are preferred to test out any endpoint that allow access via a custom role permission. +This includes controllers, REST API, and GraphQL. Examples of request specs can be found in `ee/spec/requests/custom_roles/`. In this directory you will find a sub-directory named after each permission that can be enabled via a custom role. +The `custom_roles` licensed feature must be enabled to test this functionality. + +Below is an example of the typical setup that is required to test out a +Rails Controller endpoint. + +```ruby + let_it_be(:user) { create(:user) } + let_it_be(:project) { create(:project, :repository, :in_group) } + let_it_be(:role) { create(:member_role, :guest, namespace: project.group, custom_permission: true) } + let_it_be(:membership) { create(:project_member, :guest, member_role: role, user: user, project: project) } + + before do + stub_licensed_features(custom_roles: true) + sign_in(user) + end + + describe MyController do + describe '#show' do + it 'allows access' do + get my_controller_path(project) + + expect(response).to have_gitlab_http_status(:ok) + expect(response).to render_template(:show) + end + end + end +``` + +Below is an example of the typical setup that is required to test out a GraphQL +mutation. + +```ruby + let_it_be(:user) { create(:user) } + let_it_be(:project) { create(:project, :repository, :in_group) } + let_it_be(:role) { create(:member_role, :guest, namespace: project.group, custom_permission: true) } + let_it_be(:membership) { create(:project_member, :guest, member_role: role, user: user, project: project) } + + before do + stub_licensed_features(custom_roles: true) + end + + describe MyMutation do + include GraphqlHelpers + + describe '#show' do + it 'allows access' do + post_graphql_mutation(graphql_mutation(:my_mutation, { + example: "Example" + }), current_user: user) + + expect(response).to have_gitlab_http_status(:success) + mutation_response = graphql_mutation_response(:my_mutation) + expect(mutation_response).to be_present + expect(mutation_response["errors"]).to be_empty + end + end + end +``` -You should make sure a new custom roles ability is under a feature flag. +[`GITLAB_DEBUG_POLICIES=true`](#finding-existing-abilities-checks) can be used +to troubleshoot runtime policy decisions. ## Custom abilities definition @@ -213,6 +306,7 @@ To add a new custom ability: - Use the `ee/bin/custom-ability` CLI to create the YAML definition automatically. - Perform manual steps to create a new file in `ee/config/custom_abilities/` with the filename matching the name of the ability name. 1. Add contents to the file that conform to the [schema](#schema) defined in `ee/config/custom_abilities/types/type_schema.json`. +1. Add [tests](#testing) for the new ability in `ee/spec/requests/custom_roles/` with a new directory named after the ability name. ### Schema @@ -222,7 +316,7 @@ To add a new custom ability: | `description` | yes | Human-readable description of the custom ability. | | `feature_category` | yes | Name of the feature category. For example, `vulnerability_management`. | | `introduced_by_issue` | yes | Issue URL that proposed the addition of this custom ability. | -| `introduced_by_mr` | yes | MR URL that added this custom ability. | +| `introduced_by_mr` | no | MR URL that added this custom ability. | | `milestone` | yes | Milestone in which this custom ability was added. | | `group_ability` | yes | Indicate whether this ability is checked on group level. | | `project_ability` | yes | Indicate whether this ability is checked on project level. | diff --git a/doc/development/permissions/predefined_roles.md b/doc/development/permissions/predefined_roles.md index 7cb00977e1f..577cf6192c4 100644 --- a/doc/development/permissions/predefined_roles.md +++ b/doc/development/permissions/predefined_roles.md @@ -83,10 +83,10 @@ module): - Maintainer (`40`) - Owner (`50`) -If a user is the member of both a project and the project parent groups, the +If a user is a member of both a project and the project parent groups, the highest permission is the applied access level for the project. -If a user is the member of a project, but not the parent groups, they +If a user is a member of a project, but not the parent groups, they can still view the groups and their entities (like epics). Project membership (where the group membership is already taken into account) diff --git a/doc/development/pipelines/index.md b/doc/development/pipelines/index.md index c4dfda9466a..91f4ae702ac 100644 --- a/doc/development/pipelines/index.md +++ b/doc/development/pipelines/index.md @@ -13,7 +13,7 @@ which itself includes files under for easier maintenance. We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/development/principles/#dogfooding) -GitLab [CI/CD features and best-practices](../../ci/yaml/index.md) +GitLab [CI/CD features and best-practices](../../ci/index.md) as much as possible. ## Predictive test jobs before a merge request is approved @@ -268,6 +268,9 @@ the specific list of rules. If you want to force `e2e:package-and-test` to run regardless of your changes, you can add the `pipeline:run-all-e2e` label to the merge request. +The [`e2e:test-on-gdk`](../testing_guide/end_to_end/index.md#using-the-test-on-gdk-job) child pipeline runs `:reliable` +E2E specs automatically for all `code patterns changes`. See `.qa:rules:e2e-blocking` [`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml) for specific set of rules. + Consult the [End-to-end Testing](../testing_guide/end_to_end/index.md) dedicated page for more information. ### Review app jobs @@ -297,6 +300,23 @@ set and get the `ee/` folder removed before the tests start running. The intent is to ensure that a change doesn't introduce a failure after `gitlab-org/gitlab` is synced to `gitlab-org/gitlab-foss`. +#### As-if-FOSS cross project downstream pipeline + +As an alternative to the `* as-if-foss` jobs, we can also run a cross project +FOSS pipeline exactly in the `gitlab-org/gitlab-foss` project. We trigger it +in the following cases: + +- when the `pipeline:run-as-if-foss-cross-project` label is set on the merge request + +This is still working-in-progress to replace the `* as-if-foss` jobs. The +goal is to simplify pipeline rules and make it more clear about the intention. + +##### Tokens set in the project variables + +- `AS_IF_FOSS_TOKEN`: This is a [GitLab FOSS](https://gitlab.com/gitlab-org/gitlab-foss) + project token with `developer` role and `write_repository` permission, + to push generated `as-if-foss/*` branch. + ### As-if-JH cross project downstream pipeline #### What it is @@ -396,7 +416,8 @@ flowchart TD - `ADD_JH_FILES_TOKEN`: This is a [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab) project token with `read_api` permission, to be able to download JiHu files. - `AS_IF_JH_TOKEN`: This is a [GitLab JH validation](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation) - project token with `write_repository` permission, to push generated `as-if-jh/*` branch. + project token with `developer` role and `write_repository` permission, + to push generated `as-if-jh/*` branch. ##### How we generate the as-if-JH branch @@ -610,30 +631,30 @@ Exceptions to this general guideline should be motivated and documented. ### Ruby versions testing -We're running Ruby 3.0 on GitLab.com, as well as for the default branch. -To prepare for the next Ruby version, we run merge requests in Ruby 3.1. +We're running Ruby 3.1 on GitLab.com, as well as for the default branch. +To prepare for the next Ruby version, we will run merge requests in Ruby 3.2, +starting on February 2024. Please see the roadmap at +[Ruby 3.2 epic](https://gitlab.com/groups/gitlab-org/-/epics/9684#plan) +for more details. -This takes effects at the time when -[Run merge requests in Ruby 3.1 by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134290) -is merged. See -[Ruby 3.1 epic](https://gitlab.com/groups/gitlab-org/-/epics/10034) -for the roadmap to fully make Ruby 3.1 the default. +To make sure all supported Ruby versions are working, we also run our test +suite on dedicated 2-hourly scheduled pipelines for each supported versions. -To make sure both Ruby versions are working, we also run our test suite -against both Ruby 3.0 and Ruby 3.1 on dedicated 2-hourly scheduled pipelines. +For merge requests, you can add the following labels to run the respective +Ruby version only: -For merge requests, you can add the `pipeline:run-in-ruby3_0` label to switch -the Ruby version to 3.0. When you do this, the test suite will no longer run -in Ruby 3.1 (default for merge requests). +- `pipeline:run-in-ruby3_0` +- `pipeline:run-in-ruby3_1` +- `pipeline:run-in-ruby3_2` -When the pipeline is running in a Ruby version not considered default, an -additional job `verify-default-ruby` will also run and always fail to remind -us to remove the label and run in default Ruby before merging the merge -request. At the moment both Ruby 3.0 and Ruby 3.1 are considered default. +Note that when you do this, the test suite will no longer run in the default +Ruby version for merge requests. In this case, an additional job +`verify-default-ruby` will also run and always fail to remind us to remove +the label and run in default Ruby before merging the merge request. This should let us: -- Test changes for Ruby 3.1 +- Test changes for any supported Ruby versions - Make sure it will not break anything when it's merged into the default branch ### PostgreSQL versions testing @@ -649,24 +670,27 @@ We also run our test suite against PostgreSQL 13 upon specific database library | Where? | PostgreSQL version | Ruby version | |--------------------------------------------------------------------------------------------------|-------------------------------------------------|-----------------------| -| Merge requests | 14 (default version), 13 for DB library changes | 3.1 | -| `master` branch commits | 14 (default version), 13 for DB library changes | 3.0 (default version) | -| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour) | 14 (default version), 13 for DB library changes | 3.0 (default version) | -| `maintenance` scheduled pipelines for the `ruby3_1` branch (every odd-numbered hour), see below. | 14 (default version), 13 for DB library changes | 3.1 | -| `nightly` scheduled pipelines for the `master` branch | 14 (default version), 13, 15 | 3.0 (default version) | - -There are 2 pipeline schedules used for testing Ruby 3.1. One is triggering a -pipeline in `ruby3_1-sync` branch, which updates the `ruby3_1` branch with latest -`master`, and no pipelines will be triggered by this push. The other schedule -is triggering a pipeline in `ruby3_1` 5 minutes after it, which is considered -the maintenance schedule to run test suites and update cache. - -The `ruby3_1` branch must not have any changes. The branch is only there to set -`RUBY_VERSION` to `3.1` in the maintenance pipeline schedule. - -The `gitlab` job in the `ruby3_1-sync` branch uses a `gitlab-org/gitlab` project -token with `write_repository` scope and `Maintainer` role with no expiration. -The token is stored in the `RUBY3_1_SYNC_TOKEN` variable in `gitlab-org/gitlab`. +| Merge requests | 14 (default version), 13 for DB library changes | 3.1 (default version) | +| `master` branch commits | 14 (default version), 13 for DB library changes | 3.1 (default version) | +| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour at XX:05) | 14 (default version), 13 for DB library changes | 3.1 (default version) | +| `maintenance` scheduled pipelines for the `ruby3_0` branch (every odd-numbered hour at XX:40) | 14 (default version), 13 for DB library changes | 3.0 | +| `maintenance` scheduled pipelines for the `ruby3_2` branch (every odd-numbered hour at XX:10) | 14 (default version), 13 for DB library changes | 3.2 | +| `nightly` scheduled pipelines for the `master` branch | 14 (default version), 13, 15 | 3.1 (default version) | + +For each current Ruby versions we're testing against with, we run +maintenance scheduled pipelines every 2 hours on their respective `ruby\d_\d` +branches. All these branches must not have any changes. These branches are +only there to run pipelines with their respective Ruby versions in the +scheduled maintenance pipelines. + +Additionally, we have scheduled pipelines running on `ruby-sync` branch also +every 2 hours, updating all the `ruby\d_\d` branches to be up-to-date with +the default branch `master`. No pipelines will be triggered by this push. + +The `gitlab` job in the `ruby-sync` branch uses a `gitlab-org/gitlab` project +token named `RUBY_SYNC` with `write_repository` scope and `Maintainer` role, +expiring on 2024-12-01. The token is stored in the `RUBY_SYNC_TOKEN` variable +in the pipeline schedule for `ruby-sync` branch. ### Redis versions testing @@ -692,9 +716,9 @@ We also run tests with a single database in nightly scheduled pipelines, and in Single database tests run in two modes: 1. **Single database with one connection**. Where GitLab connects to all the tables using one connection pool. -This runs through all the jobs that end with `-single-db` + This runs through all the jobs that end with `-single-db` 1. **Single database with two connections**. Where GitLab connects to `gitlab_main`, `gitlab_ci` database tables -using different database connections. This runs through all the jobs that end with `-single-db-ci-connection`. + using different database connections. This runs through all the jobs that end with `-single-db-ci-connection`. If you want to force tests to run with a single database, you can add the `pipeline:run-single-db` label to the merge request. @@ -747,28 +771,29 @@ graph LR ### Backend pipeline -[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/433316063). +[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/1118782302). ```mermaid graph RL; classDef criticalPath fill:#f66; - 1-3["compile-test-assets (5.5 minutes)"]; - class 1-3 criticalPath; + 1-1["clone-gitlab-repo (1 minute)"]; + 1-3["compile-test-assets (3 minutes)"]; click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" - 1-6["setup-test-env (3.6 minutes)"]; + 1-6["setup-test-env (4 minutes)"]; + class 1-6 criticalPath; click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" - 1-14["retrieve-tests-metadata"]; + 1-14["retrieve-tests-metadata (50 seconds)"]; click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" - 1-15["detect-tests"]; + 1-15["detect-tests (1 minute)"]; click 1-15 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=10113603&udv=1005715" - 2_5-1["rspec & db jobs (24 minutes)"]; + 2_5-1["rspec & db jobs (30~50 minutes)"]; class 2_5-1 criticalPath; click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" - 2_5-1 --> 1-3 & 1-6 & 1-14 & 1-15; + 2_5-1 --> 1-1 & 1-3 & 1-6 & 1-14 & 1-15; - ac-1["rspec:artifact-collector (2 minutes)<br/>(workaround for 'needs' limitation)"]; + ac-1["rspec:artifact-collector (30 seconds)<br/>(workaround for 'needs' limitation)"]; class ac-1 criticalPath; ac-1 --> 2_5-1; @@ -781,7 +806,6 @@ graph RL; class 4_3-1 criticalPath; click 4_3-1 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=13446492&udv=1005715" 4_3-1 --> 3_2-1; - ``` ### Review app pipeline diff --git a/doc/development/pipelines/internals.md b/doc/development/pipelines/internals.md index 1fdb2014c1f..16d0bfdfa30 100644 --- a/doc/development/pipelines/internals.md +++ b/doc/development/pipelines/internals.md @@ -48,6 +48,25 @@ from using `$FORCE_GITLAB_CI`. - [JiHu validation pipeline](https://about.gitlab.com/handbook/ceo/chief-of-staff-team/jihu-support/jihu-validation-pipelines.html) - [Gitaly downstream GitLab pipeline](https://gitlab.com/gitlab-org/gitaly/-/issues/4615) +See the next section for how we can enable pipelines without using +`$FORCE_GITLAB_CI`. + +#### Alternative to `$FORCE_GITLAB_CI` + +Essentially, we use different variables to enable different pipelines. +An example doing this is `$START_AS_IF_FOSS`. When we want to trigger a +cross project FOSS pipeline, we set `$START_AS_IF_FOSS`, along with a set of +other variables like `$ENABLE_RSPEC_UNIT`, `$ENABLE_RSPEC_SYSTEM`, and so on +so forth to enable each jobs we want to run in the as-if-foss cross project +downstream pipeline. + +The advantage of this over `$FORCE_GITLAB_CI` is that we have full control +over how we want to run the pipeline because `$START_AS_IF_FOSS` is only used +for this purpose, and changing how the pipeline behaves under this variable +will not affect other types of pipelines, while using `$FORCE_GITLAB_CI` we +do not know what exactly the pipeline is because it's used for multiple +purposes. + ## Default image The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). @@ -149,6 +168,7 @@ that are scoped to a single [configuration keyword](../../ci/yaml/index.md#job-k |------------------|-------------| | `.default-retry` | Allows a job to [retry](../../ci/yaml/index.md#retry) upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. | | `.default-before_script` | Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (for example, tests). | +| `.repo-from-artifacts` | Allows a job to fetch the repository from artifacts in `clone-gitlab-repo` instead of cloning. This should reduce GitLab.com Gitaly load and also slightly improve the speed because downloading from artifacts is faster than cloning. Note that this should be avoided to be used with jobs having `needs: []` because otherwise it'll start later and we normally want all jobs to start as soon as possible. Use this only on jobs which has other dependencies so that we don't wait longer than just cloning. Note that this behavior can be controlled via `CI_FETCH_REPO_GIT_STRATEGY`. See [Fetch repository via artifacts instead of cloning/fetching from Gitaly](performance.md#fetch-repository-via-artifacts-instead-of-cloningfetching-from-gitaly) for more details. | | `.setup-test-env-cache` | Allows a job to use a default `cache` definition suitable for setting up test environment for subsequent Ruby/Rails tasks. | | `.ruby-cache` | Allows a job to use a default `cache` definition suitable for Ruby tasks. | | `.static-analysis-cache` | Allows a job to use a default `cache` definition suitable for static analysis tasks. | @@ -199,6 +219,7 @@ and included in `rules` definitions via [YAML anchors](../../ci/yaml/yaml_optimi | `if-merge-request-title-as-if-foss` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-as-if-foss" | | | `if-merge-request-title-update-caches` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:update-cache". | | | `if-merge-request-labels-run-all-rspec` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-all-rspec". | | +| `if-merge-request-labels-run-cs-evaluation` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-CS-evaluation". | | | `if-security-merge-request` | Matches if the pipeline is for a security merge request. | | | `if-security-schedule` | Matches if the pipeline is for a security scheduled pipeline. | | | `if-nightly-master-schedule` | Matches if the pipeline is for a `master` scheduled pipeline with `$NIGHTLY` set. | | @@ -414,6 +435,8 @@ For this scenario, you have to: - `scripts/merge-simplecov` - `spec/simplecov_env_core.rb` - `spec/simplecov_env.rb` + - `prepare-as-if-foss-env` for: + - `scripts/setup/generate-as-if-foss-env.rb` Additionally, `scripts/utils.sh` is always downloaded from the API when this pattern is used (this file contains the code for `.fast-no-clone-job`). @@ -425,7 +448,7 @@ projects, only one of the following tags should be added to a job: - `gitlab-org`: Jobs randomly use privileged and unprivileged runners. - `gitlab-org-docker`: Jobs must use a privileged runner. If you need [Docker-in-Docker support](../../ci/docker/using_docker_build.md#use-docker-in-docker), -use `gitlab-org-docker` instead of `gitlab-org`. + use `gitlab-org-docker` instead of `gitlab-org`. The `gitlab-org-docker` tag is added by the `.use-docker-in-docker` job definition above. diff --git a/doc/development/pipelines/performance.md b/doc/development/pipelines/performance.md index 0cd4f4270fe..d9019e6053b 100644 --- a/doc/development/pipelines/performance.md +++ b/doc/development/pipelines/performance.md @@ -29,6 +29,45 @@ This works well for the following reasons: - We use [shallow clone](../../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) to avoid downloading the full Git history for every job. +### Fetch repository via artifacts instead of cloning/fetching from Gitaly + +Lately we see errors from Gitaly look like this: (see [the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/435456)) + +```plaintext +fatal: remote error: GitLab is currently unable to handle this request due to load. +``` + +While GitLab.com uses [pack-objects cache](../../administration/gitaly/configure_gitaly.md#pack-objects-cache), +sometimes the load is still too heavy for Gitaly to handle, and +[thundering herds](https://gitlab.com/gitlab-org/gitlab/-/issues/423830) can +also be a concern that we have a lot of jobs cloning the repository around +the same time. + +To mitigate and reduce loads for Gitaly, we changed some jobs to fetch the +repository from artifacts in a job instead of all cloning from Gitaly at once. + +For now this applies to most of the RSpec jobs, which has the most concurrent +jobs in most pipelines. This also slightly improved the speed because fetching +from the artifacts is also slightly faster than cloning, at the cost of saving +more artifacts for each pipeline. + +Based on the numbers on 2023-12-20 at [Fetch repo from artifacts for RSpec jobs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140330), +the extra storage cost was about 280M for each pipeline, and we save 15 seconds +for each RSpec jobs. + +We do not apply this to jobs having no other job dependencies because we don't +want to delay any jobs from starting. + +This behavior can be controlled by variable `CI_FETCH_REPO_GIT_STRATEGY`: + +- Set to `none` means jobs using `.repo-from-artifacts` fetch repository from + artifacts in job `clone-gitlab-repo` rather than cloning. +- Set to `clone` means jobs using `.repo-from-artifacts` clone repository + as usual. Job `clone-gitlab-repo` does not run in this case. + +To disable it, set `CI_FETCH_REPO_GIT_STRATEGY` to `clone`. To enable it, +set `CI_FETCH_REPO_GIT_STRATEGY` to `none`. + ## Caching strategy 1. All jobs must only pull caches by default. diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index 6a4a85f14ff..cef55c800e1 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -72,10 +72,10 @@ This section describes how to add new metrics for self-monitoring for [Prometheus metric names](https://prometheus.io/docs/practices/naming/#metric-names). 1. Update the list of [GitLab Prometheus metrics](../administration/monitoring/prometheus/gitlab_metrics.md). 1. Carefully choose what labels you want to add to your metric. Values with high cardinality, -like `project_path`, or `project_id` are strongly discouraged because they can affect our services -availability due to the fact that each set of labels is exposed as a new entry in the `/metrics` endpoint. -For example, a histogram with 10 buckets and a label with 100 values would generate 1000 -entries in the export endpoint. + like `project_path`, or `project_id` are strongly discouraged because they can affect our services + availability due to the fact that each set of labels is exposed as a new entry in the `/metrics` endpoint. + For example, a histogram with 10 buckets and a label with 100 values would generate 1000 + entries in the export endpoint. 1. Trigger the relevant page or code that records the new metric. 1. Check that the new metric appears at `/-/metrics`. diff --git a/doc/development/push_rules/index.md b/doc/development/push_rules/index.md index 343b199e613..96d16f5eb35 100644 --- a/doc/development/push_rules/index.md +++ b/doc/development/push_rules/index.md @@ -42,9 +42,6 @@ change the push behavior. - `EE::Gitlab::Checks::PushRules::FileSizeCheck`: Executes push rule checks related to file size rules. - Defined in `ee/lib/ee/gitlab/checks/push_rules/file_size_check.rb`. -- `EE::Gitlab::Checks::PushRules::SecretsCheck`: Executes push rule checks - related to secrets rules. - - Defined in `ee/lib/ee/gitlab/checks/push_rules/secrets_check.rb`. - `EE::Gitlab::Checks::PushRules::TagCheck`: Executes push rule checks related to tag rules. - Defined in `ee/lib/ee/gitlab/checks/push_rules/tag_check.rb`. @@ -83,11 +80,9 @@ graph TD EE::Gitlab::Checks::PushRuleCheck -->|Only if pushing to a tag| EE::Gitlab::Checks::PushRules::TagCheck EE::Gitlab::Checks::PushRuleCheck -->|Only if pushing to a branch| EE::Gitlab::Checks::PushRules::BranchCheck EE::Gitlab::Checks::PushRuleCheck --> EE::Gitlab::Checks::PushRules::FileSizeCheck - EE::Gitlab::Checks::PushRuleCheck --> EE::Gitlab::Checks::PushRules::SecretsCheck - EE::Gitlab::Checks::PushRuleCheck --> EE::Gitlab::Checks::PushRules::SecretsCheck ``` NOTE: The `PushRuleCheck` only triggers checks in parallel if the `parallel_push_checks` feature flag is enabled. Otherwise tag or branch check -runs first, then file size, then secrets. +runs first, then file size. diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index 00110d21dc0..dfe1e2d1b05 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -255,14 +255,14 @@ self.reactive_cache_hard_limit = 5.megabytes #### `self.reactive_cache_work_type` - This is the type of work performed by the `calculate_reactive_cache` method. Based on this attribute, -it's able to pick the right worker to process the caching job. Make sure to -set it as `:external_dependency` if the work performs any external request -(for example, Kubernetes, Sentry); otherwise set it to `:no_dependency`. + it's able to pick the right worker to process the caching job. Make sure to + set it as `:external_dependency` if the work performs any external request + (for example, Kubernetes, Sentry); otherwise set it to `:no_dependency`. #### `self.reactive_cache_worker_finder` - This is the method used by the background worker to find or generate the object on -which `calculate_reactive_cache` can be called. + which `calculate_reactive_cache` can be called. - By default it uses the model primary key to find the object: ```ruby diff --git a/doc/development/remote_development/index.md b/doc/development/remote_development/index.md new file mode 100644 index 00000000000..3d85c44961c --- /dev/null +++ b/doc/development/remote_development/index.md @@ -0,0 +1,16 @@ +--- +stage: Create +group: Web IDE +info: Any user with at least the Maintainer role can merge updates to this content. For details, see https://docs.gitlab.com/ee/development/development_processes.html#development-guidelines-review. +--- + +# Remote Development developer guidelines + +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105783) in GitLab 16.0. + +## Workspaces feature developer documentation + +Currently, the majority of the developer documentation for the [Remote Development Workspaces feature](../../user/workspace/index.md) +is located in the separate [`gitlab-remote-development-docs` project](https://gitlab.com/gitlab-org/remote-development/gitlab-remote-development-docs/-/blob/main/README.md). + +Parts of that documentation will eventually be migrated here. diff --git a/doc/development/ruby_upgrade.md b/doc/development/ruby_upgrade.md index 110bc6076b0..d1dd6d793d4 100644 --- a/doc/development/ruby_upgrade.md +++ b/doc/development/ruby_upgrade.md @@ -50,31 +50,31 @@ To help you estimate the scope of future upgrades, see the efforts required for Before any upgrade, consider all audiences and targets, ordered by how immediately they are affected by Ruby upgrades: 1. **Developers.** We have many contributors to GitLab and related projects both inside and outside the company. Changing files such as `.ruby-version` affects everyone using tooling that interprets these files. -The developers are affected as soon as they pull from the repository containing the merged changes. + The developers are affected as soon as they pull from the repository containing the merged changes. 1. **GitLab CI/CD.** We heavily lean on CI/CD for code integration and testing. CI/CD jobs do not interpret files such as `.ruby-version`. -Instead, they use the Ruby installed in the Docker container they execute in, which is defined in `.gitlab-ci.yml`. -The container images used in these jobs are maintained in the [`gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) repository. -When we merge an update to an image, CI/CD jobs are affected as soon as the [image is built](https://gitlab.com/gitlab-org/gitlab-build-images/#pushing-a-rebuild-image). + Instead, they use the Ruby installed in the Docker container they execute in, which is defined in `.gitlab-ci.yml`. + The container images used in these jobs are maintained in the [`gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) repository. + When we merge an update to an image, CI/CD jobs are affected as soon as the [image is built](https://gitlab.com/gitlab-org/gitlab-build-images/#pushing-a-rebuild-image). 1. **GitLab SaaS**. GitLab.com is deployed from customized Helm charts that use Docker images from [Cloud Native GitLab (CNG)](https://gitlab.com/gitlab-org/build/CNG). -Just like CI/CD, `.ruby-version` is meaningless in this environment. Instead, those Docker images must be patched to upgrade Ruby. -GitLab SaaS is affected with the next deployment. + Just like CI/CD, `.ruby-version` is meaningless in this environment. Instead, those Docker images must be patched to upgrade Ruby. + GitLab SaaS is affected with the next deployment. 1. **Self-managed GitLab.** Customers installing GitLab via [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab) use none of the above. -Instead, their Ruby version is defined by the [Ruby software bundle](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/config/software/ruby.rb) in Omnibus. -Self-managed customers are affected as soon as they upgrade to the release containing this change. + Instead, their Ruby version is defined by the [Ruby software bundle](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/config/software/ruby.rb) in Omnibus. + Self-managed customers are affected as soon as they upgrade to the release containing this change. ## Ruby upgrade approach Timing all steps in a Ruby upgrade correctly is critical. As a general guideline, consider the following: - For smaller upgrades where production behavior is unlikely to change, aim to keep the version gap between -repositories and production minimal. Coordinate with stakeholders to merge all changes closely together -(within a day or two) to avoid drift. In this scenario the likely order is to upgrade developer tooling and -environments first, production second. + repositories and production minimal. Coordinate with stakeholders to merge all changes closely together + (within a day or two) to avoid drift. In this scenario the likely order is to upgrade developer tooling and + environments first, production second. - For larger changes, the risk of going to production with a new Ruby is significant. In this case, try to get into a -position where all known incompatibilities with the new Ruby version are already fixed, then work -with production engineers to deploy the new Ruby to a subset of the GitLab production fleet. In this scenario -the likely order is to update production first, developer tooling and environments second. This makes rollbacks -easier in case of critical regressions in production. + position where all known incompatibilities with the new Ruby version are already fixed, then work + with production engineers to deploy the new Ruby to a subset of the GitLab production fleet. In this scenario + the likely order is to update production first, developer tooling and environments second. This makes rollbacks + easier in case of critical regressions in production. Either way, we found that from past experience the following approach works well, with some steps likely only necessary for minor and major upgrades. Note that some of these steps can happen in parallel or may have their @@ -110,17 +110,17 @@ for a smoother transition by supporting both old and new Ruby versions for a per There are two places that require changes: 1. **[GitLab Build Images](https://gitlab.com/gitlab-org/gitlab-build-images).** These are Docker images -we use for runners and other Docker-based pre-production environments. The kind of change necessary -depends on the scope. + we use for runners and other Docker-based pre-production environments. The kind of change necessary + depends on the scope. - For [patch level updates](https://gitlab.com/gitlab-org/gitlab-build-images/-/merge_requests/418), it should suffice to increment the patch level of `RUBY_VERSION`. -All projects building against the same minor release automatically download the new patch release. + All projects building against the same minor release automatically download the new patch release. - For [major and minor updates](https://gitlab.com/gitlab-org/gitlab-build-images/-/merge_requests/320), create a new set of Docker images that can be used side-by-side with existing images during the upgrade process. **Important:** Make sure to copy over all Ruby patch files -in the `/patches` directory to a new folder matching the Ruby version you upgrade to, or they aren't applied. + in the `/patches` directory to a new folder matching the Ruby version you upgrade to, or they aren't applied. 1. **[GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit).** -Update GDK to add the new Ruby as an additional option for -developers to choose from. This typically only requires it to be appended to `.tool-versions` so `asdf` -users will benefit from this. Other users will have to install it manually -([example](https://gitlab.com/gitlab-org/gitlab-development-kit/-/merge_requests/2136).) + Update GDK to add the new Ruby as an additional option for + developers to choose from. This typically only requires it to be appended to `.tool-versions` so `asdf` + users will benefit from this. Other users will have to install it manually + ([example](https://gitlab.com/gitlab-org/gitlab-development-kit/-/merge_requests/2136).) For larger version upgrades, consider working with [Quality Engineering](https://about.gitlab.com/handbook/engineering/quality/) to identify and set up a test plan. @@ -265,16 +265,16 @@ For GitLab SaaS, GitLab team members can inspect these log events in Kibana During the upgrade process, consider the following recommendations: - **Front-load as many changes as possible.** Especially for minor and major releases, it is likely that application -code will break or change. Any changes that are backward compatible should be merged into the main branch and -released independently ahead of the Ruby version upgrade. This ensures that we move in small increments and -get feedback from production environments early. + code will break or change. Any changes that are backward compatible should be merged into the main branch and + released independently ahead of the Ruby version upgrade. This ensures that we move in small increments and + get feedback from production environments early. - **Create an experimental branch for larger updates.** We generally try to avoid long-running topic branches, -but for purposes of feedback and experimentation, it can be useful to have such a branch to get regular -feedback from CI/CD when running a newer Ruby. This can be helpful when first assessing what problems -we might run into, as [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50640) demonstrates. -These experimental branches are not intended to be merged; they can be closed once all required changes have been broken out -and merged back independently. + but for purposes of feedback and experimentation, it can be useful to have such a branch to get regular + feedback from CI/CD when running a newer Ruby. This can be helpful when first assessing what problems + we might run into, as [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50640) demonstrates. + These experimental branches are not intended to be merged; they can be closed once all required changes have been broken out + and merged back independently. - **Give yourself enough time to fix problems ahead of a milestone release.** GitLab moves fast. -As a Ruby upgrade requires many MRs to be sent and reviewed, make sure all changes are merged at least a week -before release day. This gives us extra time to act if something breaks. If in doubt, it is better to -postpone the upgrade to the following month, as we [prioritize availability over velocity](https://about.gitlab.com/handbook/engineering/development/principles/#prioritizing-technical-decisions). + As a Ruby upgrade requires many MRs to be sent and reviewed, make sure all changes are merged at least a week + before release day. This gives us extra time to act if something breaks. If in doubt, it is better to + postpone the upgrade to the following month, as we [prioritize availability over velocity](https://about.gitlab.com/handbook/engineering/development/principles/#prioritizing-technical-decisions). diff --git a/doc/development/runner_fleet_dashboard.md b/doc/development/runner_fleet_dashboard.md index 70499e5a087..77c89f2adb5 100644 --- a/doc/development/runner_fleet_dashboard.md +++ b/doc/development/runner_fleet_dashboard.md @@ -1,197 +1,11 @@ --- -stage: Verify -group: Runner -info: >- - To determine the technical writer assigned to the Stage/Group associated with - this page, see - https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../ci/runners/runner_fleet_dashboard.md' +remove_date: '2024-03-01' --- -# Runner Fleet Dashboard **(ULTIMATE EXPERIMENT)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/424495) in GitLab 16.6 behind several [feature flags](#enable-feature-flags). +This document was moved to [another location](../ci/runners/runner_fleet_dashboard.md). -This feature is an [Experiment](../policy/experiment-beta-support.md). -To join the list of users testing this feature, contact us in -[epic 11180](https://gitlab.com/groups/gitlab-org/-/epics/11180). - -GitLab administrators can use the Runner Fleet Dashboard to assess the health of your instance runners. -The Runner Fleet Dashboard shows: - -- Recent CI errors related caused by runner infrastructure. -- Number of concurrent jobs executed on most busy runners. -- Histogram of job queue times (available only with ClickHouse). - -There is a proposal to introduce [more features](#whats-next) to the Runner Fleet Dashboard. - - - -## View the Runner Fleet Dashboard - -Prerequisites: - -- You must be an administrator. - -To view the runner fleet dashboard: - -1. On the left sidebar, at the bottom, select **Admin Area**. -1. Select **Runners**. -1. Click **Fleet dashboard**. - -Most of the dashboard works without any additional actions, with the -exception of **Wait time to pick a job** chart and [proposed features](#whats-next). -These features require setting up an additional infrastructure, described in this page. - -To test the Runner Fleet Dashboard and gather feedback, we have launched an early adopters program -for some customers to try this feature. - -## Requirements - -To test the Runner Fleet Dashboard as part of the early adopters program, you must: - -- Run GitLab 16.7 or above. -- Have an [Ultimate license](https://about.gitlab.com/pricing/). -- Be able to run ClickHouse database. We recommend using [ClickHouse Cloud](https://clickhouse.cloud/). - -## Setup - -To setup ClickHouse as the GitLab data storage: - -1. [Run ClickHouse Cluster and configure database](#run-and-configure-clickhouse). -1. [Configure GitLab connection to Clickhouse](#configure-the-gitlab-connection-to-clickhouse). -1. [Run ClickHouse migrations](#run-clickhouse-migrations). -1. [Enable the feature flags](#enable-feature-flags). - -### Run and configure ClickHouse - -The most straightforward way to run ClickHouse is with [ClickHouse Cloud](https://clickhouse.cloud/). -You can also [run ClickHouse on your own server](https://clickhouse.com/docs/en/install). Refer to the ClickHouse -documentation regarding [recommendations for self-managed instances](https://clickhouse.com/docs/en/install#recommendations-for-self-managed-clickhouse). - -When you run ClickHouse on a hosted server, various data points might impact the resource consumption, like the number -of builds that run on your instance each month, the selected hardware, the data center choice to host ClickHouse, and more. -Regardless, the cost should not be significant. - -NOTE: -ClickHouse is a secondary data store for GitLab. All your data is still stored in Postgres, -and only duplicated in ClickHouse for analytics purposes. - -To create necessary user and database objects: - -1. Generate a secure password and save it. -1. Sign in to the ClickHouse SQL console. -1. Execute the following command. Replace `PASSWORD_HERE` with the generated password. - - ```sql - CREATE DATABASE gitlab_clickhouse_main_production; - CREATE USER gitlab IDENTIFIED WITH sha256_password BY 'PASSWORD_HERE'; - CREATE ROLE gitlab_app; - GRANT SELECT, INSERT, ALTER, CREATE, UPDATE, DROP, TRUNCATE, OPTIMIZE ON gitlab_clickhouse_main_production.* TO gitlab_app; - GRANT gitlab_app TO gitlab; - ``` - -### Configure the GitLab connection to ClickHouse - -::Tabs - -:::TabTitle Linux package - -To provide GitLab with ClickHouse credentials: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['clickhouse_databases']['main']['database'] = 'gitlab_clickhouse_main_production' - gitlab_rails['clickhouse_databases']['main']['url'] = 'https://example.com/path' - gitlab_rails['clickhouse_databases']['main']['username'] = 'gitlab' - gitlab_rails['clickhouse_databases']['main']['password'] = 'PASSWORD_HERE' # replace with the actual password - ``` - -1. Save the file and reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -:::TabTitle Helm chart (Kubernetes) - -1. Save the ClickHouse password as a Kubernetes Secret: - - ```shell - kubectl create secret generic gitlab-clickhouse-password --from-literal="main_password=PASSWORD_HERE" - ``` - -1. Export the Helm values: - - ```shell - helm get values gitlab > gitlab_values.yaml - ``` - -1. Edit `gitlab_values.yaml`: - - ```yaml - global: - clickhouse: - enabled: true - main: - username: default - password: - secret: gitlab-clickhouse-password - key: main_password - database: gitlab_clickhouse_main_production - url: 'http://example.com' - ``` - -1. Save the file and apply the new values: - - ```shell - helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab - ``` - -::EndTabs - -To verify that your connection is set up successfully: - -1. Log in to [Rails console](../administration/operations/rails_console.md#starting-a-rails-console-session) -1. Execute the following: - - ```ruby - ClickHouse::Client.select('SELECT 1', :main) - ``` - - If successful, the command returns `[{"1"=>1}]` - -### Run ClickHouse migrations - -To create the required database objects execute: - -```shell -sudo gitlab-rake gitlab:clickhouse:migrate -``` - -### Enable feature flags - -Features that use ClickHouse are currently under development and are disabled by feature flags. - -To enable these features, [enable](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) -the following feature flags: - -| Feature flag name | Purpose | -|------------------------------------|---------------------------------------------------------------------------| -| `ci_data_ingestion_to_click_house` | Enables synchronization of new finished CI builds to Clickhouse database. | -| `clickhouse_ci_analytics` | Enables the **Wait time to pick a job** chart. | - -## What's next - -Support for usage and cost analysis are proposed in -[epic 11183](https://gitlab.com/groups/gitlab-org/-/epics/11183). - -## Feedback - -To help us improve the Runner Fleet Dashboard, you can provide feedback in -[issue 421737](https://gitlab.com/gitlab-org/gitlab/-/issues/421737). -In particular: - -- How easy or difficult it was to setup GitLab to make the dashboard work. -- How useful you found the dashboard. -- What other information you would like to see on that dashboard. -- Any other related thoughts and ideas. +<!-- This redirect file can be deleted after <2024-03-01>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/search/advanced_search_migration_styleguide.md b/doc/development/search/advanced_search_migration_styleguide.md index 87083d1a36f..b79ab6561d5 100644 --- a/doc/development/search/advanced_search_migration_styleguide.md +++ b/doc/development/search/advanced_search_migration_styleguide.md @@ -225,6 +225,53 @@ class MigrationName < Elastic::Migration end ``` +#### `Search::Elastic::MigrationDeleteBasedOnSchemaVersion` + +Deletes all documents in the index that stores the specified document type and has `schema_version` less than the given value. + +Requires the `DOCUMENT_TYPE` constant and `schema_version` method. +The index mapping must have a `schema_version` integer field in a `YYMM` format. + +```ruby +class MigrationName < Elastic::Migration + include ::Search::Elastic::MigrationDeleteBasedOnSchemaVersion + + DOCUMENT_TYPE = Issue + + batch_size 10_000 + batched! + throttle_delay 1.minute + retry_on_failure + + def schema_version + 23_12 + end +end +``` + +#### `Search::Elastic::MigrationDatabaseBackfillHelper` + +Reindexes all documents in the database to the elastic search index respecting the `limited_indexing` setting. + +Requires the `DOCUMENT_TYPE` constant and `respect_limited_indexing?` method. + +```ruby +class MigrationName < Elastic::Migration + include ::Search::Elastic::MigrationDatabaseBackfillHelper + + batch_size 10_000 + batched! + throttle_delay 1.minute + retry_on_failure + + DOCUMENT_TYPE = Issue + + def respect_limited_indexing? + true + end +end +``` + #### `Elastic::MigrationHelper` Contains methods you can use when a migration doesn't fit the previous examples. diff --git a/doc/development/sec/analyzer_development_guide.md b/doc/development/sec/analyzer_development_guide.md index eb59d8fcaf5..843f41300a2 100644 --- a/doc/development/sec/analyzer_development_guide.md +++ b/doc/development/sec/analyzer_development_guide.md @@ -128,7 +128,7 @@ To use Docker with `replace` in the `go.mod` file: 1. Copy the contents of `command` into the directory of the analyzer. `cp -r /path/to/command path/to/analyzer/command`. 1. Add a copy statement in the analyzer's `Dockerfile`: `COPY command /command`. 1. Update the `replace` statement to make sure it matches the destination of the `COPY` statement in the step above: -`replace gitlab.com/gitlab-org/security-products/analyzers/command/v3 => /command` + `replace gitlab.com/gitlab-org/security-products/analyzers/command/v3 => /command` ## Analyzer scripts @@ -189,11 +189,11 @@ are integrated with the existing application, iteration should not be blocked by 1. Ensure that the release source (typically the `master` or `main` branch) has a passing pipeline. 1. Create a new release for the analyzer project by selecting the **Deployments** menu on the left-hand side of the project window, then selecting the **Releases** sub-menu. 1. Select **New release** to open the **New Release** page. - 1. In the **Tag name** drop down, enter the same version used in the `CHANGELOG.md`, for example `v2.4.2`, and select the option to create the tag (`Create tag v2.4.2` here). - 1. In the **Release title** text box enter the same version used above, for example `v2.4.2`. - 1. In the `Release notes` text box, copy and paste the notes from the corresponding version in the `CHANGELOG.md`. - 1. Leave all other settings as the default values. - 1. Select **Create release**. + 1. In the **Tag name** drop down, enter the same version used in the `CHANGELOG.md`, for example `v2.4.2`, and select the option to create the tag (`Create tag v2.4.2` here). + 1. In the **Release title** text box enter the same version used above, for example `v2.4.2`. + 1. In the `Release notes` text box, copy and paste the notes from the corresponding version in the `CHANGELOG.md`. + 1. Leave all other settings as the default values. + 1. Select **Create release**. After following the above process and creating a new release, a new Git tag is created with the `Tag name` provided above. This triggers a new pipeline with the given tag version and a new analyzer Docker image is built. @@ -229,7 +229,7 @@ After the above steps have been completed, the automatic release process execute 1. After a new version of the analyzer Docker image has been tagged and deployed, test it with the corresponding test project. 1. Announce the release on the relevant group Slack channel. Example message: - > FYI I've just released `ANALYZER_NAME` `ANALYZER_VERSION`. `LINK_TO_RELEASE` + > FYI I've just released `ANALYZER_NAME` `ANALYZER_VERSION`. `LINK_TO_RELEASE` **Never delete a Git tag that has been pushed** as there is a good chance that the tag will be used and/or cached by the Go package registry. diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index a575d1ff890..d8fad6deb9c 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -182,7 +182,7 @@ For other regular expressions, here are a few guidelines: - If there's a clean non-regex solution, such as `String#start_with?`, consider using it - Ruby supports some advanced regex features like [atomic groups](https://www.regular-expressions.info/atomic.html) -and [possessive quantifiers](https://www.regular-expressions.info/possessive.html) that eliminate backtracking + and [possessive quantifiers](https://www.regular-expressions.info/possessive.html) that eliminate backtracking - Avoid nested quantifiers if possible (for example `(a+)+`) - Try to be as precise as possible in your regex and avoid the `.` if there's an alternative - For example, Use `_[^_]+_` instead of `_.*_` to match `_text here_` diff --git a/doc/development/testing_guide/end_to_end/execution_context_selection.md b/doc/development/testing_guide/end_to_end/execution_context_selection.md index f625dc466b9..f940cf289d9 100644 --- a/doc/development/testing_guide/end_to_end/execution_context_selection.md +++ b/doc/development/testing_guide/end_to_end/execution_context_selection.md @@ -48,6 +48,7 @@ Matches use: | The `nightly` and `canary` pipelines | `only: { pipeline: [:nightly, :canary] }` | ["nightly scheduled pipeline"](https://gitlab.com/gitlab-org/gitlab/-/pipeline_schedules) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | | The `ee:instance` job | `only: { job: 'ee:instance' }` | The `ee:instance` job in any pipeline | | Any `quarantine` job | `only: { job: '.*quarantine' }` | Any job ending in `quarantine` in any pipeline | +| Local development environment | `only: :local` | Any environment where `Runtime::Env.running_in_ci?` is false | | Any run where condition evaluates to a truthy value | `only: { condition: -> { ENV['TEST_ENV'] == 'true' } }` | Any run where `TEST_ENV` is set to true ```ruby diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index e11119d2c0b..189b21ea607 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -24,14 +24,14 @@ Be sure to include the `feature_flag` tag so that the test can be skipped on the - Format: `feature_flag: { name: 'feature_flag_name' }` - Used only for informational purposes at this time. It should be included to help quickly determine what -feature flag is under test. + feature flag is under test. `scope` - Format: `feature_flag: { name: 'feature_flag_name', scope: :project }` - When `scope` is set to `:global`, the test will be **skipped on all live .com environments**. This is to avoid issues with feature flag changes affecting other tests or users on that environment. - When `scope` is set to any other value (such as `:project`, `:group` or `:user`), or if no `scope` is specified, the test will only be **skipped on canary, production, and pre-production**. -This is due to the fact that administrator access is not available there. + This is due to the fact that administrator access is not available there. **WARNING:** You are strongly advised to first try and [enable feature flags only for a group, project, user](../../feature_flags/index.md#feature-actors), or [feature group](../../feature_flags/index.md#feature-groups). diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index aaea3f8958d..eabc722a26f 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -303,6 +303,75 @@ Geo requires an EE license. To visit the Geo sites in your browser, you need a r - You can increase the wait time for replication by setting `GEO_MAX_FILE_REPLICATION_TIME` and `GEO_MAX_DB_REPLICATION_TIME`. The default is 120 seconds. - To save time during tests, create a Personal Access Token with API access on the Geo primary node, and pass that value in as `GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN`. +## Group SAML Tests + +Tests that are tagged with `:group_saml` meta are orchestrated tests where the user accesses a group via SAML SSO. + +These tests depend on a SAML IDP Docker container ([jamedjo/test-SAML-idp](https://hub.docker.com/r/jamedjo/test-saml-idp)). The tests spin up the container themselves. + +To run these tests on your computer against the GDK: + +1. Add these settings to your `gitlab.yml` file: + + ```yaml + omniauth: + enabled: true + providers: + - { name: 'group_saml' } + ``` + +1. Run a group SAML test from [`gitlab/qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/d5447ebb5f99d4c72780681ddf4dc25b0738acba/qa) directory: + + ```shell + QA_DEBUG=true CHROME_HEADLESS=false bundle exec bin/qa Test::Instance::All http://localhost:3000 qa/specs/features/ee/browser_ui/1_manage/group/group_saml_enforced_sso_spec.rb -- --tag orchestrated + ``` + +For instructions on how to run these tests using the `gitlab-qa` gem, refer to [the GitLab QA documentation](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#testintegrationgroupsaml-eefull-image-address). + +## Instance SAML Tests + +Tests that are tagged with `:instance_saml` meta are orchestrated tests where the instance level sign-in happens using SAML SSO. + +These tests require a SAML IDP Docker container ([jamedjo/test-SAML-idp](https://hub.docker.com/r/jamedjo/test-saml-idp)) to be configured and running. + +To run these tests on your computer against the GDK: + +1. Add these settings to your `gitlab.yml` file: + + ```yaml + omniauth: + enabled: true + allow_single_sign_on: ["saml"] + block_auto_created_users: false + auto_link_saml_user: true + providers: + - { name: 'saml', + args: { + assertion_consumer_service_url: 'http://gdk.test:3000/users/auth/saml/callback', + idp_cert_fingerprint: '11:9b:9e:02:79:59:cd:b7:c6:62:cf:d0:75:d9:e2:ef:38:4e:44:5f', + idp_sso_target_url: 'https://gdk.test:8443/simplesaml/saml2/idp/SSOService.php', + issuer: 'http://gdk.test:3000', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' + } } + ``` + +1. Start the SAML IDP Docker container: + + ```shell + docker run --name=group_saml_qa_idp -p 8080:8080 -p 8443:8443 \ + -e SIMPLESAMLPHP_SP_ENTITY_ID=http://localhost:3000 \ + -e SIMPLESAMLPHP_SP_ASSERTION_CONSUMER_SERVICE=http://localhost:3000/users/auth/saml/callback \ + -d jamedjo/test-saml-idp + ``` + +1. Run the test from [`gitlab/qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/d5447ebb5f99d4c72780681ddf4dc25b0738acba/qa) directory: + + ```shell + QA_DEBUG=true CHROME_HEADLESS=false bundle exec bin/qa Test::Instance::All http://localhost:3000 qa/specs/features/browser_ui/1_manage/login/login_via_instance_wide_saml_sso_spec.rb -- --tag orchestrated + ``` + +For instructions on how to run these tests using the `gitlab-qa` gem, refer to [the GitLab QA documentation](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#testintegrationinstancesaml-ceeefull-image-address). + ## LDAP Tests Tests that are tagged with `:ldap_tls` and `:ldap_no_tls` meta are orchestrated tests where the sign-in happens via LDAP. diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 1895b9bdb39..4d6fac57c06 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -119,7 +119,7 @@ Adding a delay in API or controller could help reproducing the issue. - [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101728/diffs): A CSS selector only appears after a GraphQL requests has finished, and the UI has updated. - [Example 3](https://gitlab.com/gitlab-org/gitlab/-/issues/408215): A false-positive test, Capybara immediately returns true after -page visit and page is not fully loaded, or if the element is not detectable by webdriver (such as being rendered outside the viewport or behind other elements). + page visit and page is not fully loaded, or if the element is not detectable by webdriver (such as being rendered outside the viewport or behind other elements). ### Datetime-sensitive diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 81399c7ce05..99852645914 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -349,7 +349,7 @@ possible). | Tests path | Testing engine | Notes | | ---------- | -------------- | ----- | -| `spec/features/` | [Capybara](https://github.com/teamcapybara/capybara) + [RSpec](https://github.com/rspec/rspec-rails#feature-specs) | If your test has the `:js` metadata, the browser driver is [Poltergeist](https://github.com/teamcapybara/capybara#poltergeist), otherwise it's using [RackTest](https://github.com/teamcapybara/capybara#racktest). | +| `spec/features/` | [Capybara](https://github.com/teamcapybara/capybara) + [RSpec](https://github.com/rspec/rspec-rails#feature-specs) | If your test has the `:js` metadata, the browser driver is [Selenium](https://github.com/teamcapybara/capybara#selenium), otherwise it's using [RackTest](https://github.com/teamcapybara/capybara#racktest). | ### Frontend feature tests diff --git a/doc/development/ux/index.md b/doc/development/ux/index.md index ab6cd4c64f5..de4491f997c 100644 --- a/doc/development/ux/index.md +++ b/doc/development/ux/index.md @@ -20,7 +20,7 @@ To contribute to [Pajamas design system](https://design.gitlab.com/) and the [UI ## Contributing to other issues -1. Review the list of available issues that are currently [accepting UX contribution](https://gitlab.com/groups/gitlab-org/-/issues/?sort=weight&state=opened&label_name%5B%5D=UX&label_name%5B%5D=workflow%3A%3Aready%20for%20design&label_name%5B%5D=Accepting%20UX%20contributions&first_page_size=20). +1. Review the list of available UX issues that are currently [seeking community contribution](https://gitlab.com/groups/gitlab-org/-/issues/?sort=weight&state=opened&label_name%5B%5D=UX&label_name%5B%5D=Seeking%20community%20contributions&first_page_size=100). 1. Find an issue that does not have an Assignee to ensure someone else is not working on a solution. Add the `~"workflow::design"` and `~"Community contribution"` labels and mention `@gitlab-com/gitlab-ux/reviewers` to request they assign the issue to you. 1. Add your design proposal to the issue description/[design management](../../user/project/issues/design_management.md) section. Remember to keep the scope of the proposal/change small following our [MVCs guidelines](https://about.gitlab.com/handbook/values/#minimal-viable-change-mvc). 1. If you have any questions or are ready for a review of your proposal, mention `@gitlab-com/gitlab-ux/reviewers` in a comment to make your request. diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md index 725c8aa45d2..83cfb3fb385 100644 --- a/doc/development/value_stream_analytics.md +++ b/doc/development/value_stream_analytics.md @@ -28,7 +28,7 @@ Apart from the durations, we expose the record count within a stage. ## Feature availability - Group level (licensed): Requires Ultimate or Premium subscription. This version is the most -feature-full. + feature-full. - Project level (licensed): We are continually adding features to project level VSA to bring it in line with group level VSA. - Project level (FOSS): Keep it as is. diff --git a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md index 53c6721b01f..7386a83afc1 100644 --- a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md +++ b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md @@ -45,7 +45,7 @@ Benefits of the aggregated VSA backend: - Possibility to introduce further aggregations for improving the first page load time. - Better performance for large groups (with many subgroups, projects, issues and, merge requests). - Ready for database decomposition. The VSA related database tables could live in a separate -database with a minimal development effort. + database with a minimal development effort. - Ready for keyset pagination which can be useful for exporting the data. - Possibility to implement more complex event definitions. - For example, the start event can be two timestamp columns where the earliest value would be @@ -107,8 +107,7 @@ the service performs operations in batches and enforces strict application limit - Load records in batches. - Insert records in batches. -- Stop processing when a limit is reached, schedule a background job to continue the processing -later. +- Stop processing when a limit is reached, schedule a background job to continue the processing later. - Continue processing data from a specific point. As of GitLab 14.7, the data loading is done manually. Once the feature is ready, the service is @@ -267,16 +266,15 @@ database tables. This change could be implemented using array columns. The feature uses private JSON APIs for delivering the data to the frontend. On the first page load , the following requests are invoked: -- Initial HTML page load which is mostly empty. Some configuration data is exposed via `data` -attributes. +- Initial HTML page load which is mostly empty. Some configuration data is exposed via `data` attributes. - `value_streams` - Load the available value streams for the given group. - `stages` - Load the stages for the currently selected value stream. - `median` - For each stage, request the median duration. - `count` - For each stage, request the number of items in the stage (this is a -[limit count](../merge_request_concepts/performance.md#badge-counters), maximum 1000 rows). + [limit count](../merge_request_concepts/performance.md#badge-counters), maximum 1000 rows). - `average_duration_chart` - Data for the duration chart. - `summary`, `time_summary` - Top-level aggregations, most of the metrics are using different APIs/ -finders and not invoking the aggregated backend. + finders and not invoking the aggregated backend. When selecting a specific stage, the `records` endpoint is invoked, which returns the related records (paginated) for the chosen stage in a specific order. diff --git a/doc/development/work_items.md b/doc/development/work_items.md index 0c3bc4611f5..7fc0c0d9d21 100644 --- a/doc/development/work_items.md +++ b/doc/development/work_items.md @@ -245,7 +245,8 @@ Keep the following in mind when you write your migration: necessary if the new work item type is going to use the `Hierarchy` widget. In this table, you must add what work item type can have children and of what type. Also, you should specify the hierarchy depth for work items of the same type. By default a cross-hierarchy (cross group or project) relationship is disabled when creating new restrictions but - it can be enabled by specifying a value for `cross_hierarchy_enabled`. + it can be enabled by specifying a value for `cross_hierarchy_enabled`. Due to the restrictions being cached for the work item type, it's also + required to call `clear_reactive_cache!` on the associated work item types. - Optional. Create linked item restrictions. - Similarly to the `Hierarchy` widget, the `Linked items` widget also supports rules defining which work item types can be linked to other types. A restriction can specify if the source type can be related to or blocking a target type. Current restrictions: |
