diff options
Diffstat (limited to 'doc/development')
119 files changed, 3868 insertions, 1958 deletions
diff --git a/doc/development/adding_service_component.md b/doc/development/adding_service_component.md index 314065ffc10..6e47d2991dc 100644 --- a/doc/development/adding_service_component.md +++ b/doc/development/adding_service_component.md @@ -70,7 +70,7 @@ New services to be bundled with GitLab need to be available in the following env The first step of bundling a new service is to provide it in the development environment to engage in collaboration and feedback. - [Include in the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) -- [Include in the source install instructions](../install/installation.md) +- [Include in the self-compiled installation instructions](../install/installation.md) **Standard install methods** diff --git a/doc/development/advanced_search.md b/doc/development/advanced_search.md index 30e1874f1ed..805459cb4ee 100644 --- a/doc/development/advanced_search.md +++ b/doc/development/advanced_search.md @@ -52,9 +52,9 @@ during indexing and searching operations. Some of the benefits and tradeoffs to - Routing is not used if too many shards would be hit for global and group scoped searches. - Shard size imbalance might occur. -## Existing Analyzers/Tokenizers/Filters +## Existing analyzers and tokenizers -These are all defined in [`ee/lib/elastic/latest/config.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/elastic/latest/config.rb) +The following analyzers and tokenizers are defined in [`ee/lib/elastic/latest/config.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/elastic/latest/config.rb). ### Analyzers @@ -72,7 +72,7 @@ Please see the `sha_tokenizer` explanation later below for an example. #### `code_analyzer` -Used when indexing a blob's filename and content. Uses the `whitespace` tokenizer and the filters: [`code`](#code), `lowercase`, and `asciifolding` +Used when indexing a blob's filename and content. Uses the `whitespace` tokenizer and the [`word_delimiter_graph`](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-word-delimiter-graph-tokenfilter.html), `lowercase`, and `asciifolding` filters. The `whitespace` tokenizer was selected to have more control over how tokens are split. For example the string `Foo::bar(4)` needs to generate tokens like `Foo` and `bar(4)` to be properly searched. @@ -81,10 +81,6 @@ Please see the `code` filter for an explanation on how tokens are split. NOTE: The [Elasticsearch `code_analyzer` doesn't account for all code cases](../integration/advanced_search/elasticsearch_troubleshooting.md#elasticsearch-code_analyzer-doesnt-account-for-all-code-cases). -#### `code_search_analyzer` - -Not directly used for indexing, but rather used to transform a search input. Uses the `whitespace` tokenizer and the `lowercase` and `asciifolding` filters. - ### Tokenizers #### `sha_tokenizer` @@ -115,27 +111,10 @@ Example: - `'path/application.js'` - `'application.js'` -### Filters - -#### `code` - -Uses a [Pattern Capture token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pattern-capture-tokenfilter.html) to split tokens into more easily searched versions of themselves. - -Patterns: - -- `"(\\p{Ll}+|\\p{Lu}\\p{Ll}+|\\p{Lu}+)"`: captures CamelCase and lowerCamelCase strings as separate tokens -- `"(\\d+)"`: extracts digits -- `"(?=([\\p{Lu}]+[\\p{L}]+))"`: captures CamelCase strings recursively. For example: `ThisIsATest` => `[ThisIsATest, IsATest, ATest, Test]` -- `'"((?:\\"|[^"]|\\")*)"'`: captures terms inside quotes, removing the quotes -- `"'((?:\\'|[^']|\\')*)'"`: same as above, for single-quotes -- `'\.([^.]+)(?=\.|\s|\Z)'`: separate terms with periods in-between -- `'([\p{L}_.-]+)'`: some common chars in file names to keep the whole filename intact (for example `my_file-ñame.txt`) -- `'([\p{L}\d_]+)'`: letters, numbers and underscores are the most common tokens in programming. Always capture them greedily regardless of context. - ## Gotchas -- Searches can have their own analyzers. Remember to check when editing analyzers -- `Character` filters (as opposed to token filters) always replace the original character, so they're not a good choice as they can hinder exact searches +- Searches can have their own analyzers. Remember to check when editing analyzers. +- `Character` filters (as opposed to token filters) always replace the original character. These filters can hinder exact searches. ## Zero downtime reindexing with multiple indices diff --git a/doc/development/ai_architecture.md b/doc/development/ai_architecture.md index f497047ccce..84a2635b13c 100644 --- a/doc/development/ai_architecture.md +++ b/doc/development/ai_architecture.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# AI Architecture (Experiment) +# AI Architecture GitLab has created a common set of tools to support our product groups and their utilization of AI. Our goals with this common architecture are: @@ -13,79 +13,20 @@ GitLab has created a common set of tools to support our product groups and their AI is moving very quickly, and we need to be able to keep pace with changes in the area. We have built an [abstraction layer](../../ee/development/ai_features.md) to do this, allowing us to take a more "pluggable" approach to the underlying models, data stores, and other technologies. -The following diagram shows a simplified view of how the different components in GitLab interact. The abstraction layer helps avoid code duplication within the REST APIs within the `AI API` block. - -```plantuml -@startuml -skin rose - -package "Code Suggestions" { - node "Model Gateway" - node "Triton Inference Server" as Triton -} - -package "Code Suggestions Models" as CSM { - node "codegen" - node "PaLM" -} - -package "Suggested Reviewers" { - node "Model Gateway (SR)" - node "Extractor" - node "Serving Model" -} - -package "AI API" as AIF { - node "OpenAI" - node "Vertex AI" -} - -package GitLab { - node "Web IDE" - - package "Web" { - node "REST API" - node "GraphQL" - } - - package "Jobs" { - node "Sidekiq" - } -} - -package Databases { - node "Vector Database" - node "PostgreSQL" -} - -node "VSCode" - -"Model Gateway" --> Triton -Triton --> CSM -GitLab --> Databases -VSCode --> "Model Gateway" -"Web IDE" --> "Model Gateway" -"Web IDE" --> "GraphQL" -"Web IDE" --> "REST API" -"Model Gateway" -[#blue]--> "REST API": user authorized? - -"Sidekiq" --> AIF -Web --> AIF - -"Model Gateway (SR)" --> "REST API" -"Model Gateway (SR)" --> "Serving Model" -"Extractor" --> "GraphQL" -"Sidekiq" --> "Model Gateway (SR)" - -@enduml -``` +The following diagram from the [architecture blueprint](../architecture/blueprints/ai_gateway/index.md) shows a simplified view of how the different components in GitLab interact. The abstraction layer helps avoid code duplication within the REST APIs within the `AI API` block. + + ## SaaS-based AI abstraction layer -GitLab currently operates a cloud-hosted AI architecture. We are exploring how self-managed instances integrate with it. +GitLab currently operates a cloud-hosted AI architecture. We will allow access to it for licensed self managed instances using the AI-gateway. Please see [the blueprint](../architecture/blueprints/ai_gateway) for details There are two primary reasons for this: the best AI models are cloud-based as they often depend on specialized hardware designed for this purpose, and operating self-managed infrastructure capable of AI at-scale and with appropriate performance is a significant undertaking. We are actively [tracking self-managed customers interested in AI](https://gitlab.com/gitlab-org/gitlab/-/issues/409183). +## AI Gateway + +The AI Gateway (formerly the [model gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist)) is a standalone-service that will give access to AI features to all users of GitLab, no matter which instance they are using: self-managed, dedicated or GitLab.com. The SaaS-based AI abstraction layer will transition to connecting to this gateway, rather than accessing cloud-based providers directly. + ## Supported technologies As part of the AI working group, we have been investigating various technologies and vetting them. Below is a list of the tools which have been reviewed and already approved for use within the GitLab application. @@ -127,3 +68,38 @@ For optimal `probes` and `lists` values: - Use `lists` equal to `rows / 1000` for tables with up to 1 million rows and `sqrt(rows)` for larger datasets. - For `probes` start with `lists / 10` for tables up to 1 million rows and `sqrt(lists)` for larger datasets. + +### Code Suggestions + +Code Suggestions is being integrated as part of the GitLab-Rails repository which will unify the architectures between Code Suggestions and AI features that use the abstraction layer, along with offering self-managed support for the other AI features. + +The following table documents functionality that Code Suggestions offers today, and what those changes will look like as part of the unification: + +| Topic | Details | Where this happens today | Where this will happen going forward | +| ----- | ------ | -------------- | ------------------ | +| Request processing | | | | +| | Receives requests from IDEs (VSCode, GitLab WebIDE, MS Visual Studio, IntelliJ, JetBrains, VIM, Emacs, Sublime), including code before and after the cursor | AI Gateway | Abstraction Layer | +| | Authentication the current user, verifies they are authorized to use Code Suggestions for this project | AI Gateway | Abstraction layer | +| | Preprocesses the request to add context, such as including imports via TreeSitter | AI Gateway | Undecided | +| | Routes the request to the AI Provider | AI Gateway | AI Gateway | +| | Returns the response to the IDE | AI Gateway | Abstraction Layer | +| | Logs the request, including timestamp, response time, model, etc | AI Gateway | Both | +| Telemetry | | | | +| | User acceptance or rejection in the IDE | AI Gateway | [Both](https://gitlab.com/gitlab-org/gitlab/-/issues/418282) | +| | Number of unique users per day | [Abstraction Layer](https://app.periscopedata.com/app/gitlab/1143612/Code-Suggestions-Usage) | Undecided | +| | Error rate, model usage, response time, IDE usage | [AI Gateway](https://log.gprd.gitlab.net/app/dashboards#/view/6c947f80-7c07-11ed-9f43-e3784d7fe3ca?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-6h,to:now))) | Both | +| | Suggestions per language | AI Gateway |[Both](https://gitlab.com/groups/gitlab-org/-/epics/11017) | +| Monitoring | | Both | Both | +| | | | | +| Model Routing | | | | +| | Currently we are not using this functionality, but Code Suggestions is able to support routing to multiple models based on a percentage of traffic | AI Gateway | Both | +| Internal Models | | | | +| | Currently unmaintained, the ability to run models in our own instance, running them inside Triton, and routing requests to our own models | AI Gateway | AI Gateway | + +#### Code Suggestions Latency + +Code Suggestions acceptance rates are _highly_ sensitive to latency. While writing code with an AI assistant, a user will pause only for a short duration before continuing on with manually typing out a block of code. As soon as the user has pressed a subsequent keypress, the existing suggestion will be invalidated and a new request will need to be issued to the code suggestions endpoint. In turn, this request will also be highly sensitive to latency. + +In a worst case with sufficient latency, the IDE could be issuing a string of requests, each of which is then ignored as the user proceeds without waiting for the response. This adds no value for the user, while still putting load on our services. + +See our discussions [here](https://gitlab.com/gitlab-org/gitlab/-/issues/418955) around how we plan to iterate on latency for this feature. diff --git a/doc/development/ai_features.md b/doc/development/ai_features.md index 52dc37caec3..ffe151f3876 100644 --- a/doc/development/ai_features.md +++ b/doc/development/ai_features.md @@ -1,6 +1,6 @@ --- -stage: none -group: none +stage: AI-powered +group: AI Framework info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -52,6 +52,9 @@ All AI features are experimental. ## Test AI features locally +NOTE: +Use [this snippet](https://gitlab.com/gitlab-org/gitlab/-/snippets/2554994) for help automating the following section. + 1. Enable the required general feature flags: ```ruby @@ -74,6 +77,9 @@ All AI features are experimental. ### Set up the embedding database +NOTE: +Use [this snippet](https://gitlab.com/gitlab-org/gitlab/-/snippets/2554994) for help automating the following section. + For features that use the embedding database, additional setup is needed. 1. Enable [pgvector](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/pgvector.md#enable-pgvector-in-the-gdk) in GDK @@ -88,6 +94,9 @@ For features that use the embedding database, additional setup is needed. ### Set up GitLab Duo Chat +NOTE: +Use [this snippet](https://gitlab.com/gitlab-org/gitlab/-/snippets/2554994) for help automating the following section. + 1. [Enable Anthropic API features](#configure-anthropic-access). 1. [Enable OpenAI support](#configure-openai-access). 1. [Ensure the embedding database is configured](#set-up-the-embedding-database). @@ -123,6 +132,14 @@ index 5fa7ae8a2bc1..5fe996ba0345 100644 def valid? ``` +### Working with GitLab Duo Chat + +Prompts are the most vital part of GitLab Duo Chat system. Prompts are the instructions sent to the Large Language Model to perform certain tasks. + +The state of the prompts is the result of weeks of iteration. If you want to change any prompt in the current tool, you must put it behind a feature flag. + +If you have any new or updated prompts, ask members of AI Framework team to review, because they have significant experience with them. + ### Setup for GitLab documentation chat (legacy chat) To populate the embedding database for GitLab chat: @@ -130,12 +147,63 @@ To populate the embedding database for GitLab chat: 1. Open a rails console 1. Run [this script](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/10588#note_1373586079) to populate the embedding database +### Contributing to GitLab Duo Chat + +The Chat feature uses a [zero-shot agent](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/gitlab/llm/chain/agents/zero_shot/executor.rb) that includes a system prompt explaining how the large language model should interpret the question and provide an +answer. The system prompt defines available tools that can be used to gather +information to answer the user's question. + +The zero-shot agent receives the user's question and decides which tools to use to gather information to answer it. +It then makes a request to the large language model, which decides if it can answer directly or if it needs to use one +of the defined tools. + +The tools each have their own prompt that provides instructions to the large language model on how to use that tool to +gather information. The tools are designed to be self-sufficient and avoid multiple requests back and forth to +the large language model. + +After the tools have gathered the required information, it is returned to the zero-shot agent, which asks the large language +model if enough information has been gathered to provide the final answer to the user's question. + +#### Adding a new tool + +To add a new tool: + +1. Create files for the tool in the `ee/lib/gitlab/llm/chain/tools/` folder. Use existing tools like `issue_identifier` or +`resource_reader` as a template. + +1. Write a class for the tool that includes: + + - Name and description of what the tool does + - Example questions that would use this tool + - Instructions for the large language model on how to use the tool to gather information - so the main prompts that + this tool is using. + +1. Test and iterate on the prompt using RSpec tests that make real requests to the large language model. + - Prompts require trial and error, the non-deterministic nature of working with LLM can be surprising. + - Anthropic provides good [guide](https://docs.anthropic.com/claude/docs/introduction-to-prompt-design) on working on prompts. + +1. Implement code in the tool to parse the response from the large language model and return it to the zero-shot agent. + +1. Add the new tool name to the `tools` array in `ee/lib/gitlab/llm/completions/chat.rb` so the zero-shot agent knows about it. + +1. Add tests by adding questions to the test-suite for which the new tool should respond to. Iterate on the prompts as needed. + +The key things to keep in mind are properly instructing the large language model through prompts and tool descriptions, +keeping tools self-sufficient, and returning responses to the zero-shot agent. With some trial and error on prompts, +adding new tools can expand the capabilities of the chat feature. + +There are available short [videos](https://www.youtube.com/playlist?list=PL05JrBw4t0KoOK-bm_bwfHaOv-1cveh8i) covering this topic. + ### Debugging To gather more insights about the full request, use the `Gitlab::Llm::Logger` file to debug logs. +The default logging level on production is `INFO` and **must not** be used to log any data that could contain personal identifying information. + To follow the debugging messages related to the AI requests on the abstraction layer, you can use: ```shell +export LLM_DEBUG=1 +gdk start tail -f log/llm.log ``` @@ -143,7 +211,7 @@ tail -f log/llm.log In order to obtain a GCP service key for local development, please follow the steps below: -- Create a sandbox GCP environment by visiting [this page](https://about.gitlab.com/handbook/infrastructure-standards/#individual-environment) and following the instructions, or by requesting access to our existing group environment by using [this template](https://gitlab.com/gitlab-com/it/infra/issue-tracker/-/issues/new?issuable_template=gcp_group_account_iam_update_request). At this time, access to any endpoints outside of `text-bison` or `chat-bison` must be made through the group environment. +- Create a sandbox GCP environment by visiting [this page](https://about.gitlab.com/handbook/infrastructure-standards/#individual-environment) and following the instructions, or by requesting access to our existing group environment by using [this template](https://gitlab.com/gitlab-com/it/infra/issue-tracker/-/issues/new?issuable_template=gcp_group_account_iam_update_request). - In the GCP console, go to `IAM & Admin` > `Service Accounts` and click on the "Create new service account" button - Name the service account something specific to what you're using it for. Select Create and Continue. Under `Grant this service account access to project`, select the role `Vertex AI User`. Select `Continue` then `Done` - Select your new service account and `Manage keys` > `Add Key` > `Create new key`. This will download the **private** JSON credentials for your service account. @@ -174,18 +242,47 @@ Gitlab::CurrentSettings.update!(anthropic_api_key: <insert API key>) ### Testing GitLab Duo Chat with predefined questions -Because success of answers to user questions in GitLab Duo Chat heavily depends on toolchain and prompts of each tool, it's common that even a minor change in a prompt or a tool impacts processing of some questions. To make sure that a change in the toolchain doesn't break existing functionality, you can use following commands to validate answers to some predefined questions: - -1. Rake task which iterates through questions defined in CSV file and checks tools used for evaluating each question. +Because success of answers to user questions in GitLab Duo Chat heavily depends on toolchain and prompts of each tool, it's common that even a minor change in a prompt or a tool impacts processing of some questions. To make sure that a change in the toolchain doesn't break existing functionality, you can use the following rspecs to validate answers to some predefined questions: ```ruby -rake gitlab:llm:zero_shot:test:questions[<issue_url>] +export OPENAI_API_KEY='<key>' +export ANTHROPIC_API_KEY='<key>' +REAL_AI_REQUEST=1 rspec ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_spec.rb ``` -1. RSpec which iterates through resource-specific questions on predefined resources: +When you need to update the test questions that require documentation embeddings, +make sure a new fixture is generated and committed together with the change. + +#### Populating embeddings and using embeddings fixture + +To seed your development database with the embeddings for GitLab Documentation, +you may use the pre-generated embeddings and a Rake test. + +```shell +RAILS_ENV=development bundle exec rake gitlab:llm:embeddings:seed_pre_generated +``` + +The DBCleaner gem we use clear the database tables before each test runs. +Instead of fully populating the table `tanuki_bot_mvc` where we store embeddings for the documentations, +we can add a few selected embeddings to the table from a pre-generated fixture. + +For instance, to test that the question "How can I reset my password" is correctly +retrieving the relevant embeddings and answered, we can extract the top N closet embeddings +to the question into a fixture and only restore a small number of embeddings quickly. +To faciliate an extraction process, a Rake task been written. +You can add or remove the questions needed to be tested in the Rake task and run the task to generate a new fixture. + +```shell +RAILS_ENV=development bundle exec rake gitlab:llm:embeddings:extract_embeddings +``` + +In the specs where you need to use the embeddings, +use the RSpec config hook `:ai_embedding_fixtures` on a context. ```ruby -ANTHROPIC_API_KEY='<key>' REAL_AI_REQUEST=1 rspec ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_spec.rb +context 'when asking about how to use GitLab', :ai_embedding_fixtures do + # ...examples +end ``` ## Experimental REST API @@ -409,6 +506,52 @@ module EE end ``` +### Pairing requests with responses + +Because multiple users' requests can be processed in parallel, when receiving responses, +it can be difficult to pair a response with its original request. The `requestId` +field can be used for this purpose, because both the request and response are assured +to have the same `requestId` UUID. + +### Caching + +AI requests and responses can be cached. Cached conversation is being used to +display user interaction with AI features. In the current implementation, this cache +is not used to skip consecutive calls to the AI service when a user repeats +their requests. + +```graphql +query { + aiMessages { + nodes { + id + requestId + content + role + errors + timestamp + } + } +} +``` + +This cache is especially useful for chat functionality. For other services, +caching is disabled. (It can be enabled for a service by using `cache_response: true` +option.) + +Caching has following limitations: + +- Messages are stored in Redis stream. +- There is a single stream of messages per user. This means that all services + currently share the same cache. If needed, this could be extended to multiple + streams per user (after checking with the infrastructure team that Redis can handle + the estimated amount of messages). +- Only the last 50 messages (requests + responses) are kept. +- Expiration time of the stream is 3 days since adding last message. +- User can access only their own messages. There is no authorization on the caching + level, and any authorization (if accessed by not current user) is expected on + the service layer. + ### Check if feature is allowed for this resource based on namespace settings There are two settings allowed on root namespace level that restrict the use of AI features: diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 9bc62ed7095..440068d55c2 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -47,6 +47,11 @@ The GraphQL framework has some specific gotchas to be aware of, and domain exper If you are asked to review a merge request that modifies any GraphQL files or adds an endpoint, please have a look at [our GraphQL review guide](graphql_guide/reviewing.md). +## Reading GraphQL logs + +See the [Reading GraphQL logs](graphql_guide/monitoring.md) guide for tips on how to inspect logs +of GraphQL requests and monitor the performance of your GraphQL queries. + ## Authentication Authentication happens through the `GraphqlController`, right now this @@ -2045,8 +2050,8 @@ full stack: - An [argument's `default_value`](https://graphql-ruby.org/fields/arguments.html) applies correctly. - Objects resolve successfully, and there are no N+1 issues. -When adding a query, you can use the `a working graphql query` shared example to test if the query -renders valid results. +When adding a query, you can use the `a working graphql query that returns data` and +`a working graphql query that returns no data` shared examples to test if the query renders valid results. You can construct a query including all available fields using the `GraphqlHelpers#all_graphql_fields_for` helper. This makes it more straightforward to add a test rendering all possible fields for a query. @@ -2404,7 +2409,3 @@ elimination of laziness, where needed. For dealing with lazy values without forcing them, use `Gitlab::Graphql::Lazy.with_value`. - -## Monitoring GraphQL - -See the [Monitoring GraphQL](graphql_guide/monitoring.md) guide for tips on how to inspect logs of GraphQL requests and monitor the performance of your GraphQL queries. diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 2ed4d934022..45568c700c7 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -44,16 +44,16 @@ A good example is as follows: ```ruby desc 'Get all broadcast messages' do detail 'This feature was introduced in GitLab 8.12.' - success Entities::BroadcastMessage + success Entities::System::BroadcastMessage end params do optional :page, type: Integer, desc: 'Current page number' optional :per_page, type: Integer, desc: 'Number of messages per page' end get do - messages = BroadcastMessage.all + messages = System::BroadcastMessage.all - present paginate(messages), with: Entities::BroadcastMessage + present paginate(messages), with: Entities::System::BroadcastMessage end ``` diff --git a/doc/development/application_secrets.md b/doc/development/application_secrets.md index 5b0755e97e3..33bba2b3285 100644 --- a/doc/development/application_secrets.md +++ b/doc/development/application_secrets.md @@ -23,9 +23,9 @@ This page is a development guide for application secrets. |Installation type |Location | |--- |--- | -|Omnibus |[`/etc/gitlab/gitlab-secrets.json`](https://docs.gitlab.com/omnibus/settings/backups.html#backup-and-restore-omnibus-gitlab-configuration) | -|Cloud Native GitLab Charts |[Kubernetes Secrets](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/f65c3d37fc8cf09a7987544680413552fb666aac/doc/installation/secrets.md#gitlab-rails-secret)| -|Source |`<path-to-gitlab-rails>/config/secrets.yml` (Automatically generated by [01_secret_token.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/01_secret_token.rb)) | +| Linux package |[`/etc/gitlab/gitlab-secrets.json`](https://docs.gitlab.com/omnibus/settings/backups.html#backup-and-restore-omnibus-gitlab-configuration) | +| Cloud Native GitLab Charts |[Kubernetes Secrets](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/f65c3d37fc8cf09a7987544680413552fb666aac/doc/installation/secrets.md#gitlab-rails-secret)| +| Self-compiled |`<path-to-gitlab-rails>/config/secrets.yml` (Automatically generated by [01_secret_token.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/01_secret_token.rb)) | ## Warning: Before you add a new secret to application secrets @@ -38,9 +38,9 @@ GitLab.com environments prior to changing this file. **Examples** -- [Change for source installation](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/27581) -- [Change for omnibus installation](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3267) -- [Change for omnibus installation](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4158) +- [Change for self-compiled installation](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/27581) +- [Change for Linux package installation](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3267) +- [Change for Linux package installation](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4158) - [Change for Cloud Native installation](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/1318) ## Further iteration diff --git a/doc/development/application_slis/index.md b/doc/development/application_slis/index.md index f48088a6e08..30bd6011a67 100644 --- a/doc/development/application_slis/index.md +++ b/doc/development/application_slis/index.md @@ -14,40 +14,40 @@ and their success close to the implementation and allows the people building features to easily define how these features should be monitored. -Defining an SLI causes 2 -[Prometheus counters](https://prometheus.io/docs/concepts/metric_types/#counter) -to be emitted from the rails application: - -- `gitlab_sli:<sli name>:total`: incremented for each operation. -- `gitlab_sli:<sli_name>:success_total`: incremented for successful - operations. - ## Existing SLIs 1. [`rails_request`](rails_request.md) 1. `global_search_apdex` 1. `global_search_error_rate` 1. `global_search_indexing_apdex` +1. [`sidekiq_execution`](sidekiq_execution.md) ## Defining a new SLI -An SLI can be defined using the `Gitlab::Metrics::Sli::Apdex` or -`Gitlab::Metrics::Sli::ErrorRate` class. These work in broadly the same way, but -for clarity, they define different metric names: +An SLI can be defined with the `Gitlab::Metrics::Sli::Apdex` or +`Gitlab::Metrics::Sli::ErrorRate` class. When you define an SLI, two +[Prometheus counters](https://prometheus.io/docs/concepts/metric_types/#counter) +are emitted from the Rails application. Both counters work in broadly the same way and contain a total operation count. `Apdex` uses a success rate to calculate a success ratio, and `ErrorRate` uses an error rate to calculate an error ratio. -1. `Gitlab::Metrics::Sli::Apdex.new('foo')` defines: - 1. `gitlab_sli:foo_apdex:total` for the total number of measurements. - 1. `gitlab_sli:foo_apdex:success_total` for the number of successful +The following metrics are defined: + +- `Gitlab::Metrics::Sli::Apdex.new('foo')` defines: + - `gitlab_sli_foo_apdex_total` for the total number of measurements. + - `gitlab_sli_foo_apdex_success_total` for the number of successful measurements. -1. `Gitlab::Metrics::Sli::ErrorRate.new('foo')` defines: - 1. `gitlab_sli:foo:total` for the total number of measurements. - 1. `gitlab_sli:foo:error_total` for the number of error - measurements - as this is an error rate, it's more natural to talk about - errors divided by the total. +- `Gitlab::Metrics::Sli::ErrorRate.new('foo')` defines: + - `gitlab_sli_foo_total` for the total number of measurements. + - `gitlab_sli_foo_error_total` for the number of error + measurements. Because this metric is an error rate, + errors are divided by the total number. As shown in this example, they can share a base name (`foo` in this example). We recommend this when they refer to the same operation. +You should use `Apdex` to measure the performance of successful operations. You don't have to measure the performance of a failing request because that performance should be tracked with `ErrorRate`. For example, you can measure whether a request is performing within a specified latency threshold. + +You should use `ErrorRate` to measure the rate of unsuccessful operations. For example, you can measure whether a failed request returns an HTTP status greater than or equal to `500`. + Before the first scrape, it is important to have [initialized the SLI with all possible label-combinations](https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics). This avoid confusing results when using these counters in calculations. diff --git a/doc/development/application_slis/rails_request.md b/doc/development/application_slis/rails_request.md index b3ee326aa87..e5a02c20472 100644 --- a/doc/development/application_slis/rails_request.md +++ b/doc/development/application_slis/rails_request.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w NOTE: This SLI is used for service monitoring. But not for [error budgets for stage groups](../stage_group_observability/index.md#error-budget) -by default. You can [opt in](#error-budget-attribution-and-ownership). +by default. The request Apdex SLI and the error rate SLI are [SLIs defined in the application](index.md). diff --git a/doc/development/application_slis/sidekiq_execution.md b/doc/development/application_slis/sidekiq_execution.md new file mode 100644 index 00000000000..dfe7f3864d5 --- /dev/null +++ b/doc/development/application_slis/sidekiq_execution.md @@ -0,0 +1,74 @@ +--- +stage: Platforms +group: Scalability +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Sidekiq execution SLIs (service level indicators) + +> [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/700) in GitLab 16.0. This version of Sidekiq execution SLIs replaces the old version of the SLI where you can now drill down by workers in the [Application SLI Violations dashboard](https://dashboards.gitlab.net/d/general-application-sli-violations/general-application-sli-violations?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-product_stage=All&var-stage_group=All&var-component=sidekiq_execution) for stage groups. + +NOTE: +This SLI is used for service monitoring. But not for [error budgets for stage groups](../stage_group_observability/index.md#error-budget) +by default. + +The Sidekiq execution Apdex measures the duration of successful jobs completion as an indicator for +application performance. + +The error rate measures unsuccessful jobs completion when exception occurs as an indicator for +server misbehavior. + +- `gitlab_sli_sidekiq_execution_apdex_total`: This counter gets + incremented for every successful job execution that does not result in an exception. It ensures slow jobs are not + counted twice, because the job is already counted in the error SLI. + +- `gitlab_sli_sidekiq_execution_apdex_success_total`: This counter gets + incremented for every successful job that performed faster than + the [defined target duration depending on the job urgency](../sidekiq/worker_attributes.md#job-urgency). + +- `gitlab_sli_sidekiq_execution_error_total`: This counter gets + incremented for every job that encountered an exception. + +- `gitlab_sli_sidekiq_execution_total`: This counter gets + incremented for every job execution. + +These counters are labeled with: + +- `worker`: The identification of the worker. + +- `feature_category`: The feature category specified for that worker. + +- `urgency`: The urgency attribute specified for that worker. + +- `external_dependencies`: The boolean value `yes` or `no` based on the [external dependencies attribute](../sidekiq/worker_attributes.md#jobs-with-external-dependencies). + +- `queue`: The queue in which the job is running. + +For more information about these SLIs, see the [Sidekiq SLIs documentation](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/sidekiq/sidekiq-slis.md) in runbooks. + +## Adjusting job urgency + +Not all workers perform the same type of work, so it is possible to +define different urgency levels for different jobs. A job with a +lower urgency can have a longer execution duration than jobs with high urgency. + +For more information on the execution latency requirement and how to set a job's urgency, see the [Sidekiq worker attributes page](../sidekiq/worker_attributes.md#job-urgency). + +### Error budget attribution and ownership + +This SLI is used for service level monitoring. It feeds into the +[error budget for stage groups](../stage_group_observability/index.md#error-budget). + +The workers for the SLI feed into a group's error budget based on the +[feature category declared on it](../feature_categorization/index.md). + +To know which workers are included for your group, see the +Sidekiq Completion Rate panel on the +[group dashboard for your group](https://dashboards.gitlab.net/dashboards/f/stage-groups/stage-groups). +In the **Budget Attribution** row, the **Sidekiq Execution Apdex** log link shows you +how many jobs are not meeting the 10 second or 300 second target. + +## Jobs with external dependencies + +Jobs with [external dependencies](../sidekiq/worker_attributes.md#jobs-with-external-dependencies) are excluded from +the Apdex and error ratio calculation. diff --git a/doc/development/audit_event_guide/index.md b/doc/development/audit_event_guide/index.md index c49d3a243c0..b8af1341919 100644 --- a/doc/development/audit_event_guide/index.md +++ b/doc/development/audit_event_guide/index.md @@ -29,14 +29,14 @@ If you have any questions, please reach out to `@gitlab-org/govern/compliance` t To instrument an audit event, the following attributes should be provided: -| Attribute | Type | Required? | Description | -|:-------------|:---------------------|:----------|:------------------------------------------------------------------| -| `name` | String | false | Action name to be audited. Represents the [type of the event](#event-type-definitions). Used for error tracking | -| `author` | User | true | User who authors the change. Can be an [internal user](../internal_users.md). For example, [inactive project deletion](../../administration/inactive_project_deletion.md) audit events are authored by `GitLab-Admin-Bot`. | -| `scope` | User, Project, Group | true | Scope which the audit event belongs to | -| `target` | Object | true | Target object being audited | -| `message` | String | true | Message describing the action ([not translated](#i18n-and-the-audit-event-message-attribute)) | -| `created_at` | DateTime | false | The time when the action occurred. Defaults to `DateTime.current` | +| Attribute | Type | Required? | Description | +|:-------------|:------------------------------------|:----------|:------------------------------------------------------------------| +| `name` | String | false | Action name to be audited. Represents the [type of the event](#event-type-definitions). Used for error tracking | +| `author` | User | true | User who authors the change. Can be an [internal user](../internal_users.md). For example, [inactive project deletion](../../administration/inactive_project_deletion.md) audit events are authored by `GitLab-Admin-Bot`. | +| `scope` | User, Project, Group, or InstanceScope | true | Scope which the audit event belongs to | +| `target` | Object | true | Target object being audited | +| `message` | String | true | Message describing the action ([not translated](#i18n-and-the-audit-event-message-attribute)) | +| `created_at` | DateTime | false | The time when the action occurred. Defaults to `DateTime.current` | ## How to instrument new Audit Events diff --git a/doc/development/backend/ruby_style_guide.md b/doc/development/backend/ruby_style_guide.md index 714c8571d20..d6e2f7a48d2 100644 --- a/doc/development/backend/ruby_style_guide.md +++ b/doc/development/backend/ruby_style_guide.md @@ -161,6 +161,145 @@ def method end ``` +## Avoid ActiveRecord callbacks + +[ActiveRecord callbacks](https://guides.rubyonrails.org/active_record_callbacks.html) allow +you to "trigger logic before or after an alteration of an object's state." + +Use callbacks when no superior alternative exists, but employ them only if you +thoroughly understand the reasons for doing so. + +When adding new lifecycle events for ActiveRecord objects, it is preferable to +add the logic to a service class instead of a callback. + +## Why callbacks shoud be avoided + +In general, callbacks should be avoided because: + +- Callbacks are hard to reason about because invocation order is not obvious and + they break code narrative. +- Callbacks are harder to locate and navigate because they rely on reflection to + trigger rather than being ordinary method calls. +- Callbacks make it difficult to apply changes selectively to an object's state + because changes always trigger the entire callback chain. +- Callbacks trap logic in the ActiveRecord class. This tight coupling encourages + fat models that contain too much business logic, which could instead live in + service objects that are more reusable, composable, and are easier to test. +- Illegal state transitions of an object can be better enforced through + attribute validations. +- Heavy use of callbacks affects factory creation speed. With some classes + having hundreds of callbacks, creating an instance of that object for + an automated test can be a very slow operation, resulting in slow specs. + +Some of these examples are discussed in this [video from thoughtbot](https://www.youtube.com/watch?v=GLBMfB8N1G8). + +The GitLab codebase relies heavily on callbacks and it is hard to refactor them +once added due to invisible dependencies. As a result, this guideline does not +call for removing all existing callbacks. + +### When to use callbacks + +Callbacks can be used in special cases. Some examples of cases where adding a +callback makes sense: + +- A dependency uses callbacks and we would like to override the callback + behavior. +- Incrementing cache counts. +- Data normalization that only relates to data on the current model. + +### Example of moving from a callback to a service + +There is a project with the following basic data model: + +```ruby +class Project + has_one :repository +end + +class Repository + belongs_to :project +end +``` + +Say we want to create a repository after a project is created and use the +project name as the repository name. A developer familiar with Rails might +immediately think: sounds like a job for an ActiveRecord callback! And add this +code: + +```ruby +class Project + has_one :repository + + after_initialize :create_random_name + after_create :create_repository + + def create_random_name + SecureRandom.alphanumeric + end + + def create_repository + Repository.create!(project: self) + end +end + +class Repository + after_initialize :set_name + + def set_name + name = project.name + end +end + +class ProjectsController + def create + Project.create! # also creates a repository and names it + end +end +``` + +While this seems pretty harmless for a baby Rails app, adding this type of logic +via callbacks has many downsides once your Rails app becomes large and complex +(all of which are listed in this documentation). Instead, we can add this +logic to a service class: + +```ruby +class Project + has_one :repository +end + +class Repository + belongs_to :project +end + +class ProjectCreator + def self.execute + ApplicationRecord.transaction do + name = SecureRandom.alphanumeric + project = Project.create!(name: name) + Repository.create!(project: project, name: name) + end + end +end + +class ProjectsController + def create + ProjectCreator.execute + end +end +``` + +With an application this simple, it can be hard to see the benefits of the second +approach. But we already some benefits: + +- Can test `Repository` creation logic separate from `Project` creation logic. Code + no longer violates law of demeter (`Repository` class doesn't need to know + `project.name`). +- Clarity of invocation order. +- Open to change: if we decide there are some scenarios where we do not want a + repository created for a project, we can create a new service class rather + than needing to refactor to `Project` and `Repository` classes. +- Each instance of a `Project` factory does not create a second (`Repository`) object. + ## Styles we have no opinion on If a RuboCop rule is proposed and we choose not to add it, we should document that decision in this guide so it is more discoverable and link the relevant discussion as a reference. diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 56b5fa8c976..486808f25e7 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -94,7 +94,7 @@ EE: true uses system fonts for all text." - Any client-facing change to our REST and GraphQL APIs **must** have a changelog entry. See the [complete list what comprises a GraphQL breaking change](api_graphql_styleguide.md#breaking-changes). -- Any change that introduces an [advanced search migration](search/advanced_search_migration_styleguide.md#creating-a-new-advanced-search-migration) +- Any change that introduces an [advanced search migration](search/advanced_search_migration_styleguide.md#create-a-new-advanced-search-migration) **must** have a changelog entry. - A fix for a regression introduced and then fixed in the same release (such as fixing a bug introduced during a monthly release candidate) **should not** diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index dceb2da5951..41ae4fe14b4 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -109,6 +109,7 @@ A job with the `created` state isn't seen by the runner yet. To make it possible 1. The job required a manual start and it has been triggered. 1. All jobs from the previous stage have completed successfully. In this case we transition all jobs from the next stage to `pending`. 1. The job specifies DAG dependencies using `needs:` and all the dependent jobs are completed. +1. The job has not been [dropped](#dropping-stuck-builds) because of its not-runnable state by [`Ci::PipelineCreation::DropNotRunnableBuildsService`](https://gitlab.com/gitlab-org/gitlab/-/blob/v16.0.4-ee/ee/app/services/ci/pipeline_creation/drop_not_runnable_builds_service.rb). When the runner is connected, it requests the next `pending` job to run by polling the server continuously. @@ -119,11 +120,6 @@ After the server receives the request it selects a `pending` job based on the [` Once all jobs are completed for the current stage, the server "unlocks" all the jobs from the next stage by changing their state to `pending`. These can now be picked by the scheduling algorithm when the runner requests new jobs, and continues like this until all stages are completed. -If a job is not picked up by a runner in 24 hours it is automatically removed from -the processing queue after that time. If a pending job is stuck, when there is no -runner available that can process it, it is removed from the queue after 1 hour. -In both cases the job's status is changed to `failed` with an appropriate failure reason. - ### Communication between runner and GitLab server After the runner is [registered](https://docs.gitlab.com/runner/register/) using the registration token, the server knows what type of jobs it can execute. This depends on: @@ -163,6 +159,47 @@ At this point we loop through remaining `pending` jobs and we try to assign the As we increase the number of runners in the pool we also increase the chances of conflicts which would arise if assigning the same job to different runners. To prevent that we gracefully rescue conflict errors and assign the next job in the list. +### Dropping stuck builds + +There are two ways of marking builds as "stuck" and drop them. + +1. When a build is created, [`Ci::PipelineCreation::DropNotRunnableBuildsService`](https://gitlab.com/gitlab-org/gitlab/-/blob/v16.0.4-ee/ee/app/services/ci/pipeline_creation/drop_not_runnable_builds_service.rb) checks for upfront known conditions that would make jobs not executable: + - If there is not enough [CI/CD Minutes](#compute-quota) to run the build, then the build is immediately dropped with `ci_quota_exceeded`. + - [In the future](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121761), if the project is not on the plan that available runners for the build require via `allowed_plans`, then the build is immediately dropped with `no_matching_runner`. +1. If there is no available Runner to pick up a build, it is dropped after 1 hour by [`Ci::StuckBuilds::DropPendingService`](https://gitlab.com/gitlab-org/gitlab/-/blob/v16.0.4-ee/app/services/ci/stuck_builds/drop_pending_service.rb). + - If a job is not picked up by a runner in 24 hours it is automatically removed from + the processing queue after that time. + - If a pending job is **stuck**, when there is no + runner available that can process it, it is removed from the queue after 1 hour. + - In both cases the job's status is changed to `failed` with an appropriate failure reason. + +#### The reason behind this difference + +CI Minutes quota mechanism is handled early when the job is created because it is a constant decision for most of the time. +Once a project exceeds the limit, every next job matching it will be applicable for it until next month starts. +Of course, the project owner can buy additional minutes, but that is a manual action that the project need to take. + +The same mechanism will be used for `allowed_plans` [soon](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121761). +If the project is not on the required plan and a job is targeting such runner, +it will be failing constantly until the project owner changes the configuration or upgrades the namespace to the required plan. + +These two mechanisms are also very SaaS specific and at the same time are quite compute expensive when we consider SaaS' scale. +Doing the check before the job is even transitioned to pending and failing early makes a lot of sense here. + +Why we don't handle other cases for pending and drop jobs early? +In some cases, a job is in pending only because the runner is slow on taking up jobs. +This is not something that you can know at GitLab level. +Depending on the runner's configuration and capacity and the size of the queue in GitLab, a job may be taken immediately, or may need to wait. + +There may be also other reasons: + +- you are handling runner maintenance and it's not available for a while at all, +- you are updating configuration and by mistake, you've messed up the tagging and/or protected flag (or in the case of our SaaS instance runners; you've assigned a wrong cost factor or `allowed_plans` configuration). + +All of that are problems that may be temporary and mostly are not expected to happen and are expected to be detected and fixed early. +We definitely don't want to drop jobs immediately when one of these conditions is happening. +Dropping a job only because a runner is at capacity or because there is a temporary unavailability/configuration mistake would be very harmful to users. + ## The definition of "Job" in GitLab CI/CD "Job" in GitLab CI context refers a task to drive Continuous Integration, Delivery and Deployment. diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 6aedb18c15d..a8c527ad30e 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -201,7 +201,8 @@ by a reviewer before passing it to a maintainer as described in the on the line of code in question with the SQL queries so they can give their advice. 1. User-facing changes include both visual changes (regardless of how minor), and changes to the rendered DOM which impact how a screen reader may announce - the content. + the content. Groups that do not have dedicated Product + Designers do not require a Product Designer to approve feature changes, unless the changes are community contributions. 1. End-to-end changes include all files in the `qa` directory. #### Acceptance checklist @@ -605,7 +606,7 @@ When ready to merge: WARNING: **If the merge request is from a fork, also check the [additional guidelines for community contributions](#community-contributions).** -- Consider using the [Squash and merge](../user/project/merge_requests/squash_and_merge.md#squash-and-merge) +- Consider using the [Squash and merge](../user/project/merge_requests/squash_and_merge.md) feature when the merge request has a lot of commits. When merging code, a maintainer should only use the squash feature if the author has already set this option, or if the merge request clearly contains a @@ -624,7 +625,7 @@ WARNING: enough to `main`. - When you set the MR to auto-merge, you should take over subsequent revisions for anything that would be spotted after that. -- For merge requests that have had [Squash and merge](../user/project/merge_requests/squash_and_merge.md#squash-and-merge) set, +- For merge requests that have had [Squash and merge](../user/project/merge_requests/squash_and_merge.md) set, the squashed commit's default commit message is taken from the merge request title. You're encouraged to [select a commit with a more informative commit message](../user/project/merge_requests/squash_and_merge.md) before merging. diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index 58ec0de6074..59e35e658e7 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -61,7 +61,7 @@ Check these aspects both when _designing_ and _reviewing_ UI changes. ### Visual design -Check visual design properties using your browser's _elements inspector_ ([Chrome](https://developer.chrome.com/docs/devtools/css/), +Check visual design properties using your browser's elements inspector ([Chrome](https://developer.chrome.com/docs/devtools/css/), [Firefox](https://firefox-source-docs.mozilla.org/devtools-user/page_inspector/how_to/open_the_inspector/index.html)). - Use recommended [colors](https://design.gitlab.com/product-foundations/color) @@ -71,9 +71,11 @@ Check visual design properties using your browser's _elements inspector_ ([Chrom or propose new ones according to [iconography](https://design.gitlab.com/product-foundations/iconography/) and [illustration](https://design.gitlab.com/product-foundations/illustration/) guidelines. -- _Optionally_ consider [dark mode](../../user/profile/preferences.md#dark-mode). [^1] +- Optional: Consider dark mode. For more information, see [Change the syntax highlighting theme](../../user/profile/preferences.md#change-the-syntax-highlighting-theme). - [^1]: You're not required to design for [dark mode](../../user/profile/preferences.md#dark-mode) while the feature is an [Experiment](../../policy/experiment-beta-support.md#experiment). The [UX Foundations team](https://about.gitlab.com/direction/manage/foundations/) plans to improve the dark mode in the future. Until we integrate [Pajamas](https://design.gitlab.com/) components into the product and the underlying design strategy is in place to support dark mode, we cannot guarantee that we won't introduce bugs and debt to this mode. At your discretion, evaluate the need to create dark mode patches. +### Dark Mode + +You're not required to design for dark mode while the feature is an [Experiment](../../policy/experiment-beta-support.md#experiment). The [UX Foundations team](https://about.gitlab.com/direction/manage/foundations/) plans to improve the dark mode in the future. Until we integrate [Pajamas](https://design.gitlab.com/) components into the product and the underlying design strategy is in place to support dark mode, we cannot guarantee that we won't introduce bugs and debt to this mode. At your discretion, evaluate the need to create dark mode patches. ### States diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index b929a5c0f04..d2063a6836d 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -54,7 +54,7 @@ To write and test your code, you will use the GitLab Development Kit. - To run the development environment locally, download and set up the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit). See the [GDK README](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/README.md) for setup instructions - and [Troubleshooting](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/troubleshooting.md) if you get stuck. + and [Troubleshooting](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/troubleshooting/index.md) if you get stuck. - GDK is heavy. If you need to build something fast, by trial and error, consider doing so with an empty rails app and port it to GDK after. diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 50e87fc5341..228e7507777 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -7,6 +7,35 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Issues workflow +## Finding issues to work on + +GitLab has over 75,000 issues that you can work on. +You can use [labels](../../user/project/labels.md) to filter and find suitable issues to work on. +New contributors can look for [issues with the `quick win` label](https://gitlab.com/groups/gitlab-org/-/issues/?sort=created_asc&state=opened&label_name%5B%5D=quick%20win&first_page_size=20). + +The `frontend` and `backend` labels are also a good choice to refine the issue list. + +## Clarifying/validating an issue + +Many issues have not been visited or validated recently. +Before trying to solve an issue, take the following steps: + +- Ask the author if the issue is still relevant. +- Ask the community if the issue is still relevant. +- Attempt to validate whether: + - A merge request has already been created (see the related merge requests section). + Sometimes the issue is not closed/updated. + - The `type::bug` still exists (by recreating it). + - The `type::feature` has not already been implemented (by trying it). + +## Working on the issue + +Leave a note to indicate you wish to work on the issue and would like to be assigned +(mention the author and/or `@gitlab-org/coaches`). + +If you are stuck or did not properly understand the issue you can ask the author or +the community for help. + **Before you submit an issue, [search the issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues)** for similar entries. Someone else might have already had the same bug or feature proposal. If you find an existing issue, show your support with an emoji reaction and add your notes to the discussion. diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 7a0269e551d..12862cb0993 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -110,7 +110,7 @@ Commit messages should follow the guidelines below, for reasons explained by Chr **Important notes:** - If the guidelines are not met, the MR may not pass the [Danger checks](https://gitlab.com/gitlab-org/gitlab/-/blob/master/danger/commit_messages/Dangerfile). -- Consider enabling [Squash and merge](../../user/project/merge_requests/squash_and_merge.md#squash-and-merge) +- Consider enabling [Squash and merge](../../user/project/merge_requests/squash_and_merge.md) if your merge request includes "Applied suggestion to X files" commits, so that Danger can ignore those. - The prefixes in the form of `[prefix]` and `prefix:` are allowed (they can be all lowercase, as long as the message itself is capitalized). For instance, `danger: Improve Danger behavior` and @@ -172,7 +172,7 @@ the contribution acceptance criteria below: restarting the failing CI job, rebasing on top of target branch to bring in updates that may resolve the failure, or if it has not been fixed yet, ask a developer to help you fix the test. -1. The MR contains a few logically organized commits, or has [squashing commits enabled](../../user/project/merge_requests/squash_and_merge.md#squash-and-merge). +1. The MR contains a few logically organized commits, or has [squashing commits enabled](../../user/project/merge_requests/squash_and_merge.md). 1. The changes can merge without problems. If not, you should rebase if you're the only one working on your feature branch, otherwise merge the default branch into the MR branch. 1. Only one specific issue is fixed or one specific feature is implemented. Do not diff --git a/doc/development/database/avoiding_downtime_in_migrations.md b/doc/development/database/avoiding_downtime_in_migrations.md index 6a819e9f6cd..371df5b45ff 100644 --- a/doc/development/database/avoiding_downtime_in_migrations.md +++ b/doc/development/database/avoiding_downtime_in_migrations.md @@ -29,22 +29,23 @@ and upgrade processes for self-managed installations that lump together any of t ### Ignoring the column (release M) -The first step is to ignore the column in the application code. This step is -necessary because Rails caches the columns and re-uses this cache in various -places. This can be done by defining the columns to ignore. For example, to ignore +The first step is to ignore the column in the application code and remove all code references to it including +model validations. +This step is necessary because Rails caches the columns and re-uses this cache in various +places. This can be done by defining the columns to ignore. For example, in release `12.5`, to ignore `updated_at` in the User model you'd use the following: ```ruby class User < ApplicationRecord include IgnorableColumns - ignore_column :updated_at, remove_with: '12.7', remove_after: '2020-01-22' + ignore_column :updated_at, remove_with: '12.7', remove_after: '2019-12-22' end ``` Multiple columns can be ignored, too: ```ruby -ignore_columns %i[updated_at created_at], remove_with: '12.7', remove_after: '2020-01-22' +ignore_columns %i[updated_at created_at], remove_with: '12.7', remove_after: '2019-12-22' ``` If the model exists in CE and EE, the column has to be ignored in the CE model. If the @@ -52,21 +53,21 @@ model only exists in EE, then it has to be added there. We require indication of when it is safe to remove the column ignore rule with: -- `remove_with`: set to a GitLab release typically two releases (M+2) after adding the +- `remove_with`: set to a GitLab release typically two releases (M+2) (`12.7` in our example) after adding the column ignore. - `remove_after`: set to a date after which we consider it safe to remove the column - ignore, typically after the M+1 release date, during the M+2 development cycle. + ignore, typically after the M+1 release date, during the M+2 development cycle. For example, since the development cycle of `12.7` is between `2019-12-18` and `2020-01-17`, and `12.6` is the release to [drop the column](#dropping-the-column-release-m1), it's safe to set the date to the release date of `12.6` as `2019-12-22`. This information allows us to reason better about column ignores and makes sure we don't remove column ignores too early for both regular releases and deployments to GitLab.com. For example, this avoids a situation where we deploy a bulk of changes that include both changes to ignore the column and subsequently remove the column ignore (which would result in a downtime). -In this example, the change to ignore the column went into release 12.5. +In this example, the change to ignore the column went into release `12.5`. ### Dropping the column (release M+1) -Continuing our example, dropping the column goes into a _post-deployment_ migration in release 12.6: +Continuing our example, dropping the column goes into a _post-deployment_ migration in release `12.6`: Start by creating the **post-deployment migration**: @@ -135,7 +136,7 @@ for more information about database migrations. ### Removing the ignore rule (release M+2) -With the next release, in this example 12.7, we set up another merge request to remove the ignore rule. +With the next release, in this example `12.7`, we set up another merge request to remove the ignore rule. This removes the `ignore_column` line and - if not needed anymore - also the inclusion of `IgnoreableColumns`. This should only get merged with the release indicated with `remove_with` and once @@ -195,7 +196,7 @@ This step is similar to [the first step when column is dropped](#ignoring-the-co ```ruby class User < ApplicationRecord include IgnorableColumns - ignore_column :updated_at, remove_with: '12.7', remove_after: '2020-01-22' + ignore_column :updated_at, remove_with: '12.7', remove_after: '2019-12-22' end ``` diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md index 1bdd24afcad..10490df7b5e 100644 --- a/doc/development/database/batched_background_migrations.md +++ b/doc/development/database/batched_background_migrations.md @@ -13,6 +13,13 @@ in our guidelines. For example, you can use batched background migrations to migrate data that's stored in a single JSON column to a separate table instead. +NOTE: +Batched background migrations replaced the legacy background migrations framework. +Check that documentation in reference to any changes involving that framework. + +NOTE: +The batched background migrations framework has ChatOps support. Using ChatOps, GitLab engineers can interact with the batched background migrations present in the system. + ## When to use batched background migrations Use a batched background migration when you migrate _data_ in tables containing @@ -201,6 +208,13 @@ models defined in `app/models` except the `ApplicationRecord` classes). Because these migrations can take a long time to run, it's possible for new versions to deploy while the migrations are still running. +### Depending on migrated data + +Unlike a regular or a post migration, waiting for the next release is not enough to guarantee that the data was fully migrated. +That means that you shouldn't depend on the data until the BBM is finished. If having 100% of the data migrated is a requirement, +then, the `ensure_batched_background_migration_is_finished` helper can be used to guarantee that the migration was finished and the +data fully migrated. ([See an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L13-18)). + ## How to ### Generate a batched background migration @@ -538,6 +552,107 @@ Bumping the [import/export version](../../user/project/settings/import_export.md be required, if importing a project from a prior version of GitLab requires the data to be in the new format. +### Add indexes to support batched background migrations + +Sometimes it is necessary to add a new or temporary index to support a batched background migration. +To do this, create the index in a post-deployment migration that precedes the post-deployment +migration that queues the background migration. + +See the documentation for [adding database indexes](adding_database_indexes.md#analyzing-a-new-index-before-a-batched-background-migration) +for additional information about some cases that require special attention to allow the index to be used directly after +creation. + +### Execute a particular batch on the database testing pipeline + +NOTE: +Only [database maintainers](https://gitlab.com/groups/gitlab-org/maintainers/database/-/group_members?with_inherited_permissions=exclude) can view the database testing pipeline artifacts. Ask one for help if you need to use this method. + +Let's assume that a batched background migration failed on a particular batch on GitLab.com and you want to figure out which query failed and why. At the moment, we don't have a good way to retrieve query information (especially the query parameters) and rerunning the entire migration with more logging would be a long process. + +Fortunately you can leverage our [database migration pipeline](database_migration_pipeline.md) to rerun a particular batch with additional logging and/or fix to see if it solves the problem. + +For an example see [Draft: `Test PG::CardinalityViolation` fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110910) but make sure to read the entire section. + +To do that, you need to: + +1. [Find the batch `start_id` and `end_id`](#find-the-batch-start_id-and-end_id) +1. [Create a regular migration](#create-a-regular-migration) +1. [Apply a workaround for our migration helpers](#apply-a-workaround-for-our-migration-helpers-optional) (optional) +1. [Start the database migration pipeline](#start-the-database-migration-pipeline) + +#### Find the batch `start_id` and `end_id` + +You should be able to find those in [Kibana](#viewing-failure-error-logs). + +#### Create a regular migration + +Schedule the batch in the `up` block of a regular migration: + +```ruby +def up + instance = Gitlab::BackgroundMigration::YourBackgroundMigrationClass.new( + start_id: <batch start_id>, + end_id: <batch end_id>, + batch_table: <table name>, + batch_column: <batching column>, + sub_batch_size: <sub batch size>, + pause_ms: <miliseconds between batches>, + job_arguments: <job arguments if any>, + connection: connection + ) + + instance.perform +end + +def down + # no-op +end +``` + +#### Apply a workaround for our migration helpers (optional) + +If your batched background migration touches tables from a schema other than the one you specified by using `restrict_gitlab_migration` helper (example: the scheduling migration has `restrict_gitlab_migration gitlab_schema: :gitlab_main` but the background job uses tables from the `:gitlab_ci` schema) then the migration will fail. To prevent that from happening you must to monkey patch database helpers so they don't fail the testing pipeline job: + +1. Add the schema names to [`RestrictGitlabSchema`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb#L57) + +```diff +diff --git a/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb b/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb +index b8d1d21a0d2d2a23d9e8c8a0a17db98ed1ed40b7..912e20659a6919f771045178c66828563cb5a4a1 100644 +--- a/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb ++++ b/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb +@@ -55,7 +55,7 @@ def unmatched_schemas + end + + def allowed_schemas_for_connection +- Gitlab::Database.gitlab_schemas_for_connection(connection) ++ Gitlab::Database.gitlab_schemas_for_connection(connection) << :gitlab_ci + end + end + end +``` + +1. Add the schema names to [`RestrictAllowedSchemas`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb#L82) + +```diff +diff --git a/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb b/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb +index 4ae3622479f0800c0553959e132143ec9051898e..d556ec7f55adae9d46a56665ce02de782cb09f2d 100644 +--- a/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb ++++ b/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb +@@ -79,7 +79,7 @@ def restrict_to_dml_only(parsed) + tables = self.dml_tables(parsed) + schemas = self.dml_schemas(tables) + +- if (schemas - self.allowed_gitlab_schemas).any? ++ if (schemas - (self.allowed_gitlab_schemas << :gitlab_ci)).any? + raise DMLAccessDeniedError, \ + "Select/DML queries (SELECT/UPDATE/DELETE) do access '#{tables}' (#{schemas.to_a}) " \ + "which is outside of list of allowed schemas: '#{self.allowed_gitlab_schemas}'. " \ +``` + +#### Start the database migration pipeline + +Create a Draft merge request with your changes and trigger the manual `db:gitlabcom-database-testing` job. + ## Managing NOTE: @@ -683,22 +798,109 @@ migration is scheduled in the GitLab FOSS context. You can use the [generator](#generate-a-batched-background-migration) to generate an EE-only migration scaffold by passing `--ee-only` flag when generating a new batched background migration. -## Throttling batched migrations +## Debug + +### Viewing failure error logs -Because batched migrations are update-heavy and there were few incidents in the past because of the heavy load from migrations while the database was underperforming, a throttling mechanism exists to mitigate them. +You can view failures in two ways: -These database indicators are checked to throttle a migration. On getting a -stop signal, the migration is paused for a set time (10 minutes): +- Via GitLab logs: + 1. After running a batched background migration, if any jobs fail, + view the logs in [Kibana](https://log.gprd.gitlab.net/goto/4cb43f40-f861-11ec-b86b-d963a1a6788e). + View the production Sidekiq log and filter for: -- WAL queue pending archival crossing a threshold. -- Active autovacuum on the tables on which the migration works on. -- Patroni apdex SLI dropping below the SLO. + - `json.new_state: failed` + - `json.job_class_name: <Batched Background Migration job class name>` + - `json.job_arguments: <Batched Background Migration job class arguments>` -It's an ongoing effort to add more indicators to further enhance the -database health check framework. For more details, see -[epic 7594](https://gitlab.com/groups/gitlab-org/-/epics/7594). + 1. Review the `json.exception_class` and `json.exception_message` values to help + understand why the jobs failed. + + 1. Remember the retry mechanism. Having a failure does not mean the job failed. + Always check the last status of the job. + +- Via database: + + 1. Get the batched background migration `CLASS_NAME`. + 1. Execute the following query in the PostgreSQL console: + + ```sql + SELECT migration.id, migration.job_class_name, transition_logs.exception_class, transition_logs.exception_message + FROM batched_background_migrations as migration + INNER JOIN batched_background_migration_jobs as jobs + ON jobs.batched_background_migration_id = migration.id + INNER JOIN batched_background_migration_job_transition_logs as transition_logs + ON transition_logs.batched_background_migration_job_id = jobs.id + WHERE transition_logs.next_status = '2' AND migration.job_class_name = "CLASS_NAME"; + ``` + +## Testing + +Writing tests is required for: + +- The batched background migrations' queueing migration. +- The batched background migration itself. +- A cleanup migration. + +The `:migration` and `schema: :latest` RSpec tags are automatically set for +background migration specs. Refer to the +[Testing Rails migrations](../testing_guide/testing_migrations_guide.md#testing-a-non-activerecordmigration-class) +style guide. + +Remember that `before` and `after` RSpec hooks +migrate your database down and up. These hooks can result in other batched background +migrations being called. Using `spy` test doubles with +`have_received` is encouraged, instead of using regular test doubles, because +your expectations defined in a `it` block can conflict with what is +called in RSpec hooks. Refer to [issue #35351](https://gitlab.com/gitlab-org/gitlab/-/issues/18839) +for more details. + +## Best practices + +1. Know how much data you're dealing with. +1. Make sure the batched background migration jobs are idempotent. +1. Confirm the tests you write are not false positives. +1. If the data being migrated is critical and cannot be lost, the + clean-up migration must also check the final state of the data before completing. +1. Discuss the numbers with a database specialist. The migration may add + more pressure on DB than you expect. Measure on staging, + or ask someone to measure on production. +1. Know how much time is required to run the batched background migration. +1. Be careful when silently rescuing exceptions inside job classes. This may lead to + jobs being marked as successful, even in a failure scenario. + + ```ruby + # good + def perform + each_sub_batch do |sub_batch| + sub_batch.update_all(name: 'My Name') + end + end + + # acceptable + def perform + each_sub_batch do |sub_batch| + sub_batch.update_all(name: 'My Name') + rescue Exception => error + logger.error(message: error.message, class: error.class) + + raise + end + end + + # bad + def perform + each_sub_batch do |sub_batch| + sub_batch.update_all(name: 'My Name') + rescue Exception => error + logger.error(message: error.message, class: self.class.name) + end + end + ``` + +## Examples -## Example +### Routes use-case The `routes` table has a `source_type` field that's used for a polymorphic relationship. As part of a database redesign, we're removing the polymorphic relationship. One step of @@ -867,216 +1069,3 @@ background migration. After the batched migration is completed, you can safely depend on the data in `routes.namespace_id` being populated. - -## Testing - -Writing tests is required for: - -- The batched background migrations' queueing migration. -- The batched background migration itself. -- A cleanup migration. - -The `:migration` and `schema: :latest` RSpec tags are automatically set for -background migration specs. Refer to the -[Testing Rails migrations](../testing_guide/testing_migrations_guide.md#testing-a-non-activerecordmigration-class) -style guide. - -Remember that `before` and `after` RSpec hooks -migrate your database down and up. These hooks can result in other batched background -migrations being called. Using `spy` test doubles with -`have_received` is encouraged, instead of using regular test doubles, because -your expectations defined in a `it` block can conflict with what is -called in RSpec hooks. Refer to [issue #35351](https://gitlab.com/gitlab-org/gitlab/-/issues/18839) -for more details. - -## Best practices - -1. Know how much data you're dealing with. -1. Make sure the batched background migration jobs are idempotent. -1. Confirm the tests you write are not false positives. -1. If the data being migrated is critical and cannot be lost, the - clean-up migration must also check the final state of the data before completing. -1. Discuss the numbers with a database specialist. The migration may add - more pressure on DB than you expect. Measure on staging, - or ask someone to measure on production. -1. Know how much time is required to run the batched background migration. -1. Be careful when silently rescuing exceptions inside job classes. This may lead to - jobs being marked as successful, even in a failure scenario. - - ```ruby - # good - def perform - each_sub_batch do |sub_batch| - sub_batch.update_all(name: 'My Name') - end - end - - # acceptable - def perform - each_sub_batch do |sub_batch| - sub_batch.update_all(name: 'My Name') - rescue Exception => error - logger.error(message: error.message, class: error.class) - - raise - end - end - - # bad - def perform - each_sub_batch do |sub_batch| - sub_batch.update_all(name: 'My Name') - rescue Exception => error - logger.error(message: error.message, class: self.class.name) - end - end - ``` - -## Additional tips and strategies - -### ChatOps integration - -The batched background migrations framework has ChatOps support. Using ChatOps, GitLab engineers can interact with the batched background migrations present in the system. - -### Viewing failure error logs - -You can view failures in two ways: - -- Via GitLab logs: - 1. After running a batched background migration, if any jobs fail, - view the logs in [Kibana](https://log.gprd.gitlab.net/goto/4cb43f40-f861-11ec-b86b-d963a1a6788e). - View the production Sidekiq log and filter for: - - - `json.new_state: failed` - - `json.job_class_name: <Batched Background Migration job class name>` - - `json.job_arguments: <Batched Background Migration job class arguments>` - - 1. Review the `json.exception_class` and `json.exception_message` values to help - understand why the jobs failed. - - 1. Remember the retry mechanism. Having a failure does not mean the job failed. - Always check the last status of the job. - -- Via database: - - 1. Get the batched background migration `CLASS_NAME`. - 1. Execute the following query in the PostgreSQL console: - - ```sql - SELECT migration.id, migration.job_class_name, transition_logs.exception_class, transition_logs.exception_message - FROM batched_background_migrations as migration - INNER JOIN batched_background_migration_jobs as jobs - ON jobs.batched_background_migration_id = migration.id - INNER JOIN batched_background_migration_job_transition_logs as transition_logs - ON transition_logs.batched_background_migration_job_id = jobs.id - WHERE transition_logs.next_status = '2' AND migration.job_class_name = "CLASS_NAME"; - ``` - -### Executing a particular batch on the database testing pipeline - -NOTE: -Only [database maintainers](https://gitlab.com/groups/gitlab-org/maintainers/database/-/group_members?with_inherited_permissions=exclude) can view the database testing pipeline artifacts. Ask one for help if you need to use this method. - -Let's assume that a batched background migration failed on a particular batch on GitLab.com and you want to figure out which query failed and why. At the moment, we don't have a good way to retrieve query information (especially the query parameters) and rerunning the entire migration with more logging would be a long process. - -Fortunately you can leverage our [database migration pipeline](database_migration_pipeline.md) to rerun a particular batch with additional logging and/or fix to see if it solves the problem. - -<!-- vale gitlab.Substitutions = NO --> - -For an example see [Draft: Test PG::CardinalityViolation fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110910) but make sure to read the entire section. - -To do that, you need to: - -1. Find the batch `start_id` and `end_id` -1. Create a regular migration -1. Apply a workaround for our migration helpers (optional) -1. Start the database migration pipeline - -#### 1. Find the batch `start_id` and `end_id` - -You should be able to find those in [Kibana](#viewing-failure-error-logs). - -#### 2. Create a regular migration - -Schedule the batch in the `up` block of a regular migration: - -```ruby -def up - instance = Gitlab::BackgroundMigration::YourBackgroundMigrationClass.new( - start_id: <batch start_id>, - end_id: <batch end_id>, - batch_table: <table name>, - batch_column: <batching column>, - sub_batch_size: <sub batch size>, - pause_ms: <miliseconds between batches>, - job_arguments: <job arguments if any>, - connection: connection - ) - - instance.perform -end - - -def down - # no-op -end -``` - -#### 3. Apply a workaround for our migration helpers (optional) - -If your batched background migration touches tables from a schema other than the one you specified by using `restrict_gitlab_migration` helper (example: the scheduling migration has `restrict_gitlab_migration gitlab_schema: :gitlab_main` but the background job uses tables from the `:gitlab_ci` schema) then the migration will fail. To prevent that from happening you must to monkey patch database helpers so they don't fail the testing pipeline job: - -1. Add the schema names to [`RestrictGitlabSchema`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb#L57) - -```diff -diff --git a/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb b/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb -index b8d1d21a0d2d2a23d9e8c8a0a17db98ed1ed40b7..912e20659a6919f771045178c66828563cb5a4a1 100644 ---- a/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb -+++ b/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb -@@ -55,7 +55,7 @@ def unmatched_schemas - end - - def allowed_schemas_for_connection -- Gitlab::Database.gitlab_schemas_for_connection(connection) -+ Gitlab::Database.gitlab_schemas_for_connection(connection) << :gitlab_ci - end - end - end -``` - -1. Add the schema names to [`RestrictAllowedSchemas`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb#L82) - -```diff -diff --git a/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb b/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb -index 4ae3622479f0800c0553959e132143ec9051898e..d556ec7f55adae9d46a56665ce02de782cb09f2d 100644 ---- a/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb -+++ b/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb -@@ -79,7 +79,7 @@ def restrict_to_dml_only(parsed) - tables = self.dml_tables(parsed) - schemas = self.dml_schemas(tables) - -- if (schemas - self.allowed_gitlab_schemas).any? -+ if (schemas - (self.allowed_gitlab_schemas << :gitlab_ci)).any? - raise DMLAccessDeniedError, \ - "Select/DML queries (SELECT/UPDATE/DELETE) do access '#{tables}' (#{schemas.to_a}) " \ - "which is outside of list of allowed schemas: '#{self.allowed_gitlab_schemas}'. " \ -``` - -#### 4. Start the database migration pipeline - -Create a Draft merge request with your changes and trigger the manual `db:gitlabcom-database-testing` job. - -### Adding indexes to support batched background migrations - -Sometimes it is necessary to add a new or temporary index to support a batched background migration. -To do this, create the index in a post-deployment migration that precedes the post-deployment -migration that queues the background migration. - -See the documentation for [adding database indexes](adding_database_indexes.md#analyzing-a-new-index-before-a-batched-background-migration) -for additional information about some cases that require special attention to allow the index to be used directly after -creation. - -## Legacy background migrations - -Batched background migrations replaced the legacy background migrations framework. -Check that documentation in reference to any changes involving that framework. diff --git a/doc/development/database/database_lab.md b/doc/development/database/database_lab.md index 357133d8bca..7edb8ab4de5 100644 --- a/doc/development/database/database_lab.md +++ b/doc/development/database/database_lab.md @@ -146,13 +146,13 @@ the `pgai` Gem: 1. To get started, you need to gather some values from the [Postgres.ai instances page](https://console.postgres.ai/gitlab/instances): - 1. Navigate to the instance that you want to configure and the on right side of the screen. + 1. Go to the instance that you want to configure and the on right side of the screen. 1. Under **Connection**, select **Connect**. The menu might be collapsed. - A pop-up with everything that's needed for configuration appears, using this format: + A dialog with everything that's needed for configuration appears, using this format: ```shell - dblab init --url http://127.0.0.1:1234 --token TOKEN --environment-id <environment-id> + dblab init --url "http://127.0.0.1:1234" --token TOKEN --environment-id <environment-id> ``` ```shell diff --git a/doc/development/database/database_migration_pipeline.md b/doc/development/database/database_migration_pipeline.md index a9d525e2a41..280254d90b0 100644 --- a/doc/development/database/database_migration_pipeline.md +++ b/doc/development/database/database_migration_pipeline.md @@ -44,6 +44,13 @@ The next section of the comment contains detailed information for each migration calls, timings, and the number of the changed rows. - **Runtime histogram** - Indicates the distribution of query times for the migration. +### Database size increase + +Occasionally, a migration shows a +8.00 KiB size increase, even if the migration was not +expected to result in a size increase. Completing any migration adds a row to the +`schema_migrations` table, which may require a new disk page to be created. +If a new disk page is created, the size of the database will grow by exactly 8 KiB. + ## Background migration details The next section of the comment contains detailed information about each batched background migration, including: diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index c297c90918f..c75503c503b 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -68,8 +68,8 @@ The following guides provide a quick introduction and links to follow on more ad - Guide on [understanding EXPLAIN plans](understanding_explain_plans.md). - [Explaining the unexplainable series in `depesz`](https://www.depesz.com/tag/unexplainable/). -We also have licensed access to The Art of PostgreSQL. If you are interested in getting access, check out the -[issue (confidential)](https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/23). +We also have licensed access to The Art of PostgreSQL. If you are interested in getting access, GitLab team +members can check out the issue here: `https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/23`. Finally, you can find various guides in the [Database guides](index.md) page that cover more specific topics and use cases. The most frequently required during database reviewing are the following: diff --git a/doc/development/database/efficient_in_operator_queries.md b/doc/development/database/efficient_in_operator_queries.md index a0c71f49e2d..87976df5711 100644 --- a/doc/development/database/efficient_in_operator_queries.md +++ b/doc/development/database/efficient_in_operator_queries.md @@ -228,9 +228,6 @@ Gitlab::Pagination::Keyset::InOperatorOptimization::QueryBuilder.new( If it's not given, the `IN` operator optimization only makes the `ORDER BY` columns available to the end-user and not the full database row. - If it's not given, the `IN` operator optimization only makes the `ORDER BY` columns available to - the end-user and not the full database row. - The following database index on the `issues` table must be present to make the query execute efficiently: diff --git a/doc/development/database/iterating_tables_in_batches.md b/doc/development/database/iterating_tables_in_batches.md index a927242e8d8..84b82b16255 100644 --- a/doc/development/database/iterating_tables_in_batches.md +++ b/doc/development/database/iterating_tables_in_batches.md @@ -135,11 +135,24 @@ Let's consider that we want to iterate over the `users` table and print the `Use standard output. The `users` table contains millions of records, thus running one query to fetch the users likely times out. - - -This is a simplified version of the `users` table which contains several rows. We have a few +This table is a simplified version of the `users` table which contains several rows. We have a few smaller gaps in the `id` column to make the example a bit more realistic (a few records were -already deleted). Currently, we have one index on the `id` field. +already deleted). One index exists on the `id` field: + +| `ID` | `sign_in_count` | `created_at` | +| -- | :-------------: | ------------ | +| 1 | 1 | 2020-01-01 +| 2 | 4 | 2020-01-01 +| 9 | 1 | 2020-01-03 +| 300 | 5 | 2020-01-03 +| 301 | 9 | 2020-01-03 +| 302 | 8 | 2020-01-03 +| 303 | 2 | 2020-01-03 +| 350 | 1 | 2020-01-03 +| 351 | 3 | 2020-01-04 +| 352 | 0 | 2020-01-05 +| 353 | 9 | 2020-01-11 +| 354 | 3 | 2020-01-12 Loading all users into memory (avoid): diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index 7c6b94144b4..7037ab22983 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -163,6 +163,92 @@ allows you to avoid the join, while still avoiding the N+1 query. You can see a real example of this solution being used in <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67655>. +##### Remove a redundant join + +Sometimes there are cases where a query is doing excess (or redundant) joins. + +A common example occurs where a query is joining from `A` to `C`, via some +table with both foreign keys, `B`. +When you only care about counting how +many rows there are in `C` and if there are foreign keys and `NOT NULL` constraints +on the foreign keys in `B`, then it might be enough to count those rows. +For example, in +[MR 71811](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71811), it was +previously doing `project.runners.count`, which would produce a query like: + +```sql +select count(*) from projects +inner join ci_runner_projects on ci_runner_projects.project_id = projects.id +where ci_runner_projects.runner_id IN (1, 2, 3) +``` + +This was changed to avoid the cross-join by changing the code to +`project.runner_projects.count`. It produces the same response with the +following query: + +```sql +select count(*) from ci_runner_projects +where ci_runner_projects.runner_id IN (1, 2, 3) +``` + +Another common redundant join is joining all the way to another table, +then filtering by primary key when you could have instead filtered on a foreign +key. See an example in +[MR 71614](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71614). The previous +code was `joins(scan: :build).where(ci_builds: { id: build_ids })`, which +generated a query like: + +```sql +select ... +inner join security_scans +inner join ci_builds on security_scans.build_id = ci_builds.id +where ci_builds.id IN (1, 2, 3) +``` + +However, as `security_scans` already has a foreign key `build_id`, the code +can be changed to `joins(:scan).where(security_scans: { build_id: build_ids })`, +which produces the same response with the following query: + +```sql +select ... +inner join security_scans +where security_scans.build_id IN (1, 2, 3) +``` + +Both of these examples of removing redundant joins remove the cross-joins, +but they have the added benefit of producing simpler and faster +queries. + +##### Limited pluck followed by a find + +Using `pluck` or `pick` to get an array of `id`s is not advisable unless the returned +array is guaranteed to be bounded in size. Usually this is a good pattern where +you know the result will be at most 1, or in cases where you have a list of in +memory ids (or usernames) that need to be mapped to another list of equal size. +It would not be suitable when mapping a list of ids in a one-to-many +relationship as the result will be unbounded. We can then use the +returned `id`s to obtain the related record: + +```ruby +allowed_user_id = board_user_finder + .where(user_id: params['assignee_id']) + .pick(:user_id) + +User.find_by(id: allowed_user_id) +``` + +You can see an example where this was used in +<https://gitlab.com/gitlab-org/gitlab/-/merge_requests/126856> + +Sometimes it might seem easy to convert a join into a `pluck` but often this +results in loading an unbounded amount of ids into memory and then +re-serializing those in a following query back to Postgres. These cases do not +scale and we recommend attempting one of the other options. It might seem like a +good idea to just apply some `limit` to the plucked data to have bounded memory +but this introduces unpredictable results for users and often is most +problematic for our largest customers (including ourselves), and as such we +advise against it. + ##### De-normalize some foreign key to the table De-normalization refers to adding redundant precomputed (duplicated) data to @@ -263,62 +349,6 @@ logic to delete these rows if or whenever necessary in your domain. Finally, this de-normalization and new query also improves performance because it does less joins and needs less filtering. -##### Remove a redundant join - -Sometimes there are cases where a query is doing excess (or redundant) joins. - -A common example occurs where a query is joining from `A` to `C`, via some -table with both foreign keys, `B`. -When you only care about counting how -many rows there are in `C` and if there are foreign keys and `NOT NULL` constraints -on the foreign keys in `B`, then it might be enough to count those rows. -For example, in -[MR 71811](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71811), it was -previously doing `project.runners.count`, which would produce a query like: - -```sql -select count(*) from projects -inner join ci_runner_projects on ci_runner_projects.project_id = projects.id -where ci_runner_projects.runner_id IN (1, 2, 3) -``` - -This was changed to avoid the cross-join by changing the code to -`project.runner_projects.count`. It produces the same response with the -following query: - -```sql -select count(*) from ci_runner_projects -where ci_runner_projects.runner_id IN (1, 2, 3) -``` - -Another common redundant join is joining all the way to another table, -then filtering by primary key when you could have instead filtered on a foreign -key. See an example in -[MR 71614](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71614). The previous -code was `joins(scan: :build).where(ci_builds: { id: build_ids })`, which -generated a query like: - -```sql -select ... -inner join security_scans -inner join ci_builds on security_scans.build_id = ci_builds.id -where ci_builds.id IN (1, 2, 3) -``` - -However, as `security_scans` already has a foreign key `build_id`, the code -can be changed to `joins(:scan).where(security_scans: { build_id: build_ids })`, -which produces the same response with the following query: - -```sql -select ... -inner join security_scans -where security_scans.build_id IN (1, 2, 3) -``` - -Both of these examples of removing redundant joins remove the cross-joins, -but they have the added benefit of producing simpler and faster -queries. - ##### Use `disable_joins` for `has_one` or `has_many` `through:` relations Sometimes a join query is caused by using `has_one ... through:` or `has_many @@ -628,3 +658,56 @@ If this task was run against a GitLab setup that uses only a single database for both `gitlab_main` and `gitlab_ci` tables, then no tables will be locked. To undo the operation, run the opposite Rake task: `gitlab:db:unlock_writes`. + +## Truncating tables + +When the databases `main` and `ci` are fully split, we can free up disk +space by truncating tables. This results in a smaller data set: For example, +the data in `users` table on CI database is no longer read and also no +longer updated. So this data can be removed by truncating the tables. + +For this purpose, GitLab provides two Rake tasks, one for each database: + +- `gitlab:db:truncate_legacy_tables:main` will truncate the CI tables in Main database. +- `gitlab:db:truncate_legacy_tables:ci` will truncate the Main tables in CI database. + +NOTE: +These tasks can only be run when the tables in the database are +[locked for writes](#locking-writes-on-the-tables-that-dont-belong-to-the-database-schemas). + +WARNING: +The examples in this section use `DRY_RUN=true`. This ensures no data is actually +truncated. GitLab highly recommends to have a backup available before you run any of +these tasks without `DRY_RUN=true`. + +These tasks have the option to see what they do without actually changing the +data: + +```shell +$ sudo DRY_RUN=true gitlab-rake gitlab:db:truncate_legacy_tables:main +I, [2023-07-14T17:08:06.665151 #92505] INFO -- : DRY RUN: +I, [2023-07-14T17:08:06.761586 #92505] INFO -- : Truncating legacy tables for the database main +I, [2023-07-14T17:08:06.761709 #92505] INFO -- : SELECT set_config('lock_writes.ci_build_needs', 'false', false) +I, [2023-07-14T17:08:06.765272 #92505] INFO -- : SELECT set_config('lock_writes.ci_build_pending_states', 'false', false) +I, [2023-07-14T17:08:06.768220 #92505] INFO -- : SELECT set_config('lock_writes.ci_build_report_results', 'false', false) +[...] +I, [2023-07-14T17:08:06.957294 #92505] INFO -- : TRUNCATE TABLE ci_build_needs, ci_build_pending_states, ci_build_report_results, ci_build_trace_chunks, ci_build_trace_metadata, ci_builds, ci_builds_metadata, ci_builds_runner_session, ci_cost_settings, ci_daily_build_group_report_results, ci_deleted_objects, ci_editor_ai_conversation_messages, ci_freeze_periods, ci_group_variables, ci_instance_variables, ci_job_artifact_states, ci_job_artifacts, ci_job_token_project_scope_links, ci_job_variables, ci_minutes_additional_packs, ci_namespace_mirrors, ci_namespace_monthly_usages, ci_partitions, ci_pending_builds, ci_pipeline_artifacts, ci_pipeline_chat_data, ci_pipeline_messages, ci_pipeline_metadata, ci_pipeline_schedule_variables, ci_pipeline_schedules, ci_pipeline_variables, ci_pipelines, ci_pipelines_config, ci_platform_metrics, ci_project_mirrors, ci_project_monthly_usages, ci_refs, ci_resource_groups, ci_resources, ci_runner_machines, ci_runner_namespaces, ci_runner_projects, ci_runner_versions, ci_runners, ci_running_builds, ci_secure_file_states, ci_secure_files, ci_sources_pipelines, ci_sources_projects, ci_stages, ci_subscriptions_projects, ci_trigger_requests, ci_triggers, ci_unit_test_failures, ci_unit_tests, ci_variables, external_pull_requests, p_ci_builds, p_ci_builds_metadata, p_ci_job_annotations, p_ci_runner_machine_builds, taggings, tags RESTRICT +``` + +The tasks will first find out the tables that need to be truncated. Truncation will +happen in stages because we need to limit the amount of data removed in one database +transaction. The tables are processed in a specific order depending on the definition +of the foreign keys. The number of tables processed in one stage can be changed by +adding a number when invoking the task. The default value is 5: + +```shell +sudo DRY_RUN=true gitlab-rake gitlab:db:truncate_legacy_tables:main\[10\] +``` + +It is also possible to limit the number of tables to be truncated by setting the `UNTIL_TABLE` +variable. For example in this case, the process will stop when `ci_unit_test_failures` has been +truncated: + +```shell +sudo DRY_RUN=true UNTIL_TABLE=ci_unit_test_failures gitlab-rake gitlab:db:truncate_legacy_tables:main +``` diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md index 2b10e440799..e1b6868c68e 100644 --- a/doc/development/database/not_null_constraints.md +++ b/doc/development/database/not_null_constraints.md @@ -63,9 +63,10 @@ The steps required are: 1. Release `N.M` (current release) - - Ensure the constraint is enforced at the application level (that is, add a model validation). - - Add a post-deployment migration to add the `NOT NULL` constraint with `validate: false`. - - Add a post-deployment migration to fix the existing records. + 1. Ensure $ATTRIBUTE value is being set at the application level. + 1. If the attribute has a default value, add the default value to the model so the default value is set for new records. + 1. Update all places in the code where the attribute is being set to `nil`, if any, for new and existing records. + 1. Add a post-deployment migration to fix the existing records. NOTE: Depending on the size of the table, a background migration for cleanup could be required in the next release. @@ -75,22 +76,16 @@ The steps required are: 1. Release `N.M+1` (next release) - - Validate the `NOT NULL` constraint using a post-deployment migration. + 1. Make sure all existing records on GitLab.com have attribute set. If not, go back to step 1 from Release `N.M`. + 1. If step 1 seems fine and the backfill from Release `N.M` was done via a batched background migration then add a + post-deployment migration to + [finalize the background migration](batched_background_migrations.md#depending-on-migrated-data). + 1. Add a validation for the attribute in the model to prevent records with `nil` attribute as now all existing and new records should be valid. + 1. Add a post-deployment migration to add the `NOT NULL` constraint. ### Example -Considering a given release milestone, such as 13.0, a model validation has been added into `epic.rb` -to require a description: - -```ruby -class Epic < ApplicationRecord - validates :description, presence: true -end -``` - -The same constraint should be added at the database level for consistency purposes. -We only want to enforce the `NOT NULL` constraint without setting a default, as we have decided -that all epics should have a user-generated description. +Considering a given release milestone, such as 13.0. After checking our production database, we know that there are `epics` with `NULL` descriptions, so we cannot add and validate the constraint in one step. @@ -101,33 +96,16 @@ such records, so we would follow the same process either way. #### Prevent new invalid records (current release) -We first add the `NOT NULL` constraint with a `NOT VALID` parameter, which enforces consistency -when new records are inserted or current records are updated. - -In the example above, the existing epics with a `NULL` description are not affected and you are -still able to update records in the `epics` table. However, when you try to update or insert -an epic without providing a description, the constraint causes a database error. - -Adding or removing a `NOT NULL` clause requires that any application changes are deployed _first_. -Thus, adding a `NOT NULL` constraint to an existing column should happen in a post-deployment migration. +Update all the code paths where the attribute is being set to `nil`, if any, to set the attribute to non-nil value +for new and existing records. -Still in our example, for the 13.0 milestone example (current), we add the `NOT NULL` constraint -with `validate: false` in a post-deployment migration, -`db/post_migrate/20200501000001_add_not_null_constraint_to_epics_description.rb`: +An attribute with default using the +[Rails attributes API](https://api.rubyonrails.org/classes/ActiveRecord/Attributes/ClassMethods.html) has been added in +`epic.rb` so that default value is set for new records: ```ruby -class AddNotNullConstraintToEpicsDescription < Gitlab::Database::Migration[2.1] - disable_ddl_transaction! - - def up - # This will add the `NOT NULL` constraint WITHOUT validating it - add_not_null_constraint :epics, :description, validate: false - end - - def down - # Down is required as `add_not_null_constraint` is not reversible - remove_not_null_constraint :epics, :description - end +class Epic < ApplicationRecord + attribute :description, default: 'No description' end ``` @@ -176,24 +154,47 @@ class CleanupEpicsWithNullDescription < Gitlab::Database::Migration[2.1] end ``` -#### Validate the `NOT NULL` constraint (next release) +#### Check if all records are fixed (next release) + +Use postgres.ai to [create a thin clone](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/database/doc/gitlab-com-database.html#use-postgresai-to-work-with-a-thin-clone-of-the-database-includes-direct-psql-access-to-the-thin-clone) +of the production database and check if all records on GitLab.com have the attribute set. +If not go back to [Prevent new invalid records](#prevent-new-invalid-records-current-release) step and figure out where +in the code the attribute is explicitly set to `nil`. Fix the code path then reschedule the migration to fix the existing +records and wait for the next release to do the following steps. + +#### Finalize the background migration (next release) + +If the migration was done using a background migration then [finalize the migration](batched_background_migrations.md#depending-on-migrated-data). -Validating the `NOT NULL` constraint scans the whole table and make sure that each record is correct. +#### Add validation to the model (next release) -Still in our example, for the 13.1 milestone (next), we run the `validate_not_null_constraint` -migration helper in a final post-deployment migration, -`db/post_migrate/20200601000001_validate_not_null_constraint_on_epics_description.rb`: +Add a validation for the attribute to the model to prevent records with `nil` attribute as now all existing and new records should be valid. ```ruby -class ValidateNotNullConstraintOnEpicsDescription < Gitlab::Database::Migration[2.1] +class Epic < ApplicationRecord + validates :description, presence: true +end +``` + +#### Add the `NOT NULL` constraint (next release) + +Adding the `NOT NULL` constraint scans the whole table and make sure that each record is correct. + +Still in our example, for the 13.1 milestone (next), we run the `add_not_null_constraint` +migration helper in a final post-deployment migration: + +```ruby +class AddNotNullConstraintToEpicsDescription < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up - validate_not_null_constraint :epics, :description + # This will add the `NOT NULL` constraint and validate it + add_not_null_constraint :epics, :description end def down - # no-op + # Down is required as `add_not_null_constraint` is not reversible + remove_not_null_constraint :epics, :description end end ``` diff --git a/doc/development/database/pagination_guidelines.md b/doc/development/database/pagination_guidelines.md index d6550d0a515..8b07dcada05 100644 --- a/doc/development/database/pagination_guidelines.md +++ b/doc/development/database/pagination_guidelines.md @@ -218,7 +218,9 @@ We can argue that a typical user does not visit these pages, however, API users ### Keyset pagination -Keyset pagination addresses the performance concerns of "skipping" previous rows when requesting a large page, however, it's not a drop-in replacement for offset-based pagination. Keyset pagination is used only in the [GraphQL API](../graphql_guide/pagination.md) +Keyset pagination addresses the performance concerns of "skipping" previous rows when requesting a large page, however, it's not a drop-in replacement for offset-based pagination. When moving an API endpoint from offset-based pagination to keyset-based pagination, both must be supported. Removing one type of pagination entirely is a [breaking changes](../../update/terminology.md#breaking-change). + +Keyset pagination used in both the [GraphQL API](../graphql_guide/pagination.md#keyset-pagination) and the [REST API](../../api/rest/index.md#keyset-based-pagination). Consider the following `issues` table: diff --git a/doc/development/database/query_recorder.md b/doc/development/database/query_recorder.md index bae211c1618..39d54954268 100644 --- a/doc/development/database/query_recorder.md +++ b/doc/development/database/query_recorder.md @@ -20,7 +20,7 @@ This style of test works by counting the number of SQL queries executed by Activ it "avoids N+1 database queries" do control = ActiveRecord::QueryRecorder.new { visit_some_page } create_list(:issue, 5) - expect { visit_some_page }.not_to exceed_query_limit(control) + expect { visit_some_page }.to issue_same_number_of_queries_as(control) end ``` @@ -33,13 +33,15 @@ it "avoids N+1 database queries" do create_list(:issue, 5) action = ActiveRecord::QueryRecorder.new { visit_some_page } - expect(action).not_to exceed_query_limit(control) + expect(action).to issue_same_number_of_queries_as(control) end ``` As an example you might create 5 issues in between counts, which would cause the query count to increase by 5 if an N+1 problem exists. -In some cases, the query count might change slightly between runs for unrelated reasons. In this case you might need to test `exceed_query_limit(control_count + acceptable_change)`, but this should be avoided if possible. +In some cases, the query count might change slightly between runs for unrelated reasons. +In this case you might need to test `issue_same_number_of_queries_as(control_count + acceptable_change)`, +but this should be avoided if possible. If this test fails, and the control was passed as a `QueryRecorder`, then the failure message indicates where the extra queries are by matching queries on @@ -50,18 +52,45 @@ warm the cache, second one to establish a control, third one to validate that there are no N+1 queries. Rather than make an extra request to warm the cache, prefer two requests (control and test) and configure your test to ignore [cached queries](#cached-queries) in N+1 specs. +```ruby +it "avoids N+1 database queries" do + # warm up + visit_some_page + + control = ActiveRecord::QueryRecorder.new(skip_cached: true) { visit_some_page } + create_list(:issue, 5) + expect { visit_some_page }.to issue_same_number_of_queries_as(control) +end +``` + ## Cached queries -By default, QueryRecorder ignores [cached queries](../merge_request_concepts/performance.md#cached-queries) in the count. However, it may be better to count -all queries to avoid introducing an N+1 query that may be masked by the statement cache. +By default, QueryRecorder ignores [cached queries](../merge_request_concepts/performance.md#cached-queries) in the count. +However, it may be better to count all queries to avoid introducing an N+1 query that may be masked by the statement cache. To do this, this requires the `:use_sql_query_cache` flag to be set. -You should pass the `skip_cached` variable to `QueryRecorder` and use the `exceed_all_query_limit` matcher: +You should pass the `skip_cached` variable to `QueryRecorder` and use the `issue_same_number_of_queries_as` matcher: ```ruby it "avoids N+1 database queries", :use_sql_query_cache do control = ActiveRecord::QueryRecorder.new(skip_cached: false) { visit_some_page } create_list(:issue, 5) - expect { visit_some_page }.not_to exceed_all_query_limit(control) + expect { visit_some_page }.to issue_same_number_of_queries_as(control) +end +``` + +## Using RequestStore + +[`RequestStore` / `Gitlab::SafeRequestStore`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/gems/gitlab-safe_request_store/README.md) +helps us to avoid N+1 queries by caching data in memory for the duration of a request. However, it is disabled by default in tests +and can lead to false negatives when testing for N+1 queries. + +To enable `RequestStore` in tests, use the `request_store` helper when needed: + +```ruby +it "avoids N+1 database queries", :request_store do + control = ActiveRecord::QueryRecorder.new(skip_cached: true) { visit_some_page } + create_list(:issue, 5) + expect { visit_some_page }.to issue_same_number_of_queries_as(control) end ``` @@ -72,6 +101,11 @@ Use a [request spec](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/req Controller specs should not be used to write N+1 tests as the controller is only initialized once per example. This could lead to false successes where subsequent "requests" could have queries reduced (for example, because of memoization). +## Never trust a test you haven't seen fail + +Before you add a test for N+1 queries, you should first verify that the test fails without your change. +This is because the test may be broken, or the test may be passing for the wrong reasons. + ## Finding the source of the query There are multiple ways to find the source of queries. diff --git a/doc/development/database/single_table_inheritance.md b/doc/development/database/single_table_inheritance.md index dcf696b85bc..40b608bd110 100644 --- a/doc/development/database/single_table_inheritance.md +++ b/doc/development/database/single_table_inheritance.md @@ -6,22 +6,58 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Single Table Inheritance -**Summary:** don't use Single Table Inheritance (STI), use separate tables -instead. +**Summary:** Don't design new tables using Single Table Inheritance (STI). For existing tables that use STI as a pattern, avoid adding new types, and consider splitting them into separate tables. -Rails makes it possible to have multiple models stored in the same table and map -these rows to the correct models using a `type` column. This can be used to for -example store two different types of SSH keys in the same table. +STI is a database design pattern where a single table stores +different types of records. These records have a subset of shared columns and another column +that instructs the application which object that record should be represented by. +This can be used to for example store two different types of SSH keys in the same +table. ActiveRecord makes use of it and provides some features that make STI usage +more convenient. -While tempting to use one should avoid this at all costs for the same reasons as -outlined in the document ["Polymorphic Associations"](polymorphic_associations.md). +We no longer allow new STI tables because they: -## Solution +- Lead to tables with large number of rows, when we should strive to keep tables small. +- Need additional indexes, increasing our usage of lightweight locks, whose saturation can cause incidents. +- Add overhead by having to filter all of the data by a value, leading to more page accesses on read. +- Use the `class_name` to load the correct class for an object, but storing + the class name is costly and unnecessary. -The solution is very simple: just use a separate table for every type you'd -otherwise store in the same table. For example, instead of having a `keys` table -with `type` set to either `Key` or `DeployKey` you'd have two separate tables: -`keys` and `deploy_keys`. +Instead of using STI, consider the following alternatives: + +- Use a different table for each type. +- Avoid adding `*_type` columns. This is a code smell that might indicate that new types will be added in the future, and refactoring in the future will be much harder. +- If you already have a table that is effectively an STI on a `_type` column, consider: + - Splitting the existent data into multiple tables. + - Refactoring so that new types can be added as new tables while keeping existing ones (for example, move logic of the base class into a concern). + +If, **after considering all of the above downsides and alternatives**, STI +is the only solution for the problem at hand, we can at least avoid the +issues with saving the class name in the record by using an enum type +instead and the `EnumInheritance` concern: + +```ruby +class Animal < ActiveRecord::Base + include EnumInheritance + + enum species: { + dog: 1, + cat: 2 + } + + def self.inheritance_column_to_class_map = { + dog: 'Dog', + cat: 'Cat' + } + + def self.inheritance_column = 'species' +end + +class Dog < Animal; end +class Cat < Animal; end +``` + +If your table already has a `*_type`, new classes for the different types can be added as needed. ## In migrations diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index 54bdeaa7b28..adb715e219d 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -129,8 +129,6 @@ class AddExtendedTitleToSprints < Gitlab::Database::Migration[2.1] end def down - remove_text_limit :sprints, :extended_title - with_lock_retries do remove_column :sprints, :extended_title, if_exists: true end diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 42021a5ae95..bb0bfbc759b 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -29,6 +29,8 @@ A database review is required for: These metrics could have complex queries over large tables. See the [Analytics Instrumentation Guide](https://about.gitlab.com/handbook/product/analytics-instrumentation-guide/) for implementation details. +- Changes that use [`update`, `delete`, `update_all`, `delete_all` or `destroy_all`](#preparation-when-using-update-delete-update_all-delete_all-or-destroy_all) + methods on an ActiveRecord object. A database reviewer is expected to look out for overly complex queries in the change and review those closer. If the author does not @@ -216,6 +218,16 @@ Include in the MR description: - If you're adding a composite index, another index might become redundant, so remove that in the same migration. For example adding `index(column_A, column_B, column_C)` makes the indexes `index(column_A, column_B)` and `index(column_A)` redundant. +#### Preparation when using `update`, `delete`, `update_all`, `delete_all` or `destroy_all` + +Using these ActiveRecord methods requires extra care because they modify data and can perform poorly, or they +can destroy data if improperly scoped. These methods are also +[incompatible with Common Table Expression (CTE) statements](sql.md#when-to-use-common-table-expressions). +Danger will comment on a Merge Request Diff when these methods are used. + +Follow documentation for [preparation when adding or modifying queries](#preparation-when-adding-or-modifying-queries) +to add the raw SQL query and query plan to the Merge Request description, and request a database review. + ### How to review for database - Check migrations diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index af89da5ec65..d586f25ffbf 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -45,7 +45,7 @@ For versioning and upgrade details, see our [Release and Maintenance policy](../ GitLab self-managed packages are semantically versioned and follow our [maintenance policy](../../policy/maintenance.md). This process applies to features and APIs that are generally available, not beta or experimental. -This maintenance policy is in place to allow our customers to prepare for disruptive changes by establishing a clear and predictable pattern that is widely used in the software industry. For many of our customers, GitLab is a business-critical application and surprising changes can cause damages and erode trust. +This maintenance policy is in place to allow our customers to prepare for disruptive changes by establishing a clear and predictable pattern that is widely used in the software industry. For many of our customers, GitLab is a business-critical application and surprising changes can cause damages and erode trust. Introducing breaking changes in minor releases is against policy because it can disrupt our customers and introduces an element of randomness that requires customers to check for breaking changes every minor release to ensure that their business is not impacted. This does not align with our goal [to make it as easy as possible for customers to do business with GitLab](https://about.gitlab.com/company/yearlies/#fy24-yearlies) and is strongly discouraged. diff --git a/doc/development/documentation/contribute.md b/doc/development/documentation/contribute.md index b6573665f8d..2b166266ddb 100644 --- a/doc/development/documentation/contribute.md +++ b/doc/development/documentation/contribute.md @@ -1,81 +1,11 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2023-10-27' --- -# Contribute to the GitLab documentation +This document was moved to [another location](index.md). -Everyone is welcome to update the GitLab documentation. - -## Work without an issue - -You don't need an issue to update the documentation. - -On [https://docs.gitlab.com](https://docs.gitlab.com), at the bottom of any page, -you can select **View page source** or **Edit in Web IDE** and [get started with a merge request](#open-your-merge-request). - -You can alternately: - -- Choose a page [in the `/doc` directory](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc) - and edit it from there. -- Try installing and running the [Vale linting tool](testing.md#vale) - and fixing the resulting issues. - -When you're developing code, the workflow for updating docs is slightly different. -For details, see the [merge request workflow](../contributing/merge_request_workflow.md). - -## Search available issues - -If you're looking for an open issue, you can -[review the list of documentation issues curated specifically for new contributors](https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=created_date&state=opened&label_name%5B%5D=documentation&label_name%5B%5D=docs-only&label_name%5B%5D=Seeking%20community%20contributions&first_page_size=20). - -When you find an issue you'd like to work on: - -- If the issue is already assigned to someone, pick a different one. -- If the issue is unassigned, add a comment and ask to work on the issue. For a Hackathon, use `@docs-hackathon`. Otherwise, use `@gl-docsteam`. For example: - - ```plaintext - @docs-hackathon I would like to work on this issue - ``` - -- Do not ask for more than three issues at a time. - -## Open your merge request - -When you are ready to update the documentation: - -1. Go to the [GitLab repository](https://gitlab.com/gitlab-org/gitlab). -1. In the upper-right corner, select **Fork**. Forking makes a copy of the repository on GitLab.com. -1. In your fork, find the documentation page in the `\doc` directory. -1. If you know Git, make your changes and open a merge request. - If not, follow these steps: - 1. In the upper right, select **Edit > Edit single file**. - 1. In the **Commit message** text box, enter a commit message. - Use 3-5 words, start with a capital letter, and do not end with a period. - 1. Select **Commit changes**. - 1. On the left sidebar, select **Code > Merge requests**. - 1. Select **New merge request**. - 1. For the source branch, select your fork and branch. If you did not create a branch, select `master`. - For the target branch, select the [GitLab repository](https://gitlab.com/gitlab-org/gitlab) `master` branch. - 1. Select **Compare branches and continue**. A new merge request opens. - 1. Select the **Documentation** template. In the description, write a brief summary of the changes and link to the related issue, if there is one. - 1. Select **Create merge request**. - -## Ask for help - -Ask for help from the Technical Writing team if you: - -- Need help to choose the correct place for documentation. -- Want to discuss a documentation idea or outline. -- Want to request any other help. - -To identify someone who can help you: - -1. Locate the Technical Writer for the relevant - [DevOps stage group](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). -1. Either: - - If urgent help is required, directly assign the Technical Writer in the issue or in the merge request. - - If non-urgent help is required, ping the Technical Writer in the issue or merge request. - -If you are a member of the GitLab Slack workspace, you can request help in the `#docs` channel. +<!-- This redirect file can be deleted after <2023-10-27>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/documentation/graphql_styleguide.md b/doc/development/documentation/graphql_styleguide.md index 1bd8b37ff5f..79fcfc7e3f0 100644 --- a/doc/development/documentation/graphql_styleguide.md +++ b/doc/development/documentation/graphql_styleguide.md @@ -9,7 +9,7 @@ description: "Writing styles, markup, formatting, and other standards for GraphQ # Creating a GraphQL example page GraphQL APIs are different from [RESTful APIs](restful_api_styleguide.md). Reference -information is generated in our [GraphQL reference](../../api/graphql/reference/index.md). +information is generated in our [GraphQL API resources](../../api/graphql/reference/index.md) page. However, it's helpful to include examples on how to use GraphQL for different *use cases*, with samples that readers can use directly in the diff --git a/doc/development/documentation/help.md b/doc/development/documentation/help.md new file mode 100644 index 00000000000..fb730aca6f0 --- /dev/null +++ b/doc/development/documentation/help.md @@ -0,0 +1,125 @@ +--- +stage: none +group: Documentation Guidelines +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# GitLab /help + +Every GitLab instance includes documentation at `/help` (`https://gitlab.example.com/help`) +that matches the version of the instance. For example, <https://gitlab.com/help>. + +The documentation available online at <https://docs.gitlab.com> is deployed every +hour from the default branch of GitLab, Omnibus, Runner, Charts, and Operator. +After a merge request that updates documentation is merged, it is available online +in an hour or less. + +However, it's only available at `/help` on self-managed instances in the next released +version. The date an update is merged can impact which self-managed release the update +is present in. + +For example: + +1. A merge request in `gitlab` updates documentation. It has a milestone of 14.4, + with an expected release date of 2021-10-22. +1. It is merged on 2021-10-19 and available online the same day at <https://docs.gitlab.com>. +1. GitLab 14.4 is released on 2021-10-22, based on the `gitlab` codebase from 2021-10-18 + (one day *before* the update was merged). +1. The change shows up in the 14.5 self-managed release, due to missing the release cutoff + for 14.4. + +The exact cutoff date for each release is flexible, and can be sooner or later +than expected due to holidays, weekends or other events. In general, MRs merged +by the 17th should be present in the release on the 22nd, though it is not guaranteed. +If it is important that a documentation update is present in that month's release, +merge it as early as possible. + +## Linking to `/help` + +When you're building a new feature, you may need to link to the documentation +from the GitLab application. This is usually done in files inside the +`app/views/` directory, with the help of the `help_page_path` helper method. + +The `help_page_path` contains the path to the document you want to link to, +with the following conventions: + +- It's relative to the `doc/` directory in the GitLab repository. +- It omits the `.md` extension. +- It doesn't end with a forward slash (`/`). + +The help text follows the [Pajamas guidelines](https://design.gitlab.com/usability/contextual-help#formatting-help-content). + +### Linking to `/help` in HAML + +Use the following special cases depending on the context, ensuring all link text +is inside `_()` so it can be translated: + +- Linking to a doc page. In its most basic form, the HAML code to generate a + link to the `/help` page is: + + ```haml + = link_to _('Learn more.'), help_page_path('user/permissions'), target: '_blank', rel: 'noopener noreferrer' + ``` + +- Linking to an anchor link. Use `anchor` as part of the `help_page_path` + method: + + ```haml + = link_to _('Learn more.'), help_page_path('user/permissions', anchor: 'anchor-link'), target: '_blank', rel: 'noopener noreferrer' + ``` + +- Using links inline of some text. First, define the link, and then use it. In + this example, `link_start` is the name of the variable that contains the + link: + + ```haml + - link = link_to('', help_page_path('user/permissions'), target: '_blank', rel: 'noopener noreferrer') + %p= safe_format(_("This is a text describing the option/feature in a sentence. %{link_start}Learn more.%{link_end}"), tag_pair(link, :link_start, :link_end)) + ``` + +- Using a button link. Useful in places where text would be out of context with + the rest of the page layout: + + ```haml + = link_to _('Learn more.'), help_page_path('user/permissions'), class: 'btn btn-info', target: '_blank', rel: 'noopener noreferrer' + ``` + +### Linking to `/help` in JavaScript + +To link to the documentation from a JavaScript or a Vue component, use the `helpPagePath` function from [`help_page_helper.js`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/helpers/help_page_helper.js): + +```javascript +import { helpPagePath } from '~/helpers/help_page_helper'; + +helpPagePath('user/permissions', { anchor: 'anchor-link' }) +// evaluates to '/help/user/permissions#anchor-link' for GitLab.com +``` + +This is preferred over static paths, as the helper also works on instances installed under a [relative URL](../../install/relative_url.md). + +### Linking to `/help` in Ruby + +To link to the documentation from within Ruby code, use the following code block as a guide, ensuring all link text is inside `_()` so it can +be translated: + +```ruby +docs_link = link_to _('Learn more.'), help_page_url('user/permissions', anchor: 'anchor-link'), target: '_blank', rel: 'noopener noreferrer' +safe_format(_('This is a text describing the option/feature in a sentence. %{docs_link}'), docs_link: docs_link) +``` + +In cases where you need to generate a link from outside of views/helpers, where the `link_to` and `help_page_url` methods are not available, use the following code block +as a guide where the methods are fully qualified: + +```ruby +docs_link = ActionController::Base.helpers.link_to _('Learn more.'), Rails.application.routes.url_helpers.help_page_url('user/permissions', anchor: 'anchor-link'), target: '_blank', rel: 'noopener noreferrer' +safe_format(_('This is a text describing the option/feature in a sentence. %{docs_link}'), docs_link: docs_link) +``` + +Do not use `include ActionView::Helpers::UrlHelper` just to make the `link_to` method available as you might see in some existing code. Read more in +[issue 340567](https://gitlab.com/gitlab-org/gitlab/-/issues/340567). + +## `/help` tests + +Several [RSpec tests](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/features/help_pages_spec.rb) +are run to ensure GitLab documentation renders and works correctly. In particular, that [main docs landing page](../../index.md) works correctly from `/help`. +For example, [GitLab.com's `/help`](https://gitlab.com/help). diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 3f26a5a7f4e..c06a396b378 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -2,417 +2,97 @@ stage: none group: Documentation Guidelines info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -description: Learn how to contribute to GitLab Documentation. --- # Contribute to the GitLab documentation -The GitLab documentation is [intended as the single source of truth (SSOT)](https://about.gitlab.com/handbook/documentation/) for information about how to configure, use, and troubleshoot GitLab. The documentation contains use cases and usage instructions for every GitLab feature, organized by product area and subject. This includes topics and workflows that span multiple GitLab features and the use of GitLab with other applications. +The GitLab documentation is the single source of truth (SSOT) +for information about how to configure, use, and troubleshoot GitLab. +Everyone is welcome to contribute to the GitLab documentation. -In addition to this page, the following resources can help you craft and contribute to documentation: +## Work without an issue -- [Style Guide](styleguide/index.md) - What belongs in the docs, language guidelines, Markdown standards to follow, links, and more. -- [Topic types](topic_types/index.md) - Learn about the different types of topics. -- [Documentation process](workflow.md). -- [Markdown Guide](../../user/markdown.md) - A reference for all Markdown syntax supported by GitLab. -- [Site architecture](site_architecture/index.md) - How <https://docs.gitlab.com> is built. -- [Documentation for feature flags](feature_flags.md) - How to write and update documentation for GitLab features deployed behind feature flags. +You don't need an issue to update the documentation. -## Source files and rendered web locations +On <https://docs.gitlab.com>, at the bottom of any page, +you can select **View page source** or **Edit in Web IDE** and [get started with a merge request](#open-your-merge-request). -Documentation for GitLab, GitLab Runner, GitLab Operator, Omnibus GitLab, and Charts is published to <https://docs.gitlab.com>. Documentation for GitLab is also published within the application at `/help` on the domain of the GitLab instance. -At `/help`, only help for your current edition and version is included. Help for other versions is available at <https://docs.gitlab.com/archives/>. +You can alternately: -The source of the documentation exists within the codebase of each GitLab application in the following repository locations: +- Choose a page [in the `/doc` directory](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc) + and edit it from there. +- Try installing and running the [Vale linting tool](testing.md#vale) + and fixing the resulting issues. -| Project | Path | -| --- | --- | -| [GitLab](https://gitlab.com/gitlab-org/gitlab/) | [`/doc`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc) | -| [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/) | [`/docs`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/main/docs) | -| [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/) | [`/doc`](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/doc) | -| [Charts](https://gitlab.com/gitlab-org/charts/gitlab) | [`/doc`](https://gitlab.com/gitlab-org/charts/gitlab/tree/master/doc) | -| [GitLab Operator](https://gitlab.com/gitlab-org/cloud-native/gitlab-operator) | [`/doc`](https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/tree/master/doc) | +When you're developing code, the workflow for updating docs is slightly different. +For details, see the [merge request workflow](../contributing/merge_request_workflow.md). -Documentation issues and merge requests are part of their respective repositories and all have the label `Documentation`. +## Search available issues -### Branch naming +If you're looking for an open issue, you can +[review the list of documentation issues curated specifically for new contributors](https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=created_date&state=opened&label_name%5B%5D=documentation&label_name%5B%5D=docs-only&label_name%5B%5D=Seeking%20community%20contributions&first_page_size=20). -The [CI pipeline for the main GitLab project](../pipelines/index.md) is configured to automatically -run only the jobs that match the type of contribution. If your contribution contains -**only** documentation changes, then only documentation-related jobs run, and -the pipeline completes much faster than a code contribution. +When you find an issue you'd like to work on: -If you are submitting documentation-only changes to Omnibus, Charts, or Operator, -the fast pipeline is not determined automatically. Instead, create branches for -docs-only merge requests using the following guide: - -| Branch name | Valid example | -|:----------------------|:-----------------------------| -| Starting with `docs/` | `docs/update-api-issues` | -| Starting with `docs-` | `docs-update-api-issues` | -| Ending in `-docs` | `123-update-api-issues-docs` | - -## Contributing to docs - -[Contributions to GitLab docs](workflow.md) are welcome from the entire GitLab community. - -To ensure that the GitLab docs are current, there are special processes and responsibilities for all [feature changes](workflow.md), that is development work that impacts the appearance, usage, or administration of a feature. - -However, anyone can contribute [documentation improvements](workflow.md) that are not associated with a feature change. For example, adding a new document on how to accomplish a use case that's already possible with GitLab or with third-party tools and GitLab. - -## Markdown and styles - -[GitLab docs](https://gitlab.com/gitlab-org/gitlab-docs) uses [GitLab Kramdown](https://gitlab.com/gitlab-org/gitlab_kramdown) -as its Markdown rendering engine. See the [GitLab Markdown Guide](https://about.gitlab.com/handbook/markdown-guide/) for a complete Kramdown reference. - -Adhere to the [Documentation Style Guide](styleguide/index.md). If a style standard is missing, you are welcome to suggest one via a merge request. - -## Folder structure and files - -See the [Folder structure](site_architecture/folder_structure.md) page. - -## Metadata - -To provide additional directives or useful information, we add metadata in YAML -format to the beginning of each product documentation page (YAML front matter). -All values are treated as strings and are only used for the -[docs website](site_architecture/index.md). - -### Stage and group metadata - -Each page should ideally have metadata related to the stage and group it -belongs to, as well as an information block as described below: - -- `stage`: The [Stage](https://about.gitlab.com/handbook/product/categories/#devops-stages) - to which the majority of the page's content belongs. -- `group`: The [Group](https://about.gitlab.com/company/team/structure/#product-groups) - to which the majority of the page's content belongs. -- `info`: The following line, which provides direction to contributors regarding - how to contact the Technical Writer associated with the page's stage and - group: +- If the issue is already assigned to someone, pick a different one. +- If the issue is unassigned, add a comment and ask to work on the issue. For a Hackathon, use `@docs-hackathon`. Otherwise, use `@gl-docsteam`. For example: ```plaintext - To determine the technical writer assigned to the Stage/Group - associated with this page, see - https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments + @docs-hackathon I would like to work on this issue ``` -For example: - -```yaml ---- -stage: Example Stage -group: Example Group -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments ---- -``` - -### Redirection metadata - -The following metadata should be added when a page is moved to another location: - -- `redirect_to`: The relative path and filename (with an `.md` extension) of the - location to which visitors should be redirected for a moved page. - [Learn more](redirects.md). - -### Additional page metadata +- Do not ask for more than three issues at a time. -Each page can have additional, optional metadata (set in the -[default.html](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/fc3577921343173d589dfa43d837b4307e4e620f/layouts/default.html#L30-52) -Nanoc layout), which is displayed at the top of the page if defined. +## Open your merge request -### Deprecated metadata +When you are ready to update the documentation: -The `type` metadata parameter is deprecated but still exists in documentation -pages. You can safely remove the `type` metadata parameter and its values. +1. Go to the [GitLab repository](https://gitlab.com/gitlab-org/gitlab). +1. In the upper-right corner, select **Fork**. Forking makes a copy of the repository on GitLab.com. +1. In your fork, find the documentation page in the `\doc` directory. +1. If you know Git, make your changes and open a merge request. + If not, follow these steps: + 1. In the upper right, select **Edit > Edit single file**. + 1. Make your changes. + 1. When you're ready to submit your changes, in the **Commit message** text box, enter a commit message. + Use 3-5 words, start with a capital letter, and do not end with a period. + 1. Select **Commit changes**. + 1. On the left sidebar, select **Code > Merge requests**. + 1. Select **New merge request**. + 1. For the source branch, select your fork and branch. If you did not create a branch, select `master`. + For the target branch, select the [GitLab repository](https://gitlab.com/gitlab-org/gitlab) `master` branch. + 1. Select **Compare branches and continue**. A new merge request opens. + 1. Select the **Documentation** template. In the description, write a brief summary of the changes and link to the related issue, if there is one. + 1. Select **Create merge request**. -### Batch updates for TW metadata +## Ask for help -The [`CODEOWNERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/CODEOWNERS) -file contains a list of files and the associated technical writers. +Ask for help from the Technical Writing team if you: -When a merge request contains documentation, the information in the `CODEOWNERS` file determines: +- Need help to choose the correct place for documentation. +- Want to discuss a documentation idea or outline. +- Want to request any other help. -- The list of users in the **Approvers** section. -- The technical writer that the GitLab Bot pings for community contributions. +To identify someone who can help you: -You can use a Rake task to update the `CODEOWNERS` file. +1. Locate the Technical Writer for the relevant + [DevOps stage group](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). +1. Either: + - If urgent help is required, directly assign the Technical Writer in the issue or in the merge request. + - If non-urgent help is required, ping the Technical Writer in the issue or merge request. -#### Update the `CODEOWNERS` file +If you are a member of the GitLab Slack workspace, you can request help in the `#docs` channel. -When groups or [TW assignments](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) -change, you must update the `CODEOWNERS` file: +## Branch naming -1. Update the [stage and group metadata](#stage-and-group-metadata) for any affected doc pages, if necessary. If there are many changes, you can do this step in a separate MR. -1. Update the [`codeowners.rake`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/tw/codeowners.rake) file with the changes. -1. Go to the root of the `gitlab` repository. -1. Run the Rake task with this command: `bundle exec rake tw:codeowners` -1. Review the changes in the `CODEOWNERS` file. -1. Add and commit all your changes and push your branch up to `origin`. -1. Create a merge request and assign it to a technical writing manager for review. +The [CI/CD pipeline for the main GitLab project](../pipelines/index.md) is configured to +run shorter, faster pipelines on merge requests that contain only documentation changes. -When updating the `codeowners.rake` file: - -- To specify multiple writers for a single group, use a space between writer names: - - ```plaintext - CodeOwnerRule.new('Group Name', '@writer1 @writer2'), - ``` +If you submit documentation-only changes to Omnibus, Charts, or Operator, +to make the shorter pipeline run, you must follow these guidelines when naming your branch: -- For a group that does not have an assigned writer, include the group name in the file and comment out the line: - - ```plaintext - # CodeOwnerRule.new('Group Name', ''), - ``` - -## Move, rename, or delete a page - -See [redirects](redirects.md). - -## Merge requests for GitLab documentation - -Before getting started, make sure you read the introductory section -"[contributing to docs](#contributing-to-docs)" above and the -[documentation workflow](workflow.md). - -- Use the current [merge request description template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/merge_request_templates/Documentation.md) -- Label the MR `Documentation` (can only be done by people with `developer` access, for example, GitLab team members) -- Assign the correct milestone per note below (can only be done by people with `developer` access, for example, GitLab team members) - -Documentation is merged if it is an improvement on existing content, -represents a good-faith effort to follow the template and style standards, -and is believed to be accurate. - -Further needs for what would make the doc even better should be immediately addressed -in a follow-up merge request or issue. - -If the release version you want to add the documentation to has already been -released, follow the [patch release runbook](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/patch/engineers.md) -to get it merged into the current version. - -## GitLab `/help` - -Every GitLab instance includes documentation at `/help` (`https://gitlab.example.com/help`) -that matches the version of the instance. For example, <https://gitlab.com/help>. - -The documentation available online at <https://docs.gitlab.com> is deployed every -hour from the default branch of [GitLab, Omnibus, Runner, and Charts](#source-files-and-rendered-web-locations). -After a merge request that updates documentation is merged, it is available online -in an hour or less. - -However, it's only available at `/help` on self-managed instances in the next released -version. The date an update is merged can impact which self-managed release the update -is present in. - -For example: - -1. A merge request in `gitlab` updates documentation. It has a milestone of 14.4, - with an expected release date of 2021-10-22. -1. It is merged on 2021-10-19 and available online the same day at <https://docs.gitlab.com>. -1. GitLab 14.4 is released on 2021-10-22, based on the `gitlab` codebase from 2021-10-18 - (one day *before* the update was merged). -1. The change shows up in the 14.5 self-managed release, due to missing the release cutoff - for 14.4. - -The exact cutoff date for each release is flexible, and can be sooner or later -than expected due to holidays, weekends or other events. In general, MRs merged -by the 17th should be present in the release on the 22nd, though it is not guaranteed. -If it is important that a documentation update is present in that month's release, -merge it as early as possible. - -### Linking to `/help` - -When you're building a new feature, you may need to link to the documentation -from the GitLab application. This is usually done in files inside the -`app/views/` directory, with the help of the `help_page_path` helper method. - -The `help_page_path` contains the path to the document you want to link to, -with the following conventions: - -- It's relative to the `doc/` directory in the GitLab repository. -- It omits the `.md` extension. -- It doesn't end with a forward slash (`/`). - -The help text follows the [Pajamas guidelines](https://design.gitlab.com/usability/contextual-help#formatting-help-content). - -#### Linking to `/help` in HAML - -Use the following special cases depending on the context, ensuring all link text -is inside `_()` so it can be translated: - -- Linking to a doc page. In its most basic form, the HAML code to generate a - link to the `/help` page is: - - ```haml - = link_to _('Learn more.'), help_page_path('user/permissions'), target: '_blank', rel: 'noopener noreferrer' - ``` - -- Linking to an anchor link. Use `anchor` as part of the `help_page_path` - method: - - ```haml - = link_to _('Learn more.'), help_page_path('user/permissions', anchor: 'anchor-link'), target: '_blank', rel: 'noopener noreferrer' - ``` - -- Using links inline of some text. First, define the link, and then use it. In - this example, `link_start` is the name of the variable that contains the - link: - - ```haml - - link = link_to('', help_page_path('user/permissions'), target: '_blank', rel: 'noopener noreferrer') - %p= safe_format(_("This is a text describing the option/feature in a sentence. %{link_start}Learn more.%{link_end}"), tag_pair(link, :link_start, :link_end)) - ``` - -- Using a button link. Useful in places where text would be out of context with - the rest of the page layout: - - ```haml - = link_to _('Learn more.'), help_page_path('user/permissions'), class: 'btn btn-info', target: '_blank', rel: 'noopener noreferrer' - ``` - -#### Linking to `/help` in JavaScript - -To link to the documentation from a JavaScript or a Vue component, use the `helpPagePath` function from [`help_page_helper.js`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/helpers/help_page_helper.js): - -```javascript -import { helpPagePath } from '~/helpers/help_page_helper'; - -helpPagePath('user/permissions', { anchor: 'anchor-link' }) -// evaluates to '/help/user/permissions#anchor-link' for GitLab.com -``` - -This is preferred over static paths, as the helper also works on instances installed under a [relative URL](../../install/relative_url.md). - -#### Linking to `/help` in Ruby - -To link to the documentation from within Ruby code, use the following code block as a guide, ensuring all link text is inside `_()` so it can -be translated: - -```ruby -docs_link = link_to _('Learn more.'), help_page_url('user/permissions', anchor: 'anchor-link'), target: '_blank', rel: 'noopener noreferrer' -safe_format(_('This is a text describing the option/feature in a sentence. %{docs_link}'), docs_link: docs_link) -``` - -In cases where you need to generate a link from outside of views/helpers, where the `link_to` and `help_page_url` methods are not available, use the following code block -as a guide where the methods are fully qualified: - -```ruby -docs_link = ActionController::Base.helpers.link_to _('Learn more.'), Rails.application.routes.url_helpers.help_page_url('user/permissions', anchor: 'anchor-link'), target: '_blank', rel: 'noopener noreferrer' -safe_format(_('This is a text describing the option/feature in a sentence. %{docs_link}'), docs_link: docs_link) -``` - -Do not use `include ActionView::Helpers::UrlHelper` just to make the `link_to` method available as you might see in some existing code. Read more in -[issue 340567](https://gitlab.com/gitlab-org/gitlab/-/issues/340567). - -### GitLab `/help` tests - -Several [RSpec tests](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/features/help_pages_spec.rb) -are run to ensure GitLab documentation renders and works correctly. In particular, that [main docs landing page](../../index.md) works correctly from `/help`. -For example, [GitLab.com's `/help`](https://gitlab.com/help). - -## Docs site architecture - -For information on how we build and deploy <https://docs.gitlab.com>, see [Docs site architecture](site_architecture/index.md). - -### Global navigation - -See the [Global navigation](site_architecture/global_nav.md) doc for information -on how the left-side navigation menu is built and updated. - -## Previewing the changes live - -See how you can use review apps to [preview your changes live](review_apps.md). - -## Testing - -For more information about documentation testing, see the [Documentation testing](testing.md) -guide. - -## Danger Bot - -GitLab uses [Danger](https://github.com/danger/danger) for some elements in -code review. For docs changes in merge requests, whenever a change to files under `/doc` -is made, Danger Bot leaves a comment with further instructions about the documentation -process. This is configured in the `Dangerfile` in the GitLab repository under -[/danger/documentation/](https://gitlab.com/gitlab-org/gitlab/-/tree/master/danger/documentation). - -## Help and feedback section - -This section ([introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/319) in GitLab 11.4) -is displayed at the end of each document and can be omitted by adding a key into -the front matter: - -```yaml ---- -feedback: false ---- -``` - -The default is to leave it there. If you want to omit it from a document, you -must check with a technical writer before doing so. - -The click events in the feedback section are tracked with Google Tag Manager. -The conversions can be viewed on Google Analytics by navigating to -**Behavior > Events > Top events > docs**. - -## Automatic screenshot generator - -You can now set up an automatic screenshot generator to take and compress screenshots with the -help of a configuration file known as **screenshot generator**. - -### Use the tool - -To run the tool on an existing screenshot generator, take the following steps: - -1. Set up the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/gitlab_docs.md). -1. Navigate to the subdirectory with your cloned GitLab repository, typically `gdk/gitlab`. -1. Make sure that your GDK database is fully migrated: `bin/rake db:migrate RAILS_ENV=development`. -1. Install `pngquant`, see the tool website for more information: [`pngquant`](https://pngquant.org/) -1. Run `scripts/docs_screenshots.rb spec/docs_screenshots/<name_of_screenshot_generator>.rb <milestone-version>`. -1. Identify the location of the screenshots, based on the `gitlab/doc` location defined by the `it` parameter in your script. -1. Commit the newly created screenshots. - -### Extending the tool - -To add an additional **screenshot generator**, take the following steps: - -- Locate the `spec/docs_screenshots` directory. -- Add a new file with a `_docs.rb` extension. -- Be sure to include the following bits in the file: - -```ruby -require 'spec_helper' - -RSpec.describe '<What I am taking screenshots of>', :js do - include DocsScreenshotHelpers # Helper that enables the screenshots taking mechanism - - before do - page.driver.browser.manage.window.resize_to(1366, 1024) # length and width of the page - end -``` - -- In addition, every `it` block must include the path where the screenshot is saved - -```ruby - it 'user/packages/container_registry/img/project_image_repositories_list' -``` - -#### Full page screenshots - -To take a full page screenshot, `visit the page` and perform any expectation on real content (to have capybara wait till the page is ready and not take a white screenshot). - -#### Element screenshot - -To have the screenshot focuses few more steps are needed: - -- **find the area**: `screenshot_area = find('#js-registry-policies')` -- **scroll the area in focus**: `scroll_to screenshot_area` -- **wait for the content**: `expect(screenshot_area).to have_content 'Expiration interval'` -- **set the crop area**: `set_crop_data(screenshot_area, 20)` - -In particular, `set_crop_data` accepts as arguments: a `DOM` element and a -padding. The padding is added around the element, enlarging the screenshot area. - -#### Live example - -Please use `spec/docs_screenshots/container_registry_docs.rb` as a guide and as an example to create your own scripts. +| Branch name | Valid example | +|:----------------------|:-----------------------------| +| Starting with `docs/` | `docs/update-api-issues` | +| Starting with `docs-` | `docs-update-api-issues` | +| Ending in `-docs` | `123-update-api-issues-docs` | diff --git a/doc/development/documentation/site_architecture/folder_structure.md b/doc/development/documentation/site_architecture/folder_structure.md index 1c9fc1441c4..d4a3c856e0a 100644 --- a/doc/development/documentation/site_architecture/folder_structure.md +++ b/doc/development/documentation/site_architecture/folder_structure.md @@ -75,7 +75,7 @@ place for it. ## Avoid duplication Do not include the same information in multiple places. -[Link to a single source of truth instead.](../styleguide/index.md#link-instead-of-repeating-text) +Link to a single source of truth instead. For example, if you have code in a repository other than the [primary repositories](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/architecture.md), and documentation in the same repository, you can keep the documentation in that repository. diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index 849308f3086..822c7992222 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -14,6 +14,35 @@ static site generator. View the [`gitlab-docs` architecture page](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/architecture.md) for more information. +## Source files + +The documentation source files are in the same repositories as the product code. + +| Project | Path | +| --- | --- | +| [GitLab](https://gitlab.com/gitlab-org/gitlab/) | [`/doc`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc) | +| [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/) | [`/docs`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/main/docs) | +| [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/) | [`/doc`](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/doc) | +| [Charts](https://gitlab.com/gitlab-org/charts/gitlab) | [`/doc`](https://gitlab.com/gitlab-org/charts/gitlab/tree/master/doc) | +| [GitLab Operator](https://gitlab.com/gitlab-org/cloud-native/gitlab-operator) | [`/doc`](https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/tree/master/doc) | + +Documentation issues and merge requests are part of their respective repositories and all have the label `Documentation`. + +## Publication + +Documentation for GitLab, GitLab Runner, GitLab Operator, Omnibus GitLab, and Charts is published to <https://docs.gitlab.com>. + +The same documentation is included in the application. To view the in-product help, +go to the URL and add `/help` at the end. +Only help for your current edition and version is included. + +Help for other versions is available at <https://docs.gitlab.com/archives/>. + +## Updating older versions + +If you need to add or edit documentation for a GitLab version that has already been +released, follow the [patch release runbook](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/patch/engineers.md). + ## Documentation in other repositories If you have code and documentation in a repository other than the [primary repositories](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/architecture.md), @@ -41,7 +70,10 @@ Then you can use one of these approaches: The docs website supports versions and each month we add the latest one to the list. For more information, read about the [monthly release process](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/releases.md). -## Review Apps for documentation merge requests +## Danger Bot -If you are contributing to GitLab docs read how to -[create a Review App with each merge request](../index.md#previewing-the-changes-live). +GitLab uses [Danger](https://github.com/danger/danger) for some elements in +code review. For docs changes in merge requests, whenever a change to files under `/doc` +is made, Danger Bot leaves a comment with further instructions about the documentation +process. This is configured in the `Dangerfile` in the GitLab repository under +[/danger/documentation/](https://gitlab.com/gitlab-org/gitlab/-/tree/master/danger/documentation). diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index bd4ded8ca11..94bc6bba240 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -7,23 +7,12 @@ description: 'Writing styles, markup, formatting, and other standards for GitLab # Documentation Style Guide -This document defines the standards for GitLab documentation, including grammar, formatting, word use, and more. +This document defines the standards for GitLab documentation, including grammar, formatting, and more. +For guidelines on specific words, see [the word list](word_list.md). For style questions, mention `@tw-style` in an issue or merge request. If you have access to the GitLab Slack workspace, use the `#docs-processes` channel. -In addition to this page, the following resources can help you craft and contribute to documentation: - -- [Doc contribution guidelines](../index.md) -- [Recommended word list](word_list.md) -- [Doc style and consistency testing](../testing.md) -- [Guidelines for UI error messages](https://design.gitlab.com/content/voice-and-tone#clear-error-messages) -- [Documentation global navigation](../site_architecture/global_nav.md) -- [GitLab Handbook style guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines) -- [Microsoft Style Guide](https://learn.microsoft.com/en-us/style-guide/welcome/) -- [Google Developer Documentation Style Guide](https://developers.google.com/style) -- [Recent updates to this guide](https://gitlab.com/dashboard/merge_requests?scope=all&state=merged&label_name[]=tw-style¬[label_name][]=docs%3A%3Afix) - ## The GitLab voice The GitLab brand guidelines define the @@ -34,108 +23,43 @@ direct, and precise. The goal is to provide information that's easy to search an The voice in the documentation should be conversational but brief, friendly but succinct. -## Documentation is the single source of truth (SSOT) +## Documentation is the single source of truth (SSoT) -The GitLab documentation is the SSOT for all -information related to GitLab implementation, usage, and troubleshooting. It evolves -continuously, in keeping with new products and features, and with improvements -for clarity, accuracy, and completeness. +The GitLab documentation is the SSoT for all information related to implementation, +use, and troubleshooting. The documentation evolves continuously. It is updated with +new products and features, and with improvements for clarity, accuracy, and completeness. This policy prevents information silos, making it easier to find information -about GitLab products. - -It also informs decisions about the kinds of content we include in our -documentation. - -### The documentation includes all information - -Include problem-solving actions that may address rare cases or be considered -risky, but provide proper context through fully detailed -warnings and caveats. This kind of content should be included as it could be -helpful to others and, when properly explained, its benefits outweigh the risks. -If you think you have found an exception to this rule, contact the -Technical Writing team. - -GitLab adds all troubleshooting information to the documentation, no matter how -unlikely a user is to encounter a situation. - -GitLab Support maintains their own -[troubleshooting content](../../../administration/troubleshooting/index.md) -in the GitLab documentation. - -### The documentation includes all media types +about GitLab products. It also informs decisions about the kinds of content that +is included in the documentation. -Include any media types/sources if the content is relevant to readers. You can -freely include or link presentations, diagrams, and videos. No matter who -it was originally composed for, if it is helpful to any of our audiences, we can -include it. +## Topic types -- If you use an image that has a separate source file (for example, a vector or - diagram format), link the image to the source file so that anyone can update or reuse it. -- Do not copy and paste content from other sources unless it is a limited - quotation with the source cited. Typically it is better to either rephrase - relevant information in your own words or link out to the other source. +GitLab uses [topic types](../topic_types/index.md) to organize the product documentation. -### Topic types +Topic types help users digest information more quickly. They also help address these issues: -In the software industry, it is a best practice to organize documentation in -different types. For example: - -- Concepts -- Tasks -- Reference -- Troubleshooting - -At GitLab, we have not traditionally used topic types. However, we are starting to -move in this direction, so we can address these issues: - -- **Content is hard to find.** Our docs are comprehensive and include a large amount of - useful information. Topic types create repeatable patterns that make our content easier +- **Content is hard to find.** The GitLab docs are comprehensive and include a large amount of + useful information. Topic types create repeatable patterns that make the content easier to scan and parse. -- **Content is often written from the contributor's point of view.** Our docs - are written by contributors. Topic types (tasks specifically) help put +- **Content is often written from the contributor's point of view.** The GitLab docs + are written by a variety of contributors. Topic types (tasks, specifically) help put information into a format that is geared toward helping others, rather than documenting how a feature was implemented. -GitLab uses these [topic types](../topic_types/index.md). - -### Link instead of repeating text +## Docs-first methodology -Rather than repeating information from another topic, link to the single source -of truth and explain why it is important. - -### Docs-first methodology - -We employ a documentation-first methodology. This method ensures the documentation -remains a complete and trusted resource, and makes communicating about the use -of GitLab more efficient. +The product documentation should be a complete and trusted resource. - If the answer to a question exists in documentation, share the link to the documentation instead of rephrasing the information. -- When you encounter new information not available in GitLab documentation (for - example, when working on a support case or testing a feature), your first step - should be to create a merge request (MR) to add this information to the - documentation. You can then share the MR to communicate this information. - -New information that would be useful toward the future usage or troubleshooting -of GitLab should not be written directly in a forum or other messaging system, -but added to a documentation MR and then referenced, as described above. +- When you encounter information that's not available in GitLab documentation, + create a merge request (MR) to add the information to the + documentation. Then share the MR to communicate the information. The more we reflexively add information to the documentation, the more the documentation helps others efficiently accomplish tasks and solve problems. -If you have questions when considering, authoring, or editing documentation, ask -the Technical Writing team. They're available on Slack in `#docs` or in GitLab by -mentioning [the writer for](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) -the applicable [DevOps stage or group](https://about.gitlab.com/handbook/product/categories/#devops-stages). -Otherwise, forge ahead with your best effort. It does not need to be perfect; -the team is happy to review and improve upon your content. Review the -[Documentation guidelines](index.md) before you begin your first documentation MR. - -Maintaining a knowledge base separate from the documentation would -be against the documentation-first methodology because the content would overlap with -the documentation. - ## Writing for localization The GitLab documentation is not localized, but we follow guidelines that @@ -155,13 +79,18 @@ also aid in consistency, which is important for localization. ## Markdown -All GitLab documentation is written using [Markdown](https://en.wikipedia.org/wiki/Markdown). +All GitLab documentation is written in [Markdown](https://en.wikipedia.org/wiki/Markdown). -The [documentation website](https://docs.gitlab.com) uses [GitLab Kramdown](https://gitlab.com/gitlab-org/gitlab_kramdown), +The [documentation website](https://docs.gitlab.com) uses [GitLab Kramdown](https://gitlab.com/gitlab-org/ruby/gems/gitlab_kramdown), a "flavored" Kramdown engine to render pages from Markdown to HTML. The use of Kramdown features is limited by our linters, so, use regular Markdown and follow the rules in the linked style guide. You can't use Kramdown-specific markup (for example, `{:.class}`). +For a complete Kramdown reference, see the +[GitLab Markdown Guide](https://about.gitlab.com/handbook/markdown-guide/). + +The Markdown format is [tested](../testing.md) by using markdownlint and Vale. + ### HTML in Markdown Hard-coded HTML is valid, although it's discouraged from being used. HTML is permitted if: @@ -193,43 +122,92 @@ Use backticks for: - Commands, parameters, and filenames. - Values. For example: "In the **Name** text box, type `test`." -### Markdown Rules +## Metadata + +Each documentation Markdown page contains YAML front matter. +All values in the metadata are treated as strings and are used for the +docs website only. + +### Stage and group metadata + +Each page should have metadata related to the stage and group it +belongs to, as well as an information block. For example: + +```yaml +--- +stage: Example Stage +group: Example Group +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- +``` + +To populate the metadata, include this information: + +- `stage`: The [Stage](https://about.gitlab.com/handbook/product/categories/#devops-stages) + that the majority of the page's content belongs to. +- `group`: The [Group](https://about.gitlab.com/company/team/structure/#product-groups) + that the majority of the page's content belongs to. +- `info`: How to find the Technical Writer associated with the page's stage and + group. + +### Additional metadata + +Each page can have additional, optional metadata (set in the +[default.html](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/fc3577921343173d589dfa43d837b4307e4e620f/layouts/default.html#L30-52) +Nanoc layout), which is displayed at the top of the page if defined. + +### Deprecated metadata -GitLab ensures that the Markdown used across all documentation is consistent, as -well as easy to review and maintain, by [testing documentation changes](../testing.md) -with [markdownlint](../testing.md#markdownlint). This lint test fails when any -document has an issue with Markdown formatting that may cause the page to render -incorrectly in GitLab. It also fails when a document has -non-standard Markdown (which may render correctly, but is not the current -standard for GitLab documentation). +The `type` metadata parameter is deprecated but still exists in documentation +pages. You can remove the `type` metadata parameter and its values. -#### Markdown rule `MD044/proper-names` (capitalization) +### Batch updates for TW metadata -A rule that could cause confusion is `MD044/proper-names`, as it might not be -immediately clear what caused markdownlint to fail, or how to correct the -failure. This rule checks a list of known words, listed in the `.markdownlint.yml` -file in each project, to verify proper use of capitalization and backticks. -Words in backticks are ignored by markdownlint. +The [`CODEOWNERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/CODEOWNERS) +file contains a list of files and the associated technical writers. -In general, product names should follow the exact capitalization of the official -names of the products, protocols, and so on. +When a merge request contains documentation, the information in the `CODEOWNERS` file determines: -Some examples fail if incorrect capitalization is used: +- The list of users in the **Approvers** section. +- The technical writer that the GitLab Bot pings for community contributions. -- MinIO (needs capital `IO`) -- NGINX (needs all capitals) -- runit (needs lowercase `r`) +You can use a Rake task to update the `CODEOWNERS` file. -Additionally, commands, parameters, values, filenames, and so on must be -included in backticks. For example: +#### Update the `CODEOWNERS` file -- "Change the `needs` keyword in your `.gitlab-ci.yml`..." - - `needs` is a parameter, and `.gitlab-ci.yml` is a file, so both need backticks. - Additionally, `.gitlab-ci.yml` without backticks fails markdownlint because it - does not have capital G or L. -- "Run `git clone` to clone a Git repository..." - - `git clone` is a command, so it must be lowercase, while Git is the product, - so it must have a capital G. +When groups or [TW assignments](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) +change, you must update the `CODEOWNERS` file: + +1. Update the [stage and group metadata](#stage-and-group-metadata) for any affected doc pages, if necessary. If there are many changes, you can do this step in a separate MR. +1. Update the [`codeowners.rake`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/tw/codeowners.rake) file with the changes. +1. Go to the root of the `gitlab` repository. +1. Run the Rake task with this command: `bundle exec rake tw:codeowners` +1. Review the changes in the `CODEOWNERS` file. +1. Add and commit all your changes and push your branch up to `origin`. +1. Create a merge request and assign it to a technical writing manager for review. + +When you update the `codeowners.rake` file: + +- To specify multiple writers for a single group, use a space between writer names. Files are assigned to both writers. + + ```ruby + CodeOwnerRule.new('Group Name', '@writer1 @writer2'), + ``` + + - To assign different writers within a group to docs in different directories, use the `path` parameter to specify a directory: + + ```ruby + CodeOwnerRule.new('Group Name', ->(path) { path.start_with?('/doc/user') ? '@writer1' : '@writer2' }), + ``` + + In this example, `writer1` is a code owner for files related to this group that are in `/doc/user`. + For everything else, `writer2` is made code owner. For an example, see [MR 127903](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127903). + +- For a group that does not have an assigned writer, include the group name in the file and comment out the line: + + ```ruby + # CodeOwnerRule.new('Group Name', ''), + ``` ## Language @@ -386,8 +364,6 @@ For numbers in text, spell out zero through nine and use numbers for 10 and grea ## Text - [Write in Markdown](#markdown). -- Splitting long lines (preferably up to 100 characters) can make it easier to - provide feedback on small chunks of text. - Insert an empty line for new paragraphs. - Insert an empty line between different markups (for example, after every paragraph, header, list, and so on). Example: @@ -401,6 +377,14 @@ For numbers in text, spell out zero through nine and use numbers for 10 and grea - List item 2 ``` +### Line length + +To make the source content easy to read, and to more easily compare diffs, +follow these best practices when possible. + +- Split long lines at approximately 100 characters. +- Start each new sentence on a new line. + ### Comments To embed comments within Markdown, use standard HTML comments that are not rendered @@ -453,7 +437,7 @@ Do not use these punctuation characters: ### Placeholder text -You might want to provide a command or configuration that +In a code block, you might want to provide a command or configuration that uses specific values. In these cases, use [`<` and `>`](https://en.wikipedia.org/wiki/Usage_message#Pattern) @@ -465,6 +449,13 @@ For example: cp <your_source_directory> <your_destination_directory> ``` +If the placeholder is not in a code block, use [`<` and `>`] and wrap the placeholder +in a single backtick. For example: + +```plaintext +Select **Grant admin consent for `<application_name>`**. +``` + ### Keyboard commands Use the HTML `<kbd>` tag when referring to keystroke presses. For example: @@ -477,17 +468,23 @@ When the docs are generated, the output is: To stop the command, press <kbd>Control</kbd>+<kbd>C</kbd>. +### Buttons in the UI + +For elements with a visible label, use the label in bold with matching case. + +For example: `Select **Cancel**.` + ### Text entered in the UI If you want the user to type something in the UI, use backticks. For example: ```plaintext -In the **Commit message** box, type `This is my merge request`. +In the **Commit message** text box, type `This is my merge request`. ``` Backticks are more precise than quotes. For example, in this string: -- In the **Commit message** box, type "This is my merge request." +- In the **Commit message** text box, type "This is my merge request." It's not clear whether the user should include the period in the string. @@ -652,10 +649,10 @@ To help keep tables easier to maintain, you can: - Add additional spaces to make the column widths consistent. For example: ```markdown - | App name | Description | Requirements | - |----------|---------------------|----------------| - | App 1 | Description text 1. | Requirements 1 | - | App 2 | Description text 2. | None | + | App name | Description | Requirements | + |----------|---------------------|--------------| + | App 1 | Description text 1. | A, B, and C. | + | App 2 | Description text 2. | None | ``` - Skip the additional spaces in the rightmost column for tables that are very wide. @@ -669,11 +666,32 @@ To help keep tables easier to maintain, you can: | Setting 3 | `0` | Another short description. | ``` -Consider installing a plugin or extension in your editor for formatting tables: +### Editor extensions for table formatting + +To ensure consistent table formatting across all Markdown files, consider formatting your tables +with the VS Code [Markdown Table Formatter](https://github.com/fcrespo82/vscode-markdown-table-formatter). +To configure this extension to follow the guidelines above, enable the **Follow header row length** setting. +To enable the setting: + +- In the UI: + + 1. In the VS Code menu, go to **Code > Settings > Settings**. + 1. Search for `Limit Last Column Length`. + 1. In the **Limit Last Column Length** dropdown list, select **Follow header row length**. -- [Markdown Table Prettifier](https://marketplace.visualstudio.com/items?itemName=darkriszty.markdown-table-prettify) for Visual Studio Code -- [Markdown Table Formatter](https://packagecontrol.io/packages/Markdown%20Table%20Formatter) for Sublime Text -- [Markdown Table Formatter](https://atom.io/packages/markdown-table-formatter) for Atom +- In your VS Code `settings.json`, add a new line with: + + ```json + { + "markdown-table-formatter.limitLastColumnLength": "Follow header row length" + } + ``` + +To format a table with this extension, select the entire table, right-click the selection, +and select **Format selection**. + +Alternatively, if you use Sublime Text you can try the [Markdown Table Formatter](https://packagecontrol.io/packages/Markdown%20Table%20Formatter) +plugin, but it does not have a **Follow header row length** setting. ### Updates to existing tables @@ -706,31 +724,44 @@ Instead, follow the [API topic template](../restful_api_styleguide.md#api-topic- ### Footnotes To indicate a footnote, use the HTML tag `<sup>` with a number. -Put the tag at the end of the sentence or term. - -For the footnotes below the table, use a bold number followed by a sentence. - -For example: +Put the tag at the end of the sentence or term. For example: ```markdown | App name | Description | |:---------|:-------------------------------| | App A | Description text. <sup>1</sup> | | App B | Description text. <sup>2</sup> | +``` + +For the footnotes below the table, use the HTML tags `<small>`, `<ol>` and `<li>`. +For example: -1. This is the footnote. -1. This is the other footnote. +```html +<html> +<small>Footnotes + <ol> + <li>This is the footnote.</li> + <li>This is the other footnote.</li> + </ol> +</small> +</html> ``` -This text renders this output: +This text renders as this output: | App name | Description | |:---------|:-------------------------------| | App A | Description text. <sup>1</sup> | | App B | Description text. <sup>2</sup> | -1. This is the footnote. -1. This is the other footnote. +<html> +<small>Footnotes + <ol> + <li>This is the footnote.</li> + <li>This is the other footnote.</li> + </ol> +</small> +</html> ## Quotes @@ -785,7 +816,7 @@ but do not use it. Link to the page instead. If a topic title has a [product tier badge](#product-tier-badges), do not include it in the anchor link. For example, for the topic -`## This is an example **(FREE)**`, use the anchor `#this-is-an-example`. +`## This is an example **(FREE ALL)**`, use the anchor `#this-is-an-example`. With Kramdown, you can add a custom ID to an HTML element, but these IDs don't work in `/help`, so you should not use them. @@ -866,11 +897,12 @@ Sometimes they are more precise and will be maintained more actively. For each external link you add, weigh the customer benefit with the maintenance difficulties. -### Links that require permissions +### Confidential or restricted access links Don't link directly to: - [Confidential issues](../../../user/project/issues/confidential_issues.md). +- Internal handbook pages. - Project features that require [special permissions](../../../user/permissions.md) to view. @@ -881,8 +913,8 @@ These links fail for: If you must use one of these links: -- If the link is to a confidential issue, mention that the issue is visible only to GitLab team members, as in the first example. -- If the link requires a specific role or permissions, mention that information, as in the second example. +- If the link is to a confidential issue or internal handbook page, mention that the issue or page is visible only to GitLab team members. +- If the link requires a specific role or permissions, mention that information. - Put the link in backticks so that it does not cause link checkers to fail. Examples: @@ -892,6 +924,9 @@ GitLab team members can view more information in this confidential issue: `https://gitlab.com/gitlab-org/gitlab/-/issues/<issue_number>` ``` +GitLab team members can view more information in this internal handbook page: +`https://internal.gitlab.com/handbook/<link>` + ```markdown Users with the Maintainer role for the project can use the pipeline editor: `https://gitlab.com/gitlab-org/gitlab/-/ci/editor` @@ -1201,6 +1236,61 @@ include a visual representation to help readers understand it, you can: an area of the screen. - Create a short video of the interaction and link to it. +### Automatic screenshot generator + +You can use an automatic screenshot generator to take and compress screenshots. + +1. Set up the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/gitlab_docs.md). +1. Navigate to the subdirectory with your cloned GitLab repository, typically `gdk/gitlab`. +1. Make sure that your GDK database is fully migrated: `bin/rake db:migrate RAILS_ENV=development`. +1. Install `pngquant`, see the tool website for more information: [`pngquant`](https://pngquant.org/) +1. Run `scripts/docs_screenshots.rb spec/docs_screenshots/<name_of_screenshot_generator>.rb <milestone-version>`. +1. Identify the location of the screenshots, based on the `gitlab/doc` location defined by the `it` parameter in your script. +1. Commit the newly created screenshots. + +#### Extending the tool + +To add an additional **screenshot generator**, complete the following steps: + +1. Locate the `spec/docs_screenshots` directory. +1. Add a new file with a `_docs.rb` extension. +1. Be sure to include the following information in the file: + + ```ruby + require 'spec_helper' + + RSpec.describe '<What I am taking screenshots of>', :js do + include DocsScreenshotHelpers # Helper that enables the screenshots taking mechanism + + before do + page.driver.browser.manage.window.resize_to(1366, 1024) # length and width of the page + end + ``` + +1. In addition, every `it` block must include the path where the screenshot is saved: + + ```ruby + it 'user/packages/container_registry/img/project_image_repositories_list' + ``` + +##### Full page screenshots + +To take a full page screenshot, `visit the page` and perform any expectation on real content (to have capybara wait till the page is ready and not take a white screenshot). + +##### Element screenshot + +To have the screenshot focuses few more steps are needed: + +- **find the area**: `screenshot_area = find('#js-registry-policies')` +- **scroll the area in focus**: `scroll_to screenshot_area` +- **wait for the content**: `expect(screenshot_area).to have_content 'Expiration interval'` +- **set the crop area**: `set_crop_data(screenshot_area, 20)` + +In particular, `set_crop_data` accepts as arguments: a `DOM` element and a +padding. The padding is added around the element, enlarging the screenshot area. + +Use `spec/docs_screenshots/container_registry_docs.rb` as a guide and as an example to create your own scripts. + ## Emoji Don't use the Markdown emoji format, for example `:smile:`, for any purpose. Use @@ -1522,19 +1612,11 @@ Until we implement automated testing for broken links to tabs ([Issue 1355](http See [Pajamas](https://design.gitlab.com/components/tabs/#guidelines) for more details on tabs. -## Terms - -To maintain consistency through GitLab documentation, use these styles and terms. +## Plagiarism -### Describe UI elements - -Follow these styles when you're describing user interface elements in an -application: - -- For elements with a visible label, use that label in bold with matching case. - For example, `Select **Cancel**`. -- For elements with a tooltip or hover label, use that label in bold with - matching case. For example, `Select **Add status emoji**`. +Do not copy and paste content from other sources unless it is a limited +quotation with the source cited. Typically it is better to rephrase +relevant information in your own words or link out to the other source. ## Products and features @@ -1548,8 +1630,7 @@ When names change, it is more complicated to search or grep text that has line b ### Product tier badges -Tier badges are displayed as orange text next to a topic title. These badges link to the GitLab -pricing page. +Tier badges provide information about a feature and are displayed next to the topic title. You should assign a tier badge: @@ -1558,37 +1639,59 @@ You should assign a tier badge: The H1 tier badge should be the badge that applies to the lowest tier for the features on the page. -Some pages won't have a tier badge, because no obvious tier badge applies. For example: +#### Available product tier badges -- Tutorials. -- Pages that compare features from different tiers. +Tier badges must include two components, in this order: a subscription tier and an offering. +These components are surrounded by bold and parentheses, for example `**(ULTIMATE SAAS)**`. + +Subscription tiers: + +- `FREE` - Applies to all tiers. +- `PREMIUM` - Applies to Premium and Ultimate tiers. +- `ULTIMATE` - Applies to Ultimate tier only. + +Offerings: + +- `SELF` +- `SAAS` +- `ALL` - Applies to both self-managed and SaaS. + +You can also add a third component for the feature's status: + +- `EXPERIMENT` +- `BETA` #### Add a tier badge -To add a tier badge to a topic title, add the relevant tier badge -after the title text. For example: +To add a tier badge to a topic title, add the two relevant components +after the title text. You must include the subscription tier first, and then the offering. +For example: + +```markdown +# Topic title **(FREE ALL)** +``` + +Optionally, you can add the feature status as the last part of the badge: ```markdown -# Topic title **(FREE)** +# Topic title **(FREE ALL EXPERIMENT)** ``` +##### Inline tier badges + Do not add tier badges inline with other text, except for [API attributes](../restful_api_styleguide.md). The single source of truth for a feature should be the topic where the functionality is described. -#### Available product tier badges +##### Pages that don't need a tier badge -| Where feature is available | Tier badge | -|:-----------------------------------------------------------------------------------------|:----------------------| -| On GitLab self-managed and GitLab SaaS, available in all tiers. | `**(FREE)**` | -| On GitLab self-managed and GitLab SaaS, available in Premium and Ultimate. | `**(PREMIUM)**` | -| On GitLab self-managed and GitLab SaaS, available in Ultimate. | `**(ULTIMATE)**` | -| On GitLab self-managed, available in all tiers. Not available on GitLab SaaS. | `**(FREE SELF)**` | -| On GitLab self-managed, available in Premium and Ultimate. Not available on GitLab SaaS. | `**(PREMIUM SELF)**` | -| On GitLab self-managed, available in Ultimate. Not available on GitLab SaaS. | `**(ULTIMATE SELF)**` | -| On GitLab SaaS, available in all tiers. Not available on self-managed. | `**(FREE SAAS)**` | -| On GitLab SaaS, available in Premium and Ultimate. Not available on self-managed. | `**(PREMIUM SAAS)**` | -| On GitLab SaaS, available in Ultimate. Not available on self-managed. | `**(ULTIMATE SAAS)**` | +Some pages won't have a tier badge, because no obvious tier badge applies. For example: + +- Tutorials. +- Pages that compare features from different tiers. +- Pages in the `/development` folder. These pages are automatically assigned a `Contribute` badge. + +##### Administrator documentation tier badges Topics that are only for instance administrators should be badged `<TIER> SELF`. Instance administrator documentation often includes sections that mention: @@ -1605,6 +1708,25 @@ instance administrator. Certain styles should be applied to specific sections. Styles for specific sections are outlined in this section. +### Help and feedback section + +This section ([introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/319) in GitLab 11.4) +is displayed at the end of each document and can be omitted by adding a key into +the front matter: + +```yaml +--- +feedback: false +--- +``` + +The default is to leave it there. If you want to omit it from a document, you +must check with a technical writer before doing so. + +The click events in the feedback section are tracked with Google Tag Manager. +The conversions can be viewed on Google Analytics by navigating to +**Behavior > Events > Top events > docs**. + ### GitLab restart When a restart or reconfigure of GitLab is required, avoid duplication by linking @@ -1822,8 +1944,3 @@ It renders as: ``` ::EndTabs - -## Feature flags - -Learn how to [document features deployed behind flags](../feature_flags.md). For -guidance on developing GitLab with feature flags, see [Feature flags in development of GitLab](../../feature_flags/index.md). diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index 84ab2ecc4e9..d65df0b56c8 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -188,6 +188,10 @@ Instead of: - As none of the endpoints return an ID... +## as well as + +Instead of **as well as**, use **and**. + ## associate Do not use **associate** when describing adding issues to epics, or users to issues, merge requests, @@ -255,6 +259,10 @@ Use **cannot** instead of **can not**. See also [contractions](index.md#contractions). +## Chat, GitLab Duo Chat + +Use **Chat** with a capital `c` for **Chat** or **GitLab Duo Chat**. + ## checkbox Use one word for **checkbox**. Do not use **check box**. @@ -617,6 +625,14 @@ Use title case for **Geo**. Do not make **GitLab** possessive (GitLab's). This guidance follows [GitLab Trademark Guidelines](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/brand/brand-activation/trademark-guidelines/). +## GitLab Dedicated + +Do not use **Dedicated** by itself. Always use **GitLab Dedicated**. + +## GitLab Duo + +Do not use **Duo** by itself. Always use **GitLab Duo**. + ## GitLab Flavored Markdown When possible, spell out [**GitLab Flavored Markdown**](../../../user/markdown.md). @@ -1154,7 +1170,10 @@ Use lowercase for **personal access token**. ## please -Do not use **please**. For more information, see the [Microsoft Style Guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). +Do not use **please** in the product documentation. + +In UI text, use **please** when we've inconvenienced the user. For more information, +see the [Microsoft Style Guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). ## Premium @@ -1279,6 +1298,12 @@ Use lowercase for **runner managers**. These are a type of runner that can creat Use lowercase for **runner workers**. This is the process created by the runner on the host computing platform to run jobs. See also [GitLab Runner](#gitlab-runner). +## runner authentication token + +Use **runner authentication token** instead of variations like **runner token**, **authentication token**, or **token**. +Runners are assigned runner authentication tokens when they are created, and use them to authenticate with GitLab when +they execute jobs. + ## (s) Do not use **(s)** to make a word optionally plural. It can slow down comprehension. For example: @@ -1399,6 +1424,17 @@ Instead of **and/or**, use **or** or re-write the sentence. This rule also appli Do not use **slave**. Another option is **secondary**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) +## storages + +In the context of: + +- Gitaly, storage is physical and must be called a **storage**. +- Gitaly Cluster, storage can be either: + - Virtual and must be called a **virtual storage**. + - Physical and must be called a **physical storage**. + +Gitaly storages have physical paths and virtual storages have virtual paths. + ## subgroup Use **subgroup** (no hyphen) instead of **sub-group**. ([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index ab0fe7b20a9..0c65e008436 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -33,7 +33,7 @@ We also run some documentation tests in the: ## Run tests locally -Similar to [previewing your changes locally](index.md#previewing-the-changes-live), you can also +Similar to [previewing your changes locally](review_apps.md), you can also run these tests on your local computer. This has the advantage of: - Speeding up the feedback loop. You can know of any problems with the changes in your branch @@ -274,6 +274,34 @@ You can use markdownlint: - [In a code editor](#configure-editors). - [In a `pre-push` hook](#configure-pre-push-hooks). +#### Markdown rule `MD044/proper-names` (capitalization) + +A rule that can cause confusion is `MD044/proper-names`. The failure, or +how to correct it, might not be immediately clear. +This rule checks a list of known words, listed in the `.markdownlint.yml` +file in each project, to verify proper use of capitalization and backticks. +Words in backticks are ignored by markdownlint. + +In general, product names should follow the exact capitalization of the official +names of the products, protocols, and so on. + +Some examples fail if incorrect capitalization is used: + +- MinIO (needs capital `IO`) +- NGINX (needs all capitals) +- runit (needs lowercase `r`) + +Additionally, commands, parameters, values, filenames, and so on must be +included in backticks. For example: + +- "Change the `needs` keyword in your `.gitlab-ci.yml`..." + - `needs` is a parameter, and `.gitlab-ci.yml` is a file, so both need backticks. + Additionally, `.gitlab-ci.yml` without backticks fails markdownlint because it + does not have capital G or L. +- "Run `git clone` to clone a Git repository..." + - `git clone` is a command, so it must be lowercase, while Git is the product, + so it must have a capital G. + ### Vale [Vale](https://vale.sh/) is a grammar, style, and word usage linter for the @@ -395,6 +423,8 @@ In general, follow these guidelines: At a minimum, install [markdownlint](#markdownlint) and [Vale](#vale) to match the checks run in build pipelines. +These tools can be [integrated with your code editor](#configure-editors). + #### Install markdownlint You can install either `markdownlint-cli` or `markdownlint-cli2` to run `markdownlint`. @@ -417,30 +447,43 @@ the `image:docs-lint-markdown`. #### Install Vale -To install Install [`vale`](https://github.com/errata-ai/vale/releases) for: +Install [`vale`](https://github.com/errata-ai/vale/releases) using either: -- macOS using `brew`, run: `brew install vale`. -- Linux, use your distribution's package manager or a [released binary](https://github.com/errata-ai/vale/releases). +- The [`asdf-vale` plugin](https://github.com/pdemagny/asdf-vale) if using [`asdf`](https://asdf-vm.com). In a checkout + of a GitLab project with a `.tool-versions` file ([example](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.tool-versions)), + run: -These tools can be [integrated with your code editor](#configure-editors). + ```shell + asdf plugin add vale && asdf install vale + ``` + +- A package manager: + - macOS using `brew`, run: `brew install vale`. + - Linux, use your distribution's package manager or a [released binary](https://github.com/errata-ai/vale/releases). ### Update linters -It's important to use linter versions that are the same or newer than those run in -CI/CD. This provides access to new features and possible bug fixes. +It's preferable to use linter versions that are the same as those used in our CI/CD pipelines for maximum compatibility +with the linting rules we use. + +To match the versions of `markdownlint-cli` (or `markdownlint-cli2`) and `vale` used in the GitLab projects, refer to: + +- For projects managed with `asdf`, the `.tool-versions` file in the project. For example, the + [`.tool-versions` file in the `gitlab` project](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.tool-versions). +- The [versions used (see `variables:` section)](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/.gitlab-ci.yml) + when building the `image:docs-lint-markdown` Docker image containing these tools for CI/CD. -To match the versions of `markdownlint-cli` (or `markdownlint-cli2`) and `vale` used in the GitLab projects, refer to the -[versions used (see `variables:` section)](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/.gitlab-ci.yml) -when building the `image:docs-lint-markdown` Docker image containing these tools for CI/CD. +Versions set in these two locations should be the same. | Tool | Version | Command | Additional information | |:--------------------|:---------|:------------------------------------------|:---------------------------------------------------------------------------------------------| | `markdownlint-cli` | Latest | `yarn global add markdownlint-cli` | None. | | `markdownlint-cli2` | Latest | `yarn global add markdownlint-cli2` | None. | -| `markdownlint-cli` | Specific | `yarn global add markdownlint-cli@0.23.2` | The `@` indicates a specific version, and this example updates the tool to version `0.23.2`. | -| `markdownlint-cli` | Specific | `yarn global add markdownlint-cli@0.6.0` | The `@` indicates a specific version, and this example updates the tool to version `0.6.0`. | -| Vale | Latest | `brew update && brew upgrade vale` | This command is for macOS only. | -| Vale | Specific | Not applicable. | Binaries can be [directly downloaded](https://github.com/errata-ai/vale/releases). | +| `markdownlint-cli` | Specific | `yarn global add markdownlint-cli@0.35.0` | The `@` indicates a specific version, and this example updates the tool to version `0.35.0`. | +| `markdownlint-cli2` | Specific | `yarn global add markdownlint-cli2@0.8.1` | The `@` indicates a specific version, and this example updates the tool to version `0.8.1`. | +| Vale (using `asdf`) | Specific | `asdf install` | Installs the version of Vale set in `.tool-versions` file in a project. | +| Vale (other) | Specific | Not applicable. | Binaries can be [directly downloaded](https://github.com/errata-ai/vale/releases). | +| Vale (using `brew`) | Latest | `brew update && brew upgrade vale` | This command is for macOS only. | ### Configure editors @@ -465,7 +508,7 @@ To configure markdownlint in your editor, install one of the following as approp To configure Vale in your editor, install one of the following as appropriate: - Sublime Text [`SublimeLinter-vale` package](https://packagecontrol.io/packages/SublimeLinter-vale). -- Visual Studio Code [`errata-ai.vale-server` extension](https://marketplace.visualstudio.com/items?itemName=errata-ai.vale-server). +- Visual Studio Code [`ChrisChinchilla.vale-vscode` extension](https://marketplace.visualstudio.com/items?itemName=ChrisChinchilla.vale-vscode). You can configure the plugin to [display only a subset of alerts](#show-subset-of-vale-alerts). - Vim [ALE plugin](https://github.com/dense-analysis/ale). - JetBrains IDEs - No plugin exists, but diff --git a/doc/development/documentation/topic_types/index.md b/doc/development/documentation/topic_types/index.md index 381f70914c6..40039ca5b1a 100644 --- a/doc/development/documentation/topic_types/index.md +++ b/doc/development/documentation/topic_types/index.md @@ -44,9 +44,11 @@ In general, for topic titles: - Be clear and direct. Make every word count. - Use articles and prepositions. -- Follow [capitalization](../styleguide/index.md#capitalization) guidelines. +- Follow [capitalization](../styleguide/index.md#topic-titles) guidelines. - Do not repeat text from earlier topic titles. For example, if the page is about merge requests, instead of `Troubleshooting merge requests`, use only `Troubleshooting`. +- Avoid using hyphens to separate information. + For example, instead of `Internal analytics - Architecture`, use `Internal analytics architecture` or `Architecture of internal analytics`. See also [guidelines for heading levels in Markdown](../styleguide/index.md#heading-levels-in-markdown). diff --git a/doc/development/documentation/topic_types/troubleshooting.md b/doc/development/documentation/topic_types/troubleshooting.md index 70475eb0ccf..4b23117acdb 100644 --- a/doc/development/documentation/topic_types/troubleshooting.md +++ b/doc/development/documentation/topic_types/troubleshooting.md @@ -11,9 +11,25 @@ Troubleshooting topics should be the final topics on a page. If a page has more than five troubleshooting topics, put the content on a separate page that has troubleshooting information exclusively. Name the page `Troubleshooting <feature>` and in the left nav, use the word `Troubleshooting` only. -Troubleshooting can be one of three types. +## What type of troubleshooting information to include -## An introductory topic +Troubleshooting information includes: + +- Problem-solving information that might be considered risky. +- Information about rare cases. All troubleshooting information + is included, no matter how unlikely a user is to encounter a situation. + +This kind of content can be helpful to others, and the benefits outweigh the risks. +If you think you have an exception to this rule, contact the Technical Writing team. + +GitLab Support maintains their own +[troubleshooting content](../../../administration/troubleshooting/index.md). + +## Troubleshooting topic types + +Troubleshooting can be one of three types: introductory, task, or reference. + +### An introductory topic This topic introduces the troubleshooting section of a page. For example: @@ -24,12 +40,12 @@ For example: When working with <x feature>, you might encounter the following issues. ``` -## Troubleshooting task +### Troubleshooting task The title should be similar to a [standard task](task.md). For example, "Run debug tools" or "Verify syntax." -## Troubleshooting reference +### Troubleshooting reference This topic includes the error message. For example: diff --git a/doc/development/export_csv.md b/doc/development/export_csv.md index 971159e83b9..9b0205166bf 100644 --- a/doc/development/export_csv.md +++ b/doc/development/export_csv.md @@ -8,14 +8,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w This document lists the different implementations of CSV export in GitLab codebase. -| Export type | How it works | Advantages | Disadvantages | Existing examples | -|---|---|---|---|---| -| Streaming | - Query and yield data in batches to a response stream.<br>- Download starts immediately. | - Report available immediately. | - No progress indicator.<br>- Requires a reliable connection. | [Export Audit Event Log](../administration/audit_events.md#export-to-csv) | -| Downloading | - Query and write data in batches to a temporary file.<br>- Loads the file into memory.<br>- Sends the file to the client. | - Report available immediately. | - Large amount of data might cause request timeout.<br>- Memory intensive.<br>- Request expires when user navigates to a different page. | - [Export Chain of Custody Report](../user/compliance/compliance_report/index.md#chain-of-custody-report)<br>- [Export License Usage File](../subscriptions/self_managed/index.md#export-your-license-usage) | -| As email attachment | - Asynchronously process the query with background job.<br>- Email uses the export as an attachment. | - Asynchronous processing. | - Requires users use a different app (email) to download the CSV.<br>- Email providers may limit attachment size. | - [Export issues](../user/project/issues/csv_export.md)<br>- [Export merge requests](../user/project/merge_requests/csv_export.md) | -| As downloadable link in email (*) | - Asynchronously process the query with background job.<br>- Email uses an export link. | - Asynchronous processing.<br>- Bypasses email provider attachment size limit. | - Requires users use a different app (email).<br>- Requires additional storage and cleanup. | [Export User Permissions](https://gitlab.com/gitlab-org/gitlab/-/issues/1772) | -| Polling (non-persistent state) | - Asynchronously processes the query with the background job.<br>- Frontend(FE) polls every few seconds to check if CSV file is ready. | - Asynchronous processing.<br>- Automatically downloads to local machine on completion.<br>- In-app solution. | - Non-persistable request - request expires when user navigates to a different page.<br>- API is processed for each polling request. | [Export Vulnerabilities](../user/application_security/vulnerability_report/index.md#export-vulnerability-details) | -| Polling (persistent state) (*) | - Asynchronously processes the query with background job.<br>- Backend (BE) maintains the export state<br>- FE polls every few seconds to check status.<br>- FE shows 'Download link' when export is ready.<br>- User can download or regenerate a new report. | - Asynchronous processing.<br>- No database calls made during the polling requests (HTTP 304 status is returned until export status changes).<br>- Does not require user to stay on page until export is complete.<br>- In-app solution.<br>- Can be expanded into a generic CSV feature (such as dashboard / CSV API). | - Requires to maintain export states in DB.<br>- Does not automatically download the CSV export to local machine, requires users to select 'Download'. | [Export Merge Commits Report](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43055) | +| Export type | How it works | Advantages | Disadvantages | Existing examples | +|---|---|---|---|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Streaming | - Query and yield data in batches to a response stream.<br>- Download starts immediately. | - Report available immediately. | - No progress indicator.<br>- Requires a reliable connection. | [Export Audit Event Log](../administration/audit_events.md#export-to-csv) | +| Downloading | - Query and write data in batches to a temporary file.<br>- Loads the file into memory.<br>- Sends the file to the client. | - Report available immediately. | - Large amount of data might cause request timeout.<br>- Memory intensive.<br>- Request expires when user navigates to a different page. | - [Export Chain of Custody Report](../user/compliance/compliance_center/index.md#chain-of-custody-report)<br>- [Export License Usage File](../subscriptions/self_managed/index.md#export-your-license-usage) | +| As email attachment | - Asynchronously process the query with background job.<br>- Email uses the export as an attachment. | - Asynchronous processing. | - Requires users use a different app (email) to download the CSV.<br>- Email providers may limit attachment size. | - [Export issues](../user/project/issues/csv_export.md)<br>- [Export merge requests](../user/project/merge_requests/csv_export.md) | +| As downloadable link in email (*) | - Asynchronously process the query with background job.<br>- Email uses an export link. | - Asynchronous processing.<br>- Bypasses email provider attachment size limit. | - Requires users use a different app (email).<br>- Requires additional storage and cleanup. | [Export User Permissions](https://gitlab.com/gitlab-org/gitlab/-/issues/1772) | +| Polling (non-persistent state) | - Asynchronously processes the query with the background job.<br>- Frontend(FE) polls every few seconds to check if CSV file is ready. | - Asynchronous processing.<br>- Automatically downloads to local machine on completion.<br>- In-app solution. | - Non-persistable request - request expires when user navigates to a different page.<br>- API is processed for each polling request. | [Export Vulnerabilities](../user/application_security/vulnerability_report/index.md#export-vulnerability-details) | +| Polling (persistent state) (*) | - Asynchronously processes the query with background job.<br>- Backend (BE) maintains the export state<br>- FE polls every few seconds to check status.<br>- FE shows 'Download link' when export is ready.<br>- User can download or regenerate a new report. | - Asynchronous processing.<br>- No database calls made during the polling requests (HTTP 304 status is returned until export status changes).<br>- Does not require user to stay on page until export is complete.<br>- In-app solution.<br>- Can be expanded into a generic CSV feature (such as dashboard / CSV API). | - Requires to maintain export states in DB.<br>- Does not automatically download the CSV export to local machine, requires users to select 'Download'. | [Export Merge Commits Report](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43055) | NOTE: Export types marked as * are currently work in progress. diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index b00131b12f3..65b50bedb0c 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -518,9 +518,26 @@ Proper research and testing should be done to ensure compliance with [WCAG](http ## Automated accessibility testing with axe -You can use [axe-core](https://github.com/dequelabs/axe-core) [gems](https://github.com/dequelabs/axe-core-gems) +We use [axe-core](https://github.com/dequelabs/axe-core) [gems](https://github.com/dequelabs/axe-core-gems) to run automated accessibility tests in feature tests. +[We aim to conform to level AA of the World Wide Web Consortium (W3C) Web Content Accessibility Guidelines 2.1](https://design.gitlab.com/accessibility/a11y). + +### When to add accessibility tests + +When adding a new view to the application, make sure to include the accessibility check in your feature test. +We aim to have full coverage for all the views. + +One of the advantages of testing in feature tests is that we can check different states, not only +single components in isolation. + +Make sure to add assertions, when the view you are working on: + +- Has an empty state, +- Has significant changes in page structure, for example an alert is shown, or a new section is rendered. + +### How to add accessibility tests + Axe provides the custom matcher `be_axe_clean`, which can be used like the following: ```ruby @@ -530,10 +547,12 @@ it 'passes axe automated accessibility testing', :js do wait_for_requests # ensures page is fully loaded - expect(page).to be_axe_clean + expect(page).to be_axe_clean.according_to :wcag21aa end ``` +Make sure to specify the accessibility standard as `:wcag21aa`. + If needed, you can scope testing to a specific area of the page by using `within`. Axe also provides specific [clauses](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#clauses), @@ -543,20 +562,22 @@ for example: expect(page).to be_axe_clean.within '[data-testid="element"]' # run only WCAG 2.0 Level AA rules -expect(page).to be_axe_clean.according_to :wcag2aa +expect(page).to be_axe_clean.according_to :wcag21aa # specifies which rule to skip expect(page).to be_axe_clean.skipping :'link-in-text-block' # clauses can be chained expect(page).to be_axe_clean.within('[data-testid="element"]') - .according_to(:wcag2aa) + .according_to(:wcag21aa) ``` Axe does not test hidden regions, such as inactive menus or modal windows. To test hidden regions for accessibility, write tests that activate or render the regions visible and run the matcher again. +You can run accessibility tests locally in the same way as you [run any feature tests](../testing_guide/frontend_testing.md#how-to-run-a-feature-test). + ### Known accessibility violations This section documents violations where a recommendation differs with the [design system](https://design.gitlab.com/): diff --git a/doc/development/fe_guide/dark_mode.md b/doc/development/fe_guide/dark_mode.md index 55181edd64c..5e8f844172a 100644 --- a/doc/development/fe_guide/dark_mode.md +++ b/doc/development/fe_guide/dark_mode.md @@ -5,8 +5,7 @@ group: Development info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -This page is about developing dark mode for GitLab. We also have documentation on how -[to enable dark mode](../../user/profile/preferences.md#dark-mode). +This page is about developing dark mode for GitLab. For more information on how to enable dark mode, see [Change the syntax highlighting theme]](../../user/profile/preferences.md#change-the-syntax-highlighting-theme). # How dark mode works diff --git a/doc/development/fe_guide/frontend_goals.md b/doc/development/fe_guide/frontend_goals.md new file mode 100644 index 00000000000..05e9b0df389 --- /dev/null +++ b/doc/development/fe_guide/frontend_goals.md @@ -0,0 +1,31 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Frontend Goals + +This section defined the _desired state_ of the GitLab frontend how we see it over the next few years. + +## Cluster SPAs + +Currently, GitLab mostly follows Rails architecture and Rails routing which means every single time we're changing route, we have page reload. This results in long loading times because we are: + +- rendering HAML page; +- mounting Vue applications if we have any; +- fetching data for these applications + +Ideally, we should reduce the number of times user needs to go through this long process. This would be possible with converting GitLab into a single-page application but this would require significant refactoring and is not an achieavable short/mid-term goal. + +The realistic goal is to move to _multiple SPAs_ experience where we define the _clusters_ of pages that form the user flow, and move this cluster from Rails routing to a single-page application with client-side routing. This way, we can load all the relevant context from HAML only once, and fetch all the additional data from the API depending on the route. An example of a cluster could be the following pages: + +- issues list +- issue boards +- issue details page +- new issue +- editing an issue + +All of them have the same context (project path, current user etc.), we could easily fetch more data with issue-specific parameter (issue `iid`) and store the results on the client (so that opening the same issue won't require more API calls). This leads to a smooth user experience for navigating through issues. + +For navigation between clusters, we can still rely on Rails routing. These cases should be relatively more scarce than navigation within clusters. diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 00c7bb5d6c8..62183c7b349 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -416,7 +416,11 @@ Read more about local state management with Apollo in the [Vue Apollo documentat ### Using with Vuex -We do not recommend creating new applications with Vuex and Apollo Client combined +We do not recommend creating new applications with Vuex and Apollo Client combined. There are a few reasons: + +- VueX and Apollo are both **global stores**, which means sharing responsibilities and having two sources of truth. +- Keeping VueX and Apollo in sync can be high maintenance. +- Bugs that would come from the communication between Apollo and VueX would be subtle and hard to debug. ### Working on GraphQL-based features when frontend and backend are not in sync diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 8675866fce6..405e830406e 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -26,6 +26,13 @@ modern ECMAScript standards supported through [Babel](https://babeljs.io/) and E When making API calls, we use [GraphQL](graphql.md) as [the first choice](../../api/graphql/index.md#vision). There are still instances where GitLab REST API is used such as when creating new simple HAML pages or in legacy part of the codebase, but we should always default to GraphQL when possible. +We use [Apollo](https://www.apollographql.com/) as our global state manager and [GraphQL client](graphql.md). +[VueX](vuex.md) is still in use across the codebase, but it is no longer the recommended global state manager. +You should **not** [use VueX and Apollo together](graphql.md#using-with-vuex), +and should [avoid adding new VueX stores](migrating_from_vuex.md) whenever possible. + +For copy strings and translations, we have frontend utilities available. Please see the JavaScript section of [Preparing a page for translation](../i18n/externalization.md#javascript-files) for more information. + Working with our frontend assets requires Node (v12.22.1 or greater) and Yarn (v1.10.0 or greater). You can find information on how to install these on our [installation guide](../../install/installation.md#5-node). @@ -56,7 +63,7 @@ GitLab is now a large, enterprise-grade software and it often requires complex c - Building tools that solve commonly-faced problems and making them easily discoverable. - Writing better documentation on how we solve our problems. -- Writing loosly coupled components that can be easily added or removed from our codebase. +- Writing loosely coupled components that can be easily added or removed from our codebase. - Remove older technologies or pattern that we deem are no longer acceptable. By focusing on these aspects, we aim to allow engineers to contain complexity in well defined boundaries and quickly share them with their peers. @@ -67,9 +74,9 @@ Now that our values have been defined, we can base our goals on these values and - Lowest possible FID, LCP and cross-page navigation times - Minimal page reloads when interacting with the UI -- Have as little Vue applications per page as possible -- Leverage Ruby ViewComponents for simple pages and avoid Vue overhead when possible -- Migrate existing VueX stores to Apollo, but more urgently **stop using both together** +- [Have as little Vue applications per page as possible](vue.md#avoid-multiple-vue-applications-on-the-page) +- Leverage [Ruby ViewComponents](view_component.md) for simple pages and avoid Vue overhead when possible +- [Migrate away from VueX](migrating_from_vuex.md), but more urgently **stop using Apollo and VueX together** - Remove jQuery from our codebase - Add a visual testing framework - Reduce CSS bundle size to a minimum @@ -77,9 +84,11 @@ Now that our values have been defined, we can base our goals on these values and - Improve our pipelines speed - Build a better set of shared components with documentation +We have detailed description on how we see GitLab frontend in the future in [Frontend Goals](frontend_goals.md) section + ### Frontend onboarding course -The [Frontend onboarding course](onboarding_course/index.md) provides a 6-week structured curriculum to learn how to contribute to the GitLab frontend. +The [Frontend onboarding course](onboarding_course/index.md) provides a 6-week structured curriculum to learn how to contribute to the GitLab frontend. ### Browser Support diff --git a/doc/development/fe_guide/merge_request_widget_extensions.md b/doc/development/fe_guide/merge_request_widget_extensions.md index f5bf9049db1..99206d99590 100644 --- a/doc/development/fe_guide/merge_request_widget_extensions.md +++ b/doc/development/fe_guide/merge_request_widget_extensions.md @@ -351,10 +351,7 @@ To generate these known events for a single widget: ``` 1. Repeat step 6, but change the `data_source` to `redis_hll`. -1. Add each of the HLL metrics to `lib/gitlab/usage_data_counters/known_events/code_review_events.yml`: - 1. `name` = (the event) - 1. `category` = `code_review` - 1. `aggregation` = `weekly` + 1. Add each event (those listed in the command in step 7, replacing `test_reports` with the appropriate name slug) to the aggregate files: 1. `config/metrics/counts_7d/{timestamp}_code_review_category_monthly_active_users.yml` diff --git a/doc/development/fe_guide/migrating_from_vuex.md b/doc/development/fe_guide/migrating_from_vuex.md new file mode 100644 index 00000000000..8dca744e192 --- /dev/null +++ b/doc/development/fe_guide/migrating_from_vuex.md @@ -0,0 +1,318 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Migrating from Vuex + +## Why? + +We have defined [GraphQL as our primary API](../../api/graphql/index.md#vision) for all user-facing features, +so we can safely assume that whenever GraphQL is present, so will the Apollo Client. +We [do not want to use Vuex with Apollo](graphql.md#using-with-vuex), so the VueX stores count +will naturally decline over time as we move from the REST API to GraphQL. + +This section gives guidelines and methods to translate an existing VueX store to +pure Vue and Apollo, or how to rely less on VueX. + +## How? + +### Overview + +As a whole, we want to understand how complex our change will be. Sometimes, we only have a few properties that are truly worth being stored in a global state and sometimes they can safely all be extracted to pure `Vue`. `VueX` properties generally fall into one of these categories: + +- Static properties +- Reactive mutable properties +- Getters +- API data + +Therefore, the first step is to read the current VueX state and determine the category of each property. + +At a high level, we could map each category with an equivalent non-VueX code pattern: + +- Static properties: Provide/Inject from Vue API. +- Reactive mutable properties: Vue events and props, Apollo Client. +- Getters: Utils functions, Apollo `update` hook, computed properties. +- API data: Apollo Client. + +Let's go through an example. In each section we refer to this state and slowly go through migrating it fully: + +```javascript +// state.js AKA our store +export default ({ blobPath = '', summaryEndpoint = '', suiteEndpoint = '' }) => ({ + blobPath, + summaryEndpoint, + suiteEndpoint, + testReports: {}, + selectedSuiteIndex: null, + isLoading: false, + errorMessage: null, + limit : 10, + pageInfo: { + page: 1, + perPage: 20, + }, +}); +``` + +### How to migrate static values + +The easiest type of values to migrate are static values, either: + +- Client-side constants: If the static value is a client-side constant, it may have been implemented + in the store for easy access by other state properties or methods. However, it is generally + a better practice to add such values to a `constants.js` file and import it when needed. +- Rails-injected dataset: These are values that we may need to provide to our Vue apps. + They are static, so adding them to the VueX store is not necessary and it could instead + be done easily through the `provide/inject` Vue API, which would be equivalent but without the VueX overhead. This should **only** be injected inside the top-most JS file that mounts our component. + +If we take a look at our example above, we can already see that two properties contain `Endpoint` in their name, which probably means that these come from our Rails dataset. To confirm this, we would search the codebase for these properties and see where they are defined, which is the case in our example. Additionally, `blobPath` is also a static property, and a little less obvious here is that `pageInfo` is actually a constant! It is never modified and is only used as a default value that we use inside our getter: + +```javascript +// state.js AKA our store +export default ({ blobPath = '', summaryEndpoint = '', suiteEndpoint = '' }) => ({ + limit + blobPath, // Static - Dataset + summaryEndpoint, // Static - Dataset + suiteEndpoint, // Static - Dataset + testReports: {}, + selectedSuiteIndex: null, + isLoading: false, + errorMessage: null, + pageInfo: { // Static - Constant + page: 1, // Static - Constant + perPage: 20, // Static - Constant + }, +}); +``` + +### How to migrate reactive mutable values + +These values are especially useful when used by a lot of different components, so we can first evaluate how many reads and writes each property gets, and how far apart these are from each other. The fewer reads there are and the closer together they live, the easier it will be to remove these properties in favor of native Vue props and events. + +#### Simple read/write values + +If we go back to our example, `selectedSuiteIndex` is only used by **one component** and also **once inside a getter**. Additionally, this getter is only used once itself! It would be quite easy to translate this logic to Vue because this could become a `data` property on the component instance. For the getter, we can use a computed property instead, or a method on the component that returns the right item because we will have access to the index there as well. This is a perfect example of how the VueX store here complicates the application by adding a lot of abstractions when really everything could live inside the same component. + +Luckily, in our example all properties could live inside the same component. However, there are cases where it will not be possible. When this happens, we can use Vue events and props to communicate between sibling components. Store the data in question inside a parent component that should know about the state, and when a child component wants to write to the component, it can `$emit` an event with the new value and let the parent update. Then, by cascading props down to all of its children, all instances of the sibling components will share the same data. + +Sometimes, it can feel that events and props are cumbersome, especially in very deep component trees. However, it is quite important to be aware that this is mostly an inconvenience issue and not an architectural flaw or problem to fix. Passing down props, even deeply nested, is a very acceptable pattern for cross-components communication. + +#### Shared read/write values + +Let's assume that we have a property in the store that is used by multiple components for read and writes that are either so numerous or far apart that Vue props and events seem like a bad solution. Instead, we use Apollo client-side resolvers. This section requires knowledge of [Apollo Client](graphql.md), so feel free to check the apollo details as needed. + +First we need to set up our Vue app to use `VueApollo`. Then when creating our store, we pass the `resolvers` and `typedefs` (defined later) to the Apollo Client: + +```javascript +import { resolvers } from "./graphql/settings.js" +import typeDefs from './graphql/typedefs.graphql'; + +... +const apolloProvider = new VueApollo({ + defaultClient: createDefaultClient({ + resolvers, // To be written soon + { typeDefs }, // We are going to create this in a sec + }), +}); +``` + +For our example, let's call our field `app.status`, and we need is to define queries and mutations that use the `@client` directives. Let's create them right now: + +```javascript +// get_app_status.query.graphql +query getAppStatus { + app @client { + status + } +} +``` + +```javascript +// update_app_status.mutation.graphql +mutation updateAppStatus($appStatus: String) { + updateAppStatus(appStatus: $appStatus) @client +} +``` + +For fields that **do not exist in our schema**, we need to set up `typeDefs`. For example: + +```javascript +// typedefs.graphql + +type TestReportApp { + status: String! +} + +extend type Query { + app: TestReportApp +} +``` + +Now we can write our resolvers so that we can update the field with our mutation: + +```javascript +// settings.js +export const resolvers = { + Mutation: { + // appStatus is the argument to our mutation + updateAppStatus: (_, { appStatus }, { cache }) => { + cache.writeQuery({ + query: getAppStatus, + data: { + app: { + __typename: 'TestReportApp', + status: appStatus, + }, + }, + }); + }, + } +} +``` + +For querying, this works without any additional instructions because it behaves like any `Object`, because querying for `app { status }` is equivalent to `app.status`. However, we need to write either a "default" `writeQuery` (to define the very first value our field will have) or we can set up the [`typePolicies` for our `cacheConfig`](graphql.md#local-state-with-apollo) to provide this default value. + +So now when we want to read from this value, we can use our local query. When we need to update it, we can call the mutation and pass the new value as an argument. + +#### Network-related values + +There are values like `isLoading` and `errorMessage` which are tied to the network request state. These are read/write properties, but will easily be replaced later with Apollo Client's own capabilities without us doing any extra work: + +```javascript +// state.js AKA our store +export default ({ blobPath = '', summaryEndpoint = '', suiteEndpoint = '' }) => ({ + blobPath, // Static - Dataset + summaryEndpoint, // Static - Dataset + suiteEndpoint, // Static - Dataset + testReports: {}, + selectedSuiteIndex: null, // Mutable -> data property + isLoading: false, // Mutable -> tied to network + errorMessage: null, // Mutable -> tied to network + pageInfo: { // Static - Constant + page: 1, // Static - Constant + perPage: 20, // Static - Constant + }, +}); +``` + +### How to migrate getters + +Getters have to be reviewed case-by-case, but a general guideline is that it is highly possible to write a pure JavaScript util function that takes as an argument the state values we used to use inside the getter, and then return whatever value we want. Consider the following getter: + +```javascript +// getters.js +export const getSelectedSuite = (state) => + state.testReports?.test_suites?.[state.selectedSuiteIndex] || {}; +``` + +All that we do here is reference two state values, which can both become arguments to a function: + +```javascript +//new_utils.js +export const getSelectedSuite = (testReports, selectedSuiteIndex) => + testReports?.test_suites?.[selectedSuiteIndex] || {}; +``` + +This new util can then be imported and used as it previously was, but directly inside the component. Also, most of the specs for the getters can be ported to the utils quite easily because the logic is preserved. + +### How to migrate API data + +Our last property is called `testReports` and it is fetched via an `axios` call to the API. We assume that we are in a pure REST application and that GraphQL data is not yet available: + +```javascript +// actions.js +export const fetchSummary = ({ state, commit, dispatch }) => { + dispatch('toggleLoading'); + + return axios + .get(state.summaryEndpoint) + .then(({ data }) => { + commit(types.SET_SUMMARY, data); + }) + .catch(() => { + createAlert({ + message: s__('TestReports|There was an error fetching the summary.'), + }); + }) + .finally(() => { + dispatch('toggleLoading'); + }); +}; +``` + +We have two options here. If this action is only used once, there is nothing preventing us from just moving all of this code from the `actions.js` file to the component that does the fetching. Then, it would be easy to remove all the state related code in favor of `data` properties. In that case, `isLoading` and `errorMessages` would both live along with it because it's only used once. + +If we are reusing this function multiple time (or plan to), then that Apollo Client can be leveraged to do what it does best: network calls and caching. In this section, we assume Apollo Client knowledge and that you know how to set it up, but feel free to read through [the GraphQL documentation](graphql.md). + +We can use a local GraphQL query (with an `@client` directive) to structure how we want to receive the data, and then use a client-side resolver to tell Apollo Client how to resolve that query. We can take a look at our REST call in the browser network tab and determine which structure suits the use case. In our example, we could write our query like: + +```graphql +query getTestReportSummary($fullPath: ID!, $iid: ID!, endpoint: String!) { + project(fullPath: $fullPath){ + id, + pipeline(iid: $iid){ + id, + testReportSummary(endpoint: $endpoint) @client { + testSuites{ + nodes{ + name + totalTime, + # There are more fields here, but they aren't needed for our example + } + } + } + } + } +} +``` + +The structure here is arbitrary in the sense that we could write this however we want. It might be tempting to skip the `project.pipeline.testReportSummary` because this is not how the REST call is structured. However, by making the query structure compliant with the `GraphQL` API, we will not need to modify our query if we do decide to transition to `GraphQL` later, and can simply remove the `@client` directive. This also gives us **caching for free** because if we try to fetch the summary again for the same pipeline, Apollo Client knows that we already have the result! + +Additionally, we are passing an `endpoint` argument to our field `testReportSummary`. This would not be necessary in pure `GraphQL`, but our resolver is going to need that information to make the `REST` call later. + +Now we need to write a client-side resolver. When we mark a field with an `@client` directive, it is **not sent to the server**, and Apollo Client instead expects us to [define our own code to resolve the value](graphql.md#using-client-side-resolvers). We can write a client-side resolver for `testReportSummary` inside the `cacheConfig` object that we pass to Apollo Client. We want this resolver to make the Axios call and return whatever data structure we want. That this is also the perfect place to transfer a getter if it was always used when accessing the API data or massaging the data structure: + +```javascript +// graphql_config.js +export const resolvers = { + Query: { + testReportSummary(_, { summaryEndpoint }): { + return axios.get(summaryEndpoint).then(({ data }) => { + return data // we could format/massage our data here instead of using a getter + } + } +} +``` + +Any time we make a call to the `testReportSummary @client` field, this resolver is executed and returns the result of the operation, which is essentially doing the same job as the `VueX` action did. + +If we assume that our GraphQL call is stored inside a data property called `testReportSummary`, we can replace `isLoading` with `this.$apollo.queries.testReportSummary.lodaing` in any component that fires this query. Errors can be handled inside the `error` hook of the Query. + +### Migration strategy + +Now that we have gone through each type of data, let's review how to plan for the transition between a VueX-based store and one without. We are trying to avoid VueX and Apollo coexisting, so the less time where both stores are available in the same context the better. To minimize this overlap, we should start our migration by removing from the store all that does not involve adding an Apollo store. Each of the following point could be its own MR: + +1. Migrate away from Static values, both `Rails` dataset and client-side constants and use `provide/inject` and `constants.js` files instead. +1. Replace simple read/write operations with either: + - `data` properties and `methods` if in a single component. + - `props` and `emits` if shared across a localized group of components. +1. Replace shared read/write operations with Apollo Client `@client` directives. +1. Replace network data with Apollo Client, either with actual GraphQL calls when available or by using client-side resolvers to make REST calls. + +If it is impossible to quickly replace shared read/write operations or network data (for example in one or two milestones), consider making a different Vue component behind a feature flag that is exclusively functional with Apollo Client, and rename the current component that uses VueX with a `legacy-` prefix. The newer component might not be able to implement all functionality right away, but we can progressively add them as we make MRs. This way, our legacy component is exclusively using VueX as a store and the new one is only Apollo. After the new component has re-implemented all the logic, we can turn the Feature Flag on and ensure that it behaves as expected. + +## FAQ + +### What if I need a global store without any network call? + +This is a rare occurrence and should suggest the following question: "Do I **really** need a global store then?" (the answer is probably no!) If the answer is yes, then you can use the [shared read/write technique with Apollo](#how-to-migrate-reactive-mutable-values) described above. It is perfectly acceptable to use Apollo Client for client-side exclusive stores. + +### Are we going to use Pinia? + +The short answer is: we don't know, but it is unlikely. It would still mean having two global store libraries, which has the same downsides as VueX and Apollo Client coexisting. Reducing the size of our global stores is positive regardless of whether we end up using Pinia though! + +### Apollo client is really verbose for client directives. Can I mix and match with VueX? + +Mixing and matching is not recommended. There are a lot of reasons why, but think of how codebases grow organically with what is available. Even if you were really good at separating your network state and your client-side state, other developers might not share the same dedication or simply not understand how to choose what lives in which store. Over time, you will also nearly inevitably need to communicate between your VueX store and Apollo Client, which can only result in problems. diff --git a/doc/development/fe_guide/tooling.md b/doc/development/fe_guide/tooling.md index 762ef852d74..c1084ab4453 100644 --- a/doc/development/fe_guide/tooling.md +++ b/doc/development/fe_guide/tooling.md @@ -155,6 +155,19 @@ $ grep "eslint-disable.*import/no-deprecated" -r . ./app/assets/javascripts/issuable_form.js: // eslint-disable-next-line import/no-deprecated ``` +### `vue/multi-word-component-names` is disabled in my file + +Single name components are discouraged by the +[Vue style guide](https://vuejs.org/style-guide/rules-essential.html#use-multi-word-component-names). + +They are problematic because they can be confused with other HTML components: We could name a +component `<table>` and it would stop rendering an HTML `<table>`. + +To solve this, you should rename the `.vue` file and its references to use at least two words, +for example: + +- `user/table.vue` could be renamed to `user/users_table.vue` and be imported as `UsersTable` and used with `<users-table />`. + ### GraphQL schema and operations validation We use [`@graphql-eslint/eslint-plugin`](https://www.npmjs.com/package/@graphql-eslint/eslint-plugin) diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index e096d25f2d9..06d7070b855 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Vuex -[Vuex](https://vuex.vuejs.org) should no longer be considered a preferred path to store management and is currently in its legacy phase. This means it is acceptable to add upon existing `Vuex` stores, but we strongly recommend reducing store sizes over time and eventually migrating fully to [Apollo](https://www.apollographql.com/) whenever it's possible. Before adding any new `Vuex` store to an application, first ensure that the `Vue` application you plan to add it into **does not use** `Apollo`. `Vuex` and `Apollo` should not be combined unless absolutely necessary. Please consider reading through [our GraphQL documentation](../fe_guide/graphql.md) for more guidelines on how you can build `Apollo` based applications. +[Vuex](https://vuex.vuejs.org) should no longer be considered a preferred path to store management and is currently in its legacy phase. This means it is acceptable to add upon existing `Vuex` stores, but we strongly recommend reducing store sizes over time and eventually [migrating away from VueX entirely](migrating_from_vuex.md). Before adding any new `Vuex` store to an application, first ensure that the `Vue` application you plan to add it into **does not use** `Apollo`. `Vuex` and `Apollo` should not be combined unless absolutely necessary. Please consider reading through [our GraphQL documentation](../fe_guide/graphql.md) for more guidelines on how you can build `Apollo` based applications. The information included in this page is explained in more detail in the official [Vuex documentation](https://vuex.vuejs.org). @@ -40,6 +40,7 @@ applications stored in this [repository](https://gitlab.com/gitlab-org/gitlab/-/ This is the entry point for our store. You can use the following as a guide: ```javascript +// eslint-disable-next-line no-restricted-imports import Vuex from 'vuex'; import * as actions from './actions'; import * as getters from './getters'; @@ -287,6 +288,7 @@ function when mounting your Vue component: // in the Vue app's initialization script (for example, mount_show.js) import Vue from 'vue'; +// eslint-disable-next-line no-restricted-imports import Vuex from 'vuex'; import { createStore } from './stores'; import AwesomeVueApp from './components/awesome_vue_app.vue' @@ -372,6 +374,7 @@ when [providing data to a Vue app](vue.md#providing-data-from-haml-to-javascript ```javascript <script> +// eslint-disable-next-line no-restricted-imports import { mapActions, mapState, mapGetters } from 'vuex'; export default { @@ -433,6 +436,7 @@ components, we need to include the store and provide the correct state: ```javascript //component_spec.js import Vue from 'vue'; +// eslint-disable-next-line no-restricted-imports import Vuex from 'vuex'; import { mount } from '@vue/test-utils'; import { createStore } from './store'; diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index d9eb29a7b7f..8c0f7faab28 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -199,7 +199,7 @@ Only feature flags that have a YAML definition file can be used when running the ```shell $ bin/feature-flag my_feature_flag ->> Specify the group introducing the feature flag, like `group::apm`: +>> Specify the group introducing the feature flag, like `group::project management`: ?> group::application performance >> URL of the MR introducing the feature flag (enter to skip): @@ -231,6 +231,20 @@ the feature flag is set to enabled. If the feature contains any database migrati NOTE: To create a feature flag that is only used in EE, add the `--ee` flag: `bin/feature-flag --ee` +### Naming new flags + +When choosing a name for a new feature flag, consider the following guidelines: + +- A long, descriptive name is better than a short but confusing one. +- Write the name in snake case (`my_cool_feature_flag`). +- Avoid using `disable` in the name to avoid having to think (or [document](../documentation/feature_flags.md)) + with double negatives. Consider starting the name with `hide_`, `remove_`, or `disallow_`. + + In software engineering this problem is known as + ["negative names for boolean variables"](https://www.serendipidata.com/posts/naming-guidelines-for-boolean-variables). + But we can't forbid negative words altogether, to be able to introduce flags as + [disabled by default](#feature-flags-in-gitlab-development), use them to remove a feature by moving it behind a flag, or to [selectively disable a flag by actor](controls.md#selectively-disable-by-actor). + ### Risk of a broken master (main) branch WARNING: @@ -543,6 +557,8 @@ to ensure the feature works properly. If automated tests are not included for bo with the untested code path should be manually tested before deployment to production. When using the testing environment, all feature flags are enabled by default. +Flags can be disabled by default in the [`spec/spec_helper.rb` file](https://gitlab.com/gitlab-org/gitlab/-/blob/b61fba42eea2cf5bb1ca64e80c067a07ed5d1921/spec/spec_helper.rb#L274). +Please add a comment inline to explain why the flag needs to be disabled. You can also attach the issue URL for reference if possible. WARNING: This does not apply to end-to-end (QA) tests, which [do not enable feature flags by default](#end-to-end-qa-tests). There is a different [process for using feature flags in end-to-end tests](../testing_guide/end_to_end/feature_flags.md). @@ -557,6 +573,25 @@ stub_feature_flags(ci_live_trace: false) Feature.enabled?(:ci_live_trace) # => false ``` +A common pattern of testing both paths looks like: + +```ruby +it 'ci_live_trace works' do + # tests assuming ci_live_trace is enabled in tests by default + Feature.enabled?(:ci_live_trace) # => true +end + +context 'when ci_live_trace is disabled' do + before do + stub_feature_flags(ci_live_trace: false) + end + + it 'ci_live_trace does not work' do + Feature.enabled?(:ci_live_trace) # => false + end +end +``` + If you wish to set up a test where a feature flag is enabled only for some actors and not others, you can specify this in options passed to the helper. For example, to enable the `ci_live_trace` diff --git a/doc/development/features_inside_dot_gitlab.md b/doc/development/features_inside_dot_gitlab.md index f55d7e52fbc..a1f5111d6f4 100644 --- a/doc/development/features_inside_dot_gitlab.md +++ b/doc/development/features_inside_dot_gitlab.md @@ -16,4 +16,6 @@ When implementing new features, please refer to these existing features to avoid - [Route Maps](../ci/review_apps/index.md#route-maps): `.gitlab/route-map.yml`. - [Customize Auto DevOps Helm Values](../topics/autodevops/customize.md#customize-helm-chart-values): `.gitlab/auto-deploy-values.yaml`. - [Insights](../user/project/insights/index.md#configure-project-insights): `.gitlab/insights.yml`. -- [Service Desk Templates](../user/project/service_desk.md#customize-emails-sent-to-the-requester): `.gitlab/service_desk_templates/`. +- [Service Desk Templates](../user/project/service_desk/index.md#customize-emails-sent-to-the-requester): `.gitlab/service_desk_templates/`. +- [Secret Detection Custom Rulesets](../user/application_security/secret_detection/index.md#disable-predefined-analyzer-rules): `.gitlab/secret-detection-ruleset.toml` +- [Static Analysis Custom Rulesets](../user/application_security/sast/customize_rulesets.md#create-the-configuration-file): `.gitlab/sast-ruleset.toml` diff --git a/doc/development/fips_compliance.md b/doc/development/fips_compliance.md index bab4d7705f9..4f6a9feb191 100644 --- a/doc/development/fips_compliance.md +++ b/doc/development/fips_compliance.md @@ -60,7 +60,6 @@ listed here that also do not work properly in FIPS mode: - [Code Quality](../ci/testing/code_quality.md) does not support operating in FIPS-compliant mode. - [Dependency scanning](../user/application_security/dependency_scanning/index.md) support for Gradle. - [Dynamic Application Security Testing (DAST)](../user/application_security/dast/proxy-based.md) supports a reduced set of analyzers. The proxy-based analyzer is not available in FIPS mode today, however browser-based DAST, DAST API, and DAST API Fuzzing images are available. -- [License compliance](../user/compliance/license_compliance/index.md). - [Solutions for vulnerabilities](../user/application_security/vulnerabilities/index.md#resolve-a-vulnerability) for yarn projects. - [Static Application Security Testing (SAST)](../user/application_security/sast/index.md) @@ -118,48 +117,8 @@ for more details. The following instructions build on the Quick Start and are al ##### Terraform: Use a FIPS AMI -1. Follow the guide to set up Terraform and Ansible. -1. After [step 2b](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/docs/environment_quick_start_guide.md#2b-setup-config), - create a `data.tf` in your environment (for example, `gitlab-environment-toolkit/terraform/environments/gitlab-10k/inventory/data.tf`): - - ```tf - data "aws_ami" "ubuntu_20_04_fips" { - count = 1 - - most_recent = true - - filter { - name = "name" - values = ["ubuntu-pro-fips-server/images/hvm-ssd/ubuntu-focal-20.04-amd64-pro-fips-server-*"] - } - - filter { - name = "virtualization-type" - values = ["hvm"] - } - - owners = ["aws-marketplace"] - } - ``` - -1. Add the custom `ami_id` to use this AMI in `environment.tf`. For - example, in `gitlab-environment-toolkit/terraform/environments/gitlab-10k/inventory/environment.tf`: - - ```tf - module "gitlab_ref_arch_aws" { - source = "../../modules/gitlab_ref_arch_aws" - - prefix = var.prefix - ami_id = data.aws_ami.ubuntu_20_04_fips[0].id - ... - ``` - -NOTE: -GET does not allow the AMI to change on EC2 instances after it has -been deployed via `terraform apply`. Since an AMI change would tear down -an instance, this would result in data loss: not only would disks be -destroyed, but also GitLab secrets would be lost. There is a [Terraform lifecycle rule](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/blob/2aaeaff8ac8067f23cd7b6bb5bf131061649089d/terraform/modules/gitlab_aws_instance/main.tf#L40) -to ignore AMI changes. +GitLab team members can view more information in this internal handbook page on how to use FIPS AMI: +`https://internal.gitlab.com/handbook/engineering/fedramp-compliance/get-configure/#terraform---use-fips-ami` ##### Ansible: Specify the FIPS Omnibus builds @@ -167,17 +126,10 @@ The standard Omnibus GitLab releases build their own OpenSSL library, which is not FIPS-validated. However, we have nightly builds that create Omnibus packages that link against the operating system's OpenSSL library. To use this package, update the `gitlab_edition` and `gitlab_repo_script_url` fields in the Ansible -`vars.yml`. For example, you might modify -`gitlab-environment-toolkit/ansible/environments/gitlab-10k/inventory/vars.yml` -in this way: +`vars.yml`. -```yaml -all: - vars: - ... - gitlab_repo_script_url: "https://packages.gitlab.com/install/repositories/gitlab/gitlab-fips/script.deb.sh" - gitlab_edition: "gitlab-fips" -``` +GitLab team members can view more information in this internal handbook page on Ansible (AWS): +`https://internal.gitlab.com/handbook/engineering/fedramp-compliance/get-configure/#ansible-aws` #### Cloud Native Hybrid @@ -231,39 +183,16 @@ be different. Building a RHEL-based system with FIPS enabled should be possible, but there is [an outstanding issue preventing the Packer build from completing](https://github.com/aws-samples/amazon-eks-custom-amis/issues/51). -##### Terraform: Use a custom EKS AMI - -Now you can set the custom EKS AMI. - -1. In `environment.tf`, add `eks_ami_id = var.eks_ami_id` so you can pass this variable to the - AWS reference architecture module. For example, in - `gitlab-environment-toolkit/terraform/environments/gitlab-10k/inventory/environment.tf`: +Because this builds a custom AMI based on a specific version of an image, you must periodically rebuild the custom AMI to keep current with the latest security patches and upgrades. - ```tf - module "gitlab_ref_arch_aws" { - source = "../../modules/gitlab_ref_arch_aws" - - prefix = var.prefix - ami_id = data.aws_ami.ubuntu_20_04_fips[0].id - eks_ami_id = var.eks_ami_id - .... - ``` - -1. In `variables.tf`, define a `eks_ami_id` with the AMI ID in the - previous step: +##### Terraform: Use a custom EKS AMI - ```tf - variable "eks_ami_id" { - default = "ami-0a25e760cd00b027e" - } - ``` +GitLab team members can view more information in this internal handbook page on how to use a custom EKS AMI: +`https://internal.gitlab.com/handbook/engineering/fedramp-compliance/get-configure/#terraform---use-a-custom-eks-ami` ##### Ansible: Use UBI images -CNG uses a Helm Chart to manage which container images to deploy. By default, GET -deploys the latest released versions that use Debian-based containers. - -To switch to UBI-based containers, edit the Ansible `vars.yml` to use custom +CNG uses a Helm Chart to manage which container images to deploy. To use UBI-based containers, edit the Ansible `vars.yml` to use custom Charts variables: ```yaml diff --git a/doc/development/gems.md b/doc/development/gems.md index 57acac7287b..c061b33b5e4 100644 --- a/doc/development/gems.md +++ b/doc/development/gems.md @@ -13,7 +13,7 @@ We extract libraries from our codebase when their functionality is highly isolated and we want to use them in other applications ourselves or we think it would benefit the wider community. -Extracting code to a gem also ensures that the gem does not contain any hidden +Extracting code to a gem also ensures that the gem does not contain any hidden dependencies on our application code. Gems should always be used when implementing functionality that can be considered isolated, @@ -36,7 +36,7 @@ You can always start by creating a new Gem [in the same repo](#in-the-same-repo) to be used by a wider community. WARNING: -To prevent malicious actors from name-squatting the extracted Gems, follow the instructions +To prevent malicious actors from name-squatting the extracted Gems, follow the instructions to [reserve a gem name](#reserve-a-gem-name). ## Advantages of using Gems @@ -81,7 +81,7 @@ and prevents complexity (coordinating changes across repos, new permissions, mul Gems stored in the same repo should be referenced in `Gemfile` with the `path:` syntax. WARNING: -To prevent malicious actors from name-squatting the extracted Gems, follow the instructions +To prevent malicious actors from name-squatting the extracted Gems, follow the instructions to [reserve a gem name](#reserve-a-gem-name). ### Create and use a new Gem @@ -236,37 +236,61 @@ The project for a new Gem should always be created in [`gitlab-org/ruby/gems` na 1. Determine a suitable name for the gem. If it's a GitLab-owned gem, prefix the gem name with `gitlab-`. For example, `gitlab-sidekiq-fetcher`. -1. Create the gem or fork as necessary. -1. Ensure the `gitlab_rubygems` group is an owner of the new gem by running: +1. Locally create the gem or fork as necessary. +1. [Publish an empty `0.0.1` version of the gem to rubygems.org](https://guides.rubygems.org/publishing/#publishing-to-rubygemsorg) to ensure the gem name is reserved. +1. Add the [`gitlab_rubygems`](https://rubygems.org/profiles/gitlab_rubygems) and [`gitlab-qa`](https://rubygems.org/profiles/gitlab-qa) users as owners of the new gem by running: ```shell gem owner <gem-name> --add gitlab_rubygems + gem owner <gem-name> --add gitlab-qa ``` -1. [Publish the gem to rubygems.org](https://guides.rubygems.org/publishing/#publishing-to-rubygemsorg) -1. Visit `https://rubygems.org/gems/<gem-name>` and verify that the gem published - successfully and `gitlab_rubygems` is also an owner. -1. Create a project in [`gitlab-org/ruby/gems` namespace](https://gitlab.com/gitlab-org/ruby/gems/). - - - To create this project: - 1. Follow the [instructions for new projects](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#creating-a-new-project). - 1. Follow the instructions for setting up a [CI/CD configuration](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#cicd-configuration). - 1. Follow the instructions for [publishing a project](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#publishing-a-project). - - See [issue #325463](https://gitlab.com/gitlab-org/gitlab/-/issues/325463) - for an example. - - In some cases we may want to move a gem to its own namespace. Some - examples might be that it will naturally have more than one project - (say, something that has plugins as separate libraries), or that we - expect users outside GitLab to be maintainers on this project as - well as GitLab team members. - - The latter situation (maintainers from outside GitLab) could also - apply if someone who currently works at GitLab wants to maintain - the gem beyond their time working at GitLab. - -When publishing a gem to RubyGems.org, also note the section on -[gem owners](https://about.gitlab.com/handbook/developer-onboarding/#ruby-gems) -in the handbook. +1. Optional. Add some or all of the following users as co-owners: + - [Marin Jankovski](https://rubygems.org/profiles/marinjankovski) + - [Rémy Coutable](https://rubygems.org/profiles/rymai) + - [Stan Hu](https://rubygems.org/profiles/stanhu) +1. Optional. Add any other relevant developers as co-owners. +1. Visit `https://rubygems.org/gems/<gem-name>` and verify that the gem was published + successfully and `gitlab_rubygems` & `gitlab-qa` are also owners. +1. Create a project in the [`gitlab-org/ruby/gems` group](https://gitlab.com/gitlab-org/ruby/gems/). To create this project: + 1. Follow the [instructions for new projects](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#creating-a-new-project). + 1. Follow the instructions for setting up a [CI/CD configuration](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#cicd-configuration). + 1. Use the [shared CI/CD config](https://gitlab.com/gitlab-org/quality/pipeline-common/-/blob/master/ci/gem-release.yml) + to release and publish new gem versions by adding the following to their `.gitlab-ci.yml`: + + ```yaml + include: + - project: 'gitlab-org/quality/pipeline-common' + file: '/ci/gem-release.yml' + ``` + + This job will handle building and publishing the gem (it uses a `gilab-qa` Rubygems.org + API token inherited from the `gitlab-org/ruby/gems` group, in order to publish the gem + package), as well as creating the tag, release and populating its release notes by + using the + [Generate changelog data](../api/repositories.md#generate-changelog-data) + API endpoint. + + For instructions for when and how to generate a changelog entry file, see the + dedicated [Changelog entries](changelog.md) + page. + [To be consistent with the GitLab project](changelog.md), + Gem projects could also define a changelog YAML configuration file at + `.gitlab/changelog_config.yml` with the same content + as [in the `gitlab-styles` gem](https://gitlab.com/gitlab-org/ruby/gems/gitlab-styles/-/blob/master/.gitlab/changelog_config.yml). + 1. To ease the release process, you could also create a `.gitlab/merge_request_templates/Release.md` MR template with the same content + as [in the `gitlab-styles` gem](https://gitlab.com/gitlab-org/ruby/gems/gitlab-styles/-/raw/master/.gitlab/merge_request_templates/Release.md) + (make sure to replace `gitlab-styles` with the actual gem name). + 1. Follow the instructions for [publishing a project](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#publishing-a-project). + +Notes: In some cases we may want to move a gem to its own namespace. Some +examples might be that it will naturally have more than one project +(say, something that has plugins as separate libraries), or that we +expect users outside GitLab to be maintainers on this project as +well as GitLab team members. +The latter situation (maintainers from outside GitLab) could also +apply if someone who currently works at GitLab wants to maintain +the gem beyond their time working at GitLab. ## The `vendor/gems/` diff --git a/doc/development/geo.md b/doc/development/geo.md index a39f97f1241..ceaa2dbb056 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -48,8 +48,7 @@ for new events and creates background jobs for each specific event type. For example when a repository is updated, the Geo **primary** site creates a Geo event with an associated repository updated event. The Geo Log Cursor daemon picks the event up and schedules a `Geo::ProjectSyncWorker` job which -uses the `Geo::RepositorySyncService` to update the repository -and `Geo::WikiSyncService` classes to update the wiki. +uses the `Geo::RepositorySyncService` to update the repository. The Geo Log Cursor daemon can operate in High Availability mode automatically. The daemon tries to acquire a lock from time to time and once acquired, it @@ -680,7 +679,7 @@ on, check out our [self-service framework](geo/framework.md). After triggering a successful [e2e:package-and-test-ee](testing_guide/end_to_end/index.md#using-the-package-and-test-job) pipeline, you can manually trigger a job named `GET:Geo`: -1. In the [GitLab project](https://gitlab.com/gitlab-org/gitlab), select the **Pipelines** tab of a merge request. +1. In the [GitLab project](https://gitlab.com/gitlab-org/gitlab), select the **Pipelines** tab of a merge request. 1. Select the `Stage: qa` stage on the latest pipeline to expand and list all the related jobs. 1. Select `trigger-omnibus` to view the [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) pipeline corresponding to the merge request. 1. The `GET:Geo` job can be found and triggered under the `trigger-qa` stage. diff --git a/doc/development/geo/proxying.md b/doc/development/geo/proxying.md index f3136890788..45c60bc370e 100644 --- a/doc/development/geo/proxying.md +++ b/doc/development/geo/proxying.md @@ -88,44 +88,6 @@ Primary-->>Secondary: /group/project logged in response (session on primary crea Secondary-->>Client: proxy full response ``` -### Requests requiring a user session on the secondary - -At the moment, this flow only applies to Project Replication Details and Design Replication Details in the Geo Admin -Area. For more context, see -[View replication data on the primary site](../../administration/geo/index.md#view-replication-data-on-the-primary-site). - -```mermaid -sequenceDiagram -autoNumber -participant Client -participant Secondary -participant Primary - -Client->>Secondary: `admin/geo/replication/projects` request -opt secondary not signed in -Secondary-->>Client: 302 redirect -Client->>Secondary: /users/auth/geo/sign_in -Secondary-->>Client: 302 redirect -Client->>Secondary: /oauth/geo/auth/geo/sign_in -Secondary-->>Client: 302 redirect -Client->>Secondary: /oauth/authorize -Secondary->>Primary: proxy /oauth/authorize -opt primary not signed in -Primary-->>Secondary: 302 redirect -Secondary-->>Client: proxy 302 redirect -Client->>Secondary: /users/sign_in -Secondary->>Primary: proxy /users/sign_in -Note right of Primary: authentication happens, POST to same URL etc -end -Primary-->>Secondary: 302 redirect -Secondary-->>Client: proxy 302 redirect -Client->>Secondary: /oauth/geo/callback -Secondary-->>Client: 302 redirect -Client->>Secondary: admin/geo/replication/projects -end -Secondary-->>Client: admin/geo/replication/projects logged in response (session on both primary and secondary) -``` - ## Git pull For historical reasons, the `push_from_secondary` path is used to forward a Git pull. There is diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index 4a24279043d..d38be071f39 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -4,25 +4,21 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Working with the GitHub importer +# GitHub importer developer documentation -In GitLab 10.2 a new version of the GitHub importer was introduced. This new -importer performs its work in parallel using Sidekiq, greatly reducing the time -necessary to import GitHub projects into a GitLab instance. +The GitHub importer offers two different types of importers: -The GitHub importer offers two different types of importers: a sequential -importer and a parallel importer. The Rake task `import:github` uses the -sequential importer, and everything else uses the parallel importer. The -difference between these two importers is: +- A sequential importer. Used by the `import:github` Rake task. +- A parallel importer. Used by everything else. + +The difference between these two importers is: - The sequential importer does all the work in a single thread, so it's more suited for debugging purposes or Rake tasks. - The parallel importer uses Sidekiq. ## Prerequisites -- GitLab CE 10.2.0 or newer. -- Sidekiq workers that process the `github_importer` and - `github_importer_advance_stage` queues (this is enabled by default). +- Sidekiq workers that process the `github_importer` and `github_importer_advance_stage` queues (enabled by default). - Octokit (used for interacting with the GitHub API). ## Code structure @@ -221,14 +217,14 @@ long we're still performing work. GitHub has a rate limit of 5,000 API calls per hour. The number of requests necessary to import a project is largely dominated by the number of unique users -involved in a project (for example, issue authors). Other data such as issue pages -and comments typically only requires a few dozen requests to import. This is -because we need the Email address of users to map them to GitLab users. +involved in a project (for example, issue authors), because we need the email address of users to map +them to GitLab users. Other data such as issue pages and comments typically only requires a few dozen requests to import. -We handle this by doing the following: +We handle the rate limit by doing the following: -1. After we hit the rate limit all jobs automatically reschedule themselves - in such a way that they are not executed until the rate limit has been reset. +1. After we hit the rate limit, we either: + - Automatically reschedule jobs in such a way that they are not executed until the rate limit has been reset. + - Move onto another GitHub access token if multiple GitHub access tokens were passed to the API. 1. We cache the mapping of GitHub users to GitLab users in Redis. More information on user caching can be found below. @@ -253,7 +249,7 @@ Redis. For every user looked up we store three keys: - A Redis key mapping a GitHub Email addresses to a GitLab user ID. - A Redis key mapping a GitHub user ID to GitLab user ID. -There are two types of lookups we cache: +We cache two types of lookups: - A positive lookup, meaning we found a GitLab user ID. - A negative lookup, meaning we didn't find a GitLab user ID. Caching this diff --git a/doc/development/gitlab_shell/index.md b/doc/development/gitlab_shell/index.md index ef034761a1f..2cdfb68f84d 100644 --- a/doc/development/gitlab_shell/index.md +++ b/doc/development/gitlab_shell/index.md @@ -28,6 +28,15 @@ and support: - The current stable version. - The previous two major versions. +### Versions + +There are two version files relevent to GitLab Shell: + +- [Stable version](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/main/VERSION) +- [Version deployed in GitLab SaaS](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_SHELL_VERSION) + +GitLab team members can also monitor the `#announcements` internal Slack channel. + ## How GitLab Shell works When you access the GitLab server over SSH, GitLab Shell then: diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index e51542649bb..7648e84f5e8 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -146,8 +146,7 @@ Go GitLab linter plugins are maintained in the [`gitlab-org/language-tools/go/li Dependencies should be kept to the minimum. The introduction of a new dependency should be argued in the merge request, as per our [Approval Guidelines](../code_review.md#approval-guidelines). -Both [License Scanning](../../user/compliance/license_compliance/index.md) -and [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) +[Dependency Scanning](../../user/application_security/dependency_scanning/index.md) should be activated on all projects to ensure new dependencies security status and license compatibility. diff --git a/doc/development/graphql_guide/monitoring.md b/doc/development/graphql_guide/monitoring.md index 328cea2648e..f92963dbdee 100644 --- a/doc/development/graphql_guide/monitoring.md +++ b/doc/development/graphql_guide/monitoring.md @@ -4,12 +4,7 @@ group: Import and Integrate info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Monitoring GraphQL - -This page gives tips on how to analyze GraphQL data in our monitoring tools. -Please contribute your own tips to improve this document. - -## Kibana +# Reading GraphQL logs We use Kibana to filter GraphQL query logs. Sign in to [Kibana](https://log.gprd.gitlab.net/) with a `@gitlab.com` email address. @@ -20,7 +15,7 @@ In Kibana we can inspect two kinds of GraphQL logs: - Logs of the full request, which due to [query multiplexing](https://graphql-ruby.org/queries/multiplex.html) may have executed multiple queries. -### Logs of each GraphQL query +## Logs of each GraphQL query In a [multiplex query](https://graphql-ruby.org/queries/multiplex.html), each individual query is logged separately. We can use subcomponent filtering to inspect these logs. @@ -49,11 +44,11 @@ which already has a set of Kibana fields selected. Some relevant Kibana fields i | `json.query_analysis.duration_s` | Duration of query execution in seconds. | | `json.query_analysis.complexity` | The [complexity](../api_graphql_styleguide.md#max-complexity) score of the query. | -#### Useful filters +### Useful filters Combine the [subcomponent filter](#logs-of-each-graphql-query) with the following Kibana filters to further interrogate the query logs. -##### Queries that used a particular field +#### Queries that used a particular field Filter logs by queries that used a particular field: @@ -61,16 +56,16 @@ Filter logs by queries that used a particular field: 1. Filter: `json.query_analysis.used_fields` 1. Operator: `is` 1. Value: `Type.myField`, where `Type.myField` is the type name and field name as it - appears in [our GraphQL reference documentation](../../api/graphql/reference/index.md). + appears in [our GraphQL API resources documentation](../../api/graphql/reference/index.md). 1. Select **Refresh**. -##### Queries that used a deprecated field +#### Queries that used a deprecated field Filter logs of queries that used a particular deprecated field by following the [steps above](#queries-that-used-a-particular-field) but use the `json.graphql.used_deprecated_fields` filter instead. -### Logs of the full request +## Logs of the full request The full request logs encompass log data for all [multiplexed queries](https://graphql-ruby.org/queries/multiplex.html) in the request, as well as data from time spent outside of `GraphQLController#execute`. diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 12ef454d234..f4ace7491eb 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -169,9 +169,9 @@ If you need to translate strings in the Vue component's JavaScript, you can impo To test Vue translations, learn about [manually testing translations from the UI](#manually-test-translations-from-the-ui). -### Test files +### Test files (RSpec) -Test expectations against externalized contents should not be hard coded, +For RSpec tests, expectations against externalized contents should not be hard coded, because we may need to run the tests with non-default locale, and tests with hard coded contents will fail. @@ -194,18 +194,19 @@ click_button _('Submit review') expect(rendered).to have_content(_('Thank you for your feedback!')) ``` -This includes JavaScript tests: +### Test files (Jest) -Bad: - -```javascript -expect(findUpdateIgnoreStatusButton().text()).toBe('Ignore'); -``` +For Frontend Jest tests, expectations do not need to reference externalization methods. Externalization is mocked +in the Frontend test environment, so the expectations are deterministic across locales +([see relevant MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128531)). -Good: +Example: ```javascript -expect(findUpdateIgnoreStatusButton().text()).toBe(__('Ignore')); +// Bad. Not necessary in Frontend environment. +expect(findText()).toBe(__('Lorem ipsum dolar sit')); +// Good. +expect(findText()).toBe('Lorem ipsum dolar sit'); ``` #### Recommendations @@ -228,58 +229,29 @@ If strings are reused throughout a component, it can be useful to define these s </template> ``` -Also consider defining these strings in a `constants.js` file, especially if they need -to be shared across different modules. - -```javascript - javascripts - │ - └───alert_settings - │ │ constants.js - │ └───components - │ │ alert_settings_form.vue - - - // constants.js - - import { s__ } from '~/locale'; - - /* Integration constants */ +If we are reusing the same translated string in multiple components, it is tempting to add them to a `constants.js` file instead and import them across our components. However, there are multiple pitfalls to this approach: - export const MSG_ALERT_SETTINGS_FORM_ERROR = __('Failed to save alert settings.') +- It creates distance between the HTML template and the copy, adding an additional level of complexity while navigating our codebase. +- Copy strings are rarely, if ever, truly the same entity. The benefit of having a reusable variable is to have one easy place to go to update a value, but for copy it is quite common to have similar strings that aren't quite the same. +Another practice to avoid when exporting copy strings is to import them in specs. While it might seem like a much more efficient test (if we change the copy, the test will still pass!) it creates additional problems: - // alert_settings_form.vue +- There is a risk that the value we import is `undefined` and we might get a false-positive in our tests (even more so if we import an `i18n` object, see [export constants as primitives](../fe_guide/style/javascript.md#export-constants-as-primitives)). +- It is harder to know what we are testing (which copy to expect). +- There is a higher risk of typos being missed because we are not re-writing the assertion, but assuming that the value of our constant is the correct one. +- The benefit of this approach is minor. Updating the copy in our component and not updating specs is not a big enough benefit to outweigh the potential issues. - import { - MSG_ALERT_SETTINGS_FORM_ERROR, - } from '../constants'; - - <script> - export default { - MSG_ALERT_SETTINGS_FROM_ERROR, - } - </script> - - <template> - <gl-alert v-if="showAlert"> - {{ $options.MSG_ALERT_SETTINGS_FORM_ERROR }} - </gl-alert> - </template> -``` - -Using either `constants` or `$options.i18n` allows us to reference messages directly in specs: +As an example: ```javascript import { MSG_ALERT_SETTINGS_FORM_ERROR } from 'path/to/constants.js'; -// okay -expect(wrapper.text()).toEqual('this test will fail just from button text changing!'); - -// better -expect(wrapper.text()).toEqual(MyComponent.i18n.buttonLabel); -// also better -expect(wrapper.text()).toEqual(MSG_ALERT_SETTINGS_FORM_ERROR); +// Bad. What is the actual text for `MSG_ALERT_SETTINGS_FORM_ERROR`? If `wrapper.text()` returns undefined, the test may still pass with the wrong values! +expect(wrapper.text()).toBe(MSG_ALERT_SETTINGS_FORM_ERROR); +// Very bad. Same problem as above and we are going through the vm property! +expect(wrapper.text()).toBe(MyComponent.vm.i18n.buttonLabel); +// Good. What we are expecting is very clear and there can be no surprises. +expect(wrapper.text()).toBe('There was an error: Please refresh and hope for the best!'); ``` ### Dynamic translations diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 74dba53183c..cf50e417278 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -129,6 +129,8 @@ are very appreciative of the work done by translators and proofreaders! - Spanish - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) - David Elizondo - [GitLab](https://gitlab.com/daelmo), [Crowdin](https://crowdin.com/profile/daelmo) + - Pablo Reyes - [GitLab](https://gitlab.com/pabloryst9n), [Crowdin](https://crowdin.com/profile/pabloryst9n) + - Swedish - Johannes Nilsson - [GitLab](https://gitlab.com/nlssn), [Crowdin](https://crowdin.com/profile/nlssn) - Turkish diff --git a/doc/development/identity_verification.md b/doc/development/identity_verification.md index a0593905879..6895049958d 100644 --- a/doc/development/identity_verification.md +++ b/doc/development/identity_verification.md @@ -53,11 +53,35 @@ On rows where `json.event` is `Failed Attempt`, you can find valuable debugging | Reason | Description | |---------|-------------| - | `invalid_phone_number` | Either there was a typo in the phone number, or the user used a VOIP number. GitLab does not allow users to sign up with non-mobile phone numbers. | +| `invalid_phone_number` | Either there was a typo in the phone number, or the user used a VOIP number. GitLab does not allow users to sign up with non-mobile phone numbers. | | `invalid_code` | The user entered an incorrect verification code. | | `rate_limited` | The user had 10 or more failed attempts, so they were rate-limited for one hour. | | `related_to_banned_user` | The user tried a phone number already related to a banned user. | +#### View Telesign SMS status update logs + +To view logs of Telesign status updates for an SMS sent to a user: + +1. Get a `telesign_reference_id` value for an SMS sent to a specific user: + + ```plaintext + json.message: "IdentityVerification::Phone" AND json.username:<username>` + ``` + +1. Search for status update logs associated with `telesign_reference_id` value: + + ```plaintext + json.message: "IdentityVerification::Phone" AND json.event: "Telesign transaction status update" AND json.telesign_reference_id:replace_ref_id_value_here` + ``` + +Status update logs include the following fields: + +| Field | Description | +|---------|-------------| +| `telesign_status` | Delivery status of the SMS. See the [Telesign documentation](https://developer.telesign.com/enterprise/reference/smscallbacks#status-codes) for possible status codes and their descriptions. | +| `telesign_status_updated_on` | A timestamp indicating when the SMS delivery status was last updated. | +| `telesign_errors` | Errors that occurred during delivery. See the [Telesign documentation](https://developer.telesign.com/enterprise/reference/smscallbacks#status-codes) for possible error codes and their descriptions. | + ### View logs associated to a user and credit card verification To view logs associated to the [credit card stage](../security/identity_verification.md#credit-card-verification) for a user: diff --git a/doc/development/img/architecture.png b/doc/development/img/architecture.png Binary files differnew file mode 100644 index 00000000000..e63b4ba45d1 --- /dev/null +++ b/doc/development/img/architecture.png diff --git a/doc/development/img/each_batch_users_table_v13_7.png b/doc/development/img/each_batch_users_table_v13_7.png Binary files differdeleted file mode 100644 index 2e8b3fdff80..00000000000 --- a/doc/development/img/each_batch_users_table_v13_7.png +++ /dev/null diff --git a/doc/development/import_export.md b/doc/development/import_export.md index 9a7944ae308..11bddf190fa 100644 --- a/doc/development/import_export.md +++ b/doc/development/import_export.md @@ -12,10 +12,11 @@ General development guidelines and tips for the [Import/Export feature](../user/ ## Security -The Import/Export feature is constantly updated (adding new things to export), however -the code hasn't been refactored in a long time. We should perform a code audit (see -[confidential issue](../user/project/issues/confidential_issues.md) `https://gitlab.com/gitlab-org/gitlab/-/issues/20720`). +The Import/Export feature is constantly updated (adding new things to export). However, +the code hasn't been refactored in a long time. We should perform a code audit to make sure its dynamic nature does not increase the number of security concerns. +GitLab team members can view more information in this confidential issue: +`https://gitlab.com/gitlab-org/gitlab/-/issues/20720`. ### Security in the code diff --git a/doc/development/integrations/index.md b/doc/development/integrations/index.md index bd672c86b28..dd73256ce11 100644 --- a/doc/development/integrations/index.md +++ b/doc/development/integrations/index.md @@ -123,6 +123,23 @@ module Integrations end ``` +### Security enhancement features + +#### Masking channel values + +Integrations that [inherit from `Integrations::BaseChatNotification`](#define-the-integration) can hide the +values of their channel input fields. Integrations should hide these values whenever the +fields contain sensitive information such as auth tokens. + +By default, `#mask_configurable_channels?` returns `false`. To mask the channel values, override the `#mask_configurable_channels?` method in the integration to return `true`: + +```ruby +override :mask_configurable_channels? +def mask_configurable_channels? + true +end +``` + ## Define configuration test Optionally, you can define a configuration test of an integration's settings. The test is executed from the integration form's **Test** button, and results are returned to the user. @@ -156,7 +173,7 @@ module Integrations end ``` -### Customize the frontend form +## Customize the frontend form The frontend form is generated dynamically based on metadata defined in the model. @@ -181,27 +198,28 @@ This method should return an array of hashes for each field, where the keys can | `placeholder:` | string | false | | A placeholder for the form field. | `help:` | string | false | | A help text that displays below the form field. | `api_only:` | boolean | false | `false` | Specify if the field should only be available through the API, and excluded from the frontend form. +| `if:` | boolean or lambda | false | `true` | Specify if the field should be available. The value can be a boolean or a lambda. -#### Additional keys for `type: 'checkbox'` +### Additional keys for `type: 'checkbox'` | Key | Type | Required | Default | Description |:------------------|:-------|:---------|:------------------|:-- | `checkbox_label:` | string | false | Value of `title:` | A custom label that displays next to the checkbox. -#### Additional keys for `type: 'select'` +### Additional keys for `type: 'select'` | Key | Type | Required | Default | Description |:-----------|:------|:---------|:--------|:-- | `choices:` | array | true | | A nested array of `[label, value]` tuples. -#### Additional keys for `type: 'password'` +### Additional keys for `type: 'password'` | Key | Type | Required | Default | Description |:----------------------------|:-------|:---------|:------------------|:-- | `non_empty_password_title:` | string | false | Value of `title:` | An alternative label that displays when a value is already stored. | `non_empty_password_help:` | string | false | Value of `help:` | An alternative help text that displays when a value is already stored. -#### Frontend form examples +### Frontend form examples This example defines a required `url` field, and optional `username` and `password` fields: @@ -236,7 +254,7 @@ module Integrations end ``` -### Expose the integration in the REST API +## Expose the integration in the REST API To expose the integration in the [REST API](../../api/integrations.md): diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index b8815131852..c7fbb73ca36 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -66,6 +66,12 @@ To avoid external dependencies like Gitpod and a Jira Cloud instance, use the [J 1. Clone the [**Jira-connect-test-tool**](https://gitlab.com/gitlab-org/manage/integrations/jira-connect-test-tool) `git clone git@gitlab.com:gitlab-org/manage/integrations/jira-connect-test-tool.git`. 1. Start the app `bundle exec rackup`. (The app requires your GDK GitLab to be available on `http://127.0.0.1:3000`.). 1. Open `config/gitlab.yml` and uncomment the `jira_connect` config. +1. If running GDK on a domain other than `localhost`, you must add the domain to `additional_iframe_ancestors`. For example: + + ```yaml + additional_iframe_ancestors: ['localhost:*', '127.0.0.1:*', 'gdk.test:*'] + ``` + 1. Restart GDK. 1. Go to `http://127.0.0.1:3000/-/profile/personal_access_tokens`. 1. Create a new token with the `api` scope and copy the token. @@ -79,24 +85,23 @@ To avoid external dependencies like Gitpod and a Jira Cloud instance, use the [J GitLab for Jira users can authenticate with GitLab using GitLab OAuth. -WARNING: -This feature is not ready for production use. The feature flag should only be enabled in development. - The following steps describe setting up an environment to test the GitLab OAuth flow: 1. Start a Gitpod session. 1. On your GitLab instance, go to **Admin > Applications**. 1. Create a new application with the following settings: - - Name: `Jira Connect` + - Name: `GitLab for Jira` - Redirect URI: `YOUR_GITPOD_INSTANCE/-/jira_connect/oauth_callbacks` - - Scopes: `api` - Trusted: **No** - Confidential: **No** -1. Copy the Application ID. + - Scopes: `api` +1. Copy the **Application ID** value. 1. Go to **Admin > Settings > General**. 1. Expand **GitLab for Jira App**. -1. Go to [gitpod.io/variables](https://gitpod.io/variables). -1. Paste the Application ID into the **Jira Connect Application ID** field and select **Save changes**. +1. Paste the **Application ID** value into **Jira Connect Application ID**. +1. In **Jira Connect Proxy URL**, enter `YOUR_GITPOD_INSTANCE` (for example, `https://xxxx.gitpod.io`). +1. Select **Enable public key storage**. +1. Select **Save changes**. ## Troubleshooting diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 09778127050..8fda6042fcf 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -318,8 +318,8 @@ After the deprecation period for a schema version, the file is removed from GitL declare removed versions are rejected, and an error message displays on the corresponding pipeline. If a report uses a `PATCH` version that doesn't match any vendored schema version, it is validated against -the latest vendored `PATCH` version. For example, if a report version is 14.0.23 and the latest vendored -version is 14.0.6, the report is validated against version 14.0.6. +the latest vendored `PATCH` version. For example, if a report version is 15.0.23 and the latest vendored +version is 15.0.6, the report is validated against version 15.0.6. GitLab uses the [`json_schemer`](https://www.rubydoc.info/gems/json_schemer) gem to perform validation. @@ -355,12 +355,12 @@ puts(schema_validation_errors) ``` 1. Download the appropriate schema that matches your report type and declared version. For - example, you can find version `14.0.6` of the `container_scanning` report schema at - `https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/raw/v14.0.6/dist/container-scanning-report-format.json?inline=false`. + example, you can find version `15.0.6` of the `container_scanning` report schema at + `https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/raw/v15.0.6/dist/container-scanning-report-format.json?inline=false`. 1. Save the Ruby script above in a file, for example, `validate.rb`. 1. Run the script, passing the schema and report file names as arguments in order. For example: - 1. Using your local Ruby interpreter: `ruby validate.rb container-scanning-format_14-0-6.json gl-container-scanning-report.json`. - 1. Using Docker: `docker run -it --rm -v $(pwd):/ci ruby:3-slim ruby /ci/validate.rb /ci/container-scanning-format_14-0-6.json /ci/gl-container-scanning-report.json` + 1. Using your local Ruby interpreter: `ruby validate.rb container-scanning-format_15-0-6.json gl-container-scanning-report.json`. + 1. Using Docker: `docker run -it --rm -v $(pwd):/ci ruby:3 ruby /ci/validate.rb /ci/container-scanning-format_15-0-6.json /ci/gl-container-scanning-report.json` 1. Validation errors are shown on the screen. You must resolve these errors before GitLab can ingest your report. ### Report Fields @@ -577,7 +577,7 @@ All other attributes are optional. ##### SAST -The `location` of a SAST vulnerability must have a `file` that gives the path of the affected file and +The `location` of a SAST vulnerability must have a `file` that gives the path of the affected file and a `start_line` field with the affected line number. It may also have an `end_line`, a `class`, and a `method`. diff --git a/doc/development/internal_analytics/index.md b/doc/development/internal_analytics/index.md index 8abea4c2b2f..c7cf907ca92 100644 --- a/doc/development/internal_analytics/index.md +++ b/doc/development/internal_analytics/index.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Internal analytics -Learn how to implement internal analytics using: +Learn how to instrument your features on GitLab using: - [Service Ping](service_ping/index.md) - [Snowplow](snowplow/index.md) diff --git a/doc/development/internal_analytics/internal_event_tracking/architecture.md b/doc/development/internal_analytics/internal_event_tracking/architecture.md new file mode 100644 index 00000000000..de5672a4895 --- /dev/null +++ b/doc/development/internal_analytics/internal_event_tracking/architecture.md @@ -0,0 +1,11 @@ +--- +stage: Analytics +group: Analytics Instrumentation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Internal event tracking architecture + +This page is under construction. It serves as placeholder for the following information. + +- Detailed architecture and other technical details diff --git a/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md b/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md new file mode 100644 index 00000000000..7e4222ead2e --- /dev/null +++ b/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md @@ -0,0 +1,11 @@ +--- +stage: Analytics +group: Analytics Instrumentation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Internal event tracking definition guide + +This page is under construction. It serves as placeholder for the following information. + +- Explanation of all parts of an event definition diff --git a/doc/development/internal_analytics/internal_event_tracking/index.md b/doc/development/internal_analytics/internal_event_tracking/index.md new file mode 100644 index 00000000000..73e9e2d1a4c --- /dev/null +++ b/doc/development/internal_analytics/internal_event_tracking/index.md @@ -0,0 +1,17 @@ +--- +stage: Analytics +group: Analytics Instrumentation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Internal Event Tracking + +This page provides detailed guidelines on using the Internal Event Tracking system to instrument features on GitLab. + +This page is a work in progress. For any questions or clarifications, reach out to us in the Slack channel [#g_analyze_analytics_instrumentation](https://gitlab.slack.com/archives/CL3A7GFPF). + +- [Introduction to internal event tracking](introduction.md#internal-event-tracking) +- [Quick start guide](quick_start.md#quick-start-for-internal-event-tracking) +- [Event definition guide](event_definition_guide.md#internal-event-tracking-definition-guide) +- [Metrics dictionary guide](../service_ping/metrics_dictionary.md#metrics-dictionary-guide) +- [Architecture](architecture.md#internal-event-tracking-architecture) diff --git a/doc/development/internal_analytics/internal_event_tracking/introduction.md b/doc/development/internal_analytics/internal_event_tracking/introduction.md new file mode 100644 index 00000000000..ebb3caa198a --- /dev/null +++ b/doc/development/internal_analytics/internal_event_tracking/introduction.md @@ -0,0 +1,13 @@ +--- +stage: Analytics +group: Analytics Instrumentation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Internal event tracking + +This page is under construction. It serves as placeholder for the following information: + +- High level introduction +- Difference between Events and Metrics +- Basic overview of the architecture diff --git a/doc/development/internal_analytics/internal_event_tracking/quick_start.md b/doc/development/internal_analytics/internal_event_tracking/quick_start.md new file mode 100644 index 00000000000..84926657a3b --- /dev/null +++ b/doc/development/internal_analytics/internal_event_tracking/quick_start.md @@ -0,0 +1,113 @@ +--- +stage: Analytics +group: Analytics Instrumentation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Quick start for internal event tracking + +In an effort to provide a more efficient, scalable, and unified tracking API, GitLab is deprecating existing RedisHLL and Snowplow tracking. Instead, we're implementing a new `track_event` method. +With this approach, we can both update RedisHLL counters and send Snowplow events simultaneously, streamlining the tracking process. + +## Create and trigger events + +### Backend tracking + +To trigger an event, call the `Gitlab::InternalEvents.track_event` method with the desired arguments: + +```ruby +Gitlab::InternalEvents.track_event( + "i_code_review_user_apply_suggestion", + user_id: user_id, + namespace_id: namespace_id, + project_id: project_id + ) +``` + +This method automatically increments all RedisHLL metrics relating to the event `i_code_review_user_apply_suggestion`, and sends a corresponding Snowplow event with all named arguments and standard context (SaaS only). + +### Frontend tracking + +#### Vue components + +In Vue components, tracking can be done with [Vue mixin](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/tracking/internal_events.js#L29). + +To implement Vue component tracking: + +1. Import the `InternalEvents` library and call the `mixin` method: + + ```javascript + import { InternalEvents } from '~/tracking'; + const trackingMixin = InternalEvents.mixin(); + ``` + +1. Use the mixin in the component: + + ```javascript + export default { + mixins: [trackingMixin], + + data() { + return { + expanded: false, + }; + }, + }; + ``` + +1. Call the `track_event` method. Tracking options can be passed as the second parameter: + + ```javascript + this.track_event('i_code_review_user_apply_suggestion'); + ``` + + Or use the `track_event` method in the template: + + ```html + <template> + <div> + <button data-testid="toggle" @click="toggle">Toggle</button> + + <div v-if="expanded"> + <p>Hello world!</p> + <button @click="track_event('i_code_review_user_apply_suggestion')">Track another event</button> + </div> + </div> + </template> + ``` + +#### Raw JavaScript + +For tracking events directly from arbitrary frontend JavaScript code, a module for raw JavaScript is provided. This can be used outside of a component context where the Mixin cannot be utilized. + +```javascript +import { InternalEvents } from '~/tracking'; +InternalEvents.track_event('i_code_review_user_apply_suggestion'); +``` + +#### Data-track attribute + +This attribute ensures that if we want to track GitLab internal events for a button, we do not need to write JavaScript code on Click handler. Instead, we can just add a data-event-tracking attribute with event value and it should work. This can also be used with haml views. + +```html + <gl-button + data-event-tracking="i_analytics_dev_ops_adoption" + > + Click Me + </gl-button> +``` + +For Haml + +```haml += render Pajamas::ButtonComponent.new(button_options: { class: 'js-settings-toggle', data: { event_tracking: 'action' }}) do +``` + +#### Internal events on render + +Sometimes we want to send internal events when the component is rendered or loaded. In these cases, we can add the `data-event-tracking-load="true"` attribute: + +```haml += render Pajamas::ButtonComponent.new(button_options: { data: { event_tracking_load: 'true', event_tracking: 'i_devops' } }) do + = _("New project") +``` diff --git a/doc/development/internal_analytics/internal_events/index.md b/doc/development/internal_analytics/internal_events/index.md index cbc7b73a282..d987317a2b0 100644 --- a/doc/development/internal_analytics/internal_events/index.md +++ b/doc/development/internal_analytics/internal_events/index.md @@ -1,98 +1,11 @@ --- -stage: Analytics -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../internal_event_tracking/quick_start.md' +remove_date: '2023-10-27' --- -# Internal Events development guidelines +This document was moved to [another location](../internal_event_tracking/quick_start.md). -In an effort to provide a more efficient, scalable, and unified tracking API, GitLab is deprecating existing RedisHLL and Snowplow tracking. Instead, we're implementing a new `track_event` method. -With this approach, we can both update RedisHLL counters and send Snowplow events simultaneously, streamlining the tracking process. - -## Create and trigger events - -### Backend tracking - -To trigger an event, call the `Gitlab::InternalEvents.track_event` method with the desired arguments: - -```ruby -Gitlab::InternalEvents.track_event( - "i_code_review_user_apply_suggestion", - user_id: user_id, - namespace_id: namespace_id, - project_id: project_id - ) -``` - -This method automatically increments all RedisHLL metrics relating to the event `i_code_review_user_apply_suggestion`, and sends a corresponding Snowplow event with all named arguments and standard context (SaaS only). - -### Frontend tracking - -#### Vue components - -In Vue components, tracking can be done with [Vue mixin](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/tracking/internal_events.js#L29). - -To implement Vue component tracking: - -1. Import the `InternalEvents` library and call the `mixin` method: - - ```javascript - import { InternalEvents } from '~/tracking'; - const trackingMixin = InternalEvents.mixin(); - ``` - -1. Use the mixin in the component: - - ```javascript - export default { - mixins: [trackingMixin], - - data() { - return { - expanded: false, - }; - }, - }; - ``` - -1. Call the `track_event` method. Tracking options can be passed as the second parameter: - - ```javascript - this.track_event('i_code_review_user_apply_suggestion'); - ``` - - Or use the `track_event` method in the template: - - ```html - <template> - <div> - <button data-testid="toggle" @click="toggle">Toggle</button> - - <div v-if="expanded"> - <p>Hello world!</p> - <button @click="track_event('i_code_review_user_apply_suggestion')">Track another event</button> - </div> - </div> - </template> - ``` - -#### Raw JavaScript - -For tracking events directly from arbitrary frontend JavaScript code, a module for raw JavaScript is provided. This can be used outside of a component context where the Mixin cannot be utilized. - -```javascript -import { InternalEvents } from '~/tracking'; -InternalEvents.track_event('i_code_review_user_apply_suggestion'); -``` - -#### Data-track attribute - -This attribute ensures that if we want to track GitLab internal events for a button, we do not need to write JavaScript code on Click handler. Instead, we can just add a data-event-tracking attribute with event value and it should work. This can also be used with haml views. - -```html - <gl-button - data-event-tracking="i_analytics_dev_ops_adoption" - > - Click Me - </gl-button> -``` +<!-- This redirect file can be deleted after <2023-10-27>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/service_ping/implement.md b/doc/development/internal_analytics/service_ping/implement.md index 3b8243377a3..9dbfa02854d 100644 --- a/doc/development/internal_analytics/service_ping/implement.md +++ b/doc/development/internal_analytics/service_ping/implement.md @@ -21,7 +21,6 @@ To implement a new metric in Service Ping, follow these steps: 1. [Generate the SQL query](#generate-the-sql-query) 1. [Optimize queries with Database Lab](#optimize-queries-with-database-lab) 1. [Add the metric definition to the Metrics Dictionary](#add-the-metric-definition) -1. [Add the metric to the Versions Application](#add-the-metric-to-the-versions-application) 1. [Create a merge request](#create-a-merge-request) 1. [Verify your metric](#verify-your-metric) 1. [Set up and test Service Ping locally](#set-up-and-test-service-ping-locally) @@ -324,25 +323,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd/) and [P ##### Add new events -1. Define events in [`known_events`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/). - - Example event: - - ```yaml - - name: users_creating_epics - aggregation: weekly - ``` - - Keys: - - - `name`: unique event name. - - Name format for Redis HLL events `{hll_counters}_<name>` - - Example names: `users_creating_epics`, `users_triggering_security_scans`. - - - `aggregation`: may be set to a `:daily` or `:weekly` key. Defines how counting data is stored in Redis. - Aggregation on a `daily` basis does not pull more fine grained data. +1. Add an event to the required metric ([see example](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20230210200054_i_ee_code_review_merge_request_widget_license_compliance_expand_weekly.yml#L17-17)) or create a metric. 1. Use one of the following methods to track the event: @@ -447,7 +428,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd/) and [P - Using the JavaScript/Vue API helper, which calls the [`UsageData` API](#usagedata-api). - Example for an existing event already defined in [known events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/): + Example for an existing event already defined in [metric fields](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20220407125907_p_ci_templates_themekit_monthly.yml#L17-17): ```javascript import api from '~/api'; @@ -485,7 +466,6 @@ Next, get the unique events for the current week. We have the following recommendations for [adding new events](#add-new-events): -- Event aggregation: weekly. - When adding new metrics, use a [feature flag](../../../operations/feature_flags.md) to control the impact. It's recommended to disable the new feature flag by default (set `default_enabled: false`). - Events can be triggered using the `UsageData` API, which helps when there are > 10 events per change @@ -501,7 +481,7 @@ We can disable tracking completely by using the global flag: ##### Known events are added automatically in Service Data payload -Service Ping adds all events [`known_events/*.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events) to Service Data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com/-/blob/main/db/schema.rb#L213). +Service Ping adds all events to Service Data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com/-/blob/main/db/schema.rb#L213). For each event we add metrics for the weekly and monthly time frames, and totals for each where applicable: - `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#add-new-events) events and data for the last complete week for weekly [aggregation](#add-new-events) events. @@ -540,8 +520,6 @@ Example: # Redis Counters redis_usage_data(Gitlab::UsageDataCounters::WikiPageCounter) -# Define events in common.yml https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/common.yml - # Tracking events Gitlab::UsageDataCounters::HLLRedisCounter.track_event('users_expanding_vulnerabilities', values: visitor_id) @@ -678,10 +656,6 @@ For more details, see the [database review guide](../../database_review.md#prepa See the [Metrics Dictionary guide](metrics_dictionary.md) for more information. -## Add the metric to the Versions Application - -Check if the new metric must be added to the Versions Application. See the `usage_data` [schema](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com/-/blob/main/db/schema.rb#L152) and Service Data [parameters accepted](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com/-/blob/main/app/services/usage_ping.rb). Any metrics added under the `counts` key are saved in the `stats` column. - ## Create a merge request Create a merge request for the new Service Ping metric, and do the following: @@ -809,13 +783,7 @@ create metric YAML definition file following [Aggregated metric instrumentation > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45979) in GitLab 13.6. -To declare the aggregate of events collected with [Redis HLL Counters](#redis-hll-counters), -you must fulfill the following requirements: - -1. All events listed at `events` attribute must come from - [`known_events/*.yml`](#known-events-are-added-automatically-in-service-data-payload) files. -1. All events listed at `events` attribute must have the same `aggregation` attribute. -1. `time_frame` does not include `all` value, which is unavailable for Redis sourced aggregated metrics. +To declare the aggregate of events collected with [Redis HLL Counters](#redis-hll-counters), make sure `time_frame` does not include the `all` value, which is unavailable for Redis-sourced aggregated metrics. While it is possible to aggregate EE-only events together with events that occur in all GitLab editions, it's important to remember that doing so may produce high variance between data collected from EE and CE GitLab instances. diff --git a/doc/development/internal_analytics/service_ping/metrics_dictionary.md b/doc/development/internal_analytics/service_ping/metrics_dictionary.md index f56955ed422..8103db5113f 100644 --- a/doc/development/internal_analytics/service_ping/metrics_dictionary.md +++ b/doc/development/internal_analytics/service_ping/metrics_dictionary.md @@ -32,7 +32,6 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | Field | Required | Additional information | |---------------------|----------|----------------------------------------------------------------| | `key_path` | yes | JSON key path for the metric, location in Service Ping payload. | -| `name` (deprecated) | no | Metric name suggestion. Does not have any impact on the Service Ping payload, only serves as documentation. | | `description` | yes | | | `product_section` | yes | The [section](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/sections.yml). | | `product_stage` | yes | The [stage](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) for the metric. | @@ -71,42 +70,6 @@ NOTE: We can't control what the metric's `key_path` is, because some of them are generated dynamically in `usage_data.rb`. For example, see [Redis HLL metrics](implement.md#redis-hll-counters). -### Metric name (deprecated) - -WARNING: -This feature was deprecated in GitLab 16.1 -and is planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/411602) in 16.2. - -To improve metric discoverability by a wider audience, each metric with -instrumentation added at an appointed `key_path` receives a `name` attribute -filled with the name suggestion, corresponding to the metric `data_source` and instrumentation. -Metric name suggestions can contain two types of elements: - -1. **User input prompts**: enclosed by angle brackets (`< >`), these pieces should be replaced or - removed when you create a metrics YAML file. -1. **Fixed suggestion**: plaintext parts generated according to well-defined algorithms. - They are based on underlying instrumentation, and must not be changed. - -For a metric name to be valid, it must not include any prompt, and fixed suggestions -must not be changed. - -#### Generate a metric name suggestion (deprecated) - -WARNING: -This feature was deprecated in GitLab 16.1 -and is planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/411602) in 16.2. - -The metric YAML generator can suggest a metric name for you. -To generate a metric name suggestion, first instrument the metric at the provided `key_path`. -Then, generate the metric's YAML definition and -return to the instrumentation and update it. - -1. Add the metric instrumentation class to `lib/gitlab/usage/metrics/instrumentations/`. -1. Add the metric logic in the instrumentation class. -1. Run the [metrics YAML generator](metrics_dictionary.md#create-a-new-metric-definition). -1. Use the metric name suggestion to select a suitable metric name. -1. Update the metric's YAML definition with the correct `key_path`. - ### Metric statuses Metric definitions can have one of the following statuses: @@ -130,20 +93,16 @@ which has a related schema in `/config/metrics/objects_schemas/topology_schema.j ### Metric `time_frame` A metric's time frame is calculated based on the `time_frame` field and the `data_source` of the metric. -For `redis_hll` metrics, the type of aggregation is also taken into consideration. In this context, the term "aggregation" refers to [chosen events data storage interval](implement.md#add-new-events), and is **NOT** related to the Aggregated Metrics feature. -For more information about the aggregation type of each feature, see the [`common.yml` file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/common.yml). Weeks run from Monday to Sunday. - -| data_source | time_frame | aggregation | Description | -|------------------------|------------|----------------|-------------------------------------------------| -| any | `none` | not applicable | A type of data that's not tracked over time, such as settings and configuration information | -| `database` | `all` | not applicable | The whole time the metric has been active (all-time interval) | -| `database` | `7d` | not applicable | 9 days ago to 2 days ago | -| `database` | `28d` | not applicable | 30 days ago to 2 days ago | -| `redis` | `all` | not applicable | The whole time the metric has been active (all-time interval) | -| `redis_hll` | `7d` | `daily` | Most recent 7 complete days | -| `redis_hll` | `7d` | `weekly` | Most recent complete week | -| `redis_hll` | `28d` | `daily` | Most recent 28 complete days | -| `redis_hll` | `28d` | `weekly` | Most recent 4 complete weeks | + +| data_source | time_frame | Description | +|------------------------|------------|-------------------------------------------------| +| any | `none` | A type of data that's not tracked over time, such as settings and configuration information | +| `database` | `all` | The whole time the metric has been active (all-time interval) | +| `database` | `7d` | 9 days ago to 2 days ago | +| `database` | `28d` | 30 days ago to 2 days ago | +| `redis` | `all` | The whole time the metric has been active (all-time interval) | +| `redis_hll` | `7d` | Most recent complete week | +| `redis_hll` | `28d` | Most recent 4 complete weeks | ### Data category @@ -157,69 +116,6 @@ We use the following categories to classify a metric: An aggregate metric is a metric that is the sum of two or more child metrics. Service Ping uses the data category of the aggregate metric to determine whether or not the data is included in the reported Service Ping payload. -### Metric name suggestion examples (deprecated) - -WARNING: -This feature was deprecated in GitLab 16.1 -and is planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/411602) in 16.2. - -#### Metric with `data_source: database` - -For a metric instrumented with SQL: - -```sql -SELECT COUNT(DISTINCT user_id) FROM clusters WHERE clusters.management_project_id IS NOT NULL -``` - -- **Suggested name**: `count_distinct_user_id_from_<adjective describing: '(clusters.management_project_id IS NOT NULL)'>_clusters` -- **Prompt**: `<adjective describing: '(clusters.management_project_id IS NOT NULL)'>` - should be replaced with an adjective that best represents filter conditions, such as `project_management` -- **Final metric name**: For example, `count_distinct_user_id_from_project_management_clusters` - -For metric instrumented with SQL: - -```sql -SELECT COUNT(DISTINCT clusters.user_id) -FROM clusters_applications_helm -INNER JOIN clusters ON clusters.id = clusters_applications_helm.cluster_id -WHERE clusters_applications_helm.status IN (3, 5) -``` - -- **Suggested name**: `count_distinct_user_id_from_<adjective describing: '(clusters_applications_helm.status IN (3, 5))'>_clusters_<with>_<adjective describing: '(clusters_applications_helm.status IN (3, 5))'>_clusters_applications_helm` -- **Prompt**: `<adjective describing: '(clusters_applications_helm.status IN (3, 5))'>` - should be replaced with an adjective that best represents filter conditions -- **Final metric name**: `count_distinct_user_id_from_clusters_with_available_clusters_applications_helm` - -In the previous example, the prompt is irrelevant, and user can remove it. The second -occurrence corresponds with the `available` scope defined in `Clusters::Concerns::ApplicationStatus`. -It can be used as the right adjective to replace prompt. - -The `<with>` represents a suggested conjunction for the suggested name of the joined relation. -The person documenting the metric can use it by either: - -- Removing the surrounding `<>`. -- Using a different conjunction, such as `having` or `including`. - -#### Metric with `data_source: redis` or `redis_hll` - -For metrics instrumented with a Redis-based counter, the suggested name includes -only the single prompt to be replaced by the person working with metrics YAML. - -- **Prompt**: `<please fill metric name, suggested format is: {subject}_{verb}{ing|ed}_{object} eg: users_creating_epics or merge_requests_viewed_in_single_file_mode>` -- **Final metric name**: We suggest the metric name should follow the format of - `{subject}_{verb}{ing|ed}_{object}`, such as `user_creating_epics`, `users_triggering_security_scans`, - or `merge_requests_viewed_in_single_file_mode` - -#### Metric with `data_source: prometheus` or `system` - -For metrics instrumented with Prometheus or coming from the operating system, -the suggested name includes only the single prompt by person working with metrics YAML. - -- **Prompt**: `<please fill metric name>` -- **Final metric name**: Due to the variety of cases that can apply to this kind of metric, - no naming convention exists. Each person instrumenting a metric should use their - best judgment to come up with a descriptive name. - ### Example YAML metric definition The linked [`uuid`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/license/uuid.yml) diff --git a/doc/development/internal_analytics/service_ping/metrics_lifecycle.md b/doc/development/internal_analytics/service_ping/metrics_lifecycle.md index 7aa0eaa1554..bb3d6797011 100644 --- a/doc/development/internal_analytics/service_ping/metrics_lifecycle.md +++ b/doc/development/internal_analytics/service_ping/metrics_lifecycle.md @@ -103,4 +103,3 @@ To remove a metric: 1. Remove any other records related to the metric: - The feature flag YAML file at [`config/feature_flags/*/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/feature_flags). - - The entry in the known events YAML file at [`lib/gitlab/usage_data_counters/known_events/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/usage_data_counters/known_events). diff --git a/doc/development/internal_analytics/service_ping/review_guidelines.md b/doc/development/internal_analytics/service_ping/review_guidelines.md index 31b6c3f5580..8a46de7086e 100644 --- a/doc/development/internal_analytics/service_ping/review_guidelines.md +++ b/doc/development/internal_analytics/service_ping/review_guidelines.md @@ -51,7 +51,7 @@ are regular backend changes. #### The Analytics Instrumentation **reviewer** should - Perform a first-pass review on the merge request and suggest improvements to the author. -- Check the [metrics location](metrics_dictionary.md#metric-key_path) in +- Check the [metric's location](metrics_dictionary.md#metric-key_path) in the Service Ping JSON payload. - Add the `~database` label and ask for a [database review](../../database_review.md) for metrics that are based on Database. @@ -66,6 +66,7 @@ are regular backend changes. - Check the file location. Consider the time frame, and if the file should be under `ee`. - Check the tiers. - If a metric was changed or removed: Make sure the MR author notified the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) by `@` mentioning those groups in a comment on the issue for the MR and all of these groups have acknowledged the removal. +- Make sure that the new metric is available in Service Ping payload, by running: `Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values).dig(*'key_path'.split('.'))` with `key_path` substituted by the new metric's key_path. - Metrics instrumentations - Recommend using metrics instrumentation for new metrics, [if possible](metrics_instrumentation.md#support-for-instrumentation-classes). - Approve the MR, and relabel the MR with `~"analytics instrumentation::approved"`. diff --git a/doc/development/internal_analytics/service_ping/troubleshooting.md b/doc/development/internal_analytics/service_ping/troubleshooting.md index 7f7b4099d48..8f5e94506cd 100644 --- a/doc/development/internal_analytics/service_ping/troubleshooting.md +++ b/doc/development/internal_analytics/service_ping/troubleshooting.md @@ -76,7 +76,7 @@ checking the configuration file of your GitLab instance: - `/etc/gitlab/gitlab.rb` for Linux package installations and Docker. - `charts.yaml` for GitLab Helm and cloud-native Kubernetes deployments. - - `gitlab.yml` for GitLab installations from source. + - `gitlab.yml` for self-compiled installations. To check the relevant configuration file for strings that indicate whether Service Ping is disabled, you can use `grep`: diff --git a/doc/development/internal_analytics/snowplow/troubleshooting.md b/doc/development/internal_analytics/snowplow/troubleshooting.md index 2f59543e0f4..b531c6dcd56 100644 --- a/doc/development/internal_analytics/snowplow/troubleshooting.md +++ b/doc/development/internal_analytics/snowplow/troubleshooting.md @@ -21,8 +21,8 @@ You will be alarmed via a [Sisense alert](https://app.periscopedata.com/app/gitl ### Locating the problem First you need to identify at which stage in Snowplow the data pipeline the drop is occurring. -Start at [Snowplow dashboard](https://console.aws.amazon.com/systems-manager/resource-groups/cloudwatch?dashboard=SnowPlow®ion=us-east-1#) on CloudWatch, -if you do not have access to CloudWatch you need to create an [access request issue](https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/9730) first. +Start at [Snowplow dashboard](https://console.aws.amazon.com/systems-manager/resource-groups/cloudwatch?dashboard=SnowPlow®ion=us-east-1#) on CloudWatch. +If you do not have access to CloudWatch, GitLab team members can create an access request issue, similar to this one: `https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/9730`. While on CloudWatch dashboard set time range to last 4 weeks, to get better picture of system characteristics over time. Than visit following charts: 1. `ELB New Flow Count` and `Collector Auto Scaling Group Network In/Out` - they show in order: number of connections to collectors via load balancers and data volume (in bytes) processed by collectors. If there is drop visible there, it means less events were fired from the GitLab application. Proceed to [application layer guide](#troubleshooting-gitlab-application-layer) for more details @@ -42,7 +42,7 @@ or even a result of a public holiday in some regions of the world with a larger 1. Check (via [Grafana explore tab](https://dashboards.gitlab.net/explore) ) following Prometheus counters `gitlab_snowplow_events_total`, `gitlab_snowplow_failed_events_total` and `gitlab_snowplow_successful_events_total` to see how many events were fired correctly from GitLab.com. Example query to use `sum(rate(gitlab_snowplow_successful_events_total{env="gprd"}[5m])) / sum(rate(gitlab_snowplow_events_total{env="gprd"}[5m]))` would chart rate at which number of good events rose in comparison to events sent in total. If number drops from 1 it means that problem might be in communication between GitLab and AWS collectors fleet. 1. Check [logs in Kibana](https://log.gprd.gitlab.net/app/discover#) and filter with `{ "query": { "match_phrase": { "json.message": "failed to be reported to collector at" } } }` if there are some failed events logged -We conducted an investigation into an unexpected drop in snowplow events volume. +We conducted an investigation into an unexpected drop in snowplow events volume. GitLab team members can view more information in this confidential issue: `https://gitlab.com/gitlab-org/gitlab/-/issues/335206` diff --git a/doc/development/internal_users.md b/doc/development/internal_users.md index d9f40d6f5d4..1c12e541149 100644 --- a/doc/development/internal_users.md +++ b/doc/development/internal_users.md @@ -43,7 +43,7 @@ Other examples of internal users: - [GitLab Admin Bot](https://gitlab.com/gitlab-org/gitlab/-/blob/278bc9018dd1515a10cbf15b6c6cd55cb5431407/app/models/user.rb#L950-960) - [Alert Bot](../operations/incident_management/alerts.md#trigger-actions-from-alerts) - [Ghost User](../user/profile/account/delete_account.md#associated-records) -- [Support Bot](../user/project/service_desk.md#support-bot-user) +- [Support Bot](../user/project/service_desk/index.md#support-bot-user) - Visual Review Bot - Resource access tokens, including: - [Project access tokens](../user/project/settings/project_access_tokens.md). diff --git a/doc/development/merge_request_concepts/diffs/development.md b/doc/development/merge_request_concepts/diffs/development.md index 75f961e41de..95e6fcc2170 100644 --- a/doc/development/merge_request_concepts/diffs/development.md +++ b/doc/development/merge_request_concepts/diffs/development.md @@ -64,6 +64,7 @@ contents, individual commits, and the files containing changes. external_diff_store: 1, stored_externally: nil, files_count: 9, + patch_id_sha: "d504412d5b6e6739647e752aff8e468dde093f2f", sorted: true, diff_type: "regular", verification_checksum: nil> diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 8cbf18ce9f2..65dc4de30d1 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -420,6 +420,18 @@ minimum acceptable timestamp would be 20230424000000. While the above should be considered a hard rule, it is a best practice to try to keep migration timestamps to within three weeks of the date it is anticipated that the migration will be merged upstream, regardless of how much time has elapsed since the last hard stop. +To update a migration timestamp: + +1. Migrate down the migration for the `ci` and `main` DBs: + + ```ruby + rake db:migrate:down:main VERSION=<timestamp> + rake db:migrate:down:ci VERSION=<timestamp> + ``` + +1. Delete the migration file. +1. Recreate the migration following the [migration style guide](#choose-an-appropriate-migration-type). + ## Migration helpers and versioning > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339115) in GitLab 14.3. @@ -1363,7 +1375,7 @@ end When doing so be sure to explicitly set the model's table name, so it's not derived from the class name or namespace. -Be aware of the limitations [when using models in migrations](#using-models-in-migrations-discouraged). +Be aware of the limitations [when using models in migrations](#using-application-code-in-migrations-discouraged). ### Modifying existing data @@ -1423,13 +1435,25 @@ _namespaces_ that have a `project_id`. The `path` column for these rows are renamed to their previous value followed by an integer. For example: `users` would turn into `users0` -## Using models in migrations (discouraged) - -The use of models in migrations is generally discouraged. As such models are -[contraindicated for batched background migrations](database/batched_background_migrations.md#isolation), -the model needs to be declared in the migration. - -If using a model in the migrations, you should first +## Using application code in migrations (discouraged) + +The use of application code (including models) in migrations is generally +discouraged. This is because the migrations stick around for a long time and +the application code it depends on may change and break the migration in +future. In the past some background migrations needed to use +application code in order to avoid copying hundreds of lines of code spread +across multiple files into the migration. In these rare cases it's critical to +ensure the migration has good tests so that anyone refactoring the code in +future will learn if they break the migration. Using application code is also +[discouraged for batched background migrations](database/batched_background_migrations.md#isolation) +, the model needs to be declared in the migration. + +Usually you can avoid using application code (specifically models) in a +migration by defining a class that inherits from `MigrationRecord` (see +examples below). + +If using are using a model (including defined in the migration), you should +first [clear the column cache](https://api.rubyonrails.org/classes/ActiveRecord/ModelSchema/ClassMethods.html#method-i-reset_column_information) using `reset_column_information`. diff --git a/doc/development/namespaces.md b/doc/development/namespaces.md new file mode 100644 index 00000000000..e25b0f57f08 --- /dev/null +++ b/doc/development/namespaces.md @@ -0,0 +1,302 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Namespaces + +Namespaces are containers for projects and associated resources. A `Namespace` is instantiated through its subclasses of `Group`, `ProjectNamespace`, and `UserNamespace`. + +```mermaid +graph TD + Namespace -.- Group + Namespace -.- ProjectNamespace + Namespace -.- UserNamespace +``` + +A `User` has one `UserNamespace`, and can be a member of many `Namespaces`. + +```mermaid +graph TD + Namespace -.- Group + Namespace -.- ProjectNamespace + Namespace -.- UserNamespace + + User -- has one --- UserNamespace + Namespace --- Member --- User +``` + +`Group` exists in a recursive hierarchical relationship. `Groups` have many `ProjectNamespaces` which parent one `Project`. + +```mermaid +graph TD + Group -- has many --- ProjectNamespace -- has one --- Project + Group -- has many --- Group +``` + +## Querying namespaces + +There is a set of methods provided to query the namespace hierarchy. The methods produce standard Rails `ActiveRecord::Relation` objects. +The methods can be split into two similar halves. One set of methods operate on a Namespace object, while the other set operate as composable Namespace scopes. + +By their nature, the object methods will operate within a single `Namespace` hierarchy, while the scopes can span hierarchies. + +The following is a non-exhaustive list of methods to query `Namespace` hierarchies. + +### Root namespaces + +The root is the top most `Namespace` in the hierarchy. A root has a `nil` `parent_id`. + +```mermaid +graph TD + classDef active fill:#f00,color:#fff + classDef sel fill:#00f,color:#fff + + A --- A.A --- A.A.A + A.A --- A.A.B + A --- A.B --- A.B.A + A.B --- A.B.B + + class A.A.B active + class A sel +``` + +```ruby +Namespace.where(...).roots + +namespace_object.root_ancestor +``` + +### Descendant namespaces + +The descendants of a namespace are its children, their children, and so on. + +```mermaid +graph TD + classDef active fill:#f00,color:#fff + classDef sel fill:#00f,color:#fff + + A --- A.A --- A.A.A + A.A --- A.A.B + A --- A.B --- A.B.A + A.B --- A.B.B + + class A.A active + class A.A.A,A.A.B sel +``` + +We can return ourself and our descendants through `self_and_descendants`. + +```ruby +Namespace.where(...).self_and_descendants + +namespace_object.self_and_descendants +``` + +We can return only our descendants excluding ourselves: + +```ruby +Namespace.where(...).self_and_descendants(include_self: false) + +namespace_object.descendants +``` + +We could not name the scope method `.descendants` because we would override the `Object` method of the same name. + +It can be more efficient to return the descendant IDs instead of the whole record: + +```ruby +Namespace.where(...).self_and_descendant_ids +Namespace.where(...).self_and_descendant_ids(include_self: false) + +namespace_object.self_and_descendant_ids +namespace_object.descendant_ids +``` + +### Ancestor namespaces + +The ancestors of a namespace are its children, their children, and so on. + +```mermaid +graph TD + classDef active fill:#f00,color:#fff + classDef sel fill:#00f,color:#fff + + A --- A.A --- A.A.A + A.A --- A.A.B + A --- A.B --- A.B.A + A.B --- A.B.B + + class A.A active + class A sel +``` + +We can return ourself and our ancestors through `self_and_ancestors`. + +```ruby +Namespace.where(...).self_and_ancestors + +namespace_object.self_and_ancestors +``` + +We can return only our ancestors excluding ourselves: + +```ruby +Namespace.where(...).self_and_ancestors(include_self: false) + +namespace_object.ancestors +``` + +We could not name the scope method `.ancestors` because we would override the `Module` method of the same name. + +It can be more efficient to return the ancestor ids instead of the whole record: + +```ruby +Namespace.where(...).self_and_ancstor_ids +Namespace.where(...).self_and_ancestor_ids(include_self: false) + +namespace_object.self_and_ancestor_ids +namespace_object.ancestor_ids +``` + +### Hierarchies + +A Namespace hierarchy is a `Namespace`, its ancestors, and its descendants. + +```mermaid +graph TD + classDef active fill:#f00,color:#fff + classDef sel fill:#00f,color:#fff + + A --- A.A --- A.A.A + A.A --- A.A.B + A --- A.B --- A.B.A + A.B --- A.B.B + + class A.A active + class A,A.A.A,A.A.B sel +``` + +We can query a namespace hierarchy: + +```ruby +Namespace.where(...).self_and_hierarchy + +namespace_object.self_and_hierarchy +``` + +### Recursive queries + +The queries above are known as the linear queries because they use the `namespaces.traversal_ids` column to perform standard SQL queries instead of recursive CTE queries. + +A set of legacy recursive queries are also accessible if needed: + +```ruby +Namespace.where(...).recursive_self_and_descendants +Namespace.where(...).recursive_self_and_descendants(include_self: false) +Namespace.where(...).recursive_self_and_descendant_ids +Namespace.where(...).recursive_self_and_descendant_ids(include_self: false) +Namespace.where(...).recursive_self_and_ancestors +Namespace.where(...).recursive_self_and_ancestors(include_self: false) +Namespace.where(...).recursive_self_and_ancstor_ids +Namespace.where(...).recursive_self_and_ancestor_ids(include_self: false) +Namespace.where(...).recursive_self_and_hierarchy + +namespace_object.recursive_root_ancestor +namespace_object.recursive_self_and_descendants +namespace_object.recursive_descendants +namespace_object.recursive_self_and_descendant_ids +namespace_object.recursive_descendant_ids +namespace_object.recursive_self_and_ancestors +namespace_object.recursive_ancestors +namespace_object.recursive_self_and_ancestor_ids +namespace_object.recursive_ancestor_ids +namespace_object.recursive_self_and_hierarchy +``` + +## Namespace query implementation + +The linear queries are executed using the `namespaces.traversal_ids` array column. Each array represents an ordered set of `Namespace` IDs from the root `Namespace` to the current `Namespace`. + +Given the scenario: + +```mermaid +graph TD + classDef active fill:#f00,color:#fff + classDef sel fill:#00f,color:#fff + + A --- A.A --- A.A.A + A.A --- A.A.B + A --- A.B --- A.B.A + A.B --- A.B.B + + class A.A.B active +``` + +The `traversal_ids` for `Namespace` `A.A.B` would be `[A, A.A, A.A.B]`. + +The `traversal_ids` have some useful properties to keep in mind if working in this area: + +- The root of every `Namespace` is provided by `traversal_ids[1]`. Note that PostgreSQL array indexes begin at `1`. +- The ID of the current `Namespace` is provided by `traversal_ids[array_length(traversal_ids, 1)]`. +- The `Namespace` ancestors are represented by the `traversal_ids`. +- A `Namespace`'s `traversal_ids` are a subset of their descendants `traversal_ids`. A `Namespace` with `traversal_ids = [1,2,3]` will have descendants that all start with `[1,2,3,...]`. +- PostgreSQL arrays are ordered such that `[1] < [1,1] < [2]`. + +Using these properties we find the `root` and `ancestors` are already provided for by `traversal_ids`. + +With the object descendant queries we lean on the `@>` array operator which will test inclusion of an array inside another array. +The `@>` operator has been found to be quite slow as the search space grows. Another method is used for scope queries which tend to have larger search spaces. +With scope queries we combine comparison operators with the array ordering property. + +All descendants of a `Namespace` with `traversal_ids = [1,2,3]` have `traversal_ids` that are greater than `[1,2,3]` but less than `[1,2,4]`. +In this example `[1,2,3]` and `[1,2,4]` are siblings, and `[1,2,4]` is the next sibling after `[1,2,3]`. A SQL function is provided to find the next sibling of a `traversal_ids` called `next_traversal_ids_sibling`. + +```sql +gitlabhq_development=# select next_traversal_ids_sibling(ARRAY[1,2,3]); + next_traversal_ids_sibling +---------------------------- + {1,2,4} +(1 row) +``` + +We then build descendant linear query scopes using comparison operators: + +```sql +WHERE namespaces.traversal_ids > ARRAY[1,2,3] + AND namespaces.traversal_ids < next_traversal_ids_sibling(ARRAY[1,2,3]) +``` + +### Superset + +`Namespace` queries are prone to returning duplicate results. For example, consider a query to find descendants of `A` and `A.A`: + +```mermaid +graph TD + classDef active fill:#f00,color:#fff + classDef sel fill:#00f,color:#fff + + A --- A.A --- A.A.A + A.A --- A.A.B + A --- A.B --- A.B.A + A.B --- A.B.B + + class A,A.A active + class A.A.A,A.A.B,A.B,A.B.A,A.B.B sel +``` + +```ruby +namespaces = Namespace.where(name: ['A', 'A.A']) + +namespaces.self_and_descendants + +=> A.A, A.A.A, A.A.B, A.B, A.B.A, A.B.B +``` + +Searching for the descendants of both `A` and `A.A` is unnecessary because `A.A` is already a descendant of `A`. +In extreme cases this can create excessive I/O leading to poor performance. + +Redundant `Namespaces` are eliminated from a query if a `Namespace` `ID` in the `traversal_ids` attribute matches an `ID` belonging to another `Namespace` in the set of `Namespaces` being queried. +A match of this condition signifies that an ancestor exists in the set of `Namespaces` being queried, and the current `Namespace` is therefore redundant. +This optimization will result in much better performance of edge cases that would otherwise be very slow. diff --git a/doc/development/packages/new_format_development.md b/doc/development/packages/new_format_development.md index 66e6cb89661..0af0b8ad480 100644 --- a/doc/development/packages/new_format_development.md +++ b/doc/development/packages/new_format_development.md @@ -62,7 +62,7 @@ As an MVC, we recommend beginning with a project-level endpoint. A typical itera - Publish and install in a project - Install from a group -- Publish and install in an Instance (this is for Self-Managed customers) +- Publish and install in an instance (this is for self-managed customers) Using instance-level endpoints requires [stricter naming conventions](#naming-conventions). diff --git a/doc/development/packages/settings.md b/doc/development/packages/settings.md index 33caa064ab3..0fc49c4eb5d 100644 --- a/doc/development/packages/settings.md +++ b/doc/development/packages/settings.md @@ -17,6 +17,7 @@ Setting | Table | Description `npm_package_requests_forwarding` | `application_settings` | Enables or disables npm package forwarding at the instance level. `pypi_package_requests_forwarding` | `application_settings` | Enables or disables PyPI package forwarding at the instance level. `packages_cleanup_package_file_worker_capacity` | `application_settings` | Number of concurrent workers allowed for package file cleanup. +`package_registry_allow_anyone_to_pull_option` | `application_settings` | Enables or disables the `Allow anyone to pull from Package Registry` toggle. `throttle_unauthenticated_packages_api_requests_per_period` | `application_settings` | Request limit for unauthenticated package API requests in the period defined by `throttle_unauthenticated_packages_api_period_in_seconds`. `throttle_unauthenticated_packages_api_period_in_seconds` | `application_settings` | Period in seconds to measure unauthenticated package API requests. `throttle_authenticated_packages_api_requests_per_period` | `application_settings` | Request limit for authenticated package API requests in the period defined by `throttle_authenticated_packages_api_period_in_seconds`. @@ -66,6 +67,8 @@ Setting | Table | Description `maven_duplicate_exception_regex` | `namespace_package_settings` | Regex defining Maven packages that are allowed to be duplicate when duplicates are not allowed. This matches the name and version of the package. `generic_duplicates_allowed` | `namespace_package_settings` | Allow or prevent duplicate generic packages. `generic_duplicate_exception_regex` | `namespace_package_settings` | Regex defining generic packages that are allowed to be duplicate when duplicates are not allowed. +`nuget_duplicates_allowed` | `namespace_package_settings` | Allow or prevent duplicate NuGet packages. +`nuget_duplicate_exception_regex` | `namespace_package_settings` | Regex defining NuGet packages that are allowed to be duplicate when duplicates are not allowed. Dependency Proxy Cleanup Policies - `ttl` | `dependency_proxy_image_ttl_group_policies` | Number of days to retain an unused Dependency Proxy file before it is removed. Dependency Proxy - `enabled` | `dependency_proxy_image_ttl_group_policies` | Enable or disable the Dependency Proxy cleanup policy. diff --git a/doc/development/permissions/custom_roles.md b/doc/development/permissions/custom_roles.md index 9aba4035ec9..337c8f6d96b 100644 --- a/doc/development/permissions/custom_roles.md +++ b/doc/development/permissions/custom_roles.md @@ -6,12 +6,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Custom Roles -Users can create custom roles and define those roles by assigning specific abilities. For example, a user could create an "Engineer" role with `read code` and `admin merge requests` abilities, but without abilities like `admin issues`. +Ultimate customers can create custom roles and define those roles by assigning specific abilities. -In this context: +For example, a user could create an "Engineer" role with `read code` and `admin merge requests` abilities, but without abilities like `admin issues`. -- "Ability" is an action a user can do. -- "Permission" defines the policy classes. +In this context, the terms "permission" and "ability" are often used interchangeably. + +- "Ability" is an action a user can do. These map to [Declarative Policy abilities](https://gitlab.com/gitlab-org/ruby/gems/declarative-policy/-/blob/main/doc/defining-policies.md#rules) and live in Policy classes in `ee/app/policies/*`. +- "Permission" is how we refer to an ability [in user-facing documentation](../../user/permissions.md). The documentation of permissions is manually generated so there is not necessarily a 1:1 mapping of the permissions listed in documentation and the abilities defined in Policy classes. ## Custom roles vs static roles @@ -20,18 +22,21 @@ In GitLab 15.9 and earlier, GitLab only had [static roles](predefined_roles.md) With custom roles, the customers can decide which abilities they want to assign to certain user groups. For example: - In the static role system, reading of vulnerabilities is limited to a Developer role. -- In the custom role system, a customer can assign this ability to a new custom role based on the Reporter role. - -## Technical overview +- In the custom role system, a customer can assign this ability to a new custom role based on any static role. -Individual custom roles are stored in the `member_roles` table (`MemberRole` model) and can be defined only for top-level groups. This table includes individual abilities and a `base_access_level` value. This value defines the minimum access level of: +Like static roles, custom roles are [inherited](../../user/project/members/index.md#inherited-membership) within a group hierarchy. If a user has custom role for a group, that user will also have a custom role for any projects or subgroups within the group. -- Users who can be assigned to the custom role. -- Every ability. - -For example, the `read_vulnerability` ability has a minimum access level of `Reporter`. That means only member role records with `base_access_level = REPORTER` (20) or higher can have the `read_vulnerability` value set to `true`. Also, only users who have at least the Reporter role can be assigned that ability. +## Technical overview -For now, custom role abilities are supported only at project level. +- Individual custom roles are stored in the `member_roles` table (`MemberRole` model). +- A `member_roles` record is associated with top-level groups (not subgroups) via the `namespace_id` foreign key. +- A Group or project membership (`members` record) is associated with a custom role via the `member_role_id` foreign key. +- A Group or project membership can be associated with any custom role that is defined on the root-level group of the group or project. +- The `member_roles` table includes individual permissions and a `base_access_level` value. +- The `base_access_level` must be a [valid access level](../../api/access_requests.md#valid-access-levels). +The `base_access_level` determines which abilities are included in the custom role. For example, if the `base_access_level` is `10`, the custom role will include any abilities that a static Guest role would receive, plus any additional abilities that are enabled by the `member_roles` record by setting an attribute, such as `read_code`, to true. +- A custom role can enable additional abilities for a `base_access_level` but it cannot disable a permission. As a result, custom roles are "additive only". The rationale for this choice is [in this comment](https://gitlab.com/gitlab-org/gitlab/-/issues/352891#note_1059561579). +- For now, custom role abilities are supported only at project level. There is an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/411851) to add support for custom group abilities. ## How to implement a new ability for custom roles @@ -153,6 +158,26 @@ There might be features that require additional abilities but try to minimalize This is also where your work should begin. Take all the abilities for the feature you work on, and consolidate those abilities into `read_`, `admin_`, or additional abilities if necessary. +Many abilities in the `GroupPolicy` and `ProjectPolicy` classes have many +redundant policies. There is an [epic for consolidating these Policy classes](https://gitlab.com/groups/gitlab-org/-/epics/6689). +If you encounter similar permissions in these classes, consider refactoring so +that they have the same name. + +For example, you see in `GroupPolicy` that there is an ability called +`read_group_security_dashboard` and in `ProjectPolicy` has an ability called +`read_project_security_dashboard`. You'd like to make both customizable. Rather +than adding a row to the `member_roles` table for each ability, consider +renaming them to `read_security_dashboard` and adding `read_security_dashboard` +to the `member_roles` table. This is more expected because it means that +enabling `read_security_dashboard` on the parent group will enable the custom +For example, `GroupPolicy` has an ability called `read_group_security_dashboard` and `ProjectPolicy` has an ability +called `read_project_security_dashboard`. If you would like to make both customizable, rather than adding a row to the +`member_roles` table for each ability, consider renaming them to `read_security_dashboard` and adding +`read_security_dashboard` to the `member_roles` table. This convention means that enabling `read_security_dashboard` on +the parent group will allow the custom role to access the group security dashboard and the project security dashboard +for each project in that group. Enabling the same permission on a specific project will allow access to that projects' +security dashboard. + ### Implement a new ability To add a new ability to a custom role: diff --git a/doc/development/pipelines/internals.md b/doc/development/pipelines/internals.md index c0d7bbd3713..1a4e4e738a8 100644 --- a/doc/development/pipelines/internals.md +++ b/doc/development/pipelines/internals.md @@ -294,3 +294,126 @@ qa:selectors-as-if-foss: extends: - .qa:rules:as-if-foss ``` + +### Extend the `.fast-no-clone-job` job + +Downloading the branch for the canonical project takes between 20 and 30 seconds. + +Some jobs only need a limited number of files, which we can download via the GitLab API. + +You can skip a job `git clone`/`git fetch` by adding the following pattern to a job. + +#### Scenario 1: no `before_script` is defined in the job + +This applies to the parent sections the job extends from as well. + +You can just extend the `.fast-no-clone-job`: + +**Before:** + +```yaml + # Note: No `extends:` is present in the job + a-job: + script: + - source scripts/rspec_helpers.sh scripts/slack + - echo "No need for a git clone!" +``` + +**After:** + +```yaml + # Note: No `extends:` is present in the job + a-job: + extends: + - .fast-no-clone-job + variables: + FILES_TO_DOWNLOAD: > + scripts/rspec_helpers.sh + scripts/slack + script: + - source scripts/rspec_helpers.sh scripts/slack + - echo "No need for a git clone!" +``` + +#### Scenario 2: a `before_script` block is already defined in the job (or in jobs it extends) + +For this scenario, you have to: + +1. Extend the `.fast-no-clone-job` as in the first scenario (this will merge the `FILES_TO_DOWNLOAD` variable with the other variables) +1. Make sure the `before_script` section from `.fast-no-clone-job` is referenced in the `before_script` we use for this job. + +**Before:** + +```yaml + .base-job: + before_script: + echo "Hello from .base-job" + + a-job: + extends: + - .base-job + script: + - source scripts/rspec_helpers.sh scripts/slack + - echo "No need for a git clone!" +``` + +**After:** + +```yaml + .base-job: + before_script: + echo "Hello from .base-job" + + a-job: + extends: + - .base-job + - .fast-no-clone-job + variables: + FILES_TO_DOWNLOAD: > + scripts/rspec_helpers.sh + scripts/slack + before_script: + - !reference [".fast-no-clone-job", before_script] + - !reference [".base-job", before_script] + script: + - source scripts/rspec_helpers.sh scripts/slack + - echo "No need for a git clone!" +``` + +#### Caveats + +- This pattern does not work if a script relies on `git` to access the repository, because we don't have the repository without cloning or fetching. +- The job using this pattern needs to have `curl` available. +- If you need to run `bundle install` in the job (even using `BUNDLE_ONLY`), you need to: + - Download the gems that are stored in the `gitlab-org/gitlab` project. + - You can use the `download_local_gems` shell command for that purpose. + - Include the `Gemfile`, `Gemfile.lock` and `Gemfile.checksum` (if applicable) + +#### Where is this pattern used? + +- For now, we use this pattern for the following jobs, and those do not block private repositories: + - `review-build-cng-env` for: + - `GITALY_SERVER_VERSION` + - `GITLAB_ELASTICSEARCH_INDEXER_VERSION` + - `GITLAB_KAS_VERSION` + - `GITLAB_METRICS_EXPORTER_VERSION` + - `GITLAB_PAGES_VERSION` + - `GITLAB_SHELL_VERSION` + - `scripts/trigger-build.rb` + - `VERSION` + - `review-deploy` for: + - `GITALY_SERVER_VERSION` + - `GITLAB_SHELL_VERSION` + - `scripts/review_apps/review-apps.sh` + - `scripts/review_apps/seed-dast-test-data.sh` + - `VERSION` + - `rspec:coverage` for: + - `config/bundler_setup.rb` + - `Gemfile` + - `Gemfile.checksum` + - `Gemfile.lock` + - `scripts/merge-simplecov` + - `spec/simplecov_env_core.rb` + - `spec/simplecov_env.rb` + +Additionally, `scripts/utils.sh` is always downloaded from the API when this pattern is used (this file contains the code for `.fast-no-clone-job`). diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 0305cbf25b4..b15c4eca5ae 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -46,7 +46,7 @@ bin/rake "gitlab:seed:issues[group-path/project-path]" By default, this seeds an average of 2 issues per week for the last 5 weeks per project. -#### Seeding issues for Insights charts **(ULTIMATE)** +#### Seeding issues for Insights charts **(ULTIMATE ALL)** You can seed issues specifically for working with the [Insights charts](../user/group/insights/index.md) with the @@ -526,7 +526,7 @@ NOTE: This Rake task needs `docker` to be installed. To update generated code for OpenAPI client located in -`vendor/gems/error_tracking_open_api` run the following commands: +`gems/error_tracking_open_api` run the following commands: ```shell # Run rake task @@ -535,7 +535,7 @@ bundle exec rake gems:error_tracking_open_api:generate # Review and test the changes # Commit the changes -git commit -m 'Update ErrorTrackingOpenAPI from OpenAPI definition' vendor/gems/error_tracking_open_api +git commit -m 'Update ErrorTrackingOpenAPI from OpenAPI definition' gems/error_tracking_open_api ``` ## Update banned SSH keys diff --git a/doc/development/redis/new_redis_instance.md b/doc/development/redis/new_redis_instance.md index 00cc102b427..bc58bae45ec 100644 --- a/doc/development/redis/new_redis_instance.md +++ b/doc/development/redis/new_redis_instance.md @@ -30,8 +30,8 @@ Before we can switch any features to using the new instance, we have to support configuring it and referring to it in the codebase. We must support the main installation types: -- Source installs (including development environments) - [example MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62767) -- Omnibus - [example MR](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5316) +- Self-compiled installations (including development environments) - [example MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62767) +- Linux package installations - [example MR](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5316) - Helm charts - [example MR](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/2031) ### Fallback instance diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index bfb568d2fd2..4a964f7d3c9 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -38,7 +38,7 @@ projects = if current_user && params[:authorized_only].presence && !current_user_related? current_user.authorized_projects elsif group - finder_options = { include_subgroups: params[:include_subgroups], only_owned: true } + finder_options = { include_subgroups: params[:include_subgroups], exclude_shared: true } GroupProjectsFinder.new(group: group, current_user: current_user, options: finder_options).execute else ProjectsFinder.new(current_user: current_user).execute diff --git a/doc/development/ruby_upgrade.md b/doc/development/ruby_upgrade.md index ccd65b4e7e9..21c19c31b0a 100644 --- a/doc/development/ruby_upgrade.md +++ b/doc/development/ruby_upgrade.md @@ -209,7 +209,7 @@ prudent to skip this step until you have verified that it runs smoothly in produ rollout. In this case, go to the next step first, and then, after the verification period has passed, promote the new Ruby to be the new default. -### Update CNG and Omnibus, merge the GitLab MR +### Update CNG, Omnibus, Self-compiled and merge the GitLab MR The last step is to use the new Ruby in production. This requires updating Omnibus and production Docker images to use the new version. @@ -220,6 +220,7 @@ To use the new Ruby in production, update the following projects: - [Cloud-native GitLab Docker Images (CNG)](https://gitlab.com/gitlab-org/build/CNG) ([example](https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/739)) - [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) ([example](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5545)) +- [Self-compiled installations](../install/installation.md): update the [Ruby system version check](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/system_check/app/ruby_version_check.rb) If you submit a change management request, coordinate the rollout with infrastructure engineers. When dealing with larger upgrades, involve [Release Managers](https://about.gitlab.com/community/release-managers/) diff --git a/doc/development/search/advanced_search_migration_styleguide.md b/doc/development/search/advanced_search_migration_styleguide.md index d7c0dddee7b..10c4fa3dcc6 100644 --- a/doc/development/search/advanced_search_migration_styleguide.md +++ b/doc/development/search/advanced_search_migration_styleguide.md @@ -6,13 +6,25 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Advanced search migration style guide -## Creating a new advanced search migration - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/234046) in GitLab 13.6. +## Create a new advanced search migration NOTE: This functionality is only supported for indices created in GitLab 13.0 and later. +### With a script + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/414674) in GitLab 16.3. + +Execute `scripts/elastic-migration` and follow the prompts to create: + +- A migration file to define the migration: `ee/elastic/migrate/YYYYMMDDHHMMSS_migration_name.rb` +- A spec file to test the migration: `ee/spec/elastic/migrate/YYYYMMDDHHMMSS_migration_name_spec.rb` +- A dictionary file to identify the migration: `ee/elastic/docs/YYYYMMDDHHMMSS_migration_name.yml` + +### Manually + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/234046) in GitLab 13.6. + In the [`ee/elastic/migrate/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/elastic/migrate) folder, create a new file with the filename format `YYYYMMDDHHMMSS_migration_name.rb`. This format is the same for Rails database migrations. ```ruby @@ -198,6 +210,27 @@ class MigrationName < Elastic::Migration end ``` +#### `Search::Elastic::MigrationReindexBasedOnSchemaVersion` + +Reindexes all documents in the index that stores the specified document type and updates `schema_version`. + +Requires the `DOCUMENT_TYPE` and `NEW_SCHEMA_VERSION` constants. +The index mapping must have a `schema_version` integer field in a `YYMM` format. + +```ruby +class MigrationName < Elastic::Migration + include Search::Elastic::MigrationReindexBasedOnSchemaVersion + + batched! + batch_size 9_000 + throttle_delay 1.minute + + DOCUMENT_TYPE = WorkItem + NEW_SCHEMA_VERSION = 23_08 + UPDATE_BATCH_SIZE = 100 +end +``` + #### `Elastic::MigrationHelper` Contains methods you can use when a migration doesn't fit the previous examples. diff --git a/doc/development/sec/analyzer_development_guide.md b/doc/development/sec/analyzer_development_guide.md index af8d1758713..c76b7f5e55f 100644 --- a/doc/development/sec/analyzer_development_guide.md +++ b/doc/development/sec/analyzer_development_guide.md @@ -342,7 +342,7 @@ This issue will guide you through the whole release process. In general, you hav - Check the list of supported technologies in GitLab documentation. - [Supported languages in SAST](../../user/application_security/sast/index.md#supported-languages-and-frameworks) - [Supported languages in DS](../../user/application_security/dependency_scanning/index.md#supported-languages-and-package-managers) - - [Supported languages in LM](../../user/compliance/license_compliance/index.md#supported-languages-and-package-managers) + - [Supported languages in LS](../../user/compliance/license_scanning_of_cyclonedx_files/index.md#supported-languages-and-package-managers) - Check that CI **_job definitions are still accurate_** in vendored CI/CD templates and **_all of the ENV vars are propagated_** to the Docker containers upon `docker run` per tool. diff --git a/doc/development/sec/generate_test_vulnerabilities.md b/doc/development/sec/generate_test_vulnerabilities.md new file mode 100644 index 00000000000..75fb7edd68e --- /dev/null +++ b/doc/development/sec/generate_test_vulnerabilities.md @@ -0,0 +1,32 @@ +--- +type: reference, howto +stage: Govern +group: Threat Insights +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Generate test vulnerabilities + +You can generate test vulnerabilities for the [Vulnerability Report](../../user/application_security/vulnerability_report/index.md) to test GitLab +vulnerability management features without running a pipeline. + +1. Log in to GitLab. +1. Go to `/-/profile/personal_access_tokens` and generate a personal access token with `api` permissions. +1. Go to your project page and find the project ID. You can find the project ID below the project title. +1. [Clone the GitLab repository](../../gitlab-basics/start-using-git.md#clone-a-repository) to your local machine. +1. Open a terminal and go to `gitlab/qa` directory. +1. Run `bundle install` +1. Run the following command: + +```shell +GITLAB_QA_ACCESS_TOKEN=<your_personal_access_token> GITLAB_URL="<address:port>" bundle exec rake vulnerabilities:setup\[<your_project_id>,<vulnerability_count>\] --trace +``` + +Make sure you do the following: + +- Replace `<your_personal_access_token>` with the token you generated in step one. +- Double check the `GITLAB_URL`. It should point to address and port of your GitLab instance, for example `http://localhost:3000` if you are running GDK +- Replace `<your_project_id>` with the ID you obtained in step three above. +- Replace `<vulnerability_count>` with the number of vulnerabilities you'd like to generate. + +The script creates the specified number of placeholder vulnerabilities in the project. diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 8d6f36bb189..186239cc547 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -1412,3 +1412,64 @@ If circumstances dictate that local storage is the only option, a couple of prec - Local storage should only be used for the minimal amount of data possible. Consider alternative storage formats. - If you have to store sensitive data using local storage, do so for the minimum time possible, calling `localStorage.removeItem` on the item as soon as we're done with it. Another alternative is to call `localStorage.clear()`. + +## Logging + +Logging is the tracking of events that happen in the system for the purposes of future investigation or processing. + +### Purpose of logging + +Logging helps track events for debugging. Logging also allows the application to generate an audit trail that you can use for security incident identification and analysis. + +### What type of events should be logged + +- Failures + - Login failures + - Input/output validation failures + - Authentication failures + - Authorization failures + - Session management failures + - Timeout errors +- Account lockouts +- Use of invalid access tokens +- Authentication and authorization events + - Access token creation/revocation/expiry + - Configuration changes by administrators + - User creation or modification + - Password change + - User creation + - Email change +- Sensitive operations + - Any operation on sensitive files or resources + - New runner registration + +### What should be captured in the logs + +- The application logs must record attributes of the event, which helps auditors identify the time/date, IP, user ID, and event details. +- To avoid resource depletion, make sure the proper level for logging is used (for example, `information`, `error`, or `fatal`). + +### What should not be captured in the logs + +- Personal data, except for integer-based identifiers and UUIDs, or IP address, which can be logged when necessary. +- Credentials like access tokens or passwords. If credentials must be captured for debugging purposes, log the internal ID of the credential (if available) instead. Never log credentials under any circumstances. + - When [debug logging](../ci/variables/index.md#enable-debug-logging) is enabled, all masked CI/CD variables are visible in job logs. Consider using [protected variables](../ci/variables/index.md#protect-a-cicd-variable) when possible so that sensitive CI/CD variables are only available to pipelines running on protected branches or protected tags. +- Any data supplied by the user without proper validation. +- Any information that might be considered sensitive (for example, credentials, passwords, tokens, keys, or secrets). Here is an [example](https://gitlab.com/gitlab-org/gitlab/-/issues/383142) of sensitive information being leaked through logs. + +### Protecting log files + +- Access to the log files should be restricted so that only the intended party can modify the logs. +- External user input should not be directly captured in the logs without any validation. This could lead to unintended modification of logs through log injection attacks. +- An audit trail for log edits must be available. +- To avoid data loss, logs must be saved on different storage. + +### Who to contact if you have questions + +For general guidance, contact the [Application Security](https://about.gitlab.com/handbook/security/security-engineering/application-security/) team. + +### Related topics + +- [Log system in GitLab](../administration/logs/index.md) +- [Audit event development guidelines](../development/audit_event_guide/index.md)) +- [Security logging overview](https://about.gitlab.com/handbook/security/security-engineering/security-logging/) +- [OWASP logging cheat sheet](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html) diff --git a/doc/development/sidekiq/index.md b/doc/development/sidekiq/index.md index 936388a14fe..1b3b319ef28 100644 --- a/doc/development/sidekiq/index.md +++ b/doc/development/sidekiq/index.md @@ -135,7 +135,7 @@ Sidekiq workers are deferred by two ways, sidekiq_options retry: 3 include ChaosQueue - defer_on_database_health_signal :gitlab_main, 1.minute, [:users] + defer_on_database_health_signal :gitlab_main, [:users], 1.minute def perform(duration_s) Gitlab::Chaos.sleep(duration_s) diff --git a/doc/development/sidekiq/worker_attributes.md b/doc/development/sidekiq/worker_attributes.md index 1e3104c5e86..814b61c746b 100644 --- a/doc/development/sidekiq/worker_attributes.md +++ b/doc/development/sidekiq/worker_attributes.md @@ -16,11 +16,11 @@ have to redefine them if you want to override their values. Jobs can have an `urgency` attribute set, which can be `:high`, `:low`, or `:throttled`. These have the below targets: -| **Urgency** | **Queue Scheduling Target** | **Execution Latency Requirement** | -|--------------|-----------------------------|------------------------------------| -| `:high` | 10 seconds | p50 of 1 second, p99 of 10 seconds | -| `:low` | 1 minute | Maximum run time of 5 minutes | -| `:throttled` | None | Maximum run time of 5 minutes | +| **Urgency** | **Queue Scheduling Target** | **Execution Latency Requirement** | +|--------------- | ----------------------------- | ------------------------------------ | +| `:high` | 10 seconds | 10 seconds | +| `:low` (default) | 1 minute | 5 minutes | +| `:throttled` | None | 5 minutes | To set a job's urgency, use the `urgency` class method: @@ -326,3 +326,42 @@ end For [idempotent jobs](idempotent_jobs.md) that declare either `:sticky` or `:delayed` data consistency, we are [preserving the latest WAL location](idempotent_jobs.md#preserve-the-latest-wal-location-for-idempotent-jobs) while deduplicating, ensuring that we read from the replica that is fully caught up. + +## Job pause control + +With the `pause_control` property, you can conditionally pause job processing. If the strategy is active, the job +is stored in a separate `ZSET` and re-enqueued when the strategy becomes inactive. `PauseControl::ResumeWorker` is a cron +worker that checks if any paused jobs must be restarted. + +To use `pause_control`, you can: + +- Use one of the strategies defined in `lib/gitlab/sidekiq_middleware/pause_control/strategies/`. +- Define a custom strategy in `lib/gitlab/sidekiq_middleware/pause_control/strategies/` and add the strategy to `lib/gitlab/sidekiq_middleware/pause_control/strategies.rb`. + +For example: + +```ruby +module Gitlab + module SidekiqMiddleware + module PauseControl + module Strategies + class CustomStrategy < Base + def should_pause? + ApplicationSetting.current.elasticsearch_pause_indexing? + end + end + end + end + end +end +``` + +```ruby +class PausedWorker + include ApplicationWorker + + pause_control :custom_strategy + + # ... +end +``` diff --git a/doc/development/software_design.md b/doc/development/software_design.md index 33a6dfcb4a7..e2749df372d 100644 --- a/doc/development/software_design.md +++ b/doc/development/software_design.md @@ -140,6 +140,27 @@ end If classes that are defined into a namespace have a lot in common with classes in other namespaces, chances are that these two namespaces are part of the same bounded context. +## Distinguish domain code from generic code + +The [guidelines above](#use-namespaces-to-define-bounded-contexts) refer primarily to the domain code. +For domain code we should put Ruby classes under a namespace that represents a given bounded context +(a cohesive set of features and capabilities). + +The domain code is unique to GitLab product. It describes the business logic, policies and data. +This code should live in the GitLab repository. The domain code is split between `app/` and `lib/` +primarily. + +In an application codebase there is also generic code that allows to perform more infrastructure level +actions. This can be loggers, instrumentation, clients for datastores like Redis, database utilities, etc. + +Although vital for an application to run, generic code doesn't describe any business logic that is +unique to GitLab product. It could be rewritten or replaced by off-the-shelf solutions without impacting +the business logic. +This means that generic code should be separate from the domain code. + +Today a lot of the generic code lives in `lib/` but it's mixed with domain code. +We should extract gems into `gems/` directory instead, as described in our [Gems development guidelines](gems.md). + ## Taming Omniscient classes We must consider not adding new data and behavior to [omniscient classes](https://en.wikipedia.org/wiki/God_object) (also known as god objects). diff --git a/doc/development/sql.md b/doc/development/sql.md index 5dbfd8f3ddb..101ccc239e7 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -315,6 +315,30 @@ union = Gitlab::SQL::Union.new([projects, more_projects, ...]) Project.from("(#{union.to_sql}) projects") ``` +The `FromUnion` model concern provides a more convenient method to produce the same result as above: + +```ruby +class Project + include FromUnion + ... +end + +Project.from_union(projects, more_projects, ...) +``` + +`UNION` is common through the codebase, but it's also possible to use the other SQL set operators of `EXCEPT` and `INTERSECT`: + +```ruby +class Project + include FromIntersect + include FromExcept + ... +end + +intersected = Project.from_intersect(all_projects, project_set_1, project_set_2) +excepted = Project.from_except(all_projects, project_set_1, project_set_2) +``` + ### Uneven columns in the `UNION` sub-queries When the `UNION` query has uneven columns in the `SELECT` clauses, the database returns an error. @@ -354,7 +378,10 @@ values (the new column is missing), because the values are cached within the `Ac cache. These values are usually populated when the application boots up. At this point, the only fix would be a full application restart so that the schema cache gets -updated. +updated. Since [GitLab 16.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121957), +the schema cache will be automatically reset so that subsequent queries +will succeed. This reset can be disabled by disabling the `ops` feature +flag `reset_column_information_on_statement_invalid`. The problem can be avoided if we always use `SELECT users.*` or we always explicitly define the columns. @@ -460,6 +487,96 @@ Both methods use subtransactions internally if executed within the context of an existing transaction. This can significantly impact overall performance, especially if more than 64 live subtransactions are being used inside a single transaction. +### Can I use `.safe_find_or_create_by`? + +If your code is generally isolated (for example it's executed in a worker only) and not wrapped with another transaction, then you can use `.safe_find_or_create_by`. However, there is no tooling to catch cases when someone else calls your code within a transaction. Using `.safe_find_or_create_by` will definitely carry some risks that cannot be eliminated completely at the moment. + +Additionally, we have a RuboCop rule `Performance/ActiveRecordSubtransactionMethods` that prevents the usage of `.safe_find_or_create_by`. This rule can be disabled on a case by case basis via `# rubocop:disable Performance/ActiveRecordSubtransactionMethods`. + +## Alternative 1: `UPSERT` + +The [`.upsert`](https://api.rubyonrails.org/v7.0.5/classes/ActiveRecord/Persistence/ClassMethods.html#method-i-upsert) method can be an alternative solution when the table is backed by a unique index. + +Simple usage of the `.upsert` method: + +```ruby +BuildTrace.upsert( + { + build_id: build_id, + title: title + }, + unique_by: :build_id +) +``` + +A few things to be careful about: + +- The sequence for the primary key will be incremented, even if the record was only updated. +- The created record is not returned. The `returning` option only returns data when an `INSERT` happens (new record). +- `ActiveRecord` validations are not executed. + +An example of the `.upsert` method with validations and record loading: + +```ruby +params = { + build_id: build_id, + title: title +} + +build_trace = BuildTrace.new(params) + +unless build_trace.valid? + raise 'notify the user here' +end + +BuildTrace.upsert(params, unique_by: :build_id) + +build_trace = BuildTrace.find_by!(build_id: build_id) + +# do something with build_trace here +``` + +The code snippet above will not work well if there is a model-level uniqueness validation on the `build_id` column because we invoke the validation before calling `.upsert`. + +To work around this, we have two options: + +- Remove the unqueness validation from the `ActiveRecord` model. +- Use the [`on` keyword](https://guides.rubyonrails.org/active_record_validations.html#on) and implement context-specific validation. + +### Alternative 2: Check existence and rescue + +When the chance of concurrently creating the same record is very low, we can use a simpler approach: + +```ruby +def my_create_method + params = { + build_id: build_id, + title: title + } + + build_trace = BuildTrace + .where(build_id: params[:build_id]) + .first + + build_trace = BuildTrace.new(params) if build_trace.blank? + + build_trace.update!(params) + +rescue ActiveRecord::RecordInvalid => invalid + retry if invalid.record&.errors&.of_kind?(:build_id, :taken) +end +``` + +The method does the following: + +1. Look up the model by the unique column. +1. If no record found, build a new one. +1. Persist the record. + +There is a short race condition between the lookup query and the persist query where another process could insert the record and cause an `ActiveRecord::RecordInvalid` exception. + +The code rescues this particular exception and retries the operation. For the second run, the record would be successfully located. For example check [this block of code](https://gitlab.com/gitlab-org/gitlab/-/blob/0b51d7fbb97d4becf5fd40bc3b92f732bece85bd/ee/app/services/compliance_management/standards/gitlab/prevent_approval_by_author_service.rb#L20-30) in `PreventApprovalByAuthorService`. + ## Monitor SQL queries in production GitLab team members can monitor slow or canceled queries on GitLab.com @@ -468,3 +585,129 @@ searchable using Kibana. See [the runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/pg_collect_query_data.md#searching-postgresql-logs-with-kibanaelasticsearch) for more details. + +## When to use common table expressions + +You can use common table expressions (CTEs) to create a temporary result set within a more complex query. +You can also use a recursive CTE to reference the CTE's result set within +the query itself. The following example queries a chain of +`personal access tokens` referencing each other in the +`previous_personal_access_token_id` column. + +```sql +WITH RECURSIVE "personal_access_tokens_cte" AS ( +( + SELECT + "personal_access_tokens".* + FROM + "personal_access_tokens" + WHERE + "personal_access_tokens"."previous_personal_access_token_id" = 15) + UNION ( + SELECT + "personal_access_tokens".* + FROM + "personal_access_tokens", + "personal_access_tokens_cte" + WHERE + "personal_access_tokens"."previous_personal_access_token_id" = "personal_access_tokens_cte"."id")) +SELECT + "personal_access_tokens".* +FROM + "personal_access_tokens_cte" AS "personal_access_tokens" + + id | previous_personal_access_token_id +----+----------------------------------- + 16 | 15 + 17 | 16 + 18 | 17 + 19 | 18 + 20 | 19 + 21 | 20 +(6 rows) +``` + +As CTEs are temporary result sets, you can use them within another `SELECT` +statement. Using CTEs with `UPDATE`, or `DELETE` could lead to unexpected +behavior: + +Consider the following method: + +```ruby +def personal_access_token_chain(token) + cte = Gitlab::SQL::RecursiveCTE.new(:personal_access_tokens_cte) + personal_access_token_table = Arel::Table.new(:personal_access_tokens) + + cte << PersonalAccessToken + .where(personal_access_token_table[:previous_personal_access_token_id].eq(token.id)) + cte << PersonalAccessToken + .from([personal_access_token_table, cte.table]) + .where(personal_access_token_table[:previous_personal_access_token_id].eq(cte.table[:id])) + PersonalAccessToken.with.recursive(cte.to_arel).from(cte.alias_to(personal_access_token_table)) +end +``` + +It works as expected when it is used to query data: + +```sql +> personal_access_token_chain(token) + +WITH RECURSIVE "personal_access_tokens_cte" AS ( +( + SELECT + "personal_access_tokens".* + FROM + "personal_access_tokens" + WHERE + "personal_access_tokens"."previous_personal_access_token_id" = 11) + UNION ( + SELECT + "personal_access_tokens".* + FROM + "personal_access_tokens", + "personal_access_tokens_cte" + WHERE + "personal_access_tokens"."previous_personal_access_token_id" = "personal_access_tokens_cte"."id")) +SELECT + "personal_access_tokens".* +FROM + "personal_access_tokens_cte" AS "personal_access_tokens" +``` + +However, the CTE is dropped when used with `#update_all`. As a result, the method +updates the entire table: + +```sql +> personal_access_token_chain(token).update_all(revoked: true) + +UPDATE + "personal_access_tokens" +SET + "revoked" = TRUE +``` + +To work around this behavior: + +1. Query the `ids` of the records: + + ```ruby + > token_ids = personal_access_token_chain(token).pluck_primary_key + => [16, 17, 18, 19, 20, 21] + ``` + +1. Use this array to scope `PersonalAccessTokens`: + + ```ruby + PersonalAccessToken.where(id: token_ids).update_all(revoked: true) + ``` + +Alternatively, combine these two steps: + +```ruby +PersonalAccessToken + .where(id: personal_access_token_chain(token).pluck_primary_key) + .update_all(revoked: true) +``` + +NOTE: +Avoid updating large volumes of unbounded data. If there are no [application limits](application_limits.md) on the data, or you are unsure about the data volume, you should [update the data in batches](database/iterating_tables_in_batches.md). diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 5a4ac99f5c5..65787f7a355 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -488,7 +488,9 @@ We collect information about tests duration in [`rspec_profiling_stats`](https:/ With [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/375983) we defined thresholds for tests duration that can act a guide. -For tests that are not meeting the thresholds it is recommended to create issues and improve the tests duration. +For tests that are not meeting the thresholds, we create [issues](https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=created_date&state=opened&label_name%5B%5D=rspec%3Aslow%20test&first_page_size=100) automatically in order to improve them. + +For tests that are slow for a legitimate reason and to skip issue creation, add `allowed_to_be_slow: true`. | Date | Feature tests | Controllers and Requests tests | Unit | Other | Method | | :-: | :-: | :-: | :-: | :-: | :-: | @@ -619,11 +621,11 @@ You can use the `be_axe_clean` matcher to run [axe automated accessibility testi ##### Externalized contents -Test expectations against externalized contents should call the same +For RSpec tests, expectations against externalized contents should call the same externalizing method to match the translation. For example, you should use the `_` -method in Ruby and `__` method in JavaScript. +method in Ruby. -See [Internationalization for GitLab - Test files](../i18n/externalization.md#test-files) for details. +See [Internationalization for GitLab - Test files (RSpec)](../i18n/externalization.md#test-files-rspec) for details. ##### Actions @@ -908,6 +910,9 @@ so we need to set some guidelines for their use going forward: ### Common test setup +NOTE: +`before_all` does not work with the `:delete` strategy. For more information, see [issue 420379](https://gitlab.com/gitlab-org/gitlab/-/issues/420379). + In some cases, there is no need to recreate the same object for tests again for each example. For example, a project and a guest of that project are needed to test issues on the same project, so one project and user are enough for the entire file. @@ -1027,6 +1032,58 @@ after :all do end ``` +#### Timestamp truncation + +Active Record timestamps are [set by the Rails’ `ActiveRecord::Timestamp`](https://github.com/rails/rails/blob/1eb5cc13a2ed8922b47df4ae47faf5f23faf3d35/activerecord/lib/active_record/timestamp.rb#L105) +module [using `Time.now`](https://github.com/rails/rails/blob/1eb5cc13a2ed8922b47df4ae47faf5f23faf3d35/activerecord/lib/active_record/timestamp.rb#L78). +Time precision is [OS-dependent](https://ruby-doc.org/core-2.6.3/Time.html#method-c-new), +and as the docs state, may include fractional seconds. + +When Rails models are saved to the database, +any timestamps they have are stored using a type in PostgreSQL called `timestamp without time zone`, +which has microsecond resolution—i.e., six digits after the decimal. +So if `1577987974.6472975` is sent to PostgreSQL, +it truncates the last digit of the fractional part and instead saves `1577987974.647297`. + +The results of this can be a simple test like: + +```ruby +let_it_be(:contact) { create(:contact) } + +data = Gitlab::HookData::IssueBuilder.new(issue).build + +expect(data).to include('customer_relations_contacts' => [contact.hook_attrs]) +``` + +Failing with an error along the lines of: + +```shell +expected { +"assignee_id" => nil, "...1 +0000 } to include {"customer_relations_contacts" => [{:created_at => "2023-08-04T13:30:20Z", :first_name => "Sidney Jones3" }]} + +Diff: + @@ -1,35 +1,69 @@ + -"customer_relations_contacts" => [{:created_at=>"2023-08-04T13:30:20Z", :first_name=>"Sidney Jones3" }], + +"customer_relations_contacts" => [{"created_at"=>2023-08-04 13:30:20.245964000 +0000, "first_name"=>"Sidney Jones3" }], +``` + +The fix is to ensure we `.reload` the object from the database to get the timestamp with correct precision: + +```ruby +let_it_be(:contact) { create(:contact) } + +data = Gitlab::HookData::IssueBuilder.new(issue).build + +expect(data).to include('customer_relations_contacts' => [contact.reload.hook_attrs]) +``` + +This explanation was taken from [a blog post](https://www.toptal.com/ruby-on-rails/timestamp-truncation-rails-activerecord-tale) +by Maciek Rząsa. + +You can see a [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/126530#note_1500580985) +where this problem arose and the [backend pairing session](https://www.youtube.com/watch?v=nMCjEeuYFDA) +where it was discussed. + ### Feature flags in tests This section was moved to [developing with feature flags](../feature_flags/index.md). diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index adf679c44a2..4e7ef6f29a2 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -18,6 +18,8 @@ together. We use [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) to build GitLab packages and then we test these packages using the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) tool to run the end-to-end tests located in the `qa` directory. +Additionally, we use the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) (GDK) as a test environment that can be deployed quickly for faster test feedback. + ### Testing nightly builds We run scheduled pipelines each night to test nightly builds created by Omnibus. @@ -43,10 +45,9 @@ Docker image built from your merge request's changes.** Manual action that starts end-to-end tests is also available in [`gitlab-org/omnibus-gitlab` merge requests](https://docs.gitlab.com/omnibus/build/team_member_docs.html#i-have-an-mr-in-the-omnibus-gitlab-project-and-want-a-package-or-docker-image-to-test-it). -#### How does it work? +##### How does it work? -Currently, we are using _multi-project pipeline_-like approach to run end-to-end -pipelines. +Currently, we are using _multi-project pipeline_-like approach to run end-to-end pipelines against Omnibus GitLab. ```mermaid graph TB @@ -98,7 +99,22 @@ work-around was suggested in <https://gitlab.com/gitlab-org/omnibus-gitlab/-/iss A feature proposal to segregate access control regarding running pipelines from ability to push/merge was also created at <https://gitlab.com/gitlab-org/gitlab/-/issues/24585>. For more technical details on CI/CD setup and documentation on adding new test jobs to `e2e:package-and-test` pipeline, see -[`e2e:package_and_test` setup documentation](package_and_test_pipeline.md). +[`e2e:package_and_test` setup documentation](test_pipelines.md). + +#### Using the `test-on-gdk` job + +The `e2e:test-on-gdk` job is run automatically in most merge requests, which triggers a [child-pipeline](../../../ci/pipelines/downstream_pipelines.md#parent-child-pipelines) +that builds and installs a GDK instance from your merge request's changes, and then executes end-to-end tests against that GDK instance. + +##### How does it work? + +In the [`gitlab-org/gitlab` pipeline](https://gitlab.com/gitlab-org/gitlab): + +1. The [`build-gdk-image` job](https://gitlab.com/gitlab-org/gitlab/-/blob/07504c34b28ac656537cd60810992aa15e9e91b8/.gitlab/ci/build-images.gitlab-ci.yml#L32) + uses the code from the merge request to build a Docker image for a GDK instance. +1. The `e2e:test-on-gdk` trigger job creates a child pipeline that executes the end-to-end tests against GDK instances launched from the image built in the previous job. + +For more details, see the [documentation for the `e2e:test-on-gdk` pipeline](test_pipelines.md#e2etest-on-gdk). #### With merged results pipelines diff --git a/doc/development/testing_guide/end_to_end/package_and_test_pipeline.md b/doc/development/testing_guide/end_to_end/package_and_test_pipeline.md index b0257e7b02c..240db2cbfe5 100644 --- a/doc/development/testing_guide/end_to_end/package_and_test_pipeline.md +++ b/doc/development/testing_guide/end_to_end/package_and_test_pipeline.md @@ -1,134 +1,11 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: 'test_pipelines.md' +remove_date: '2023-11-08' --- -# e2e:package-and-test +This document was moved to [another location](test_pipelines.md). -The `e2e:package-and-test` child pipeline is the main executor of E2E testing for the GitLab platform. The pipeline definition has several dynamic -components to reduce the number of tests being executed in merge request pipelines. - -## Setup - -Pipeline setup consists of: - -- The `e2e-test-pipeline-generate` job in the `prepare` stage of the main GitLab pipeline. -- The `e2e:package-and-test` job in the `qa` stage, which triggers the child pipeline that is responsible for building the `omnibus` package and - running E2E tests. - -### e2e-test-pipeline-generate - -This job consists of two components that implement selective test execution: - -- The [`detect_changes`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/tasks/ci.rake) Rake task determines which e2e specs should be executed - in a particular merge request pipeline. This task analyzes changes in a particular merge request and determines which specs must be executed. - Based on that, a `dry-run` of every [scenario](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test) executes and determines if a - scenario contains any executable tests. Selective test execution uses [these criteria](index.md#selective-test-execution) to determine which specific - tests to execute. -- [`generate-e2e-pipeline`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/generate-e2e-pipeline) is executed, which generates a child - pipeline YAML definition file with appropriate environment variables. - -### e2e:package-and-test - -E2E test execution pipeline consists of several stages which all support execution of E2E tests. - -#### .pre - -This stage is responsible for the following tasks: - -- Fetching `knapsack` reports that support [parallel test execution](index.md#run-tests-in-parallel). -- Triggering downstream pipeline which builds the [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) Docker image. - -#### test - -This stage runs e2e tests against different types of GitLab configurations. The number of jobs executed is determined dynamically by -[`e2e-test-pipeline-generate`](package_and_test_pipeline.md#e2e-test-pipeline-generate) job. - -#### report - -This stage is responsible for [allure test report](index.md#allure-report) generation. - -## Adding new jobs - -Selective test execution depends on a set of rules present in every job definition. A typical job contains the following attributes: - -```yaml -variables: - QA_SCENARIO: Test::Integration::MyNewJob -rules: - - !reference [.rules:test:qa, rules] - - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ - - !reference [.rules:test:manual, rules] -``` - -In this example: - -- `QA_SCENARIO: Test::Integration::MyNewJob`: name of the scenario class that is passed to the - [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md) executor. -- `!reference [.rules:test:qa, rules]`: main rule definition that is matched for pipelines that should execute all tests. For example, when changes to - `qa` framework are present. -- `if: $QA_SUITES =~ /Test::Integration::MyNewJob/`: main rule responsible for selective test execution. `QA_SUITE` is the name of the scenario - abstraction located in [`qa framework`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test). - - `QA_SUITE` is not the same as `QA_SCENARIO`, which is passed to the `gitlab-qa` executor. For consistency, it usually has the same name. `QA_SUITE` - abstraction class usually contains information on what tags to run and optionally some additional setup steps. -- `!reference [.rules:test:manual, rules]`: final rule that is always matched and sets the job to `manual` so it can still be executed on demand, - even if not set to execute by selective test execution. - -Considering example above, perform the following steps to create a new job: - -1. Create new scenario type `my_new_job.rb` in the [`integration`](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/master/lib/gitlab/qa/scenario/test/integration) directory - of the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa) project and release new version so it's generally available. -1. Create new scenario `my_new_job.rb` in [`integration`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test/integration) directory of the - [`qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa) framework. In the most simple case, this scenario would define RSpec tags that should be executed: - - ```ruby - module QA - module Scenario - module Test - module Integration - class MyNewJob < Test::Instance::All - tags :some_special_tag - end - end - end - end - end - ``` - -1. Add new job definition in the [`main.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/package-and-test/main.gitlab-ci.yml) pipeline definition: - - ```yaml - ee:my-new-job: - extends: .qa - variables: - QA_SCENARIO: Test::Integration::MyNewJob - rules: - - !reference [.rules:test:qa, rules] - - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ - - !reference [.rules:test:manual, rules] - ``` - -### Parallel jobs - -For selective execution to work correctly with job types that require running multiple parallel jobs, -a job definition typically must be split into parallel and selective variants. Splitting is necessary so that when selective execution -executes only a single spec, multiple unnecessary jobs are not spawned. For example: - -```yaml -ee:my-new-job-selective: - extends: .qa - variables: - QA_SCENARIO: Test::Integration::MyNewJob - rules: - - !reference [.rules:test:qa-selective, rules] - - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ -ee:my-new-job: - extends: - - .parallel - - ee:my-new-job-selective - rules: - - !reference [.rules:test:qa-parallel, rules] - - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ -``` +<!-- This redirect file can be deleted after <2023-11-08>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index f9e136a86df..becdc375c63 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -347,7 +347,7 @@ end ## Creating resources in your tests To create a resource in your tests, you can call the `.fabricate!` method on -the resource class. +the resource class, or use the [factory](#factories) to create it. Note that if the resource class supports API fabrication, this uses this fabrication by default. @@ -390,6 +390,23 @@ end In this case, the result is similar to calling `Resource::Shirt.fabricate!`. +### Factories + +You may also use FactoryBot invocations to create resources within your tests. + +```ruby +# create a project via the API to use in the test +let(:project) { create(:project) } + +# create an issue belonging to a project via the API to use in the test +let(:issue) { create(:issue, project: project) } + +# create a private project via the API with a specific name +let(:project) { create(:project, :private, name: 'my-project-name', add_name_uuid: false) } +``` + +All factories are defined in [`qa/qa/factories`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/factories/). + ### Resources cleanup We have a mechanism to [collect](https://gitlab.com/gitlab-org/gitlab/-/blob/44345381e89d6bbd440f7b4c680d03e8b75b86de/qa/qa/tools/test_resource_data_processor.rb#L32) diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index f13686981a9..93adc492d5a 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -15,6 +15,7 @@ This is a partial list of the [RSpec metadata](https://rspec.info/features/3-12/ |-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. | | `:except` | The test is to be run in their typical execution contexts _except_ as specified. See [test execution context selection](execution_context_selection.md) for more information. | +| `:external_api_calls` | The test requires interaction with a network external to the Docker network | | `:feature_flag` | The test uses a feature flag and therefore requires an administrator account to run. When `scope` is set to `:global`, the test will be skipped on all live .com environments. Otherwise, it will be skipped only on Canary, Production, and Pre-production. See [testing with feature flags](../../../development/testing_guide/end_to_end/feature_flags.md) for more details. | | `:framework` | The test makes sanity assertions around the QA framework itself | | `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. | diff --git a/doc/development/testing_guide/end_to_end/test_pipelines.md b/doc/development/testing_guide/end_to_end/test_pipelines.md new file mode 100644 index 00000000000..b47b75e398a --- /dev/null +++ b/doc/development/testing_guide/end_to_end/test_pipelines.md @@ -0,0 +1,190 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# End-to-end test pipelines + +## e2e:package-and-test + +The `e2e:package-and-test` child pipeline is the main executor of E2E testing for the GitLab platform. The pipeline definition has several dynamic +components to reduce the number of tests being executed in merge request pipelines. + +### Setup + +Pipeline setup consists of: + +- The `e2e-test-pipeline-generate` job in the `prepare` stage of the main GitLab pipeline. +- The `e2e:package-and-test` job in the `qa` stage, which triggers the child pipeline that is responsible for building the `omnibus` package and + running E2E tests. + +#### e2e-test-pipeline-generate + +This job consists of two components that implement selective test execution: + +- The [`detect_changes`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/tasks/ci.rake) Rake task determines which e2e specs should be executed + in a particular merge request pipeline. This task analyzes changes in a particular merge request and determines which specs must be executed. + Based on that, a `dry-run` of every [scenario](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test) executes and determines if a + scenario contains any executable tests. Selective test execution uses [these criteria](index.md#selective-test-execution) to determine which specific + tests to execute. +- [`generate-e2e-pipeline`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/generate-e2e-pipeline) is executed, which generates a child + pipeline YAML definition file with appropriate environment variables. + +#### e2e:package-and-test + +E2E test execution pipeline consists of several stages which all support execution of E2E tests. + +##### .pre + +This stage is responsible for the following tasks: + +- Fetching `knapsack` reports that support [parallel test execution](index.md#run-tests-in-parallel). +- Triggering downstream pipeline which builds the [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) Docker image. + +##### test + +This stage runs e2e tests against different types of GitLab configurations. The number of jobs executed is determined dynamically by +[`e2e-test-pipeline-generate`](test_pipelines.md#e2e-test-pipeline-generate) job. + +##### report + +This stage is responsible for [allure test report](index.md#allure-report) generation. + +### Adding new jobs + +Selective test execution depends on a set of rules present in every job definition. A typical job contains the following attributes: + +```yaml +variables: + QA_SCENARIO: Test::Integration::MyNewJob +rules: + - !reference [.rules:test:qa, rules] + - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ + - !reference [.rules:test:manual, rules] +``` + +In this example: + +- `QA_SCENARIO: Test::Integration::MyNewJob`: name of the scenario class that is passed to the + [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md) executor. +- `!reference [.rules:test:qa, rules]`: main rule definition that is matched for pipelines that should execute all tests. For example, when changes to + `qa` framework are present. +- `if: $QA_SUITES =~ /Test::Integration::MyNewJob/`: main rule responsible for selective test execution. `QA_SUITE` is the name of the scenario + abstraction located in [`qa framework`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test). + + `QA_SUITE` is not the same as `QA_SCENARIO`, which is passed to the `gitlab-qa` executor. For consistency, it usually has the same name. `QA_SUITE` + abstraction class usually contains information on what tags to run and optionally some additional setup steps. +- `!reference [.rules:test:manual, rules]`: final rule that is always matched and sets the job to `manual` so it can still be executed on demand, + even if not set to execute by selective test execution. + +Considering example above, perform the following steps to create a new job: + +1. Create new scenario type `my_new_job.rb` in the [`integration`](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/master/lib/gitlab/qa/scenario/test/integration) directory + of the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa) project and release new version so it's generally available. +1. Create new scenario `my_new_job.rb` in [`integration`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test/integration) directory of the + [`qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa) framework. In the most simple case, this scenario would define RSpec tags that should be executed: + + ```ruby + module QA + module Scenario + module Test + module Integration + class MyNewJob < Test::Instance::All + tags :some_special_tag + end + end + end + end + end + ``` + +1. Add new job definition in the [`main.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/package-and-test/main.gitlab-ci.yml) pipeline definition: + + ```yaml + ee:my-new-job: + extends: .qa + variables: + QA_SCENARIO: Test::Integration::MyNewJob + rules: + - !reference [.rules:test:qa, rules] + - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ + - !reference [.rules:test:manual, rules] + ``` + +#### Parallel jobs + +For selective execution to work correctly with job types that require running multiple parallel jobs, +a job definition typically must be split into parallel and selective variants. Splitting is necessary so that when selective execution +executes only a single spec, multiple unnecessary jobs are not spawned. For example: + +```yaml +ee:my-new-job-selective: + extends: .qa + variables: + QA_SCENARIO: Test::Integration::MyNewJob + rules: + - !reference [.rules:test:qa-selective, rules] + - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ +ee:my-new-job: + extends: + - .parallel + - ee:my-new-job-selective + rules: + - !reference [.rules:test:qa-parallel, rules] + - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ +``` + +## `e2e:test-on-gdk` + +The `e2e:test-on-gdk` child pipeline supports development of the GitLab platform by providing feedback to engineers on +end-to-end test execution faster than via `e2e:package-and-test` or [Review Apps](../review_apps.md). + +This is achieved by running tests against the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) (GDK), +which can be built and installed in less time than when testing against [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab). +The trade-off is that Omnibus GitLab can be used to deploy a production installation, whereas the GDK is a development +environment. Tests that run against the GDK might not catch bugs that depend on part of the process of preparing GitLab +to run in a production environment, including pre-compiling assets, assigning configuration defaults as part of an official +installation package, deploying GitLab services to multiple servers, and more. On the other hand, engineers who use the +GDK day-to-day can benefit from automated tests catching bugs that only appear on the GDK. + +### Setup + +The pipeline setup consists of several jobs in the main GitLab pipeline: + +- The [`e2e-test-pipeline-generate` job](https://gitlab.com/gitlab-org/gitlab/-/blob/9456299b995084bfceb8bc6d082229c0198a0f72/.gitlab/ci/setup.gitlab-ci.yml#L158) + in the `prepare` stage. This is the same job as in the [`e2e:package-and-test`](#e2epackage-and-test) pipeline. +- The [`build-gdk-image` job](https://gitlab.com/gitlab-org/gitlab/-/blob/07504c34b28ac656537cd60810992aa15e9e91b8/.gitlab/ci/build-images.gitlab-ci.yml#L32) + in the `build-images` stage. +- The `e2e:test-on-gdk` trigger job in the `qa` stage, which triggers the child pipeline that runs E2E tests. + +#### `build-gdk-image` + +[This job](https://gitlab.com/gitlab-org/gitlab/-/blob/07504c34b28ac656537cd60810992aa15e9e91b8/.gitlab/ci/build-images.gitlab-ci.yml#L32) +uses the code from the merge request to build a Docker image that can be used in test jobs to launch a GDK instance in a container. The image is pushed to the Container Registry. + +The job also runs in pipelines on the default branch to build a base image that includes the GDK and GitLab components. +This avoids building the entire image from scratch in merge requests. However, if the merge request includes changes to +[certain GitLab components or code](https://gitlab.com/gitlab-org/gitlab/-/blob/24109c1a7ae1f29d4f6f1aeba3a13cbd8ea0e8e6/.gitlab/ci/rules.gitlab-ci.yml#L911) +the job will rebuild the base image before building the image that will be used in the test jobs. + +#### `e2e:test-on-gdk` child pipeline + +Like the [`e2e:package-and-test`](#e2epackage-and-test) pipeline, the `e2e:test-on-gdk` pipeline consists of several stages +that support execution of E2E tests. + +##### .pre + +This stage is responsible for fetching `knapsack` reports that support [parallel test execution](index.md#run-tests-in-parallel). + +##### test + +This stage runs e2e tests against different types of GitLab configurations. The number of jobs executed is determined dynamically by the +[`e2e-test-pipeline-generate`](test_pipelines.md#e2e-test-pipeline-generate) job. + +Each job starts a container from the GDK Docker image created in the `build-gdk-image` job, and then executes the end-to-end +tests against the GDK instance running in the container. + +##### report + +This stage is responsible for [allure test report](index.md#allure-report) generation. diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 1b50bd410c2..8da4350074d 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -228,16 +228,13 @@ it('exists', () => { // Bad wrapper.find('.js-foo'); wrapper.find('.btn-primary'); - wrapper.find('.qa-foo-component'); }); ``` -It is recommended to use `kebab-case` for `data-testid` attribute. +You should use `kebab-case` for `data-testid` attribute. It is not recommended that you add `.js-*` classes just for testing purposes. Only do this if there are no other feasible options available. -Do not use `.qa-*` class attributes for any tests other than QA end-to-end testing. - ### Querying for child components When testing Vue components with `@vue/test-utils` another possible approach is querying for child @@ -387,7 +384,7 @@ If your tests require `window.location.href` to take a particular value, use the `setWindowLocation` helper: ```javascript -import setWindowLocation from 'helpers/set_window_location'; +import setWindowLocation from 'helpers/set_window_location_helper'; it('passes', () => { setWindowLocation('https://gitlab.test/foo?bar=true'); diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 38fda1bb742..68d30ff6656 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -10,8 +10,8 @@ This document describes various guidelines and best practices for automated testing of the GitLab project. It is meant to be an _extension_ of the -[Thoughtbot testing style guide](https://github.com/thoughtbot/guides/tree/master/testing-rspec). If -this guide defines a rule that contradicts the Thoughtbot guide, this guide +[thoughtbot testing style guide](https://github.com/thoughtbot/guides/tree/master/testing-rspec). If +this guide defines a rule that contradicts the thoughtbot guide, this guide takes precedence. Some guidelines may be repeated verbatim to stress their importance. diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 480a53bbefe..4e4dc671c03 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -197,16 +197,8 @@ graph RL #### What to mock in component tests -- **DOM**: - Operating on the real DOM is significantly slower than on the virtual DOM. -- **Properties and state of the component under test**: - Similar to testing classes, modifying the properties directly (rather than relying on methods of the component) avoids side effects. -- **Vuex store**: - To avoid side effects and keep component tests simple, Vuex stores are replaced with mocks. -- **All server requests**: - Similar to unit tests, when running component tests, the backend may not be reachable, so all outgoing requests need to be mocked. -- **Asynchronous background operations**: - Similar to unit tests, background operations cannot be stopped or waited on. This means they continue running in the following tests and cause side effects. +- **Side effects**: + Anything that can change external state (for example, a network request) should be mocked. - **Child components**: Every component is tested individually, so child components are mocked. See also [`shallowMount()`](https://v1.test-utils.vuejs.org/api/#shallowmount) @@ -215,8 +207,10 @@ graph RL - **Methods or computed properties of the component under test**: By mocking part of the component under test, the mocks are tested and not the real component. -- **Functions and classes independent from Vue**: - All plain JavaScript code is already covered by unit tests and needs not to be mocked in component tests. +- **Vuex**: + Keep Vuex unmocked to avoid fragile and false-positive tests. + Set the Vuex to a proper state using mutations. + Mock the side-effects, not the Vuex actions. ## Integration tests diff --git a/doc/development/testing_guide/testing_migrations_guide.md b/doc/development/testing_guide/testing_migrations_guide.md index f69f0d1febb..abe04274b72 100644 --- a/doc/development/testing_guide/testing_migrations_guide.md +++ b/doc/development/testing_guide/testing_migrations_guide.md @@ -21,11 +21,15 @@ We don't enforce tests on post migrations that only perform schema changes. ## How does it work? -Adding a `:migration` tag to a test signature enables some custom RSpec +All specs in `(ee/)spec/migrations/` and `spec/lib/(ee/)background_migrations` are automatically +tagged with the `:migration` RSpec tag. This tag enables some custom RSpec `before` and `after` hooks in our [`spec/support/migration.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/f81fa6ab1dd788b70ef44b85aaba1f31ffafae7d/spec/support/migration.rb) to run. If performing a migration against a database schema other than -`:gitlab_main` (for example `:gitlab_ci`), then you must specify it as well: `migration: :gitlab_ci`. See [spec/migrations/change_public_projects_cost_factor_spec.rb](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/migrations/change_public_projects_cost_factor_spec.rb#L6-6) for an example. +`:gitlab_main` (for example `:gitlab_ci`), then you must explicitly specify it +with an RSpec tag like: `migration: :gitlab_ci`. See +[spec/migrations/change_public_projects_cost_factor_spec.rb](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/migrations/change_public_projects_cost_factor_spec.rb#L6-6) +for an example. A `before` hook reverts all migrations to the point that a migration under test is not yet migrated. diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md index 30244407e38..955fc88c713 100644 --- a/doc/development/value_stream_analytics.md +++ b/doc/development/value_stream_analytics.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Value stream analytics development guidelines -For information on how to configure value stream analytics (VSA) in GitLab, see our [analytics documentation](../user/analytics/value_stream_analytics.md). +For information on how to configure value stream analytics (VSA) in GitLab, see our [analytics documentation](../user/group/value_stream_analytics/index.md). ## How does Value Stream Analytics work? @@ -290,7 +290,7 @@ considered legacy, which will be phased out at some point. ## Frontend -[Project VSA](../user/analytics/value_stream_analytics.md) is available for all users and: +[Project VSA](../user/group/value_stream_analytics/index.md) is available for all users and: - Includes a mixture of key and DORA metrics based on the tier. - Uses the set of [default stages](#default-stages). diff --git a/doc/development/workhorse/configuration.md b/doc/development/workhorse/configuration.md index 84fb557d9ec..d19e85f3f7a 100644 --- a/doc/development/workhorse/configuration.md +++ b/doc/development/workhorse/configuration.md @@ -189,7 +189,9 @@ You can also set the `GITLAB_WORKHORSE_SENTRY_ENVIRONMENT` environment variable use the Sentry environment feature to separate staging, production and development. -Omnibus GitLab (`/etc/gitlab/gitlab.rb`): +::Tabs + +:::TabTitle Linux package (Omnibus) ```ruby gitlab_workhorse['env'] = { @@ -198,13 +200,15 @@ gitlab_workhorse['env'] = { } ``` -Source installations (`/etc/default/gitlab`): +:::TabTitle Self-compiled (source) ```plaintext export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar' export GITLAB_WORKHORSE_SENTRY_ENVIRONMENT='production' ``` +::EndTabs + ## Distributed tracing Workhorse supports distributed tracing through [LabKit](https://gitlab.com/gitlab-org/labkit/) |
