diff options
Diffstat (limited to 'doc/development')
138 files changed, 2714 insertions, 1089 deletions
diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index d29d9b9651f..3ce794514a3 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -40,6 +40,13 @@ GraphiQL is an interactive GraphQL API explorer where you can play around with e You can access it in any GitLab environment on `https://<your-gitlab-site.com>/-/graphql-explorer`. For example, the one for [GitLab.com](https://gitlab.com/-/graphql-explorer). +## Reviewing merge requests with GraphQL changes + +The GraphQL framework has some specific gotchas to be aware of, and domain expertise is required to ensure they are satisfied. + +If you are asked to review a merge request that modifies any GraphQL files or adds an endpoint, please have a look at +[our GraphQL review guide](graphql_guide/reviewing.md). + ## Authentication Authentication happens through the `GraphqlController`, right now this @@ -287,13 +294,13 @@ or by calling `#to_global_id` on an object that has mixed in the `GlobalID::Identification` module. Using an example from -[`Types::Notes::DiscussionType`](https://gitlab.com/gitlab-org/gitlab/-/blob/3c95bd9/app/graphql/types/notes/discussion_type.rb#L24-26): +[`Types::Notes::DiscussionType`](https://gitlab.com/gitlab-org/gitlab/-/blob/af48df44/app/graphql/types/notes/discussion_type.rb#L22-30): ```ruby -field :reply_id, GraphQL::Types::ID +field :reply_id, Types::GlobalIDType[Discussion] def reply_id - ::Gitlab::GlobalId.build(object, id: object.reply_id) + Gitlab::GlobalId.build(object, id: object.reply_id) end ``` @@ -785,7 +792,7 @@ The documentation mentions that the old Global ID style is now deprecated. ## Mark schema items as Alpha You can mark GraphQL schema items (fields, arguments, enum values, and mutations) as -[Alpha](../policy/alpha-beta-support.md#alpha-features). +[Alpha](../policy/alpha-beta-support.md#experiment). An item marked as Alpha is [exempt from the deprecation process](#breaking-change-exemptions) and can be removed at any time without notice. Mark an item as Alpha when it is @@ -1132,7 +1139,9 @@ you need to do something more custom however, remember, if you encounter an object the `current_user` does not have access to when resolving a field, then the entire field should resolve to `null`. -### Deriving resolvers (`BaseResolver.single` and `BaseResolver.last`) +### Deriving resolvers + +(including `BaseResolver.single` and `BaseResolver.last`) For some use cases, we can derive resolvers from others. The main use case for this is one resolver to find all items, and another to diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 828ef21ba9a..34e043a30d6 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -338,12 +338,13 @@ Also see [verifying N+1 performance](#verifying-with-tests) in tests. When throwing an error with a message that is meant to be user-facing, you should use the error message utility function contained in `lib/gitlab/utils/error_message.rb`. It adds a prefix to the error message, making it distinguishable from non-user-facing error messages. -Please make sure that the Frontend is aware of the prefix usage and is using the according utils. ```ruby Gitlab::Utils::ErrorMessage.to_user_facing('Example user-facing error-message') ``` +Please make sure that the Frontend is aware of the prefix usage and is using the according utils. See [Error handling](fe_guide/style/javascript.md#error-handling) in JavaScript style guide for more information. + ## Include a changelog entry All client-facing changes **must** include a [changelog entry](changelog.md). diff --git a/doc/development/application_secrets.md b/doc/development/application_secrets.md index 526cc6c3f61..e640a94f0a6 100644 --- a/doc/development/application_secrets.md +++ b/doc/development/application_secrets.md @@ -17,7 +17,7 @@ This page is a development guide for application secrets. |`db_key_base` | The base key to encrypt the data for `attr_encrypted` columns | |`openid_connect_signing_key` | The signing key for OpenID Connect | | `encrypted_settings_key_base` | The base key to encrypt settings files with | -| `ci_jwt_signing_key` | The base key for encrypting the `CI_JOB_JWT` and `CI_JOB_JWT_V2` predefined CI/CD variables | +| `ci_jwt_signing_key` | The base key for encrypting the `CI_JOB_JWT` and `CI_JOB_JWT_V2` predefined CI/CD variables. `CI_JOB_JWT` and `CI_JOB_JWT_V2` were [deprecated in GitLab 15.9](../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and are scheduled to be removed in GitLab 16.0. | ## Where the secrets are stored diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 7a9c3572406..d1e16b7f137 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -558,7 +558,7 @@ GitLab CI/CD is the open-source continuous integration service included with Git #### GitLab Workhorse -- [Project page](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/README.md) +- [Project page](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/workhorse/index.md) - Configuration: - [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) - [Charts](https://docs.gitlab.com/charts/charts/gitlab/webservice/) @@ -567,7 +567,7 @@ GitLab CI/CD is the open-source continuous integration service included with Git - Process: `gitlab-workhorse` - GitLab.com: [Service Architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#service-architecture) -[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from Puma. You can read more about the [historical reasons for developing](https://about.gitlab.com/blog/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/workhorse) is a program designed at GitLab to help alleviate pressure from Puma. You can read more about the [historical reasons for developing](https://about.gitlab.com/blog/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. #### Grafana diff --git a/doc/development/auto_devops.md b/doc/development/auto_devops.md index b2281f9c032..1f98a37ac9d 100644 --- a/doc/development/auto_devops.md +++ b/doc/development/auto_devops.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/chatops_on_gitlabcom.md b/doc/development/chatops_on_gitlabcom.md index 378639c4a43..002470f7325 100644 --- a/doc/development/chatops_on_gitlabcom.md +++ b/doc/development/chatops_on_gitlabcom.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/code_owners/index.md b/doc/development/code_owners/index.md index fa0385777e8..d962a36b497 100644 --- a/doc/development/code_owners/index.md +++ b/doc/development/code_owners/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219916) in GitLab 15.10. This document was created to help contributors understand the code design of -[Code Owners](../../user/project/code_owners.md). You should read this +[Code Owners](../../user/project/codeowners/index.md). You should read this document before making changes to the code for this feature. This document is intentionally limited to an overview of how the code is @@ -28,6 +28,15 @@ namespace. Code Owners is an EE-only feature, so the files only exist in the `./ - `Gitlab::CodeOwners`: the main module used to interact with the code owner rules. - Defined in `./ee/lib/gitlab/code_owners.rb`. +- `Gitlab::CodeOwners::File`: wraps a `CODEOWNERS` file and exposes the data through + the class' public methods. + - Defined in `./ee/lib/gitlab/code_owners/file.rb`. +- `Gitlab::CodeOwners::Section`: wraps a section heading from a + `CODEOWNERS` file and parses the different parts. + - Defined in `./ee/lib/gitlab/code_owners/section.rb`. +- `Gitlab::CodeOwners::Entry`: wraps an entry (a pattern and owners line) in a + `CODEOWNERS` file and exposes the data through the class' public methods. + - Defined in `./ee/lib/gitlab/code_owners/entry.rb`. - `Gitlab::CodeOwners::Loader`: finds the correct `CODEOWNER` file and loads the content into a `Gitlab::CodeOwners::File` instance. - Defined in `./ee/lib/gitlab/code_owners/loader.rb`. @@ -40,12 +49,6 @@ namespace. Code Owners is an EE-only feature, so the files only exist in the `./ - `Gitlab::CodeOwners::GroupsLoader`: finds the correct `CODEOWNER` file and loads the content into a `Gitlab::CodeOwners::File` instance. - Defined in `./ee/lib/gitlab/code_owners/groups_loader.rb`. -- `Gitlab::CodeOwners::File`: wraps a `CODEOWNERS` file and exposes the data through - the class' public methods. - - Defined in `./ee/lib/gitlab/code_owners/file.rb`. -- `Gitlab::CodeOwners::Entry`: wraps an entry (a pattern and owners line) in a - `CODEOWNERS` file and exposes the data through the class' public methods. - - Defined in `./ee/lib/gitlab/code_owners/entry.rb`. - `Gitlab::CodeOwners::Validator`: validates no files in the `CODEOWNERS` entries have been changed when a user pushes to a protected branch with `require_code_owner_approval` enabled. - Defined in `./ee/lib/gitlab/code_owners/validator.rb`. diff --git a/doc/development/code_review.md b/doc/development/code_review.md index ed9404187ae..5588afe0713 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -546,7 +546,9 @@ people who add commits to an MR are not authorized to approve or merge the MR an must seek a maintainer who has not contributed to the MR to approve and merge it. This policy is in place to satisfy the CHG-04 control of the GitLab -[Change Management Controls](https://about.gitlab.com/handbook/security/security-assurance/security-compliance/guidance/change-management.html). +[Change Management Controls](https://about.gitlab.com/handbook/security/change-management-policy.html). + +<!-- Or should it link to: https://about.gitlab.com/handbook/engineering/infrastructure/change-management/ ? --> To implement this policy in `gitlab-org/gitlab`, we have enabled the following settings to ensure MRs get an approval from a top-level CODEOWNERS maintainer: diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index 210370f9816..d68bc194266 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -73,7 +73,7 @@ Check visual design properties using your browser's _elements inspector_ ([Chrom guidelines. - _Optionally_ consider [dark mode](../../user/profile/preferences.md#dark-mode). [^1] - [^1]: You're not required to design for [dark mode](../../user/profile/preferences.md#dark-mode) while the feature is in [alpha](../../policy/alpha-beta-support.md#alpha-features). The [UX Foundations team](https://about.gitlab.com/direction/manage/foundations/) plans to improve the dark mode in the future. Until we integrate [Pajamas](https://design.gitlab.com/) components into the product and the underlying design strategy is in place to support dark mode, we cannot guarantee that we won't introduce bugs and debt to this mode. At your discretion, evaluate the need to create dark mode patches. + [^1]: You're not required to design for [dark mode](../../user/profile/preferences.md#dark-mode) while the feature is an [Experiment](../../policy/alpha-beta-support.md#experiment). The [UX Foundations team](https://about.gitlab.com/direction/manage/foundations/) plans to improve the dark mode in the future. Until we integrate [Pajamas](https://design.gitlab.com/) components into the product and the underlying design strategy is in place to support dark mode, we cannot guarantee that we won't introduce bugs and debt to this mode. At your discretion, evaluate the need to create dark mode patches. ### States diff --git a/doc/development/contributing/first_contribution.md b/doc/development/contributing/first_contribution.md new file mode 100644 index 00000000000..0c4b5b21171 --- /dev/null +++ b/doc/development/contributing/first_contribution.md @@ -0,0 +1,340 @@ +--- +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Tutorial: Make a GitLab contribution + +Anyone can contribute to the development of GitLab. + +Maybe you want to add functionality that you feel is missing. Or maybe +you noticed some UI text that you want to improve. + +This tutorial will walk you through the process of updating UI text +and related files by using the GitLab Development Kit and the GitLab community fork. +You can follow this example exactly to familiarize yourself with the process, +or you can choose other UI text to update. + +## Steps + +To make a contribution, you will: + +1. [Configure the GitLab Development Kit](#step-1-configure-the-gitlab-development-kit) +1. [Make your code updates](#step-2-change-the-code) +1. [Push your changes to the community fork](#step-3-push-your-changes-to-the-community-fork) +1. [Create a merge request](#step-4-create-a-merge-request) + +## Prerequisites + +On your local machine: + +- Ensure Git is installed. + (From the command line, type `git -v`. If you get a result, you have Git installed.) +- Install a source code editor, or decide which tool you're going to use to edit files. + +On GitLab.com: + +- Create an account. Ensure you can successfully sign in. +- Go to the [`gitlab-community/community-members` group](https://gitlab.com/gitlab-community/community-members) + and select **Request access**. This action will give you access to the GitLab + community fork, where you'll author your changes. + +## Step 1: Configure the GitLab Development Kit + +The GitLab Development Kit (GDK) is a local version of GitLab that's yours to play with. +It's just like an installation of self-managed GitLab. It includes sample projects you +can use to test functionality, and it gives you access to administrator functionality. +You can run it on your local machine, or use GitPod to run a remote version. + + + +If you've never used the GDK before, and you think you might contribute +more than once, you should install it. +If you already have a working GDK, you should +[update it to use the community fork](#an-existing-gdk-installation). + +### A new GDK installation + +Set aside about two hours to install the GDK. If all goes smoothly, it +should take about an hour to install. + +Sometimes the installation needs some tweaks to make it work, so you should +also set aside some time for troubleshooting. +It might seem like a lot of work, but after you have the GDK running, +you'll be able to contribute much more often and more easily. + +To install the GDK: + +1. Ensure you're on + [one of the supported platforms](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/main/#supported-platforms) + (macOS, Ubuntu, etc.). +1. Choose the directory where you want to install the GDK. + In this location, a repository called `gitlab-development-kit` will be created, + and the application will be installed. +1. From the command line, go to that directory. + In this example, we will use the `development` directory. + + ```shell + cd development + ``` + +1. Run the one-line installation command: + + ```shell + curl "https://gitlab.com/gitlab-org/gitlab-development-kit/-/raw/main/support/install" | bash + ``` + +1. For the message `Where would you like to install the GDK? [./gitlab-development-kit]`, + press Enter to accept the default location. + +1. For the message `Which GitLab repo URL would you like to clone?`, enter the GitLab community fork: + + ```shell + https://gitlab.com/gitlab-community/gitlab.git + ``` + +While the installation is running, copy any messages that are displayed. +If you have any problems with the installation, you can use this output as +part of troubleshooting. + +When the installation is complete: + +1. Go to the directory where the GDK was installed: + + ```shell + cd gitlab-development-kit + ``` + +1. Start the GDK: + + ```shell + gdk start + ``` + +1. Connect to the GDK by using the URL provided. It should be something like <http://127.0.0.1:3000>. +1. Use the username `root` and the password `5iveL!fe`. You will be prompted + to reset your password the first time you sign in. + +If you have any problems, try going to the `gitlab-development-kit/gitlab` +directory and running these commands: + +```shell +yarn install && bundle install +bundle exec rails db:migrate RAILS_ENV=development +``` + +From the `gitlab-development-kit` folder, you can also try running `gdk doctor`. + +For more advanced troubleshooting, see +[the troubleshooting docs](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/main/doc/troubleshooting). + +### An existing GDK installation + +If you have an existing GDK installation, you should update it so it's +using the community fork. + +1. Delete the existing `gitlab-development-kit/gitlab` directory. +1. Clone the community fork into that location: + + ```shell + cd gitlab-development-kit + git clone https://gitlab.com/gitlab-community/gitlab.git + ``` + +To confirm it was successful: + +1. Ensure the `gitlab-development-kit/gitlab` directory exists. +1. Go to the top `gitlab-development-kit` directory and run `gdk stop` and `gdk start`. + +If you get errors, run `gdk doctor` to troubleshoot. For more advanced troubleshooting, see +[the troubleshooting docs](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/main/doc/troubleshooting). + +## Step 2: Change the code + +Now for the fun part. Let's edit some code. + +In this example, I found some UI text I'd like to change. +In the upper-right corner in GitLab, I selected my avatar and then **Preferences**. +I want to change this text: + + + +Other settings on the page start with the word `Customize` and skip the `This setting allows you to` part. +I'll update this phrase to match the others. + +1. Search the `gitlab-development-kit/gitlab` directory for the string `This setting allows you to customize`. + + The results show one `.haml` file, two `.md` files, one `.pot` file, and + several `.po` files. + +1. Open the `.haml` file. This file is where the UI text resides. +1. Update the string. In this case, I'll remove the words before `customize` + and start the word `customize` with a capital `C`. +1. Save the file. + +You can check that you were successful: + +- In the `gitlab-development-kit/gitlab` directory, type `git status` + to show the file you modified: + + ```shell + modified: app/views/profiles/preferences/show.html.haml + ``` + +- Refresh the web browser where you're viewing the GDK. + The changes should be displayed. Take a screenshot. + +  + +### Update the translation files + +English UI strings are localized into many languages. +These strings are saved in a `.pot` file, which you must update +any time you update UI text. + +To generate the localization file: + +1. Ensure you are in the `gitlab-development-kit/gitlab` directory. +1. Run the following command: + + ```shell + bin/rake gettext:compile + ``` + + After several minutes, a `.pot` file is generated in the `/locale` directory. + +Now, in the `gitlab-development-kit/gitlab` directory, if you type `git status` +you should have both files listed: + +```shell + modified: app/views/profiles/preferences/show.html.haml + modified: locale/gitlab.pot +``` + +For more information about localization, see [internationalization](../i18n/externalization.md). + +### Update the documentation + +Documentation for GitLab is published on <https://docs.gitlab.com>. +When you add or update a feature, you must update the docs as well. + +1. To find the documentation for a feature, the easiest thing is to search the + docs site. In this case, the setting is described on this documentation page: + + ```plaintext + https://docs.gitlab.com/ee/user/profile/preferences.html + ``` + +1. The URL shows you the location of the file in the `/doc` directory. + In this case, the location is: + + ```plaintext + doc/user/profile/preferences.md + ``` + +1. Go to this location in your local `gitlab` repository and update the `.md` file + and any related images. + +Now when you run `git status`, you should have something like: + +```plaintext + modified: app/views/profiles/preferences/show.html.haml + modified: doc/user/profile/img/profile-preferences-syntax-themes.png + modified: doc/user/profile/preferences.md + modified: locale/gitlab.pot +``` + +To view these changes in action, you can +[check out a merge request where these changes have already been made](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116472). + +## Step 3: Push your changes to the community fork + +Now you're going to push your changes to the community fork. This is the next step +in getting your changes put into the main GitLab repository. + +1. Ensure you are in the `gitlab-development-kit/gitlab` directory. +1. Create a branch. You don't want to work in the `master` branch. + Instead, you want to create a branch for your work. In this example, + we're going to call the branch `ui-updates`. + + ```shell + git checkout -b ui-updates + ``` + +1. Add the files to the staging area. + + ```shell + git add . + ``` + +1. Provide a commit message. GitLab has somewhat strict + [commit message guidelines](merge_request_workflow.md#commit-messages-guidelines). + To be safe, a general rule is to use three to five words, start with a capital letter, + and do **not** end with a period. + + ```shell + git commit -m "Updating UI text + + Standardizing the text on this page so + that each area uses consistent language. + + Changelog: changed" + ``` + + The `Changelog: changed` is because we're changing an existing feature. If we were adding a feature, we'd + use `Changelog: added`. For details, see [changelog entries](../changelog.md). + +1. Push the changes to the community fork. At the same time, set the fork as your upstream, + so that it will be in sync for any future contributions. + + ```shell + git push --set-upstream origin ui-updates + ``` + +## Step 4: Create a merge request + +Now you're ready to push changes from the community fork to the main GitLab repository! + +1. Go to [the community fork on GitLab.com](https://gitlab.com/gitlab-community/gitlab). + You should see a message like this one: + +  + + Select **Create merge request**. + If you don't see this message, on the left sidebar, select **Merge requests > New merge request**. + +1. Take a look at the branch names. You should be merging from your branch + in the community fork to the `master` branch in the GitLab repository. + +  + +1. Fill out the information and then select **Save changes**. + Don't worry if your merge request is not complete. If you don't want anyone + from GitLab to review it, you can select the **Mark as draft** checkbox. + If you're not happy with the merge request after you create it, you can + close it, no harm done. + +1. Select the **Changes** tab. It should look something like this: + +  + + The red text shows the code before you made changes. The green shows what + the code looks like now. + +1. If you're happy with this merge request and want to start the review process, type + `@gitlab-bot ready` in a comment and then select **Comment**. + +  + +Someone from GitLab will look at your request and let you know what the next steps are. + +Now, any time you want to make a contribution to GitLab, you can just +go to the `gitlab-development-kit` folder and run `gdk update`. Then make +your changes in the `gitlab` directory and push them to the fork. + +If you need help at any point in the process, type `@gitlab-bot help` in a comment +or initiate a [mentor session](https://about.gitlab.com/community/contribute/mentor-sessions/) +on [Discord](https://discord.gg/gitlab). + +Congratulations on submitting your request, and thank you for your contribution! diff --git a/doc/development/contributing/img/bot_ready.png b/doc/development/contributing/img/bot_ready.png Binary files differnew file mode 100644 index 00000000000..85116c8957b --- /dev/null +++ b/doc/development/contributing/img/bot_ready.png diff --git a/doc/development/contributing/img/changes_tab.png b/doc/development/contributing/img/changes_tab.png Binary files differnew file mode 100644 index 00000000000..2158e9dd183 --- /dev/null +++ b/doc/development/contributing/img/changes_tab.png diff --git a/doc/development/contributing/img/gdk_home.png b/doc/development/contributing/img/gdk_home.png Binary files differnew file mode 100644 index 00000000000..77505850609 --- /dev/null +++ b/doc/development/contributing/img/gdk_home.png diff --git a/doc/development/contributing/img/mr_button.png b/doc/development/contributing/img/mr_button.png Binary files differnew file mode 100644 index 00000000000..5bf08b44412 --- /dev/null +++ b/doc/development/contributing/img/mr_button.png diff --git a/doc/development/contributing/img/new_merge_request.png b/doc/development/contributing/img/new_merge_request.png Binary files differnew file mode 100644 index 00000000000..34512d06dd2 --- /dev/null +++ b/doc/development/contributing/img/new_merge_request.png diff --git a/doc/development/contributing/img/ui_text_after.png b/doc/development/contributing/img/ui_text_after.png Binary files differnew file mode 100644 index 00000000000..3a54e7ef653 --- /dev/null +++ b/doc/development/contributing/img/ui_text_after.png diff --git a/doc/development/contributing/img/ui_text_before.png b/doc/development/contributing/img/ui_text_before.png Binary files differnew file mode 100644 index 00000000000..afd75a11ac2 --- /dev/null +++ b/doc/development/contributing/img/ui_text_before.png diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 6a9dd34e703..ef3242b89c2 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -8,10 +8,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Contribute to GitLab Thank you for your interest in contributing to GitLab. This guide details how -to contribute to GitLab in a way that is easy for everyone. +to contribute to the development of GitLab in a way that is easy for everyone. -For a first-time step-by-step guide to the contribution process, see our -[Contributing to GitLab](https://about.gitlab.com/community/contribute/) page. +For a first-time step-by-step guide to the contribution process for development, see [Tutorial: Make a GitLab contribution](first_contribution.md). + +For other ways to contribute see our [Contributing to GitLab](https://about.gitlab.com/community/contribute/) page. Looking for something to work on? See the [How to contribute](#how-to-contribute) section for more information. @@ -43,27 +44,10 @@ If you would like to contribute to GitLab: slowest tests. These tests are good candidates for improving and checking if any of [best practices](../testing_guide/best_practices.md) could speed them up. -- Consult the [Contribution Flow](#contribution-flow) section to learn the process. - -### Contribution flow - -The general flow of contributing to GitLab is: -1. [Read about](https://gitlab.com/gitlab-community/meta) and [request access](https://gitlab.com/gitlab-community/meta#request-access-to-community-forks) - to the GitLab Community forks. In some cases, you will want to set up the - [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) to - [develop against the GitLab community fork](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/index.md#develop-in-your-own-gitlab-fork). -1. Create a feature branch and make changes in the [GitLab community fork](https://gitlab.com/gitlab-community/gitlab). -1. When you're ready, [create a new merge request](../../user/project/merge_requests/creating_merge_requests.md). -1. In the merge request's description: - - Ensure you provide complete and accurate information. - - Review the provided checklist. -1. Once you're ready, mark your MR as ready for review with `@gitlab-bot ready`. - - This will add the `~"workflow::ready for review"` label, and then automatically assign a merge request coach as reviewer. - - If you know a relevant reviewer (for example, someone that was involved a related issue), you can also - assign them directly with `@gitlab-bot ready @username`. +For a walkthrough of the contribution process, see [Tutorial: Make a GitLab contribution](first_contribution.md). -#### Review process +### Review process When you submit code to GitLab, we really want it to get merged! However, we always review submissions carefully, and this takes time. Code submissions will usually be reviewed by two @@ -95,7 +79,7 @@ Lastly, keep the following in mind when submitting merge requests: be merged, as well as some guidance. The maintainers will be open to discussion about how to change the code so it can be approved and merged in the future. -#### Getting attention on your merge request +### Getting attention on your merge request GitLab will do its best to review community contributions as quickly as possible. Specially appointed developers review community contributions daily. Look at the @@ -119,7 +103,7 @@ This [documentation](issue_workflow.md) outlines the current issue workflow: - [Technical and UX debt](../labels/index.md#technical-and-ux-debt) - [Technical debt in follow-up issues](issue_workflow.md#technical-debt-in-follow-up-issues) -#### Merge requests workflow +### Merge requests workflow This [documentation](merge_request_workflow.md) outlines the current merge request process. @@ -136,9 +120,9 @@ This [documentation](merge_request_workflow.md) outlines the current merge reque ## Getting an Enterprise Edition License If you need a license for contributing to an EE-feature, see -[relevant information](https://about.gitlab.com/handbook/marketing/community-relations/code-contributor-program/operations/#contributing-to-the-gitlab-enterprise-edition-ee). +[relevant information](https://about.gitlab.com/handbook/marketing/community-relations/contributor-success/community-contributors-workflows.html#contributing-to-the-gitlab-enterprise-edition-ee). ## Finding help - [Get help](https://about.gitlab.com/get-help/). -- Join the community-run [Discord server](https://discord.com/invite/gitlab) and find other contributors in the `#contribute` channel. +- Join the community-run [Discord server](https://discord.gg/gitlab) and find other contributors in the `#contribute` channel. diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index f39d93a39bc..a67459e02ed 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -55,66 +55,25 @@ within the MR. ## Merge request guidelines for contributors -### Getting started - -Merge requests should be submitted to the appropriate project at GitLab.com, for example -[GitLab](https://gitlab.com/gitlab-org/gitlab/-/merge_requests), -[GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests), or -[Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests). - -To start developing GitLab locally, download the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) -and see the [Development section](../../index.md) for the required guidelines. - -NOTE: -Consider placing your code behind a feature flag if you think it might affect production availability. -Not sure? Read [When to use feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags). - -If the change is non-trivial, we encourage you to -start a discussion with [a product manager or a member of the team](https://about.gitlab.com/handbook/product/categories/). -You can do -this by tagging them in an MR before submitting the code for review. Talking -to team members can be helpful when making design decisions. Communicating the -intent behind your changes can also help expedite merge request reviews. - -### Creating a merge request - -To create a merge request: - -1. [Read about](https://gitlab.com/gitlab-community/meta) and [request access](https://gitlab.com/gitlab-community/meta#request-access-to-community-forks) - to the GitLab Community forks. In some cases, you will want to set up the - [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) to - [develop against the GitLab community fork](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/index.md#develop-in-your-own-gitlab-fork). -1. Create a feature branch in the [GitLab community fork](https://gitlab.com/gitlab-community/gitlab) - (don't work off the [default branch](../../user/project/repository/branches/default.md)). -1. Follow the [commit messages guidelines](#commit-messages-guidelines). -1. If you have multiple commits, combine them into a few logically organized commits. -1. Push the commits to your working branch in the fork. -1. Submit a merge request (MR) against the default branch of the upstream project. -1. The MR title should describe the change you want to make. -1. The MR description should give a reason for your change. - 1. If you are contributing code, fill in the description according to the default - template already provided in the "Description" field. - 1. If you are contributing documentation, choose `Documentation` from the - "Choose a template" menu and fill in the description according to the template. - 1. Use the syntax `Solves #XXX`, `Closes #XXX`, or `Refs #XXX` to mention the issues your merge - request addresses. Referenced issues do not [close automatically](../../user/project/issues/managing_issues.md#closing-issues-automatically). - You must close them manually once the merge request is merged. -1. If you're allowed to, set a relevant milestone and [labels](issue_workflow.md). - MR labels should generally match the corresponding issue (if there is one). - The group label should reflect the group that executed or coached the work, - not necessarily the group that owns the feature. -1. Read and adhere to - [The responsibility of the merge request author](../code_review.md#the-responsibility-of-the-merge-request-author). -1. Read and follow - [Having your merge request reviewed](../code_review.md#having-your-merge-request-reviewed). -1. Make sure the merge request meets the [Definition of done](#definition-of-done). - -If you would like quick feedback on your merge request feel free to mention someone -from the [core team](https://about.gitlab.com/community/core-team/) or one of the -[merge request coaches](https://about.gitlab.com/company/team/). When having your code reviewed -and when reviewing merge requests, please keep the [code review guidelines](../code_review.md) -in mind. And if your code also makes changes to the database, or does expensive queries, -check the [database review guidelines](../database_review.md). +For a walkthrough of the contribution process, see [Tutorial: Make a GitLab contribution](first_contribution.md). + +### Best practices + +- If the change is non-trivial, we encourage you to start a discussion with + [a product manager or a member of the team](https://about.gitlab.com/handbook/product/categories/). + You can do this by tagging them in an MR before submitting the code for review. Talking + to team members can be helpful when making design decisions. Communicating the + intent behind your changes can also help expedite merge request reviews. + +- Consider placing your code behind a feature flag if you think it might affect production availability. + Not sure? Read [When to use feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags). + +- If you would like quick feedback on your merge request feel free to mention someone + from the [core team](https://about.gitlab.com/community/core-team/) or one of the + [merge request coaches](https://about.gitlab.com/company/team/). When having your code reviewed + and when reviewing merge requests, please keep the [code review guidelines](../code_review.md) + in mind. And if your code also makes changes to the database, or does expensive queries, + check the [database review guidelines](../database_review.md). ### Keep it simple @@ -381,3 +340,8 @@ issue) that are incremental improvements, such as: Tag a merge request with ~"Stuff that should Just Work" to track work in this area. + +## Related topics + +- [The responsibility of the merge request author](../code_review.md#the-responsibility-of-the-merge-request-author) +- [Having your merge request reviewed](../code_review.md#having-your-merge-request-reviewed) diff --git a/doc/development/database/adding_database_indexes.md b/doc/development/database/adding_database_indexes.md index 1e3a1de9b69..7b29b1b14de 100644 --- a/doc/development/database/adding_database_indexes.md +++ b/doc/development/database/adding_database_indexes.md @@ -38,6 +38,15 @@ when adding a new index: 1. Is the overhead of maintaining the index worth the reduction in query timings? +In some situations, an index might not be required: + +- The table is small (less than `1,000` records) and it's not expected to exponentially grow in size. +- Any existing indexes filter out enough rows. +- The reduction in query timings after the index is added is not significant. + +Additionally, wide indexes are not required to match all filter criteria of queries. We just need +to cover enough columns so that the index lookup has a small enough selectivity. + ## Re-using Queries The first step is to make sure your query re-uses as many existing indexes as @@ -183,6 +192,29 @@ for `index_exists?`, causing a required index to not be created properly. By always requiring a name for certain types of indexes, the chance of error is greatly reduced. +## Testing for existence of indexes + +The easiest way to test for existence of an index by name is to use the `index_name_exists?` method, but the `index_exists?` method can also be used with a name option. For example: + +```ruby +class MyMigration < Gitlab::Database::Migration[2.1] + INDEX_NAME = 'index_name' + + def up + # an index must be conditionally created due to schema inconsistency + unless index_exists?(:table_name, :column_name, name: INDEX_NAME) + add_index :table_name, :column_name, name: INDEX_NAME + end + end + + def down + # no op + end +end +``` + +Keep in mind that concurrent index helpers like `add_concurrent_index`, `remove_concurrent_index`, and `remove_concurrent_index_by_name` already perform existence checks internally. + ## Temporary indexes There may be times when an index is only needed temporarily. @@ -448,7 +480,7 @@ You must test the database index changes locally before creating a merge request the post-deploy migration has been executed in the production database. For more information, see [How to determine if a post-deploy migration has been executed on GitLab.com](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/post_deploy_migration/readme.md#how-to-determine-if-a-post-deploy-migration-has-been-executed-on-gitlabcom). 1. In the case of an [index removed asynchronously](#schedule-the-index-to-be-removed), wait - until the next week so that the index can be created over a weekend. + until the next week so that the index can be removed over a weekend. 1. Use Database Lab [to check if removal was successful](database_lab.md#checking-indexes). [Database Lab](database_lab.md) should report an error when trying to find the removed index. If not, the index may still exist. diff --git a/doc/development/database/avoiding_downtime_in_migrations.md b/doc/development/database/avoiding_downtime_in_migrations.md index 8e1eeee7a42..25310554c24 100644 --- a/doc/development/database/avoiding_downtime_in_migrations.md +++ b/doc/development/database/avoiding_downtime_in_migrations.md @@ -605,7 +605,7 @@ See example [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_request ### Remove the trigger and old `integer` columns (release N + 2) Using post-deployment migration and the provided `cleanup_conversion_of_integer_to_bigint` helper, -drop the database trigger and the old `integer` columns ([see an example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69714)). +drop the database trigger and the old `integer` columns ([see an example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70351)). ### Remove ignore rules (release N + 3) diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md index 67dccd99a6c..7f3c5889017 100644 --- a/doc/development/database/batched_background_migrations.md +++ b/doc/development/database/batched_background_migrations.md @@ -597,6 +597,37 @@ for more details. more pressure on DB than you expect. Measure on staging, or ask someone to measure on production. 1. Know how much time is required to run the batched background migration. +1. Be careful when silently rescuing exceptions inside job classes. This may lead to + jobs being marked as successful, even in a failure scenario. + + ```ruby + # good + def perform + each_sub_batch do |sub_batch| + sub_batch.update_all(name: 'My Name') + end + end + + # acceptable + def perform + each_sub_batch do |sub_batch| + sub_batch.update_all(name: 'My Name') + rescue Exception => error + logger.error(message: error.message, class: error.class) + + raise + end + end + + # bad + def perform + each_sub_batch do |sub_batch| + sub_batch.update_all(name: 'My Name') + rescue Exception => error + logger.error(message: error.message, class: self.class.name) + end + end + ``` ## Additional tips and strategies @@ -768,7 +799,7 @@ To do that, you need to: #### 1. Find the batch `start_id` and `end_id` -You should be able to find those in [Kibana][#viewing-failure-error-logs]. +You should be able to find those in [Kibana](#viewing-failure-error-logs). #### 2. Create a regular migration @@ -809,7 +840,7 @@ index b8d1d21a0d2d2a23d9e8c8a0a17db98ed1ed40b7..912e20659a6919f771045178c6682856 +++ b/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb @@ -55,7 +55,7 @@ def unmatched_schemas end - + def allowed_schemas_for_connection - Gitlab::Database.gitlab_schemas_for_connection(connection) + Gitlab::Database.gitlab_schemas_for_connection(connection) << :gitlab_ci @@ -828,7 +859,7 @@ index 4ae3622479f0800c0553959e132143ec9051898e..d556ec7f55adae9d46a56665ce02de78 @@ -79,7 +79,7 @@ def restrict_to_dml_only(parsed) tables = self.dml_tables(parsed) schemas = self.dml_schemas(tables) - + - if (schemas - self.allowed_gitlab_schemas).any? + if (schemas - (self.allowed_gitlab_schemas << :gitlab_ci)).any? raise DMLAccessDeniedError, \ diff --git a/doc/development/database/ci_mirrored_tables.md b/doc/development/database/ci_mirrored_tables.md index bf3a744b936..1e37739bdc4 100644 --- a/doc/development/database/ci_mirrored_tables.md +++ b/doc/development/database/ci_mirrored_tables.md @@ -76,9 +76,8 @@ the source and the target tables in sync: 1. Deleting namespaces/projects. ```mermaid -graph TD - - subgraph "CI database (tables)" +graph LR + subgraph CI["CI Tables"] E[other CI tables] F{queries with joins allowed} G[ci_project_mirrors] @@ -89,17 +88,18 @@ graph TD F---H end - A---B - B---C - B---D + Main["Main Tables"]---L["⛔ ← Joins are not allowed → ⛔"] + L---CI -L["⛔ ← Joins are not allowed → ⛔"] - - subgraph "Main database (tables)" + subgraph Main["Main Tables"] A[other main tables] B{queries with joins allowed} C[projects] D[namespaces] + + A---B + B---C + B---D end ``` diff --git a/doc/development/database/index.md b/doc/development/database/index.md index 2cb8509e203..f532e054849 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -64,6 +64,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Hash indexes](hash_indexes.md) - [Insert into tables in batches](insert_into_tables_in_batches.md) - [Iterating tables in batches](iterating_tables_in_batches.md) +- [Load balancing](load_balancing.md) - [`NOT NULL` constraints](not_null_constraints.md) - [Ordering table columns](ordering_table_columns.md) - [Pagination guidelines](pagination_guidelines.md) diff --git a/doc/development/database/iterating_tables_in_batches.md b/doc/development/database/iterating_tables_in_batches.md index 6357bed8b00..a927242e8d8 100644 --- a/doc/development/database/iterating_tables_in_batches.md +++ b/doc/development/database/iterating_tables_in_batches.md @@ -44,9 +44,13 @@ all of the arguments that `in_batches` supports. You should always use ## Iterating over non-unique columns -One should proceed with extra caution. When you iterate over an attribute that is not unique, -even with the applied max batch size, there is no guarantee that the resulting batches do not -surpass it. The following snippet demonstrates this situation when one attempt to select +You should not use the `each_batch` method with a non-unique column (in the context of the relation) as it +[may result in an infinite loop](https://gitlab.com/gitlab-org/gitlab/-/issues/285097). +Additionally, the inconsistent batch sizes cause performance issues when you +iterate over non-unique columns. Even when you apply a max batch size +when iterating over an attribute, there's no guarantee that the resulting +batches don't surpass it. The following snippet demonstrates this situation +when you attempt to select `Ci::Build` entries for users with `id` between `1` and `10,000`, the database returns `1 215 178` matching rows. @@ -465,6 +469,58 @@ Issue.each_batch(of: 1000) do |relation| end ``` +### Counting records + +For tables with a large amount of data, counting records through queries can result +in timeouts. The `EachBatch` module provides an alternative way to iteratively count +records. The downside of using `each_batch` is the extra count query which is executed +on the yielded relation object. + +The `each_batch_count` method is a more efficient approach that eliminates the need +for the extra count query. By invoking this method, the iteration process can be +paused and resumed as needed. This feature is particularly useful in situations +where error budget violations are triggered after five minutes, such as when performing +counting operations within Sidekiq workers. + +To illustrate, counting records using `EachBatch` involves invoking an additional +count query as follows: + +```ruby +count = 0 + +Issue.each_batch do |relation| + count += relation.count +end + +puts count +``` + +On the other hand, the `each_batch_count` method enables the counting process to be +performed more efficiently (counting is part of the iteration query) without invoking +an extra count query: + +```ruby +count, _last_value = Issue.each_batch_count # last value can be ignored here +``` + +Furthermore, the `each_batch_count` method allows the counting process to be paused +and resumed at any point. This capability is demonstrated in the following code snippet: + +```ruby +stop_at = Time.current + 3.minutes + +count, last_value = Issue.each_batch_count do + Time.current > stop_at # condition for stopping the counting +end + +# Continue the counting later +stop_at = Time.current + 3.minutes + +count, last_value = Issue.each_batch_count(last_count: count, last_value: last_value) do + Time.current > stop_at +end +``` + ### `EachBatch` vs `BatchCount` When adding new counters for Service Ping, the preferred way to count records is using the diff --git a/doc/development/database/load_balancing.md b/doc/development/database/load_balancing.md new file mode 100644 index 00000000000..f623ad1eab0 --- /dev/null +++ b/doc/development/database/load_balancing.md @@ -0,0 +1,59 @@ +--- +stage: Data Stores +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Database load balancing + +With database load balancing, read-only queries can be distributed across multiple +PostgreSQL nodes to increase performance. + +This documentation provides a technical overview on how database load balancing +is implemented in GitLab Rails and Sidekiq. + +## Nomenclature + +1. **Host**: Each database host. It could be a primary or a replica. +1. **Primary**: Primary PostgreSQL host that is used for write-only and read-and-write operations. +1. **Replica**: Secondary PostgreSQL hosts that are used for read-only operations. +1. **Workload**: a Rails request or a Sidekiq job that requires database connections. + +## Components + +F few Ruby classes are involved in the load balancing process. All of them are +in the namespace `Gitlab::Database::LoadBalancing`: + +1. `Host` +1. `LoadBalancer` +1. `ConnectionProxy` +1. `Session` + +Each workload begins with a new instance of `Gitlab::Database::LoadBalancing::Session`. +The `Session` keeps track of the database operations that have been performed. It then +determines if the workload requires a connection to either the primary host or a replica host. + +When the workload requires a database connection through `ActiveRecord`, +`ConnectionProxy` first redirects the connection request to `LoadBalancer`. +`ConnectionProxy` requests either a `read` or `read_write` connection from the `LoadBalancer` +depending on a few criteria: + +1. Whether the query is a read-only or it requires write. +1. Whether the `Session` has recorded a write operation previously. +1. Whether any special blocks have been used to prefer primary or replica, such as: + - `use_primary` + - `ignore_writes` + - `use_replicas_for_read_queries` + - `fallback_to_replicas_for_ambiguous_queries` + +`LoadBalancer` then yields the requested connection from the respective database connection pool. +It yields either: + +- A `read_write` connection from the primary's connection pool. +- A `read` connection from the replicas' connection pools. + +When responding to a request for a `read` connection, `LoadBalancer` would +first attempt to load balance the connection across the replica hosts. +It looks for the next `online` replica host and yields a connection from the host's connection pool. +A replica host is considered `online` if it is up-to-date with the primary, based on +either the replication lag size or time. The thresholds for these requirements are configurable. diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index daa022a3de2..91a22d8c26b 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -64,7 +64,7 @@ The tool ensures that all aspects of swapping a foreign key are covered. This in - Creating a migration to remove a foreign key. - Updating `db/structure.sql` with the new migration. -- Updating `lib/gitlab/database/gitlab_loose_foreign_keys.yml` to add the new loose foreign key. +- Updating `config/gitlab_loose_foreign_keys.yml` to add the new loose foreign key. - Creating or updating a model's specs to ensure that the loose foreign key is properly supported. The tool is located at `scripts/decomposition/generate-loose-foreign-key`: diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index c1b30a4cbbe..6adfdc90cf2 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -545,7 +545,7 @@ end ``` Don't hesitate to reach out to the -[pods group](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/pods/) +[Pods group](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/tenant-scale/) for advice. ##### Avoid `dependent: :nullify` and `dependent: :destroy` across databases @@ -580,6 +580,24 @@ or records that point to nowhere, which might lead to bugs. As such we created ["loose foreign keys"](loose_foreign_keys.md) which is an asynchronous process of cleaning up orphaned records. +## Testing for multiple databases + +In our testing CI pipelines, we test GitLab by default with multiple databases set up, using +both `main` and `ci` databases. But in merge requests, for example when we modify some database-related code or +add the label `~"pipeline:run-single-db"` to the MR, we additionally run our tests in +[two other database modes](../pipelines/index.md#single-database-testing): +`single-db` and `single-db-ci-connection`. + +To handle situations where our tests need to run in specific database modes, we have some RSpec helpers +to limit the modes where tests can run, and skip them on any other modes. + +| Helper name | Test runs | +|---------------------------------------------| --- | +| `skip_if_shared_database(:ci)` | On **multiple databases** | +| `skip_if_database_exists(:ci)` | On **single-db** and **single-db-ci-connection** | +| `skip_if_multiple_databases_are_setup(:ci)` | Only on **single-db** | +| `skip_if_multiple_databases_not_setup(:ci)` | On **single-db-ci-connection** and **multiple databases** | + ## Locking writes on the tables that don't belong to the database schemas When the CI database is promoted and the two databases are fully split, diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index 47e89c1ce0f..5cd325bfa56 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -68,9 +68,17 @@ is held for a brief amount of time, the time `add_column` needs to complete its depending on how frequently the table is accessed. For example, acquiring an exclusive lock for a very frequently accessed table may take minutes in GitLab.com and requires the use of `with_lock_retries`. -For these reasons, it is advised to add the text limit on a separate migration than the `add_column` one. +When adding a text limit, transactions must be disabled with `disable_ddl_transaction!`. This means adding the column is not rolled back +in case the migration fails afterwards. An attempt to re-run the migration will raise an error because of the already existing column. -For example, consider a migration that adds a new text column `extended_title` to table `sprints`, +For these reasons, adding a text column to an existing table can be done by either: + +- [Add the column and limit in separate migrations.](#add-the-column-and-limit-in-separate-migrations) +- [Add the column and limit in one migration with checking if the column already exists.](#add-the-column-and-limit-in-one-migration-with-checking-if-the-column-already-exists) + +### Add the column and limit in separate migrations + +Consider a migration that adds a new text column `extended_title` to table `sprints`, `db/migrate/20200501000001_add_extended_title_to_sprints.rb`: ```ruby @@ -103,6 +111,33 @@ class AddTextLimitToSprintsExtendedTitle < Gitlab::Database::Migration[2.1] end ``` +### Add the column and limit in one migration with checking if the column already exists + +Consider a migration that adds a new text column `extended_title` to table `sprints`, +`db/migrate/20200501000001_add_extended_title_to_sprints.rb`: + +```ruby +class AddExtendedTitleToSprints < Gitlab::Database::Migration[2.1] + disable_ddl_transaction! + + def up + with_lock_retries do + add_column :sprints, :extended_title, :text, if_not_exists: true + end + + add_text_limit :sprints, :extended_title, 512 + end + + def down + remove_text_limit :sprints, :extended_title + + with_lock_retries do + remove_column :sprints, :extended_title, if_exists: true + end + end +end +``` + ## Add a text limit constraint to an existing column Adding text limits to existing database columns requires multiple steps split into at least two different releases: diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md index 0d5e3c233f6..81733b126b6 100644 --- a/doc/development/database/table_partitioning.md +++ b/doc/development/database/table_partitioning.md @@ -6,6 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Database table partitioning +WARNING: +If you have questions not answered below, check for and add them +to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/398650). +Tag `@gitlab-org/database-team/triage` and we'll get back to you with an +answer as soon as possible. If you get an answer in Slack, document +it on the issue as well so we can update this document in the future. + Table partitioning is a powerful database feature that allows a table's data to be split into smaller physical tables that act as a single large table. If the application is designed to work with partitioning in mind, @@ -32,31 +39,38 @@ several releases. Due to the limitations of partitioning and the related migrations, you should understand how partitioning fits your use case before attempting to leverage this feature. -## Determining when to use partitioning +## Determine when to use partitioning While partitioning can be very useful when properly applied, it's imperative to identify if the data and workload of a table naturally fit a -partitioning scheme. There are a few details you have to understand -to decide if partitioning is a good fit for your particular -problem. - -First, a table is partitioned on a partition key, which is a column or -set of columns which determine how the data is split across the -partitions. The partition key is used by the database when reading or -writing data, to decide which partitions must be accessed. The -partition key should be a column that would be included in a `WHERE` -clause on almost all queries accessing that table. - -Second, it's necessary to understand the strategy the database uses -to split the data across the partitions. The scheme supported by the -GitLab migration helpers is date-range partitioning, where each partition -in the table contains data for a single month. In this case, the partitioning -key must be a timestamp or date column. In order for this type of +partitioning scheme. Understand a few details to decide if partitioning +is a good fit for your particular problem: + +- **Table partitioning**. A table is partitioned on a partition key, which is a + column or set of columns which determine how the data is split across the + partitions. The partition key is used by the database when reading or + writing data, to decide which partitions must be accessed. The + partition key should be a column that would be included in a `WHERE` + clause on almost all queries accessing that table. + +- **How the data is split**. What strategy does the database use + to split the data across the partitions? The available choices are `range`, + `hash`, and `list`. + +## Determine the appropriate partitioning strategy + +The available partitioning strategy choices are `range`, `hash`, and `list`. + +### Range partitioning + +The scheme best supported by the GitLab migration helpers is date-range partitioning, +where each partition in the table contains data for a single month. In this case, +the partitioning key must be a timestamp or date column. For this type of partitioning to work well, most queries must access data in a certain date range. -For a more concrete example, the `audit_events` table can be used, which -was the first table to be partitioned in the application database +For a more concrete example, consider using the `audit_events` table. +It was the first table to be partitioned in the application database (scheduled for deployment with the GitLab 13.5 release). This table tracks audit entries of security events that happen in the application. In almost all cases, users want to see audit activity that @@ -142,6 +156,31 @@ substantial. Partitioning should only be leveraged if the access patterns of the data support the partitioning strategy, otherwise performance suffers. +### Hash Partitioning + +Hash partitioning splits a logical table into a series of partitioned +tables. Each partition corresponds to the ID range that matches +a hash and remainder. For example, if partitioning `BY HASH(id)`, rows +with `hash(id) % 64 == 1` would end up in the partition +`WITH (MODULUS 64, REMAINDER 1)`. + +When hash partitioning, you must include a `WHERE hashed_column = ?` condition in +every performance-sensitive query issued by the application. If this is not possible, +hash partitioning may not be the correct fit for your use case. + +Hash partitioning has one main advantage: it is the only type of partitioning that +can enforce uniqueness on a single numeric `id` column. (While also possible with +range partitioning, it's rarely the correct choice). + +Hash partitioning has downsides: + +- The number of partitions must be known up-front. +- It's difficult to move new data to an extra partition if current partitions become too large. +- Range queries, such as `WHERE id BETWEEN ? and ?`, are unsupported. +- Lookups by other keys, such as `WHERE other_id = ?`, are unsupported. + +For this reason, it's often best to choose a large number of hash partitions to accommodate future table growth. + ## Partitioning a table (Range) Unfortunately, tables can only be partitioned at their creation, making @@ -257,6 +296,18 @@ for use by the application. This section will be updated when the migration helper is ready, for now development can be followed in the [Tracking Issue](https://gitlab.com/gitlab-org/gitlab/-/issues/241267). +## Partitioning a table (Hash) + +Hash partitioning divides data into partitions based on a hash of their ID. +It works well only if most queries against the table include a clause like `WHERE id = ?`, +so that PostgreSQL can decide which partition to look in based on the ID or ids being requested. + +Another key downside is that hash partitioning does not allow adding additional partitions after table creation. +The correct number of partitions must be chosen up-front. + +Hash partitioning is the only type of partitioning (aside from some complex uses of list partitioning) that can guarantee +uniqueness of an ID across multiple partitions at the database level. + ## Partitioning a table (List) > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96815) in GitLab 15.4. @@ -504,3 +555,48 @@ class Model < ApplicationRecord self.sequence_name = 'model_id_seq' end ``` + +If the partitioning constraint migration takes [more than 10 minutes](../migration_style_guide.md#how-long-a-migration-should-take) to finish, +it can be made to run asynchronously to avoid running the post-migration during busy hours. + +Prepend the following migration `AsyncPrepareTableConstraintsForListPartitioning` +and use `async: true` option. This change marks the partitioning constraint as `NOT VALID` +and enqueues a scheduled job to validate the existing data in the table during the weekend. + +Then the second post-migration `PrepareTableConstraintsForListPartitioning` only +marks the partitioning constraint as validated, because the existing data is already +tested during the previous weekend. + +For example: + +```ruby +class AsyncPrepareTableConstraintsForListPartitioning < Gitlab::Database::Migration[2.1] + include Gitlab::Database::PartitioningMigrationHelpers::TableManagementHelpers + + disable_ddl_transaction! + + TABLE_NAME = :table_name + PARENT_TABLE_NAME = :p_table_name + FIRST_PARTITION = 100 + PARTITION_COLUMN = :partition_id + + def up + prepare_constraint_for_list_partitioning( + table_name: TABLE_NAME, + partitioning_column: PARTITION_COLUMN, + parent_table_name: PARENT_TABLE_NAME, + initial_partitioning_value: FIRST_PARTITION, + async: true + ) + end + + def down + revert_preparing_constraint_for_list_partitioning( + table_name: TABLE_NAME, + partitioning_column: PARTITION_COLUMN, + parent_table_name: PARENT_TABLE_NAME, + initial_partitioning_value: FIRST_PARTITION + ) + end +end +``` diff --git a/doc/development/database/transaction_guidelines.md b/doc/development/database/transaction_guidelines.md index 26bb6c2ce8f..1648cf58937 100644 --- a/doc/development/database/transaction_guidelines.md +++ b/doc/development/database/transaction_guidelines.md @@ -12,7 +12,7 @@ For further reference, check PostgreSQL documentation about [transactions](https ## Database decomposition and sharding -The [Pods Group](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/pods/) plans +The [Pods group](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/tenant-scale/) plans to split the main GitLab database and move some of the database tables to other database servers. We start decomposing the `ci_*`-related database tables first. To maintain the current application diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 1f653f8af94..feac0b3098d 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -125,7 +125,7 @@ the following preparations into account. test its execution using `CREATE INDEX CONCURRENTLY` in [Database Lab](database/database_lab.md) and add the execution time to the MR description: - Execution time largely varies between Database Lab and GitLab.com, but an elevated execution time from Database Lab can give a hint that the execution on GitLab.com is also considerably high. - - If the execution from Database Lab is longer than `1h`, the index should be moved to a [post-migration](database/post_deployment_migrations.md). + - If the execution from Database Lab is longer than `10 minutes`, the [index](database/adding_database_indexes.md) should be moved to a [post-migration](database/post_deployment_migrations.md). Keep in mind that in this case you may need to split the migration and the application changes in separate releases to ensure the index is in place when the code that needs it is deployed. - Manually trigger the [database testing](database/database_migration_pipeline.md) job (`db:gitlabcom-database-testing`) in the `test` stage. diff --git a/doc/development/diffs.md b/doc/development/diffs.md deleted file mode 100644 index c84bf57e085..00000000000 --- a/doc/development/diffs.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'merge_request_concepts/diffs/index.md' -remove_date: '2023-04-10' ---- - -This document was moved to [another location](merge_request_concepts/diffs/index.md). - -<!-- This redirect file can be deleted after <2023-04-10>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index bd9e7035290..823a71eb130 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -22,6 +22,14 @@ Distributed tracing adds minimal overhead when disabled, but imposes only small enabled and is therefore capable in any environment, including production. For this reason, it can be useful in diagnosing production issues, particularly performance problems. +Services have different levels of support for distributed tracing. Custom +instrumentation code must be added to the application layer in addition to +pre-built instrumentation for the most common libraries. + +For service-specific information, see: + +- [Using Jaeger for Gitaly local development](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/jaeger_for_local_development.md) + ## Using Correlation IDs to investigate distributed requests The GitLab application passes correlation IDs between the various components in a request. A @@ -77,15 +85,56 @@ In this example, we have the following hypothetical values: they should be URL encoded. Multiple values should be separated by `&` characters like a URL. +GitLab Rails provides pre-implemented instrumentations for common types of +operations that offer a detailed view of the requests. However, the detailed +information comes at a cost. The resulting traces are long and can be difficult +to process, making it hard to identify bigger underlying issues. To address this +concern, some instrumentations are disabled by default. To enable those disabled +instrumentations, set the following environment variables: + +- `GITLAB_TRACING_TRACK_CACHES`: enable tracking cache operations, such as cache +read, write, or delete. +- `GITLAB_TRACING_TRACK_REDIS`: enable tracking Redis operations. Most Redis +operations are for caching, though. + ## Using Jaeger in the GitLab Development Kit The first tracing implementation that GitLab supports is Jaeger, and the -[GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit/) supports distributed tracing with -Jaeger out-of-the-box. +[GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit/) +supports distributed tracing with Jaeger out-of-the-box. GDK automatically adds +`GITLAB_TRACING` environment variables to add services. + +Configure GDK for Jaeger by editing the `gdk.yml` file and adding the following +settings: + +```yaml +tracer: + build_tags: tracer_static tracer_static_jaeger + jaeger: + enabled: true + listen_address: 127.0.0.1 + version: 1.43.0 +``` -The easiest way to access tracing from a GDK environment is through the -[performance-bar](../administration/monitoring/performance/performance_bar.md). This can be shown -by typing `p` `b` in the browser window. +After modifying the `gdk.yml` file, reconfigure your GDK by running +the `gdk reconfigure` command. This ensures that your GDK is properly configured +and ready to use. + +The above configuration sets the `tracer_static` and `tracer_static_jaeger` +build tags when rebuilding services written in Go for the first time. Any +changes made afterward require rebuilding them with those build tags. You can +either: + +- Add those build tags to the default set of build tags. +- Manually attach them to the build command. For example, Gitaly supports adding + build tag out of the box. You can run + `make all WITH_BUNDLED_GIT=YesPlease BUILD_TAGS="tracer_static tracer_static_jaeger"`. + +After reconfiguration, Jaeger dashboard is available at +`http://localhost:16686`. Another way to access tracing from a GDK environment +is through the +[performance-bar](../administration/monitoring/performance/performance_bar.md). +This can be shown by typing `p` `b` in the browser window. Once the performance bar is enabled, select **Trace** in the performance bar to go to the Jaeger UI. diff --git a/doc/development/documentation/alpha_beta.md b/doc/development/documentation/alpha_beta.md index 4b5c24d512f..d421aae3cb9 100644 --- a/doc/development/documentation/alpha_beta.md +++ b/doc/development/documentation/alpha_beta.md @@ -4,40 +4,46 @@ stage: none group: unassigned --- -# Document Alpha, Beta, LA features +# Documenting Experiment and Beta features Some features are not generally available and are instead considered -[Alpha, Beta, or Limited Availability](../../policy/alpha-beta-support.md). +[Experiment or Beta](../../policy/alpha-beta-support.md). When you document a feature in one of these three statuses: -- Add `(Alpha)`, `(Beta)`, or `(Limited Availability)` in parentheses after the page or topic title. -- Do not include `(Alpha)`, `(Beta)`, or `(Limited Availability)` in the left nav. +- Add `(Experiment)` or `(Beta)` in parentheses after the page or topic title. +- Do not include `(Experiment)` or `(Beta)` in the left nav. - Ensure the version history lists the feature's status. These features are usually behind a feature flag, which follow [these documentation guidelines](feature_flags.md). If you add details of how users should enroll, or how to contact the team with issues, -the `FLAG:` note should be above these details. For example: +the `FLAG:` note should be above these details. + +For example: ```markdown -## Great new feature (Alpha) +## Great new feature (Experiment) + +> [Introduced](link) in GitLab 15.10. This feature is an [Experiment](<link_to>/policy/alpha-beta-support.md). FLAG: On self-managed GitLab, by default this feature is not available. -To make it available, ask an administrator to enable the feature flag named example_flag. +To make it available, ask an administrator to enable the feature flag named `example_flag`. On GitLab.com, this feature is not available. This feature is not ready for production use. Use this great new feature when you need to do this new thing. -This feature is in Alpha. To join the list of users testing this feature, -do this thing. If you find a bug, [open an issue](link). +This feature is an [Experiment](<link_to>/policy/alpha-beta-support.md). To join +the list of users testing this feature, do this thing. If you find a bug, +[open an issue](link). ``` -When the feature is released, remove: +When the feature is ready for production, remove: - The text in parentheses. -- Any language about the feature not being ready for production. -- The feature flag information. +- Any language about the feature not being ready for production in the body + description. +- The feature flag information if available. -Ensure the version history is up-to-date. +Ensure the version history is up-to-date by adding a note about the production release. diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 986252eedac..854faa839ef 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -17,7 +17,7 @@ When the state of a feature flag changes, the developer who made the change Every feature introduced to the codebase, even if it's behind a disabled feature flag, must be documented. For more information, see -[the discussion that led to this decision](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47917#note_459984428). [Alpha, Beta, or Limited Availability](../../policy/alpha-beta-support.md) features are usually behind a feature flag, and must also be documented. For more information, see [Document Alpha, Beta, LA features](alpha_beta.md). +[the discussion that led to this decision](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47917#note_459984428). [Experiment or Beta](../../policy/alpha-beta-support.md) features are usually behind a feature flag, and must also be documented. For more information, see [Document Experiment or Beta features](alpha_beta.md). When the feature is [implemented in multiple merge requests](../feature_flags/index.md#feature-flags-in-gitlab-development), discuss the plan with your technical writer. diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index d1f5ee8f9f3..c9f31b36e3f 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -114,21 +114,6 @@ The following metadata should be added when a page is moved to another location: - `redirect_to`: The relative path and filename (with an `.md` extension) of the location to which visitors should be redirected for a moved page. [Learn more](redirects.md). -- `disqus_identifier`: Identifier for Disqus commenting system. Used to keep - comments with a page that has been moved to a new URL. - [Learn more](redirects.md#redirections-for-pages-with-disqus-comments). - -### Comments metadata - -The [docs website](site_architecture/index.md) has comments (provided by Disqus) -enabled by default. In case you want to disable them (for example in index pages), -set it to `false`: - -```yaml ---- -comments: false ---- -``` ### Additional page metadata @@ -353,28 +338,6 @@ feedback: false The default is to leave it there. If you want to omit it from a document, you must check with a technical writer before doing so. -## Disqus - -We have integrated the docs site with Disqus (introduced by -[!151](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/151)), -allowing our users to post comments. - -To omit only the comments from the feedback section, use the following key in -the front matter: - -```yaml ---- -comments: false ---- -``` - -We're hiding comments only in main index pages, such as [the main documentation index](../../index.md), -since its content is too broad to comment on. Before omitting Disqus, you must -check with a technical writer. - -Note that after adding `feedback: false` to the front matter, it will omit -Disqus, therefore, don't add both keys to the same document. - The click events in the feedback section are tracked with Google Tag Manager. The conversions can be viewed on Google Analytics by navigating to **Behavior > Events > Top events > docs**. diff --git a/doc/development/documentation/redirects.md b/doc/development/documentation/redirects.md index 4cfe8be713a..068c1e84a0f 100644 --- a/doc/development/documentation/redirects.md +++ b/doc/development/documentation/redirects.md @@ -81,7 +81,6 @@ To redirect a page to another page in the same repository: - Replace both instances of `../newpath/to/file/index.md` with the new file path. - Replace both instances of `YYYY-MM-DD` with the expiration date, as explained in the template. -1. If the page has Disqus comments, follow [the steps for pages with Disqus comments](#redirections-for-pages-with-disqus-comments). 1. If the page had images that aren't used on any other pages, delete them. After your changes are committed, search for and update all links that point to the old file: @@ -158,26 +157,3 @@ If you create a new page and then rename it before it's added to a release on th Instead of following that procedure, ask a Technical Writer to manually add the redirect to [`redirects.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/redirects.yaml). - -## Redirections for pages with Disqus comments - -If the documentation page being relocated already has Disqus comments, -we need to preserve the Disqus thread. - -Disqus uses an identifier per page, and for <https://docs.gitlab.com>, the page identifier -is configured to be the page URL. Therefore, when we change the document location, -we need to preserve the old URL as the same Disqus identifier. - -To do that, add to the front matter the variable `disqus_identifier`, -using the old URL as value. For example, let's say we moved the document -available under `https://docs.gitlab.com/my-old-location/README.html` to a new location, -`https://docs.gitlab.com/my-new-location/index.html`. - -Into the **new document** front matter, we add the following information. You must -include the filename in the `disqus_identifier` URL, even if it's `index.html` or `README.html`. - -```yaml ---- -disqus_identifier: 'https://docs.gitlab.com/my-old-location/README.html' ---- -``` diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index ef6f3c0ae18..3e0534220a8 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -69,8 +69,7 @@ Sometimes pages for deprecated features are not in the global nav, depending on All other pages should be in the global nav. The technical writing team runs a report to determine which pages are not in the nav. -For now this report is manual, but [an issue exists](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/1212) -to automate it. +The team reviews this list each month. ### Where to add diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index d4553614fad..849308f3086 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -33,7 +33,7 @@ Then you can use one of these approaches: of the documentation in the external repository. The landing page is indexed and searchable on <https://docs.gitlab.com>, but the rest of the documentation is not. For example, the [GitLab Workflow extension for VS Code](../../../user/project/repository/vscode.md). - We do not encourage the use of [pages with lists of links](../topic_types/index.md#topics-and-resources), + We do not encourage the use of [pages with lists of links](../topic_types/index.md#pages-and-topics-to-avoid), so only use this option if the recommended options are not feasible. ## Monthly release process (versions) diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 7f69a1f0d54..2925a0f0371 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -60,7 +60,7 @@ GitLab adds all troubleshooting information to the documentation, no matter how unlikely a user is to encounter a situation. GitLab Support maintains their own -[troubleshooting content](../../../administration/index.md#support-team-documentation) +[troubleshooting content](../../../administration/troubleshooting/index.md) in the GitLab documentation. ### The documentation includes all media types @@ -718,6 +718,15 @@ This is overridden by the [documentation-specific punctuation rules](#punctuatio Links help the docs adhere to the [single source of truth](#documentation-is-the-single-source-of-truth-ssot) principle. +However, you should avoid putting too many links on any page. Too many links can hinder readability. + +- Do not duplicate links on the same page. For example, on **Page A**, do not link to **Page B** multiple times. +- Avoid multiple links in a single paragraph. +- Avoid multiple links in a single task. +- On any one page, try not to use more than 15 links to other pages. +- Consider using [Related topics](../topic_types/index.md#related-topics) to reduce links that interrupt the flow of a task. +- Try to avoid anchor links to sections on the same page. Let users rely on the right navigation instead. + ### Links within the same repository To link to another page in the same repository, @@ -1449,7 +1458,7 @@ For tab titles, be brief and consistent. Ensure they are parallel, and start eac For example: - `Linux package (Omnibus)`, `Helm chart (Kubernetes)` (when documenting configuration edits, follow the - [configuration edits guide](#configuration-documentation-for-different-installation-methods)) + [configuration edits guide](#how-to-document-different-installation-methods)) - `15.1 and earlier`, `15.2 and later` Until we implement automated testing for broken links to tabs ([Issue 1355](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/1355)), do not link directly to a single tab, even though they do have unique URL parameters. @@ -1546,24 +1555,28 @@ If the document resides outside of the `doc/` directory, use the full path instead of the relative link: `https://docs.gitlab.com/ee/administration/restart_gitlab.html`. -### Installation guide - -In [step 2 of the installation guide](../../../install/installation.md#2-ruby), -we install Ruby from source. To update the guide for a new Ruby version: +### How to document different installation methods -- Change the version throughout the code block. -- Replace the sha256sum. It's available on the - [downloads page](https://www.ruby-lang.org/en/downloads/) of the Ruby website. +GitLab supports five official installation methods. If you're referring to +words as part of sentences and titles, use the following phrases: -### Configuration documentation for different installation methods +- Linux package +- Helm chart +- GitLab Operator +- Docker +- Self-compiled -GitLab supports four installation methods: +It's OK to add the explanatory parentheses when +[using tabs](#use-tabs-to-describe-a-self-managed-configuration-procedure): - Linux package (Omnibus) - Helm chart (Kubernetes) +- GitLab Operator (Kubernetes) - Docker - Self-compiled (source) +### Use tabs to describe a self-managed configuration procedure + Configuration procedures can require users to edit configuration files, reconfigure GitLab, or restart GitLab. In this case: @@ -1576,8 +1589,8 @@ GitLab, or restart GitLab. In this case: - The final step to reconfigure or restart GitLab can be used verbatim since it's the same every time. -You can copy and paste the following snippet when describing a configuration -edit: +When describing a configuration edit, you can use and edit to your liking the +following snippet: <!-- markdownlint-disable tabs-blank-lines --> ````markdown diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index 1ea95367d01..eb0576d0c05 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -149,13 +149,6 @@ Instead of: This phrasing is more active and is from the user perspective, rather than the person who implemented the feature. [View details in the Microsoft style guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/a/allow-allows). -## Alpha - -Use uppercase for **Alpha**. For example: **The XYZ feature is in Alpha.** or **This Alpha release is ready to test.** - -You might also want to link to [this section](../../../policy/alpha-beta-support.md#alpha-features) -in the handbook when writing about Alpha features. - ## analytics Use lowercase for **analytics** and its variations, like **contribution analytics** and **issue analytics**. @@ -214,7 +207,7 @@ Try to avoid **below** when referring to an example or table in a documentation Use uppercase for **Beta**. For example: **The XYZ feature is in Beta.** or **This Beta release is ready to test.** -You might also want to link to [this section](../../../policy/alpha-beta-support.md#beta-features) +You might also want to link to [this section](../../../policy/alpha-beta-support.md#beta) in the handbook when writing about Beta features. ## blacklist @@ -286,8 +279,7 @@ CI/CD is always uppercase. No need to spell it out on first use. ## CI/CD minutes -Use **CI/CD minutes** instead of **CI minutes**, **pipeline minutes**, **pipeline minutes quota**, or -**CI pipeline minutes**. This decision was made in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/342813). +Do not use **CI/CD minutes**. This term was renamed to [**compute credits**](#compute-credits). ## click @@ -297,7 +289,7 @@ Do not use **click**. Instead, use **select** with buttons, links, menu items, a ## cloud native When you're talking about using a Kubernetes cluster to host GitLab, you're talking about a **cloud-native version of GitLab**. -This version is different than the larger, more monolithic **Omnibus package** that is used to deploy GitLab. +This version is different than the larger, more monolithic **Linux package** that is used to deploy GitLab. You can also use **cloud-native GitLab** for short. It should be hyphenated and lowercase. @@ -305,6 +297,25 @@ You can also use **cloud-native GitLab** for short. It should be hyphenated and Use **collapse** instead of **close** when you are talking about expanding or collapsing a section in the UI. +## command line + +Use **From the command line** to introduce commands. + +Hyphenate when using as an adjective. For example, **a command-line tool**. + +## compute credits + +Use **compute credits** instead of these (or similar) terms: + +- **CI/CD minutes** +- **CI minutes** +- **pipeline minutes** +- **CI pipeline minutes** +- **pipeline minutes quota** + +This language is still being standardized in the documentation and UI beginning in March, 2023. +For more information, see [issue 5218](https://gitlab.com/gitlab-com/Product/-/issues/5218). + ## confirmation dialog Use **confirmation dialog** to describe the dialog box that asks you to confirm your action. For example: @@ -480,6 +491,13 @@ Instead of: Use **expand** instead of **open** when you are talking about expanding or collapsing a section in the UI. +## Experiment + +Use uppercase for **Experiment**. For example: **The XYZ feature is an Experiment.** or **This Experiment is ready to test.** + +You might also want to link to [this section](../../../policy/alpha-beta-support.md#experiment) +in the handbook when writing about Experiment features. + ## FAQ We want users to find information quickly, and they rarely search for the term **FAQ**. @@ -518,6 +536,17 @@ Filtering is different from [searching](#search). Do not use **foo** in product documentation. You can use it in our API and contributor documentation, but try to use a clearer and more meaningful example instead. +## fork + +A **fork** is a project that was created from a **upstream project** by using the +[forking process](../../../topics/git/terminology.md#fork). + +The **upstream project** (also known as the **source project**) and the **fork** have a **fork relationship** and are +**linked**. + +If the [**fork relationship** is removed](../../../user/project/repository/forking_workflow.md#unlink-a-fork), the +**fork** is **unlinked** from the **upstream project**. + ## future tense When possible, use present tense instead of future tense. For example, use **after you execute this command, GitLab displays the result** instead of **after you execute this command, GitLab will display the result**. ([Vale](../testing.md#vale) rule: [`FutureTense.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FutureTense.yml)) @@ -552,6 +581,10 @@ Do not use **the `gitlab` chart**, **the GitLab Chart**, or **the cloud-native c You use the **GitLab Helm chart** to deploy **cloud-native GitLab** in a Kubernetes cluster. +If you use it in a context of describing the +[different installation methods](index.md#how-to-document-different-installation-methods). +use `Helm chart (Kubernetes)`. + ## GitLab Pages For consistency and branding, use **GitLab Pages** rather than **Pages**. @@ -561,7 +594,14 @@ you can use **Pages** thereafter. ## GitLab Runner -Use title case for **GitLab Runner**. This is the product you install. See also [runners](#runner-runners) and [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529). +Use title case for **GitLab Runner**. This is the product you install. For more information about the decision for this usage, +see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529). + +See also: + +- [runners](#runner-runners) +- [runner managers](#runner-manager-runner-managers) +- [runner workers](#runner-worker-runner-workers) ## GitLab SaaS @@ -643,13 +683,29 @@ Do not use Latin abbreviations. Use **that is** instead. ([Vale](../testing.md#v Do not use **in order to**. Use **to** instead. ([Vale](../testing.md#vale) rule: [`Wordy.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Wordy.yml)) +## Installation from source + +When referring to the installation method using the self-compiled code, refer to it +as **self-compiled**. + +Use: + +- For self-compiled installations... + +Instead of: + +- For installations from source... + +For more information, see the +[different installation methods](index.md#how-to-document-different-installation-methods). + ## -ing words Remove **-ing** words whenever possible. They can be difficult to translate, and more precise terms are usually available. For example: - Instead of **The files using storage are deleted**, use **The files that use storage are deleted**. -- Instead of **Delete files using the Edit button**, use **Delete files by using the Edit button**. +- Instead of **Delete files using the Edit button**, use **Use the Edit button to delete files**. - Instead of **Replicating your server is required**, use **You must replicate your server**. ## issue @@ -814,6 +870,23 @@ Use lowercase for **merge requests**. If you use **MR** as the acronym, spell it Use lowercase for **milestones**. +## Minimal Access + +When writing about the Minimal Access role: + +- Use a capital **M** and a capital **A**. +- Write it out: + - Use: if you are assigned the Minimal Access role + - Instead of: if you are a Minimal Access user + +- When the Minimal Access role is the minimum required role: + - Use: at least the Minimal Access role + - Instead of: the Minimal Access role or higher + +Do not use bold. + +Do not use **Minimal Access permissions**. A user who is assigned the Minimal Access role has a set of associated permissions. + ## n/a, N/A, not applicable When possible, use **not applicable**. Spelling out the phrase helps non-English speaking users and avoids @@ -876,6 +949,22 @@ Instead of: - Note that you can change the settings. +## Omnibus GitLab + +When referring to the installation method that uses the Linux package, refer to it +as **Linux package**. + +Use: + +- For installations that use the Linux package... + +Instead of: + +- For installations that use Omnibus GitLab... + +For more information, see the +[different installation methods](index.md#how-to-document-different-installation-methods). + ## on When documenting how to select high-level UI elements, use the word **on**. @@ -969,6 +1058,10 @@ Use lowercase for **personal access token**. Do not use **please**. For details, see the [Microsoft style guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). +## Premium + +Use **Premium**, in uppercase, for the subscription tier. + ## prerequisites Use **prerequisites** when documenting the steps before a task. Do not use **requirements**. @@ -1079,6 +1172,14 @@ Use lowercase for **runners**. These are the agents that run CI/CD jobs. See als When referring to runners, if you have to specify that the runners are installed on a customer's GitLab instance, use **self-managed** rather than **self-hosted**. +## runner manager, runner managers + +Use lowercase for **runner managers**. These are a type of runner that can create multiple runners for autoscaling. See also [GitLab Runner](#gitlab-runner). + +## runner worker, runner workers + +Use lowercase for **runner workers**. This is the process created by the runner on the host computing platform to run jobs. See also [GitLab Runner](#gitlab-runner). + ## (s) Do not use **(s)** to make a word optionally plural. It can slow down comprehension. For example: @@ -1239,6 +1340,13 @@ Use lowercase for **terminal**. For example: - Open a terminal. - From a terminal, run the `docker login` command. +## Terraform Module Registry + +Use title case for the GitLab Terraform Module Registry, but use lowercase `m` when +talking about non-specific modules. For example: + +- You can publish a Terraform module to your project's Terraform Module Registry. + ## text box Use **text box** instead of **field** or **box** when referring to the UI element. @@ -1314,6 +1422,10 @@ For example: See also [**enter**](#enter). +## Ultimate + +Use **Ultimate**, in uppercase, for the subscription tier. + ## units of measurement Use a space between the number and the unit of measurement. For example, **128 GB**. diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index 4f61b2424eb..4a7e206da20 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -49,17 +49,21 @@ To run tests locally, it's important to: ### Lint checks Lint checks are performed by the [`lint-doc.sh`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/lint-doc.sh) -script and can be executed as follows: +script and can be executed with the help of a Rake task as follows: -1. Go to the `gitlab` directory. +1. Go to your `gitlab` directory. 1. Run: ```shell - MD_DOC_PATH=path/to/my_doc.md scripts/lint-doc.sh + rake lint:markdown ``` -Where `MD_DOC_PATH` points to the file or directory you would like to run lint checks for. -If you omit it completely, it defaults to the `doc/` directory. +To specify a single file or directory you would like to run lint checks for, run: + +```shell +MD_DOC_PATH=path/to/my_doc.md rake lint:markdown +``` + The output should be similar to: ```plaintext @@ -77,7 +81,7 @@ The output should be similar to: This requires you to either: - Have the [required lint tools installed](#local-linters) on your computer. -- A working Docker installation, in which case an image with these tools pre-installed is used. +- A working Docker or containerd installation, to use an image with these tools pre-installed. ### Documentation link tests diff --git a/doc/development/documentation/topic_types/glossary.md b/doc/development/documentation/topic_types/glossary.md new file mode 100644 index 00000000000..4985101a391 --- /dev/null +++ b/doc/development/documentation/topic_types/glossary.md @@ -0,0 +1,70 @@ +--- +stage: none +group: Style Guide +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Glossary topic type + +A glossary provides a list of unfamiliar terms and their definitions to help users understand a specific +GitLab feature. + +Each glossary item provides a single term and its associated definition. The definition should answer the questions: + +- **What** is this? +- **Why** would you use it? + +For glossary terms: + +- Do not use jargon. +- Do not use internal terminology or acronyms. +- Ensure the correct usage is defined in the [word list](../styleguide/word_list.md). + +## Alternatives to glossaries + +Glossaries should provide short, concise term-definition pairs. + +- If a definition requires more than a brief explanation, use a concept topic instead. +- If you find yourself explaining how to use the feature, use a task topic instead. + +## Glossary example + +Glossary topics should be in this format. Use an unordered list primarily. You can use a table if you need to apply +additional categorization. + +Try to include glossary topics on pages that explain the feature, rather than as a standalone page. + +```markdown +## FeatureName glossary + +This glossary provides definitions for terms related to FeatureName. + +- **Term A**: Term A does this thing. +- **Term B**: Term B does this thing. +- **Term C**: Term C does this thing. +``` + +If you use the table format: + +```markdown +## FeatureName glossary + +This glossary provides definitions for terms related to FeatureName. + +| Term | Definition | Additional category | +|--------|-------------------------|---------------------| +| Term A | Term A does this thing. | | +| Term B | Term B does this thing. | | +| Term C | Term C does this thing. | | +``` + +## Glossary topic titles + +Use `FeatureName glossary`. + +Don't use alternatives to `glossary`. For example: + +- `Terminology` +- `Glossary of terms` +- `Glossary of common terms` +- `Definitions` diff --git a/doc/development/documentation/topic_types/index.md b/doc/development/documentation/topic_types/index.md index cfc231c268a..37ed876fc59 100644 --- a/doc/development/documentation/topic_types/index.md +++ b/doc/development/documentation/topic_types/index.md @@ -16,22 +16,52 @@ Each topic on a page should be one of the following topic types: Even if a page is short, the page usually starts with a concept and then includes a task or reference topic. -The tech writing team sometimes uses the acronym `CTRT` to refer to our topic types. +The tech writing team sometimes uses the acronym `CTRT` to refer to the topic types. The acronym refers to the first letter of each topic type. -In addition to the four primary topic types, we also have a page type for -[Tutorials](tutorial.md) and [Get started](#get-started). +## Other page and topic types + +In addition to the four primary topic types, you can use the following: + +- Page types: [Tutorials](tutorial.md) and [Get started](#get-started) +- Topic type: [Related topics](#related-topics) +- Page or topic type: [Glossaries](glossary.md) + +## Pages and topics to avoid + +You should avoid: + +- Pages that are exclusively links to other pages. The only exception are + top-level pages that aid with navigation. +- Topics that have one or two sentences only. In these cases: + - Incorporate the information in another topic. + - If the sentence links to another page, use a [Related topics](#related-topics) link instead. + +## Topic title guidelines + +In general, for topic titles: + +- Be clear and direct. Make every word count. +- Use articles and prepositions. +- Follow [capitalization](../styleguide/index.md#capitalization) guidelines. +- Do not repeat text from earlier topic titles. For example, if the page is about merge requests, + instead of `Troubleshooting merge requests`, use only `Troubleshooting`. + +See also [guidelines for heading levels in Markdown](../styleguide/index.md#heading-levels-in-markdown). ## Related topics If inline links are not sufficient, you can create a topic called **Related topics** and include an unordered list of related topics. This topic should be above the Troubleshooting section. +Links in this section should be brief and scannable. They are usually not +full sentences, and so should not end in a period. + ```markdown ## Related topics -- [Configure your pipeline](link-to-topic). -- [Trigger a pipeline manually](link-to-topic). +- [CI/CD variables](link-to-topic) +- [Environment variables](link-to-topic) ``` ## Get started @@ -57,22 +87,3 @@ consider using subsections for each distinct task. In the left nav, use `Get started` as the text. On the page itself, spell out the full name. For example, `Get started with application security`. - -## Topics and resources - -Some pages are solely a list of links to other documentation. - -We do not encourage this page type. Lists of links can get out-of-date quickly -and offer little value to users, who prefer to search to find information. - -## Topic title guidelines - -In general, for topic titles: - -- Be clear and direct. Make every word count. -- Use articles and prepositions. -- Follow [capitalization](../styleguide/index.md#capitalization) guidelines. -- Do not repeat text from earlier topic titles. For example, if the page is about merge requests, - instead of `Troubleshooting merge requests`, use only `Troubleshooting`. - -See also [guidelines for heading levels in Markdown](../styleguide/index.md#heading-levels-in-markdown). diff --git a/doc/development/documentation/topic_types/task.md b/doc/development/documentation/topic_types/task.md index f596ffc9b8b..2ddfd841ec3 100644 --- a/doc/development/documentation/topic_types/task.md +++ b/doc/development/documentation/topic_types/task.md @@ -155,4 +155,4 @@ how to write the phrase for each role. ## Related topics -- [View the format for writing task steps](../styleguide/index.md#navigation). +- [How to write task steps](../styleguide/index.md#navigation) diff --git a/doc/development/documentation/topic_types/tutorial.md b/doc/development/documentation/topic_types/tutorial.md index 1b1426a0465..b75b1e6b629 100644 --- a/doc/development/documentation/topic_types/tutorial.md +++ b/doc/development/documentation/topic_types/tutorial.md @@ -13,9 +13,17 @@ In general, you might consider using a tutorial when: of sub-steps. - The steps cover a variety of GitLab features or third-party tools. -Tutorials are learning aids that complement our core documentation. -They do not introduce new features. -Always use the primary [topic types](index.md) to document new features. +## Tutorial guidance + +- Tutorials are not [tasks](task.md). A task gives instructions for one procedure. + A tutorial combines multiple tasks to achieve a specific goal. +- Tutorials provide a working example. Ideally the reader can create the example the + tutorial describes. If they can't replicate it exactly, they should be able + to replicate something similar. +- Tutorials do not introduce new features. +- Tutorials do not need to adhere to the Single Source of Truth tenet. While it's not + ideal to duplicate content that is available elsewhere, it's worse to force the reader to + leave the page to find what they need. ## Tutorial format @@ -31,7 +39,9 @@ To create a website: 1. [Do the first task](#do-the-first-task) 1. [Do the second task](#do-the-second-task) -Prerequisites (optional): +## Prerequisites + +This topic is optional. - Thing 1 - Thing 2 diff --git a/doc/development/emails.md b/doc/development/emails.md index 2d03d8bca2f..fdcdcec814d 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -151,37 +151,10 @@ fork of [`MailRoom`](https://github.com/tpitale/mail_room/), to ensure that we can make updates quickly to the gem if necessary. We try to upstream changes as soon as possible and keep the two projects in sync. -Updating the `gitlab-mail_room` dependency in `Gemfile` is deprecated at -the moment in favor of updating the version in -[Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/config/software/mail_room.rb) -(see [example merge request](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5816)) -and Helm Chart configuration (see [example merge request](https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/854)). +To update MailRoom: -#### Rationale - -This was done because to avoid [thread deadlocks](https://github.com/ruby/net-imap/issues/14), `MailRoom` needs -an updated version of the `net-imap` gem. However, this -[version of the net-imap cannot be installed by an unprivileged user](https://github.com/ruby/net-imap/issues/14) due to -[an error installing the digest gem](https://github.com/ruby/digest/issues/14). -[This bug in the Ruby interpreter](https://bugs.ruby-lang.org/issues/17761) was fixed in Ruby -3.0.2. - -Updating the gem directly in the GitLab Rails `Gemfile` caused a [production incident](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4053) -since Kubernetes pods do not run as privileged users. - -Note that Omnibus GitLab runs `/opt/gitlab/embedded/bin/mail_room` -instead of `bundle exec rake` to avoid loading the older version. With a -Kubernetes install, the MailRoom version has always been explicitly set -in the Helm Chart configuration rather than the `Gemfile`. - -#### Preserving backwards compatibility - -Removing the `Gemfile` would break incoming email processing for source -installs. For now, source installs are advised to upgrade manually to -the version specified in Omnibus and run `bin/mail_room` directly as -done with Omnibus. - -We can reconsider this deprecation once we upgrade to Ruby 3.0. +1. Update `Gemfile` in GitLab Rails (see [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116494)). +1. Update the Helm Chart configuration (see [example merge request](https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/854)). --- diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index fd5b0ea15e6..5e68921510e 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -29,7 +29,7 @@ the documentation on that project if you want to dig into more advanced topics o aware that the documentation there reflects what's in the main branch and may not be the same as the version being used in GitLab. -## Glossary of terms +## Glossary To ensure a shared language, you should understand these fundamental terms we use when communicating about experiments: diff --git a/doc/development/fe_guide/content_editor.md b/doc/development/fe_guide/content_editor.md index 5c7fe68fec5..25140a067ca 100644 --- a/doc/development/fe_guide/content_editor.md +++ b/doc/development/fe_guide/content_editor.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Editor +stage: Plan +group: Knowledge info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/fe_guide/customizable_dashboards.md b/doc/development/fe_guide/customizable_dashboards.md index 26c73b26126..9e45c660745 100644 --- a/doc/development/fe_guide/customizable_dashboards.md +++ b/doc/development/fe_guide/customizable_dashboards.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Customizable dashboards -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98610) in GitLab 15.5 as an [Alpha feature](../../policy/alpha-beta-support.md#alpha-features). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98610) in GitLab 15.5 as an [Experiment](../../policy/alpha-beta-support.md#experiment). Customizable dashboards provide a dashboard structure that allows users to create their own dashboards and commit the structure to a repository. diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 5f0855d8f91..da3a6eff79d 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -1129,53 +1129,7 @@ We use [subscriptions](https://www.apollographql.com/docs/react/data/subscriptio **NOTE:** We cannot test subscriptions using GraphiQL, because they require an ActionCable client, which GraphiQL does not support at the moment. -Subscriptions don't require any additional configuration of Apollo Client instance, you can use them in the application right away. To distinguish subscriptions from queries and mutations, we recommend naming them with `.subscription.graphql` extension: - -```graphql -// ~/sidebar/queries/issuable_assignees.subscription.graphql - -subscription issuableAssigneesUpdated($issuableId: IssuableID!) { - issuableAssigneesUpdated(issuableId: $issuableId) { - ... on Issue { - assignees { - nodes { - ...User - status { - availability - } - } - } - } - } -} -``` - -When using GraphQL subscriptions in Vue application, we recommend updating existing Apollo query results with [subscribeToMore](https://apollo.vuejs.org/guide/apollo/subscriptions.html#subscribe-to-more) option: - -```javascript -import issuableAssigneesSubscription from '~/sidebar/queries/issuable_assignees.subscription.graphql' - -apollo: { - issuable: { - query() { - return assigneesQueries[this.issuableType].query; - }, - subscribeToMore: { - // Specify the subscription that will update the query - document() { - return issuableAssigneesSubscription; - }, - variables() { - return { - issuableId: convertToGraphQLId(this.issuableClass, this.issuableId), - }; - }, - }, - }, -}, -``` - -We would need also to define a field policy similarly like we do it for the [paginated queries](#defining-field-merge-policy) +Refer to the [Real-time widgets developer guide](../real_time.md) for a comprehensive introduction to subscriptions. ### Best Practices @@ -1446,6 +1400,24 @@ describe('when query times out', () => { }); ``` +Previously, we've used `{ mocks: { $apollo ...}}` on `mount` to test Apollo functionality. This approach is discouraged - proper `$apollo` mocking leaks a lot of implementation details to the tests. Consider replacing it with mocked Apollo provider + +```javascript +wrapper = mount(SomeComponent, { + mocks: { + // avoid! Mock real graphql queries and mutations instead + $apollo: { + mutate: jest.fn(), + queries: { + groups: { + loading, + }, + }, + }, + }, +}); +``` + #### Testing `@client` queries ##### Using mock resolvers diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 6403cff549f..ef506f260f7 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -11,7 +11,7 @@ across the GitLab frontend team. ## Overview -GitLab is built on top of [Ruby on Rails](https://rubyonrails.org). It uses [Haml](https://haml.info/) and a JavaScript-based frontend with [Vue.js](https://vuejs.org). +GitLab is built on top of [Ruby on Rails](https://rubyonrails.org). It uses [Haml](https://haml.info/) and a JavaScript-based frontend with [Vue.js](https://vuejs.org). If you are not sure when to use Vue on top of Haml-page, please read [this explanation](vue.md#when-to-add-vue-application). <!-- vale gitlab.Spelling = NO --> diff --git a/doc/development/fe_guide/merge_request_widget_extensions.md b/doc/development/fe_guide/merge_request_widget_extensions.md index 4242dd837f8..07029aec015 100644 --- a/doc/development/fe_guide/merge_request_widget_extensions.md +++ b/doc/development/fe_guide/merge_request_widget_extensions.md @@ -329,7 +329,6 @@ To generate these known events for a single widget: 1. `product_section` = `dev` 1. `product_stage` = `create` 1. `product_group` = `code_review` - 1. `product_category` = `code_review` 1. `introduced_by_url` = `'[your MR]'` 1. `options.events` = (the event in the command from above that generated this file, like `i_code_review_merge_request_widget_test_reports_count_view`) - This value is how the telemetry events are linked to "metrics" so this is probably one of the more important values. diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index 20609718217..432e66bee33 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -474,4 +474,4 @@ General tips: - [WebPage Test](https://www.webpagetest.org) for testing site loading time and size. - [Google PageSpeed Insights](https://pagespeed.web.dev/) grades web pages and provides feedback to improve the page. - [Profiling with Chrome DevTools](https://developer.chrome.com/docs/devtools/) -- [Browser Diet](https://browserdiet.com/) is a community-built guide that catalogues practical tips for improving web page performance. +- [Browser Diet](https://github.com/zenorocha/browser-diet) was a community-built guide that cataloged practical tips for improving web page performance. diff --git a/doc/development/fe_guide/registry_architecture.md b/doc/development/fe_guide/registry_architecture.md index d86f8416db6..cf267e80619 100644 --- a/doc/development/fe_guide/registry_architecture.md +++ b/doc/development/fe_guide/registry_architecture.md @@ -14,7 +14,7 @@ Existing registries: - Package Registry - Container Registry -- Infrastructure Registry +- Terraform Module Registry - Dependency Proxy ## Frontend architecture diff --git a/doc/development/fe_guide/source_editor.md b/doc/development/fe_guide/source_editor.md index 9f52850041d..2c115effcf9 100644 --- a/doc/development/fe_guide/source_editor.md +++ b/doc/development/fe_guide/source_editor.md @@ -17,6 +17,14 @@ GitLab features use it, including: - [Web Editor](../../user/project/repository/web_editor.md) - [Security Policies](../../user/application_security/policies/index.md) +## When to use Source Editor + +Use Source Editor only when users need to edit the file content. +If you only need to display source code, consider using the [`BlobContent`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/blob/components/blob_content.vue) component. + +If the page you're working on is already loading the Source Editor, +displaying read-only content in the Source Editor is still a valid option. + ## How to use Source Editor Source Editor is framework-agnostic and can be used in any application, including both @@ -56,7 +64,7 @@ An instance of Source Editor accepts the following configuration options: | `blobContent` | `false` | `String`: The initial content to render in the editor. | | `extensions` | `false` | `Array`: Extensions to use in this instance. | | `blobGlobalId` | `false` | `String`: An auto-generated property.<br>**Note:** This property may go away in the future. Do not pass `blobGlobalId` unless you know what you're doing.| -| Editor Options | `false` | `Object(s)`: Any property outside of the list above is treated as an Editor Option for this particular instance. Use this field to override global Editor Options on the instance level. A full [index of Editor Options](https://microsoft.github.io/monaco-editor/api/enums/monaco.editor.EditorOption.html) is available. | +| Editor Options | `false` | `Object(s)`: Any property outside of the list above is treated as an Editor Option for this particular instance. Use this field to override global Editor Options on the instance level. A full [index of Editor Options](https://microsoft.github.io/monaco-editor/docs.html#enums/editor.EditorOption.html) is available. | ## API @@ -68,7 +76,7 @@ with additional functions on the instance level: | --------------------- | ----- | ----- | | `updateModelLanguage` | `path`: String | Updates the instance's syntax highlighting to follow the extension of the passed `path`. Available only on the instance level. | | `use` | Array of objects | Array of extensions to apply to the instance. Accepts only an array of **objects**. The extensions' ES6 modules must be fetched and resolved in your views or components before they're passed to `use`. Available on the instance and global editor (all instances) levels. | -| Monaco Editor options | See [documentation](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.IStandaloneCodeEditor.html) | Default Monaco editor options. | +| Monaco Editor options | See [documentation](https://microsoft.github.io/monaco-editor/docs.html#interfaces/editor.IStandaloneCodeEditor.html) | Default Monaco editor options. | ## Tips diff --git a/doc/development/fe_guide/style/javascript.md b/doc/development/fe_guide/style/javascript.md index b35ffdd8669..987543642f0 100644 --- a/doc/development/fe_guide/style/javascript.md +++ b/doc/development/fe_guide/style/javascript.md @@ -2,7 +2,6 @@ stage: none group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -disqus_identifier: 'https://docs.gitlab.com/ee/development/fe_guide/style_guide_js.html' --- # JavaScript style guide @@ -332,19 +331,23 @@ Only export the constants as a collection (array, or object) when there is a nee ## Error handling -When catching a server-side error you should use the error message +When catching a server-side error, you should use the error message utility function contained in `app/assets/javascripts/lib/utils/error_message.js`. -This utility parses the received error message and checks for a prefix that indicates -whether the message is meant to be user-facing or not. The utility returns -an object with the message, and a boolean indicating whether the message is meant to be user-facing or not. Please make sure that the Backend is aware of the utils usage and is adding the prefix -to the error message accordingly. +This utility accepts two parameters: the error object received from the server response and a +default error message. The utility examines the message in the error object for a prefix that +indicates whether the message is meant to be user-facing or not. If the message is intended +to be user-facing, the utility returns it as is. Otherwise, it returns the default error +message passed as a parameter. ```javascript import { parseErrorMessage } from '~/lib/utils/error_message'; onError(error) { - const { message, userFacing } = parseErrorMessage(error); - - const errorMessage = userFacing ? message : genericErrorText; + const errorMessage = parseErrorMessage(error, genericErrorText); } ``` + +To benefit from this parsing mechanism, the utility user should ensure that the server-side +code is aware of this utility's usage and prefixes the error messages where appropriate +before sending them back to the user. See +[Error handling for API](../../api_styleguide.md#error-handling) for more information. diff --git a/doc/development/fe_guide/style/scss.md b/doc/development/fe_guide/style/scss.md index aed7310e95d..e760b0adaaa 100644 --- a/doc/development/fe_guide/style/scss.md +++ b/doc/development/fe_guide/style/scss.md @@ -2,7 +2,6 @@ stage: none group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -disqus_identifier: 'https://docs.gitlab.com/ee/development/fe_guide/style_guide_scss.html' --- # SCSS style guide @@ -77,7 +76,7 @@ These mixins should be used to replace _magic values_ in our code. For example a `margin-top: 8px` is a good candidate for the `@include gl-mt-3` mixin replacement. Avoid using utility mixins for [pre-defined CSS keywords](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Values_and_Units#pre-defined_keyword_values). -For example prefer `display: flex` over `@include gl-display-flex`. +For example prefer `display: flex` over `@include gl-display-flex`. Utility mixins are particularly useful for encapsulating our design system but there is no need to encapsulate simple properties. ```scss // Bad diff --git a/doc/development/fe_guide/style/vue.md b/doc/development/fe_guide/style/vue.md index e9d2a724d9d..a3ab1c1af30 100644 --- a/doc/development/fe_guide/style/vue.md +++ b/doc/development/fe_guide/style/vue.md @@ -650,6 +650,18 @@ over [`expect.objectContaining`](https://jestjs.io/docs/expect#expectobjectconta }); ``` +### Testing props validation + +1. When checking component props use `assertProps` helper. Props validation failures will be thrown as errors + +```javascript +import { assertProps } from 'helpers/assert_props' + +// ... + +expect(() => assertProps(SomeComponent, { invalidPropValue: '1', someOtherProp: 2 })).toThrow() +``` + ## The JavaScript/Vue Accord The goal of this accord is to make sure we are all on the same page. diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 119a91e33b5..65ceb9f0220 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -16,11 +16,23 @@ What is described in the following sections can be found in these examples: - [Security products](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/app/assets/javascripts/vue_shared/security_reports) - [Registry](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/app/assets/javascripts/registry/stores) +## When to add Vue application + +Sometimes, HAML page is enough to satisfy requirements. This statement is correct primarily for the static pages or pages that have very little logic. How do we know it's worth adding a Vue application to the page? The answer is "when we need to maintain application state and synchronize the rendered page with it". + +To better explain this, let's imagine the page that has one toggle, and toggling it sends an API request. This case does not involve any state we want to maintain, we send the request and switch the toggle. However, if we add ont more toggle that should always be the opposite to the first one, we need a _state_: one toggle should be "aware" about the state of another one. When written in plain JavaScript, this logic usually involves listening to DOM event and reacting with modifying DOM. Cases like this are much easier to handle with Vue.js so we should create a Vue application here. + +### What are some flags signaling that you might need Vue application? + +- when you need to define complex conditionals based on multiple factors and update them on user interaction; +- when you have to maintain any form of application state and share it between tags/elements; +- when you expect complex logic to be added in the future - it's easier to start with basic Vue application than having to rewrite JS/HAML to Vue on the next step. + ## Vue architecture -All new features built with Vue.js must follow a [Flux architecture](https://facebook.github.io/flux/). +All new features built with Vue.js must follow a [Flux architecture](https://facebookarchive.github.io/flux/). The main goal we are trying to achieve is to have only one data flow, and only one data entry. -To achieve this goal we use [Vuex](#vuex). +To achieve this goal we use [Vuex](#vuex) or [Apollo Client](graphql.md#libraries) You can also read about this architecture in Vue documentation about [state management](https://v2.vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch) @@ -121,7 +133,7 @@ export default { ```javascript //index.js import MyComponent from './my_component.vue' -import initSimpleApp from '~/helpers/init_simple_app_helper' +import { initSimpleApp } from '~/helpers/init_simple_app_helper' initSimpleApp('#js-my-element', MyComponent) ``` @@ -391,6 +403,87 @@ You can read more about components in Vue.js site, [Component System](https://v2 Check this [page](vuex.md) for more details. +### Vue Router + +To add [Vue Router](https://router.vuejs.org/) to a page: + +1. Add a catch-all route to the Rails route file using a wildcard named `*vueroute`: + + ```ruby + # example from ee/config/routes/project.rb + + resources :iteration_cadences, path: 'cadences(/*vueroute)', action: :index + ``` + + The above example serves the `index` page from `iteration_cadences` controller to any route + matching the start of the `path`, for example `groupname/projectname/-/cadences/123/456/`. +1. Pass the base route (everything before `*vueroute`) to the frontend to use as the `base` parameter to initialize Vue Router: + + ```haml + .js-my-app{ data: { base_path: project_iteration_cadences_path(project) } } + ``` + +1. Initialize the router: + + ```javascript + Vue.use(VueRouter); + + export function createRouter(basePath) { + return new VueRouter({ + routes: createRoutes(), + mode: 'history', + base: basePath, + }); + } + ``` + +1. Add a fallback for unrecognised routes with `path: '*'`. Either: + - Add a redirect to the end of your routes array: + + ```javascript + const routes = [ + { + path: '/', + name: 'list-page', + component: ListPage, + }, + { + path: '*', + redirect: '/', + }, + ]; + ``` + + - Add a fallback component to the end of your routes array: + + ```javascript + const routes = [ + { + path: '/', + name: 'list-page', + component: ListPage, + }, + { + path: '*', + component: NotFound, + }, + ]; + ``` + +1. Optional. To also allow using the path helper for child routes, add `controller` and `action` + parameters to use the parent controller. + + ```ruby + resources :iteration_cadences, path: 'cadences(/*vueroute)', action: :index do + resources :iterations, only: [:index, :new, :edit, :show], constraints: { id: /\d+/ }, controller: :iteration_cadences, action: :index + end + ``` + + This means routes like `/cadences/123/iterations/456/edit` can be validated on the backend, + for example to check group or project membership. + It also means we can use the `_path` helper, which means we can load the page in feature specs + without manually building the `*vueroute` part of the path.. + ### Mixing Vue and jQuery - Mixing Vue and jQuery is not recommended. diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md index c3cee596d8a..5828855a725 100644 --- a/doc/development/feature_development.md +++ b/doc/development/feature_development.md @@ -80,7 +80,7 @@ Consult these topics for information on contributing to specific GitLab features - [Sidekiq guidelines](sidekiq/index.md) for working with Sidekiq workers - [Working with Gitaly](gitaly.md) - [Advanced search integration docs](advanced_search.md) -- [Working with merge request diffs](diffs.md) +- [Working with merge request diffs](merge_request_concepts/diffs/index.md) - [Approval Rules](merge_request_concepts/approval_rules.md) - [Repository mirroring](repository_mirroring.md) - [Uploads development guide](uploads/index.md) @@ -130,9 +130,8 @@ See [database guidelines](database/index.md). The following integration guides are internal. Some integrations require access to administrative accounts of third-party services and are available only for GitLab team members to contribute to: -- [Jira app development](https://gitlab.com/gitlab-org/manage/integrations/team/-/blob/main/integrations/jira.md) -- [Slack app development](https://gitlab.com/gitlab-org/manage/integrations/team/-/blob/main/integrations/slack.md) -- [ZenTao development](https://gitlab.com/gitlab-org/manage/integrations/team/-/blob/main/integrations/zentao.md) +- [Jira app development](https://gitlab.com/gitlab-org/manage/integrate/team/-/blob/main/integrations/jira.md) +- [GitLab for Slack app development](https://gitlab.com/gitlab-org/manage/integrate/team/-/blob/main/integrations/slack.md) ## Testing guides diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index f8a592f98f5..0e88c68cc99 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -40,7 +40,7 @@ easier to measure the impact of both separately. The GitLab feature library (using [Flipper](https://github.com/jnunemaker/flipper), and covered in the -[Feature Flags process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) guide) supports rolling out changes to a percentage of +[Feature flags process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) guide) supports rolling out changes to a percentage of time to users. This in turn can be controlled using [GitLab ChatOps](../../ci/chatops/index.md). For an up to date list of feature flag commands please see @@ -115,13 +115,13 @@ incidents or in-progress change issues, for example: 2021-06-29 Canary deployment failing QA tests ``` -Before enabling a feature flag, verify that you are not violating any [Production Change Lock periods](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl) and are in compliance with the [Feature Flags and the Change Management Process](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process). +Before enabling a feature flag, verify that you are not violating any [Production Change Lock periods](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl) and are in compliance with the [Feature flags and the Change Management Process](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process). The following `/chatops` commands should be performed in the Slack `#production` channel. When you begin to enable the feature, please link to the relevant -Feature Flag Rollout Issue within a Slack thread of the first `/chatops` +feature flag rollout issue within a Slack thread of the first `/chatops` command you make so people can understand the change if they need to. To enable a feature for 25% of the time, run the following in Slack: @@ -358,7 +358,7 @@ After turning on the feature flag, you need to [monitor the relevant graphs](htt In this illustration, you can see that the Apdex score started to decline after the feature flag was enabled at `09:46`. The feature flag was then deactivated at `10:31`, and the service returned to the original value: - + ### Feature flag change logging diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 9b6876ac0bc..40a5d2a2abc 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -83,7 +83,7 @@ used for deploying unfinished code to production. Most feature flags used at GitLab are the `development` type. A `development` feature flag must have a rollout issue -created from the [Feature Flag Roll Out template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md). +created from the [Feature flag Roll Out template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md). The format for `development` feature flags is `Feature.<state>(:<dev_flag_name>)`. To enable and disable them, run on the GitLab Rails console: @@ -252,7 +252,7 @@ deleting feature flags. ## Develop with a feature flag -There are two main ways of using Feature Flags in the GitLab codebase: +There are two main ways of using feature flags in the GitLab codebase: - [Backend code (Rails)](#backend) - [Frontend code (VueJS)](#frontend) diff --git a/doc/development/features_inside_dot_gitlab.md b/doc/development/features_inside_dot_gitlab.md index dda2c05288f..f1915327c8c 100644 --- a/doc/development/features_inside_dot_gitlab.md +++ b/doc/development/features_inside_dot_gitlab.md @@ -13,9 +13,9 @@ When implementing new features, please refer to these existing features to avoid - [Issue Templates](../user/project/description_templates.md#create-an-issue-template): `.gitlab/issue_templates/`. - [Merge request Templates](../user/project/description_templates.md#create-a-merge-request-template): `.gitlab/merge_request_templates/`. - [GitLab agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/configuration_repository.md#layout): `.gitlab/agents/`. -- [CODEOWNERS](../user/project/code_owners.md#set-up-code-owners): `.gitlab/CODEOWNERS`. +- [CODEOWNERS](../user/project/codeowners/index.md#set-up-code-owners): `.gitlab/CODEOWNERS`. - [Route Maps](../ci/review_apps/index.md#route-maps): `.gitlab/route-map.yml`. - [Customize Auto DevOps Helm Values](../topics/autodevops/customize.md#customize-helm-chart-values): `.gitlab/auto-deploy-values.yaml`. - [Insights](../user/project/insights/index.md#configure-project-insights): `.gitlab/insights.yml`. -- [Service Desk Templates](../user/project/service_desk.md#using-customized-email-templates): `.gitlab/service_desk_templates/`. +- [Service Desk Templates](../user/project/service_desk.md#create-customized-email-templates): `.gitlab/service_desk_templates/`. - [Web IDE](../user/project/web_ide/index.md#web-ide-configuration-file): `.gitlab/.gitlab-webide.yml`. diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md index 38c0d77bcae..d4dc66bae8f 100644 --- a/doc/development/gemfile.md +++ b/doc/development/gemfile.md @@ -65,6 +65,17 @@ This means that new dependencies should, at a minimum, meet the following criter - If the project uses a C extension, consider requesting an additional review from a C or MRI domain expert. C extensions can greatly impact GitLab stability and performance. +## Gems that require a domain expert approval + +Changes to the following gems require a domain expert review and approval by a backend team member of the group. + +For gems not listed in this table, it's still recommended but not required that you find a domain expert to review changes. + +| Gem | Requires approval by | +| ------ | ------ | +| `doorkeeper` | [Manage:Authentication and Authorization](https://about.gitlab.com/handbook/product/categories/#authentication-and-authorization-group) | +| `doorkeeper-openid_connect` | [Manage:Authentication and Authorization](https://about.gitlab.com/handbook/product/categories/#authentication-and-authorization-group) | + ## Request an Appsec review When adding a new gem to our `Gemfile` or even changing versions in diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index 6014ccbfb39..e98ebe5efe1 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -23,15 +23,17 @@ At the Git level, we achieve deduplication by using Git alternates is a mechanism that lets a repository borrow objects from another repository on the same machine. -If we want repository A to borrow from repository B, we first write a -path that resolves to `B.git/objects` in the special file -`A.git/objects/info/alternates`. This establishes the alternates link. -Next, we must perform a Git repack in A. After the repack, any objects -that are duplicated between A and B are deleted from A. Repository -A is now no longer self-contained, but it still has its own refs and -configuration. Objects in A that are not in B remain in A. For this -to work, it is of course critical that **no objects ever get deleted from -B** because A might need them. +To make repository A borrow from repository B: + +1. Establish the alternates link in the special file `A.git/objects/info/alternates` + by writing a path that resolves to `B.git/objects`. +1. In repository A, run `git repack` to remove all objects in repository A that + also exist in repository B. + +After the repack, repository A is no longer self-contained, but still contains its +own refs and configuration. Objects in A that are not in B remain in A. For this +configuration to work, **objects must not be deleted from repository B** because +repository A might need them. WARNING: Do not run `git prune` or `git gc` in object pool repositories, which are diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index addab512f89..408f3d5fa3c 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -54,16 +54,6 @@ they are merged. - See [below](#running-tests-with-a-locally-modified-version-of-gitaly) for instructions on running GitLab tests with a modified version of Gitaly. - In GDK run `gdk install` and restart GDK using `gdk restart` to use a locally modified Gitaly version for development -### `gitaly-ruby` - -It is possible to implement and test RPCs in Gitaly using Ruby code, -in -[`gitaly-ruby`](https://gitlab.com/gitlab-org/gitaly/tree/master/ruby). -This should make it easier to contribute for developers who are less -comfortable writing Go code. - -For more information, see the [Beginner's guide to Gitaly contributions](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/beginners_guide.md). - ## Gitaly-Related Test Failures If your test-suite is failing with Gitaly issues, as a first step, try running: @@ -249,7 +239,7 @@ Re-run steps 2-5 each time you want to try out new changes. [Return to Development documentation](index.md) -## Wrapping RPCs in Feature Flags +## Wrapping RPCs in feature flags Here are the steps to gate a new feature in Gitaly behind a feature flag. diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index 2d9ae768d33..86142efa679 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -70,7 +70,12 @@ don't need to perform this work in parallel. This worker imports all pull requests. For every pull request a job for the `Gitlab::GithubImport::ImportPullRequestWorker` worker is scheduled. -### 5. Stage::ImportPullRequestsMergedByWorker +### 5. Stage::ImportCollaboratorsWorker + +This worker imports all repository collaborators. For every collaborator, we schedule a job +for the `Gitlab::GithubImport::ImportCollaboratorWorker` worker. + +### 6. Stage::ImportPullRequestsMergedByWorker This worker imports the pull requests' _merged-by_ user information. The [_List pull requests_](https://docs.github.com/en/rest/pulls#list-pull-requests) @@ -79,21 +84,21 @@ individually to import this information. A `Gitlab::GithubImport::ImportPullRequestMergedByWorker` job is scheduled for each fetched pull request. -### 6. Stage::ImportPullRequestsReviewRequestsWorker +### 7. Stage::ImportPullRequestsReviewRequestsWorker This worker imports assigned reviewers of pull requests. For each pull request, this worker: - Fetches all assigned review requests. - Schedules a `Gitlab::GithubImport::PullRequests::ImportReviewRequestWorker` job for each fetched review request. -### 7. Stage::ImportPullRequestsReviewsWorker +### 8. Stage::ImportPullRequestsReviewsWorker This worker imports reviews of pull requests. For each pull request, this worker: - Fetches all the pages of reviews. - Schedules a `Gitlab::GithubImport::ImportPullRequestReviewWorker` job for each fetched review. -### 8. Stage::ImportIssuesAndDiffNotesWorker +### 9. Stage::ImportIssuesAndDiffNotesWorker This worker imports all issues and pull request comments. For every issue, we schedule a job for the `Gitlab::GithubImport::ImportIssueWorker` worker. For @@ -109,7 +114,7 @@ label links in the same worker removes the need for performing a separate crawl through the API data, reducing the number of API calls necessary to import a project. -### 9. Stage::ImportIssueEventsWorker +### 10. Stage::ImportIssueEventsWorker This worker imports all issues and pull request events. For every event, we schedule a job for the `Gitlab::GithubImport::ImportIssueEventWorker` worker. @@ -125,7 +130,7 @@ Therefore, both issues and pull requests have a common API for most related thin NOTE: This stage is optional and can consume significant extra import time (controlled by `Gitlab::GithubImport::Settings`). -### 10. Stage::ImportNotesWorker +### 11. Stage::ImportNotesWorker This worker imports regular comments for both issues and pull requests. For every comment, we schedule a job for the @@ -136,7 +141,7 @@ returns comments for both issues and pull requests. This means we have to wait for all issues and pull requests to be imported before we can import regular comments. -### 11. Stage::ImportAttachmentsWorker +### 12. Stage::ImportAttachmentsWorker This worker imports note attachments that are linked inside Markdown. For each entity with Markdown text in the project, we schedule a job of: @@ -155,7 +160,7 @@ Each job: NOTE: It's an optional stage that could consume significant extra import time (controlled by `Gitlab::GithubImport::Settings`). -### 12. Stage::ImportProtectedBranchesWorker +### 13. Stage::ImportProtectedBranchesWorker This worker imports protected branch rules. For every rule that exists on GitHub, we schedule a job of @@ -164,7 +169,7 @@ For every rule that exists on GitHub, we schedule a job of Each job compares the branch protection rules from GitHub and GitLab and applies the strictest of the rules to the branches in GitLab. -### 13. Stage::FinishImportWorker +### 14. Stage::FinishImportWorker This worker completes the import process by performing some housekeeping (such as flushing any caches) and by marking the import as completed. diff --git a/doc/development/gitlab_flavored_markdown/index.md b/doc/development/gitlab_flavored_markdown/index.md index 064a7ecbfa9..cde83bff32e 100644 --- a/doc/development/gitlab_flavored_markdown/index.md +++ b/doc/development/gitlab_flavored_markdown/index.md @@ -1,6 +1,6 @@ --- stage: Create -group: Editor +group: IDE info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/gitlab_flavored_markdown/specification_guide/index.md b/doc/development/gitlab_flavored_markdown/specification_guide/index.md index 806ac3837bf..ae78daa3687 100644 --- a/doc/development/gitlab_flavored_markdown/specification_guide/index.md +++ b/doc/development/gitlab_flavored_markdown/specification_guide/index.md @@ -1,6 +1,6 @@ --- stage: Create -group: Editor +group: IDE info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -178,7 +178,7 @@ strings in the standard In this context, it should not be confused with other similar or related meanings of _example_, such as -[RSpec examples](https://relishapp.com/rspec/rspec-core/docs/example-groups/basic-structure-describe-it). +[RSpec examples](https://rspec.info/features/3-12/rspec-core/example-groups/basic-structure/). See the section on the [`glfm_official_specification.md`](#glfm_official_specificationmd) file for more details on the backtick-delimited Markdown+HTML example syntax. @@ -297,7 +297,7 @@ The Markdown dialect used in the GitLab application has a dual requirement for r 1. Rendering to static read-only HTML format, to be displayed in various places throughout the application. 1. Rendering editable content in the - [Content Editor](https://about.gitlab.com/direction/create/editor/content_editor/), + [Content Editor](https://about.gitlab.com/direction/plan/knowledge/content_editor/), a ["What You See Is What You Get" (WYSIWYG)](https://en.wikipedia.org/wiki/WYSIWYG) editor. The Content Editor supports real-time instant switching between an editable Markdown source and an editable WYSIWYG document. diff --git a/doc/development/gitlab_shell/index.md b/doc/development/gitlab_shell/index.md index 3a1af0fc9e9..0663341f806 100644 --- a/doc/development/gitlab_shell/index.md +++ b/doc/development/gitlab_shell/index.md @@ -217,6 +217,6 @@ sequenceDiagram ## Related topics -- [LICENSE](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/main/LICENSE). +- [LICENSE](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/main/LICENSE) - [PROCESS.md](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/main/PROCESS.md) - [Using the GitLab Shell chart](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/) diff --git a/doc/development/graphql_guide/reviewing.md b/doc/development/graphql_guide/reviewing.md new file mode 100644 index 00000000000..9c32179a89d --- /dev/null +++ b/doc/development/graphql_guide/reviewing.md @@ -0,0 +1,96 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# GraphQL API merge request checklist + +The GitLab GraphQL API has a fair degree of complexity so it's important that merge requests containing GraphQL changes be reviewed by someone familiar with GraphQL. +You can ping one via the `@gitlab-org/graphql-experts` group in a MR or in the [`#f_graphql` channel](https://gitlab.slack.com/archives/C6MLS3XEU) in Slack (available to GitLab team members only). + +GraphQL queries need to be reviewed for: + +- breaking changes +- authorization +- performance + +## Review criteria + +This is not an exhaustive list. + +### Description with sample query + +Ensure that the description includes a sample query with setup instructions. +Try running the query in [GraphiQL](../api_graphql_styleguide.md#graphiql) on your local GDK instance. + +### No breaking changes (unless after full deprecation cycle) + +Check the MR for any [breaking changes](../api_graphql_styleguide.md#breaking-changes). + +If a feature is marked as an [Experiment](../api_graphql_styleguide.md#mark-schema-items-as-alpha), you can make breaking changes immediately, with no deprecation period. + +For more information, see [deprecation and removal process](../../api/graphql/index.md#deprecation-and-removal-process). + +### Multiversion compatibility + +Ensure that multi-version compatibility is guaranteed. +This generally means frontend and backend code for the same GraphQL feature can't be shipped in the same release. + +For details, see [multiple version compatibility](../multi_version_compatibility.md). + +### Technical writing review + +Changes to the generated API docs require a technical writer review. + +### Changelog + +Public-facing changes that are not marked as an [Experiment](../api_graphql_styleguide.md#mark-schema-items-as-alpha) require a [changelog entry](../changelog.md). + +### Use the framework + +GraphQL is a framework with many moving parts. It's important that the framework is followed. + +- Do not manually invoke framework bits. For example, do not instantiate resolvers during execution and instead let the framework do that. +- You can subclass resolvers, as in `MyResolver.single` (see [deriving resolvers](../api_graphql_styleguide.md#deriving-resolvers)). +- Use the `ready?` method for more complex argument logic (see [correct use of resolver#ready](../api_graphql_styleguide.md#correct-use-of-resolverready)). +- Use the `prepare` method for more complex argument validation (see [validating arguments](../api_graphql_styleguide.md#validating-arguments)). + +For details, see [resolver guide](../api_graphql_styleguide.md#writing-resolvers). + +### Authorization + +Ensure proper authorization is followed and that `authorize :some_ability` is tested in the specs. + +For details, see [authorization guide](authorization.md). + +### Performance + +Ensure: + +- You avoid N+1s with [BatchLoader](batchloader.md) or [Lookahead](../api_graphql_styleguide.md#look-ahead) when appropriate. +- You use [laziness](../api_graphql_styleguide.md#laziness) appropriately. + +### Use appropriate types + +For example: + +- [`TimeType`](../api_graphql_styleguide.md#typestimetype) for Ruby `Time` and `DateTime` objects. +- Global IDs for `id` fields + +For details, see [types](../api_graphql_styleguide.md#types). + +### Appropriate complexity + +Query complexity is a way of quantifying how expensive a query is likely to be. Query complexity limits are defined as constants in the schema. +When a resolver or type is expensive to call we need to ensure that the query complexity reflects that. + +For details, see [max complexity](../api_graphql_styleguide.md#max-complexity), [field complexity](../api_graphql_styleguide.md#field-complexity) and [query limits](../api_graphql_styleguide.md#query-limits). + +### Testing + +- Resolver (unit) specs are deprecated in favour of request (integration) specs. +- Many aspects of our framework are outside the `resolve` method and a request spec is the only way to ensure they behave properly. +- Every GraphQL change MR should ideally have changes to API specs. + +For details, see [testing guide](../api_graphql_styleguide.md#testing). diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 1957662317a..964bee8620e 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -19,6 +19,8 @@ are very appreciative of the work done by translators and proofreaders! - Tsegaselassie Tadesse - [GitLab](https://gitlab.com/tsega), [Crowdin](https://crowdin.com/profile/tsegaselassi) - Arabic - Proofreaders needed. +- Belarusian + - Anton Katsuba - [GitLab](https://gitlab.com/coinvariant), [Crowdin](https://crowdin.com/profile/aerialfiddle) - Bosnian - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) - Bulgarian @@ -126,6 +128,7 @@ are very appreciative of the work done by translators and proofreaders! - Proofreaders needed. - Spanish - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) + - David Elizondo - [GitLab](https://gitlab.com/daelmo), [Crowdin](https://crowdin.com/profile/daelmo) - Swedish - Johannes Nilsson - [GitLab](https://gitlab.com/nlssn), [Crowdin](https://crowdin.com/profile/nlssn) - Turkish diff --git a/doc/development/index.md b/doc/development/index.md index a39a83b257a..b8e67748bc9 100644 --- a/doc/development/index.md +++ b/doc/development/index.md @@ -1,5 +1,4 @@ --- -comments: false type: index, dev stage: none group: Development @@ -14,8 +13,8 @@ Learn how to contribute to the development of the GitLab product. This content is intended for GitLab team members as well as members of the wider community. - [Contribute to GitLab development](contributing/index.md) -- [Contribute to Omnibus development](https://docs.gitlab.com/omnibus/development/) -- [Contribute to GitLab Pages development](pages/index.md) - [Contribute to GitLab Runner development](https://docs.gitlab.com/runner/development/) -- [Contribute to Helm chart development](https://docs.gitlab.com/charts/development/) +- [Contribute to GitLab Pages development](pages/index.md) +- [Contribute to GitLab distribution development](distribution/index.md) +- [Contribute to the GitLab Design System](https://design.gitlab.com/get-started/contributing) - [Contribute to the GitLab documentation](documentation/index.md) diff --git a/doc/development/integrations/index.md b/doc/development/integrations/index.md index 6f42cb4d4fe..efe70e5948a 100644 --- a/doc/development/integrations/index.md +++ b/doc/development/integrations/index.md @@ -23,9 +23,11 @@ if you need clarification or spot any outdated information. - For example, `Integrations::FooBar` in `app/models/integrations/foo_bar.rb`. - For certain types of integrations, you can also build on these base classes: - `Integrations::BaseChatNotification` + - `Integrations::BaseCi` - `Integrations::BaseIssueTracker` - `Integrations::BaseMonitoring` - `Integrations::BaseSlashCommands` + - `Integrations::BaseThirdPartyWiki` - For integrations that primarily trigger HTTP calls to external services, you can also use the `Integrations::HasWebHook` concern. This reuses the [webhook functionality](../../user/project/integrations/webhooks.md) in GitLab through an associated `ServiceHook` model, and automatically records request logs @@ -37,9 +39,6 @@ if you need clarification or spot any outdated information. has_one :foo_bar_integration, class_name: 'Integrations::FooBar' ``` -1. TEMPORARY: Accommodate the current migration to [rename "services" to "integrations"](#rename-services-to-integrations): - - Add the integration's camel-cased name (`'FooBar'`) to `Gitlab::Integrations::StiType::NAMESPACED_INTEGRATIONS`. - ### Define properties Integrations can define arbitrary properties to store their configuration with the class method `Integration.prop_accessor`. @@ -237,9 +236,7 @@ module Integrations end ``` -### Expose the integration in the API - -#### REST API +### Expose the integration in the REST API To expose the integration in the [REST API](../../api/integrations.md): @@ -258,46 +255,6 @@ Sensitive fields are not exposed over the API. Sensitive fields are those fields - `token` - `webhook` -#### GraphQL API - -Integrations use the `Types::Projects::ServiceType` type by default, -which only exposes the `type` and `active` properties. - -To expose additional properties, you can write a class implementing `ServiceType`: - -```ruby -# in app/graphql/types/project/services/foo_bar_service_type.rb -module Types - module Projects - module Services - class FooBarServiceType < BaseObject - graphql_name 'FooBarService' - implements(Types::Projects::ServiceType) - authorize :read_project - - field :frobinity, - GraphQL::Types::Float, - null: true, - description: 'The level of frobinity.' - - field :foo_label, - GraphQL::Types::String, - null: true, - description: 'The foo label to apply.' - end - end - end -end -``` - -Each property you want to expose should have a field defined for it. You can also expose any public instance method of the integration. - -Contact a member of the Integrations team to discuss the best authorization. - -Reference documentation for GraphQL is automatically generated. - -You can also refer to our [GraphQL API style guide](../api_graphql_styleguide.md). - ## Availability of integrations By default, integrations are available on the project, group, and instance level. @@ -345,26 +302,8 @@ The strings should use the integration name as [namespace](../i18n/externalizati ## Ongoing migrations and refactorings -The Integrations team is in the process of some larger migrations that developers should be aware of. - -### [Rename "services" to "integrations"](https://gitlab.com/groups/gitlab-org/-/epics/2504) - -The "integrations" in GitLab were historically called "services", which frequently caused -confusion with our "service" classes in `app/services`. We sometimes also called -them "project services" because they were initially only available on projects, which is -not the case anymore. - -We decided to change the naming from "services" and "project services" to "integrations". -This refactoring is an ongoing effort, and there are still references to the old names in some places. - -Developers should be especially aware that we still use the old class names for the STI column -`integrations.type`. For example, a class `Integrations::FooBar` still stores -the old name `FooBarService` in the database. This mapping is handled via `Gitlab::Integrations::StiType` -and should be mostly transparent to the rest of the app. - -### [Consolidate integration settings](https://gitlab.com/groups/gitlab-org/-/epics/3955) - -We want to unify the way integration properties are defined. +Developers should be aware that the Integrations team is in the process of +[unifying the way integration properties are defined](https://gitlab.com/groups/gitlab-org/-/epics/3955). ## Integration examples diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index 4789280c09d..3d5e6aae25a 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -72,22 +72,6 @@ To avoid external dependencies like Gitpod and a Jira Cloud instance, use the [J 1. Go to `http://localhost:9292`. 1. Paste the token and select **Install GitLab.com Jira Cloud app**. -### Troubleshooting - -If the app install failed, you might need to delete `jira_connect_installations` from your database. - -1. Open the [database console](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md#access-postgresql). -1. Run `TRUNCATE TABLE jira_connect_installations CASCADE;`. - -#### Not authorized to access the file - -If you use Gitpod and you get an error about Jira not being able to access the descriptor file, you might need to make the GDK port public by following these steps: - -1. Open your GitLab workspace in Gitpod. -1. When the GDK is running, select **Ports** in the bottom-right corner. -1. On the left sidebar, select the port the GDK is listening to (typically `3000`). -1. If the port is marked as private, select the lock icon to make it public. - ## Test the GitLab OAuth authentication flow > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81126) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `jira_connect_oauth`. Disabled by default. @@ -123,3 +107,21 @@ The following steps describe setting up an environment to test the GitLab OAuth 1. Scroll down and expand the GitLab for Jira App section. 1. Go to [gitpod.io/variables](https://gitpod.io/variables). 1. Paste the Application ID into the **Jira Connect Application ID** field and select **Save changes**. + +## Troubleshooting + +### App installation fails + +If the app installation fails, you might need to delete `jira_connect_installations` from your database. + +1. Open the [database console](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md#access-postgresql). +1. Run `TRUNCATE TABLE jira_connect_installations CASCADE;`. + +### Not authorized to access the file + +If you use Gitpod and you get an error about Jira not being able to access the descriptor file, you might need to make the GDK port public by following these steps: + +1. Open your GitLab workspace in Gitpod. +1. When the GDK is running, select **Ports** in the bottom-right corner. +1. On the left sidebar, select the port the GDK is listening to (typically `3000`). +1. If the port is marked as private, select the lock icon to make it public. diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index 853541144fb..ab94cad3bdc 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -42,7 +42,7 @@ best place to integrate your own product and its results into GitLab. - Pipeline jobs serve a variety of purposes. Jobs can do scanning for and have implications for app security, corporate policy, or compliance. When complete, the job reports back on its status and creates a - [job artifact](../../ci/pipelines/job_artifacts.md) as a result. + [job artifact](../../ci/jobs/job_artifacts.md) as a result. - The [Merge Request Security Widget](../../ci/testing/index.md#security-reports) displays the results of the pipeline's security checks and the developer can review them. The developer can review both a summary and a detailed version @@ -84,7 +84,7 @@ and complete an integration with the Secure stage. to successfully display your own product's results with the rest of GitLab. - See detailed [technical directions](secure.md) for this step. - Read more about [job report artifacts](../../ci/yaml/index.md#artifactsreports). - - Read about [job artifacts](../../ci/pipelines/job_artifacts.md). + - Read about [job artifacts](../../ci/jobs/job_artifacts.md). - Your report artifact must be in one of our currently supported formats. For more information, see the [documentation on reports](secure.md#report). - Documentation for [SAST reports](../../user/application_security/sast/index.md#reports-json-format). @@ -95,7 +95,7 @@ and complete an integration with the Secure stage. and add the label `devops::secure`. - Once the job is completed, the data can be seen: - In the [Merge Request Security Report](../../ci/testing/index.md#security-reports) ([MR Security Report data flow](https://gitlab.com/snippets/1910005#merge-request-view)). - - While [browsing a Job Artifact](../../ci/pipelines/job_artifacts.md). + - While [browsing a Job Artifact](../../ci/jobs/job_artifacts.md). - In the [Security Dashboard](../../user/application_security/security_dashboard/index.md) ([Dashboard data flow](https://gitlab.com/snippets/1910005#project-and-group-dashboards)). 1. Optional: Provide a way to interact with results as Vulnerabilities: - Users can interact with the findings from your artifact within their workflow. They can dismiss the findings or accept them and create a backlog issue. diff --git a/doc/development/interacting_components.md b/doc/development/interacting_components.md index f4becd06245..7fb711795c1 100644 --- a/doc/development/interacting_components.md +++ b/doc/development/interacting_components.md @@ -29,5 +29,5 @@ See also [File Storage in GitLab](file_storage.md). ### Forks GitLab supports a great amount of features for [merge requests](../user/project/merge_requests/index.md). One -of them is the ability to create merge requests from and to [forks](../user/project/repository/forking_workflow.md#creating-a-fork), +of them is the ability to create merge requests from and to [forks](../user/project/repository/forking_workflow.md#create-a-fork), which should also be highly considered and tested upon development phase. diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index aa10bbeda9c..c1c0177609b 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -809,6 +809,107 @@ Example response: - CustomersDot +## Storage limit exclusions + +The namespace storage limit exclusion endpoints manage storage limit exclusions on top-level namespaces on GitLab.com. +These endpoints can only be consumed in the Admin Area of GitLab.com. + +### Retrieve storage limit exclusions + +Use a GET request to retrieve all `Namespaces::Storage::LimitExclusion` records. + +```plaintext +GET /namespaces/storage/limit_exclusions +``` + +Example request: + +```shell +curl --request GET \ + --url "https://gitlab.com/v4/namespaces/storage/limit_exclusions" \ + --header 'PRIVATE-TOKEN: <admin access token>' +``` + +Example response: + +```json +[ + { + "id": 1, + "namespace_id": 1234, + "namespace_name": "A Namespace Name", + "reason": "a reason to exclude the Namespace" + }, + { + "id": 2, + "namespace_id": 4321, + "namespace_name": "Another Namespace Name", + "reason": "another reason to exclude the Namespace" + }, +] +``` + +### Create a storage limit exclusion + +Use a POST request to create an `Namespaces::Storage::LimitExclusion`. + +```plaintext +POST /namespaces/:id/storage/limit_exclusion +``` + +| Attribute | Type | Required | Description | +|:------------|:--------|:---------|:------------| +| `reason` | string | yes | The reason to exclude the namespace. | + +Example request: + +```shell +curl --request POST \ + --url "https://gitlab.com/v4/namespaces/123/storage/limit_exclusion" \ + --header 'Content-Type: application/json' \ + --header 'PRIVATE-TOKEN: <admin access token>' \ + --data '{ + "reason": "a reason to exclude the Namespace" + }' +``` + +Example response: + +```json +{ + "id": 1, + "namespace_id": 1234, + "namespace_name": "A Namespace Name", + "reason": "a reason to exclude the Namespace" +} +``` + +### Delete a storage limit exclusion + +Use a DELETE request to delete a `Namespaces::Storage::LimitExclusion` for a namespace. + +```plaintext +DELETE /namespaces/:id/storage/limit_exclusion +``` + +Example request: + +```shell +curl --request DELETE \ + --url "https://gitlab.com/v4/namespaces/123/storage/limit_exclusion" \ + --header 'PRIVATE-TOKEN: <admin access token>' +``` + +Example response: + +```plaintext +204 +``` + +### Known consumers + +- GitLab.com Admin Area + ## CI/CD minutes provisioning The CI/CD Minutes endpoints are used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) diff --git a/doc/development/internal_users.md b/doc/development/internal_users.md index 67000d969af..541a7117c6a 100644 --- a/doc/development/internal_users.md +++ b/doc/development/internal_users.md @@ -44,5 +44,8 @@ Other examples of internal users: - [Ghost User](../user/profile/account/delete_account.md#associated-records) - [Support Bot](../user/project/service_desk.md#support-bot-user) - Visual Review Bot -- Resource access tokens (including [project access tokens](../user/project/settings/project_access_tokens.md)). - These are implemented as `project_bot` users with a `PersonalAccessToken`. +- Resource access tokens, including: + - [Project access tokens](../user/project/settings/project_access_tokens.md). + - [Group access tokens](../user/group/settings/group_access_tokens.md). + + These are implemented as `project_{project_id}_bot_{random_string}` i.e. `group_{group_id}_bot_{random_string}` users, with a `PersonalAccessToken`. diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md index 99737e71b22..e537b4c4c76 100644 --- a/doc/development/kubernetes.md +++ b/doc/development/kubernetes.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/lfs.md b/doc/development/lfs.md index 8f6d54e08e3..bb327bf5d04 100644 --- a/doc/development/lfs.md +++ b/doc/development/lfs.md @@ -3,12 +3,144 @@ stage: Create group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- - # Git LFS development guidelines This page contains developer-centric information for GitLab team members. For the user documentation, see [Git Large File Storage](../topics/git/lfs/index.md). +## Controllers and Services + +### Repositories::GitHttpClientController + +The methods for authentication defined here are inherited by all the other LFS controllers. + +### Repositories::LfsApiController + +#### `#batch` + +After authentication the `batch` action is the first action called by the Git LFS +client during downloads and uploads (such as pull, push, and clone). + +### Repositories::LfsStorageController + +#### `#upload_authorize` + +Provides payload to Workhorse including a path for Workhorse to save the file to. Could be remote object storage. + +#### `#upload_finalize` + +Handles requests from Workhorse that contain information on a file that workhorse already uploaded (see [this middleware](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/middleware/multipart.rb)) so that `gitlab` can either: + +- Create an `LfsObject`. +- Connect an existing `LfsObject` to a project with an `LfsObjectsProject`. + +### LfsObject and LfsObjectsProject + +- Only one `LfsObject` is created for a file with a given `oid` (a SHA256 checksum of the file) and file size. +- `LfsObjectsProject` associate `LfsObject`s with `Project`s. They determine if a file can be accessed through a project. +- These objects are also used for calculating the amount of LFS storage a given project is using. + For more information, see + [`ProjectStatistics#update_lfs_objects_size`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/project_statistics.rb#L82-84). + +### Repositories::LfsLocksApiController + +Handles the lock API for LFS. Delegates mostly to corresponding services: + +- `Lfs::LockFileService` +- `Lfs::UnlockFileService` +- `Lfs::LocksFinderService` + +These services create and delete `LfsFileLock`. + +#### `#verify` + +- This endpoint responds with a payload that allows a client to check if there are any files being pushed that have locks that belong to another user. +- A client-side `lfs.locksverify` configuration can be set so that the client aborts the push if locks exist that belong to another user. +- The existence of locks belonging to other users is also [validated on the server side](https://gitlab.com/gitlab-org/gitlab/-/blob/65f0c6e59121b62c9b0f89b810ef5186969bb4d2/lib/gitlab/checks/diff_check.rb#L69). + +## Example authentication + +```mermaid +sequenceDiagram +autonumber + alt Over HTTPS + Git client-->>Git client: user-supplied credentials + else Over SSH + Git client->>gitlab-shell: git-lfs-authenticate + activate gitlab-shell + activate GitLab Rails + gitlab-shell->>GitLab Rails: POST /api/v4/internal/lfs_authenticate + GitLab Rails-->>gitlab-shell: token with expiry + deactivate gitlab-shell + deactivate GitLab Rails + end +``` + +1. Clients can be configured to store credentials in a few different ways. + See the [Git LFS documentation on authentication](https://github.com/git-lfs/git-lfs/blob/bea0287cdd3acbc0aa9cdf67ae09b6843d3ffcf0/docs/api/authentication.md#git-credentials). +1. Running `gitlab-lfs-authenticate` on `gitlab-shell`. See the [Git LFS documentation concerning `gitlab-lfs-authenticate`](https://github.com/git-lfs/git-lfs/blob/bea0287cdd3acbc0aa9cdf67ae09b6843d3ffcf0/docs/api/server-discovery.md#ssh). +1. `gitlab-shell`makes a request to the GitLab API. +1. [Responding to shell with token](https://gitlab.com/gitlab-org/gitlab/-/blob/7a2f7a31a88b6085ea89b8ba188a4d92d5fada91/lib/api/internal/base.rb#L168) which is used in subsequent requests. See [Git LFS documentation concerning authentication](https://github.com/git-lfs/git-lfs/blob/bea0287cdd3acbc0aa9cdf67ae09b6843d3ffcf0/docs/api/authentication.md). + +## Example clone + +```mermaid +sequenceDiagram + Note right of Git client: Typical Git clone things happen first + Note right of Git client: Authentication for LFS comes next + activate GitLab Rails + autonumber + Git client->>GitLab Rails: POST project/namespace/info/lfs/objects/batch + GitLab Rails-->>Git client: payload with objects + deactivate GitLab Rails + loop each object in payload + Git client->>GitLab Rails: GET project/namespace/gitlab-lfs/objects/:oid/ (<- This URL is from the payload) + GitLab Rails->>Workhorse: SendfileUpload + Workhorse-->> Git client: Binary data + end +``` + +1. Git LFS requests the ability to download files with authorization header from authorization. +1. `gitlab` responds with the list of objects and where to find them. See + [LfsApiController#batch](https://gitlab.com/gitlab-org/gitlab/-/blob/7a2f7a31a88b6085ea89b8ba188a4d92d5fada91/app/controllers/repositories/lfs_api_controller.rb#L25). +1. Git LFS makes a request for each file for the `href` in the previous response. See + [how downloads are handled with the basic transfer mode](https://github.com/git-lfs/git-lfs/blob/bea0287cdd3acbc0aa9cdf67ae09b6843d3ffcf0/docs/api/basic-transfers.md#downloads). +1. `gitlab` redirects to the remote URL if remote object storage is enabled. See + [SendFileUpload](https://gitlab.com/gitlab-org/gitlab/-/blob/7a2f7a31a88b6085ea89b8ba188a4d92d5fada91/app/controllers/concerns/send_file_upload.rb#L4). + +## Example push + +```mermaid +sequenceDiagram + Note right of Git client: Typical Git push things happen first. + Note right of Git client: Suthentication for LFS comes next. + autonumber + activate GitLab Rails + Git client ->> GitLab Rails: POST project/namespace/info/lfs/objects/batch + GitLab Rails-->>Git client: payload with objects + deactivate GitLab Rails + loop each object in payload + Git client->>Workhorse: PUT project/namespace/gitlab-lfs/objects/:oid/:size (URL is from payload) + Workhorse->>GitLab Rails: PUT project/namespace/gitlab-lfs/objects/:oid/:size/authorize + GitLab Rails-->>Workhorse: response with where path to upload + Workhorse->>Workhorse: Upload + Workhorse->>GitLab Rails: PUT project/namespace/gitlab-lfs/objects/:oid/:size/finalize + end +``` + +1. Git LFS requests the ability to upload files. +1. `gitlab` responds with the list of objects and uploads to find them. See + [LfsApiController#batch](https://gitlab.com/gitlab-org/gitlab/-/blob/7a2f7a31a88b6085ea89b8ba188a4d92d5fada91/app/controllers/repositories/lfs_api_controller.rb#L27). +1. Git LFS makes a request for each file for the `href` in the previous response. See + [how uploads are handled with the basic transfer mode](https://github.com/git-lfs/git-lfs/blob/bea0287cdd3acbc0aa9cdf67ae09b6843d3ffcf0/docs/api/basic-transfers.md#uploads). +1. `gitlab` responds with a payload including a path for Workhorse to save the file to. + Could be remote object storage. See + [LfsStorageController#upload_authorize](https://gitlab.com/gitlab-org/gitlab/-/blob/96250de93a410e278ef659a3d38b056f12024636/app/controllers/repositories/lfs_storage_controller.rb#L42). +1. Workhorse does the work of saving the file. +1. Workhorse makes a request to `gitlab` with information on the uploaded file so + that `gitlab` can create an `LfsObject`. See + [LfsStorageController#upload_finalize](https://gitlab.com/gitlab-org/gitlab/-/blob/96250de93a410e278ef659a3d38b056f12024636/app/controllers/repositories/lfs_storage_controller.rb#L51). + ## Deep Dive In April 2019, Francisco Javier López hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/-/issues/1`) diff --git a/doc/development/logging.md b/doc/development/logging.md index 6840445a0a5..084bad70b90 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -202,7 +202,7 @@ suffix and `duration` within its name (for example, `view_duration_s`). ## Multi-destination Logging -GitLab is transitioning from unstructured/plaintext logs to structured/JSON logs. During this transition period some logs are recorded in multiple formats through multi-destination logging. +GitLab transitioned from structured to JSON logs. However, through multi-destination logging, the logs can be recorded in multiple formats. ### How to use multi-destination logging diff --git a/doc/development/merge_request_concepts/diffs/frontend.md b/doc/development/merge_request_concepts/diffs/frontend.md index 6bd6d80af94..ff163050e1f 100644 --- a/doc/development/merge_request_concepts/diffs/frontend.md +++ b/doc/development/merge_request_concepts/diffs/frontend.md @@ -84,40 +84,9 @@ The most important part of the metadata response is the diff file names. This da app to render the file browser inside of the diffs app, without waiting for all batch diffs requests to complete. -When the metadata response is received, the diff file data gets sent to a web worker. The web worker -exists to allow for this data, which for larger merge requests could be huge, to be processed off -the main thread. Processing this data involves getting the data into the correct structure +When the metadata response is received, the diff file data is processed into the correct structure that the frontend requires to render the file browser in either tree view or list view. -```mermaid -graph TD - A[fetchDiffFilesMeta] - B[Create web worker] - C[Fetch data] - D[SET_DIFF_METADATA] - E[Post worker data] - F[Worker message event listener] - K[SET_TREE_DATA] - - G[TreeWorker] - H[onMessage] - I[generateTreeList] - J[postMessage sortTree] - - A -->B - E -->F - B -->C - C -->D - D -->E - - G -->H - H -->I - I -->J - J -->F - - F -->K -``` - The structure for this file object is: ```javascript @@ -128,6 +97,11 @@ The structure for this file object is: "type": "", "tree": [], "changed": true, + "diffLoaded": false, + "filePaths": { + "old": file.old_path, + "new": file.new_path + }, "tempFile": false, "deleted": false, "fileHash": "", diff --git a/doc/development/merge_request_concepts/index.md b/doc/development/merge_request_concepts/index.md index 8e9b586d5b0..8bec5e0c0b0 100644 --- a/doc/development/merge_request_concepts/index.md +++ b/doc/development/merge_request_concepts/index.md @@ -61,7 +61,7 @@ Additionally, approval settings provide configuration options to define how thos Examples of approval rules and settings include: - [Merge request approval rules](../../user/project/merge_requests/approvals/rules.md) -- [Code owner approvals](../../user/project/code_owners.md) +- [Code owner approvals](../../user/project/codeowners/index.md) - [Security approvals](../../user/application_security/index.md#security-approvals-in-merge-requests) - [Prevent editing approval rules](../../user/project/merge_requests/approvals/settings.md#prevent-editing-approval-rules-in-merge-requests) - [Remove all approvals when commits are added](../../user/project/merge_requests/approvals/settings.md#remove-all-approvals-when-commits-are-added-to-the-source-branch) diff --git a/doc/development/merge_request_concepts/performance.md b/doc/development/merge_request_concepts/performance.md index 3b2a097ea2d..174307fc6a7 100644 --- a/doc/development/merge_request_concepts/performance.md +++ b/doc/development/merge_request_concepts/performance.md @@ -158,7 +158,7 @@ query. This in turn makes it much harder for this code to overload a database. ## Use read replicas when possible -In a DB cluster we have many read replicas and one primary. A classic use of scaling the DB is to have read-only actions be performed by the replicas. We use [load balancing](../../administration/postgresql/database_load_balancing.md) to distribute this load. This allows for the replicas to grow as the pressure on the DB grows. +In a DB cluster we have many read replicas and one primary. A classic use of scaling the DB is to have read-only actions be performed by the replicas. We use [load balancing](../database/load_balancing.md) to distribute this load. This allows for the replicas to grow as the pressure on the DB grows. By default, queries use read-only replicas, but due to [primary sticking](../../administration/postgresql/database_load_balancing.md#primary-sticking), GitLab uses the diff --git a/doc/development/merge_request_diffs.md b/doc/development/merge_request_diffs.md deleted file mode 100644 index 9ec7e6cae8b..00000000000 --- a/doc/development/merge_request_diffs.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'merge_request_concepts/diffs/development.md' -remove_date: '2023-04-10' ---- - -This document was moved to [another location](merge_request_concepts/diffs/development.md). - -<!-- This redirect file can be deleted after <2023-04-10>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md deleted file mode 100644 index 1af81a8af9f..00000000000 --- a/doc/development/merge_request_performance_guidelines.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'merge_request_concepts/performance.md' -remove_date: '2023-04-10' ---- - -This document was moved to [another location](merge_request_concepts/performance.md). - -<!-- This redirect file can be deleted after <2023-04-10>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 9f2ad1f7769..47580f91af6 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -156,7 +156,7 @@ regenerate a clean `db/structure.sql` for the migrations you're adding. This script applies all migrations found in `db/migrate` or `db/post_migrate`, so if there are any migrations you don't want to commit to the schema, rename or remove them. If your branch is not -targeting `main` you can set the `TARGET` environment variable. +targeting the default Git branch, you can set the `TARGET` environment variable. ```shell # Regenerate schema against `main` @@ -644,89 +644,14 @@ for more details. ## Adding indexes -Before adding an index, consider if this one is necessary. There are situations in which an index -might not be required, like: - -- The table is small (less than `1,000` records) and it's not expected to exponentially grow in size. -- Any existing indexes filter out enough rows. -- The reduction in query timings after the index is added is not significant. - -Additionally, wide indexes are not required to match all filter criteria of queries, we just need -to cover enough columns so that the index lookup has a small enough selectivity. Please review our -[Adding Database indexes](database/adding_database_indexes.md) guide for more details. - -When adding an index to a non-empty table make sure to use the method -`add_concurrent_index` instead of the regular `add_index` method. -The `add_concurrent_index` method automatically creates concurrent indexes -when using PostgreSQL, removing the need for downtime. - -To use this method, you must disable single-transactions mode -by calling the method `disable_ddl_transaction!` in the body of your migration -class like so: - -```ruby -class MyMigration < Gitlab::Database::Migration[2.1] - disable_ddl_transaction! - - INDEX_NAME = 'index_name' - - def up - add_concurrent_index :table, :column, name: INDEX_NAME - end - - def down - remove_concurrent_index :table, :column, name: INDEX_NAME - end -end -``` - -You must explicitly name indexes that are created with more complex -definitions beyond table name, column names, and uniqueness constraint. -Consult the [Adding Database Indexes](database/adding_database_indexes.md#requirements-for-naming-indexes) -guide for more details. - -If you need to add a unique index, please keep in mind there is the possibility -of existing duplicates being present in the database. This means that should -always _first_ add a migration that removes any duplicates, before adding the -unique index. - -For a small table (such as an empty one or one with less than `1,000` records), -it is recommended to use `add_index` in a single-transaction migration, combining it with other -operations that don't require `disable_ddl_transaction!`. +Before adding an index, consider if one is necessary. The [Adding Database indexes](database/adding_database_indexes.md) guide contains more details to help you decide if an index is necessary and provides best practices for adding indexes. ## Testing for existence of indexes -If a migration requires conditional logic based on the absence or -presence of an index, you must test for existence of that index using -its name. This helps avoids problems with how Rails compares index definitions, -which can lead to unexpected results. For more details, review the -[Adding Database Indexes](database/adding_database_indexes.md#why-explicit-names-are-required) -guide. - -The easiest way to test for existence of an index by name is to use the -`index_name_exists?` method, but the `index_exists?` method can also -be used with a name option. For example: - -```ruby -class MyMigration < Gitlab::Database::Migration[2.1] - INDEX_NAME = 'index_name' - - def up - # an index must be conditionally created due to schema inconsistency - unless index_exists?(:table_name, :column_name, name: INDEX_NAME) - add_index :table_name, :column_name, name: INDEX_NAME - end - end - - def down - # no op - end -end -``` +If a migration requires conditional logic based on the absence or presence of an index, you must test for existence of that index using its name. This helps avoids problems with how Rails compares index definitions, which can lead to unexpected results. -Keep in mind that concurrent index helpers like `add_concurrent_index`, -`remove_concurrent_index`, and `remove_concurrent_index_by_name` already -perform existence checks internally. +For more details, review the [Adding Database Indexes](database/adding_database_indexes.md#testing-for-existence-of-indexes) +guide. ## Adding foreign-key constraints diff --git a/doc/development/navigation_sidebar.md b/doc/development/navigation_sidebar.md index 495f30a796c..90b87e9518d 100644 --- a/doc/development/navigation_sidebar.md +++ b/doc/development/navigation_sidebar.md @@ -20,6 +20,10 @@ To enable the new navigation sidebar: - Enable the `super_sidebar_nav` feature flag. - Select your avatar, then turn on the **New navigation** toggle. +## Adding items to the sidebar + +Before adding an item to the sidebar, ensure you follow [this process](https://about.gitlab.com/handbook/product/ux/navigation/#how-to-propose-a-change-that-impacts-navigation). + ## Adding page-specific Vue content Pages can render arbitrary content into the sidebar using the `SidebarPortal` @@ -36,3 +40,16 @@ subclassing `::Sidebars::Panel`. NOTE: Do not use the `SidebarPortalTarget` component. It is internal to the sidebar. + +## Snowplow Tracking + +All clicks on the nav items should be automatically tracked in Snowplow, but may require additional input. +We use `data-tracking` attributes on all the elements in the nav to send the data up to Snowplow. +You can test that they're working by [setting up snowplow on your GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/snowplow_micro.md). + +| Field | Data attribute | Notes | Example | +| -- | -- | -- | -- | +| Category | `data-tracking-category` | The page that the user was on when the item was clicked. | `groups:show` | +| Action | `data-tracking-action` | The action taken. In most cases this is `click_link` or `click_menu_item` | `click_link` | +| Label | `data-tracking-label` | A descriptor for what was clicked on. This is inferred by the ID of the item in most cases, but falls back to `item_without_id`. This is one to look out for. | `group_issue_list` | +| Property | `data-tracking-property` | This describes where in the nav the link was clicked. If it's in the main nav panel, then it needs to describe which panel. | `nav_panel_group` | diff --git a/doc/development/organization/index.md b/doc/development/organization/index.md index 6a643603170..3a23e50caf9 100644 --- a/doc/development/organization/index.md +++ b/doc/development/organization/index.md @@ -1,5 +1,4 @@ --- -comments: false type: index, dev stage: Data Stores group: Tenant Scale @@ -39,87 +38,14 @@ A solution for this problem is to consolidate groups and projects into a single entity, `namespace`. The work on this solution is split into several phases and is tracked in [epic 6473](https://gitlab.com/groups/gitlab-org/-/epics/6473). -### Phase 1 - -- [Phase 1 epic](https://gitlab.com/groups/gitlab-org/-/epics/6697). -- **Goals**: - 1. Ensure every project receives a corresponding record in the `namespaces` - table with `type='Project'`. - 1. For user namespaces, the type changes from `NULL` to `User`. - -We should make sure that projects, and the project namespace, are equivalent: - -- **Create project:** use Rails callbacks to ensure a new project namespace is - created for each project. Project namespace records should contain `created_at` and - `updated_at` attributes equal to the project's `created_at`/`updated_at` attributes. -- **Update project:** use the `after_save` callback in Rails to ensure some - attributes are kept in sync between project and project namespaces. - Read [`project#after_save`](https://gitlab.com/gitlab-org/gitlab/blob/6d26634e864d7b748dda0e283eb2477362263bc3/app/models/project.rb#L101-L101) - for more information. -- **Delete project:** use FKs cascade delete or Rails callbacks to ensure when a `Project` - or its `ProjectNamespace` is removed, its corresponding `ProjectNamespace` or `Project` - is also removed. -- **Transfer project to a different group:** make sure that when a project is transferred, - its corresponding project namespace is transferred to the same group. -- **Transfer group:** make sure when transferring a group that all of its sub-projects, - either direct or through descendant groups, have their corresponding project - namespaces transferred correctly as well. -- **Export or import project** - - **Export project** continues to export only the project, and not its project namespace, - in this phase. The project namespace does not contain any specific information - to export at this point. Eventually we want the project namespace to be exported as well. - - **Import project** creates a new project, so the project namespace is created through - Rails `after_save` callback on the project model. -- **Export or import group:** when importing or exporting a `Group`, projects are not - included in the operation. If that feature is changed to include `Project` when its group is - imported or exported, the logic must include their corresponding project namespaces - in the import or export. - -After ensuring these points, run a database migration to create a `ProjectNamespace` -record for every `Project`. Project namespace records created during the migration -should have `created_at` and `updated_at` attributes set to the migration runtime. -The project namespaces' `created_at` and `updated_at` attributes would not match -their corresponding project's `created_at` and `updated_at` attributes. We want -the different dates to help audit any of the created project namespaces, in case we need it. -After this work completes, we must migrate data as described in -[Backfill `ProjectNamespace` for every Project](https://gitlab.com/gitlab-org/gitlab/-/issues/337100). - -### Phase 2 - -- [Phase 2 epic](https://gitlab.com/groups/gitlab-org/-/epics/6768). -- **Goal**: Link `ProjectNamespace` to other entities on the database level. - -In this phase: - -- Communicate the changes company-wide at the engineering level. We want to make - engineers aware of the upcoming changes, even though teams are not expected to - collaborate actively until phase 3. -- Raise awareness to avoid regressions, and conflicting or duplicate work that - can be dealt with before phase 3. - -### Phase 3 - -- [Phase 3 epic](https://gitlab.com/groups/gitlab-org/-/epics/6585). -- **Goal**: Achieve feature parity between the namespace types. -Problems to solve as part of this phase: - -- Routes handling through `ProjectNamespace` rather than `Project`. -- Memberships handling. -- Policies handling. -- Import and export. -- Other interactions between project namespace and project models. - -Phase 3 is when the active migration of features from `Project` to `ProjectNamespace`, -or directly to `Namespace`, happens. - -### How to plan features that interact with Group and ProjectNamespace +## How to plan features that interact with Group and ProjectNamespace As of now, every Project in the system has a record in the `namespaces` table. This makes it possible to use common interface to create features that are shared between Groups and Projects. Shared behavior can be added using a concerns mechanism. Because the `Namespace` model is responsible for `UserNamespace` methods as well, it is discouraged to use the `Namespace` model for shared behavior for Projects and Groups. -#### Resource-based features +### Resource-based features To migrate resource-based features, existing functionality will need to be supported. This can be achieved in two Phases. @@ -141,7 +67,7 @@ To migrate resource-based features, existing functionality will need to be suppo Introducing new functionality is very much dependent on every single team and feature. -#### Settings-related features +### Settings-related features Right now, cascading settings are available for `NamespaceSettings`. By creating `ProjectNamespace`, we can use this framework to make sure that some settings are applicable on the project level as well. @@ -150,7 +76,7 @@ When working on settings, we need to make sure that: - They are not used in `join` queries or modify those queries. - Updating settings is taken into consideration. -- If we want to move from project to project namespace, we follow a similar database process to the one described in [Phase 1](#phase-1). +- If we want to move from project to project namespace, we follow a similar database process to the one described in Phase 1. ## Related topics diff --git a/doc/development/pages/index.md b/doc/development/pages/index.md index e71d7df642c..4769dbf427d 100644 --- a/doc/development/pages/index.md +++ b/doc/development/pages/index.md @@ -1,7 +1,7 @@ --- type: reference, dev -stage: Create -group: Editor +stage: Plan +group: Knowledge info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: "GitLab's development guidelines for GitLab Pages" --- @@ -255,7 +255,7 @@ incidents and downtime. To add a new feature flag to GitLab Pages: 1. Create the feature flag in [`internal/feature/feature.go`](https://gitlab.com/gitlab-org/gitlab-pages/-/blob/master/internal/feature/feature.go), which must be **off** by default. -1. Create an issue to track the feature flag using the `Feature Flag` template. +1. Create an issue to track the feature flag using the `Feature flag` template. 1. Add the `~"feature flag"` label to any merge requests that handle feature flags. For GitLab Pages, the feature flags are controlled by environment variables at a global level. diff --git a/doc/development/pipelines/index.md b/doc/development/pipelines/index.md index 51cdd12a8c1..44fd20df40d 100644 --- a/doc/development/pipelines/index.md +++ b/doc/development/pipelines/index.md @@ -47,22 +47,49 @@ In summary: - RSpec tests are dependent on the backend code. - Jest tests are dependent on both frontend and backend code, the latter through the frontend fixtures. +### Predictive Tests Dashboards + +- <https://app.periscopedata.com/app/gitlab/1116767/Test-Intelligence-Accuracy> +- <https://app.periscopedata.com/app/gitlab/899368/EP---Predictive-testing-analysis> + +### The `detect-tests` CI job + +Most CI/CD pipelines for `gitlab-org/gitlab` will run a [`detect-tests` CI job](https://gitlab.com/gitlab-org/gitlab/-/blob/0c6058def8f182b4a2410db5d08a9550b951b2d8/.gitlab/ci/setup.gitlab-ci.yml#L101-146) in the `prepare` stage to detect which backend/frontend tests should be run based on the files that changed in the given MR. + +The `detect-tests` job will create many files that will contain the backend/frontend tests that should be run. Those files will be read in subsequent jobs in the pipeline, and only those tests will be executed. + ### RSpec predictive jobs #### Determining predictive RSpec test files in a merge request -To identify the RSpec tests that are likely to fail in a merge request, we use the [`test_file_finder` gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder), with two strategies: +To identify the RSpec tests that are likely to fail in a merge request, we use *static mappings* and *dynamic mappings*. + +##### Static mappings -- dynamic mapping from test coverage tracing (generated via the [`Crystalball` gem](https://github.com/toptal/crystalball)) - ([see where it's used](https://gitlab.com/gitlab-org/gitlab/-/blob/47d507c93779675d73a05002e2ec9c3c467cd698/tooling/bin/find_tests#L15)) -- static mapping maintained in the [`tests.yml` file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/tests.yml) for special cases that cannot - be mapped via coverage tracing ([see where it's used](https://gitlab.com/gitlab-org/gitlab/-/blob/47d507c93779675d73a05002e2ec9c3c467cd698/tooling/bin/find_tests#L12)) +We use the [`test_file_finder` gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder), with a static mapping maintained in the [`tests.yml` file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/tests.yml) for special cases that cannot + be mapped via coverage tracing ([see where it's used](https://gitlab.com/gitlab-org/gitlab/-/blob/5ab06422826c0d69c615655982a6f969a7f3c6ea/tooling/lib/tooling/find_tests.rb#L17)). The test mappings contain a map of each source files to a list of test files which is dependent of the source file. -In the `detect-tests` job, we use this mapping to identify the predictive tests needed for the current merge request. +##### Dynamic mappings + +First, we use the [`test_file_finder` gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder), with a dynamic mapping strategy from test coverage tracing (generated via the [`Crystalball` gem](https://github.com/toptal/crystalball)) + ([see where it's used](https://gitlab.com/gitlab-org/gitlab/-/blob/master/tooling/lib/tooling/find_tests.rb#L20)). -Later on in [the `rspec fail-fast` job](#fail-fast-job-in-merge-request-pipelines), we run the predictive tests for the current merge request. +In addition to `test_file_finder`, we have added several advanced mappings to detect even more tests to run: + +- [`FindChanges`](https://gitlab.com/gitlab-org/gitlab/-/blob/28943cbd8b6d7e9a350d00e5ea5bb52123ee14a4/tooling/lib/tooling/find_changes.rb) ([!74003](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74003)) + - Automatically detect Jest tests to run upon backend changes (via frontend fixtures) +- [`PartialToViewsMappings`](https://gitlab.com/gitlab-org/gitlab/-/blob/28943cbd8b6d7e9a350d00e5ea5bb52123ee14a4/tooling/lib/tooling/mappings/partial_to_views_mappings.rb) ([#395016](https://gitlab.com/gitlab-org/gitlab/-/issues/395016)) + - Run view specs when Rails partials included in those views are changed in an MR +- [`JsToSystemSpecsMappings`](https://gitlab.com/gitlab-org/gitlab/-/blob/28943cbd8b6d7e9a350d00e5ea5bb52123ee14a4/tooling/lib/tooling/mappings/js_to_system_specs_mappings.rb) ([#386754](https://gitlab.com/gitlab-org/gitlab/-/issues/386754)) + - Run certain system specs if a JavaScript file was changed in an MR +- [`GraphqlBaseTypeMappings`](https://gitlab.com/gitlab-org/gitlab/-/blob/28943cbd8b6d7e9a350d00e5ea5bb52123ee14a4/tooling/lib/tooling/mappings/graphql_base_type_mappings.rb) ([#386756](https://gitlab.com/gitlab-org/gitlab/-/issues/386756)) + - If a GraphQL type class changed, we should try to identify the other GraphQL types that potentially include this type, and run their specs. +- [`ViewToSystemSpecsMappings`](https://gitlab.com/gitlab-org/gitlab/-/blob/28943cbd8b6d7e9a350d00e5ea5bb52123ee14a4/tooling/lib/tooling/mappings/view_to_system_specs_mappings.rb) ([#395017](https://gitlab.com/gitlab-org/gitlab/-/issues/395017)) + - When a view gets changed, we try to find feature specs that would test that area of the code. +- [`ViewToJsMappings`](https://gitlab.com/gitlab-org/gitlab/-/blob/8d7dfb7c043adf931128088b9ffab3b4a39af6f5/tooling/lib/tooling/mappings/view_to_js_mappings.rb) ([#386719](https://gitlab.com/gitlab-org/gitlab/-/issues/386719)) + - If a JS file is changed, we should try to identify the system specs that are covering this JS component. #### Exceptional cases @@ -74,6 +101,10 @@ In addition, there are a few circumstances where we would always run the full RS - when the merge request is created in a security mirror - when any CI configuration file is changed (for example, `.gitlab-ci.yml` or `.gitlab/ci/**/*`) +#### Have you encountered a problem with backend predictive tests? + +If so, please have a look at [the Engineering Productivity RUNBOOK on predictive tests](https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/blob/main/runbooks/predictive-tests.md) for instructions on how to act upon predictive tests issues. Additionally, if you identified any test selection gaps, please let `@gl-quality/eng-prod` know so that we can take the necessary steps to optimize test selections. + ### Jest predictive jobs #### Determining predictive Jest test files in a merge request @@ -95,6 +126,10 @@ In addition, there are a few circumstances where we would always run the full Je The `rules` definitions for full Jest tests are defined at `.frontend:rules:jest` in [`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/42321b18b946c64d2f6f788c38844499a5ae9141/.gitlab/ci/rules.gitlab-ci.yml#L938-955). +#### Have you encountered a problem with frontend predictive tests? + +If so, please have a look at [the Engineering Productivity RUNBOOK on predictive tests](https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/blob/main/runbooks/predictive-tests.md) for instructions on how to act upon predictive tests issues. + ### Fork pipelines We run only the predictive RSpec & Jest jobs for fork pipelines, unless the `pipeline:run-all-rspec` @@ -104,8 +139,7 @@ See the [experiment issue](https://gitlab.com/gitlab-org/quality/quality-enginee ## Fail-fast job in merge request pipelines -To provide faster feedback when a merge request breaks existing tests, we are experimenting with a -fail-fast mechanism. +To provide faster feedback when a merge request breaks existing tests, we implemented a fail-fast mechanism. An `rspec fail-fast` job is added in parallel to all other `rspec` jobs in a merge request pipeline. This job runs the tests that are directly related to the changes @@ -150,8 +184,8 @@ This number can be overridden by setting a CI/CD variable named `RSPEC_FAIL_FAST ## Re-run previously failed tests in merge request pipelines -In order to reduce the feedback time after resolving failed tests for a merge request, the `rspec rspec-pg12-rerun-previous-failed-tests` -and `rspec rspec-ee-pg12-rerun-previous-failed-tests` jobs run the failed tests from the previous MR pipeline. +In order to reduce the feedback time after resolving failed tests for a merge request, the `rspec rspec-pg13-rerun-previous-failed-tests` +and `rspec rspec-ee-pg13-rerun-previous-failed-tests` jobs run the failed tests from the previous MR pipeline. This was introduced on August 25th 2021, with <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69053>. @@ -159,7 +193,7 @@ This was introduced on August 25th 2021, with <https://gitlab.com/gitlab-org/git 1. The `detect-previous-failed-tests` job (`prepare` stage) detects the test files associated with failed RSpec jobs from the previous MR pipeline. -1. The `rspec rspec-pg12-rerun-previous-failed-tests` and `rspec rspec-ee-pg12-rerun-previous-failed-tests` jobs +1. The `rspec rspec-pg13-rerun-previous-failed-tests` and `rspec rspec-ee-pg13-rerun-previous-failed-tests` jobs will run the test files gathered by the `detect-previous-failed-tests` job. ```mermaid @@ -169,13 +203,27 @@ graph LR end subgraph "test stage"; - B["rspec rspec-pg12-rerun-previous-failed-tests"]; - C["rspec rspec-ee-pg12-rerun-previous-failed-tests"]; + B["rspec rspec-pg13-rerun-previous-failed-tests"]; + C["rspec rspec-ee-pg13-rerun-previous-failed-tests"]; end A --"artifact: list of test files"--> B & C ``` +## Merge Trains + +### Why do we need to have a “stable” master branch to enable merge trains? + +If the master branch is unstable (i.e. CI/CD pipelines for the master branch are failing frequently), all of the merge requests pipelines that were added AFTER a faulty merge request pipeline would have to be **cancelled** and **added back to the train**, which would create a lot of delays if the merge train is long. + +### How stable does the master branch have to be for us to enable merge trains? + +We don't have a specific number, but we need to have better numbers for flaky tests failures and infrastructure failures (see the [Master Broken Incidents RCA Dashboard](https://app.periscopedata.com/app/gitlab/1082465/Master-Broken-Incidents-Root-Cause-Analysis)). + +### Could we gradually move to merge trains in our CI/CD configuration? + +There was a proposal from a contributor, but the approach is not without some downsides: [see the original proposal and discussion](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/195#note_1117151994). + ## Faster feedback for some merge requests ### Broken Master Fixes @@ -465,10 +513,9 @@ Exceptions to this general guideline should be motivated and documented. ### Ruby versions testing -We're running Ruby 3.0 for the merge requests and the default branch. However, -we're still running Ruby 2.7 for GitLab.com and there are older versions that -we need to maintain, so we also run our test suite against Ruby 2.7 on a -dedicated 2-hourly scheduled pipelines. +We're running Ruby 3.0 on GitLab.com, as well as for merge requests and the default branch. +However, there are older versions for which we need to support Ruby 2.7, so we also run our +test suite against Ruby 2.7 on a dedicated 2-hourly scheduled pipelines. For merge requests, you can add the `pipeline:run-in-ruby2` label to switch the Ruby version used for running the whole test suite to 2.7. When you do @@ -483,20 +530,22 @@ This should let us: ### PostgreSQL versions testing -Our test suite runs against PG12 as GitLab.com runs on PG12 and -[Omnibus defaults to PG12 for new installs and upgrades](../../administration/package_information/postgresql_versions.md). +Our test suite runs against PG13 as GitLab.com runs on PG13 and +[Omnibus defaults to PG13 for new installs and upgrades](../../administration/package_information/postgresql_versions.md). We do run our test suite against PG13 on nightly scheduled pipelines. +We also run our test suite against PG13 upon specific database library changes in MRs and `main` pipelines (with the `rspec db-library-code pg13` job). + #### Current versions testing -| Where? | PostgreSQL version | Ruby version | -|------------------------------------------------------------------------------------------------|--------------------------|-----------------------| -| Merge requests | 12 (default version) | 3.0 (default version) | -| `master` branch commits | 12 (default version) | 3.0 (default version) | -| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour) | 12 (default version) | 3.0 (default version) | -| `maintenance` scheduled pipelines for the `ruby2` branch (every odd-numbered hour), see below. | 12 (default version) | 2.7 | -| `nightly` scheduled pipelines for the `master` branch | 12 (default version), 13 | 3.0 (default version) | +| Where? | PostgreSQL version | Ruby version | +|------------------------------------------------------------------------------------------------|-------------------------------------------------|-----------------------| +| Merge requests | 13 (default version), 12 for DB library changes | 3.0 (default version) | +| `master` branch commits | 13 (default version), 12 for DB library changes | 3.0 (default version) | +| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour) | 13 (default version), 12 for DB library changes | 3.0 (default version) | +| `maintenance` scheduled pipelines for the `ruby2` branch (every odd-numbered hour), see below. | 13 (default version), 12 for DB library changes | 2.7 | +| `nightly` scheduled pipelines for the `master` branch | 13 (default version), 12, 14 | 3.0 (default version) | There are 2 pipeline schedules used for testing Ruby 2.7. One is triggering a pipeline in `ruby2-sync` branch, which updates the `ruby2` branch with latest @@ -511,15 +560,6 @@ The `gitlab` job in the `ruby2-sync` branch uses a `gitlab-org/gitlab` project token with `write_repository` scope and `Maintainer` role with no expiration. The token is stored in the `RUBY2_SYNC_TOKEN` variable in `gitlab-org/gitlab`. -#### Long-term plan - -We follow the [PostgreSQL versions shipped with Omnibus GitLab](../../administration/package_information/postgresql_versions.md): - -| PostgreSQL version | 14.1 (July 2021) | 14.2 (August 2021) | 14.3 (September 2021) | 14.4 (October 2021) | 14.5 (November 2021) | 14.6 (December 2021) | -| -------------------| ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | -| PG12 | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | -| PG13 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | - ### Redis versions testing Our test suite runs against Redis 6 as GitLab.com runs on Redis 6 and @@ -541,6 +581,13 @@ By default, all tests run with [multiple databases](../database/multiple_databas We also run tests with a single database in nightly scheduled pipelines, and in merge requests that touch database-related files. +Single database tests run in two modes: + +1. **Single database with one connection**. Where GitLab connects to all the tables using one connection pool. +This runs through all the jobs that end with `-single-db` +1. **Single database with two connections**. Where GitLab connects to `gitlab_main`, `gitlab_ci` database tables +using different database connections. This runs through all the jobs that end with `-single-db-ci-connection`. + If you want to force tests to run with a single database, you can add the `pipeline:run-single-db` label to the merge request. ## Monitoring @@ -561,7 +608,7 @@ In general, pipelines for an MR fall into one of the following types (from short - [Documentation pipeline](#documentation-pipeline): For MRs that touch documentation. - [Backend pipeline](#backend-pipeline): For MRs that touch backend code. -- [Frontend pipeline](#frontend-pipeline): For MRs that touch frontend code. +- [Review app pipeline](#review-app-pipeline): For MRs that touch frontend code. - [End-to-end pipeline](#end-to-end-pipeline): For MRs that touch code in the `qa/` folder. A "pipeline type" is an abstract term that mostly describes the "critical path" (for example, the chain of jobs for which the sum @@ -598,10 +645,10 @@ graph LR graph RL; classDef criticalPath fill:#f66; - 1-3["compile-test-assets (6 minutes)"]; + 1-3["compile-test-assets (5.5 minutes)"]; class 1-3 criticalPath; click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" - 1-6["setup-test-env (4 minutes)"]; + 1-6["setup-test-env (3.6 minutes)"]; click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" 1-14["retrieve-tests-metadata"]; click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" @@ -613,19 +660,19 @@ graph RL; click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" 2_5-1 --> 1-3 & 1-6 & 1-14 & 1-15; - 3_2-1["rspec:coverage (5.35 minutes)"]; + 3_2-1["rspec:coverage (5 minutes)"]; class 3_2-1 criticalPath; click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0" 3_2-1 -.->|"(don't use needs<br/>because of limitations)"| 2_5-1; - 4_3-1["rspec:undercoverage (3.5 minutes)"]; + 4_3-1["rspec:undercoverage (1.3 minutes)"]; class 4_3-1 criticalPath; click 4_3-1 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=13446492&udv=1005715" 4_3-1 --> 3_2-1; ``` -### Frontend pipeline +### Review app pipeline [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/431913287). @@ -635,15 +682,15 @@ graph RL; 1-2["build-qa-image (2 minutes)"]; click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" - 1-5["compile-production-assets (16 minutes)"]; + 1-5["compile-production-assets (12 minutes)"]; class 1-5 criticalPath; click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" - 2_3-1["build-assets-image (1.3 minutes)"]; + 2_3-1["build-assets-image (1.1 minutes)"]; class 2_3-1 criticalPath; 2_3-1 --> 1-5 - 2_6-1["start-review-app-pipeline (49 minutes)"]; + 2_6-1["start-review-app-pipeline (52 minutes)"]; class 2_6-1 criticalPath; click 2_6-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" 2_6-1 --> 2_3-1 & 1-2; @@ -659,17 +706,17 @@ graph RL; 1-2["build-qa-image (2 minutes)"]; click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" - 1-5["compile-production-assets (16 minutes)"]; + 1-5["compile-production-assets (12 minutes)"]; class 1-5 criticalPath; click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" 1-15["detect-tests"]; click 1-15 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=10113603&udv=1005715" - 2_3-1["build-assets-image (1.3 minutes)"]; + 2_3-1["build-assets-image (1.1 minutes)"]; class 2_3-1 criticalPath; 2_3-1 --> 1-5 - 2_4-1["e2e:package-and-test (102 minutes)"]; + 2_4-1["e2e:package-and-test-ee (103 minutes)"]; class 2_4-1 criticalPath; click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0" 2_4-1 --> 1-2 & 2_3-1 & 1-15; diff --git a/doc/development/pipelines/internals.md b/doc/development/pipelines/internals.md index bd96f2f2872..678297eb3e5 100644 --- a/doc/development/pipelines/internals.md +++ b/doc/development/pipelines/internals.md @@ -136,10 +136,10 @@ that are scoped to a single [configuration keyword](../../ci/yaml/index.md#job-k | `.qa-cache` | Allows a job to use a default `cache` definition suitable for QA tasks. | | `.yarn-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that do a `yarn install`. | | `.assets-compile-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that compile assets. | -| `.use-pg12` | Allows a job to use the `postgres` 12 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | -| `.use-pg12-ee` | Same as `.use-pg12` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | | `.use-pg13` | Allows a job to use the `postgres` 13 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | | `.use-pg13-ee` | Same as `.use-pg13` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | +| `.use-pg14` | Allows a job to use the `postgres` 14 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | +| `.use-pg14-ee` | Same as `.use-pg14` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | | `.use-kaniko` | Allows a job to use the `kaniko` tool to build Docker images. | | `.as-if-foss` | Simulate the FOSS project by setting the `FOSS_ONLY='1'` CI/CD variable. | | `.use-docker-in-docker` | Allows a job to use Docker in Docker. | @@ -187,7 +187,6 @@ and included in `rules` definitions via [YAML anchors](../../ci/yaml/yaml_optimi | `if-dot-com-gitlab-org-and-security-merge-request` | Limit jobs creation to merge requests for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | | `if-dot-com-gitlab-org-and-security-tag` | Limit jobs creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | | `if-dot-com-ee-schedule` | Limits jobs to scheduled pipelines for the `gitlab-org/gitlab` project on GitLab.com. | | -| `if-security-pipeline-merge-result` | Matches if the pipeline is for a security merge request triggered by `@gitlab-release-tools-bot`. | | <!-- vale gitlab.Substitutions = YES --> diff --git a/doc/development/product_qualified_lead_guide/index.md b/doc/development/product_qualified_lead_guide/index.md index 41237dcdff4..fb8ec478840 100644 --- a/doc/development/product_qualified_lead_guide/index.md +++ b/doc/development/product_qualified_lead_guide/index.md @@ -117,7 +117,7 @@ expect(wrapper.findComponent(HandRaiseLeadButton).exists()).toBe(true); The flow of a PQL lead is as follows: 1. A user triggers a [`HandRaiseLeadButton` component](#embed-a-hand-raise-lead-form) on `gitlab.com`. -1. The `HandRaiseLeadButton` submits any information to the following API endpoint: `/-/trials/create_hand_raise_lead`. +1. The `HandRaiseLeadButton` submits any information to the following API endpoint: `/-/subscriptions/hand_raise_leads`. 1. That endpoint reposts the form to the CustomersDot `trials/create_hand_raise_lead` endpoint. 1. CustomersDot records the form data to the `leads` table and posts the form to [Workato](https://about.gitlab.com/handbook/marketing/marketing-operations/workato/). 1. Workato sends the form to Marketo. @@ -170,7 +170,7 @@ sequenceDiagram ```mermaid sequenceDiagram HandRaiseForm Vue Component->>TrialsController#create_hand_raise_lead: GitLab.com frontend sends [lead] to backend - TrialsController#create_hand_raise_lead->>CreateHandRaiseLeadService: [lead] + Subscriptions::HandRaiseLeadsController#create->>CreateHandRaiseLeadService: [lead] CreateHandRaiseLeadService->>SubscriptionPortalClient: [lead] SubscriptionPortalClient->>CustomersDot|TrialsController#create_hand_raise_lead: GitLab.com sends [lead] to CustomersDot ``` diff --git a/doc/development/project_templates.md b/doc/development/project_templates.md index 31537b21527..cc53ef77c62 100644 --- a/doc/development/project_templates.md +++ b/doc/development/project_templates.md @@ -8,110 +8,63 @@ info: "To determine the technical writer assigned to the Stage/Group associated ## Adding a new built-in project template -This page provides instructions about how to contribute a -[built-in project template](../user/project/index.md#create-a-project-from-a-built-in-template). - -To contribute a built-in project template, you must complete the following tasks: - -1. [Create a project template for GitLab review](#create-a-project-template-for-review) -1. [Add the template SVG icon to GitLab SVGs](#add-the-template-svg-icon-to-gitlab-svgs) -1. [Create a merge request with vendor details](#create-a-merge-request-with-vendor-details) - -You can contribute the following types of project templates: - -- Enterprise: For users with GitLab Premium and above. -- Non-enterprise: For users with GitLab Free and above. - -### Prerequisites - -To add or update an existing template, you must have the following tools -installed: - -- `wget` -- `tar` - -### Create a project template for review - -1. In your selected namespace, create a public project. -1. Add the project content you want to use in the template. Do not include unnecessary assets or dependencies. For an example, -[see this project](https://gitlab.com/gitlab-org/project-templates/dotnetcore). -1. When the project is ready for review, [create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues) with a link to your project. - In your issue, mention the Create:Source Code [Backend Engineering Manager and Product Manager](https://about.gitlab.com/handbook/product/categories/#source-code-group) - for the Templates feature. - -### Add the template SVG icon to GitLab SVGs - -If the project template has an SVG icon, you must add it to the -[GitLab SVGs project](https://gitlab.com/gitlab-org/gitlab-svgs/-/blob/main/README.md#adding-icons-or-illustrations) -before you can create a merge request with vendor details. - -### Create a merge request with vendor details - -Before GitLab can implement the project template, you must [create a merge request](../user/project/merge_requests/creating_merge_requests.md) in [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) that includes vendor details about the project. - -1. [Export the project](../user/project/settings/import_export.md#export-a-project-and-its-data) - and save the file as `<name>.tar.gz`, where `<name>` is the short name of the project. - Move this file to the root directory of `gitlab-org/gitlab`. -1. In `gitlab-org/gitlab`, create and check out a new branch. -1. Edit the following files to include the project template: - - For **non-Enterprise** project templates: - - In `lib/gitlab/project_template.rb`, add details about the template - in the `localized_templates_table` method. In the following example, - the short name of the project is `hugo`: - - ```ruby - ProjectTemplate.new('hugo', 'Pages/Hugo', _('Everything you need to create a GitLab Pages site using Hugo'), 'https://gitlab.com/pages/hugo', 'illustrations/logos/hugo.svg'), - ``` - - If the project doesn't have an SVG icon, exclude `, 'illustrations/logos/hugo.svg'`. - - - In `spec/support/helpers/project_template_test_helper.rb`, append the short name - of the template in the `all_templates` method. - - In `app/assets/javascripts/projects/default_project_templates.js`, - add details of the template. For example: - - ```javascript - hugo: { - text: s__('ProjectTemplates|Pages/Hugo'), - icon: '.template-option .icon-hugo', - }, - ``` - - If the project doesn't have an SVG icon, use `.icon-gitlab_logo` - instead. - - For **Enterprise** project templates: - - In `ee/lib/ee/gitlab/project_template.rb`, in the `localized_ee_templates_table` method, add details about the template. For example: - - ```ruby - ::Gitlab::ProjectTemplate.new('hipaa_audit_protocol', 'HIPAA Audit Protocol', _('A project containing issues for each audit inquiry in the HIPAA Audit Protocol published by the U.S. Department of Health & Human Services'), 'https://gitlab.com/gitlab-org/project-templates/hipaa-audit-protocol', 'illustrations/logos/asklepian.svg') - ``` - - - In `ee/spec/lib/gitlab/project_template_spec.rb`, add the short name - of the template in the `.all` test. - - In `ee/app/assets/javascripts/projects/default_project_templates.js`, - add the template details. For example: - - ```javascript - hipaa_audit_protocol: { - text: s__('ProjectTemplates|HIPAA Audit Protocol'), - icon: '.template-option .icon-hipaa_audit_protocol', - }, - ``` - -1. Run the following Rake task, where `<path>/<name>` is the - name you gave the template in `lib/gitlab/project_template.rb`: +If you'd like to contribute a new built-in project template to be distributed with GitLab, please do the following: + +1. Create a new public project with the project content you'd like to contribute in a namespace of your choosing. You can view a working example [here](https://gitlab.com/gitlab-org/project-templates/dotnetcore). + - Projects should be as simple as possible and free of any unnecessary assets or dependencies. +1. When the project is ready for review, please create a new issue in [GitLab](https://gitlab.com/gitlab-org/gitlab/issues) with a link to your project. + - In your issue, `@` mention the relevant Backend Engineering Manager and Product Manager for the [Create:Source Code group](https://about.gitlab.com/handbook/product/categories/#source-code-group). + +To make the project template available when creating a new project, the vendoring process will have to be completed: + +1. Create a working template ([example](https://gitlab.com/gitlab-org/project-templates/dotnetcore)) + - 2 types of built-in templates are available within GitLab: + - **Normal templates**: Available in GitLab Core, Starter and above (this is the most common type of built-in template). + - To contribute a normal template: + - Add details of the template in the `localized_templates_table` method in `gitlab/lib/gitlab/project_template.rb`, + - Add details of the template in `spec/lib/gitlab/project_template_spec.rb`, in the test for the `all` method, and + - Add details of the template in `gitlab/app/assets/javascripts/projects/default_project_templates.js`. + - See MR [!25318](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25318) for an example + - **Enterprise templates**: Introduced in GitLab 12.10, that are available only in GitLab Gold & Ultimate. + - To contribute an Enterprise template: + - Add details of the template in the `localized_ee_templates_table` method in `gitlab/ee/lib/ee/gitlab/project_template.rb`, + - Add details of the template in `gitlab/ee/spec/lib/gitlab/project_template_spec.rb`, in the `enterprise_templates` method, and + - Add details of the template in `gitlab/ee/app/assets/javascripts/projects/default_project_templates.js`. + - See MR [!28187](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28187) for an example. + +1. Run the following in the `gitlab` project, where `$name` is the name you gave the template in `gitlab/project_template.rb`: ```shell - bin/rake gitlab:update_project_templates\[<path>/<name>\] + bin/rake gitlab:update_project_templates[$name] ``` -1. Regenerate `gitlab.pot`: - - ```shell - bin/rake gettext:regenerate - ``` - -1. After you run the scripts, there is one new file in `vendor/project_templates/` and four changed files. Commit all changes and push your branch to update the merge request. For an example, see this [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25318). +1. Run the `bundle_repo` script. Make sure to pass the correct arguments, or the script may damage the folder structure. +1. Add exported project (`$name.tar.gz`) to `gitlab/vendor/project_templates` and remove the resulting build folders `tar-base` and `project`. +1. Run `bin/rake gettext:regenerate` in the `gitlab` project and commit new `.pot` file. +1. Add a changelog entry in the commit message (for example, `Changelog: added`). + For more information, see [Changelog entries](changelog.md). +1. Add an icon to [`gitlab-svgs`](https://gitlab.com/gitlab-org/gitlab-svgs), as shown in + [this example](https://gitlab.com/gitlab-org/gitlab-svgs/merge_requests/195). If a logo + is not available for the project, use the default 'Tanuki' logo instead. +1. Run `yarn run svgs` on `gitlab-svgs` project and commit result. +1. Forward changes in `gitlab-svgs` project to master. This involves: + - Merging your MR in `gitlab-svgs` + - [The bot](https://gitlab.com/gitlab-org/frontend/renovate-gitlab-bot/) + will pick the new release up and create an MR in `gitlab-org/gitlab`. +1. Once the bot-created MR created above is merged, you can rebase your template MR onto the updated `master` to pick up the new svgs. +1. Test everything is working. + +### Contributing an improvement to an existing template + +Existing templates are available in the [project-templates](https://gitlab.com/gitlab-org/project-templates) +group. + +To contribute a change, please open a merge request in the relevant project +and mention `@gitlab-org/manage/import/backend` when you are ready for a review. + +Then, if your merge request gets accepted, either open an issue on +`gitlab` to ask for it to get updated, or open a merge request updating +the vendored template using [these instructions](rake_tasks.md#update-project-templates). ### Test your built-in project with the GitLab Development Kit @@ -124,17 +77,6 @@ Complete the following steps to test the project template in your own GitLab Dev bin/rake gitlab:update_project_templates\[<path>/<name>\] ``` -## Contribute an improvement to an existing template - -To update an existing built-in project template, changes are usually made to the existing template, found in the [project-templates](https://gitlab.com/gitlab-org/project-templates) group. A merge request is made directly against the template and the Create:Source Code [Backend Engineering Manager and Product Manager](https://about.gitlab.com/handbook/product/categories/#source-code-group) pinged for review. - -Sometimes it is necessary to completely replace the template files. In this case the process would be: - -1. Create a merge request in the relevant project of the `project-templates` and `pages` group and mention `@gitlab-org/manage/import/backend` when you are ready for a review. -1. If your merge request is accepted, either: - - [Create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues) to ask for the template to get updated. - - [Create a merge request with vendor details](#create-a-merge-request-with-vendor-details) to update the template. - ## For GitLab team members Please ensure the merge request has been reviewed by the Security Counterpart before merging. diff --git a/doc/development/projections.md b/doc/development/projections.md index 7c727fc0901..caa54e7b912 100644 --- a/doc/development/projections.md +++ b/doc/development/projections.md @@ -28,8 +28,6 @@ You can find a basic list of projection options in - [Alternate File](https://marketplace.visualstudio.com/items?itemName=will-wow.vscode-alternate-file) - [projectionist](https://github.com/jarsen/projectionist) - [`jumpto`](https://github.com/gmdayley/jumpto) -- Atom - - [projectionist-atom](https://atom.io/packages/projectionist-atom) - Command-line - [projectionist](https://github.com/glittershark/projectionist) @@ -41,4 +39,4 @@ This started as a [plugin for vim by tpope](https://github.com/tpope/vim-projectionist) It has since become editor-agnostic and ported to most modern editors. -<!-- vale gitlab.Spelling = YES -->
\ No newline at end of file +<!-- vale gitlab.Spelling = YES --> diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index 590ac547c2d..f72805e0fe9 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -78,3 +78,15 @@ For example, a histogram with 10 buckets and a label with 100 values would gener entries in the export endpoint. 1. Trigger the relevant page or code that records the new metric. 1. Check that the new metric appears at `/-/metrics`. + +For metrics that are not bounded to a specific context (`request`, `process`, `machine`, `namespace`, etc), +generate them from a cron-based Sidekiq job: + +- For Geo related metrics, check `Geo::MetricsUpdateService`. +- For other "global" / instance-wide metrics, check: `Metrics::GlobalMetricsUpdateService`. + +When exporting data from Sidekiq in an installation with more than one Sidekiq instance, +you are not guaranteed that the same exporter will always be queried. + +You can read more and understand the caveats in [issue 406583](https://gitlab.com/gitlab-org/gitlab/-/issues/406583), +where we also discuss a possible solution using a push-gateway. diff --git a/doc/development/real_time.md b/doc/development/real_time.md index 99d09170646..017e308fc03 100644 --- a/doc/development/real_time.md +++ b/doc/development/real_time.md @@ -1,29 +1,347 @@ --- -stage: Plan -group: Project Management +stage: Data Stores +group: Application Performance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Real-time features +# Build and deploy real-time view components -This guide contains instructions on how to safely roll out new real-time -features. +GitLab provides an interactive user experience through individual view components that accept +user input and reflect state changes back to the user. For example, on the Merge Request +page, users can approve, leave comments, interact with the CI/CD pipeline, and more. -Real-time features are implemented using GraphQL Subscriptions. -[Developer documentation](api_graphql_styleguide.md#subscriptions) is available. +However, GitLab often does not reflect state updates in a timely manner. +This means parts of the page display stale data that only update after users reload the page. + +To address this, GitLab has introduced technology and programming APIs that allow view components +to receive state updates in real-time over a WebSocket. + +The following documentation tells you how to build and deploy view components that +receive updates in real-time from the GitLab Ruby on Rails server. + +NOTE: +Action Cable and GraphQL subscriptions are a work-in-progress and under active development. +Developers must evaluate their use case to check if these are the right tools to use. +If you are not sure, ask for help in the [`#f_real-time` internal Slack channel](https://gitlab.slack.com/archives/CUX9Z2N66). + +## Build real-time view components + +Prerequisites: + +Read the: + +- [GraphQL development guide](fe_guide/graphql.md). +- [Vue development guide](fe_guide/vue.md). + +To build a real-time view component on GitLab, you must: + +- Integrate a Vue component with Apollo subscriptions in the GitLab frontend. +- Add and trigger GraphQL subscriptions from the GitLab Ruby on Rails backend. + +### Integrate a Vue component with Apollo subscriptions + +NOTE: +Our current real-time stack assumes that client code is built using Vue as the rendering layer and +Apollo as the state and networking layer. If you are working with a part of +the GitLab frontend that has not been migrated to Vue + Apollo yet, complete that task first. + +Consider a hypothetical `IssueView` Vue component that observes and renders GitLab `Issue` data. +For simplicity, we assume here that all it does is render an issue's title and description: + +```javascript +import issueQuery from '~/issues/queries/issue_view.query.graqhql'; + +export default { + props: { + issueId: { + type: Number, + required: false, + default: null, + }, + }, + apollo: { + // Name of the Apollo query object. Must match the field name bound by `data`. + issue: { + // Query used for the initial fetch. + query: issueQuery, + // Bind arguments used for the initial fetch query. + variables() { + return { + iid: this.issueId, + }; + }, + // Map response data to view properties. + update(data) { + return data.project?.issue || {}; + }, + }, + }, + // Reactive Vue component data. Apollo updates these when queries return or subscriptions fire. + data() { + return { + issue: {}, // It is good practice to return initial state here while the view is loading. + }; + }, +}; + +// The <template> code is omitted for brevity as it is not relevant to this discussion. +``` + +The query should: + +- Be defined at `app/assets/javascripts/issues/queries/issue_view.query.graqhql`. +- Contain the following GraphQL operation: + + ```plaintext + query gitlabIssue($iid: String!) { + # We hard-code the path here only for illustration. Don't do this in practice. + project(fullPath: "gitlab-org/gitlab") { + issue(iid: $iid) { + title + description + } + } + } + ``` + +So far this view component only defines the initial fetch query to populate itself with data. +This is an ordinary GraphQL `query` operation sent as an HTTP POST request, initiated by the view. +Any subsequent updates on the server would make this view stale. For it to receive updates from the server, you must: + +1. Add a GraphQL subscription definition. +1. Define an Apollo subscription hook. + +#### Add a GraphQL subscription definition + +A subscription defines a GraphQL query as well, but it is wrapped inside a GraphQL `subscription` operation. +This query is initiated by the backend and its results pushed over a WebSocket into the view component. + +Similar to the initial fetch query, you must: + +- Define the subscription file at `app/assets/javascripts/issues/queries/issue_updated.subscription.graqhql`. +- Include the following GraphQL operation in the file: + + ```plaintext + subscription issueUpdatedSubscription($iid: String!) { + issueUpdated($issueId: IssueID!) { + issue(issueId: $issueId) { + title + description + } + } + } + ``` + +When adding new subscriptions, use the following naming guidelines: + +- End the subscription's operation name with `Subscription`, or `SubscriptionEE` if it's exclusive to GitLab EE. + For example, `issueUpdatedSubscription`, or `issueUpdatedSubscriptionEE`. +- Use a "has happened" action verb in the subscription's event name. For example, `issueUpdated`. + +While subscription definitions look similar to ordinary queries, there are some key differences that are important to understand: + +- The `query`: + - Originates from the frontend. + - Uses an internal ID (`iid`, numeric), which is how entities are usually referenced in URLs. + Because the internal ID is relative to the enclosing namespace (in this example, the `project`), you must nest the query under the `fullPath`. +- The `subscription`: + - Is a request from the frontend to the backend to receive future updates. + - Consists of: + - The operation name describing the subscription itself (`issueUpdatedSubscription` in this example). + - A nested event query (`issueUpdated` in this example). The nested event query: + - Executes when running the [GraphQL trigger](#trigger-graphql-subscriptions) of the same name, + so the event name used in the subscription must match the trigger field used in the backend. + - Uses a global ID string instead of a numeric internal ID, which is the preferred way to identify resources in GraphQL. + For more information, see [GraphQL global IDs](fe_guide/graphql.md#global-ids). + +#### Define an Apollo subscription hook + +After defining the subscription, add it to the view component using Apollo's `subscribeToMore` property: + +```javascript +import issueQuery from '~/issues/queries/issue_view.query.graqhql'; +import issueUpdatedSubscription from '~/issues/queries/issue_updated.subscription.graqhql'; + +export default { + // As before. + // ... + apollo: { + issue: { + // As before. + // ... + // This Apollo hook enables real-time pushes. + subscribeToMore: { + // Subscription operation that returns future updates. + document: issueUpdatedSubscription, + // Bind arguments used for the subscription operation. + variables() { + return { + iid: this.issueId, + }; + }, + // Implement this to return true|false if subscriptions should be disabled. + // Useful when using feature-flags. + skip() { + return this.shouldSkipRealTimeUpdates; + }, + }, + }, + }, + // As before. + // ... + computed: { + shouldSkipRealTimeUpdates() { + return false; // Might check a feature flag here. + }, + }, +}; +``` + +Now you can enable the view component to receive updates over a WebSocket connection through Apollo. +Next, we cover how events are triggered from the backend to initiate a push update to the frontend. + +### Trigger GraphQL subscriptions + +Writing a view component that can receive updates from a WebSocket is only half the story. +In the GitLab Rails application, we need to perform the following steps: + +1. Implement a `GraphQL::Schema::Subscription` class. This class: + - Is used by `graphql-ruby` to resolve the `subscription` operation sent by the frontend. + - Defines the arguments a subscription takes and the payload returned to the caller, if any. + - Runs any necessary business logic to ensure that the caller is authorized to create this subscription. +1. Add a new `field` to the `Types::SubscriptionType` class. This field maps the event name used + [when integrating the Vue component](#integrate-a-vue-component-with-apollo-subscriptions) to the + `GraphQL::Schema::Subscription` class. +1. Add a method matching the event name to `GraphqlTriggers` that runs the corresponding GraphQL trigger. +1. Use a service or Active Record model class to execute the new trigger as part of your domain logic. + +#### Implement the subscription + +If you subscribe to a an event that is already implemented as a `GraphQL::Schema::Subscription`, this step is optional. +Otherwise, create a new class under `app/graphql/subscriptions/` +that implements the new subscription. For the example of an `issueUpdated` event happening in response to an `Issue` being updated, +the subscription implementation is as follows: + +```ruby +module Subscriptions + class IssueUpdated < BaseSubscription + include Gitlab::Graphql::Laziness + + payload_type Types::IssueType + + argument :issue_id, Types::GlobalIDType[Issue], + required: true, + description: 'ID of the issue.' + + def authorized?(issue_id:) + issue = force(GitlabSchema.find_by_gid(issue_id)) + + unauthorized! unless issue && Ability.allowed?(current_user, :read_issue, issue) + + true + end + end +end +``` + +When creating this new class: + +- Make sure every subscription type inherits from `Subscriptions::BaseSubscription`. +- Use an appropriate `payload_type` to indicate what data subscribed queries may access, + or define the individual `field`s you want to expose. +- You may also define custom `subscribe` and `update` hooks that are called each time a client subscribes or + an event fires. Refer to the [official documentation](https://graphql-ruby.org/subscriptions/subscription_classes) + for how to use these methods. +- Implement `authorized?` to perform any necessary permission checks. These checks execute for each call + to `subscribe` or `update`. + +Read more about GraphQL subscription classes [in the official documentation](https://graphql-ruby.org/subscriptions/subscription_classes). + +#### Hook up the subscription + +Skip this step if you did not implement a new subscription class. + +After you implement a new subscription class, you must map that class to a `field` on the `SubscriptionType` before +it can execute. Open the `Types::SubscriptionType` class and add the new field: + +```ruby +module Types + class SubscriptionType < ::Types::BaseObject + graphql_name 'Subscription' + + # Existing fields + # ... + + field :issue_updated, + subscription: Subscriptions::IssueUpdated, null: true, + description: 'Triggered when an issue is updated.' + end +end +``` + +NOTE: +If you are connecting an EE subscription, update `EE::Types::SubscriptionType` instead. + +Make sure the `:issue_updated` argument matches the name used in the `subscription` request sent by the frontend in camel-case (`issueUpdated`), or `graphql-ruby` does not know which subscribers to inform. The event can now trigger. + +#### Add the new trigger + +Skip this step if you can reuse an existing trigger. + +We use a facade around `GitlabSchema.subscriptions.trigger` to make it simpler to trigger an event. +Add the new trigger to `GraphqlTriggers`: + +```ruby +module GraphqlTriggers + # Existing triggers + # ... + + def self.issue_updated(issue) + GitlabSchema.subscriptions.trigger(:issue_updated, { issue_id: issue.to_gid }, issue) + end +end +``` + +NOTE: +If the trigger is for an EE subscription, update `EE::GraphqlTriggers` instead. + +- The first argument, `:issue_updated`, must match the `field` name used in the previous + step. +- The argument hash specifies the issue for which we publish the event. + GraphQL uses this hash to identify the topic it should publish the event to. + +The final step is to call into this trigger function. + +#### Execute the trigger + +The implementation of this step depends on what exactly it is you are building. In the example +of the issue's fields changing, we could extend `Issues::UpdateService` to call `GraphqlTriggers.issue_updated`. + +The real-time view component is now functional. Updates to an issue should now propagate immediately into the GitLab UI. + +## Deploy real-time view components WebSockets are a relatively new technology at GitLab, and supporting them at scale introduces some challenges. For that reason, new features should be rolled out using the instructions below. -## Reuse an existing WebSocket connection +### Shipping a real-time component + +You can work on the frontend and backend at the same time, because updates over WebSockets +are difficult to simulate without the necessary backend code in place. + +However, it is safer to send changes in separate Merge Requests and deploy the backend changes first. +This ensures that when the frontend starts subscribing to events, the backend is already prepared +to service them. + +### Reuse an existing WebSocket connection Features reusing an existing connection incur minimal risk. Feature flag rollout is recommended to give more control to self-hosting customers. However, it is not necessary to roll out in percentages, or to estimate new connections for GitLab.com. -## Introduce a new WebSocket connection +### Introduce a new WebSocket connection Any change that introduces a WebSocket connection to part of the GitLab application incurs some scalability risk, both to nodes responsible for maintaining open @@ -70,7 +388,7 @@ of the feature flag ensures that effects can be observed on the 1. Copy in a member of the Plan and Scalability teams to estimate a percentage-based roll-out plan. -## Backward compatibility +### Backward compatibility For the duration of the feature flag roll-out and indefinitely thereafter, real-time features must be backward-compatible, or at least degrade @@ -80,18 +398,200 @@ needs to be done before Action Cable can be enabled by default. Making real-time a requirement represents a breaking change, so the next opportunity to do this is version 15.0. -## Enable Real-Time by default - -Mounting the Action Cable library adds minimal memory footprint. However, -serving WebSocket requests introduces additional memory requirements. For this -reason, enabling Action Cable by default requires additional work; perhaps -to reduce overall memory usage, including a known issue with Workhorse, but at -least to revise Reference Architectures. - -## Real-time infrastructure on GitLab.com +### Real-time infrastructure on GitLab.com On GitLab.com, WebSocket connections are served from dedicated infrastructure, entirely separate from the regular Web fleet and deployed with Kubernetes. This limits risk to nodes handling requests but not to shared services. For more information on the WebSockets Kubernetes deployment see [this epic](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/355). + +## The GitLab real-time stack in depth + +Because a push initiated by the server needs to propagate over the network and trigger a view update +in the client without any user interaction whatsoever, real-time features can only be understood +by looking at the entire stack including frontend and backend. + +NOTE: +For historic reasons, the controller routes that service updates in response to clients polling +for changes are called `realtime_changes`. They use conditional GET requests and are unrelated +to the real-time behavior covered in this guide. + +Any real-time update pushed into a client originates from the GitLab Rails application. We use the following +technologies to initiate and service these updates: + +In the GitLab Rails backend: + +- Redis PubSub to handle subscription state. +- Action Cable to handle WebSocket connections and data transport. +- `graphql-ruby` to implement GraphQL subscriptions and triggers. + +In the GitLab frontend: + +- Apollo Client to handle GraphQL requests, routing and caching. +- Vue.js to define and render view components that update in real-time. + +The following figure illustrates how data propagates between these layers. + +```mermaid +sequenceDiagram + participant V as Vue Component + participant AP as Apollo Client + participant P as Rails/GraphQL + participant AC as Action Cable/GraphQL + participant R as Redis PubSub + AP-->>V: injected + AP->>P: HTTP GET /-/cable + AC-->>P: Hijack TCP connection + AC->>+R: SUBSCRIBE(client) + R-->>-AC: channel subscription + AC-->>AP: HTTP 101: Switching Protocols + par + V->>AP: query(gql) + Note over AP,P: Fetch initial data for this view + AP->>+P: HTTP POST /api/graphql (initial query) + P-->>-AP: initial query response + AP->>AP: cache and/or transform response + AP->>V: trigger update + V->>V: re-render + and + Note over AP,AC: Subscribe to future updates for this view + V->>AP: subscribeToMore(event, gql) + AP->>+AC: WS: subscribe(event, query) + AC->>+R: SUBSCRIBE(event) + R-->>-AC: event subscription + AC-->>-AP: confirm_subscription + end + Note over V,R: time passes + P->>+AC: trigger event + AC->>+R: PUBLISH(event) + R-->>-AC: subscriptions + loop For each subscriber + AC->>AC: run GQL query + AC->>+R: PUBLISH(client, query_result) + R-->>-AC: callback + AC->>-AP: WS: push query result + end + AP->>AP: cache and/or transform response + AP->>V: trigger update + V->>V: re-render +``` + +In the subsequent sections we explain each element of this stack in detail. + +### Action Cable and WebSockets + +[Action Cable](https://guides.rubyonrails.org/action_cable_overview.html) is a library that adds +[WebSocket](https://www.rfc-editor.org/rfc/rfc6455) support to Ruby on Rails. +WebSockets were developed as an HTTP-friendly solution to enhance existing HTTP-based servers and +applications with bidirectional communication over a single TCP connection. +A client first sends an ordinary HTTP request to the server, asking it to upgrade the connection +to a WebSocket instead. When successful, the same TCP connection can then be used by both client +and server to send and receive data in either direction. + +Because the WebSocket protocol does not prescribe how the transmitted data is encoded or structured, +we need libraries like Action Cable that take care of these concerns. Action Cable: + +- Handles the initial connection upgrade from HTTP to the WebSocket protocol. Subsequent requests using + the `ws://` scheme are then handled by the Action Cable server and not Action Pack. +- Defines how data transmitted over the WebSocket is encoded. Action Cable specifies this to be JSON. This allows the + application to provide data as a Ruby Hash and Action Cable (de)serializes it from and to JSON. +- Provides callback hooks to handle clients connecting or disconnecting and client authentication. +- Provides `ActionCable::Channel` as a developer abstraction to implement publish/subscribe and remote procedure calls. + +Action Cable supports different implementations to track which client is subscribed to which +`ActionCable::Channel`. At GitLab we use the Redis adapter, which uses +[Redis PubSub](https://redis.io/docs/manual/pubsub/) channels as a distributed message bus. +Shared storage is necessary because different clients might connect to the same Action Cable channel +from different Puma instances. + +NOTE: +Do not confuse Action Cable channels with Redis PubSub channels. An Action Cable `Channel` object is a +programming abstraction to classify and handle the various kinds of data going over the WebSocket connection. +In Action Cable, the underlying PubSub channel is referred to as a broadcasting instead and the association +between a client and a broadcasting is called a subscription. In particular, there can be many broadcastings +(PubSub channels) and subscriptions for each Action Cable `Channel`. + +Because Action Cable allows us to express different kinds of behavior through its `Channel` API, and because +updates to any `Channel` can use the same WebSocket connection, we only require a single WebSocket connection +to be established for each GitLab page to enhance a view component on that page with real-time behavior. + +To implement real-time updates on a GitLab page, we do not write individual `Channel` implementations. +Instead, we provide the `GraphqlChannel` to which all pages that require push-based updates on GitLab +subscribe. + +### GraphQL subscriptions: Backend + +GitLab supports [GraphQL](https://graphql.org) for clients to request structured data from the server +using GraphQL queries. Refer to the [GitLab GraphQL overview](../api/graphql/index.md) to learn about why we adopted GraphQL. +GraphQL support in the GitLab backend is provided by the [`graphql-ruby`](https://graphql-ruby.org) gem. + +Ordinarily, GraphQL queries are client-initiated HTTP POST requests that follow the standard request-response cycle. +For real-time functionality, we use GraphQL subscriptions instead, which are an implementation of the publish/subscribe pattern. +In this approach the client first sends a subscription request to the `GraphqlChannel` with the: + +- Name of the subscription `field` (the event name). +- GraphQL query to run when this event triggers. + +This information is used by the server to create a `topic` that represents this event stream. The topic is a unique name +derived from the subscription arguments and event name and is used to identify all subscribers +that need to be informed if the event triggers. More than one client can subscribe to the +same topic. For example, `issuableAssigneesUpdated:issuableId:<hashed_id>` might serve as the topic +that clients subscribe to if they wish to be updated whenever the assignees for the issue with the +given ID change. + +The backend is responsible for triggering a subscription, typically in response to a domain +event such as "issue added to epic" or "user assigned to issue". At GitLab, this could be a service object +or an ActiveRecord model object. +A trigger is executed by calling into [`GitlabSchema.subscriptions.trigger`](https://gitlab.com/gitlab-org/gitlab/-/blob/5e3c334116178eec5f50fc5fee2ec0b3841a2504/app/graphql/graphql_triggers.rb) with the respective event name and arguments, +from which `graphql-ruby` derives the topic. It then finds all subscribers for this topic, executes the query for +each subscriber, and pushes the result back to all topic subscribers. + +Because we use Action Cable as the underlying transport for GraphQL subscriptions, topics are implemented +as Action Cable broadcastings, which as mentioned above represent Redis PubSub channels. +This means that for each subscriber, two PubSub channels are used: + +- One `graphql-event:<namespace>:<topic>` channel per each topic. This channel is used to track which client is subscribed + to which event and is shared among all potential clients. The use of a `namespace` is optional and it can be blank. +- One `graphql-subscription:<subscription-id>` channel per each client. This channel is used to transmit the query result + back to the respective client and hence cannot be shared between different clients. + +The next section describes how the GitLab frontend uses GraphQL subscriptions to implement real-time updates. + +### GraphQL subscriptions: Frontend + +Because the GitLab frontend executes JavaScript, not Ruby, we need a different GraphQL implementation +to send GraphQL queries, mutations, and subscriptions from the client to the server. +We use [Apollo](https://www.apollographql.com) to do this. + +Apollo is a comprehensive implementation of GraphQL in JavaScript and is split into `apollo-server` and `apollo-client` +as well as additional utility modules. Because we run a Ruby backend, we use `apollo-client` instead of `apollo-server`. + +It simplifies: + +- Networking, connection management and request routing. +- Client-side state management and response caching. +- Integrating GraphQL with view components using a bridge module. + +NOTE: +When reading the Apollo Client documentation, it assumes that React.js is used for view rendering. We do not use React.js +at GitLab. We use Vue.js, which integrates with Apollo using the [Vue.js adapter](https://apollo.vuejs.org/). + +Apollo provides functions and hooks with which you define how: + +- Views send queries, mutations or subscriptions. +- Responses should be dealt with. +- Response data is cached. + +The entry point is `ApolloClient`, which is a GraphQL client object that: + +- Is shared between all view components on a single page. +- All view components use internally to communicate with the server. + +To decide how different types of requests should be routed, Apollo uses the `ApolloLink` abstraction. Specifically, +it splits real-time server subscriptions from other GraphQL requests using the `ActionCableLink`. This: + +- Establishes the WebSocket connection to Action Cable. +- Maps server pushes to an `Observable` event stream in the client that views can subscribe to in order to update themselves. + +For more information about Apollo and Vue.js, see the [GitLab GraphQL development guide](fe_guide/graphql.md). diff --git a/doc/development/redis/new_redis_instance.md b/doc/development/redis/new_redis_instance.md index c1f69f4d103..00cc102b427 100644 --- a/doc/development/redis/new_redis_instance.md +++ b/doc/development/redis/new_redis_instance.md @@ -119,7 +119,7 @@ Migration Requirements: - No downtime. - No loss of stored data until the TTL for storing data expires. -- Partial rollout using Feature Flags or ENV vars or combinations of both. +- Partial rollout using feature flags or ENV vars or combinations of both. - Monitoring of the switch. - Prometheus metrics in place. - Easy rollback without downtime in case the new instance or logic does not behave as expected. diff --git a/doc/development/scalability.md b/doc/development/scalability.md index de9c57c2f2a..733e94cb5a7 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -123,7 +123,7 @@ the read replicas. [Omnibus ships with Patroni](../administration/postgresql/rep #### Load-balancing -GitLab EE has [application support for load balancing using read replicas](../administration/postgresql/database_load_balancing.md). This load balancer does +GitLab EE has [application support for load balancing using read replicas](database/load_balancing.md). This load balancer does some actions that aren't traditionally available in standard load balancers. For example, the application considers a replica only if its replication lag is low (for example, WAL data behind by less than 100 MB). diff --git a/doc/development/sec/index.md b/doc/development/sec/index.md index f74d31a1fb3..b887d13c267 100644 --- a/doc/development/sec/index.md +++ b/doc/development/sec/index.md @@ -52,7 +52,7 @@ The Analyzers are mainly written in Go. Some 3rd party integrators also make additional Scanners available by following our [integration documentation](../integrations/secure.md), which leverages the same architecture. -The results of the scans are exported as JSON reports that must comply with the [Secure report format](../../user/application_security/terminology/index.md#secure-report-format) and are uploaded as [CI/CD Job Report artifacts](../../ci/pipelines/job_artifacts.md) to make them available for processing after the pipelines completes. +The results of the scans are exported as JSON reports that must comply with the [Secure report format](../../user/application_security/terminology/index.md#secure-report-format) and are uploaded as [CI/CD Job Report artifacts](../../ci/jobs/job_artifacts.md) to make them available for processing after the pipelines completes. ### Processing, visualization, and management diff --git a/doc/development/sec/security_report_ingestion_overview.md b/doc/development/sec/security_report_ingestion_overview.md index 688986e0eb1..aca33990b0f 100644 --- a/doc/development/sec/security_report_ingestion_overview.md +++ b/doc/development/sec/security_report_ingestion_overview.md @@ -7,15 +7,56 @@ type: concepts # Security report ingestion overview -## Definitions +WARNING: +The `Vulnerability::Feedback` model is currently undergoing deprecation and should be actively avoided in all further development. It is currently maintained with feature parity to enable revert should any issues arise, but is intended to be removed in 16.0. Any interactions relating to the Feedback model are superseded by the `StateTransition`, `IssueLink`, and `MergeRequestLink` models. You can find out more on [in this epic](https://gitlab.com/groups/gitlab-org/-/epics/5629). -- **Vulnerability Finding** – an instance of `Vulnerabilities::Finding` class. This class was previously called `Vulnerabilities::Occurrence`; after renaming the class, we kept the associated table name `vulnerability_occurrences` due to the effort involved in renaming large tables. -- **Vulnerability** – an instance of `Vulnerability` class. They are created based on information available in `Vulnerabilities::Finding` class. Every `Vulnerability` **must have** a corresponding `Vulnerabilities::Finding` object to be valid, however this is not enforced at the database level. -- **Security Finding** – an instance of `Security::Finding` class. They store **partial** finding data to improve performance of the pipeline security report. We are working on extending this class to store almost all required information so we can stop relying on job artifacts. -- **Feedback** – an instance of `Vulnerabilities::Feedback` class. They are created to keep track of users' interactions with Vulnerability Findings before they are promoted to a Vulnerability. We are in the process of removing this model via [Deprecate and remove Vulnerabilities::Feedback epic](https://gitlab.com/groups/gitlab-org/-/epics/5629). -- **Issue Link** – an instance of `Vulnerabilities::IssueLink` class. They are used to link `Vulnerability` objects to `Issue` objects. +## Commonly used terms -## Vulnerability creation from security reports +### Feedback + +An instance of `Vulnerabilities::Feedback` class. They are created to keep track of users' interactions with Vulnerability Findings before they are promoted to a Vulnerability. This model is deprecated and due to be removed by GitLab 16.0 as part of the [Deprecate and remove Vulnerabilities::Feedback epic](https://gitlab.com/groups/gitlab-org/-/epics/5629). + +### Issue Link + +An instance of `Vulnerabilities::IssueLink` class. They are used to link `Vulnerability` records to `Issue` records. + +### Merge Request Link + +An instance of `Vulnerabilities::MergeRequestLink` class. They are used to link `Vulnerability` records to `MergeRequest` records. + +### Security Finding + +An instance of `Security::Finding` class. These serve as a meta-data store of a specific vulnerability detected in a specific `Security::Scan`. They currently store **partial** finding data to improve performance of the pipeline security report. This class has been extended to store almost all required scan information so we can stop relying on job artifacts and is [due to be used in favor of `Vulnerability::Findings` soon.](https://gitlab.com/gitlab-org/gitlab/-/issues/393394) + +### Security Scan + +An instance of the `Security::Scan` class. Security scans are representative of a `Ci::Build` which output a `Job Artifact` which has been output as a security scan result, which GitLab acknowledges and ingests the findings of as `Security::Finding` records. + +### State Transition + +An instance of the `Vulnerabilities::StateTransition` class. This model represents a state change of a respecitve Vulnerability record, for example the dismissal of a vulnerability which has been determined to be safe. + +### Vulnerability + +An instance of `Vulnerability` class. A `Vulnerability` is representative of a `Vulnerability::Finding` which has been detected in the default branch of the project, or if the `present_on_default_branch` flag is false, is representative of a finding which has been interacted with in some way outside of the default branch, such as if it is dismissed (`State Transition`), or linked to an `Issue` or `Merge Request`. They are created based on information available in `Vulnerabilities::Finding` class. Every `Vulnerability` **must have** a corresponding `Vulnerabilities::Finding` object to be valid, however this is not enforced at the database level. + +### Finding + +An instance of `Vulnerabilities::Finding` class. A `Vulnerability::Finding` is a database only representation of a security finding which has been merged into the default branch of a project, as the same `Vulnerability` may be present in multiple places within a project. This class was previously called `Vulnerabilities::Occurrence`; after renaming the class, we kept the associated table name `vulnerability_occurrences` due to the effort involved in renaming large tables. + +### Identifier + +An instance of the `Vulnerabilities::Identifier` class. Each vulnerability is given a unique identifier that can be derived from it's finding, enabling multiple Findings of the same `Vulnerability` to be correlated accordingly. + +### Vulnerability Read + +An instance of the `Vulnerabilities::Read` class. This is a denormalised record of `Vulnerability` and `Vulnerability::Finding` data to improve performance of filtered querying of vulnerability data to the front end. + +### Remediation + +An instance of the `Vulnerabilities::Remediation` class. A remediation is representative of a known solution to a detected `Vulnerability`. These enable GitLab to recommend a change to resolve a specific `Vulnerability`. + +## Vulnerability creation from Security Reports Assumptions: @@ -24,51 +65,53 @@ Assumptions: - No Vulnerabilities are present in the database - All pipelines perform security scans -1. Code is pushed to a branch that's **not** the default branch. +### Scan runs in a pipeline for a non-default branch + +1. Code is pushed to the branch. 1. GitLab CI runs a new pipeline for that branch. 1. Pipeline status transitions to any of [`::Ci::Pipeline.completed_statuses`](https://gitlab.com/gitlab-org/gitlab/-/blob/354261b2fe4fc5b86d1408467beadd90e466ce0a/app/models/concerns/ci/has_status.rb#L12). 1. `Security::StoreScansWorker` is called and it schedules `Security::StoreScansService`. -1. `Security::StoreScansService` calls `Security::StoreGroupedScansService`. +1. `Security::StoreScansService` calls `Security::StoreGroupedScansService` and schedules `ScanSecurityReportSecretsWorker`. 1. `Security::StoreGroupedScansService` calls `Security::StoreScanService`. 1. `Security::StoreScanService` calls `Security::StoreFindingsService`. -1. At this point we have `Security::Finding` objects **only**. +1. `ScanSecurityReportSecretsWorker` calls `Security::TokenRevocationService` to automatically revoke any leaked keys that were detected. +1. At this point we **only** have `Security::Finding` records as these findings are not present in the default branch of the project. -At this point, the following things can happen to the `Security::Finding`: +At this point, the following things can happen to the `Security::Finding` which would result in its promotion to a `Vulnerability::Finding` with a respective `Vulnerability` record: -- Dismissal -- Issue creation -- Promotion to a Vulnerability +### Scan runs in a pipeline for the default branch -If the pipeline ran on the default branch then the following, additional steps are done: +If the pipeline ran on the default branch then the following steps, in addition to the steps in [Scan runs in a pipeline for a non-default branch](#scan-runs-in-a-pipeline-for-a-non-default-branch), are executed: -1. `Security::StoreScansService` gets called and schedules `Security::StoreSecurityReportsWorker`. -1. `Security::StoreSecurityReportsWorker` executes `Security::Ingestion::IngestReportsService`. +1. `Security::StoreScansService` gets called and schedules `StoreSecurityReportsWorker`. +1. `StoreSecurityReportsWorker` executes `Security::Ingestion::IngestReportsService`. 1. `Security::Ingestion::IngestReportsService` takes all reports from a given Pipeline and calls `Security::Ingestion::IngestReportService` and then calls `Security::Ingestion::MarkAsResolvedService`. 1. `Security::Ingestion::IngestReportService` calls `Security::Ingestion::IngestReportSliceService` which executes a number of tasks for a report slice. ### Dismissal -If you select `Dismiss vulnerability`, a Feedback is created. You can also dismiss it with a comment. +If you change the state of a vulnerability, such as selecting `Dismiss vulnerability` the following things currently happen: -#### After Feedback removal +- A `Feedback` record of `dismissal` type is created to record the current state. +- If they do not already exist, a `Vulnerability Finding` and a `Vulnerability` with `present_on_default_branch: false` attribute get created, to which a `State Transition` reflecting the state change is related. -If there is only a Security Finding, a Vulnerability Finding and a Vulnerability get created. At the same time we create a `Vulnerabilities::StateTransition` record to indicate the Vulnerability was dismissed. +You can optionally add a comment to the state change which is recorded on both the `Feedback` and the `State Transition`. -### Issue creation +### Issue or Merge Request creation -If you select `Create issue`, a Vulnerabilities::Feedback record is created as well. The Feedback has a different `feedback_type` and an `issue_id` that's not `NULL`. +If you select `Create issue` or `Create merge request` the following things currently happen: -NOTE: -Vulnerabilities::Feedback are in the process of being [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/5629). This will later create a `Vulnerabilities::IssueLink` record. +- A `Vulnerabilities::Feedback` record is created. The Feedback will have a `feedback_type` of `issue` or `merge request` and an `issue_id` or `merge_request_id` that's not `NULL` respective to the attachment. +- If they do not already exist, a `Vulnerability Finding` and a `Vulnerability` with `present_on_default_branch: false` attribute get created, to which a `Issue Link` or `Merge Request Link` will be related respective to the action taken. -#### After Feedback removal +## Vulnerabilities in the Default Branch -If there's only a Security Finding, a Vulnerability Finding and a Vulnerability gets created. At the same time, we create an Issue and a Issue Link. +Security Findings detected in scan run on the default branch are saved as `Vulnerabilities` with the `present_on_default_branch: true` attribute and respective `Vulnerability Finding` records. `Vulnerability` records that already exist from interactions outside of the default branch will be updated to `present_on_default_branch: true` -## Promotion to a Vulnerability +`Vulnerabilities` which have already been interacted with will retain all existing `State Transitions`, `Merge Request Links` and `Issue Links`, as well as a corresponding `Vulnerability Feedback`. -If the branch with a Security Finding gets merged into the default branch, all Security Findings get promoted into Vulnerabilities. Promotion is the process of creating Vulnerability Findings and Vulnerability records from those Security Findings. +## Vulnerability Read Creation -If there's a dismissal Feedback present for that Security Finding, the created Vulnerability is marked as dismissed. +`Vulnerability::Read` records are created via postgres database trigger upon the creation of a `Vulnerability::Finding` record and as such are part of our ingestion process, though they have no impact on it bar it's denormalization performance benefits on the report pages. -If there's an issue Feedback present for that Security Finding, we also create an Issue Link for that Vulnerability. +This style of creation was intended to be fast and seamless, but has proven difficult to debug and maintain and may be [migrated to the application layer later](https://gitlab.com/gitlab-org/gitlab/-/issues/393912). diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 4f644dd018e..f1342d24fb4 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -1290,6 +1290,7 @@ This sensitive data must be handled carefully to avoid leaks which could lead to - Credentials must be encrypted while at rest (database or file) with `attr_encrypted`. See [issue #26243](https://gitlab.com/gitlab-org/gitlab/-/issues/26243) before using `attr_encrypted`. - Store the encryption keys separately from the encrypted credentials with proper access control. For instance, store the keys in a vault, KMS, or file. Here is an [example](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/user.rb#L70-74) use of `attr_encrypted` for encryption with keys stored in separate access controlled file. - When the intention is to only compare secrets, store only the salted hash of the secret instead of the encrypted value. +- Salted hashes should be used to store any sensitive value where the plaintext value itself does not need to be retrieved. - Never commit credentials to repositories. - The [Gitleaks Git hook](https://gitlab.com/gitlab-com/gl-security/security-research/gitleaks-endpoint-installer) is recommended for preventing credentials from being committed. - Never log credentials under any circumstance. Issue [#353857](https://gitlab.com/gitlab-org/gitlab/-/issues/353857) is an example of credential leaks through log file. @@ -1306,6 +1307,32 @@ This sensitive data must be handled carefully to avoid leaks which could lead to In the event of credential leak through an MR, issue, or any other medium, [reach out to SIRT team](https://about.gitlab.com/handbook/security/security-operations/sirt/#-engaging-sirt). +### Examples + +Encrypting a token with `attr_encrypted` so that the plaintext can be retrieved and used later: + +```ruby +module AlertManagement + class HttpIntegration < ApplicationRecord + + attr_encrypted :token, + mode: :per_attribute_iv, + key: Settings.attr_encrypted_db_key_base_32, + algorithm: 'aes-256-gcm' +``` + +Hashing a sensitive value with `CryptoHelper` so that it can be compared in future, but the plaintext is irretrievable: + +```ruby +class WebHookLog < ApplicationRecord + before_save :set_url_hash, if: -> { interpolated_url.present? } + + def set_url_hash + self.url_hash = Gitlab::CryptoHelper.sha256(interpolated_url) + end +end +``` + ## Serialization Serialization of active record models can leak sensitive attributes if they are not protected. diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 5bfb81c1d00..54352a43010 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index e938de9e253..9d3f3d37dca 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index 90ce776af19..f36a97bcf6b 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -37,7 +37,6 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | `product_section` | yes | The [section](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/sections.yml). | | `product_stage` | yes | The [stage](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) for the metric. | | `product_group` | yes | The [group](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) that owns the metric. | -| `product_category` | no | The [product category](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml) for the metric. | | `value_type` | yes | `string`; one of [`string`, `number`, `boolean`, `object`](https://json-schema.org/understanding-json-schema/reference/type.html). | | `status` | yes | `string`; [status](#metric-statuses) of the metric, may be set to `active`, `removed`, `broken`. | | `time_frame` | yes | `string`; may be set to a value like `7d`, `28d`, `all`, `none`. | @@ -218,7 +217,6 @@ instance unique identifier. ```yaml key_path: uuid description: GitLab instance unique identifier -product_category: collection product_section: analytics product_stage: analytics product_group: product_intelligence diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md index 14ec4b8c9a6..7441a2d1bd4 100644 --- a/doc/development/service_ping/metrics_instrumentation.md +++ b/doc/development/service_ping/metrics_instrumentation.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md index 7fcbafe50f7..3c51eefc4b4 100644 --- a/doc/development/service_ping/metrics_lifecycle.md +++ b/doc/development/service_ping/metrics_lifecycle.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/service_ping/performance_indicator_metrics.md b/doc/development/service_ping/performance_indicator_metrics.md index 4c1c61aa05b..0ca663ce09a 100644 --- a/doc/development/service_ping/performance_indicator_metrics.md +++ b/doc/development/service_ping/performance_indicator_metrics.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/service_ping/review_guidelines.md b/doc/development/service_ping/review_guidelines.md index 9813c9e0b12..f260d9700a3 100644 --- a/doc/development/service_ping/review_guidelines.md +++ b/doc/development/service_ping/review_guidelines.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -44,7 +44,7 @@ are regular backend changes. - Assign an [engineer](https://gitlab.com/groups/gitlab-org/analytics-section/product-intelligence/engineers/-/group_members?with_inherited_permissions=exclude) from the Product Intelligence team for a review. - Set the correct attributes in the metric's YAML definition: - - `product_section`, `product_stage`, `product_group`, `product_category` + - `product_section`, `product_stage`, `product_group` - Provide a clear description of the metric. - Add a changelog [according to guidelines](../changelog.md). @@ -59,12 +59,11 @@ are regular backend changes. metrics that are based on Database. - Add `~Data Warehouse::Impact Check` for any database metric that has a query change. Changes in queries can affect [data operations](https://about.gitlab.com/handbook/business-technology/data-team/how-we-work/triage/#gitlabcom-db-structure-changes). - For tracking using Redis HLL (HyperLogLog): - - Check the Redis slot. - Check if a [feature flag is needed](implement.md#recommendations). - For a metric's YAML definition: - Check the metric's `description`. - Check the metric's `key_path`. - - Check the `product_section`, `product_stage`, `product_group`, and `product_category` fields. + - Check the `product_section`, `product_stage`, and `product_group` fields. Read the [stages file](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml). - Check the file location. Consider the time frame, and if the file should be under `ee`. - Check the tiers. diff --git a/doc/development/service_ping/troubleshooting.md b/doc/development/service_ping/troubleshooting.md index 3b7cd092d97..1b896efb726 100644 --- a/doc/development/service_ping/troubleshooting.md +++ b/doc/development/service_ping/troubleshooting.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/service_ping/usage_data.md b/doc/development/service_ping/usage_data.md index 1c7a212ed64..b3bdaedd60a 100644 --- a/doc/development/service_ping/usage_data.md +++ b/doc/development/service_ping/usage_data.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/sidekiq/worker_attributes.md b/doc/development/sidekiq/worker_attributes.md index a3bfe5f27cc..1e3104c5e86 100644 --- a/doc/development/sidekiq/worker_attributes.md +++ b/doc/development/sidekiq/worker_attributes.md @@ -242,7 +242,7 @@ can put unsustainable load on the primary database server. We therefore added th By configuring a worker's `data_consistency` field, we can then allow the scheduler to target read replicas under several strategies outlined below. -## Trading immediacy for reduced primary load +### Trading immediacy for reduced primary load We require Sidekiq workers to make an explicit decision around whether they need to use the primary database node for all reads and writes, or whether reads can be served from replicas. This is @@ -259,7 +259,8 @@ that mostly or exclusively perform writes, or workers that read their own writes into data consistency issues should a stale record be read back from a replica. **Try to avoid these scenarios, since `:always` should be considered the exception, not the rule.** -To allow for reads to be served from replicas, we added two additional consistency modes: `:sticky` and `:delayed`. +To allow for reads to be served from replicas, we added two additional consistency modes: `:sticky` and `:delayed`. A RuboCop rule +reminds the developer when `:always` data consistency mode is used. If workers require the primary database, you can disable the rule in-line. When you declare either `:sticky` or `:delayed` consistency, workers become eligible for database load-balancing. @@ -268,18 +269,17 @@ In both cases, if the replica is not up-to-date and the time from scheduling the the jobs sleep up to the minimum delay interval (0.8 seconds). This gives the replication process time to finish. The difference is in what happens when there is still replication lag after the delay: `sticky` workers switch over to the primary right away, whereas `delayed` workers fail fast and are retried once. -If they still encounter replication lag, they also switch to the primary instead. -**If your worker never performs any writes, it is strongly advised to apply one of these consistency settings, -since it never needs to rely on the primary database node.** +If the workers still encounter replication lag, they switch to the primary instead. **If your worker never performs any writes, +it is strongly advised to apply `:sticky` or `:delayed` consistency settings, since the worker never needs to rely on the primary database node.** The table below shows the `data_consistency` attribute and its values, ordered by the degree to which they prefer read replicas and wait for replicas to catch up: -| **Data Consistency** | **Description** | -|--------------|-----------------------------| -| `:always` | The job is required to use the primary database (default). It should be used for workers that primarily perform writes, have strict requirements around data consistency when reading their own writes, or are cron jobs. | -| `:sticky` | The job prefers replicas, but switches to the primary for writes or when encountering replication lag. It should be used for jobs that require to be executed as fast as possible but can sustain a small initial queuing delay. | -| `:delayed` | The job prefers replicas, but switches to the primary for writes. When encountering replication lag before the job starts, the job is retried once. If the replica is still not up to date on the next retry, it switches to the primary. It should be used for jobs where delaying execution further typically does not matter, such as cache expiration or web hooks execution. | +| **Data consistency** | **Description** | **Guideline** | +|--------------|-----------------------------|----------| +| `:always` | The job is required to use the primary database (default). | It should be used for workers that primarily perform writes, have strict requirements around data consistency when reading their own writes, or are cron jobs. | +| `:sticky` | The job prefers replicas, but switches to the primary for writes or when encountering replication lag. | It should be used for jobs that require to be executed as fast as possible but can sustain a small initial queuing delay. | +| `:delayed` | The job prefers replicas, but switches to the primary for writes. When encountering replication lag before the job starts, the job is retried once. If the replica is still not up to date on the next retry, it switches to the primary. | It should be used for jobs where delaying execution further typically does not matter, such as cache expiration or web hooks execution. | In all cases workers read either from a replica that is fully caught up, or from the primary node, so data consistency is always ensured. diff --git a/doc/development/snowplow/event_dictionary_guide.md b/doc/development/snowplow/event_dictionary_guide.md index dc2214a40ed..8825a38627a 100644 --- a/doc/development/snowplow/event_dictionary_guide.md +++ b/doc/development/snowplow/event_dictionary_guide.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md index ae90b45b1f0..05137a4410f 100644 --- a/doc/development/snowplow/implementation.md +++ b/doc/development/snowplow/implementation.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index 276b5913890..4ccb90c22a6 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/snowplow/infrastructure.md b/doc/development/snowplow/infrastructure.md index ae416f40c98..ac146542630 100644 --- a/doc/development/snowplow/infrastructure.md +++ b/doc/development/snowplow/infrastructure.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/snowplow/review_guidelines.md b/doc/development/snowplow/review_guidelines.md index da7f2bc2781..2cf13385179 100644 --- a/doc/development/snowplow/review_guidelines.md +++ b/doc/development/snowplow/review_guidelines.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/snowplow/schemas.md b/doc/development/snowplow/schemas.md index da58cd5f2e5..0ef6ed04aa3 100644 --- a/doc/development/snowplow/schemas.md +++ b/doc/development/snowplow/schemas.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/snowplow/troubleshooting.md b/doc/development/snowplow/troubleshooting.md index 47df3e43d57..92267dfcb0c 100644 --- a/doc/development/snowplow/troubleshooting.md +++ b/doc/development/snowplow/troubleshooting.md @@ -1,6 +1,6 @@ --- stage: Analytics -group: Product Intelligence +group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/stage_group_observability/dashboards/error_budget_detail.md b/doc/development/stage_group_observability/dashboards/error_budget_detail.md index a8f932b78c0..c45c31c5bd1 100644 --- a/doc/development/stage_group_observability/dashboards/error_budget_detail.md +++ b/doc/development/stage_group_observability/dashboards/error_budget_detail.md @@ -100,28 +100,3 @@ In the graphs, there is a single line per service. In the previous example image Sidekiq is not included in this dashboard. We're tracking this in [epic 700](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/700). - -### SLI detail - - - -The SLI details row shows a breakdown of a specific SLI based on the -labels present on the source metrics. - -For example, in the previous image, the `rails_requests` SLI has an `endpoint_id` label. -We can show how much a certain endpoint was requested (RPS), and how much it contributed to the error -budget spend. - -For Apdex we show the **Apdex Attribution** panel. The more prominent -color is the one that contributed most to the spend. To see the -top spending endpoint over the entire range, sort by the average. - -For error ratio we show an error rate. To see which label contributed most to the spend, sort by the -average. - -We don't have endpoint information available for Rails errors. This work is being planned in -[epic 663](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/663). - -The number of series to be loaded in the SLI details graphs is very -high when compared to the other aggregations. Because of this, it's not possible to -load more than a few days' worth of data. diff --git a/doc/development/stage_group_observability/dashboards/img/error_budget_detail_sli_detail.png b/doc/development/stage_group_observability/dashboards/img/error_budget_detail_sli_detail.png Binary files differdeleted file mode 100644 index 99530886ae9..00000000000 --- a/doc/development/stage_group_observability/dashboards/img/error_budget_detail_sli_detail.png +++ /dev/null diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index a9d17472a9f..3a2b40a6cd5 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -313,7 +313,28 @@ NOTE: `stub_method` does not support method existence and method arity checks. WARNING: -`stub_method` is supposed to be used in factories only. It's strongly discouraged to be used elsewhere. Please consider using [RSpec's mocks](https://relishapp.com/rspec/rspec-mocks/v/3-10/docs/basics) if available. +`stub_method` is supposed to be used in factories only. It's strongly discouraged to be used elsewhere. Please consider using [RSpec mocks](https://rspec.info/features/3-12/rspec-mocks/) if available. + +#### Stubbing member access level + +To stub [member access level](../../user/permissions.md#roles) for factory stubs like `Project` or `Group` use +[`stub_member_access_level`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/stub_member_access_level.rb): + +```ruby +let(:project) { build_stubbed(:project) } +let(:maintainer) { build_stubbed(:user) } +let(:policy) { ProjectPolicy.new(maintainer, project) } + +it 'allows admin_project ability' do + stub_member_access_level(project, maintainer: maintainer) + + expect(policy).to be_allowed(:admin_project) +end +``` + +NOTE: +Refrain from using this stub helper if the test code relies on persisting +`project_authorizations` or `Member` records. Use `Project#add_member` or `Group#add_member` instead. #### Identify slow tests @@ -434,7 +455,7 @@ results are available, and not just the first failure. - When using `evaluate_script("$('.js-foo').testSomething()")` (or `execute_script`) which acts on a given element, use a Capybara matcher beforehand (such as `find('.js-foo')`) to ensure the element actually exists. - Use `focus: true` to isolate parts of the specs you want to run. -- Use [`:aggregate_failures`](https://relishapp.com/rspec/rspec-core/docs/expectation-framework-integration/aggregating-failures) when there is more than one expectation in a test. +- Use [`:aggregate_failures`](https://rspec.info/features/3-12/rspec-core/expectation-framework-integration/aggregating-failures/) when there is more than one expectation in a test. - For [empty test description blocks](https://github.com/rubocop-hq/rspec-style-guide#it-and-specify), use `specify` rather than `it do` if the test is self-explanatory. - Use `non_existing_record_id`/`non_existing_record_iid`/`non_existing_record_access_level` when you need an ID/IID/access level that doesn't actually exists. Using 123, 1234, @@ -451,6 +472,10 @@ You can use `if: Gitlab.ee?` or `unless: Gitlab.ee?` on context/spec blocks to e Example: [SchemaValidator reads a different path depending on the license](https://gitlab.com/gitlab-org/gitlab/-/blob/7cdcf9819cfa02c701d6fa9f18c1e7a8972884ed/spec/lib/gitlab/ci/parsers/security/validators/schema_validator_spec.rb#L571) +### Tests depending on SaaS + +You can use the `:saas` RSpec metadata tag helper on context/spec blocks to test code that only runs on GitLab.com. This helper sets `Gitlab.config.gitlab['url']` to `Gitlab::Saas.com_url`. + ### Coverage [`simplecov`](https://github.com/colszowka/simplecov) is used to generate code test coverage reports. @@ -867,8 +892,6 @@ it 'is overdue' do travel_to(3.days.from_now) do expect(issue).to be_overdue end - - travel_back # Returns the current time back to its original state end ``` @@ -1017,7 +1040,7 @@ a single test triggers the rate limit, the `:disable_rate_limit` can be used ins #### Stubbing File methods In the situations where you need to -[stub](https://relishapp.com/rspec/rspec-mocks/v/3-9/docs/basics/allowing-messages) +[stub](https://rspec.info/features/3-12/rspec-mocks/basics/allowing-messages/) methods such as `File.read`, make sure to: 1. Stub `File.read` for only the file path you are interested in. diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index 07a9bcb2f5c..78e6af4a347 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -34,9 +34,8 @@ For more information, see [End-to-end testing Best Practices](best_practices.md) ## Determine if end-to-end tests are needed -Check the code coverage of a specific feature before writing end-to-end tests, -for both [GitLab Community Edition](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) -and [GitLab Enterprise Edition](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) projects. +Check the code coverage of a specific feature before writing end-to-end tests +for the [GitLab](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) project. Does sufficient test coverage exist at the unit, feature, or integration levels? If you answered *yes*, then you *don't* need an end-to-end test. @@ -53,9 +52,8 @@ For information about the distribution of tests per level in GitLab, see the feature and the lower-level tests. WARNING: -Check both [GitLab Community Edition](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) and -[GitLab Enterprise Edition](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) coverage projects -for previously-written tests for this feature. For analyzing the code coverage, +Check the [GitLab](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) coverage project +for previously written tests for this feature. To analyze code coverage, you must understand which application files implement specific features. In this tutorial we're writing a login end-to-end test, even though it has been @@ -225,7 +223,7 @@ end 1. Check if the user avatar *does not* appear in the top navigation. Behind the scenes, `be_signed_in` is a -[predicate matcher](https://relishapp.com/rspec/rspec-expectations/v/3-8/docs/built-in-matchers/predicate-matchers) +[predicate matcher](https://rspec.info/features/3-12/rspec-expectations/built-in-matchers/predicates/) that [implements checking the user avatar](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/page/main/menu.rb#L92). ## De-duplicate your code diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index fd184c13e5e..c1f06bb9a66 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -413,7 +413,7 @@ except(page).to have_no_text('hidden') ``` Unfortunately, that's not automatically the case for the predicate methods that we add to our -[page objects](page_objects.md). We need to [create our own negatable matchers](https://relishapp.com/rspec/rspec-expectations/v/3-9/docs/custom-matchers/define-a-custom-matcher#matcher-with-separate-logic-for-expect().to-and-expect().not-to). +[page objects](page_objects.md). We need to [create our own negatable matchers](https://rspec.info/features/3-12/rspec-expectations/custom-matchers/define-matcher/). The initial example uses the `have_job` matcher which is derived from the [`has_job?` predicate method of the `Page::Project::Pipeline::Show` page object](https://gitlab.com/gitlab-org/gitlab/-/blob/87864b3047c23b4308f59c27a3757045944af447/qa/qa/page/project/pipeline/show.rb#L53). diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index c9e60c06732..a42d5e3df1d 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # RSpec metadata for end-to-end tests -This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec-core/docs/metadata/user-defined-metadata) +This is a partial list of the [RSpec metadata](https://rspec.info/features/3-12/rspec-core/metadata/user-defined/) (a.k.a. tags) that are used in our end-to-end tests. <!-- Please keep the tags in alphabetical order --> diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 0d6d7a161c7..bf4d750423d 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -428,6 +428,8 @@ For instructions on how to run these tests using the `gitlab-qa` gem, please ref Tests that are tagged with `:mobile` can be run against specified mobile devices using cloud emulator/simulator services. +These tests run in the [nightly pipeline](https://gitlab.com/gitlab-org/quality/nightly/-/pipelines) in the `ce:remote_mobile_safari` job. + ### How to run mobile tests with Sauce Labs Running directly against an environment like staging is not recommended because Sauce Labs test logs expose credentials. Therefore, it is best practice and the default to use a tunnel. diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index c626a4fa81c..a7a5e7a97d5 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -130,6 +130,7 @@ Adding a delay in API or controller could help reproducing the issue. **Examples:** - [Example 1](https://gitlab.com/gitlab-org/gitlab/-/issues/118612): A test that breaks after some time passed. +- [Example 2](https://gitlab.com/gitlab-org/gitlab/-/issues/403332): A test that breaks in the last day of the month. ### Unstable infrastructure @@ -150,7 +151,7 @@ usually a good idea. ## Quarantined tests -When a test frequently fails in `master`, +When we have a flaky test in `master`, quarantine the test after the first failure and create [a ~"failure::flaky-test" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master). If the test cannot be fixed in a timely fashion, there is an impact on the @@ -158,6 +159,7 @@ productivity of all the developers, so it should be quarantined. There are two w ### RSpec +Add the corresponding category and group labels to issue using [`feature_category` metadata](../feature_categorization/index.md#rspec-examples). For RSpec tests, you can use the `:quarantine` metadata with the issue URL. ```ruby @@ -300,6 +302,12 @@ If a spec hangs, it might be caused by a [bug in Rails](https://github.com/rails - <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81112> - <https://gitlab.com/gitlab-org/gitlab/-/issues/337039> +## Suggestions + +### Split the test file + +It could help to split the large RSpec files in multiple files in order to narrow down the context and identify the problematic tests. + ## Resources - [Flaky Tests: Are You Sure You Want to Rerun Them?](https://semaphoreci.com/blog/2017/04/20/flaky-tests.html) diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 6134e0f9959..a3b580a43fa 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -567,6 +567,53 @@ Example }); ``` +### Testing local-only Apollo queries and mutations + +To add a new query or mutation before it is added to the backend, we can use the `@client` directive. For example: + +```graphql +mutation setActiveBoardItemEE($boardItem: LocalBoardItem, $isIssue: Boolean = true) { + setActiveBoardItem(boardItem: $boardItem) @client { + ...Issue @include(if: $isIssue) + ...EpicDetailed @skip(if: $isIssue) + } +} +``` + +When writing test cases for such calls, we can use resolvers to make sure they are called with the correct parameters. + +For example, when creating the wrapper, we should make sure the resolver is mapped to the query or mutation. +The mutation we are mocking here is `setActiveBoardItem`: + +```javascript +const mockSetActiveBoardItemResolver = jest.fn(); +const mockApollo = createMockApollo([], { + Mutation: { + setActiveBoardItem: mockSetActiveBoardItemResolver, + }, +}); +``` + +In the following code, we must pass four arguments. The second one must be the collection of input variables of the query or mutation mocked. +To test that the mutation is called with the correct parameters: + +```javascript +it('calls setActiveBoardItemMutation on close', async () => { + wrapper.findComponent(GlDrawer).vm.$emit('close'); + + await waitForPromises(); + + expect(mockSetActiveBoardItemResolver).toHaveBeenCalledWith( + {}, + { + boardItem: null, + }, + expect.anything(), + expect.anything(), + ); +}); +``` + ### Jest best practices > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34209) in GitLab 13.2. @@ -819,10 +866,10 @@ often using fixtures to validate correct integration with the backend code. ### Use fixtures -To import a JSON fixture, `import` it using the `test_fixtures` alias. +To import a JSON or HTML fixture, `import` it using the `test_fixtures` alias. ```javascript -import responseBody from 'test_fixtures/some/fixture.json' // loads spec/frontend/fixtures/some/fixture.json +import responseBody from 'test_fixtures/some/fixture.json' // loads tmp/tests/frontend/fixtures-ee/some/fixture.json it('makes a request', () => { axiosMock.onGet(endpoint).reply(200, responseBody); @@ -833,23 +880,6 @@ it('makes a request', () => { }); ``` -For other fixtures, Jest uses `spec/frontend/__helpers__/fixtures.js` to import them in tests. - -The following are examples of tests that work for Jest: - -```javascript -it('uses some HTML element', () => { - loadHTMLFixture('some/page.html'); // loads spec/frontend/fixtures/some/page.html and adds it to the DOM - - const element = document.getElementById('#my-id'); - - // ... - - // Jest does not clean up the DOM automatically - resetHTMLFixture(); -}); -``` - ### Generate fixtures You can find code to generate test fixtures in: diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 7958d6db706..1e5ee9f3003 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -47,8 +47,8 @@ Maintainers can elect to use the [process for merging during broken `master`](ht ## Performance Metrics -On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage, the -`review-performance` job is automatically started: this job does basic +On every Review App child pipeline in the `qa` stage, the +`browser_performance` job is automatically started: this job does basic browser performance testing using a [Sitespeed.io Container](../../ci/testing/browser_performance_testing.md). @@ -227,7 +227,7 @@ Review apps are automatically stopped 2 days after the last deployment thanks to the [Environment auto-stop](../../ci/environments/index.md#stop-an-environment-after-a-certain-time-period) feature. If you need your review app to stay up for a longer time, you can -[pin its environment](../../ci/environments/index.md#override-a-deployments-scheduled-stop-time) or retry the +[pin its environment](../../ci/environments/index.md#override-a-environments-scheduled-stop-date-and-time) or retry the `review-deploy` job to update the "latest deployed at" time. The `review-cleanup` job that automatically runs in scheduled diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 187d30c12be..480a53bbefe 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -55,6 +55,7 @@ records should use stubs/doubles as much as possible. | `lib/` | `spec/lib/` | RSpec | | | `lib/tasks/` | `spec/tasks/` | RSpec | | | `rubocop/` | `spec/rubocop/` | RSpec | | +| `spec/support/` | `spec/support_specs/` | RSpec | | ### Frontend unit tests @@ -65,7 +66,7 @@ that is not directly perceivable by a user. graph RL plain[Plain JavaScript]; Vue[Vue Components]; - feature-flags[Feature Flags]; + feature-flags[Feature flags]; license-checks[License Checks]; plain---Vuex; @@ -149,7 +150,7 @@ Component tests cover the state of a single component that is perceivable by a u graph RL plain[Plain JavaScript]; Vue[Vue Components]; - feature-flags[Feature Flags]; + feature-flags[Feature flags]; license-checks[License Checks]; plain---Vuex; @@ -243,7 +244,7 @@ Their abstraction level is comparable to how a user would interact with the UI. graph RL plain[Plain JavaScript]; Vue[Vue Components]; - feature-flags[Feature Flags]; + feature-flags[Feature flags]; license-checks[License Checks]; plain---Vuex; @@ -366,13 +367,12 @@ See also: - The [RSpec testing guidelines](../testing_guide/best_practices.md#rspec). - System / Feature tests in the [Testing Best Practices](best_practices.md#system--feature-tests). -- [Issue #26159](https://gitlab.com/gitlab-org/gitlab/-/issues/26159) which aims at combining those guidelines with this page. ```mermaid graph RL plain[Plain JavaScript]; Vue[Vue Components]; - feature-flags[Feature Flags]; + feature-flags[Feature flags]; license-checks[License Checks]; plain---Vuex; diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md index 634c337793f..5f500640841 100644 --- a/doc/development/value_stream_analytics.md +++ b/doc/development/value_stream_analytics.md @@ -221,6 +221,7 @@ graph LR; MergeRequestLastBuildFinished --> MergeRequestLabelRemoved; MergeRequestLabelAdded --> MergeRequestLabelAdded; MergeRequestLabelAdded --> MergeRequestLabelRemoved; + MergeRequestLabelAdded --> MergeRequestMerged; MergeRequestLabelRemoved --> MergeRequestLabelAdded; MergeRequestLabelRemoved --> MergeRequestLabelRemoved; ``` diff --git a/doc/development/wikis.md b/doc/development/wikis.md index 2f97931f924..c9ca181575b 100644 --- a/doc/development/wikis.md +++ b/doc/development/wikis.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Editor +stage: Plan +group: Knowledge info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: "GitLab's development guidelines for Wikis" --- @@ -30,7 +30,7 @@ Some notable gems that are used for wikis are: | Component | Description | Gem name | GitLab project | Upstream project | |:--------------|:-----------------------------------------------|:-------------------------------|:--------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------| | `gitlab` | Markup renderer, depends on various other gems | `gitlab-markup` | [`gitlab-org/gitlab-markup`](https://gitlab.com/gitlab-org/gitlab-markup) | [`github/markup`](https://github.com/github/markup) | -| `gitaly-ruby` | Main Gollum library | `gitlab-gollum-lib` | [`gitlab-org/gollum-lib`](https://gitlab.com/gitlab-org/gollum-lib) | [`gollum/gollum-lib`](https://github.com/gollum/gollum-lib) | +| `gollum-lib` | Main Gollum library | `gitlab-gollum-lib` | [`gitlab-org/gollum-lib`](https://gitlab.com/gitlab-org/gollum-lib) | [`gollum/gollum-lib`](https://github.com/gollum/gollum-lib) | | | Gollum Git adapter for Rugged | `gitlab-gollum-rugged_adapter` | [`gitlab-org/gitlab-gollum-rugged_adapter`](https://gitlab.com/gitlab-org/gitlab-gollum-rugged_adapter) | [`gollum/rugged_adapter`](https://github.com/gollum/rugged_adapter) | | | Rugged (also used in Gitaly itself) | `rugged` | - | [`libgit2/rugged`](https://github.com/libgit2/rugged) | diff --git a/doc/development/work_items_widgets.md b/doc/development/work_items_widgets.md index 5b9602595bb..bafceccdafe 100644 --- a/doc/development/work_items_widgets.md +++ b/doc/development/work_items_widgets.md @@ -137,3 +137,27 @@ Each record in the table defines mapping of a widget to a work item type. Curren | 4 | 1 | 1 | 1 | 0 | MyWeight | false | | 5 | 2 | 0 | 1 | 1 | Other Weight | false | | 6 | 3 | 0 | 1 | 1 | Weight | true | + +## Backend architecture + +You can update widgets using custom fine-grained mutations (for example, `WorkItemCreateFromTask`) or as part of the +`workItemCreate` or `workItemUpdate` mutations. + +### Widget callbacks + +When updating the widget together with the work item's mutation, backend code should be implemented using +callback classes that inherit from `Issuable::Callbacks::Base`. These classes have callback methods +that are named similar to ActiveRecord callbacks and behave similarly. + +Callback classes with the same name as the widget are automatically used. For example, `Issuable::Callbacks::Milestone` +is called when the work item has the milestone widget. To use a different class, you can override the `callback_class` +class method. + +#### Available callbacks + +- `after_initialize` is called after the work item is initialized by the `BuildService` and before + the work item is saved by the `CreateService` and `UpdateService`. This callback runs outside the + creation or update database transaction. +- `after_update_commit` is called after the DB update transaction is committed by the `UpdateService`. +- `after_save_commit` is called after the creation or DB update transaction is committed by the + `CreateService` or `UpdateService`. diff --git a/doc/development/workhorse/configuration.md b/doc/development/workhorse/configuration.md index 9fc106b8f04..e69f16c41f8 100644 --- a/doc/development/workhorse/configuration.md +++ b/doc/development/workhorse/configuration.md @@ -296,4 +296,4 @@ GITLAB_CONTINUOUS_PROFILING="stackdriver?service=workhorse&service_version=1.0.1 ## Related topics -- [LabKit monitoring documentation](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go). +- [LabKit monitoring documentation](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go) diff --git a/doc/development/workhorse/index.md b/doc/development/workhorse/index.md index 91795336f78..680dd0c205b 100644 --- a/doc/development/workhorse/index.md +++ b/doc/development/workhorse/index.md @@ -15,9 +15,14 @@ Workhorse itself is not a feature, but there are The canonical source for Workhorse is [`gitlab-org/gitlab/workhorse`](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse). -Prior to [epic #4826](https://gitlab.com/groups/gitlab-org/-/epics/4826), it was -[`gitlab-org/gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse/tree/master), -but that repository is no longer used for development. + +## Learning Resources + +- Workhorse documentation (this page) +- [GitLab Workhorse Deep Dive: Dependency Proxy](https://www.youtube.com/watch?v=9cRd-k0TRqI) _video_ +- [How Dependency Proxy via Workhorse works](https://gitlab.com/gitlab-org/gitlab/-/issues/370235) +- [Workhorse overview for the Dependency Proxy](https://www.youtube.com/watch?v=WmBibT9oQms) +- [Workhorse architecture discussion](https://www.youtube.com/watch?v=QlHdh-yudtw) ## Install Workhorse |
