diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-10-20 12:40:42 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-10-20 12:40:42 +0300 |
| commit | ee664acb356f8123f4f6b00b73c1e1cf0866c7fb (patch) | |
| tree | f8479f94a28f66654c6a4f6fb99bad6b4e86a40e /doc/development | |
| parent | 62f7d5c5b69180e82ae8196b7b429eeffc8e7b4f (diff) | |
Add latest changes from gitlab-org/gitlab@15-5-stable-eev15.5.0-rc42
Diffstat (limited to 'doc/development')
359 files changed, 2847 insertions, 1196 deletions
diff --git a/doc/development/adding_service_component.md b/doc/development/adding_service_component.md index c00d9da5d16..0048b10c9da 100644 --- a/doc/development/adding_service_component.md +++ b/doc/development/adding_service_component.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Adding a new Service Component to GitLab diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 673ec692bf4..d3053a1ca5f 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GraphQL API style guide @@ -436,7 +436,7 @@ By default, fields add `1` to a query's complexity score. This can be overridden [providing a custom `complexity`](https://graphql-ruby.org/queries/complexity_and_depth.html) value for a field. Developers should specify higher complexity for fields that cause more _work_ to be performed -by the server in order to return data. Fields that represent data that can be returned +by the server to return data. Fields that represent data that can be returned with little-to-no _work_, for example in most cases; `id` or `title`, can be given a complexity of `0`. ### `calls_gitaly` diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 7f7d78bb58e..f74ed8db50f 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # API style guide diff --git a/doc/development/application_limits.md b/doc/development/application_limits.md index ceb3c124d1a..edf159a116a 100644 --- a/doc/development/application_limits.md +++ b/doc/development/application_limits.md @@ -1,7 +1,7 @@ --- stage: Systems group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Application limits development diff --git a/doc/development/application_secrets.md b/doc/development/application_secrets.md index 93e43856b34..526cc6c3f61 100644 --- a/doc/development/application_secrets.md +++ b/doc/development/application_secrets.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Application secrets diff --git a/doc/development/application_slis/index.md b/doc/development/application_slis/index.md index cb2eb9b8d90..75dd066680e 100644 --- a/doc/development/application_slis/index.md +++ b/doc/development/application_slis/index.md @@ -1,7 +1,7 @@ --- stage: Platforms group: Scalability -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab Application Service Level Indicators (SLIs) @@ -26,6 +26,8 @@ to be emitted from the rails application: 1. [`rails_request_apdex`](rails_request_apdex.md) 1. `global_search_apdex` +1. `global_search_error_rate` +1. `global_search_indexing_apdex` ## Defining a new SLI diff --git a/doc/development/application_slis/rails_request_apdex.md b/doc/development/application_slis/rails_request_apdex.md index 033bffbf608..2fa9f5f4869 100644 --- a/doc/development/application_slis/rails_request_apdex.md +++ b/doc/development/application_slis/rails_request_apdex.md @@ -1,7 +1,7 @@ --- stage: Platforms group: Scalability -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Rails request Apdex SLI diff --git a/doc/development/approval_rules.md b/doc/development/approval_rules.md index 922a0cd46a2..ce774b1e8f9 100644 --- a/doc/development/approval_rules.md +++ b/doc/development/approval_rules.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Approval Rules development guide **(FREE)** diff --git a/doc/development/appsec/index.md b/doc/development/appsec/index.md deleted file mode 100644 index 8361170c3d2..00000000000 --- a/doc/development/appsec/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../index.md' -remove_date: '2022-09-23' ---- - -This document was moved to [another location](../index.md). - -<!-- This redirect file can be deleted after <2022-09-23>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/architecture.md b/doc/development/architecture.md index a813072a976..a731e661a80 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab architecture overview diff --git a/doc/development/audit_event_guide/index.md b/doc/development/audit_event_guide/index.md index 50d7eeed107..5c8938aa46a 100644 --- a/doc/development/audit_event_guide/index.md +++ b/doc/development/audit_event_guide/index.md @@ -1,7 +1,7 @@ --- stage: Govern group: Compliance -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Audit Event Guide @@ -19,7 +19,7 @@ actions performed across the application. While any events could trigger an Audit Event, not all events should. In general, events that are not good candidates for audit events are: - Not attributable to one specific user. -- Not of specific interest to an admin or owner persona. +- Not of specific interest to an administrator or owner persona. - Are tracking information for product feature adoption. - Are covered in the direction page's discussion on [what is not planned](https://about.gitlab.com/direction/manage/compliance/audit-events/#what-is-not-planned-right-now). diff --git a/doc/development/auto_devops.md b/doc/development/auto_devops.md index b9b8770207e..7a684f64d64 100644 --- a/doc/development/auto_devops.md +++ b/doc/development/auto_devops.md @@ -1,7 +1,7 @@ --- stage: Configure group: Configure -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Auto DevOps development guide **(FREE)** diff --git a/doc/development/avoiding_downtime_in_migrations.md b/doc/development/avoiding_downtime_in_migrations.md deleted file mode 100644 index d4c225b62c5..00000000000 --- a/doc/development/avoiding_downtime_in_migrations.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/avoiding_downtime_in_migrations.md' -remove_date: '2022-07-08' ---- - -This document was moved to [another location](database/avoiding_downtime_in_migrations.md). - -<!-- This redirect file can be deleted after <2022-07-08>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/backend/create_source_code_be/gitaly_touch_points.md b/doc/development/backend/create_source_code_be/gitaly_touch_points.md index 5ac362e709f..6ee2c7f0e51 100644 --- a/doc/development/backend/create_source_code_be/gitaly_touch_points.md +++ b/doc/development/backend/create_source_code_be/gitaly_touch_points.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Source Code - Gitaly Touch Points diff --git a/doc/development/backend/create_source_code_be/index.md b/doc/development/backend/create_source_code_be/index.md index a1322b3fa25..8a1a541fac9 100644 --- a/doc/development/backend/create_source_code_be/index.md +++ b/doc/development/backend/create_source_code_be/index.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Create: Source Code Backend diff --git a/doc/development/backend/create_source_code_be/rest_endpoints.md b/doc/development/backend/create_source_code_be/rest_endpoints.md index dd43bb914c9..2130d76d014 100644 --- a/doc/development/backend/create_source_code_be/rest_endpoints.md +++ b/doc/development/backend/create_source_code_be/rest_endpoints.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Source Code REST endpoints diff --git a/doc/development/backend/ruby_style_guide.md b/doc/development/backend/ruby_style_guide.md index 6ba5b8dd2c5..9b5a68e4292 100644 --- a/doc/development/backend/ruby_style_guide.md +++ b/doc/development/backend/ruby_style_guide.md @@ -2,7 +2,7 @@ type: reference, dev stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Ruby style guide diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md deleted file mode 100644 index 3c9c34bccf8..00000000000 --- a/doc/development/background_migrations.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/background_migrations.md' -remove_date: '2022-07-08' ---- - -This document was moved to [another location](database/background_migrations.md). - -<!-- This redirect file can be deleted after <2022-07-08>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/batched_background_migrations.md b/doc/development/batched_background_migrations.md deleted file mode 100644 index f5f3655944b..00000000000 --- a/doc/development/batched_background_migrations.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/batched_background_migrations.md' -remove_date: '2022-07-26' ---- - -This document was moved to [another location](database/batched_background_migrations.md). - -<!-- This redirect file can be deleted after <2022-07-26>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/build_test_package.md b/doc/development/build_test_package.md index 97dd24fc522..70aa328bf8a 100644 --- a/doc/development/build_test_package.md +++ b/doc/development/build_test_package.md @@ -1,7 +1,7 @@ --- stage: Systems group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Building a package for testing diff --git a/doc/development/bulk_import.md b/doc/development/bulk_import.md index a2620faed35..92236594f8e 100644 --- a/doc/development/bulk_import.md +++ b/doc/development/bulk_import.md @@ -1,7 +1,7 @@ --- stage: Manage group: Import -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab Group Migration diff --git a/doc/development/cached_queries.md b/doc/development/cached_queries.md index 7af4c302e93..fbb857106be 100644 --- a/doc/development/cached_queries.md +++ b/doc/development/cached_queries.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Application Performance -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Cached queries guidelines diff --git a/doc/development/caching.md b/doc/development/caching.md index 5ae6484436e..36fbfc7010e 100644 --- a/doc/development/caching.md +++ b/doc/development/caching.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Caching guidelines diff --git a/doc/development/cascading_settings.md b/doc/development/cascading_settings.md index 56699ff5ffc..22f146c3f5a 100644 --- a/doc/development/cascading_settings.md +++ b/doc/development/cascading_settings.md @@ -1,7 +1,7 @@ --- stage: Manage group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Cascading Settings diff --git a/doc/development/changelog.md b/doc/development/changelog.md index c0296a6d75e..7dc3ae0a80b 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Changelog entries @@ -114,7 +114,7 @@ making it both concise and descriptive, err on the side of descriptive. - **Bad:** Go to a project order. - **Good:** Show a user's starred projects at the top of the "Go to project" - dropdown. + dropdown list. The first example provides no context of where the change was made, or why, or how it benefits the user. @@ -126,9 +126,9 @@ how it benefits the user. Again, the first example is too vague and provides no context. - **Bad:** Fixes and Improves CSS and HTML problems in mini pipeline graph and - builds dropdown. + builds dropdown list. - **Good:** Fix tooltips and hover states in mini pipeline graph and builds - dropdown. + dropdown list. The first example is too focused on implementation details. The user doesn't care that we changed CSS and HTML, they care about the _end result_ of those diff --git a/doc/development/chaos_endpoints.md b/doc/development/chaos_endpoints.md index 838235fd795..008c3700253 100644 --- a/doc/development/chaos_endpoints.md +++ b/doc/development/chaos_endpoints.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Generating chaos in a test GitLab instance @@ -77,7 +77,7 @@ curl "http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10&token=s ## CPU spin -This endpoint attempts to fully utilise a single core, at 100%, for the given period. +This endpoint attempts to fully use a single core, at 100%, for the given period. Depending on your rack server setup, your request may timeout after a predetermined period (normally 60 seconds). @@ -100,7 +100,7 @@ curl "http://localhost:3000/-/chaos/cpu_spin?duration_s=60&token=secret" ## DB spin -This endpoint attempts to fully utilise a single core, and interleave it with DB request, for the given period. +This endpoint attempts to fully use a single core, and interleave it with DB request, for the given period. This endpoint can be used to model yielding execution to another threads when running concurrently. Depending on your rack server setup, your request may timeout after a predetermined period (normally 60 seconds). diff --git a/doc/development/chatops_on_gitlabcom.md b/doc/development/chatops_on_gitlabcom.md index 7309b92c702..16dc17dd229 100644 --- a/doc/development/chatops_on_gitlabcom.md +++ b/doc/development/chatops_on_gitlabcom.md @@ -1,7 +1,7 @@ --- stage: Configure group: Configure -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # ChatOps on GitLab.com diff --git a/doc/development/cicd/cicd_reference_documentation_guide.md b/doc/development/cicd/cicd_reference_documentation_guide.md index 0da1717f53c..8c75e66c33a 100644 --- a/doc/development/cicd/cicd_reference_documentation_guide.md +++ b/doc/development/cicd/cicd_reference_documentation_guide.md @@ -1,7 +1,7 @@ --- stage: Verify group: Pipeline Authoring -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Documenting the `.gitlab-ci.yml` keywords **(FREE)** diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index 7a2dfa01d1e..cd31038ddd1 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -1,7 +1,7 @@ --- stage: Verify group: Pipeline Execution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: index, concepts, howto --- @@ -96,6 +96,9 @@ when transitioning to the `pending` state. This job is responsible for creating a multi-project or child pipeline. The workflow loop starts again from the `CreatePipelineService` every time a downstream pipeline is triggered. +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +You can watch a walkthrough of the architecture in [CI Backend Architectural Walkthrough](https://www.youtube.com/watch?v=ew4BwohS5OY). + ## Job scheduling When a Pipeline is created all its jobs are created at once for all stages, with an initial state of `created`. This makes it possible to visualize the full content of a pipeline. diff --git a/doc/development/cicd/pipeline_wizard.md b/doc/development/cicd/pipeline_wizard.md index 227a49d85db..eb6d1c60bb1 100644 --- a/doc/development/cicd/pipeline_wizard.md +++ b/doc/development/cicd/pipeline_wizard.md @@ -1,7 +1,7 @@ --- stage: none group: Incubation Engineering -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Pipeline Wizard diff --git a/doc/development/cicd/schema.md b/doc/development/cicd/schema.md index ee5b5e4359a..22896594443 100644 --- a/doc/development/cicd/schema.md +++ b/doc/development/cicd/schema.md @@ -1,7 +1,7 @@ --- stage: Verify group: Pipeline Authoring -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: index, howto --- diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index d0c56fb18bc..85ac58e749d 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -1,13 +1,13 @@ --- stage: Verify group: Pipeline Authoring -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: index, concepts, howto --- # Development guide for GitLab CI/CD templates **(FREE)** -This document explains how to develop [GitLab CI/CD templates](../../ci/examples/index.md). +This document explains how to develop [GitLab CI/CD templates](../../ci/examples/index.md#cicd-templates). ## Requirements for CI/CD templates @@ -391,7 +391,7 @@ This is useful information for reviewers to make sure the template is safe to be ### Make sure the new template can be selected in UI Templates located under some directories are also [selectable in the **New file** UI](#template-directories). -When you add a template into one of those directories, make sure that it correctly appears in the dropdown: +When you add a template into one of those directories, make sure that it correctly appears in the dropdown list:  diff --git a/doc/development/code_comments.md b/doc/development/code_comments.md index b1db781b9ee..306f371f306 100644 --- a/doc/development/code_comments.md +++ b/doc/development/code_comments.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Code comments diff --git a/doc/development/code_intelligence/index.md b/doc/development/code_intelligence/index.md index 87697a5e252..f8fb794053d 100644 --- a/doc/development/code_intelligence/index.md +++ b/doc/development/code_intelligence/index.md @@ -1,7 +1,7 @@ --- stage: Create group: Code Review -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Code Intelligence **(FREE)** diff --git a/doc/development/code_review.md b/doc/development/code_review.md index c320540401f..5b745f06d22 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Code Review Guidelines @@ -57,7 +57,7 @@ We make the following assumption with regards to automatically being considered - Team members working on a specific feature (for example, search) are considered domain experts for that feature. We default to assigning reviews to team members with domain expertise. -When a suitable [domain expert](#domain-experts) isn't available, you can choose any team member to review the MR, or simply follow the [Reviewer roulette](#reviewer-roulette) recommendation. +When a suitable [domain expert](#domain-experts) isn't available, you can choose any team member to review the MR, or follow the [Reviewer roulette](#reviewer-roulette) recommendation. To find a domain expert: @@ -75,9 +75,9 @@ NOTE: Reviewer roulette is an internal tool for use on GitLab.com, and not available for use on customer installations. The [Danger bot](dangerbot.md) randomly picks a reviewer and a maintainer for -each area of the codebase that your merge request seems to touch. It only makes -**recommendations** and you should override it if you think someone else is a better -fit! +each area of the codebase that your merge request seems to touch. It makes +**recommendations** for developer reviewers and you should override it if you think someone else is a better +fit. User-facing changes are required to have a UX review, too. Default to the recommended UX reviewer suggested. It picks reviewers and maintainers from the list at the [engineering projects](https://about.gitlab.com/handbook/engineering/projects/) @@ -149,7 +149,7 @@ with [domain expertise](#domain-experts). | `~UX` user-facing changes (*3*) | [Product Designer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_UX). Refer to the [design and user interface guidelines](contributing/design.md) for details. | | Adding a new JavaScript library (*1*) | - [Frontend foundations member](https://about.gitlab.com/direction/ecosystem/foundations/) if the library significantly increases the [bundle size](https://gitlab.com/gitlab-org/frontend/playground/webpack-memory-metrics/-/blob/master/doc/report.md).<br/>- A [legal department member](https://about.gitlab.com/handbook/legal/) if the license used by the new library hasn't been approved for use in GitLab.<br/><br/>More information about license compatibility can be found in our [GitLab Licensing and Compatibility documentation](licensing.md). | | A new dependency or a file system change | - [Distribution team member](https://about.gitlab.com/company/team/). See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/systems/distribution/#how-to-work-with-distribution) for more details.<br/>- For Rubygems, request an [AppSec review](gemfile.md#request-an-appsec-review). | -| `~documentation` changes | [Technical writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages). | +| `~documentation` or `~UI text` changes | [Technical writer](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages). | | Changes to development guidelines | Follow the [review process](development_processes.md#development-guidelines-review) and get the approvals accordingly. | | End-to-end **and** non-end-to-end changes (*4*) | [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors). | | Only End-to-end changes (*4*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors) | [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa). | @@ -208,7 +208,7 @@ See the [test engineering process](https://about.gitlab.com/handbook/engineering 1. I have included changelog trailers, or I have decided that they are not needed. - [Does this MR need a changelog?](changelog.md#what-warrants-a-changelog-entry) 1. I have added/updated documentation or decided that documentation changes are unnecessary for this MR. - - [Is documentation required?](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#when-documentation-is-required) + - [Is documentation required?](https://about.gitlab.com/handbook/product/ux/technical-writing/workflow/#when-documentation-is-required) ##### Security @@ -442,8 +442,7 @@ first time. ### Requesting a review When you are ready to have your merge request reviewed, -you should [request an initial review](../user/project/merge_requests/getting_started.md#reviewer) by selecting a reviewer from your group or team. -However, you can also assign it to any reviewer. The list of reviewers can be found on [Engineering projects](https://about.gitlab.com/handbook/engineering/projects/) page. +you should [request an initial review](../user/project/merge_requests/getting_started.md#reviewer) by selecting a reviewer based on the [approval guidelines](#approval-guidelines). When a merge request has multiple areas for review, it is recommended you specify which area a reviewer should be reviewing, and at which stage (first or second). This will help team members who qualify as a reviewer for multiple areas to know which area they're being requested to review. @@ -662,9 +661,10 @@ Enterprise Edition instance. This has some implications: - Regular migrations run before the new code is running on the instance. - [Post-deployment migrations](database/post_deployment_migrations.md) run _after_ the new code is deployed, when the instance is configured to do that. - - [Background migrations](database/background_migrations.md) run in Sidekiq, and - should only be done for migrations that would take an extreme amount of - time at GitLab.com scale. + - [Batched background migrations](database/batched_background_migrations.md) run in Sidekiq, and + should be used for migrations that + [exceed the post-deployment migration time limit](migration_style_guide.md#how-long-a-migration-should-take) + GitLab.com scale. 1. **Sidekiq workers** [cannot change in a backwards-incompatible way](sidekiq/compatibility_across_updates.md): 1. Sidekiq queues are not drained before a deploy happens, so there are workers in the queue from the previous version of GitLab. diff --git a/doc/development/contributing/community_roles.md b/doc/development/contributing/community_roles.md index 37c3c24a7d1..8aa219d72a1 100644 --- a/doc/development/contributing/community_roles.md +++ b/doc/development/contributing/community_roles.md @@ -1,7 +1,7 @@ --- stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Community members & roles @@ -12,7 +12,7 @@ GitLab community members and their privileges/responsibilities. |-------|------------------|--------------| | Maintainer | Accepts merge requests on several GitLab projects | Added to the [team page](https://about.gitlab.com/company/team/). An expert on code reviews and knows the product/codebase | | Reviewer | Performs code reviews on MRs | Added to the [team page](https://about.gitlab.com/company/team/) | -| Developer |Has access to GitLab internal infrastructure & issues (for example, HR-related) | GitLab employee or a Core Team member (with an NDA) | +| Developer | Has access to GitLab internal infrastructure & issues (for example, HR-related) | GitLab employee or a Core Team member (with an NDA) | | Contributor | Can make contributions to all GitLab public projects | Have a GitLab.com account | -[List of current reviewers/maintainers](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce). +[List of current reviewers/maintainers](https://about.gitlab.com/handbook/engineering/projects/#gitlab). diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index 04915985dc8..9e54b92337a 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -2,7 +2,7 @@ type: reference, dev stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Design and user interface changes @@ -17,11 +17,15 @@ with additions and improvements. ## Merge request reviews -As a merge request (MR) author, you must include _Before_ and _After_ +As a merge request (MR) author, you must: + +- Include _Before_ and _After_ screenshots (or videos) of your changes in the description, as explained in our [MR workflow](merge_request_workflow.md). These screenshots/videos are very helpful for all reviewers and can speed up the review process, especially if the changes are small. +- Attach the ~UX label to any merge request that impacts the user experience. This will enable Product Designers to [review](https://about.gitlab.com/handbook/engineering/ux/product-designer/mr-reviews/#stage-group-mrs/) any user facing changes. +- Assign the Product Designer suggested by Reviewer Roulette as the reviewer of your merge request. The reviewer does not have to be the domain expert unless this is a community contribution. ## Checklist @@ -35,7 +39,7 @@ Check these aspects both when _designing_ and _reviewing_ UI changes. - Use clear and consistent [terminology](https://design.gitlab.com/content/terminology/). - Check grammar and spelling. - Consider help content and follow its [guidelines](https://design.gitlab.com/usability/helping-users/). -- Request review from the [appropriate Technical Writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments), +- Request review from the [appropriate Technical Writer](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments), indicating any specific files or lines they should review, and how to preview or understand the location/context of the text from the user's perspective. @@ -100,7 +104,7 @@ When the design is ready, _before_ starting its implementation: - Share design specifications in the related issue, preferably through a [Figma link](https://help.figma.com/hc/en-us/articles/360040531773-Share-Files-with-anyone-using-Link-Sharing#copy-link) link or [GitLab Designs feature](../../user/project/issues/design_management.md). - See [when you should use each tool](https://about.gitlab.com/handbook/engineering/ux/product-designer/#deliver). + See [when you should use each tool](https://about.gitlab.com/handbook/product/ux/product-designer/#deliver). - Document user flow and states (for example, using [Mermaid flowcharts in Markdown](../../user/markdown.md#mermaid)). - Document animations and transitions. - Document responsive behaviors. diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 2ff51e765a3..cbccd832d78 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -2,7 +2,7 @@ type: reference, dev stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Contribute to GitLab @@ -26,11 +26,6 @@ Throughout this guide you will see references to CE and EE for abbreviation. To get an overview of GitLab community membership, including those that would review or merge your contributions, visit [the community roles page](community_roles.md). -If you want to know how the GitLab [core team](https://about.gitlab.com/community/core-team/) -operates, see [the GitLab contributing process](https://gitlab.com/gitlab-org/gitlab/-/blob/master/PROCESS.md). - -GitLab Inc engineers should refer to the [engineering workflow document](https://about.gitlab.com/handbook/engineering/workflow/). - ## Security vulnerability disclosure Report suspected security vulnerabilities by following the diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index d557319b41f..df337bb2809 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -2,7 +2,7 @@ type: reference, dev stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Issues workflow @@ -373,7 +373,7 @@ below will make it easy to manage this, without unnecessary overhead. which might lead to many hard problems to solve. Changing some text in GitLab is probably 1, adding a new Git Hook maybe 4 or 5, big features 7-9. 1. If something is very large, it should probably be split up in multiple - issues or chunks. You can simply not set the weight of a parent issue and set + issues or chunks. You can not set the weight of a parent issue and set weights to children issues. ## Regression issues @@ -432,7 +432,7 @@ original merge request - or not tracked at all! The overheads of scheduling, and rate of change in the GitLab codebase, mean that the cost of a trivial technical debt issue can quickly exceed the value of tracking it. This generally means we should resolve these in the original merge -request - or simply not create a follow-up issue at all. +request - or not create a follow-up issue at all. For example, a typo in a comment that is being copied between files is worth fixing in the same MR, but not worth creating a follow-up issue for. Renaming a diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 0e9ac569558..b40df01cbe9 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -2,7 +2,7 @@ type: reference, dev stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Merge requests workflow @@ -14,7 +14,7 @@ label, but you are free to contribute to any issue you want. If an issue is marked for the current milestone at any time, even when you are working on it, a GitLab Inc. team member may take over the merge request -in order to ensure the work is finished before the release date. +to ensure the work is finished before the release date. If you want to add a new feature that is not labeled, it is best to first create an issue (if there isn't one already) and leave a comment asking for it diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 2e696cf517b..9d04e1590d0 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -2,7 +2,7 @@ type: reference, dev stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Style guides @@ -41,13 +41,13 @@ We were using Overcommit prior to Lefthook, so you may want to uninstall it firs bundle exec lefthook install ``` -1. Test Lefthook is working by running the Lefthook `prepare-commit-msg` Git hook: +1. Test Lefthook is working by running the Lefthook `pre-push` Git hook: ```shell - bundle exec lefthook run prepare-commit-msg + bundle exec lefthook run pre-push ``` -This should return a fully qualified path command with no other output. +This should return the lefthook version and the list of executable commands with output. ### Lefthook configuration @@ -153,11 +153,37 @@ to add it to our [GitLab Styles](https://gitlab.com/gitlab-org/gitlab-styles) ge If the Cop targets rules that only apply to the main GitLab application, it should be added to [GitLab](https://gitlab.com/gitlab-org/gitlab) instead. +### Cop grace period + +A cop is in a "grace period" if it is enabled and has `Details: grace period` defined in its TODO YAML configuration. + +On the default branch, all of the offenses from cops in the ["grace period"](../rake_tasks.md#run-rubocop-in-graceful-mode) will not fail the RuboCop CI job. The job will notify Slack in the `#f_rubocop` channel when offenses have been silenced in the scheduled pipeline. However, on merge request pipelines, the RuboCop job will fail. + +A grace period can safely be lifted as soon as there are no warnings for 2 weeks in the `#f_rubocop` channel on Slack. + +### Enabling a new cop + +1. Enable the new cop in `.rubocop.yml` (if not already done via [`gitlab-styles`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-styles)). +1. [Generate TODOs for the new cop](../rake_tasks.md#generate-initial-rubocop-todo-list). +1. [Set the new cop to "grace period"](#cop-grace-period). +1. Create an issue to fix TODOs and encourage Community contributions (via ~"good for new contributors" and/or ~"Seeking community contributions"). [See some examples](https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=created_date&state=opened&label_name%5B%5D=good%20for%20new%20contributors&label_name%5B%5D=static%20code%20analysis&first_page_size=20). +1. Create an issue to remove "grace period" after 2 weeks silence in `#f_rubocop` Slack channel. ([See an example](https://gitlab.com/gitlab-org/gitlab/-/issues/374903).) + +### Silenced offenses + +When offenses are silenced for cops in ["grace period"](#cop-grace-period), +the `#f_rubocop` Slack channel receives a notification message every two hours. + +To fix this issue: + +1. Find cops with silenced offenses in the linked CI job. +1. [Generate TODOs](../rake_tasks.md#generate-initial-rubocop-todo-list) for these cops. + #### RuboCop node pattern When creating [node patterns](https://docs.rubocop.org/rubocop-ast/node_pattern.html) to match Ruby's AST, you can use [`scripts/rubocop-parse`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/rubocop-parse) -to display the AST of a Ruby expression, in order to help you create the matcher. +to display the AST of a Ruby expression, to help you create the matcher. See also [!97024](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97024). ### Resolving RuboCop exceptions diff --git a/doc/development/contributing/verify/index.md b/doc/development/contributing/verify/index.md index 09b206d59aa..e94b7583904 100644 --- a/doc/development/contributing/verify/index.md +++ b/doc/development/contributing/verify/index.md @@ -52,7 +52,7 @@ and they serve us and our users well. Some examples of these principles are that - The feedback delivered by GitLab CI/CD and data produced by the platform should be accurate. If a job fails and we notify a user that it was successful, it can have severe negative consequences. -- Feedback needs to be available when a user needs it and data can not disappear unexpectedly when engineers need it. +- Feedback needs to be available when a user needs it and data cannot disappear unexpectedly when engineers need it. - It all doesn't matter if the platform is not secure and we are leaking credentials or secrets. - When a user provides a set of preconditions in a form of CI/CD configuration, the result should be deterministic each time a pipeline runs, because otherwise the platform might not be trustworthy. @@ -62,7 +62,7 @@ are leaking credentials or secrets. ### Measure before you optimize, and make data-informed decisions -It is very difficult to optimize something that you can not measure. How would you +It is very difficult to optimize something that you cannot measure. How would you know if you succeeded, or how significant the success was? If you are working on a performance or reliability improvement, make sure that you measure things before you optimize them. diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index d2b231ebc7c..52503f2d9c8 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Danger bot @@ -13,7 +13,7 @@ Danger is a gem that runs in the CI environment, like any other analysis tool. What sets it apart from (for example, RuboCop) is that it's designed to allow you to easily write arbitrary code to test properties of your code or changes. To this end, it provides a set of common helpers and access to information about what -has actually changed in your environment, then simply runs your code! +has actually changed in your environment, then runs your code! If Danger is asking you to change something about your merge request, it's best just to make the change. If you want to learn how Danger works, or make changes diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md index 4be3296b2bb..1609e00531e 100644 --- a/doc/development/database/add_foreign_key_to_existing_column.md +++ b/doc/development/database/add_foreign_key_to_existing_column.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Add a foreign key constraint to an existing column @@ -126,7 +126,7 @@ Validating the foreign key scans the whole table and makes sure that each relati Fortunately, this does not lock the source table (`users`) while running. NOTE: -When using [background migrations](background_migrations.md), foreign key validation should happen in the next GitLab release. +When using [batched background migrations](batched_background_migrations.md), foreign key validation should happen in the next GitLab release. Migration file for validating the foreign key: diff --git a/doc/development/database/adding_database_indexes.md b/doc/development/database/adding_database_indexes.md index 774703bd54f..040c6780316 100644 --- a/doc/development/database/adding_database_indexes.md +++ b/doc/development/database/adding_database_indexes.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Adding Database Indexes diff --git a/doc/development/database/avoiding_downtime_in_migrations.md b/doc/development/database/avoiding_downtime_in_migrations.md index 79c76b351c8..57f5a66a9ee 100644 --- a/doc/development/database/avoiding_downtime_in_migrations.md +++ b/doc/development/database/avoiding_downtime_in_migrations.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Avoiding downtime in migrations @@ -296,7 +296,7 @@ when migrating a column in a large table (for example, `issues`). Background migrations spread the work / load over a longer time period, without slowing down deployments. -For more information, see [the documentation on cleaning up background migrations](background_migrations.md#cleaning-up). +For more information, see [the documentation on cleaning up batched background migrations](batched_background_migrations.md#cleaning-up). ## Adding Indexes diff --git a/doc/development/database/background_migrations.md b/doc/development/database/background_migrations.md index 9b596eb7379..8e6f29b9eb8 100644 --- a/doc/development/database/background_migrations.md +++ b/doc/development/database/background_migrations.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Background migrations @@ -44,7 +44,7 @@ into this category. ## Isolation -Background migrations must be isolated and can not use application code (for example, +Background migrations must be isolated and cannot use application code (for example, models defined in `app/models` except the `ApplicationRecord` classes). Since these migrations can take a long time to run it's possible for new versions to be deployed while they are still running. @@ -86,7 +86,7 @@ migration classes must be defined in the namespace Scheduling a background migration should be done in a post-deployment migration that includes `Gitlab::Database::MigrationHelpers` -To do so, simply use the following code while +To do so, use the following code while replacing the class name and arguments with whatever values are necessary for your migration: @@ -110,7 +110,7 @@ You also need to make sure that newly created data is either migrated, or saved in both the old and new version upon creation. For complex and time consuming migrations it's best to schedule a background job using an `after_create` hook so this doesn't affect response timings. The same applies to -updates. Removals in turn can be handled by simply defining foreign keys with +updates. Removals in turn can be handled by defining foreign keys with cascading deletes. ### Rescheduling background migrations diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md index 192cd0d3e49..a48a9c42e27 100644 --- a/doc/development/database/batched_background_migrations.md +++ b/doc/development/database/batched_background_migrations.md @@ -2,7 +2,7 @@ type: reference, dev stage: Data Stores group: Database -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" --- # Batched background migrations @@ -41,7 +41,7 @@ into this category. ## Isolation -Batched background migrations must be isolated and can not use application code (for example, +Batched background migrations must be isolated and cannot use application code (for example, models defined in `app/models` except the `ApplicationRecord` classes). Because these migrations can take a long time to run, it's possible for new versions to deploy while the migrations are still running. @@ -532,6 +532,119 @@ for more details. ## Additional tips and strategies +### ChatOps integration + +The batched background migrations framework has ChatOps support. Using ChatOps, GitLab engineers can interact with the batched background migrations present in the system. + +#### List batched background migrations + +To list the batched background migrations in the system, run this command: + +`/chatops run batched_background_migrations list` + +This command supports the following options: + +- Database selection: + - `--database DATABASE_NAME`: Connects to the given database: + - `main`: Uses the main database (default). + - `ci`: Uses the CI database. +- Environment selection: + - `--dev`: Uses the `dev` environment. + - `--staging`: Uses the `staging` environment. + - `--staging_ref`: Uses the `staging_ref` environment. + - `--production` : Uses the `production` environment (default). + +Output example: + + + +NOTE: +ChatOps returns 20 batched background migrations order by `created_at` (DESC). + +#### Monitor the progress and status of a batched background migration + +To see the status and progress of a specific batched background migration, run this command: + +`/chatops run batched_background_migrations status MIGRATION_ID` + +This command supports the following options: + +- Database selection: + - `--database DATABASE_NAME`: Connects to the given database: + - `main`: Uses the main database (default) + - `ci`: Uses the CI database +- Environment selection: + - `--dev`: Uses the `dev` environment. + - `--staging`: Uses the `staging` environment. + - `--staging_ref`: Uses the `staging_ref` environment. + - `--production` : Uses the `production` environment (default). + +Output example: + + + +`Progress` represents the percentage of the background migration that has been completed. + +Definitions of the batched background migration states: + +- **Active:** Either: + - Ready to be picked by the runner. + - Running batched jobs. +- **Finalizing:** Running batched jobs. +- **Failed:** Failed batched background migration. +- **Finished:** Completed batched background migration. +- **Paused:** Not visible to the runner. + +#### Pause a batched background migration + +If you want to pause a batched background migration, you need to run the following command: + +`/chatops run batched_background_migrations pause MIGRATION_ID` + +This command supports the following options: + +- Database selection: + - `--database DATABASE_NAME`: Connects to the given database: + - `main`: Uses the main database (default). + - `ci`: Uses the CI database. +- Environment selection: + - `--dev`: Uses the `dev` environment. + - `--staging`: Uses the `staging` environment. + - `--staging_ref`: Uses the `staging_ref` environment. + - `--production` : Uses the `production` environment (default). + +Output example: + + + +NOTE: +You can pause only `active` batched background migrations. + +#### Resume a batched background migration + +If you want to resume a batched background migration, you need to run the following command: + +`/chatops run batched_background_migrations resume MIGRATION_ID` + +This command supports the following options: + +- Database selection: + - `--database DATABASE_NAME`: Connects to the given database: + - `main`: Uses the main database (default). + - `ci`: Uses the CI database. +- Environment selection: + - `--dev`: Uses the `dev` environment. + - `--staging`: Uses the `staging` environment. + - `--staging_ref`: Uses the `staging_ref` environment. + - `--production` : Uses the `production` environment (default). + +Output example: + + + +NOTE: +You can resume only `active` batched background migrations + ### Viewing failure error logs You can view failures in two ways: @@ -565,3 +678,8 @@ You can view failures in two ways: ON transition_logs.batched_background_migration_job_id = jobs.id WHERE transition_logs.next_status = '2' AND migration.job_class_name = "CLASS_NAME"; ``` + +## Legacy background migrations + +Batched background migrations replaced the [legacy background migrations framework](background_migrations.md). +Check that documentation in reference to any changes involving that framework. diff --git a/doc/development/database/ci_mirrored_tables.md b/doc/development/database/ci_mirrored_tables.md index 1d285e607fa..bf3a744b936 100644 --- a/doc/development/database/ci_mirrored_tables.md +++ b/doc/development/database/ci_mirrored_tables.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # CI mirrored tables diff --git a/doc/development/database/client_side_connection_pool.md b/doc/development/database/client_side_connection_pool.md index 3143391a553..3c9f8e76d89 100644 --- a/doc/development/database/client_side_connection_pool.md +++ b/doc/development/database/client_side_connection_pool.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Client-side connection-pool diff --git a/doc/development/database/constraint_naming_convention.md b/doc/development/database/constraint_naming_convention.md index 2f0b8bf0463..e9e130495e6 100644 --- a/doc/development/database/constraint_naming_convention.md +++ b/doc/development/database/constraint_naming_convention.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Constraints naming conventions diff --git a/doc/development/database/creating_enums.md b/doc/development/database/creating_enums.md index 450cb97d978..73c3f546728 100644 --- a/doc/development/database/creating_enums.md +++ b/doc/development/database/creating_enums.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Creating enums @@ -103,7 +103,7 @@ This looks working as a workaround, however, this approach has some downsides th Therefore, you need to be careful of that the offset doesn't exceed the maximum value of 2 bytes integer. As a conclusion, you should define all of the key/value pairs in FOSS. -For example, you can simply write the following code in the above case: +For example, you can write the following code in the above case: ```ruby class Pipeline < ApplicationRecord diff --git a/doc/development/database/database_debugging.md b/doc/development/database/database_debugging.md index 591e526cc96..4dc6a3bdcfa 100644 --- a/doc/development/database/database_debugging.md +++ b/doc/development/database/database_debugging.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Troubleshooting and debugging the database diff --git a/doc/development/database/database_dictionary.md b/doc/development/database/database_dictionary.md index c330c5e67bd..bd6dbc54316 100644 --- a/doc/development/database/database_dictionary.md +++ b/doc/development/database/database_dictionary.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Database Dictionary diff --git a/doc/development/database/database_lab.md b/doc/development/database/database_lab.md index 1d584a4ec6f..b60091fa37c 100644 --- a/doc/development/database/database_lab.md +++ b/doc/development/database/database_lab.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Database Lab and Postgres.ai diff --git a/doc/development/database/database_migration_pipeline.md b/doc/development/database/database_migration_pipeline.md index 496bd09bf1d..148dc1e94a0 100644 --- a/doc/development/database/database_migration_pipeline.md +++ b/doc/development/database/database_migration_pipeline.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Database migration pipeline diff --git a/doc/development/database/database_query_comments.md b/doc/development/database/database_query_comments.md index 2798071bc06..946947db502 100644 --- a/doc/development/database/database_query_comments.md +++ b/doc/development/database/database_query_comments.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Database query comments with Marginalia diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index a85f399a447..aa92134018d 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Database Reviewer Guidelines diff --git a/doc/development/database/db_dump.md b/doc/development/database/db_dump.md index f2076cbc410..b09c601537c 100644 --- a/doc/development/database/db_dump.md +++ b/doc/development/database/db_dump.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Importing a database dump into a staging environment diff --git a/doc/development/database/dbcheck-migrations-job.md b/doc/development/database/dbcheck-migrations-job.md index 49f8b183272..4ebfb40cd03 100644 --- a/doc/development/database/dbcheck-migrations-job.md +++ b/doc/development/database/dbcheck-migrations-job.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # db:check-migrations job diff --git a/doc/development/database/deleting_migrations.md b/doc/development/database/deleting_migrations.md index 8354cb62d0c..bd03e6c11ae 100644 --- a/doc/development/database/deleting_migrations.md +++ b/doc/development/database/deleting_migrations.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Delete existing migrations diff --git a/doc/development/database/efficient_in_operator_queries.md b/doc/development/database/efficient_in_operator_queries.md index a2481577e8c..fb7ff3c1cc2 100644 --- a/doc/development/database/efficient_in_operator_queries.md +++ b/doc/development/database/efficient_in_operator_queries.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Efficient `IN` operator queries @@ -670,6 +670,64 @@ records_by_id.each do |id, _| end ``` +#### Ordering by `JOIN` columns + +Ordering records by mixed columns where one or more columns are coming from `JOIN` tables +works with limitations. It requires extra configuration (CTE). The trick is to use a +non-materialized CTE to act as a "fake" table which exposes all required columns. + +NOTE: +The query performance might not improve compared to the standard `IN` query. Always +check the query plan. + +Example: order issues by `projects.name, issues.id` within the group hierarchy + +The first step is to create a CTE, where all required columns are collected in the `SELECT` +clause. + +```ruby +cte_query = Issue + .select('issues.id AS id', 'issues.project_id AS project_id', 'projects.name AS projects_name') + .joins(:project) + +cte = Gitlab::SQL::CTE.new(:issue_with_projects, cte_query, materialized: false) +``` + +Custom order object configuration: + +```ruby +order = Gitlab::Pagination::Keyset::Order.build([ + Gitlab::Pagination::Keyset::ColumnOrderDefinition.new( + attribute_name: 'projects_name', + order_expression: Issue.arel_table[:projects_name].asc, + sql_type: 'character varying', + nullable: :nulls_last, + distinct: false + ), + Gitlab::Pagination::Keyset::ColumnOrderDefinition.new( + attribute_name: :id, + order_expression: Issue.arel_table[:id].asc + ) + ]) +``` + +Generate the query: + +```ruby +scope = cte.apply_to(Issue.where({}).reorder(order)) + +opts = { + scope: scope, + array_scope: Project.where(namespace_id: top_level_group.self_and_descendants.select(:id)).select(:id), + array_mapping_scope: -> (id_expression) { Issue.where(Issue.arel_table[:project_id].eq(id_expression)) } +} + +records = Gitlab::Pagination::Keyset::InOperatorOptimization::QueryBuilder + .new(**opts) + .execute + .limit(20) +``` + #### Batch iteration Batch iteration over the records is possible via the keyset `Iterator` class. @@ -1052,6 +1110,6 @@ Performance comparison for the `gitlab-org` group: | Optimized `IN` query | 9783 | 450ms | 22ms | NOTE: -Before taking measurements, the group lookup query was executed separately in order to make +Before taking measurements, the group lookup query was executed separately to make the group data available in the buffer cache. Since it's a frequently called query, it hits many shared buffers during the query execution in the production environment. diff --git a/doc/development/database/filtering_by_label.md b/doc/development/database/filtering_by_label.md index 29b0c75298e..a30322176ba 100644 --- a/doc/development/database/filtering_by_label.md +++ b/doc/development/database/filtering_by_label.md @@ -1,7 +1,7 @@ --- stage: Plan group: Project Management -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Filtering by label diff --git a/doc/development/database/foreign_keys.md b/doc/development/database/foreign_keys.md index 7834e7d53c3..d9506ae614a 100644 --- a/doc/development/database/foreign_keys.md +++ b/doc/development/database/foreign_keys.md @@ -1,10 +1,10 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Foreign Keys & Associations +# Foreign keys and associations When adding an association to a model you must also add a foreign key. For example, say you have the following model: @@ -20,7 +20,7 @@ that data consistency is enforced on database level. Foreign keys also mean that the database can very quickly remove associated data (for example, when removing a user), instead of Rails having to do this. -## Adding Foreign Keys In Migrations +## Adding foreign keys in migrations Foreign keys can be added concurrently using `add_concurrent_foreign_key` as defined in `Gitlab::Database::MigrationHelpers`. See the @@ -31,7 +31,7 @@ you have removed any orphaned rows. The method `add_concurrent_foreign_key` does not take care of this so you must do so manually. See [adding foreign key constraint to an existing column](add_foreign_key_to_existing_column.md). -## Updating Foreign Keys In Migrations +## Updating foreign keys in migrations Sometimes a foreign key constraint must be changed, preserving the column but updating the constraint condition. For example, moving from @@ -45,64 +45,64 @@ To replace a foreign key: 1. [Add the new foreign key without validation](add_foreign_key_to_existing_column.md#prevent-invalid-records) - The name of the foreign key constraint must be changed to add a new - foreign key before removing the old one. + The name of the foreign key constraint must be changed to add a new + foreign key before removing the old one. - ```ruby - class ReplaceFkOnPackagesPackagesProjectId < Gitlab::Database::Migration[2.0] - disable_ddl_transaction! + ```ruby + class ReplaceFkOnPackagesPackagesProjectId < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! - NEW_CONSTRAINT_NAME = 'fk_new' + NEW_CONSTRAINT_NAME = 'fk_new' - def up - add_concurrent_foreign_key(:packages_packages, :projects, column: :project_id, on_delete: :nullify, validate: false, name: NEW_CONSTRAINT_NAME) - end + def up + add_concurrent_foreign_key(:packages_packages, :projects, column: :project_id, on_delete: :nullify, validate: false, name: NEW_CONSTRAINT_NAME) + end - def down - with_lock_retries do - remove_foreign_key_if_exists(:packages_packages, column: :project_id, on_delete: :nullify, name: NEW_CONSTRAINT_NAME) - end - end - end - ``` + def down + with_lock_retries do + remove_foreign_key_if_exists(:packages_packages, column: :project_id, on_delete: :nullify, name: NEW_CONSTRAINT_NAME) + end + end + end + ``` 1. [Validate the new foreign key](add_foreign_key_to_existing_column.md#validate-the-foreign-key) - ```ruby - class ValidateFkNew < Gitlab::Database::Migration[2.0] - NEW_CONSTRAINT_NAME = 'fk_new' + ```ruby + class ValidateFkNew < Gitlab::Database::Migration[2.0] + NEW_CONSTRAINT_NAME = 'fk_new' - # foreign key added in <link to MR or path to migration adding new FK> - def up - validate_foreign_key(:packages_packages, name: NEW_CONSTRAINT_NAME) - end + # foreign key added in <link to MR or path to migration adding new FK> + def up + validate_foreign_key(:packages_packages, name: NEW_CONSTRAINT_NAME) + end - def down - # no-op - end - end - ``` + def down + # no-op + end + end + ``` 1. Remove the old foreign key: - ```ruby - class RemoveFkOld < Gitlab::Database::Migration[2.0] - OLD_CONSTRAINT_NAME = 'fk_old' + ```ruby + class RemoveFkOld < Gitlab::Database::Migration[2.0] + OLD_CONSTRAINT_NAME = 'fk_old' - # new foreign key added in <link to MR or path to migration adding new FK> - # and validated in <link to MR or path to migration validating new FK> - def up - remove_foreign_key_if_exists(:packages_packages, column: :project_id, on_delete: :cascade, name: OLD_CONSTRAINT_NAME) - end + # new foreign key added in <link to MR or path to migration adding new FK> + # and validated in <link to MR or path to migration validating new FK> + def up + remove_foreign_key_if_exists(:packages_packages, column: :project_id, on_delete: :cascade, name: OLD_CONSTRAINT_NAME) + end - def down - # Validation is skipped here, so if rolled back, this will need to be revalidated in a separate migration - add_concurrent_foreign_key(:packages_packages, :projects, column: :project_id, on_delete: :cascade, validate: false, name: OLD_CONSTRAINT_NAME) - end - end - ``` + def down + # Validation is skipped here, so if rolled back, this will need to be revalidated in a separate migration + add_concurrent_foreign_key(:packages_packages, :projects, column: :project_id, on_delete: :cascade, validate: false, name: OLD_CONSTRAINT_NAME) + end + end + ``` -## Cascading Deletes +## Cascading deletes Every foreign key must define an `ON DELETE` clause, and in 99% of the cases this should be set to `CASCADE`. @@ -124,7 +124,7 @@ have a foreign key constraint. So if that spec fails, don't add the column to `IGNORED_FK_COLUMNS`, but instead add the FK constraint, or consider naming it differently. -## Dependent Removals +## Dependent removals Don't define options such as `dependent: :destroy` or `dependent: :delete` when defining an association. Defining these options means Rails handles the @@ -182,9 +182,9 @@ create_table :user_configs, id: false do |t| end ``` -Setting `default: nil` ensures a primary key sequence is not created, and since the primary key +Setting `default: nil` ensures a primary key sequence is not created, and because the primary key automatically gets an index, we set `index: false` to avoid creating a duplicate. -You also need to add the new primary key to the model: +You must also add the new primary key to the model: ```ruby class UserConfig < ActiveRecord::Base diff --git a/doc/development/database/hash_indexes.md b/doc/development/database/hash_indexes.md index 731639b6f06..aba23dbea70 100644 --- a/doc/development/database/hash_indexes.md +++ b/doc/development/database/hash_indexes.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Hash Indexes diff --git a/doc/development/database/img/list_v15_4.png b/doc/development/database/img/list_v15_4.png Binary files differnew file mode 100644 index 00000000000..f5c478321c1 --- /dev/null +++ b/doc/development/database/img/list_v15_4.png diff --git a/doc/development/database/img/pause_v15_4.png b/doc/development/database/img/pause_v15_4.png Binary files differnew file mode 100644 index 00000000000..744a2f13243 --- /dev/null +++ b/doc/development/database/img/pause_v15_4.png diff --git a/doc/development/database/img/resume_v15_4.png b/doc/development/database/img/resume_v15_4.png Binary files differnew file mode 100644 index 00000000000..919fd337595 --- /dev/null +++ b/doc/development/database/img/resume_v15_4.png diff --git a/doc/development/database/img/status_v15_4.png b/doc/development/database/img/status_v15_4.png Binary files differnew file mode 100644 index 00000000000..01396649094 --- /dev/null +++ b/doc/development/database/img/status_v15_4.png diff --git a/doc/development/database/index.md b/doc/development/database/index.md index 8cf9a2eec04..87b1b4a9d87 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Database guides @@ -26,7 +26,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Different types of migrations](../migration_style_guide.md#choose-an-appropriate-migration-type) - [Create a regular migration](../migration_style_guide.md#create-a-regular-schema-migration), including creating new models - [Post-deployment migrations guidelines](post_deployment_migrations.md) and [how to create one](post_deployment_migrations.md#creating-migrations) -- [Background migrations guidelines](background_migrations.md) +- [Legacy Background migrations guidelines](background_migrations.md) - [Batched background migrations guidelines](batched_background_migrations.md) - [Deleting migrations](deleting_migrations.md) - [Running database migrations](database_debugging.md#migration-wrangling) @@ -36,7 +36,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Migrations style guide](../migration_style_guide.md) for creating safe SQL migrations - [Testing Rails migrations](../testing_guide/testing_migrations_guide.md) guide - [Post deployment migrations](post_deployment_migrations.md) -- [Background migrations](background_migrations.md) - [Swapping tables](swapping_tables.md) - [Deleting migrations](deleting_migrations.md) - [SQL guidelines](../sql.md) for working with SQL queries diff --git a/doc/development/database/insert_into_tables_in_batches.md b/doc/development/database/insert_into_tables_in_batches.md index ebed3d16319..78aa793f28b 100644 --- a/doc/development/database/insert_into_tables_in_batches.md +++ b/doc/development/database/insert_into_tables_in_batches.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: "Sometimes it is necessary to store large amounts of records at once, which can be inefficient when iterating collections and performing individual `save`s. With the arrival of `insert_all` in Rails 6, which operates at the row level (that is, using `Hash`es), GitLab has added a set @@ -181,7 +181,7 @@ end ``` You can still save relations that are not `BulkInsertSafe` in this block; they -simply are treated as if you had invoked `save` from outside the block. +are treated as if you had invoked `save` from outside the block. ## Known limitations diff --git a/doc/development/database/iterating_tables_in_batches.md b/doc/development/database/iterating_tables_in_batches.md index 6d7a57ecacb..6357bed8b00 100644 --- a/doc/development/database/iterating_tables_in_batches.md +++ b/doc/development/database/iterating_tables_in_batches.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Iterating tables in batches @@ -114,7 +114,7 @@ falling into an endless loop as described in following When dealing with data migrations the preferred way to iterate over a large volume of data is using `EachBatch`. -A special case of data migration is a [background migration](background_migrations.md#scheduling) +A special case of data migration is a [batched background migration](batched_background_migrations.md) where the actual data modification is executed in a background job. The migration code that determines the data ranges (slices) and schedules the background jobs uses `each_batch`. @@ -196,7 +196,7 @@ value is "excluded". The query looks at the index to get the location of the fiv rows on the disk and read the rows from the table. The returned array is processed in Ruby. The first iteration is done. For the next iteration, the last `id` value is reused from the -previous iteration in order to find out the next end `id` value. +previous iteration to find out the next end `id` value. ```sql SELECT "users"."id" FROM "users" WHERE "users"."id" >= 302 ORDER BY "users"."id" ASC LIMIT 1 OFFSET 5 diff --git a/doc/development/database/keyset_pagination.md b/doc/development/database/keyset_pagination.md index 4aec64b8cce..21bce41012e 100644 --- a/doc/development/database/keyset_pagination.md +++ b/doc/development/database/keyset_pagination.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Keyset pagination diff --git a/doc/development/database/layout_and_access_patterns.md b/doc/development/database/layout_and_access_patterns.md index 99a50b503aa..06ac24c284c 100644 --- a/doc/development/database/layout_and_access_patterns.md +++ b/doc/development/database/layout_and_access_patterns.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Best practices for data layout and access patterns diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index 0af12939629..abf66368548 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Loose foreign keys @@ -311,7 +311,7 @@ end The "`it has loose foreign keys`" shared example can be used to test the presence of the `ON DELETE` trigger and the loose foreign key definitions. -Simply add to the model test file: +Add to the model test file: ```ruby it_behaves_like 'it has loose foreign keys' do diff --git a/doc/development/database/maintenance_operations.md b/doc/development/database/maintenance_operations.md index 85df185c024..bfe18068aab 100644 --- a/doc/development/database/maintenance_operations.md +++ b/doc/development/database/maintenance_operations.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Maintenance operations diff --git a/doc/development/database/migrations_for_multiple_databases.md b/doc/development/database/migrations_for_multiple_databases.md index df9607f5672..b4d2656121b 100644 --- a/doc/development/database/migrations_for_multiple_databases.md +++ b/doc/development/database/migrations_for_multiple_databases.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Migrations for Multiple databases diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index 034a2c2e438..e5b6cfb8866 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Pods -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Multiple Databases diff --git a/doc/development/database/namespaces_storage_statistics.md b/doc/development/database/namespaces_storage_statistics.md index 702129b9ddb..c653cfb145d 100644 --- a/doc/development/database/namespaces_storage_statistics.md +++ b/doc/development/database/namespaces_storage_statistics.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Database case study: Namespaces storage statistics diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md index cd2adc3ca28..dccaff2df00 100644 --- a/doc/development/database/not_null_constraints.md +++ b/doc/development/database/not_null_constraints.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # `NOT NULL` constraints @@ -93,7 +93,7 @@ We only want to enforce the `NOT NULL` constraint without setting a default, as that all epics should have a user-generated description. After checking our production database, we know that there are `epics` with `NULL` descriptions, -so we can not add and validate the constraint in one step. +so we cannot add and validate the constraint in one step. NOTE: Even if we did not have any epic with a `NULL` description, another instance of GitLab could have @@ -202,7 +202,7 @@ end If you have to clean up a nullable column for a [high-traffic table](../migration_style_guide.md#high-traffic-tables) (for example, the `artifacts` in `ci_builds`), your background migration goes on for a while and -it needs an additional [background migration cleaning up](background_migrations.md#cleaning-up) +it needs an additional [batched background migration cleaning up](batched_background_migrations.md#cleaning-up) in the release after adding the data migration. In that rare case you need 3 releases end-to-end: diff --git a/doc/development/database/ordering_table_columns.md b/doc/development/database/ordering_table_columns.md index a16df6a4499..286313fb3ce 100644 --- a/doc/development/database/ordering_table_columns.md +++ b/doc/development/database/ordering_table_columns.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Ordering Table Columns in PostgreSQL diff --git a/doc/development/database/pagination_guidelines.md b/doc/development/database/pagination_guidelines.md index fe2e3b46939..54b315b9dd9 100644 --- a/doc/development/database/pagination_guidelines.md +++ b/doc/development/database/pagination_guidelines.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Pagination guidelines @@ -50,9 +50,9 @@ When we list records on the page we often provide additional filters and differe For the MVC version, consider the following: - Reduce the number of sort options to the minimum. -- Reduce the number of filters (dropdown, search bar) to the minimum. +- Reduce the number of filters (dropdown list, search bar) to the minimum. -To make sorting and pagination efficient, for each sort option we need at least two database indexes (ascending, descending order). If we add filter options (by state or by author), we might need more indexes to maintain good performance. Note that indexes are not free, they can significantly affect the `UPDATE` query timings. +To make sorting and pagination efficient, for each sort option we need at least two database indexes (ascending, descending order). If we add filter options (by state or by author), we might need more indexes to maintain good performance. Indexes are not free, they can significantly affect the `UPDATE` query timings. It's not possible to make all filter and sort combinations performant, so we should try optimizing the performance by usage patterns. @@ -154,7 +154,7 @@ Here we're leveraging the ordered property of the b-tree database index. Values ##### `COUNT(*)` on a large dataset -Kaminari by default executes a count query to determine the number of pages for rendering the page links. Count queries can be quite expensive for a large table. In an unfortunate scenario the queries simply time out. +Kaminari by default executes a count query to determine the number of pages for rendering the page links. Count queries can be quite expensive for a large table. In an unfortunate scenario the queries time out. To work around this, we can run Kaminari without invoking the count SQL query. @@ -264,7 +264,7 @@ Looking at the query execution plan, we can see that this query read only 5 rows ##### No page numbers -Offset pagination provides an easy way to request a specific page. We can simply edit the URL and modify the `page=` URL parameter. Keyset pagination cannot provide page numbers because the paging logic might depend on different columns. +Offset pagination provides an easy way to request a specific page. We can edit the URL and modify the `page=` URL parameter. Keyset pagination cannot provide page numbers because the paging logic might depend on different columns. In the previous example, the column is the `id`, so we might see something like this in the `URL`: diff --git a/doc/development/database/pagination_performance_guidelines.md b/doc/development/database/pagination_performance_guidelines.md index 0fef246f133..0f98b50d95c 100644 --- a/doc/development/database/pagination_performance_guidelines.md +++ b/doc/development/database/pagination_performance_guidelines.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Pagination performance guidelines diff --git a/doc/development/database/polymorphic_associations.md b/doc/development/database/polymorphic_associations.md index ac4dc7720a5..f3c9bf1276f 100644 --- a/doc/development/database/polymorphic_associations.md +++ b/doc/development/database/polymorphic_associations.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Polymorphic Associations diff --git a/doc/development/database/post_deployment_migrations.md b/doc/development/database/post_deployment_migrations.md index 8166fcc8905..0d0047531f9 100644 --- a/doc/development/database/post_deployment_migrations.md +++ b/doc/development/database/post_deployment_migrations.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Post Deployment Migrations diff --git a/doc/development/database/query_count_limits.md b/doc/development/database/query_count_limits.md index a888bbfc6e7..b5fd4395b36 100644 --- a/doc/development/database/query_count_limits.md +++ b/doc/development/database/query_count_limits.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Query Count Limits diff --git a/doc/development/database/query_performance.md b/doc/development/database/query_performance.md index 41dbd08d726..61fd80338fe 100644 --- a/doc/development/database/query_performance.md +++ b/doc/development/database/query_performance.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Query performance guidelines diff --git a/doc/development/database/query_recorder.md b/doc/development/database/query_recorder.md index da5c6c8e6cb..3fc38c10d68 100644 --- a/doc/development/database/query_recorder.md +++ b/doc/development/database/query_recorder.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # QueryRecorder diff --git a/doc/development/database/rename_database_tables.md b/doc/development/database/rename_database_tables.md index d6827cb9e03..641bba1f131 100644 --- a/doc/development/database/rename_database_tables.md +++ b/doc/development/database/rename_database_tables.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Rename table without downtime diff --git a/doc/development/database/serializing_data.md b/doc/development/database/serializing_data.md index 97e6f665484..b25c8d49b64 100644 --- a/doc/development/database/serializing_data.md +++ b/doc/development/database/serializing_data.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Serializing Data diff --git a/doc/development/database/setting_multiple_values.md b/doc/development/database/setting_multiple_values.md index cba15a73430..fb85386785d 100644 --- a/doc/development/database/setting_multiple_values.md +++ b/doc/development/database/setting_multiple_values.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Update multiple database objects diff --git a/doc/development/database/sha1_as_binary.md b/doc/development/database/sha1_as_binary.md index dab9b0fe72e..5ad9d106086 100644 --- a/doc/development/database/sha1_as_binary.md +++ b/doc/development/database/sha1_as_binary.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Storing SHA1 Hashes As Binary diff --git a/doc/development/database/single_table_inheritance.md b/doc/development/database/single_table_inheritance.md index c8d082e8a67..ad0101e1594 100644 --- a/doc/development/database/single_table_inheritance.md +++ b/doc/development/database/single_table_inheritance.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Single Table Inheritance diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index 4b5d1fc8f72..fb005e51902 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Strings and the Text data type @@ -15,7 +15,7 @@ When adding new columns to store strings or other textual information: the `#text ... limit: 100` helper (see below) when creating a table, or by using the `add_text_limit` when altering an existing table. -The standard Rails `text` column type can not be defined with a limit, but we extend `create_table` to +The standard Rails `text` column type cannot be defined with a limit, but we extend `create_table` to add a `limit: 255` option. Outside of `create_table`, `add_text_limit` can be used to add a [check constraint](https://www.postgresql.org/docs/11/ddl-constraints.html) to an already existing column. @@ -50,7 +50,7 @@ For example, consider a migration that creates a table with two text columns, `db/migrate/20200401000001_create_db_guides.rb`: ```ruby -class CreateDbGuides < Gitlab::Database::Migration[1.0] +class CreateDbGuides < Gitlab::Database::Migration[2.0] def change create_table :db_guides do |t| t.bigint :stars, default: 0, null: false @@ -74,7 +74,7 @@ For example, consider a migration that adds a new text column `extended_title` t `db/migrate/20200501000001_add_extended_title_to_sprints.rb`: ```ruby -class AddExtendedTitleToSprints < Gitlab::Database::Migration[1.0] +class AddExtendedTitleToSprints < Gitlab::Database::Migration[2.0] # rubocop:disable Migration/AddLimitToTextColumns # limit is added in 20200501000002_add_text_limit_to_sprints_extended_title @@ -89,7 +89,7 @@ A second migration should follow the first one with a limit added to `extended_t `db/migrate/20200501000002_add_text_limit_to_sprints_extended_title.rb`: ```ruby -class AddTextLimitToSprintsExtendedTitle < Gitlab::Database::Migration[1.0] +class AddTextLimitToSprintsExtendedTitle < Gitlab::Database::Migration[2.0] disable_ddl_transaction! def up @@ -131,7 +131,7 @@ Issues is a pretty busy and large table with more than 25 million rows, so we do other processes that try to access it while running the update. Also, after checking our production database, we know that there are `issues` with more characters in -their title than the 1024 character limit, so we can not add and validate the constraint in one step. +their title than the 1024 character limit, so we cannot add and validate the constraint in one step. NOTE: Even if we did not have any record with a title larger than the provided limit, another @@ -165,7 +165,7 @@ in a post-deployment migration, `db/post_migrate/20200501000001_add_text_limit_migration.rb`: ```ruby -class AddTextLimitMigration < Gitlab::Database::Migration[1.0] +class AddTextLimitMigration < Gitlab::Database::Migration[2.0] disable_ddl_transaction! def up @@ -196,7 +196,7 @@ to add a background migration for the 13.0 milestone (current), `db/post_migrate/20200501000002_schedule_cap_title_length_on_issues.rb`: ```ruby -class ScheduleCapTitleLengthOnIssues < Gitlab::Database::Migration[1.0] +class ScheduleCapTitleLengthOnIssues < Gitlab::Database::Migration[2.0] # Info on how many records will be affected on GitLab.com # time each batch needs to run on average, etc ... BATCH_SIZE = 5000 @@ -207,30 +207,25 @@ class ScheduleCapTitleLengthOnIssues < Gitlab::Database::Migration[1.0] disable_ddl_transaction! - class Issue < ::ApplicationRecord - include EachBatch - - self.table_name = 'issues' - end - def up - queue_background_migration_jobs_by_range_at_intervals( - Issue.where('char_length(title_html) > 1024'), - ISSUES_MIGRATION, - DELAY_INTERVAL, + queue_batched_background_migration( + ISSUES_BACKGROUND_MIGRATION, + :issues, + :id, + job_interval: DELAY_INTERVAL, batch_size: BATCH_SIZE ) end def down - # no-op : the part of the title_html after the limit is lost forever + delete_batched_background_migration(ISSUES_BACKGROUND_MIGRATION, :issues, :id, []) end end ``` To keep this guide short, we skipped the definition of the background migration and only provided a high level example of the post-deployment migration that is used to schedule the batches. -You can find more information on the guide about [background migrations](background_migrations.md) +You can find more information on the guide about [batched background migrations](batched_background_migrations.md) #### Validate the text limit (next release) @@ -241,7 +236,7 @@ helper in a final post-deployment migration, `db/post_migrate/20200601000001_validate_text_limit_migration.rb`: ```ruby -class ValidateTextLimitMigration < Gitlab::Database::Migration[1.0] +class ValidateTextLimitMigration < Gitlab::Database::Migration[2.0] disable_ddl_transaction! def up @@ -260,7 +255,7 @@ Increasing text limits on existing database columns can be safely achieved by fi and then dropping the previous limit: ```ruby -class ChangeMaintainerNoteLimitInCiRunner < Gitlab::Database::Migration[1.0] +class ChangeMaintainerNoteLimitInCiRunner < Gitlab::Database::Migration[2.0] disable_ddl_transaction! def up @@ -278,7 +273,7 @@ end If you have to clean up a text column for a really [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3) (for example, the `artifacts` in `ci_builds`), your background migration goes on for a while and -it needs an additional [background migration cleaning up](background_migrations.md#cleaning-up) +it needs an additional [batched background migration cleaning up](batched_background_migrations.md#cleaning-up) in the release after adding the data migration. In that rare case you need 3 releases end-to-end: diff --git a/doc/development/database/swapping_tables.md b/doc/development/database/swapping_tables.md index efb481ccf35..f659d1408d3 100644 --- a/doc/development/database/swapping_tables.md +++ b/doc/development/database/swapping_tables.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Swapping Tables diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md index 582c988bef9..bf12329473d 100644 --- a/doc/development/database/table_partitioning.md +++ b/doc/development/database/table_partitioning.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Database table partitioning @@ -142,7 +142,7 @@ substantial. Partitioning should only be leveraged if the access patterns of the data support the partitioning strategy, otherwise performance suffers. -## Partitioning a table +## Partitioning a table (Range) Unfortunately, tables can only be partitioned at their creation, making it nontrivial to apply to a busy database. A suite of migration @@ -256,3 +256,217 @@ The final step of the migration makes the partitioned table ready for use by the application. This section will be updated when the migration helper is ready, for now development can be followed in the [Tracking Issue](https://gitlab.com/gitlab-org/gitlab/-/issues/241267). + +## Partitioning a table (List) + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96815) in GitLab 15.4. + +Add the partitioning key column to the table you are partitioning. +Include the partitioning key in the following constraints: + +- The primary key. +- All foreign keys referencing the table to be partitioned. +- All unique constraints. + +### Step 1 - Add partition key + +Add the partitioning key column. For example, in a rails migration: + +```ruby +class AddPartitionNumberForPartitioning < Gitlab::Database::Migration[2.0] + enable_lock_retries! + + TABLE_NAME = :table_name + COLUMN_NAME = :partition_id + DEFAULT_VALUE = 100 + + def change + add_column(TABLE_NAME, COLUMN_NAME, :bigint, default: 100) + end +end +``` + +### Step 2 - Create required indexes + +Add indexes including the partitioning key column. For example, in a rails migration: + +```ruby +class PrepareIndexesForPartitioning < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + TABLE_NAME = :table_name + INDEX_NAME = :index_name + + def up + add_concurrent_index(TABLE_NAME, [:id, :partition_id], unique: true, name: INDEX_NAME) + end + + def down + remove_concurrent_index_by_name(TABLE_NAME, INDEX_NAME) + end +end +``` + +### Step 3 - Swap primary key + +Swap the primary key including the partitioning key column. For example, in a rails migration: + +```ruby +class PreparePrimaryKeyForPartitioning < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + TABLE_NAME = :table_name + PRIMARY_KEY = :primary_key + OLD_INDEX_NAME = :old_index_name + NEW_INDEX_NAME = :new_index_name + + def up + swap_primary_key(TABLE_NAME, PRIMARY_KEY, NEW_INDEX_NAME) + end + + def down + add_concurrent_index(TABLE_NAME, :id, unique: true, name: OLD_INDEX_NAME) + add_concurrent_index(TABLE_NAME, [:id, :partition_id], unique: true, name: NEW_INDEX_NAME) + + unswap_primary_key(TABLE_NAME, PRIMARY_KEY, OLD_INDEX_NAME) + end +end +``` + +NOTE: +Do not forget to set the primary key explicitly in your model as `ActiveRecord` does not support composite primary keys. + +```ruby +class Model < ApplicationRecord + self.primary_key = :id +end +``` + +### Step 4 - Enforce unique constraint + +Enforce unique indexes including the partitioning key column. For example, in a rails migration: + +```ruby +class PrepareUniqueContraintForPartitioning < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + TABLE_NAME = :table_name + OLD_UNIQUE_INDEX_NAME = :index_name_unique + NEW_UNIQUE_INDEX_NAME = :new_index_name + + def up + add_concurrent_index(TABLE_NAME, [:some_column, :partition_id], unique: true, name: NEW_UNIQUE_INDEX_NAME) + + remove_concurrent_index_by_name(TABLE_NAME, OLD_UNIQUE_INDEX_NAME) + end + + def down + add_concurrent_index(TABLE_NAME, :some_column, unique: true, name: OLD_UNIQUE_INDEX_NAME) + + remove_concurrent_index_by_name(TABLE_NAME, NEW_UNIQUE_INDEX_NAME) + end +end +``` + +### Step 5 - Enforce foreign key constraint + +Enforce foreign keys including the partitioning key column. For example, in a rails migration: + +```ruby +class PrepareForeignKeyForPartitioning < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + SOURCE_TABLE_NAME = :source_table_name + TARGET_TABLE_NAME = :target_table_name + COLUMN = :foreign_key_id + TARGET_COLUMN = :id + CONSTRAINT_NAME = :fk_365d1db505_p + PARTITION_COLUMN = :partition_id + + def up + add_concurrent_foreign_key( + SOURCE_TABLE_NAME, + TARGET_TABLE_NAME, + column: [PARTITION_COLUMN, COLUMN], + target_column: [PARTITION_COLUMN, TARGET_COLUMN], + validate: false + name: CONSTRAINT_NAME + ) + + validate_foreign_key(TARGET_TABLE_NAME, CONSTRAINT_NAME) + end + + def down + drop_constraint(TARGET_TABLE_NAME, CONSTRAINT_NAME) + end +end +``` + +### Step 6 - Create parent table and attach existing table as the initial partition + +You can now create the parent table attaching the existing table as the initial +partition by using the following helpers provided by the database team. + +For example, using list partitioning in Rails post migrations: + +```ruby +class PrepareTableConstraintsForListPartitioning < Gitlab::Database::Migration[2.0] + include Gitlab::Database::PartitioningMigrationHelpers::TableManagementHelpers + + disable_ddl_transaction! + + TABLE_NAME = :table_name + PARENT_TABLE_NAME = :p_table_name + FIRST_PARTITION = 100 + PARTITION_COLUMN = :partition_id + + def up + prepare_constraint_for_list_partitioning( + table_name: TABLE_NAME, + partitioning_column: PARTITION_COLUMN, + parent_table_name: PARENT_TABLE_NAME, + initial_partitioning_value: FIRST_PARTITION + ) + end + + def down + revert_preparing_constraint_for_list_partitioning( + table_name: TABLE_NAME, + partitioning_column: PARTITION_COLUMN, + parent_table_name: PARENT_TABLE_NAME, + initial_partitioning_value: FIRST_PARTITION + ) + end +end +``` + +```ruby +class ConvertTableToListPartitioning < Gitlab::Database::Migration[2.0] + include Gitlab::Database::PartitioningMigrationHelpers::TableManagementHelpers + + disable_ddl_transaction! + + TABLE_NAME = :table_name + PARENT_TABLE_NAME = :p_table_name + FIRST_PARTITION = 100 + PARTITION_COLUMN = :partition_id + + def up + convert_table_to_first_list_partition( + table_name: TABLE_NAME, + partitioning_column: PARTITION_COLUMN, + parent_table_name: PARENT_TABLE_NAME, + initial_partitioning_value: FIRST_PARTITION + ) + end + + def down + revert_converting_table_to_first_list_partition( + table_name: TABLE_NAME, + partitioning_column: PARTITION_COLUMN, + parent_table_name: PARENT_TABLE_NAME, + initial_partitioning_value: FIRST_PARTITION + ) + end +end +``` diff --git a/doc/development/database/transaction_guidelines.md b/doc/development/database/transaction_guidelines.md index 1583bbc02c2..26bb6c2ce8f 100644 --- a/doc/development/database/transaction_guidelines.md +++ b/doc/development/database/transaction_guidelines.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Transaction guidelines @@ -139,5 +139,5 @@ end ``` The `ApplicationRecord` class uses a different database connection than the `Ci::Build` records. -The two statements in the transaction block are not part of the transaction and are -rolled back in case something goes wrong. They act as 3rd part calls. +The two statements in the transaction block are not part of the transaction and are not +rolled back in case something goes wrong. They act as third-party calls. diff --git a/doc/development/database/understanding_explain_plans.md b/doc/development/database/understanding_explain_plans.md index c3cb408b35f..fff9d755e9a 100644 --- a/doc/development/database/understanding_explain_plans.md +++ b/doc/development/database/understanding_explain_plans.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Understanding EXPLAIN plans @@ -546,7 +546,7 @@ improve this query, other than _not_ running it at all. What is important here is that while some may recommend to straight up add an index the moment you see a sequential scan, it is _much more important_ to first understand what your query does, how much data it retrieves, and so on. After -all, you can not optimize something you do not understand. +all, you cannot optimize something you do not understand. ### Cardinality and selectivity @@ -719,7 +719,7 @@ and through its [web interface](https://console.postgres.ai/gitlab/joe-instances With Joe Bot you can execute DDL statements (like creating indexes, tables, and columns) and get query plans for `SELECT`, `UPDATE`, and `DELETE` statements. -For example, in order to test new index on a column that is not existing on production yet, you can do the following: +For example, to test new index on a column that is not existing on production yet, you can do the following: Create the column: diff --git a/doc/development/database/verifying_database_capabilities.md b/doc/development/database/verifying_database_capabilities.md index 55347edf4ec..c1dd29082ba 100644 --- a/doc/development/database/verifying_database_capabilities.md +++ b/doc/development/database/verifying_database_capabilities.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Verifying Database Capabilities diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 14d73437c36..58776c5330c 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Database Review Guidelines @@ -240,9 +240,11 @@ Include in the MR description: - Check queries timing (If any): In a single transaction, cumulative query time executed in a migration needs to fit comfortably in `15s` - preferably much less than that - on GitLab.com. - For column removals, make sure the column has been [ignored in a previous release](database/avoiding_downtime_in_migrations.md#dropping-columns) -- Check [background migrations](database/background_migrations.md): +- Check [batched background migrations](database/batched_background_migrations.md): - Establish a time estimate for execution on GitLab.com. For historical purposes, it's highly recommended to include this estimation on the merge request description. + This can be the number of expected batches times the delay interval. + - Manually trigger the [database testing](database/database_migration_pipeline.md) job (`db:gitlabcom-database-testing`) in the `test` stage. - If a single `update` is below than `1s` the query can be placed directly in a regular migration (inside `db/migrate`). - Background migrations are normally used, but not limited to: @@ -253,8 +255,6 @@ Include in the MR description: it's suggested to treat background migrations as [post migrations](migration_style_guide.md#choose-an-appropriate-migration-type): place them in `db/post_migrate` instead of `db/migrate`. - - If a migration [has tracking enabled](database/background_migrations.md#background-jobs-tracking), - ensure `mark_all_as_succeeded` is called even if no work is done. - Check [timing guidelines for migrations](migration_style_guide.md#how-long-a-migration-should-take) - Check migrations are reversible and implement a `#down` method - Check new table migrations: diff --git a/doc/development/deleting_migrations.md b/doc/development/deleting_migrations.md deleted file mode 100644 index 5d5ca431598..00000000000 --- a/doc/development/deleting_migrations.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/deleting_migrations.md' -remove_date: '2022-07-08' ---- - -This document was moved to [another location](database/deleting_migrations.md). - -<!-- This redirect file can be deleted after <2022-07-08>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/dependencies.md b/doc/development/dependencies.md index 329539f0cc2..3b935ceba22 100644 --- a/doc/development/dependencies.md +++ b/doc/development/dependencies.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Dependencies diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index 1e9d3ebda77..a940cd9404c 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Deprecating GitLab features diff --git a/doc/development/developing_with_solargraph.md b/doc/development/developing_with_solargraph.md index d7e41187ace..7b9912a865e 100644 --- a/doc/development/developing_with_solargraph.md +++ b/doc/development/developing_with_solargraph.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Using Solargraph diff --git a/doc/development/development_processes.md b/doc/development/development_processes.md index 27ebe98bc63..10818b749ab 100644 --- a/doc/development/development_processes.md +++ b/doc/development/development_processes.md @@ -1,7 +1,7 @@ --- stage: none group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" --- # Development processes @@ -22,7 +22,7 @@ Must-reads: Complementary reads: -- [GitLab core team & GitLab Inc. contribution process](https://gitlab.com/gitlab-org/gitlab/-/blob/master/PROCESS.md) +- [Contribute to GitLab](contributing/index.md) - [Security process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#security-releases-critical-non-critical-as-a-developer) - [Patch release process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/patch/process.md#process-for-developers) - [Guidelines for implementing Enterprise Edition features](ee_features.md) @@ -56,10 +56,10 @@ group. For example, if you're documenting a new internal API used exclusively by a given group, request an engineering review from one of the group's members. After the engineering review is complete, assign the MR to the -[Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) +[Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) in the modified documentation page's metadata. If the page is not assigned to a specific group, follow the -[Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). +[Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines). #### Broader changes @@ -79,12 +79,12 @@ In these cases, use the following workflow: - [Frontend](https://about.gitlab.com/handbook/engineering/frontend/) - [Backend](https://about.gitlab.com/handbook/engineering/) - [Database](https://about.gitlab.com/handbook/engineering/development/database/) - - [User Experience (UX)](https://about.gitlab.com/handbook/engineering/ux/) + - [User Experience (UX)](https://about.gitlab.com/handbook/product/ux/) - [Security](https://about.gitlab.com/handbook/engineering/security/) - [Quality](https://about.gitlab.com/handbook/engineering/quality/) - [Engineering Productivity](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/) - [Infrastructure](https://about.gitlab.com/handbook/engineering/infrastructure/) - - [Technical Writing](https://about.gitlab.com/handbook/engineering/ux/technical-writing/) + - [Technical Writing](https://about.gitlab.com/handbook/product/ux/technical-writing/) You can skip this step for MRs authored by EMs or Staff Engineers responsible for their area. @@ -100,10 +100,10 @@ In these cases, use the following workflow: @clefelhocz1. 1. After all approvals are complete, assign the MR to the - [Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) + [Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) in the modified documentation page's metadata. If the page is not assigned to a specific group, follow the - [Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). + [Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines). The Technical Writer may ask for additional approvals as previously suggested before merging the MR. ### Reviewer values diff --git a/doc/development/diffs.md b/doc/development/diffs.md index 5f03ba93a4d..b38fcea4f00 100644 --- a/doc/development/diffs.md +++ b/doc/development/diffs.md @@ -1,7 +1,7 @@ --- stage: Create group: Code Review -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Working with diffs @@ -148,7 +148,7 @@ This limit is hardcoded and only applied on GitLab. Diff Viewers, which can be found on `models/diff_viewer/*` are classes used to map metadata about each type of Diff File. It has information whether it's a binary, which partial should be used to render it or which File extensions this class accounts for. -`DiffViewer::Base` validates _blobs_ (old and new versions) content, extension and file type in order to check if it can be rendered. +`DiffViewer::Base` validates _blobs_ (old and new versions) content, extension and file type to check if it can be rendered. ## Merge request diffs against the `HEAD` of the target branch @@ -169,7 +169,7 @@ In order to display an up-to-date diff, in GitLab 12.9 we [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27008) merge request diffs compared against `HEAD` of the target branch: the target branch is artificially merged into the source branch, then the resulting -merge ref is compared to the source branch in order to calculate an accurate +merge ref is compared to the source branch to calculate an accurate diff. Until we complete the epics ["use merge refs for diffs"](https://gitlab.com/groups/gitlab-org/-/epics/854) diff --git a/doc/development/directory_structure.md b/doc/development/directory_structure.md index f909fda897f..27384fa559f 100644 --- a/doc/development/directory_structure.md +++ b/doc/development/directory_structure.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Backend directory structure diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index 9d62f2061ca..0ca10bc93dd 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Distributed Tracing - development guidelines **(FREE)** diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 87d2930dcb5..dae62fea603 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -1,5 +1,5 @@ --- -info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-other-projects-and-subjects. +info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. stage: none group: unassigned description: "GitLab development - how to document features deployed behind feature flags" @@ -34,7 +34,7 @@ Possible version history entries are: > - [Enabled on GitLab.com](issue-link) in GitLab X.X. > - [Enabled on GitLab.com](issue-link) in GitLab X.X. Available to GitLab.com administrators only. > - [Enabled on self-managed](issue-link) in GitLab X.X. -> - [Generally available](issue-link) in GitLab X.Y. [Feature flag `flag_name`](issue-link) removed. +> - [Generally available](issue-link) in GitLab X.Y. Feature flag `flag_name` removed. ``` You can combine entries if they happened in the same release: @@ -115,5 +115,5 @@ And, when the feature is done and fully available to all users: > - Introduced in GitLab 13.7 [with a flag](../../administration/feature_flags.md) named `forti_token_cloud`. Disabled by default. > - [Enabled on self-managed](https://gitlab.com/issue/etc) in GitLab 13.8. > - [Enabled on GitLab.com](https://gitlab.com/issue/etc) in GitLab 13.9. -> - [Generally available](issue-link) in GitLab 14.0. [Feature flag `forti_token_cloud`](issue-link) removed. +> - [Generally available](issue-link) in GitLab 14.0. Feature flag `forti_token_cloud` removed. ``` diff --git a/doc/development/documentation/graphql_styleguide.md b/doc/development/documentation/graphql_styleguide.md index ad19a40a3f5..1bd8b37ff5f 100644 --- a/doc/development/documentation/graphql_styleguide.md +++ b/doc/development/documentation/graphql_styleguide.md @@ -2,7 +2,7 @@ type: reference, dev stage: none group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" description: "Writing styles, markup, formatting, and other standards for GraphQL API's GitLab Documentation." --- diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 73c1874f09e..c52b3b5adae 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -1,7 +1,7 @@ --- stage: none group: Documentation Guidelines -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: Learn how to contribute to GitLab Documentation. --- @@ -94,7 +94,7 @@ belongs to, as well as an information block as described below: ```plaintext To determine the technical writer assigned to the Stage/Group associated with this page, see - https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments + https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments ``` For example: @@ -103,7 +103,7 @@ For example: --- stage: Example Stage group: Example Group -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- ``` @@ -171,7 +171,7 @@ contains an array of groups and their assigned technical writer. This task: To prepare an update to the `CODEOWNERS` file: -1. Update `lib/tasks/gitlab/tw/codeowners.rake` with the latest [TW team assignments](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments). +1. Update `lib/tasks/gitlab/tw/codeowners.rake` with the latest [TW team assignments](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). Make this update in a standalone merge request, as it runs a long pipeline and requires backend maintainer review. Make sure this is merged before you update `CODEOWNERS` in another merge request. @@ -465,7 +465,7 @@ RSpec.describe '<What I am taking screenshots of>', :js do #### Full page screenshots -To take a full page screenshot simply `visit the page` and perform any expectation on real content (to have capybara wait till the page is ready and not take a white screenshot). +To take a full page screenshot, `visit the page` and perform any expectation on real content (to have capybara wait till the page is ready and not take a white screenshot). #### Element screenshot diff --git a/doc/development/documentation/redirects.md b/doc/development/documentation/redirects.md index 1bc697f2878..1118dc97926 100644 --- a/doc/development/documentation/redirects.md +++ b/doc/development/documentation/redirects.md @@ -1,7 +1,7 @@ --- stage: none group: Documentation Guidelines -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: Learn how to contribute to GitLab Documentation. --- diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index 2991c1b3224..dc84f3a08dd 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -1,5 +1,5 @@ --- -info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-other-projects-and-subjects. +info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. stage: none group: unassigned description: 'Writing styles, markup, formatting, and other standards for the GitLab RESTful APIs.' diff --git a/doc/development/documentation/review_apps.md b/doc/development/documentation/review_apps.md index a50efceb307..cb04f0909c1 100644 --- a/doc/development/documentation/review_apps.md +++ b/doc/development/documentation/review_apps.md @@ -1,7 +1,7 @@ --- stage: none group: Documentation Guidelines -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: Learn how documentation review apps work. --- diff --git a/doc/development/documentation/site_architecture/deployment_process.md b/doc/development/documentation/site_architecture/deployment_process.md index 8a9c2e1e8d7..b5c3a59b0eb 100644 --- a/doc/development/documentation/site_architecture/deployment_process.md +++ b/doc/development/documentation/site_architecture/deployment_process.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Documentation deployments diff --git a/doc/development/documentation/site_architecture/folder_structure.md b/doc/development/documentation/site_architecture/folder_structure.md index 7f29d3fba9e..8a1d9b9fee3 100644 --- a/doc/development/documentation/site_architecture/folder_structure.md +++ b/doc/development/documentation/site_architecture/folder_structure.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Folder structure for documentation diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index 05e697869b9..37d10b16fcd 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: "Learn how GitLab docs' global navigation works and how to add new items." --- diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index 44e4aac2756..d4553614fad 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Documentation site architecture diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index bc79bf0fbe2..d629bc0b87e 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -1,5 +1,5 @@ --- -info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-other-projects-and-subjects. +info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. stage: none group: unassigned description: 'Writing styles, markup, formatting, and other standards for GitLab Documentation.' @@ -17,9 +17,9 @@ In addition to this page, the following resources can help you craft and contrib - [Doc contribution guidelines](../index.md) - [Recommended word list](word_list.md) - [Doc style and consistency testing](../testing.md) -- [UI text guidelines](https://design.gitlab.com/content/error-messages/) +- [Guidelines for UI error messages](https://design.gitlab.com/content/error-messages/) - [GitLab Handbook style guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines) -- [Microsoft Style Guide](https://docs.microsoft.com/en-us/style-guide/welcome/) +- [Microsoft Style Guide](https://learn.microsoft.com/en-us/style-guide/welcome/) - [Google Developer Documentation Style Guide](https://developers.google.com/style) - [Recent updates to this guide](https://gitlab.com/dashboard/merge_requests?scope=all&state=merged&label_name[]=tw-style¬[label_name][]=docs%3A%3Afix) @@ -115,7 +115,7 @@ the documentation helps others efficiently accomplish tasks and solve problems. If you have questions when considering, authoring, or editing documentation, ask the Technical Writing team. They're available on Slack in `#docs` or in GitLab by -mentioning [the writer for](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) +mentioning [the writer for](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) the applicable [DevOps stage or group](https://about.gitlab.com/handbook/product/categories/#devops-stages). Otherwise, forge ahead with your best effort. It does not need to be perfect; the team is happy to review and improve upon your content. Review the @@ -333,7 +333,7 @@ When possible, try to avoid acronyms in headings. ### Numbers -When using numbers in text, spell out zero through nine, and use numbers for 10 and greater. For details, see the [Microsoft Style Guide](https://docs.microsoft.com/en-us/style-guide/numbers). +When using numbers in text, spell out zero through nine, and use numbers for 10 and greater. For details, see the [Microsoft Style Guide](https://learn.microsoft.com/en-us/style-guide/numbers). ## Text @@ -465,10 +465,18 @@ When using code block style: ## Lists -- Always start list items with a capital letter, unless they're parameters or - commands that are in backticks, or similar. -- Always leave a blank line before and after a list. -- Begin a line with spaces (not tabs) to denote a [nested sub-item](#nesting-inside-a-list-item). +- Use a period after every sentence, including those that complete an introductory phrase. + Do not use semicolons or commas. +- Majority rules. Use either full sentences or all fragments. Avoid a mix. +- Always start list items with a capital letter. +- Separate the introductory phrase from explanatory text with a colon (`:`). For example: + + ```markdown + You can: + + - Do this thing. + - Do this other thing. + ``` ### Choose between an ordered or unordered list @@ -492,40 +500,27 @@ These things are imported: - Thing 3 ``` -You can choose to introduce either list with a colon, but you do not have to. - -### Markup +### List markup - Use dashes (`-`) for unordered lists instead of asterisks (`*`). -- Prefix `1.` to every item in an ordered list. When rendered, the list items - display with sequential numbering. - -### Punctuation - -- Don't add commas (`,`) or semicolons (`;`) to the ends of list items. -- If a list item is a complete sentence (with a subject and a verb), add a period at the end. -- Majority rules. If the majority of items do not end in a period, do not end any of the items in a period. -- Separate list items from explanatory text with a colon (`:`). For example: - - ```markdown - The list is as follows: - - - First item: this explains the first item. - - Second item: this explains the second item. - ``` +- Start every item in an ordered list with `1.`. When rendered, the list items + are sequential. +- Leave a blank line before and after a list. +- Begin a line with spaces (not tabs) to denote a [nested sub-item](#nesting-inside-a-list-item). ### Nesting inside a list item -It's possible to nest items under a list item, so that they render with the same -indentation as the list item. This can be done with: +You can nest items under a list item, so they render with the same +indentation as the list item. You can do this with: - [Code blocks](#code-blocks) - [Blockquotes](#blockquotes) - [Alert boxes](#alert-boxes) - [Images](#images) +- [Tabs](#tabs) -Items nested in lists should always align with the first character of the list -item. In unordered lists (using `-`), this means two spaces for each level of +Nested items should always align with the first character of the list +item. For unordered lists (using `-`), use two spaces for each level of indentation: ````markdown @@ -555,26 +550,9 @@ For ordered lists, use three spaces for each level of indentation: 1. Ordered list item 1 A line nested using 3 spaces to align with the `O` above. - -1. Ordered list item 2 - - > A quote block that will nest - > inside list item 2. - -1. Ordered list item 3 - - ```plaintext - a code block that nests inside list item 3 - ``` - -1. Ordered list item 4 - -  ```` -You can nest full lists inside other lists using the same rules as above. If you -want to mix types, that's also possible, if you don't mix items at the same -level: +You can nest lists in other lists. ```markdown 1. Ordered list item one. @@ -676,125 +654,66 @@ For other punctuation rules, refer to the [Pajamas Design System Punctuation section](https://design.gitlab.com/content/punctuation/). This is overridden by the [documentation-specific punctuation rules](#punctuation). -### Anchor links +## Links -Headings generate anchor links when rendered. `## This is an example` generates -the anchor `#this-is-an-example`. +Links help the docs adhere to the +[single source of truth](#documentation-is-the-single-source-of-truth-ssot) principle. -NOTE: -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39717) in -GitLab 13.4, [product badges](#product-tier-badges) used in headings aren't -included in the generated anchor links. For example, when you link to -`## This is an example **(FREE)**`, use the anchor `#this-is-an-example`. +### Links within the same repository -Keep in mind that the GitLab user interface links to many documentation pages -and anchor links to take the user to the right spot. When you change -a heading, search `doc/*`, `app/views/*`, and `ee/app/views/*` for the old -anchor. If you do not fix these links, the [`ui-docs-lint` job](../testing.md#ui-link-tests) -in your merge request fails. - -Important: +To link to another page in the same repository, +use a relative file path. For example, `../user/gitlab_com/index.md`. -- Avoid crosslinking documentation to headings unless you need to link to a - specific section of the document. This avoids breaking anchors in the - future in case the heading is changed. -- If possible, avoid changing headings, because they're not only linked internally. - There are various links to GitLab documentation on the internet, such as - tutorials, presentations, StackOverflow posts, and other sources. -- Do not link to `h1` headings. +Use inline link Markdown markup `[Text](https://example.com)`, +rather than reference-style links, like `[Text][identifier]`. -Note that with Kramdown, it's possible to add a custom ID to an HTML element -with Markdown markup, but they don't work in `/help`. Because of this, don't use -this option. +Put the entire link on a single line so that [linters](../testing.md) can find it. -## Links +### Links in separate repositories -Links are important in GitLab documentation. Use links instead of -summarizing to help preserve a [single source of truth](#documentation-is-the-single-source-of-truth-ssot) -in GitLab documentation. - -We include guidance for links in these categories: - -- How to set up [anchor links](#anchor-links) for headings. -- How to set up [criteria](#basic-link-criteria) for configuring a link. -- What to set up when [linking to a `help`](../../documentation/index.md#linking-to-help) - page. -- How to set up [links to internal documentation](#links-to-internal-documentation) - for cross-references. -- How to set up [links to external documentation](#links-to-external-documentation) - for authoritative sources. -- When to use [links requiring permissions](#links-requiring-permissions). -- How to set up a [link to a video](#link-to-video). -- How to [link to specific lines of code](#link-to-specific-lines-of-code) - -### Basic link criteria - -- Use inline link Markdown markup `[Text](https://example.com)`. - It's easier to read, review, and maintain. Do not use `[Text][identifier]` reference-style links. -- Use meaningful anchor text. - For example, instead of writing something like `Read more about merge requests [here](LINK)`, - write `Read more about [merge requests](LINK)`. -- Put the entire link on a single line. Some of our [linters](../testing.md) do not - validate links when split over multiple lines, and incorrect or broken links could - slip through. - -### Links to internal documentation +To link to a page in a different repository, use an absolute URL. +For example, to link from a page in the GitLab repo to the Charts repo, +use a URL like `https://docs.gitlab.com/charts/`. -NOTE: -**Internal** refers to documentation in the same project. When linking to -documentation in separate projects (for example, linking to Omnibus documentation -from GitLab documentation), you must use absolute URLs. +### Anchor links -Do not use absolute URLs like `https://docs.gitlab.com/ee/index.html` to -cross-link to other documentation in the same project. Use relative links to -the file, like `../index.md`. (These are converted to HTML when the site is -rendered.) +Each heading has an anchor link. For example, a topic with the title +`## This is an example` has the anchor `#this-is-an-example`. -Relative linking enables crosslinks to work: +The first topic on a page (the `h1`) has an anchor link, +but do not use it. Link to the page instead. -- in Review Apps, local previews, and `/help`. -- when working on the documentation locally, so you can verify that they work as - early as possible in the process. -- in the GitLab user interface when browsing doc files in their respective - repositories. For example, the links displayed at - `https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/README.md`. +If a heading has a [product tier badge](#product-tier-badges), +do not include it in the anchor link. For example, for the heading +`## This is an example **(FREE)**`, use the anchor `#this-is-an-example`. -To link to internal documentation: +With Kramdown, you can add a custom ID to an HTML element, but these IDs +don't work in `/help`, so you should not use them. -- Use relative links to Markdown files in the same repository. -- Do not use absolute URLs or URLs from `docs.gitlab.com`. -- Use `../` to navigate to higher-level directories. -- Don't prepend `./` to links to files or directories. To link to a file in the - same directory or one of its sub-directories, use the syntax `path/to/file.md`. -- Don't link relative to root. For example, `/ee/user/gitlab_com/index.md`. +#### Changing links and titles - Don't: +When you change a heading, the anchor link changes. To ensure you update +any related links, search these directories: - - `https://docs.gitlab.com/ee/administration/geo/replication/troubleshooting.html` - - `/ee/administration/geo/replication/troubleshooting.md` - - `./troubleshooting.md` +- `doc/*` +- `app/views/*` +- `ee/app/views/*` - Do: `../../geo/replication/troubleshooting.md` +If you do not fix these links, the [`ui-docs-lint` job](../testing.md#ui-link-tests) +in your merge request fails. -- Always add the filename `file.md` at the end of the link with the `.md` - extension, not `.html`. +### Text for links - Don't: +Use descriptive text for links, rather than words like `here` or `this page.` - - `../../merge_requests/` - - `../../issues/tags.html` - - `../../issues/tags.html#stages` +For example, instead of: - Do: +- `For more information, see [this page](LINK).` +- `For more information, go [here](LINK).` - - `../../merge_requests/index.md` - - `../../issues/tags.md` - - `../../issues/tags.md#stages` - - `issues/tags.md` +Use: -NOTE: -Using the Markdown extension is necessary for the [`/help`](../index.md#gitlab-help) -section of GitLab. +- `For more information, see [merge requests](LINK)`. ### Links to external documentation @@ -900,6 +819,28 @@ To open group settings: 1. Expand **General pipelines**. ``` +To open either project or group settings: + +```markdown +1. On the top bar, select **Main menu**, and: + - For a project, select ***Projects** and find your project. + - For a group, select **Groups** and find your group. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +``` + +To create a project: + +```markdown +1. On the top bar, select **Create new... > New project**. +``` + +To create a group: + +```markdown +1. On the top bar, select **Create new... > New group**. +``` + To open the Admin Area: ```markdown @@ -1127,6 +1068,14 @@ Do not use words to describe the icon: - Avoid: `Select **Erase job log** (the trash icon).` - Use instead: `Select **Erase job log** (**{remove}**).` This generates as: Select **Erase job log** (**{remove}**). +When the button doesn't have any hover text, you can describe the icon. +Follow up by creating a +[UX bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Bug) +to add hover text to the button to improve accessibility. + +- Avoid: `Select **{ellipsis_v}**.` +- Use instead: `Select the vertical ellipsis (**{ellipsis_v}**).` This generates as: Select the vertical ellipsis (**{ellipsis_v}**). + ## Videos Adding GitLab YouTube video tutorials to the documentation is highly diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index 029c7389290..65ad8dea688 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -1,7 +1,7 @@ --- stage: none group: Style Guide -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: 'Writing styles, markup, formatting, and other standards for GitLab Documentation.' --- @@ -17,7 +17,7 @@ recommends these word choices. In addition: For guidance not on this page, we defer to these style guides: -- [Microsoft Style Guide](https://docs.microsoft.com/en-us/style-guide/welcome/) +- [Microsoft Style Guide](https://learn.microsoft.com/en-us/style-guide/welcome/) - [Google Developer Documentation Style Guide](https://developers.google.com/style) <!-- vale off --> @@ -125,7 +125,7 @@ Instead of: - This feature enables users to add files to their repository. This phrasing is more active and is from the user perspective, rather than the person who implemented the feature. -[View details in the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/a/allow-allows). +[View details in the Microsoft style guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/a/allow-allows). ## Alpha @@ -141,7 +141,7 @@ Instead of **and/or**, use **or** or rewrite the sentence to spell out both opti ## and so on Do not use **and so on**. Instead, be more specific. For details, see -[the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/a/and-so-on). +[the Microsoft style guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/a/and-so-on). ## area @@ -218,7 +218,7 @@ Instead of: ## cannot, can not -Use **cannot** instead of **can not**. You can also use **can't**. +Use **cannot** instead of **can not**. See also [contractions](index.md#contractions). @@ -316,7 +316,7 @@ Do not use **Developer permissions**. A user who is assigned the Developer role ## disable -See [the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/d/disable-disabled) for guidance on **disable**. +See [the Microsoft style guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/d/disable-disabled) for guidance on **disable**. Use **inactive** or **off** instead. ([Vale](../testing.md#vale) rule: [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml)) ## disallow @@ -363,9 +363,13 @@ Do not use Latin abbreviations. Use **for example**, **such as**, **for instance Do not use **e-mail** with a hyphen. When plural, use **emails** or **email messages**. ([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) +## emojis + +Use **emojis** to refer to the plural form of **emoji**. + ## enable -See [the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/e/enable-enables) for guidance on **enable**. +See [the Microsoft style guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/e/enable-enables) for guidance on **enable**. Use **active** or **on** instead. ([Vale](../testing.md#vale) rule: [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml)) ## enter @@ -509,13 +513,16 @@ For example, **Snowplow Guide**. Instead, speak about the feature itself, and ho When writing about the Guest role: - Use a capital **G**. -- Do not use bold. -- Do not use the phrase, **if you are a guest** to mean someone who is assigned the Guest - role. Instead, write it out. For example, **if you are assigned the Guest role**. -- To describe a situation where the Guest role is the minimum required: +- Write it out: + - Use: if you are assigned the Guest role + - Instead of: if you are a guest + +- When the Guest role is the minimum required role: - Use: at least the Guest role - Instead of: the Guest role or higher +Do not use bold. + Do not use **Guest permissions**. A user who is assigned the Guest role has a set of associated permissions. ## handy @@ -656,13 +663,16 @@ Instead of: When writing about the Maintainer role: - Use a capital **M**. -- Do not use bold. -- Do not use the phrase, **if you are a maintainer** to mean someone who is assigned the Maintainer - role. Instead, write it out. For example, **if you are assigned the Maintainer role**. -- To describe a situation where the Maintainer role is the minimum required: +- Write it out. + - Use: if you are assigned the Maintainer role + - Instead of: if you are a maintainer + +- When the Maintainer role is the minimum required role: - Use: at least the Maintainer role - Instead of: the Maintainer role or higher +Do not use bold. + Do not use **Maintainer permissions**. A user who is assigned the Maintainer role has a set of associated permissions. ## mankind @@ -796,11 +806,14 @@ For example, a log file might overwrite a log file of the same name. When writing about the Owner role: - Use a capital **O**. -- Do not use bold. -- Do not use the phrase, **if you are an owner** to mean someone who is assigned the Owner - role. Instead, write it out. For example, **if you are assigned the Owner role**. +- Write it out. + - Use: if you are assigned the Owner role + - Instead of: if you are an owner +Do not use bold. + Do not use **Owner permissions**. A user who is assigned the Owner role has a set of associated permissions. +An Owner is the highest role a user can have. ## Package Registry @@ -818,7 +831,7 @@ Use lowercase for **personal access token**. ## please -Do not use **please**. For details, see the [Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). +Do not use **please**. For details, see the [Microsoft style guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). ## press @@ -851,13 +864,16 @@ Use **register** instead of **sign up** when talking about creating an account. When writing about the Reporter role: - Use a capital **R**. -- Do not use bold. -- Do not use the phrase, **if you are a reporter** to mean someone who is assigned the Reporter - role. Instead, write it out. For example, **if you are assigned the Reporter role**. -- To describe a situation where the Reporter role is the minimum required: +- Write it out. + - Use: if you are assigned the Reporter role + - Instead of: if you are a reporter + +- When the Reporter role is the minimum required role: - Use: at least the Reporter role - Instead of: the Reporter role or higher +Do not use bold. + Do not use **Reporter permissions**. A user who is assigned the Reporter role has a set of associated permissions. ## Repository Mirroring diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index 59a078bdec0..c801bb9f877 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -1,7 +1,7 @@ --- stage: none group: Documentation Guidelines -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: Learn how to contribute to GitLab Documentation. --- diff --git a/doc/development/documentation/topic_types/concept.md b/doc/development/documentation/topic_types/concept.md index a20bb93a97f..dfd003b642d 100644 --- a/doc/development/documentation/topic_types/concept.md +++ b/doc/development/documentation/topic_types/concept.md @@ -1,7 +1,7 @@ --- stage: none group: Style Guide -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Concept topic type diff --git a/doc/development/documentation/topic_types/index.md b/doc/development/documentation/topic_types/index.md index af3e66fe87a..8403fd26517 100644 --- a/doc/development/documentation/topic_types/index.md +++ b/doc/development/documentation/topic_types/index.md @@ -1,7 +1,7 @@ --- stage: none group: Style Guide -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Documentation topic types (CTRT) diff --git a/doc/development/documentation/topic_types/reference.md b/doc/development/documentation/topic_types/reference.md index 42f4f5f6f94..e7ee8b20925 100644 --- a/doc/development/documentation/topic_types/reference.md +++ b/doc/development/documentation/topic_types/reference.md @@ -1,7 +1,7 @@ --- stage: none group: Style Guide -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Reference topic type diff --git a/doc/development/documentation/topic_types/task.md b/doc/development/documentation/topic_types/task.md index 3488cb90cf9..60508fbf6ee 100644 --- a/doc/development/documentation/topic_types/task.md +++ b/doc/development/documentation/topic_types/task.md @@ -1,7 +1,7 @@ --- stage: none group: Style Guide -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Task topic type diff --git a/doc/development/documentation/topic_types/troubleshooting.md b/doc/development/documentation/topic_types/troubleshooting.md index 35187bd892e..e2136de2e06 100644 --- a/doc/development/documentation/topic_types/troubleshooting.md +++ b/doc/development/documentation/topic_types/troubleshooting.md @@ -1,7 +1,7 @@ --- stage: none group: Style Guide -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Troubleshooting topic type diff --git a/doc/development/documentation/versions.md b/doc/development/documentation/versions.md index 85733603cfe..12381b84c2c 100644 --- a/doc/development/documentation/versions.md +++ b/doc/development/documentation/versions.md @@ -1,5 +1,5 @@ --- -info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-other-projects-and-subjects. +info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. stage: none group: unassigned description: 'Writing styles, markup, formatting, and other standards for GitLab Documentation.' @@ -139,7 +139,7 @@ To remove a page: --- stage: Data Stores group: Global Search - info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments + info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments remove_date: '2022-08-02' redirect_to: '../newpath/to/file/index.md' --- @@ -154,7 +154,7 @@ To remove a page: 1. Remove the page's entry from the global navigation by editing [`navigation.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/content/_data/navigation.yaml) in `gitlab-docs`. This content is removed from the documentation as part of the Technical Writing team's -[regularly scheduled tasks](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#regularly-scheduled-tasks). +[regularly scheduled tasks](https://about.gitlab.com/handbook/product/ux/technical-writing/#regularly-scheduled-tasks). ### Remove a topic @@ -179,7 +179,7 @@ To remove a topic: ``` This content is removed from the documentation as part of the Technical Writing team's -[regularly scheduled tasks](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#regularly-scheduled-tasks). +[regularly scheduled tasks](https://about.gitlab.com/handbook/product/ux/technical-writing/#regularly-scheduled-tasks). ## Which versions are removed diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index fb43a2e995a..85d3d5e9cfc 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # How to update GitLab documentation @@ -13,7 +13,7 @@ Anyone can contribute to the GitLab documentation! You can create a merge reques accomplish their work with GitLab. If you are working on a feature or enhancement, use the -[feature workflow process described in the GitLab Handbook](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#for-a-product-change). +[feature workflow process described in the GitLab Handbook](https://about.gitlab.com/handbook/product/ux/technical-writing/workflow/#for-a-product-change). ## How to update the docs @@ -54,7 +54,7 @@ Ask for help from the Technical Writing team if you: To identify someone who can help you: 1. Locate the Technical Writer for the relevant - [DevOps stage group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments). + [DevOps stage group](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). 1. Either: - If urgent help is required, directly assign the Technical Writer in the issue or in the merge request. - If non-urgent help is required, ping the Technical Writer in the issue or merge request. @@ -86,7 +86,7 @@ merge documentation changes. Maintainers must make a good-faith effort to ensure - Meets the [Documentation Guidelines](index.md) and [Style Guide](styleguide/index.md). If the author or reviewer has any questions, they can mention the writer who is assigned to the relevant -[DevOps stage group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments). +[DevOps stage group](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). The process involves the following: @@ -96,7 +96,7 @@ The process involves the following: - Technical Writer (Optional). If not completed for a merge request before merging, must be scheduled post-merge. Schedule post-merge reviews only if an urgent merge is required. To request a: - Pre-merge review, assign the Technical Writer listed for the applicable - [DevOps stage group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments). + [DevOps stage group](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). - Post-merge review, see [Post-merge reviews](#post-merge-reviews). - Maintainer. For merge requests, Maintainers: - Can always request any of the above reviews. diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 869cb0bab0a..2516196d2e0 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Guidelines for implementing Enterprise Edition features @@ -72,7 +72,7 @@ To guard your licensed feature: ``` 1. Optional. If your global feature is also available to namespaces with a paid plan, combine two -feature identifiers to allow both admins and group users. For example: +feature identifiers to allow both administrators and group users. For example: ```ruby License.feature_available?(:my_feature_name) || group.licensed_feature_available?(:my_feature_name_for_namespace) # Both admins and group members can see this EE feature @@ -176,12 +176,19 @@ This works because for every path that is present in CE's eager-load/auto-load paths, we add the same `ee/`-prepended path in [`config/application.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/925d3d4ebc7a2c72964ce97623ae41b8af12538d/config/application.rb#L42-52). This also applies to views. -#### Testing EE-only features +#### Testing EE-only backend features To test an EE class that doesn't exist in CE, create the spec file as you normally would in the `ee/spec` directory, but without the second `ee/` subdirectory. For example, a class `ee/app/models/vulnerability.rb` would have its tests in `ee/spec/models/vulnerability_spec.rb`. +By default, licensed features are disabled while specs are running. To effectively test your feature +you must explicitly enable the feature using the `stub_licensed_features` helper, for example: + +```ruby + stub_licensed_features(my_awesome_feature_name: true) +``` + ### Extend CE features with EE backend code For features that build on existing CE features, write a module in the `EE` @@ -817,7 +824,7 @@ end Sometimes we need EE-specific behavior in some of the APIs. Normally we could use EE methods to override CE methods, however API routes are not methods and -therefore can't be simply overridden. We need to extract them into a standalone +therefore cannot be overridden. We need to extract them into a standalone method, or introduce some "hooks" where we could inject behavior in the CE route. Something like this: @@ -868,8 +875,8 @@ end #### EE `route_setting` -It's very hard to extend this in an EE module, and this is simply storing -some meta-data for a particular route. Given that, we could simply leave the +It's very hard to extend this in an EE module, and this is storing +some meta-data for a particular route. Given that, we could leave the EE `route_setting` in CE as it doesn't hurt and we don't use those meta-data in CE. @@ -1047,7 +1054,7 @@ FactoryBot.define do end ``` -## Separate of EE code in the frontend +## Separation of EE code in the frontend To separate EE-specific JS-files, move the files into an `ee` folder. @@ -1089,10 +1096,13 @@ ee/app/assets/javascripts/ee_only_feature/index.js Feature guarding `licensed_feature_available?` and `License.feature_available?` typical occurs in the controller, as described in the [backend guide](#ee-only-features). -#### Test EE-only features +#### Testing EE-only frontend features Add your EE tests to `ee/spec/frontend/` following the same directory structure you use for CE. +Check the note under [Testing EE-only backend features](#testing-ee-only-backend-features) regarding +enabling licensed features. + ### Extend CE features with EE frontend code Use the [`push_licensed_feature`](#guard-your-ee-feature) to guard frontend features that extend @@ -1406,5 +1416,5 @@ to avoid conflicts during CE to EE merge. ### GitLab-svgs Conflicts in `app/assets/images/icons.json` or `app/assets/images/icons.svg` can -be resolved simply by regenerating those assets with +be resolved by regenerating those assets with [`yarn run svg`](https://gitlab.com/gitlab-org/gitlab-svgs). diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index b3996e16fa1..ab2d241a781 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Global Search -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Elasticsearch knowledge **(PREMIUM SELF)** @@ -64,7 +64,7 @@ Please see the `sha_tokenizer` explanation later below for an example. Used when indexing a blob's filename and content. Uses the `whitespace` tokenizer and the filters: [`code`](#code), `lowercase`, and `asciifolding` -The `whitespace` tokenizer was selected in order to have more control over how tokens are split. For example the string `Foo::bar(4)` needs to generate tokens like `Foo` and `bar(4)` in order to be properly searched. +The `whitespace` tokenizer was selected to have more control over how tokens are split. For example the string `Foo::bar(4)` needs to generate tokens like `Foo` and `bar(4)` to be properly searched. Please see the `code` filter for an explanation on how tokens are split. @@ -94,7 +94,7 @@ Example: #### `path_tokenizer` -This is a custom tokenizer that uses the [`path_hierarchy` tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pathhierarchy-tokenizer.html) with `reverse: true` in order to allow searches to find paths no matter how much or how little of the path is given as input. +This is a custom tokenizer that uses the [`path_hierarchy` tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pathhierarchy-tokenizer.html) with `reverse: true` to allow searches to find paths no matter how much or how little of the path is given as input. Example: @@ -461,8 +461,8 @@ _from "Disk-based Shard Allocation | Elasticsearch Reference" [5.6](https://www. The use of Elasticsearch in GitLab is only ever as a secondary data store. This means that all of the data stored in Elasticsearch can always be derived again from other data sources, specifically PostgreSQL and Gitaly. Therefore if -the Elasticsearch data store is ever corrupted for whatever reason you can -simply reindex everything from scratch. +the Elasticsearch data store is ever corrupted for whatever reason you can reindex +everything from scratch. If your Elasticsearch index is incredibly large it may be too time consuming or cause too much downtime to reindex from scratch. There aren't any built in diff --git a/doc/development/emails.md b/doc/development/emails.md index c997916aa21..2d03d8bca2f 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Working with email in development @@ -176,7 +176,7 @@ in the Helm Chart configuration rather than the `Gemfile`. #### Preserving backwards compatibility -Removing the `Gemfile` would break incoming e-mail processing for source +Removing the `Gemfile` would break incoming email processing for source installs. For now, source installs are advised to upgrade manually to the version specified in Omnibus and run `bin/mail_room` directly as done with Omnibus. diff --git a/doc/development/event_store.md b/doc/development/event_store.md index 37035083e23..b9200d3be25 100644 --- a/doc/development/event_store.md +++ b/doc/development/event_store.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab EventStore diff --git a/doc/development/experiment_guide/experiment_code_reviews.md b/doc/development/experiment_guide/experiment_code_reviews.md index 07bc0f59463..7242acfbdcf 100644 --- a/doc/development/experiment_guide/experiment_code_reviews.md +++ b/doc/development/experiment_guide/experiment_code_reviews.md @@ -1,7 +1,7 @@ --- stage: Growth group: Acquisition -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Experiment code reviews diff --git a/doc/development/experiment_guide/experiment_rollout.md b/doc/development/experiment_guide/experiment_rollout.md index e68520f7812..b2e1b1168d3 100644 --- a/doc/development/experiment_guide/experiment_rollout.md +++ b/doc/development/experiment_guide/experiment_rollout.md @@ -1,7 +1,7 @@ --- stage: Growth group: Acquisition -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Experiment rollouts and feature flags diff --git a/doc/development/experiment_guide/gitlab_experiment.md b/doc/development/experiment_guide/gitlab_experiment.md deleted file mode 100644 index 5ddbe9b3de9..00000000000 --- a/doc/development/experiment_guide/gitlab_experiment.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2022-08-05' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2022-08-05. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/experiment_guide/implementing_experiments.md b/doc/development/experiment_guide/implementing_experiments.md index 19200d48637..e1407473731 100644 --- a/doc/development/experiment_guide/implementing_experiments.md +++ b/doc/development/experiment_guide/implementing_experiments.md @@ -1,7 +1,7 @@ --- stage: Growth group: Acquisition -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Implementing an A/B/n experiment @@ -136,7 +136,7 @@ somewhat abstract and hard to understand initially, but this approach enables us communicate about experiments as something that's wider than just user behavior. NOTE: -Using `actor:` utilizes cookies if the `current_user` is nil. If you don't need +Using `actor:` uses cookies if the `current_user` is nil. If you don't need cookies though - meaning that the exposed functionality would only be visible to signed in users - `{ user: current_user }` would be just as effective. @@ -318,7 +318,7 @@ Given that we've defined a class for our experiment, and have defined the varian The first way is by running the experiment. Assuming the experiment has been run, it surfaces in the client layer without having to do anything special. -The second way doesn't run the experiment and is intended to be used if the experiment must only surface in the client layer. To accomplish this we can `.publish` the experiment. This does not run any logic, but does surface the experiment details in the client layer so they can be utilized there. +The second way doesn't run the experiment and is intended to be used if the experiment must only surface in the client layer. To accomplish this we can `.publish` the experiment. This does not run any logic, but does surface the experiment details in the client layer so they can be used there. An example might be to publish an experiment in a `before_action` in a controller. Assuming we've defined the `PillColorExperiment` class, like we have above, we can surface it to the client by publishing it instead of running it: diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index 500a19fe1ad..e185cd702a4 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -1,7 +1,7 @@ --- stage: Growth group: Acquisition -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Experiment Guide diff --git a/doc/development/experiment_guide/testing_experiments.md b/doc/development/experiment_guide/testing_experiments.md index 78a5d606490..2e88029a27f 100644 --- a/doc/development/experiment_guide/testing_experiments.md +++ b/doc/development/experiment_guide/testing_experiments.md @@ -1,7 +1,7 @@ --- stage: Growth group: Acquisition -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Testing experiments diff --git a/doc/development/export_csv.md b/doc/development/export_csv.md index 29e80f676da..5d35c880ffd 100644 --- a/doc/development/export_csv.md +++ b/doc/development/export_csv.md @@ -1,7 +1,7 @@ --- stage: Manage group: Import -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Export to CSV diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index bdd6c5d6e84..67166a93cb4 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Accessibility @@ -68,9 +68,9 @@ We should ensure that: To provide markup with accessible names, ensure every: -- `input` has an associated `label`. -- `button` and `a` have child text, or `aria-label` when child text isn't present, such as for an icon button with no content. -- `img` has an `alt` attribute. +- input has an [associated `label`](#examples-of-providing-accessible-names). +- button and link have [visible text](#buttons-and-links-with-descriptive-accessible-names), or `aria-label` when there is no visible text, such as for an icon button with no content. +- image has an [`alt` attribute](#images-with-accessible-names). - `fieldset` has `legend` as its first child. - `figure` has `figcaption` as its first child. - `table` has `caption` as its first child. @@ -100,12 +100,12 @@ Text input examples: ```html <!-- Input with label --> <gl-form-group :label="__('Issue title')" label-for="issue-title"> - <gl-form-input id="issue-title" v-model="title" name="title" /> + <gl-form-input id="issue-title" v-model="title" /> </gl-form-group> <!-- Input with hidden label --> -<gl-form-group :label="__('Issue title')" label-for="issue-title" :label-sr-only="true"> - <gl-form-input id="issue-title" v-model="title" name="title" /> +<gl-form-group :label="__('Issue title')" label-for="issue-title" label-sr-only> + <gl-form-input id="issue-title" v-model="title" /> </gl-form-group> ``` @@ -114,12 +114,12 @@ Textarea examples: ```html <!-- Textarea with label --> <gl-form-group :label="__('Issue description')" label-for="issue-description"> - <gl-form-textarea id="issue-description" v-model="description" name="description" /> + <gl-form-textarea id="issue-description" v-model="description" /> </gl-form-group> <!-- Textarea with hidden label --> -<gl-form-group :label="__('Issue description')" label-for="issue-description" :label-sr-only="true"> - <gl-form-textarea id="issue-description" v-model="description" name="description" /> +<gl-form-group :label="__('Issue description')" label-for="issue-description" label-sr-only> + <gl-form-textarea id="issue-description" v-model="description" /> </gl-form-group> ``` @@ -128,11 +128,11 @@ Alternatively, you can use a plain `label` element: ```html <!-- Input with label using `label` --> <label for="issue-title">{{ __('Issue title') }}</label> -<gl-form-input id="issue-title" v-model="title" name="title" /> +<gl-form-input id="issue-title" v-model="title" /> <!-- Input with hidden label using `label` --> <label for="issue-title" class="gl-sr-only">{{ __('Issue title') }}</label> -<gl-form-input id="issue-title" v-model="title" name="title" /> +<gl-form-input id="issue-title" v-model="title" /> ``` #### Select inputs with accessible names @@ -142,12 +142,12 @@ Select input examples: ```html <!-- Select input with label --> <gl-form-group :label="__('Issue status')" label-for="issue-status"> - <gl-form-select id="issue-status" v-model="status" name="status" :options="options" /> + <gl-form-select id="issue-status" v-model="status" :options="options" /> </gl-form-group> <!-- Select input with hidden label --> -<gl-form-group :label="__('Issue status')" label-for="issue-status" :label-sr-only="true"> - <gl-form-select id="issue-status" v-model="status" name="status" :options="options" /> +<gl-form-group :label="__('Issue status')" label-for="issue-status" label-sr-only> + <gl-form-select id="issue-status" v-model="status" :options="options" /> </gl-form-group> ``` @@ -157,12 +157,12 @@ Single checkbox: ```html <!-- Single checkbox with label --> -<gl-form-checkbox v-model="status" name="status" value="task-complete"> +<gl-form-checkbox v-model="status" value="task-complete"> {{ __('Task complete') }} </gl-form-checkbox> <!-- Single checkbox with hidden label --> -<gl-form-checkbox v-model="status" name="status" value="task-complete"> +<gl-form-checkbox v-model="status" value="task-complete"> <span class="gl-sr-only">{{ __('Task complete') }}</span> </gl-form-checkbox> ``` @@ -172,24 +172,24 @@ Multiple checkboxes: ```html <!-- Multiple labeled checkboxes grouped within a fieldset --> <gl-form-group :label="__('Task list')"> - <gl-form-checkbox name="task-list" value="task-1">{{ __('Task 1') }}</gl-form-checkbox> - <gl-form-checkbox name="task-list" value="task-2">{{ __('Task 2') }}</gl-form-checkbox> + <gl-form-checkbox value="task-1">{{ __('Task 1') }}</gl-form-checkbox> + <gl-form-checkbox value="task-2">{{ __('Task 2') }}</gl-form-checkbox> </gl-form-group> <!-- Or --> <gl-form-group :label="__('Task list')"> - <gl-form-checkbox-group v-model="selected" :options="options" name="task-list" /> + <gl-form-checkbox-group v-model="selected" :options="options" /> </gl-form-group> <!-- Multiple labeled checkboxes grouped within a fieldset with hidden legend --> -<gl-form-group :label="__('Task list')" :label-sr-only="true"> - <gl-form-checkbox name="task-list" value="task-1">{{ __('Task 1') }}</gl-form-checkbox> - <gl-form-checkbox name="task-list" value="task-2">{{ __('Task 2') }}</gl-form-checkbox> +<gl-form-group :label="__('Task list')" label-sr-only> + <gl-form-checkbox value="task-1">{{ __('Task 1') }}</gl-form-checkbox> + <gl-form-checkbox value="task-2">{{ __('Task 2') }}</gl-form-checkbox> </gl-form-group> <!-- Or --> -<gl-form-group :label="__('Task list')" :label-sr-only="true"> - <gl-form-checkbox-group v-model="selected" :options="options" name="task-list" /> +<gl-form-group :label="__('Task list')" label-sr-only> + <gl-form-checkbox-group v-model="selected" :options="options" /> </gl-form-group> ``` @@ -199,12 +199,12 @@ Single radio input: ```html <!-- Single radio with a label --> -<gl-form-radio v-model="status" name="status" value="opened"> +<gl-form-radio v-model="status" value="opened"> {{ __('Opened') }} </gl-form-radio> <!-- Single radio with a hidden label --> -<gl-form-radio v-model="status" name="status" value="opened"> +<gl-form-radio v-model="status" value="opened"> <span class="gl-sr-only">{{ __('Opened') }}</span> </gl-form-radio> ``` @@ -214,24 +214,24 @@ Multiple radio inputs: ```html <!-- Multiple labeled radio inputs grouped within a fieldset --> <gl-form-group :label="__('Issue status')"> - <gl-form-radio name="status" value="opened">{{ __('Opened') }}</gl-form-radio> - <gl-form-radio name="status" value="closed">{{ __('Closed') }}</gl-form-radio> + <gl-form-radio value="opened">{{ __('Opened') }}</gl-form-radio> + <gl-form-radio value="closed">{{ __('Closed') }}</gl-form-radio> </gl-form-group> <!-- Or --> <gl-form-group :label="__('Issue status')"> - <gl-form-radio-group v-model="selected" :options="options" name="status" /> + <gl-form-radio-group v-model="selected" :options="options" /> </gl-form-group> <!-- Multiple labeled radio inputs grouped within a fieldset with hidden legend --> -<gl-form-group :label="__('Issue status')" :label-sr-only="true"> - <gl-form-radio name="status" value="opened">{{ __('Opened') }}</gl-form-radio> - <gl-form-radio name="status" value="closed">{{ __('Closed') }}</gl-form-radio> +<gl-form-group :label="__('Issue status')" label-sr-only> + <gl-form-radio value="opened">{{ __('Opened') }}</gl-form-radio> + <gl-form-radio value="closed">{{ __('Closed') }}</gl-form-radio> </gl-form-group> <!-- Or --> -<gl-form-group :label="__('Issue status')" :label-sr-only="true"> - <gl-form-radio-group v-model="selected" :options="options" name="status" /> +<gl-form-group :label="__('Issue status')" label-sr-only> + <gl-form-radio-group v-model="selected" :options="options" /> </gl-form-group> ``` @@ -242,11 +242,11 @@ File input examples: ```html <!-- File input with a label --> <label for="attach-file">{{ __('Attach a file') }}</label> -<input id="attach-file" type="file" name="attach-file" /> +<input id="attach-file" type="file" /> <!-- File input with a hidden label --> <label for="attach-file" class="gl-sr-only">{{ __('Attach a file') }}</label> -<input id="attach-file" type="file" name="attach-file" /> +<input id="attach-file" type="file" /> ``` #### GlToggle components with an accessible names @@ -337,7 +337,7 @@ element is interactive you must ensure: - It can receive keyboard focus. - It has a visible focus state. -Use semantic HTML, such as `a` and `button`, which provides these behaviours by default. +Use semantic HTML, such as `a` (`GlLink`) and `button` (`GlButton`), which provides these behaviours by default. Keep in mind that: @@ -351,7 +351,7 @@ See the [Pajamas Keyboard-only page](https://design.gitlab.com/accessibility-aud Prefer **no** `tabindex` to using `tabindex`, since: -- Using semantic HTML such as `button` implicitly provides `tabindex="0"`. +- Using semantic HTML such as `button` (`GlButton`) implicitly provides `tabindex="0"`. - Tabbing order should match the visual reading order and positive `tabindex`s interfere with this. ### Avoid using `tabindex="0"` to make an element interactive @@ -359,8 +359,8 @@ Prefer **no** `tabindex` to using `tabindex`, since: Use interactive elements instead of `div` and `span` tags. For example: -- If the element should be clickable, use a `button`. -- If the element should be text editable, use an `input` or `textarea`. +- If the element should be clickable, use a `button` (`GlButton`). +- If the element should be text editable, use an [`input` or `textarea`](#text-inputs-with-accessible-names). Once the markup is semantically complete, use CSS to update it to its desired visual state. diff --git a/doc/development/fe_guide/architecture.md b/doc/development/fe_guide/architecture.md index 1d08296eafc..0c85a21fdf4 100644 --- a/doc/development/fe_guide/architecture.md +++ b/doc/development/fe_guide/architecture.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Architecture diff --git a/doc/development/fe_guide/axios.md b/doc/development/fe_guide/axios.md index b42a17d7870..f90a2b37a1c 100644 --- a/doc/development/fe_guide/axios.md +++ b/doc/development/fe_guide/axios.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Axios diff --git a/doc/development/fe_guide/content_editor.md b/doc/development/fe_guide/content_editor.md index f262e48b6da..8cc274c732e 100644 --- a/doc/development/fe_guide/content_editor.md +++ b/doc/development/fe_guide/content_editor.md @@ -1,7 +1,7 @@ --- stage: Create group: Editor -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Content Editor development guidelines **(FREE)** diff --git a/doc/development/fe_guide/customizable_dashboards.md b/doc/development/fe_guide/customizable_dashboards.md new file mode 100644 index 00000000000..38ee750d421 --- /dev/null +++ b/doc/development/fe_guide/customizable_dashboards.md @@ -0,0 +1,99 @@ +--- +stage: Analytics +group: Product Analytics +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Customizable dashboards **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98610) in GitLab 15.5 as an [Alpha feature](../../policy/alpha-beta-support.md#alpha-features). + +Customizable dashboards provide a dashboard structure that allows users to create +their own dashboards and commit the structure to a repository. + +## Usage + +To use customizable dashboards: + +1. Create your dashboard component. +1. Render an instance of `CustomizableDashboard`. +1. Pass a list of widgets to render. + +For example, a customizable dashboard for users over time: + +```vue +<script> +import CustomizableDashboard from 'ee/vue_shared/components/customizable_dashboard/customizable_dashboard.vue'; +import { s__ } from '~/locale'; + +export default { + name: 'AnalyticsDashboard', + components: { + CustomizableDashboard, + }, + data() { + return { + widgets: [ + { + component: 'CubeLineChart', // The name of the widget component. + title: s__('ProductAnalytics|Users / Time'), // The title shown on the widget component. + // Gridstack settings based upon https://github.com/gridstack/gridstack.js/tree/master/doc#item-options. + // All values are grid row/column numbers up to 12. + // We use the default 12 column grid https://github.com/gridstack/gridstack.js#change-grid-columns. + gridAttributes: { + size: { + height: 4, + width: 6, + minHeight: 4, + minWidth: 6, + }, + position: { + xPos: 0, + yPos: 0, + }, + }, + // Options that are used to set bespoke values for each widget. + // Available customizations are determined by the widget itself. + customizations: {}, + // Chart options defined by the charting library being used by the widget. + chartOptions: { + xAxis: { name: __('Time'), type: 'time' }, + yAxis: { name: __('Counts') }, + }, + // The data for the widget. + // This could be imported or in this case, a query passed to be used by the widgets API. + // Each widget type determines how it handles this property. + data: { + query: { + users: { + measures: ['Jitsu.count'], + dimensions: ['Jitsu.eventType'], + }, + }, + }, + }, + ] + }; + }, +}; +</script> + +<template> + <h1>{{ s__('ProductAnalytics|Analytics dashboard') }}</h1> + <customizable-dashboard :widgets="widgets" /> +</template> +``` + +The widgets data can be retrieved from a file or API request, or imported through HTML data attributes. + +For each widget, a `component` is defined. Each `component` is a component declaration and should be included in +[`vue_shared/components/customizable_dashboard/widgets_base.vue`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/assets/javascripts/vue_shared/components/customizable_dashboard/widgets_base.vue) +as a dynamic import, to keep the memory usage down until it is used. + +For example: + +```javascript +components: { + CubeLineChart: () => import('ee/product_analytics/dashboards/components/widgets/cube_line_chart.vue') +} +``` diff --git a/doc/development/fe_guide/dark_mode.md b/doc/development/fe_guide/dark_mode.md index 34901bbb1e6..d687d9740c9 100644 --- a/doc/development/fe_guide/dark_mode.md +++ b/doc/development/fe_guide/dark_mode.md @@ -2,7 +2,7 @@ type: reference, dev stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- This page is about developing dark mode for GitLab. We also have documentation on how diff --git a/doc/development/fe_guide/dependencies.md b/doc/development/fe_guide/dependencies.md index c4f30fd36c9..b9bacada499 100644 --- a/doc/development/fe_guide/dependencies.md +++ b/doc/development/fe_guide/dependencies.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Frontend dependencies diff --git a/doc/development/fe_guide/design_anti_patterns.md b/doc/development/fe_guide/design_anti_patterns.md index 580f488bd33..07de86f5810 100644 --- a/doc/development/fe_guide/design_anti_patterns.md +++ b/doc/development/fe_guide/design_anti_patterns.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Design Anti-patterns @@ -146,7 +146,7 @@ Even in these scenarios, consider avoiding the Singleton pattern. #### Utility Functions -When no state needs to be managed, we can simply export utility functions from a module without +When no state needs to be managed, we can export utility functions from a module without messing with any class instantiation. ```javascript diff --git a/doc/development/fe_guide/design_patterns.md b/doc/development/fe_guide/design_patterns.md index 03575f7e7f9..3c273ab18e9 100644 --- a/doc/development/fe_guide/design_patterns.md +++ b/doc/development/fe_guide/design_patterns.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Design Patterns diff --git a/doc/development/fe_guide/development_process.md b/doc/development/fe_guide/development_process.md index 3273263de3b..fc91ff55b24 100644 --- a/doc/development/fe_guide/development_process.md +++ b/doc/development/fe_guide/development_process.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Frontend Development Process @@ -26,7 +26,7 @@ Use your best judgment when to use it and contribute new points through merge re - [ ] Check the current set weight of the issue, does it fit your estimate? - [ ] Are all [departments](https://about.gitlab.com/handbook/engineering/#engineering-teams) that are needed from your perspective already involved in the issue? (For example is UX missing?) - [ ] Is the specification complete? Are you missing decisions? How about error handling/defaults/edge cases? Take your time to understand the needed implementation and go through its flow. -- [ ] Are all necessary UX specifications available that you will need in order to implement? Are there new UX components/patterns in the designs? Then contact the UI component team early on. How should error messages or validation be handled? +- [ ] Are all necessary UX specifications available that you will need to implement? Are there new UX components/patterns in the designs? Then contact the UI component team early on. How should error messages or validation be handled? - [ ] **Library usage** Use Vuex as soon as you have even a medium state to manage, use Vue router if you need to have different views internally and want to link from the outside. Check what libraries we already have for which occasions. - [ ] **Plan your implementation:** - [ ] **Architecture plan:** Create a plan aligned with GitLab's architecture, how you are going to do the implementation, for example Vue application setup and its components (through [onion skinning](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/35873#note_39994091)), Store structure and data flow, which existing Vue components can you reuse. It's a good idea to go through your plan with another engineer to refine it. diff --git a/doc/development/fe_guide/emojis.md b/doc/development/fe_guide/emojis.md index 3c7fc20440b..3296783a616 100644 --- a/doc/development/fe_guide/emojis.md +++ b/doc/development/fe_guide/emojis.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Emojis diff --git a/doc/development/fe_guide/frontend_faq.md b/doc/development/fe_guide/frontend_faq.md index 6a645416c0a..3b349d880c0 100644 --- a/doc/development/fe_guide/frontend_faq.md +++ b/doc/development/fe_guide/frontend_faq.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Frontend FAQ diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 6dcc57b0ff5..9ed3e551ff2 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -2,7 +2,7 @@ type: reference, dev stage: none group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" --- # GraphQL @@ -330,7 +330,7 @@ Along with creating local data, we can also extend existing GraphQL types with ` ##### Mocking API response with local Apollo cache -Using local Apollo Cache is helpful when we have a need to mock some GraphQL API responses, queries, or mutations locally (such as when they're still not added to our actual API). +Using local Apollo Cache is helpful when we have a reason to mock some GraphQL API responses, queries, or mutations locally (such as when they're still not added to our actual API). For example, we have a [fragment](#fragments) on `DesignVersion` used in our queries: @@ -341,7 +341,7 @@ fragment VersionListItem on DesignVersion { } ``` -We also need to fetch the version author and the `created at` property to display in the versions dropdown. But, these changes are still not implemented in our API. We can change the existing fragment to get a mocked response for these new fields: +We also must fetch the version author and the `created at` property to display in the versions dropdown list. But, these changes are still not implemented in our API. We can change the existing fragment to get a mocked response for these new fields: ```javascript fragment VersionListItem on DesignVersion { @@ -627,7 +627,7 @@ GraphQL entities are not yet part of the schema, or if they are feature-flagged ### Manually triggering queries Queries on a component's `apollo` property are made automatically when the component is created. -Some components instead want the network request made on-demand, for example a dropdown with lazy-loaded items. +Some components instead want the network request made on-demand, for example a dropdown list with lazy-loaded items. There are two ways to do this: @@ -1318,7 +1318,7 @@ automatically find and index the schema. #### Testing Apollo components -If we use `ApolloQuery` or `ApolloMutation` in our components, in order to test their functionality we need to add a stub first: +If we use `ApolloQuery` or `ApolloMutation` in our components, to test their functionality we need to add a stub first: ```javascript import { ApolloMutation } from 'vue-apollo'; diff --git a/doc/development/fe_guide/haml.md b/doc/development/fe_guide/haml.md index 7e72570454e..d76d718e8e5 100644 --- a/doc/development/fe_guide/haml.md +++ b/doc/development/fe_guide/haml.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # HAML diff --git a/doc/development/fe_guide/icons.md b/doc/development/fe_guide/icons.md index 73f196ef51f..df296f13f48 100644 --- a/doc/development/fe_guide/icons.md +++ b/doc/development/fe_guide/icons.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Icons and SVG Illustrations diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index d90c270153e..6403cff549f 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Frontend Development Guidelines diff --git a/doc/development/fe_guide/keyboard_shortcuts.md b/doc/development/fe_guide/keyboard_shortcuts.md index aab252da305..aeeee72e6be 100644 --- a/doc/development/fe_guide/keyboard_shortcuts.md +++ b/doc/development/fe_guide/keyboard_shortcuts.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Implementing keyboard shortcuts diff --git a/doc/development/fe_guide/logging.md b/doc/development/fe_guide/logging.md index 26633eade43..30cf290fbe8 100644 --- a/doc/development/fe_guide/logging.md +++ b/doc/development/fe_guide/logging.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Client-side logging for frontend development diff --git a/doc/development/fe_guide/merge_request_widget_extensions.md b/doc/development/fe_guide/merge_request_widget_extensions.md index a2ff10cc57f..e5e813bf921 100644 --- a/doc/development/fe_guide/merge_request_widget_extensions.md +++ b/doc/development/fe_guide/merge_request_widget_extensions.md @@ -1,7 +1,7 @@ --- stage: Create group: Code Review -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Merge request widget extensions **(FREE)** diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index 2e1fabd739c..3aa901093f0 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Performance diff --git a/doc/development/fe_guide/principles.md b/doc/development/fe_guide/principles.md index 1bf37c8d008..6d1a3238b73 100644 --- a/doc/development/fe_guide/principles.md +++ b/doc/development/fe_guide/principles.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Principles diff --git a/doc/development/fe_guide/registry_architecture.md b/doc/development/fe_guide/registry_architecture.md index be14d5d920c..a87d38634d5 100644 --- a/doc/development/fe_guide/registry_architecture.md +++ b/doc/development/fe_guide/registry_architecture.md @@ -1,7 +1,7 @@ --- stage: Package group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Registry architecture diff --git a/doc/development/fe_guide/security.md b/doc/development/fe_guide/security.md index 6f500c8f0fa..4b7ce6d11e3 100644 --- a/doc/development/fe_guide/security.md +++ b/doc/development/fe_guide/security.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Security diff --git a/doc/development/fe_guide/source_editor.md b/doc/development/fe_guide/source_editor.md index 88508e94380..b7cd903485e 100644 --- a/doc/development/fe_guide/source_editor.md +++ b/doc/development/fe_guide/source_editor.md @@ -1,7 +1,7 @@ --- stage: Create group: Editor -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Source Editor **(FREE)** diff --git a/doc/development/fe_guide/storybook.md b/doc/development/fe_guide/storybook.md index a3a1fa2160f..e57c117bc39 100644 --- a/doc/development/fe_guide/storybook.md +++ b/doc/development/fe_guide/storybook.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Storybook diff --git a/doc/development/fe_guide/style/html.md b/doc/development/fe_guide/style/html.md index 90ff88bc975..b1cce29bc61 100644 --- a/doc/development/fe_guide/style/html.md +++ b/doc/development/fe_guide/style/html.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # HTML style guide diff --git a/doc/development/fe_guide/style/index.md b/doc/development/fe_guide/style/index.md index f3da78647be..94ed9544cf5 100644 --- a/doc/development/fe_guide/style/index.md +++ b/doc/development/fe_guide/style/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab development style guides diff --git a/doc/development/fe_guide/style/javascript.md b/doc/development/fe_guide/style/javascript.md index b86bdfafa21..38fb926197b 100644 --- a/doc/development/fe_guide/style/javascript.md +++ b/doc/development/fe_guide/style/javascript.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/development/fe_guide/style_guide_js.html' --- diff --git a/doc/development/fe_guide/style/scss.md b/doc/development/fe_guide/style/scss.md index 17e80762a38..aacf07fda92 100644 --- a/doc/development/fe_guide/style/scss.md +++ b/doc/development/fe_guide/style/scss.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/development/fe_guide/style_guide_scss.html' --- diff --git a/doc/development/fe_guide/style/vue.md b/doc/development/fe_guide/style/vue.md index 39dc9cc9c4e..a21d7c4577b 100644 --- a/doc/development/fe_guide/style/vue.md +++ b/doc/development/fe_guide/style/vue.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Vue.js style guide diff --git a/doc/development/fe_guide/tooling.md b/doc/development/fe_guide/tooling.md index 4b55580c33c..c9efb8e939d 100644 --- a/doc/development/fe_guide/tooling.md +++ b/doc/development/fe_guide/tooling.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Tooling diff --git a/doc/development/fe_guide/troubleshooting.md b/doc/development/fe_guide/troubleshooting.md index ab10c5bf988..873a14b8f14 100644 --- a/doc/development/fe_guide/troubleshooting.md +++ b/doc/development/fe_guide/troubleshooting.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Troubleshooting frontend development issues diff --git a/doc/development/fe_guide/view_component.md b/doc/development/fe_guide/view_component.md index 7ddce609ee7..662d1ad32fc 100644 --- a/doc/development/fe_guide/view_component.md +++ b/doc/development/fe_guide/view_component.md @@ -1,7 +1,7 @@ --- -stage: Ecosystem +stage: Manage group: Foundations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # ViewComponent diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 5cb461c8ca0..00d9588d087 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Vue @@ -408,11 +408,11 @@ export function useCount(initialValue) { const count = ref(initialValue) function incrementCount() { - ref.value += 1 + count.value += 1 } function decrementCount() { - ref.value -= 1 + count.value -= 1 } return { count, incrementCount, decrementCount } diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index 068b0c5b475..c3ce42a80a5 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Migration to Vue 3 diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 2d1569b7812..5047f1b7f89 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Vuex diff --git a/doc/development/fe_guide/widgets.md b/doc/development/fe_guide/widgets.md index c6bb89d1fe8..3364c778d76 100644 --- a/doc/development/fe_guide/widgets.md +++ b/doc/development/fe_guide/widgets.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Widgets diff --git a/doc/development/feature_categorization/index.md b/doc/development/feature_categorization/index.md index a93ed58336d..0bf506b53ba 100644 --- a/doc/development/feature_categorization/index.md +++ b/doc/development/feature_categorization/index.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Infrastructure -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Feature Categorization diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md index fd1c7f4afa5..760ba033633 100644 --- a/doc/development/feature_development.md +++ b/doc/development/feature_development.md @@ -1,7 +1,7 @@ --- stage: none group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" --- # Feature development @@ -114,6 +114,7 @@ Consult these topics for information on contributing to specific GitLab features - [Cached queries guidelines](cached_queries.md), for tracking down N+1 queries masked by query caching, memory profiling and why should we avoid cached queries. +- [JSON guidelines](json.md) for how to handle JSON in a performant manner. ## Database guides @@ -128,6 +129,12 @@ See [database guidelines](database/index.md). - [How to run Jenkins in development environment](integrations/jenkins.md) - [How to run local `Codesandbox` integration for Web IDE Live Preview](integrations/codesandbox.md) +The following integration guides are internal. Some integrations require access to administrative accounts of third-party services and are available only for GitLab team members to contribute to: + +- [Jira app development](https://gitlab.com/gitlab-org/manage/integrations/team/-/blob/main/integrations/jira.md) +- [Slack app development](https://gitlab.com/gitlab-org/manage/integrations/team/-/blob/main/integrations/slack.md) +- [ZenTao development](https://gitlab.com/gitlab-org/manage/integrations/team/-/blob/main/integrations/zentao.md) + ## Testing guides - [Testing standards and style guidelines](testing_guide/index.md) diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index 63dad3070c7..e6c3c20d50b 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -2,15 +2,18 @@ type: reference, dev stage: none group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" --- -# Feature flag controls +# Use ChatOps to enable and disable feature flags -## Access +NOTE: +This document explains how to contribute to the development of the GitLab product. +If you want to use feature flags to show and hide functionality in your own applications, +view [this feature flags information](../../operations/feature_flags.md) instead. -To be able to turn on/off features behind feature flags in any of the -GitLab Inc. provided environments such as staging and production, you need to +To turn on/off features behind feature flags in any of the +GitLab-provided environments, like staging and production, you need to have access to the [ChatOps](../chatops_on_gitlabcom.md) bot. The ChatOps bot is currently running on the ops instance, which is different from [GitLab.com](https://gitlab.com) or [`dev.gitlab.org`](https://dev.gitlab.org). @@ -47,12 +50,12 @@ Note that all the examples in that file must be preceded by If you get an error "Whoops! This action is not allowed. This incident will be reported." that means your Slack account is not allowed to -change feature flags or you do not [have access](#access). +change feature flags or you do not have access. ### Enabling a feature for pre-production testing As a first step in a feature rollout, you should enable the feature on -[`about.staging.gitlab.com`](https://about.staging.gitlab.com) +[`staging.gitlab.com`](https://staging.gitlab.com) and [`dev.gitlab.org`](https://dev.gitlab.org). These two environments have different scopes. @@ -173,7 +176,7 @@ For project level features: Feature.enabled?(:feature_ice_cold_projects, project) ``` -If you are not certain what percentages to use, simply use the following steps: +If you are not certain what percentages to use, use the following steps: 1. 25% 1. 50% diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 444b53f9c8d..f1cde4ae1ea 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -2,24 +2,20 @@ type: reference, dev stage: none group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" --- # Feature flags in the development of GitLab NOTE: -The documentation below covers feature flags used by GitLab to deploy its own features, which **is not** the same -as the [feature flags offered as part of the product](../../operations/feature_flags.md). - -This document provides guidelines on how to use feature flags -for the development of GitLab to conditionally and/or incrementally enable features -and test them in production/staging. +This document explains how to contribute to the development of the GitLab product. +If you want to use feature flags to show and hide functionality in your own applications, +view [this feature flags information](../../operations/feature_flags.md) instead. WARNING: All newly-introduced feature flags should be [disabled by default](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#feature-flags-in-gitlab-development). -NOTE: -This document is the subject of continued work as part of an epic to [improve internal usage of Feature Flags](https://gitlab.com/groups/gitlab-org/-/epics/3551). Raise any suggestions as new issues and attach them to the epic. +This document is the subject of continued work as part of an epic to [improve internal usage of feature flags](https://gitlab.com/groups/gitlab-org/-/epics/3551). Raise any suggestions as new issues and attach them to the epic. For an [overview of the feature flag lifecycle](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle), or if you need help deciding [if you should use a feature flag](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags) or not, please see the [feature flag lifecycle](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) handbook page. @@ -56,7 +52,7 @@ When the feature implementation is delivered among multiple merge requests: 1. When the feature is ready to be announced, create a merge request that adds documentation about the feature, including [documentation for the feature flag itself](../documentation/feature_flags.md), and a [changelog entry](#changelog). In the same merge request either flip the feature flag to - be **on by default** or remove it entirely in order to enable the new behavior. + be **on by default** or remove it entirely to enable the new behavior. One might be tempted to think that feature flags will delay the release of a feature by at least one month (= one release). This is not the case. A feature diff --git a/doc/development/features_inside_dot_gitlab.md b/doc/development/features_inside_dot_gitlab.md index f30a041931e..7a46cd40da1 100644 --- a/doc/development/features_inside_dot_gitlab.md +++ b/doc/development/features_inside_dot_gitlab.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Features inside the `.gitlab/` directory @@ -16,6 +16,6 @@ When implementing new features, please refer to these existing features to avoid - [CODEOWNERS](../user/project/code_owners.md#set-up-code-owners): `.gitlab/CODEOWNERS`. - [Route Maps](../ci/review_apps/index.md#route-maps): `.gitlab/route-map.yml`. - [Customize Auto DevOps Helm Values](../topics/autodevops/customize.md#customize-values-for-helm-chart): `.gitlab/auto-deploy-values.yaml`. -- [Insights](../user/project/insights/index.md#configure-your-insights): `.gitlab/insights.yml`. +- [Insights](../user/project/insights/index.md#configure-project-insights): `.gitlab/insights.yml`. - [Service Desk Templates](../user/project/service_desk.md#using-customized-email-templates): `.gitlab/service_desk_templates/`. - [Web IDE](../user/project/web_ide/index.md#web-ide-configuration-file): `.gitlab/.gitlab-webide.yml`. diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index 04e2f381c97..11acb8a2161 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # File Storage in GitLab diff --git a/doc/development/fips_compliance.md b/doc/development/fips_compliance.md index 1029ed88eac..475c1a49000 100644 --- a/doc/development/fips_compliance.md +++ b/doc/development/fips_compliance.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # FIPS compliance @@ -67,7 +67,7 @@ listed here that also do not work properly in FIPS mode: - [Static Application Security Testing (SAST)](../user/application_security/sast/index.md) supports a reduced set of [analyzers](../user/application_security/sast/index.md#fips-enabled-images) when operating in FIPS-compliant mode. -- Advanced Search is currently not included in FIPS mode. It must not be enabled in order to be FIPS-compliant. +- Advanced Search is currently not included in FIPS mode. It must not be enabled to be FIPS-compliant. - [Gravatar or Libravatar-based profile images](../administration/libravatar.md) are not FIPS-compliant. Additionally, these package repositories are disabled in FIPS mode: @@ -417,7 +417,7 @@ In this environment, OpenSSL refuses to perform cryptographic operations forbidden by the FIPS standards. This enables you to reproduce FIPS-related bugs, and validate fixes. -You should be able to open a web browser inside the virtual machine and log in +You should be able to open a web browser inside the virtual machine and sign in to the GitLab instance. You can disable FIPS mode again by running this command, then restarting the diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md index 87304a761ea..7d3531afb49 100644 --- a/doc/development/gemfile.md +++ b/doc/development/gemfile.md @@ -1,20 +1,67 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Gemfile guidelines -When adding a new entry to `Gemfile` or upgrading an existing dependency pay +When adding a new entry to `Gemfile`, or upgrading an existing dependency pay attention to the following rules. +## Bundler checksum verification + +In [GitLab 15.5 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98508), gem +checksums are checked before installation. This verification is still +experimental so it is only active for CI. + +If the downloaded gem's checksum does not match the checksum record in +`Gemfile.checksum`, you will see an error saying that Bundler cannot continue +installing a gem because there is a potential security issue. + +You will see this error as well if you updated, or added a new gem without +updating `Gemfile.checksum`. To fix this error, +[update the Gemfile.checksum](#updating-the-checksum-file). + +You can opt-in to this verification locally by setting the +`BUNDLER_CHECKSUM_VERIFICATION_OPT_IN` environment variable: + +```shell +export BUNDLER_CHECKSUM_VERIFICATION_OPT_IN=1 +bundle install +``` + +### Updating the checksum file + +This needs to be done for any new, or updated gems. + +1. When updating `Gemfile.lock`, make sure to also update `Gemfile.checksum` with: + + ```shell + bundle exec bundler-checksum init + ``` + +1. Check and commit the changes for `Gemfile.checksum`. + ## No gems fetched from Git repositories We do not allow gems that are fetched from Git repositories. All gems have to be available in the RubyGems index. We want to minimize external build dependencies and build times. +## Review the new dependency for quality + +We should not add 3rd-party dependencies to GitLab that would not pass our own quality standards. +This means that new dependencies should, at a minimum, meet the following criteria: + +- They have an active developer community. At the minimum a maintainer should still be active + to merge change requests in case of emergencies. +- There are no issues open that we know may impact the availablity or performance of GitLab. +- The project is tested using some form of test automation. The test suite must be passing + using the Ruby version currently used by GitLab. +- If the project uses a C extension, consider requesting an additional review from a C or MRI + domain expert. C extensions can greatly impact GitLab stability and performance. + ## Request an Appsec review When adding a new gem to our `Gemfile` or even changing versions in diff --git a/doc/development/geo.md b/doc/development/geo.md index 1ae9d9ee32b..884c09cc174 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo (development) **(PREMIUM SELF)** @@ -69,8 +69,8 @@ the lock, it switches to standby mode. Geo uses [streaming replication](#streaming-replication) to replicate the database from the **primary** to the **secondary** sites. This replication gives the **secondary** sites access to all the data saved -in the database. So users can log in on the **secondary** and read all -the issues, merge requests, and so on, on the **secondary** site. +in the database, so users can sign in to the **secondary** site and read, +for example, all the issues and merge requests. ### Repository replication diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index 18774b9b3fd..3624d280f86 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo self-service framework diff --git a/doc/development/geo/proxying.md b/doc/development/geo/proxying.md index d4cb611e965..67d4129a9d0 100644 --- a/doc/development/geo/proxying.md +++ b/doc/development/geo/proxying.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo proxying diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index a20bdf633cd..dcfcd2e864a 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -1,7 +1,7 @@ --- stage: Systems group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # How Git object deduplication works in GitLab diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index 66b535682f5..a570b5dd7eb 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -1,7 +1,7 @@ --- stage: Systems group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Gitaly developers guide @@ -219,7 +219,7 @@ as a [CI/CD variable](../ci/variables/index.md). If you are making changes to the RPC client, such as adding a new endpoint or adding a new parameter to an existing endpoint, follow the guide for -[Gitaly protobuf specifications](https://gitlab.com/gitlab-org/gitaly/blob/master/proto/README.md). After pushing +[Gitaly protobuf specifications](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/protobuf.md). After pushing the branch with the changes (`new-feature-branch`, for example): 1. Change the `gitaly` line in the Rails' `Gemfile` to: diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index 9740ae04553..9ba375439f4 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Working with the GitHub importer @@ -114,6 +114,9 @@ GitHub are stored in a single table. Therefore, they have globally-unique IDs an Therefore, both issues and pull requests have a common API for most related things. +NOTE: +This stage is optional and can consume significant extra import time (controlled by `Gitlab::GithubImport::Settings`). + ### 9. Stage::ImportNotesWorker This worker imports regular comments for both issues and pull requests. For @@ -127,16 +130,23 @@ comments. ### 10. Stage::ImportAttachmentsWorker -This worker imports release notes attachments that are linked inside Markdown. -For every release of the project, we schedule a job of -`Gitlab::GithubImport::ImportReleaseAttachmentsWorker` for every comment. +This worker imports note attachments that are linked inside Markdown. +For each entity with Markdown text in the project, we schedule a job of: + +- `Gitlab::GithubImport::ImportReleaseAttachmentsWorker` for every release. +- `Gitlab::GithubImport::ImportNoteAttachmentsWorker` for every note. +- `Gitlab::GithubImport::ImportIssueAttachmentsWorker` for every issue. +- `Gitlab::GithubImport::ImportMergeRequestAttachmentsWorker` for every merge request. Each job: -1. Iterates over all attachment links inside of a specific release note. +1. Iterates over all attachment links inside of a specific record. 1. Downloads the attachment. 1. Replaces the old link with a newly-generated link to GitLab. +NOTE: +It's an optional stage that could consume significant extra import time (controlled by `Gitlab::GithubImport::Settings`). + ### 11. Stage::ImportProtectedBranchesWorker This worker imports protected branch rules. @@ -197,7 +207,7 @@ GitHub has a rate limit of 5,000 API calls per hour. The number of requests necessary to import a project is largely dominated by the number of unique users involved in a project (for example, issue authors). Other data such as issue pages and comments typically only requires a few dozen requests to import. This is -because we need the Email address of users in order to map them to GitLab users. +because we need the Email address of users to map them to GitLab users. We handle this by doing the following: diff --git a/doc/development/gitlab_flavored_markdown/index.md b/doc/development/gitlab_flavored_markdown/index.md index 7f7781cbc62..0af31892726 100644 --- a/doc/development/gitlab_flavored_markdown/index.md +++ b/doc/development/gitlab_flavored_markdown/index.md @@ -1,7 +1,7 @@ --- stage: Create group: Editor -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab Flavored Markdown (GLFM) developer documentation **(FREE)** diff --git a/doc/development/gitlab_flavored_markdown/specification_guide/index.md b/doc/development/gitlab_flavored_markdown/specification_guide/index.md index c1227e5d33f..95d06907aa6 100644 --- a/doc/development/gitlab_flavored_markdown/specification_guide/index.md +++ b/doc/development/gitlab_flavored_markdown/specification_guide/index.md @@ -1,34 +1,67 @@ --- stage: Create group: Editor -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab Flavored Markdown (GLFM) Specification Guide **(FREE)** +## Summary + +- _GitLab_ Flavored Markdown (GLFM) is based on + [_GitHub_ Flavored Markdown](https://github.github.com/gfm/) (GFM), + which is based on [CommonMark](https://spec.commonmark.org/current/). +- GLFM is divided into two "sets" of Markdown syntax: + - An "[official specification](#official-specifications)", + which is not dependent upon any specific + implementation or environment, and can be supported in any editor. + - "[Internal extensions](#internal-extensions)", which may be + dependent upon the GitLab environment and metadata. +- Everything in each of these sets of syntax is specified by + [special Markdown files](#input-specification-files) + based on the [CommonMark specification syntax](https://spec.commonmark.org/0.30/#about-this-document), + which contain side-by-side "examples" of Markdown and the corresponding + generated HTML, and associated documentation describing each example. +- There are also [YAML metadata files](#input-specification-files), which + may contain additional information on how individual Markdown/HTML examples + should be processed and rendered. +- These Markdown/YAML files and the examples they contain serve multiple goals: + - They are the canonical "source of truth" for how GLFM should be rendered. + - They support rendering a [formatted HTML document](#spechtml) containing all + of the examples and associated documentation, as the + [GFM and CommonMark specs](#various-markdown-specifications) also do. + - They support running standard CommonMark [conformance testing](#markdown-conformance-testing) + against the official specification. + - They support [snapshot testing](#markdown-snapshot-testing) of GitLab + internal GLFM processing logic. This is accomplished by automatically + generating YAML ["example snapshot files"](#example-snapshot-files) + which are used as fixtures to drive automated testing within the GitLab app. +- There are [various scripts and logic](#scripts) + which are used to accomplish the above goals. + +## Introduction + GitLab supports Markdown in various places. The Markdown dialect we use is called -GitLab Flavored Markdown, or GLFM. +GitLab Flavored Markdown (GLFM). + +NOTE: +In this document, _GFM_ refers to _GitHub_ Flavored Markdown, not _GitLab_ Flavored Markdown. +Refer to the [section on acronyms](#acronyms-glfm-ghfm-gfm-commonmark) +for a detailed explanation of the various acronyms used in this document. The specification for the GLFM dialect is based on the [GitHub Flavored Markdown (GFM) specification](https://github.github.com/gfm/), which is in turn based on the [CommonMark specification](https://spec.commonmark.org/current/). The GLFM specification includes -[several extensions](../../../user/markdown.md#differences-between-gitlab-flavored-markdown-and-standard-markdown) -to the GFM specification. +[many additions](../../../user/markdown.md#differences-between-gitlab-flavored-markdown-and-standard-markdown) +compared to the GFM specification. -See the [section on acronyms](#acronyms-glfm-ghfm-gfm-commonmark) for a -detailed explanation of the various acronyms used in this document. This guide is a developer-facing document that describes the various terms and definitions, goals, tools, and implementations related to the GLFM specification. It is intended to support and augment the [user-facing documentation](../../../user/markdown.md) for GitLab Flavored Markdown. NOTE: -In this document, _GFM_ refers to _GitHub_ Flavored Markdown, not _GitLab_ Flavored Markdown. -Refer to the [section on acronyms](#acronyms-glfm-ghfm-gfm-commonmark) -for a detailed explanation of the various acronyms used in this document. - -NOTE: This guide and the implementation and files described in it are still a work in progress. As the work progresses, rewrites and consolidation between this guide and the [user-facing documentation](../../../user/markdown.md) @@ -43,7 +76,7 @@ to by the acronym GFM, and this document follows that convention as well. _GitLab_ Flavored Markdown is referred to as GLFM in this document, to distinguish it from GitHub Flavored Markdown. -Unfortunately, this convention is not followed consistently in the rest +Unfortunately, this convention is not yet followed consistently in the rest of the documentation or GitLab codebase. In many places, the GFM acronym is used to refer to _GitLab_ Flavored Markdown. An [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/24592) exists to resolve @@ -53,7 +86,7 @@ Some places in the code refer to both the GitLab and GitHub specifications simultaneous in the same areas of logic. In these situations, _GitHub_ Flavored Markdown may be referred to with variable or constant names like `ghfm_` to avoid confusion. For example, we use the `ghfm` acronym for the -[`ghfm_spec_v_0.29.txt` GitHub Flavored Markdown specification file](#github-flavored-markdown-specification) +[`ghfm_spec_v_0.29.md` GitHub Flavored Markdown specification file](#github-flavored-markdown-specification), which is committed to the `gitlab` repository and used as input to the [`update_specification.rb` script](#update-specificationrb-script). @@ -77,7 +110,7 @@ Here are the HTML-rendered versions of the specifications: NOTE: The creation of the -[GitLab Flavored Markdown (GLFM) specification](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output/spec.html) +[HTML-rendered version of the GitLab Flavored Markdown (GLFM) specification](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output/spec.html) file is still pending. However, GLFM has more complex parsing, rendering, and testing requirements than @@ -134,7 +167,8 @@ Markdown for issues, pull requests, or merge requests within the editor or IDE. ### Markdown examples Everywhere in the context of the specification and this guide, the term -_examples_ is specifically used to refer to the Markdown + HTML pairs used +_examples_ is specifically used to refer to the convention of using +backtick-delimited Markdown + HTML pairs to illustrate the canonical parsing (or rendering) behavior of various Markdown source strings in the standard [CommonMark specification format](https://spec.commonmark.org/0.30/#example-1). @@ -143,6 +177,9 @@ In this context, it should not be confused with other similar or related meaning _example_, such as [RSpec examples](https://relishapp.com/rspec/rspec-core/docs/example-groups/basic-structure-describe-it). +See the section on the [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd) file +for more details on the backtick-delimited Markdown+HTML example syntax. + ### Parsers and renderers To understand the various ways in which a specification is used, and how it related @@ -169,11 +206,20 @@ of specification-driven testing referred to in this documentation and elsewhere. #### Markdown conformance testing +NOTE: +Markdown conformance testing for GLFM is not yet implemented. + _Markdown conformance testing_ refers to the standard testing method used by all CommonMark Markdown dialects to verify that a specific implementation conforms to the CommonMark Markdown specification. It is enforced by running the standard CommonMark tool [`spec_tests.py`](https://github.com/github/cmark-gfm/blob/master/test/spec_tests.py) -against a given `spec.txt` specification and the implementation. +against a given `spec.txt` specification and the implementation, as +[described in the specification itself](https://github.github.com/gfm/#about-this-document) + +Conformance testing is _only_ to be run against GLFM [official specification](#official-specifications) +examples, and is _not_ to be run against [internal extension](#internal-extensions) examples. +This is because the internal extension examples may have dependencies on the GitLab environment +or metadata, but the standard CommonMark conformance testing tool does not support this. NOTE: `spec_tests.py` may eventually be re-implemented in Ruby, to not have a dependency on Python. @@ -187,7 +233,14 @@ tests which use the fixture data. This fixture data is contained in YAML files. are generated and updated based on the Markdown examples in the specification, and the existing GLFM parser and render implementations. They may also be manually updated as necessary to test-drive incomplete implementations. -Regarding the terminology used here: + +Snapshot testing is intended to be comprehensive, so it is run against _all_ examples - both the GLFM +[official specification](#official-specifications) and [internal extension](#internal-extensions) examples. +This means that it uses configuration files to support providing GitLab-specific environment or metadata +which is required by internal extension examples, such +as [`glfm_example_metadata.yml`](#glfm_example_metadatayml). + +Regarding the terminology used for Markdown snapshot testing: <!-- vale gitlab.InclusionCultural = NO --> @@ -270,13 +323,14 @@ implementations: ### Multiple versions of rendered HTML Both of these GLFM renderer implementations (static and WYSIWYG) produce -HTML which differs from the canonical HTML examples from the specification. -For every Markdown example in the GLFM specification, three +HTML which may differ from the canonical HTML examples in the +<abbr title="GitLab Flavored Markdown">GLFM</abbr> [official specification](#official-specifications). +Therefore, for every Markdown example in the GLFM specification, three versions of HTML can potentially be rendered from the example: -- Static HTML. -- WYSIWYG HTML. -- Canonical HTML. +- Static HTML +- WYSIWYG HTML +- Canonical HTML #### Static HTML @@ -286,22 +340,57 @@ added for dynamically creating an issue from a task list item. The GitLab [Markdown API](../../../api/markdown.md) generates HTML for a given Markdown string using this method. +The Markdown specified in the [Markdown examples](#markdown-examples) is used to automatically generate HTML in +[`glfm_specification/example_snapshots/html.yml`](#glfm_specificationexample_snapshotshtmlyml) via +[`update-example-snapshots.rb`](#update-example-snapshotsrb-script). These examples are +used when running [Markdown snapshot testing](#markdown-snapshot-testing). + #### WYSIWYG HTML **WYSIWYG HTML** is HTML produced by the frontend (JavaScript) Content Editor, which includes parsing and rendering logic. It is used to present an editable document in the ProseMirror WYSIWYG editor. +Just like static HTML, +the Markdown specified in the [Markdown examples](#markdown-examples) is used to automatically generate HTML in +[`glfm_specification/example_snapshots/html.yml`](#glfm_specificationexample_snapshotshtmlyml) via +[`update-example-snapshots.rb`](#update-example-snapshotsrb-script). These examples are +used when running [Markdown snapshot testing](#markdown-snapshot-testing). + #### Canonical HTML -**Canonical HTML** is the clean, basic version of HTML rendered from Markdown. +**Canonical HTML** is the clean, basic version of HTML rendered from Markdown, with no +unnecessary classes/elements related to styling or any other implementation-specific behavior. + +Its purpose is to support [Markdown conformance testing](#markdown-conformance-testing) against the +GLFM [`spec.txt`](#spectxt). + +Always hardcoded and manually curated, the HTML is never automatically generated. +The [Markdown examples](#markdown-examples) specifying it are contained +in different files depending on which [Markdown specification](#various-markdown-specifications) +a given example originally comes from. + +Canonical HTML is **_always specified_** for all [Markdown examples](#markdown-examples) +in the CommonMark, <abbr title="GitHub Flavored Markdown">GFM</abbr>, and <abbr title="GitLab Flavored Markdown">GLFM</abbr> +[official specifications](#official-specifications). + +However, it is **_never specified_** for GLFM [internal extensions](#internal-extensions) in the [Markdown examples](#markdown-examples). +**This is because the internal extensions are never tested via [Markdown conformance testing](#markdown-conformance-testing). +Therefore, canonical HTML for internal extension examples is never used by any scripts or automated testing.** + +Here are more details on the sources of canonical HTML examples: -1. For the examples which come from the CommonMark specification and +1. For the examples which are part of the CommonMark specification and GFM extensions specification, the canonical HTML is the exact identical HTML found in the - GFM `spec.txt` example blocks. -1. For GLFM extensions to the <abbr title="GitHub Flavored Markdown">GFM</abbr> / CommonMark - specification, a `glfm_canonical_examples.txt` [input specification file](#input-specification-files) - contains the Markdown examples and corresponding canonical HTML examples. + [GFM `spec.txt`](https://github.com/github/cmark-gfm/blob/master/test/spec.txt) [Markdown example](#markdown-examples) blocks. + These examples are copied verbatim from the GFM `spec.txt` into the GLFM + version of [`spec.txt`](#spectxt). +1. For the examples which are part of the GLFM [_official specification_](#official-specifications), + the canonical HTML is manually maintained and curated via the examples contained in the + [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd) [input specification file](#input-specification-files). +1. For the examples which are part of the GLFM [_internal extensions_](#internal-extensions), + the canonical HTML **is never specified**, and **must be left empty in all examples** contained in + the [`glfm_internal_extension_examples.md`](#glfm_internal_extension_examplesmd) [input specification file](#input-specification-files). ### Canonicalization of HTML @@ -309,7 +398,11 @@ The rendered [static HTML](#static-html) and [WYSIWYG HTML](#wysiwyg-html) from the backend (Ruby) and frontend (JavaScript) renderers usually contains extra styling or HTML elements, to support specific appearance and behavioral requirements. -Neither the backend nor the frontend rendering logic can directly render the clean, basic canonical HTML. +Neither the backend nor the frontend rendering logic can directly render the clean, basic HTML +which is necessary to perform comparison to the [canonical HTML](#canonical-html) +when running [Markdown conformance testing](#markdown-conformance-testing) +for the [GLFM official specification examples](#glfm_official_specification_examplesmd). + Nor should they be able to, because: - It's not a direct requirement to support any GitLab application feature. @@ -318,16 +411,9 @@ Nor should they be able to, because: Instead, the rendered static or WYSIWYG HTML is converted to canonical HTML by a _canonicalization_ process. This process can strip all the extra styling and behavioral HTML from the static or WYSIWYG HTML, resulting in canonical HTML which exactly -matches the Markdown + HTML examples in a standard `spec.txt` specification. +matches the canonical HTML examples in a standard `spec.txt` specification. Use the [`canonicalize-html.rb` script](#canonicalize-htmlrb-script) for this process. -More explanation about this canonicalization process in the sections below. - -NOTE: -Some of the static or WYSIWYG HTML examples may not be representable as canonical -HTML. (For example, when they are represented as an image.) In these cases, the Markdown -conformance test for the example can be skipped by setting `skip_update_example_snapshots: true` -for the example in `glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml`. ### Normalization @@ -416,7 +502,7 @@ of how the normalizations are specified. ## Goals -Given the constraints above, we have a few goals related to the GLFM +Given all the constraints above, we can summarize the various goals related to the GLFM specification and testing infrastructure: 1. A canonical `spec.txt` exists, and represents the official specification for @@ -425,17 +511,18 @@ specification and testing infrastructure: (GFM) specification, just as <abbr title="GitHub Flavored Markdown">GFM</abbr> is a strict superset [of the CommonMark specification](https://github.github.com/gfm/#what-is-github-flavored-markdown-). - Therefore, it contains the superset of all canonical Markdown + HTML examples - for CommonMark, GFM, and GLFM. + 1. Therefore, it contains the superset of all [Markdown examples](#markdown-examples) + for CommonMark and GFM, as well as the GLFM + [official specification](#official-specifications) and [internal extensions](#internal-extensions). 1. It contains a prose introduction section which is specific to GitLab and GLFM. 1. It contains all other non-introduction sections verbatim from the - GFM - `spec.txt`. - 1. It contains a new extra section for the GLFM GitLab-specific extensions, - with both prose and examples describing the extensions. - 1. It should be in the standard format which can processed by the standard - CommonMark tools [`spec_tests.py`](https://github.com/github/cmark-gfm/blob/master/test/spec_tests.py), - which is a [script used to run the Markdown conformance tests](https://github.github.com/gfm/#about-this-document) + [GFM specification](#github-flavored-markdown-specification). + 1. It contains new, extra sections for all the additional Markdown contained in the GLFM + [official specification](#official-specifications) and [internal extensions](#internal-extensions), + with [Markdown examples](#markdown-examples) and accompanying prose, just like the CommonMark and GFM examples. + 1. All its headers and [Markdown examples](#markdown-examples) should be in the standard format which can be processed by the standard + CommonMark tool [`spec_tests.py`](https://github.com/github/cmark-gfm/blob/master/test/spec_tests.py) used to perform + [Markdown conformance testing](#markdown-conformance-testing) against all examples contained in a `spec.txt`. 1. The GLFM parsers and HTML renderers for both the static backend (Ruby) and WYSIWYG frontend (JavaScript) implementations @@ -445,9 +532,9 @@ specification and testing infrastructure: NOTE: Consistent does not mean that both of these implementations render to the identical HTML. They each have different implementation-specific additions - to the HTML they render, so therefore their rendered HTML is - ["canonicalized"](#canonicalization-of-html) to canonical HTML prior running - the Markdown conformance tests. + to the HTML they render, so their rendered HTML is + ["canonicalized"](#canonicalization-of-html) to canonical HTML prior to running + [Markdown conformance testing](#markdown-conformance-testing). 1. For _both_ the static backend (Ruby) and WYSIWYG frontend (JavaScript) implementations, a set of example snapshots exists in the form of YAML files, which correspond to every Markdown example in the GLFM `spec.txt`. These example snapshots @@ -519,7 +606,7 @@ them from the corresponding implementation class entry point files under #### `update-specification.rb` script -The `scripts/glfm/update-specification.rb` script uses specification input files to +The `scripts/glfm/update-specification.rb` script uses [input specification files](#input-specification-files) to generate and update `spec.txt` (Markdown) and `spec.html` (HTML). The `spec.html` is generated by passing the generated (or updated) `spec.txt` Markdown to the backend API for rendering to static HTML: @@ -531,14 +618,15 @@ subgraph script: A --> B{Backend Markdown API} end subgraph input:<br/>input specification files - C[ghfm_spec_v_0.29.txt] --> A - D[glfm_intro.txt] --> A - E[glfm_canonical_examples.txt] --> A + C[ghfm_spec_v_0.29.md] --> A + D[glfm_intro.md] --> A + E[glfm_official_specification_examples.md] --> A + F[glfm_internal_extension_examples.md] --> A end subgraph output:<br/>GLFM specification files - A --> F[spec.txt] - F --> B - B --> G[spec.html] + A --> G[spec.txt] + G --> B + B --> H[spec.html] end ``` @@ -650,38 +738,65 @@ subgraph output:<br/>test results/output end ``` +#### `verify-all-generated-files-are-up-to-date.rb` script + +The `scripts/glfm/verify-all-generated-files-are-up-to-date.rb` script +runs the [`update-specification.rb`](#update-specificationrb-script). +[`update-example-snapshots.rb`](#update-example-snapshotsrb-script) scripts, +It fails with an exception and non-zero return code if running these scripts +results in any diffs to the generated and committed +[output specification files](#output-specification-files) or +[example snapshot files](#example-snapshot-files). + +This script is run via the `glfm-verify` CI job to ensure that all changes to the +[input specification files](#input-specification-files) +are reflected in the generated output specification and example snapshot files. + ### Specification files These files represent the GLFM specification itself. They are all -located under the root `glfm_specification`, and are further divided into two -subfolders: - -- `input`: Contains files which are imported or manually edited. -- `output`: Contains files which are automatically generated. +located under the root `glfm_specification` and are further divided into +subcategories based on their usage and purpose: + +- `glfm_specification` + - `input`: Contains files that are downloaded or manually edited. + These are the original input to drive all other automated GLFM + specification scripts, processes, or tests. + - `github_flavored_markdown`: Contains only the downloaded and committed + [`ghfm_spec_v_0.29.md`](#github-flavored-markdown-specification) specification. + - `gitlab_flavored_markdown`: Contains all `glfm_*` files. + - `*.md` [input specification files](#input-specification-files), + which represent the GLFM specification itself. + - `*.yml` [input specification configuration files](#input-specification-configuration-files), + which control various aspects of the automated GLFM scripts and processes. + - `output`: Contains [output specification files](#output-specification-files), + which are automatically generated from the + input files by running the [`update-specification.rb`](#update-specificationrb-script) script. #### Input specification files -The `glfm_specification/input` directory contains files which are the original -input to drive all other automated GLFM specification scripts/processes/tests. -They are either downloaded, as in the case of the -GFM `spec.txt` file, or manually -updated, as in the case of all GFM files. +Input specification files are manually curated Markdown files that represent the specification itself. +They are located at `glfm_specification/input/github_flavored_markdown/*.md` and +`glfm_specification/input/gitlab_flavored_markdown/*.md`. + +See the main [specification files](#specification-files) section for more context and details. ##### GitHub Flavored Markdown specification -[`glfm_specification/input/github_flavored_markdown/ghfm_spec_v_0.29.txt`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/github_flavored_markdown/ghfm_spec_v_0.29.txt) -is the official latest [GFM `spec.txt`](https://github.com/github/cmark-gfm/blob/master/test/spec.txt). +[`glfm_specification/input/github_flavored_markdown/ghfm_spec_v_0.29.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/github_flavored_markdown/ghfm_spec_v_0.29.md) +is a copy of the official latest [GFM `spec.txt`](https://github.com/github/cmark-gfm/blob/master/test/spec.txt). -- It is automatically downloaded and updated by `update-specification.rb` script. +- It is automatically downloaded and updated by the `update-specification.rb` script. - When it is downloaded, the version number is added to the filename. +- The extension is changed from `*.txt` to `*.md` so that it can be handled better by Markdown editors. NOTE: -This file uses the `ghfm` acronym instead of `gfm`, as +For extra clarity, this file uses the `ghfm` acronym in its name instead of `gfm`, as explained in the [Acronyms section](#acronyms-glfm-ghfm-gfm-commonmark). -##### `glfm_intro.txt` +##### `glfm_intro.md` -[`glfm_specification/input/gitlab_flavored_markdown/glfm_intro.txt`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_intro.txt) +[`glfm_specification/input/gitlab_flavored_markdown/glfm_intro.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_intro.md) is the GitLab-specific version of the prose in the introduction section of the GLFM specification. - It is manually updated. @@ -690,13 +805,24 @@ is the GitLab-specific version of the prose in the introduction section of the G ##### `glfm_canonical_examples.txt` -[`glfm_specification/input/gitlab_flavored_markdown/glfm_canonical_examples.txt`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_canonical_examples.txt) -is the manually updated canonical Markdown+HTML examples for GLFM extensions. +The `glfm_canonical_examples.txt` file is deprecated and no longer exists. It has been replaced by two files: -- It contains examples in the [standard backtick-delimited `spec.txt` format](#various-markdown-specifications), - each of which contain a Markdown example and the corresponding canonical HTML. +- [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd) + which contains the [GLFM official specification](#official-specifications) examples. +- [`glfm_internal_extension_examples.md`](#glfm_internal_extension_examplesmd) + which contains the [GLFM internal extension](#internal-extensions) examples. + +##### `glfm_official_specification_examples.md` + +[`glfm_specification/input/gitlab_flavored_markdown/glfm_official_specification_examples.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_official_specification_examples.md) +consists of the manually updated Markdown+HTML examples for the +[GLFM official specification](#official-specifications), and their associated documentation and descriptions. + +- It contains [Markdown examples](#markdown-examples) in the + [standard backtick-delimited `spec.txt` format](#various-markdown-specifications), + each of which contains Markdown and the corresponding canonical HTML that should be rendered. - For all GitLab examples, the "extension" annotation after the backticks should consist of only - `example gitlab`. It does not currently include any additional extension annotations describing + `example`. It does not currently include any additional extension annotations describing the specific Markdown, unlike the GitHub Flavored Markdown examples, which do include these additional annotations (such as `example strikethrough`). - The `update-specification.rb` script inserts it as new sections before the appendix @@ -706,7 +832,53 @@ is the manually updated canonical Markdown+HTML examples for GLFM extensions. - `H3` header sections must be nested within `H2` header sections. They cannot be nested directly within `H1` header sections. -`glfm_specification/input/gitlab_flavored_markdown/glfm_canonical_examples.txt` sample entries: +`glfm_specification/input/gitlab_flavored_markdown/glfm_official_specification_examples.md` sample entries: + +<!-- markdownlint-disable MD048 --> + +~~~plaintext +# Section with GLFM official specification examples + +## Strong + +### Strong with two asterisks + +```````````````````````````````` example +**bold** +. +<p><strong>bold</strong></p> +```````````````````````````````` + +### Strong with HTML + +```````````````````````````````` example +<strong> +bold +</strong> +. +<p><strong> +bold +</strong></p> +```````````````````````````````` +~~~ + +<!-- markdownlint-enable MD048 --> + +##### `glfm_internal_extension_examples.md` + +[`glfm_specification/input/gitlab_flavored_markdown/glfm_internal_extension_examples.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_internal_extension_examples.md) +consists of the manually updated Markdown examples for the +[GLFM internal extensions](#internal-extensions), and their associated documentation and descriptions. + +Its general format is identical to [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd), +consisting of `H1`, `H2`, or `H3` sections containing [Markdown examples](#markdown-examples) in the +[standard backtick-delimited `spec.txt` format](#various-markdown-specifications). + +However, as described in the [canonical HTML section](#canonical-html), only the Markdown portion of each +example is specified, and the HTML portion is left empty, because internal extension examples are +never used for [Markdown conformance testing](#markdown-conformance-testing). + +`glfm_specification/input/gitlab_flavored_markdown/glfm_official_specification_examples.md` sample entries: NOTE: All lines in this example are prefixed with a `|` character. This prefix helps avoid false @@ -714,31 +886,40 @@ errors when this file is checked by `markdownlint`, and possible errors in other The actual file should not have these prefixed `|` characters. ```plaintext -|# First GitLab-Specific Section with Examples +|# Section with GLFM Internal Extension Examples | -|## Strong but with two asterisks +|## Video | -|```````````````````````````````` example gitlab -|**bold** +|```````````````````````````````` example +| |. -|<p><strong>bold</strong></p> -|```````````````````````````````` -| -|# Second GitLab-Specific Section with Examples -| -|## Strong but with HTML -| -|```````````````````````````````` example gitlab -|<strong> -|bold -|</strong> -|. -|<p><strong> -|bold -|</strong></p> |```````````````````````````````` ``` +#### Input specification configuration files + +Input specification configuration files are manually curated YAML files that control various +aspects of the automated GLFM scripts and processes. They are located at +`glfm_specification/input/gitlab_flavored_markdown/*.yml`. + +See the main [specification files](#specification-files) section for more context and details. + +##### Configuration file validation + +All of the manually curated example names in the configuration files must correspond to +an existing [Markdown example](#markdown-examples) name found in +[`example_snapshots/examples_index.yml`](#glfm_specificationexample_snapshotsexamples_indexyml), +which is automatically generated based on the [input specification files](#input-specification-files). + +If there is an invalid reference to an example name that does not exist, the +[`scripts/glfm/update-example-snapshots.rb`](#update-example-snapshotsrb-script) +script fails with a descriptive error. + +The only exceptions to this validation are example names beginning with `00_`, which are reserved +for [YAML aliases](https://yaml.org/spec/1.2.2/#692-node-anchors). +See the section on [`glfm_example_normalizations.yml`](#glfm_example_normalizationsyml) +for more details and examples. + ##### `glfm_example_status.yml` [`glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml) @@ -801,8 +982,8 @@ The following optional entries are supported for each example. They all default ##### `glfm_example_normalizations.yml` [`glfm_specification/input/gitlab_flavored_markdown/glfm_example_normalizations.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_example_normalizations.yml) -controls the [normalization](#normalization) process. It allows one or more `regex`/`replacement` pairs -to be specified for a Markdown example. +is used to control the [fixture-based normalization](#fixture-based-normalization) process. +It allows one or more `regex`/`replacement` pairs to be specified for a Markdown example. - It is manually updated. - It has a nested structure corresponding to the example and type of entry it refers to. @@ -811,6 +992,11 @@ to be specified for a Markdown example. - The YAML anchors use a naming convention based on the index number of the example, to ensure unique anchor names and avoid naming conflicts. +NOTE: +Other approaches to [normalization](#normalization) such as [fixture-based normalization](#fixture-based-normalization) +or [environment-variable-based normalization](#environment-variable-based-normalization) are always preferable to +[fixture-based normalization](#fixture-based-normalization). + `glfm_specification/input/gitlab_flavored_markdown/glfm_example_normalizations.yml` sample entries: ```yaml @@ -891,11 +1077,11 @@ allows control over other aspects of the snapshot example generation process. #### Output specification files The `glfm_specification/output` directory contains the CommonMark standard format -`spec.txt` file which represents the canonical GLFM specification which is generated +`spec.txt` file which represents the GLFM specification which is generated by the `update-specification.rb` script. It also contains the rendered `spec.html` which is generated based on the `spec.txt` as input. -These output `spec.*` files, which represent the official, canonical GLFM specification, +These output `spec.*` files, which represent the GLFM specification, are colocated under the same parent folder `glfm_specification` with the other `input` specification files. They're located here both for convenience, and because they are all a mix of manually edited and generated files. @@ -909,12 +1095,16 @@ move or copy a hosted version of the rendered HTML `spec.html` version to anothe [`glfm_specification/output/spec.txt`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output/spec.txt) is a Markdown specification file, in the standard format -with prose and Markdown + canonical HTML examples. It is generated or updated by the -`update-specification.rb` script. +with prose and Markdown + canonical HTML examples. It also serves as input for other scripts such as `update-example-snapshots.rb` and `run-spec-tests.sh`. +It is generated or updated by the `update-specification.rb` script, using the +[input specification files](#input-specification-files) as input. +See the [`update-specification.rb` script section](#update-specificationrb-script) +for a diagram and more description on this process. + ##### spec.html [`glfm_specification/output/spec.html`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output/spec.html) @@ -945,14 +1135,14 @@ be manually edited as necessary to help drive the implementations. [`glfm_specification/example_snapshots/examples_index.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/example_snapshots/examples_index.yml) is the main list of all -CommonMark, GFM, and GLFM example names, each with a unique canonical name. +CommonMark, GFM, and GLFM example names, each with a uniquely identifying name. - It is generated from the hierarchical sections and examples in the GFM `spec.txt` specification. - For CommonMark and GFM examples, these sections originally came from the GFM `spec.txt`. -- For GLFM examples, it is generated from `glfm_canonical_examples.txt`, which is - the additional Section 7 in the GLFM `spec.txt`. +- For GLFM examples, it is generated from + [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd) and [`glfm_internal_extension_examples.md`](#glfm_internal_extension_examplesmd). - It also contains extra metadata about each example, such as: 1. `spec_txt_example_position` - The position of the example in the generated GLFM `spec.txt` file. - This value is the index order of each individual Markdown + HTML5 example in the file. It is _not_ @@ -998,8 +1188,8 @@ for each entry in `glfm_specification/example_snapshots/examples_index.yml` it is generated (or updated) from the standard GFM `spec.txt` using the `update-example-snapshots.rb` script. - For GLFM, it is generated (or updated) from the - `glfm_specification/input/gitlab_flavored_markdown/glfm_canonical_examples.txt` - input specification file. + [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd) and [`glfm_internal_extension_examples.md`](#glfm_internal_extension_examplesmd) + input specification files. `glfm_specification/example_snapshots/markdown.yml` sample entry: @@ -1018,8 +1208,8 @@ Three types of entries exist, with different HTML for each: - **Canonical** - The ["Canonical"](#canonicalization-of-html) HTML. - For CommonMark and GFM examples, the HTML comes from the examples in `spec.txt`. - - For GLFM examples, it is generated/updated from - `glfm_specification/input/gitlab_flavored_markdown/glfm_canonical_examples.txt`. + - For [GLFM official specification](#official-specifications) examples, it is generated/updated from + [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd). - **Static** - This is the static (backend (Ruby)-generated) HTML for each entry in `glfm_specification/example_snapshots/examples_index.yml`. @@ -1094,13 +1284,14 @@ This section describes how the scripts can be used to manage the GLFM specificat 1. Run [`update-specification.rb`](#update-specificationrb-script) to update the GLFM specification [output specification files](#output-specification-files). 1. Visually inspect and confirm any resulting changes to the [output specification files](#output-specification-files). -1. Run [`run-spec-tests.sh`](http://gdk.test:3005/ee/development/gitlab_flavored_markdown/specification_guide/index.html#run-spec-testssh-script) to run the conformance tests against the canonicalized GLFM specification. +1. Run [`run-spec-tests.sh`](#run-spec-testssh-script) to run the conformance tests against the canonicalized GLFM specification. 1. Commit any changes to the [output specification files](#output-specification-files). ### Update the example snapshots and run snapshot tests 1. If you are working on an in-progress feature or bug, make any necessary manual updates to the [input specification files](#input-specification-files). This may include: - 1. Updating the canonical Markdown or HTML examples in `glfm_specification/input/gitlab_flavored_markdown/glfm_canonical_examples.txt`. + 1. Updating the canonical Markdown or HTML examples in + [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd) or [`glfm_internal_extension_examples.md`](#glfm_internal_extension_examplesmd). 1. Updating `glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml` to reflect the current status of the examples or tests. 1. Run [`update-specification.rb`](#update-specificationrb-script) to update the `spec.txt` to reflect any changes which were made to the [input specification files](#input-specification-files). 1. Visually inspect and confirm any resulting changes to the [output specification files](#output-specification-files). diff --git a/doc/development/go_guide/dependencies.md b/doc/development/go_guide/dependencies.md index 7cad5bbf417..f2b1ae9e7b8 100644 --- a/doc/development/go_guide/dependencies.md +++ b/doc/development/go_guide/dependencies.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Dependency Management in Go diff --git a/doc/development/go_guide/go_upgrade.md b/doc/development/go_guide/go_upgrade.md index 4e2a0d95910..d931140d9da 100644 --- a/doc/development/go_guide/go_upgrade.md +++ b/doc/development/go_guide/go_upgrade.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Managing Go versions @@ -158,6 +158,8 @@ if you need help finding the correct person or labels: | GitLab Quality Images | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-build-images/-/issues) | | GitLab Shell | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-shell/-/issues) | | GitLab Workhorse | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) | +| GitLab Browser-based DAST | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) | +| GitLab Coverage Fuzzer | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) | | LabKit | [Issue Tracker](https://gitlab.com/gitlab-org/labkit/-/issues) | | [Node Exporter](https://github.com/prometheus/node_exporter) | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) | | [PgBouncer Exporter](https://github.com/prometheus-community/pgbouncer_exporter) | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) | diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 3adafa8750f..197c7616d82 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Go standards and style guidelines @@ -357,7 +357,7 @@ Every binary ideally must have structured (JSON) logging in place as it helps with searching and filtering the logs. At GitLab we use structured logging in JSON format, as all our infrastructure assumes that. When using [Logrus](https://github.com/sirupsen/logrus) you can turn on structured -logging simply by using the build in [JSON formatter](https://github.com/sirupsen/logrus#formatters). This follows the +logging by using the build in [JSON formatter](https://github.com/sirupsen/logrus#formatters). This follows the same logging type we use in our [Ruby applications](../logging.md#use-structured-json-logging). #### How to use Logrus @@ -436,6 +436,25 @@ of the Code Review Comments page on the Go wiki for more details. Most editors/IDEs allow you to run commands before/after saving a file, you can set it up to run `goimports -local gitlab.com/gitlab-org` so that it's applied to every file when saving. +### Naming branches + +Only use the characters `a-z`, `0-9` or `-` in branch names. This restriction is due to the fact that `go get` doesn't work as expected when a branch name contains certain characters, such as a slash `/`: + +```shell +$ go get -u gitlab.com/gitlab-org/security-products/analyzers/report/v3@some-user/some-feature + +go get: gitlab.com/gitlab-org/security-products/analyzers/report/v3@some-user/some-feature: invalid version: version "some-user/some-feature" invalid: disallowed version string +``` + +If a branch name contains a slash, it forces us to refer to the commit SHA instead, which is less flexible. For example: + +```shell +$ go get -u gitlab.com/gitlab-org/security-products/analyzers/report/v3@5c9a4279fa1263755718cf069d54ba8051287954 + +go: downloading gitlab.com/gitlab-org/security-products/analyzers/report/v3 v3.15.3-0.20221012172609-5c9a4279fa12 +... +``` + ### Initializing slices If initializing a slice, provide a capacity where possible to avoid extra diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index af11340737f..25651639170 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Gotchas diff --git a/doc/development/graphql_guide/authorization.md b/doc/development/graphql_guide/authorization.md index 717a6d29fbc..7b69ce36071 100644 --- a/doc/development/graphql_guide/authorization.md +++ b/doc/development/graphql_guide/authorization.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GraphQL Authorization diff --git a/doc/development/graphql_guide/batchloader.md b/doc/development/graphql_guide/batchloader.md index 492d3bc9007..ef0b97f4f62 100644 --- a/doc/development/graphql_guide/batchloader.md +++ b/doc/development/graphql_guide/batchloader.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GraphQL BatchLoader @@ -134,7 +134,7 @@ z.sync ``` NOTE: -There is no dependency analysis in the use of batch-loading. There is simply +There is no dependency analysis in the use of batch-loading. There is a pending queue of requests, and as soon as any one result is needed, all pending requests are evaluated. @@ -146,7 +146,7 @@ def publisher ::Gitlab::Graphql::Loaders::BatchModelLoader.new(::Publisher, object.publisher_id).find end -# Here we need the publisher in order to generate the catalog URL +# Here we need the publisher to generate the catalog URL def catalog_url ::Gitlab::Graphql::Lazy.with_value(publisher) do |p| UrlHelpers.book_catalog_url(publisher, object.isbn) diff --git a/doc/development/graphql_guide/graphql_pro.md b/doc/development/graphql_guide/graphql_pro.md index 3170f0cfdc2..ec28ceb4f20 100644 --- a/doc/development/graphql_guide/graphql_pro.md +++ b/doc/development/graphql_guide/graphql_pro.md @@ -1,7 +1,7 @@ --- -stage: Ecosystem +stage: Manage group: Integrations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GraphQL Pro @@ -16,6 +16,6 @@ The main purpose is for support. Per the website: > creator and maintainer. Get prioritized support for issues and requests. Note that we **cannot** use the Pro version directly in our product, since we are -an Open Core product - we can not require customers to purchase the Pro version, nor can we ship it. +an Open Core product - we cannot require customers to purchase the Pro version, nor can we ship it. Details on the billing account and gem licensing can be found in the Engineering 1Password vault. diff --git a/doc/development/graphql_guide/index.md b/doc/development/graphql_guide/index.md index 412825e06d3..9c6a2559e5c 100644 --- a/doc/development/graphql_guide/index.md +++ b/doc/development/graphql_guide/index.md @@ -1,7 +1,7 @@ --- -stage: Ecosystem +stage: Manage group: Integrations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GraphQL development guidelines diff --git a/doc/development/graphql_guide/monitoring.md b/doc/development/graphql_guide/monitoring.md index 28d1a4a9046..1e4c083653e 100644 --- a/doc/development/graphql_guide/monitoring.md +++ b/doc/development/graphql_guide/monitoring.md @@ -1,7 +1,7 @@ --- -stage: Ecosystem +stage: Manage group: Integrations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Monitoring GraphQL diff --git a/doc/development/graphql_guide/pagination.md b/doc/development/graphql_guide/pagination.md index 72f321a1fd2..7b9d2158c60 100644 --- a/doc/development/graphql_guide/pagination.md +++ b/doc/development/graphql_guide/pagination.md @@ -1,7 +1,7 @@ --- -stage: Ecosystem +stage: Manage group: Integrations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GraphQL pagination diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index ea22b33f1bc..158eb19764b 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -1,7 +1,7 @@ --- stage: Manage group: Import -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Internationalization for GitLab diff --git a/doc/development/i18n/index.md b/doc/development/i18n/index.md index 0870a0e6750..757bcc0ebf1 100644 --- a/doc/development/i18n/index.md +++ b/doc/development/i18n/index.md @@ -1,7 +1,7 @@ --- stage: Manage group: Import -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Translate GitLab to your language diff --git a/doc/development/i18n/merging_translations.md b/doc/development/i18n/merging_translations.md index 43f19f8a9d6..2a086c796c9 100644 --- a/doc/development/i18n/merging_translations.md +++ b/doc/development/i18n/merging_translations.md @@ -1,7 +1,7 @@ --- stage: Manage group: Import -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Merging translations from Crowdin @@ -79,7 +79,7 @@ recreate it with the following steps: ## Manually update the translation levels There's no automated way to pull the translation levels from Crowdin, to display -this information in the language selection dropdown. Therefore, the translation +this information in the language selection dropdown list. Therefore, the translation levels are hard-coded in the `TRANSLATION_LEVELS` constant in [`i18n.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/i18n.rb), and must be regularly updated. diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index f986a852567..b5d57f9912b 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -1,7 +1,7 @@ --- stage: Manage group: Import -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Proofread Translations @@ -65,6 +65,7 @@ are very appreciative of the work done by translators and proofreaders! - Michael Hahnle - [GitLab](https://gitlab.com/mhah), [Crowdin](https://crowdin.com/profile/mhah) - Katrin Leinweber - [GitLab](https://gitlab.com/katrinleinweber), [Crowdin](https://crowdin.com/profile/katrinleinweber) - Vladislav Wanner - [GitLab](https://gitlab.com/RumBugen), [Crowdin](https://crowdin.com/profile/RumBugen) + - Daniel Ziegenberg - [GitLab](https://gitlab.com/ziegenberg), [Crowdin](https://crowdin.com/profile/ziegenberg) - Greek - Proofreaders needed. - Hebrew diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index 61f9d7a25c3..2c8d7eac2a6 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -1,7 +1,7 @@ --- stage: Manage group: Import -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Translating GitLab diff --git a/doc/development/image_scaling.md b/doc/development/image_scaling.md index 2078db8294c..4b19c21a457 100644 --- a/doc/development/image_scaling.md +++ b/doc/development/image_scaling.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Application Performance -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Image scaling guide @@ -13,7 +13,7 @@ For a general introduction to the history of image scaling at GitLab, you might ## Why image scaling? -Since version 13.6, GitLab scales down images on demand in order to reduce the page data footprint. +Since version 13.6, GitLab scales down images on demand to reduce the page data footprint. This both reduces the amount of data "on the wire", but also helps with rendering performance, since the browser has less work to do. @@ -68,7 +68,7 @@ controller mixin. Upon receiving a request coming from a client through Workhors it should trigger the image scaler as per the criteria mentioned above, and if so, render a special response header field (`Gitlab-Workhorse-Send-Data`) with the necessary parameters for Workhorse to carry out the scaling request. If Rails decides the request does not constitute a valid image scaling request, -we simply follow the path we take to serve any ordinary upload. +we follow the path we take to serve any ordinary upload. ### Workhorse diff --git a/doc/development/import_export.md b/doc/development/import_export.md index c66ac0418ac..f864dd3b678 100644 --- a/doc/development/import_export.md +++ b/doc/development/import_export.md @@ -1,7 +1,7 @@ --- stage: Manage group: Import -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Import/Export development documentation diff --git a/doc/development/import_project.md b/doc/development/import_project.md index 1f3bf860257..b879d635350 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Test Import Project @@ -16,7 +16,7 @@ There are several ways to import a project. ### Importing via UI -The first option is to simply [import the Project tarball file via the GitLab UI](../user/project/settings/import_export.md#import-a-project-and-its-data): +The first option is to [import the Project tarball file via the GitLab UI](../user/project/settings/import_export.md#import-a-project-and-its-data): 1. Create the group `qa-perf-testing` 1. Import the [GitLab FOSS project tarball](https://gitlab.com/gitlab-org/quality/performance-data/-/blob/master/projects_export/gitlabhq_export.tar.gz) into the Group. @@ -59,7 +59,7 @@ This script was introduced in GitLab 12.6 for importing large GitLab project exp As part of this script we also disable direct and background upload to avoid situations where a huge archive is being uploaded to GCS (while being inside a transaction, which can cause idle transaction timeouts). -We can simply run this script from the terminal: +We can run this script from the terminal: Parameters: @@ -111,7 +111,7 @@ The specified project export file in `archive_path` is missing. ##### `Exception: Permission denied @ rb_sysopen - (filename)` -The specified project export file can not be accessed by the `git` user. +The specified project export file cannot be accessed by the `git` user. Setting the file owner to `git:git`, changing the file permissions to `0400`, and moving it to a public folder (for example `/tmp/`) fixes the issue. diff --git a/doc/development/index.md b/doc/development/index.md index 34e6f466664..d4a34051bc8 100644 --- a/doc/development/index.md +++ b/doc/development/index.md @@ -3,7 +3,7 @@ comments: false type: index, dev stage: none group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" description: "Development Guidelines: learn how to contribute to GitLab." --- diff --git a/doc/development/integrations/codesandbox.md b/doc/development/integrations/codesandbox.md index 8fd1f0e331d..d7d0ea48db0 100644 --- a/doc/development/integrations/codesandbox.md +++ b/doc/development/integrations/codesandbox.md @@ -1,7 +1,7 @@ --- stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Set up local CodeSandbox development environment diff --git a/doc/development/integrations/index.md b/doc/development/integrations/index.md index 0387ba2e4dd..a0a81775391 100644 --- a/doc/development/integrations/index.md +++ b/doc/development/integrations/index.md @@ -1,7 +1,7 @@ --- -stage: Ecosystem +stage: Manage group: Integrations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: "GitLab's development guidelines for Integrations" --- diff --git a/doc/development/integrations/jenkins.md b/doc/development/integrations/jenkins.md index bb541d26df2..f314f4536e2 100644 --- a/doc/development/integrations/jenkins.md +++ b/doc/development/integrations/jenkins.md @@ -1,7 +1,7 @@ --- -stage: Ecosystem +stage: Manage group: Integrations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # How to run Jenkins in development environment (on macOS) **(FREE)** diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index ade81e29ffb..fb0d239db46 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -1,7 +1,7 @@ --- -stage: Ecosystem +stage: Manage group: Integrations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Set up a development environment **(FREE)** diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 2c5dd1c0500..f40caf29cfa 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -1,7 +1,7 @@ --- stage: Secure group: Static Analysis -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Security scanner integration @@ -180,7 +180,7 @@ Here are some examples to get you started: As documented in the [Docker Official Images](https://github.com/docker-library/official-images#tags-and-aliases) project, it is strongly encouraged that version number tags be given aliases which allows the user to easily refer to the "most recent" release of a particular series. -See also [Docker Tagging: Best practices for tagging and versioning Docker images](https://docs.microsoft.com/en-us/archive/blogs/stevelasker/docker-tagging-best-practices-for-tagging-and-versioning-docker-images). +See also [Docker Tagging: Best practices for tagging and versioning Docker images](https://learn.microsoft.com/en-us/archive/blogs/stevelasker/docker-tagging-best-practices-for-tagging-and-versioning-docker-images). ## Command line @@ -358,6 +358,10 @@ analyzer to use the latest available schemas. After the deprecation period for a schema version, the file is removed from GitLab. Reports that declare removed versions are rejected, and an error message displays on the corresponding pipeline. +If a report uses a `PATCH` version that doesn't match any vendored schema version, it is validated against +the latest vendored `PATCH` version. For example, if a report version is 14.0.23 and the latest vendored +version is 14.0.6, the report is validated against version 14.0.6. + GitLab uses the [`json_schemer`](https://www.rubydoc.info/gems/json_schemer) gem to perform validation. @@ -430,10 +434,29 @@ The value of the `category` field matches the report type: - `sast` - `dast` -##### Scanner +##### Scan + +The `scan` field is an object that embeds meta information about the scan itself: the `analyzer` +and `scanner` that performed the scan, the `start_time` and `end_time` the scan executed, +and `status` of the scan (either "success" or "failure"). + +Both the `analyzer` and `scanner` fields are objects that embeds a human-readable `name` and a technical `id`. +The `id` should not collide with any other analyzers or scanners another integrator would provide. + +##### Scan Primary Identifiers + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368284) in GitLab 15.4 [with a flag](../../administration/feature_flags.md) named `sec_mark_dropped_findings_as_resolved`. Disabled by default. + +The `scan.primary_identifiers` field is an optional field containing an array of +[primary identifiers](../../user/application_security/terminology/index.md#primary-identifier)). +This is an exhaustive list of all rulesets for which the analyzer performed the scan. + +Even when the [`Vulnerabilities`](#vulnerabilities) array for a given scan may be empty, this optional field +should contain the complete list of potential identifiers to inform the Rails application of which +rules were executed. -The `scanner` field is an object that embeds a human-readable `name` and a technical `id`. -The `id` should not collide with any other scanner another integrator would provide. +When populated, the Rails application will automatically resolve previously detected vulnerabilities as no +longer relevant when their primary identifier is not included. ##### Name, message, and description diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index 63f86a3f95d..ba0654971d3 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -1,7 +1,7 @@ --- stage: Secure group: Static Analysis -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Secure Partner Integration - Onboarding Process diff --git a/doc/development/interacting_components.md b/doc/development/interacting_components.md index c137faf464c..f4becd06245 100644 --- a/doc/development/interacting_components.md +++ b/doc/development/interacting_components.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Developing against interacting components or features diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index c35d40a7b7f..9b77a20a0d6 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -1,13 +1,13 @@ --- stage: Create group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" type: reference, api --- # Internal API **(FREE)** -The internal API is used by different GitLab components, it can not be +The internal API is used by different GitLab components, it cannot be used by other consumers. This documentation is intended for people working on the GitLab codebase. @@ -21,7 +21,7 @@ Before adding a new internal endpoint, consider if the API would potentially be useful to the wider GitLab community and can be made externally accessible. One reason we might favor internal API endpoints sometimes is when using such an endpoint requires -internal data that external actors can not have. For example, in the internal Pages API we might use +internal data that external actors cannot have. For example, in the internal Pages API we might use a secret token that identifies a request as internal or sign a request with a public key that is not available to a wider community. @@ -665,8 +665,7 @@ Example response: ## Subscriptions -The subscriptions endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) -in order to apply subscriptions including trials, and add-on purchases, for personal namespaces or top-level groups within GitLab.com. +The subscriptions endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) to apply subscriptions including trials, and add-on purchases, for personal namespaces or top-level groups within GitLab.com. ### Creating a subscription @@ -966,3 +965,253 @@ Example response: ### Known consumers - CustomersDot + +## SCIM API **(PREMIUM SAAS)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/9388) in GitLab 11.10. + +The SCIM API implements the [RFC7644 protocol](https://www.rfc-editor.org/rfc/rfc7644). As this API is for +**system** use for SCIM provider integration, it is subject to change without notice. + +To use this API, [Group SSO](../../user/group/saml_sso/index.md) must be enabled for the group. +This API is only in use where [SCIM for Group SSO](../../user/group/saml_sso/scim_setup.md) is enabled. It's a prerequisite to the creation of SCIM identities. + +Not to be confused with the [main SCIM API](../../api/scim.md). + +### Get a list of SCIM provisioned users + +This endpoint is used as part of the SCIM syncing mechanism. It only returns +a single user based on a unique ID which should match the `extern_uid` of the user. + +```plaintext +GET /api/scim/v2/groups/:group_path/Users +``` + +Parameters: + +| Attribute | Type | Required | Description | +|:----------|:--------|:---------|:----------------------------------------------------------------------------------------------------------------------------------------| +| `filter` | string | no | A [filter](#available-filters) expression. | +| `group_path` | string | yes | Full path to the group. | +| `startIndex` | integer | no | The 1-based index indicating where to start returning results from. A value of less than one will be interpreted as 1. | +| `count` | integer | no | Desired maximum number of query results. | + +NOTE: +Pagination follows the [SCIM spec](https://www.rfc-editor.org/rfc/rfc7644#section-3.4.2.4) rather than GitLab pagination as used elsewhere. If records change between requests it is possible for a page to either be missing records that have moved to a different page or repeat records from a previous request. + +Example request: + +```shell +curl "https://gitlab.example.com/api/scim/v2/groups/test_group/Users?filter=id%20eq%20%220b1d561c-21ff-4092-beab-8154b17f82f2%22" \ + --header "Authorization: Bearer <your_scim_token>" \ + --header "Content-Type: application/scim+json" +``` + +Example response: + +```json +{ + "schemas": [ + "urn:ietf:params:scim:api:messages:2.0:ListResponse" + ], + "totalResults": 1, + "itemsPerPage": 20, + "startIndex": 1, + "Resources": [ + { + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:User" + ], + "id": "0b1d561c-21ff-4092-beab-8154b17f82f2", + "active": true, + "name.formatted": "Test User", + "userName": "username", + "meta": { "resourceType":"User" }, + "emails": [ + { + "type": "work", + "value": "name@example.com", + "primary": true + } + ] + } + ] +} +``` + +### Get a single SCIM provisioned user + +```plaintext +GET /api/scim/v2/groups/:group_path/Users/:id +``` + +Parameters: + +| Attribute | Type | Required | Description | +|:----------|:--------|:---------|:----------------------------------------------------------------------------------------------------------------------------------------| +| `id` | string | yes | External UID of the user. | +| `group_path` | string | yes | Full path to the group. | + +Example request: + +```shell +curl "https://gitlab.example.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2" \ + --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +``` + +Example response: + +```json +{ + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:User" + ], + "id": "0b1d561c-21ff-4092-beab-8154b17f82f2", + "active": true, + "name.formatted": "Test User", + "userName": "username", + "meta": { "resourceType":"User" }, + "emails": [ + { + "type": "work", + "value": "name@example.com", + "primary": true + } + ] +} +``` + +### Create a SCIM provisioned user + +```plaintext +POST /api/scim/v2/groups/:group_path/Users/ +``` + +Parameters: + +| Attribute | Type | Required | Description | +|:---------------|:----------|:----|:--------------------------| +| `externalId` | string | yes | External UID of the user. | +| `userName` | string | yes | Username of the user. | +| `emails` | JSON string | yes | Work email. | +| `name` | JSON string | yes | Name of the user. | +| `meta` | string | no | Resource type (`User`). | + +Example request: + +```shell +curl --verbose --request POST "https://gitlab.example.com/api/scim/v2/groups/test_group/Users" \ + --data '{"externalId":"test_uid","active":null,"userName":"username","emails":[{"primary":true,"type":"work","value":"name@example.com"}],"name":{"formatted":"Test User","familyName":"User","givenName":"Test"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],"meta":{"resourceType":"User"}}' \ + --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +``` + +Example response: + +```json +{ + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:User" + ], + "id": "0b1d561c-21ff-4092-beab-8154b17f82f2", + "active": true, + "name.formatted": "Test User", + "userName": "username", + "meta": { "resourceType":"User" }, + "emails": [ + { + "type": "work", + "value": "name@example.com", + "primary": true + } + ] +} +``` + +Returns a `201` status code if successful. + +### Update a single SCIM provisioned user + +Fields that can be updated are: + +| SCIM/IdP field | GitLab field | +|:---------------------------------|:-----------------------------------------------------------------------------| +| `id/externalId` | `extern_uid` | +| `name.formatted` | `name` ([Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/363058)) | +| `emails\[type eq "work"\].value` | `email` ([Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/363058)) | +| `active` | Identity removal if `active` = `false` | +| `userName` | `username` ([Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/363058)) | + +```plaintext +PATCH /api/scim/v2/groups/:group_path/Users/:id +``` + +Parameters: + +| Attribute | Type | Required | Description | +|:----------|:--------|:---------|:----------------------------------------------------------------------------------------------------------------------------------------| +| `id` | string | yes | External UID of the user. | +| `group_path` | string | yes | Full path to the group. | +| `Operations` | JSON string | yes | An [operations](#available-operations) expression. | + +Example request: + +```shell +curl --verbose --request PATCH "https://gitlab.example.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2" \ + --data '{ "Operations": [{"op":"Add","path":"name.formatted","value":"New Name"}] }' \ + --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +``` + +Returns an empty response with a `204` status code if successful. + +### Remove a single SCIM provisioned user + +Removes the user's SSO identity and group membership. + +```plaintext +DELETE /api/scim/v2/groups/:group_path/Users/:id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| ------------ | ------ | -------- | ------------------------- | +| `id` | string | yes | External UID of the user. | +| `group_path` | string | yes | Full path to the group. | + +Example request: + +```shell +curl --verbose --request DELETE "https://gitlab.example.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2" \ + --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +``` + +Returns an empty response with a `204` status code if successful. + +### Available filters + +They match an expression as specified in [the RFC7644 filtering section](https://www.rfc-editor.org/rfc/rfc7644#section-3.4.2.2). + +| Filter | Description | +| ----- | ----------- | +| `eq` | The attribute matches exactly the specified value. | + +Example: + +```plaintext +id eq a-b-c-d +``` + +### Available operations + +They perform an operation as specified in [the RFC7644 update section](https://www.rfc-editor.org/rfc/rfc7644#section-3.5.2). + +| Operator | Description | +| ----- | ----------- | +| `Replace` | The attribute's value is updated. | +| `Add` | The attribute has a new value. | + +Example: + +```json +{ "op": "Add", "path": "name.formatted", "value": "New Name" } +``` diff --git a/doc/development/internal_api/internal_api_allowed.md b/doc/development/internal_api/internal_api_allowed.md index 83dbb54babd..89c76c88df4 100644 --- a/doc/development/internal_api/internal_api_allowed.md +++ b/doc/development/internal_api/internal_api_allowed.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" type: reference, api --- diff --git a/doc/development/internal_users.md b/doc/development/internal_users.md index 24785c0cf48..67000d969af 100644 --- a/doc/development/internal_users.md +++ b/doc/development/internal_users.md @@ -3,7 +3,7 @@ description: "Internal users documentation." type: concepts, reference, dev stage: none group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" --- # Internal users diff --git a/doc/development/issuable-like-models.md b/doc/development/issuable-like-models.md index 42dfe2e0f2f..0b48883fe58 100644 --- a/doc/development/issuable-like-models.md +++ b/doc/development/issuable-like-models.md @@ -1,7 +1,7 @@ --- stage: Plan group: Project Management -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Issuable-like Rails models utilities diff --git a/doc/development/issue_types.md b/doc/development/issue_types.md index 820c37aeb14..465f539e026 100644 --- a/doc/development/issue_types.md +++ b/doc/development/issue_types.md @@ -1,7 +1,7 @@ --- stage: Plan group: Project Management -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Issue Types (DEPRECATED) @@ -42,7 +42,7 @@ issues or when linking objects of the type from Epics. It should use the same - You can't close it from a commit or a merge request. - You can't mark it as related to another issue. -If an Issue type can not be used you can still define a first-class type and +If an Issue type cannot be used you can still define a first-class type and then include concerns such as `Issuable` or `Noteable` to reuse functionality which is common for all our issue-related resources. But you still need to define the interface for working with the new resource and update some other diff --git a/doc/development/json.md b/doc/development/json.md new file mode 100644 index 00000000000..8a2575401fb --- /dev/null +++ b/doc/development/json.md @@ -0,0 +1,46 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# JSON Guidelines + +At GitLab we handle a lot of JSON data. To best ensure we remain performant +when handling large JSON encodes or decodes, we use our own JSON class +instead of the default methods. + +## `Gitlab::Json` + +This class should be used in place of any calls to the default `JSON` class, +`.to_json` calls, and the like. It implements the majority of the public +methods provided by `JSON`, such as `.parse`, `.generate`, `.dump`, etc, and +should be entirely identical in its response. + +The difference being that by sending all JSON handling through `Gitlab::Json` +we can change the gem being used in the background. We use `oj` +instead of the `json` gem, which uses C extensions and is therefore notably +faster. + +This class came into existence because, due to the age of the GitLab application, +it was proving impossible to just replace the `json` gem with `oj` by default because: + +- The number of tests with exact expectations of the responses. +- The subtle variances between different JSON processors, particularly + around formatting. + +The `Gitlab::Json` class takes this into account and can +vary the adapter based on the use case, and account for outdated formatting +expectations. + +## `Gitlab::Json::PrecompiledJson` + +This class is used by our hooks into the Grape framework to ensure that +already-generated JSON is not then run through JSON generation +a second time when returning the response. + +## `Gitlab::Json::LimitedEncoder` + +This class can be used to generate JSON but fail with an error if the +resulting JSON would be too large. The default limit for the `.encode` +method is 25 MB, but this can be customized when using the method. diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md index 1c83bef5620..e7764326db3 100644 --- a/doc/development/kubernetes.md +++ b/doc/development/kubernetes.md @@ -1,7 +1,7 @@ --- stage: Configure group: Configure -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Kubernetes integration - development guidelines **(FREE)** diff --git a/doc/development/lfs.md b/doc/development/lfs.md index 20157e9e805..4d3371af1d4 100644 --- a/doc/development/lfs.md +++ b/doc/development/lfs.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Git LFS **(FREE)** diff --git a/doc/development/licensing.md b/doc/development/licensing.md index a50c514733e..d7dfdb7357d 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab Licensing and Compatibility @@ -18,7 +18,7 @@ Some gems may not include their license information in their `gemspec` file, and ### License Finder commands -There are a few basic commands License Finder provides that you need in order to manage license detection. +There are a few basic commands License Finder provides that you need to manage license detection. To verify that the checks are passing, and/or to see what dependencies are causing the checks to fail: diff --git a/doc/development/logging.md b/doc/development/logging.md index 467fb68f3ae..44a47f96de1 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab Developers Guide to Logging **(FREE)** diff --git a/doc/development/maintenance_mode.md b/doc/development/maintenance_mode.md index f70cca1040e..b61b7472385 100644 --- a/doc/development/maintenance_mode.md +++ b/doc/development/maintenance_mode.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Internal workings of GitLab Maintenance Mode **(PREMIUM SELF)** diff --git a/doc/development/mass_insert.md b/doc/development/mass_insert.md index 4b1716ff00a..24f6127a3de 100644 --- a/doc/development/mass_insert.md +++ b/doc/development/mass_insert.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Mass inserting Rails models diff --git a/doc/development/merge_request_application_and_rate_limit_guidelines.md b/doc/development/merge_request_application_and_rate_limit_guidelines.md index 62bf62f6275..07a48ad7723 100644 --- a/doc/development/merge_request_application_and_rate_limit_guidelines.md +++ b/doc/development/merge_request_application_and_rate_limit_guidelines.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Application and rate limit guidelines diff --git a/doc/development/merge_request_concepts/index.md b/doc/development/merge_request_concepts/index.md index d463f6ba290..14d9582ad84 100644 --- a/doc/development/merge_request_concepts/index.md +++ b/doc/development/merge_request_concepts/index.md @@ -2,7 +2,7 @@ type: reference, dev stage: Create group: Code Review -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" --- # Merge request concepts diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index 895fc6f92a4..9d1489836fb 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Merge Request Performance Guidelines @@ -44,7 +44,7 @@ and running. Can the queries used potentially take down any critical services and result in engineers being woken up in the night? Can a malicious user abuse the code to -take down a GitLab instance? Do my changes simply make loading a certain page +take down a GitLab instance? Do my changes make loading a certain page slower? Does execution time grow exponentially given enough load or data in the database? @@ -281,7 +281,7 @@ be clearly mentioned in the merge request description. ## Batch process **Summary:** Iterating a single process to external services (for example, PostgreSQL, Redis, Object Storage) -should be executed in a **batch-style** in order to reduce connection overheads. +should be executed in a **batch-style** to reduce connection overheads. For fetching rows from various tables in a batch-style, please see [Eager Loading](#eager-loading) section. @@ -488,7 +488,7 @@ We can consider the following types of storages: - **Object-based persistent storage** (long term storage) this type of storage uses external services like [AWS S3](https://en.wikipedia.org/wiki/Amazon_S3). The Object Storage can be treated as infinitely scalable and redundant. Accessing this storage usually requires - downloading the file in order to manipulate it. The Object Storage can be considered as an ultimate + downloading the file to manipulate it. The Object Storage can be considered as an ultimate solution, as by definition it can be assumed that it can handle unlimited concurrent uploads and downloads of files. This is also ultimate solution required to ensure that application can run in containerized deployments (Kubernetes) at ease. diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 4e569579f37..f59c1fd8368 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Migration Style Guide @@ -175,6 +175,14 @@ git checkout origin/master db/structure.sql VERSION=<migration ID> bundle exec rails db:migrate:main ``` +### Adding new tables to GitLab Schema + +GitLab connects to two different Postgres databases: `main` and `ci`. New tables should be defined in [`lib/gitlab/database/gitlab_schemas.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/gitlab_schemas.yml) with the databases they need to be added to. + + ```yaml + <TABLE_NAME>: :gitlab_main + ``` + ## Avoiding downtime The document ["Avoiding downtime in migrations"](database/avoiding_downtime_in_migrations.md) specifies @@ -267,7 +275,7 @@ in that limit. Singular query timings should fit within the [standard limit](dat In case you need to insert, update, or delete a significant amount of data, you: - Must disable the single transaction with `disable_ddl_transaction!`. -- Should consider doing it in a [Background Migration](database/background_migrations.md). +- Should consider doing it in a [batched background migration](database/batched_background_migrations.md). ## Migration helpers and versioning @@ -604,7 +612,7 @@ Note that it is not necessary to check if the index exists prior to removing it, however it is required to specify the name of the index that is being removed. This can be done either by passing the name as an option to the appropriate form of `remove_index` or `remove_concurrent_index`, -or more simply by using the `remove_concurrent_index_by_name` method. Explicitly +or by using the `remove_concurrent_index_by_name` method. Explicitly specifying the name is important to ensure the correct index is removed. For a small table (such as an empty one or one with less than `1,000` records), @@ -982,6 +990,43 @@ NOTE: `add_sequence` should be avoided for columns with foreign keys. Adding sequence to these columns is **only allowed** in the down method (restore previous schema state). +## Swapping primary key + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98645) in GitLab 15.5. + +Swapping the primary key is required to partition a table as the **partition key must be included in the primary key**. + +You can use the `swap_primary_key` method provided by the database team. + +Under the hood, it works like this: + +- Drop the primary key constraint. +- Add the primary key using the index defined beforehand. + +```ruby +class SwapPrimaryKey < Gitlab::Database::Migration[2.0] + TABLE_NAME = :table_name + PRIMARY_KEY = :table_name_pkey + OLD_INDEX_NAME = :old_index_name + NEW_INDEX_NAME = :new_index_name + + def up + swap_primary_key(TABLE_NAME, PRIMARY_KEY, NEW_INDEX_NAME) + end + + def down + add_concurrent_index(TABLE_NAME, :id, unique: true, name: OLD_INDEX_NAME) + add_concurrent_index(TABLE_NAME, [:id, :partition_id], unique: true, name: NEW_INDEX_NAME) + + unswap_primary_key(TABLE_NAME, PRIMARY_KEY, OLD_INDEX_NAME) + end +end +``` + +NOTE: +Make sure to introduce the new index beforehand in a separate migration in order +to swap the primary key. + ## Integer column type By default, an integer column can hold up to a 4-byte (32-bit) number. That is @@ -1221,7 +1266,7 @@ by an integer. For example: `users` would turn into `users0` ## Using models in migrations (discouraged) The use of models in migrations is generally discouraged. As such models are -[contraindicated for background migrations](database/background_migrations.md#isolation), +[contraindicated for batched background migrations](database/batched_background_migrations.md#isolation), the model needs to be declared in the migration. If using a model in the migrations, you should first @@ -1236,7 +1281,7 @@ in a previous migration. ### Example: Add a column `my_column` to the users table -It is important not to leave out the `User.reset_column_information` command, in order to ensure that the old schema is dropped from the cache and ActiveRecord loads the updated schema information. +It is important not to leave out the `User.reset_column_information` command, to ensure that the old schema is dropped from the cache and ActiveRecord loads the updated schema information. ```ruby class AddAndSeedMyColumn < Gitlab::Database::Migration[2.0] diff --git a/doc/development/module_with_instance_variables.md b/doc/development/module_with_instance_variables.md index 8e39186d396..733823d7f6e 100644 --- a/doc/development/module_with_instance_variables.md +++ b/doc/development/module_with_instance_variables.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Modules with instance variables could be considered harmful diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index 0f3531cf5dd..808554d6027 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Backwards compatibility across updates diff --git a/doc/development/omnibus.md b/doc/development/omnibus.md index 4e2f2b0c763..a0ba2751a06 100644 --- a/doc/development/omnibus.md +++ b/doc/development/omnibus.md @@ -1,7 +1,7 @@ --- stage: Systems group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # What you should know about Omnibus packages diff --git a/doc/development/packages.md b/doc/development/packages.md deleted file mode 100644 index a79f5f09677..00000000000 --- a/doc/development/packages.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'packages/index.md' -remove_date: '2022-08-19' ---- - -This document was moved to [another location](packages/index.md). - -<!-- This redirect file can be deleted after <2022-08-19>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/packages/debian_repository.md b/doc/development/packages/debian_repository.md index a417ced2e65..71f6c916652 100644 --- a/doc/development/packages/debian_repository.md +++ b/doc/development/packages/debian_repository.md @@ -1,7 +1,7 @@ --- stage: Package group: Package -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Debian Repository diff --git a/doc/development/packages/dependency_proxy.md b/doc/development/packages/dependency_proxy.md new file mode 100644 index 00000000000..f3e323fb9fa --- /dev/null +++ b/doc/development/packages/dependency_proxy.md @@ -0,0 +1,197 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Dependency Proxy + +The Dependency Proxy is a pull-through-cache for registry images from DockerHub. This document describes how this +feature is constructed in GitLab. + +## Container registry + +The Dependency Proxy for the container registry acts a stand-in for a remote container registry. In our case, +the remote registry is the public DockerHub registry. + +```mermaid +flowchart TD + id1([$ docker]) --> id2([GitLab Dependency Proxy]) + id2 --> id3([DockerHub]) +``` + +From the user's perspective, the GitLab instance is just a container registry that they are interacting with to +pull images by using `docker login gitlab.com` + +When you use `docker login gitlab.com`, the Docker client uses the [v2 API](https://docs.docker.com/registry/spec/api/) +to make requests. + +To support authentication, we must include one route: + +- [API Version Check](https://docs.docker.com/registry/spec/api/#api-version-check) + +To support `docker pull` requests, we must include two additional routes: + +- [Pulling an image manifest](https://docs.docker.com/registry/spec/api/#pulling-an-image-manifest) +- [Pulling an image layer (blob)](https://docs.docker.com/registry/spec/api/#pulling-a-layer) + +These routes are defined in [`gitlab-org/gitlab/config/routes/group.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/3f76455ac9cf90a927767e55c837d6b07af818df/config/routes/group.rb#L164-175). + +In its simplest form, the Dependency Proxy manages three requests: + +- Logging in / returning a JWT +- Fetching a manifest +- Fetching a blob + +Here is what the general request sequence looks like for the Dependency Proxy: + +```mermaid +sequenceDiagram + Client->>+GitLab: Login? / request token + GitLab->>+Client: JWT + Client->>+GitLab: request a manifest for an image + GitLab->>+ExternalRegistry: request JWT + ExternalRegistry->>+GitLab : JWT + GitLab->>+ExternalRegistry : request manifest + ExternalRegistry->>+GitLab : return manifest + GitLab->>+GitLab : store manifest + GitLab->>+Client : return manifest + loop request image layers + Client->>+GitLab: request a blob from the manifest + GitLab->>+ExternalRegistry: request JWT + ExternalRegistry->>+GitLab : JWT + GitLab->>+ExternalRegistry : request blob + ExternalRegistry->>+GitLab : return blob + GitLab->>+GitLab : store blob + GitLab->>+Client : return blob + end +``` + +### Authentication and authorization + +When a Docker client authenticates with a registry, the registry tells the client where to get a JSON Web Token +(JWT) and to use it for all subsequent requests. This allows the authentication service to live in a separate +application from the registry. For example, the GitLab container registry directs Docker clients to get a token +from `https://gitlab.com/jwt/auth`. This endpoint is part of the `gitlab-org/gitlab` project, also known as the +rails project or web service. + +When a user tries to sign in to the dependency proxy with a Docker client, we must tell it where to get a JWT. We +can use the same endpoint we use with the container registry: `https://gitlab.com/jwt/auth`. But in our case, +we tell the Docker client to specify `service=dependency_proxy` in the parameters so can use a separate underlying +service to generate the token. + +This sequence diagram shows the request flow for logging into the Dependency Proxy. + +```mermaid +sequenceDiagram + autonumber + participant C as Docker CLI + participant R as GitLab (Dependency Proxy) + + Note right of C: User tries `docker login gitlab.com` and enters username/password + C->>R: GET /v2/ + Note left of R: Check for Authorization header, return 401 if none, return 200 if token exists and is valid + R->>C: 401 Unauthorized with header "WWW-Authenticate": "Bearer realm=\"http://gitlab.com/jwt/auth\",service=\"registry.docker.io\"" + Note right of C: Request Oauth token using HTTP Basic Auth + C->>R: GET /jwt/auth + Note left of R: Token is returned + R->>C: 200 OK (with Bearer token included) + Note right of C: original request is tested again + C->>R: GET /v2/ (this time with `Authorization: Bearer [token]` header) + Note right of C: Login Succeeded + R->>C: 200 OK +``` + +The dependency proxy uses its own authentication service, separate from the authentication managed by the UI +(`ApplicationController`) and API (`ApiGuard`). Once the service has created a JWT, the `DependencyProxy::ApplicationController` +manages authentication and authorization for the rest of the requests. It manages the user by using `GitLab::Auth::Result` and +is similar to the authentication implemented by the Git client requests in `GitHttpClientController`. + +### Caching + +Blobs are cached artifacts with no logic around them. We cache them by digest. When we receive a request for a new blob, +we check to see if we have a blob with the requested digest, and return it. Otherwise we fetch it from the external +registry and cache it. + +Manifests are more complicated, partially due to [rate limiting on DockerHub](https://www.docker.com/increase-rate-limits/). +A manifest is essentially a recipe for creating an image. It has a list of blobs to create a certain image. So +`alpine:latest` has a manifest associated with it that specifies the blobs needed to create the `alpine:latest` +image. The interesting part is that `alpine:latest` can change over time, so we can't just cache the manifest and +assume it is OK to use forever. Instead, we must check the digest of the manifest, which is an Etag. This gets +interesting because the requests for manifests often don't include the digest. So how do we know if the manifest +we have cached is still the most up-to-date `alpine:latest`? DockerHub allows free HEAD requests that don't count +toward their rate limit. The HEAD request returns the manifest digest so we can tell whether or not the one we +have is stale. + +With this knowledge, we have built the following logic to manage manifest requests: + +```mermaid +graph TD + A[Receive manifest request] --> | We have the manifest cached.| B{Docker manifest HEAD request} + A --> | We do not have manifest cached.| C{Docker manifest GET request} + B --> | Digest matches the one in the DB | D[Fetch manifest from cache] + B --> | Network failure, cannot reach DockerHub | D[Fetch manifest from cache] + B --> | Digest does not match the one in DB | C + C --> E[Save manifest to cache, save digest to database] + D --> F + E --> F[Return manifest] +``` + +### Workhorse for file handling + +Management of file uploads and caching happens in [Workhorse](../workhorse/index.md). This explains the additional +[`POST` routes](https://gitlab.com/gitlab-org/gitlab/-/blob/3f76455ac9cf90a927767e55c837d6b07af818df/config/routes/group.rb#L170-173) +that we have for the Dependency Proxy. + +The [send_dependency](https://gitlab.com/gitlab-org/gitlab/-/blob/7359d23f4e078479969c872924150219c6f1665f/app/helpers/workhorse_helper.rb#L46-53) +method makes a request to Workhorse including the previously fetched JWT from the external registry. Workhorse then +can use that token to request the manifest or blob the user originally requested. The Workhorse code lives in +[`workhorse/internal/dependencyproxy/dependencyproxy.go`](https://gitlab.com/gitlab-org/gitlab/-/blob/b8f44a8f3c26efe9932c2ada2df75ef7acb8417b/workhorse/internal/dependencyproxy/dependencyproxy.go#L4). + +Once we put it all together, the sequence for requesting an image file looks like this: + +```mermaid +sequenceDiagram + Client->>Workhorse: GET /v2/*group_id/dependency_proxy/containers/*image/manifests/*tag + Workhorse->>Rails: GET /v2/*group_id/dependency_proxy/containers/*image/manifests/*tag + Rails->>Rails: Check DB. Is manifest persisted in cache? + + alt In Cache + Rails->>Workhorse: Respond with send-url injector + Workhorse->>Client: Send the file to the client + else Not In Cache + Rails->>Rails: Generate auth token and download URL for the manifest in upstream registry + Rails->>Workhorse: Respond with send-dependency injector + Workhorse->>External Registry: Request the manifest + External Registry->>Workhorse: Download the manifest + Workhorse->>Rails: GET /v2/*group_id/dependency_proxy/containers/*image/manifest/*tag/authorize + Rails->>Workhorse: Respond with upload instructions + Workhorse->>Client: Send the manifest file to the client with original headers + Workhorse->>Object Storage: Save the manifest file with some of it's header values + Workhorse->>Rails: Finalize the upload + end +``` + +### Cleanup policies + +The cleanup policies for the Dependency Proxy work as time-to-live policies. They allow users to set the number +of days a file is allowed to remain cached if it has been unread. Since there is no way to associate the blobs +with the images they belong to (to do this, we would need to build the metadata database that the container registry +folks built), we can set up rules like "if this blob has not been pulled in 90 days, delete it". This means that +any files that are continuously getting pulled will not be removed from the cache, but if, for example, +`alpine:latest` changes and one of the underlying blobs is no longer used, it will eventually get cleaned up +because it has stopped getting pulled. We use the `read_at` attribute to track the last time a given +`dependency_proxy_blob` or `dependency_proxy_manifest` was pulled. + +These work using a cron worker, [DependencyProxy::CleanupDependencyProxyWorker](https://gitlab.com/gitlab-org/gitlab/-/blob/7359d23f4e078479969c872924150219c6f1665f/app/workers/dependency_proxy/cleanup_dependency_proxy_worker.rb#L4), +that will kick off two [limited capacity](../sidekiq/limited_capacity_worker.md) workers: one to delete blobs, +and one to delete manifests. The capacity is set in an [application setting](settings.md#container-registry). + +### Historic reference links + +- [Dependency proxy for private groups](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46042) - initial authentication implementation +- [Manifest caching](https://gitlab.com/gitlab-org/gitlab/-/issues/241639) - initial manifest caching implementation +- [Workhorse for blobs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71890) - initial workhorse implementation +- [Workhorse for manifest](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73033) - moving manifest cache logic to Workhorse +- [Deploy token support](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64363) - authorization largely updated +- [SSO support](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67373) - changes how policies are checked diff --git a/doc/development/packages/index.md b/doc/development/packages/index.md index 55deaa229ba..41d6e1b84fd 100644 --- a/doc/development/packages/index.md +++ b/doc/development/packages/index.md @@ -1,13 +1,15 @@ --- stage: Package group: Package -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Package Registry Development Development and Architectural documentation for the package registry +- [Debian repository structure](debian_repository.md) +- [Dependency proxy structure](dependency_proxy.md) - [Developing a new format](new_format_development.md) - [Settings](settings.md) - [Structure / Schema](structure.md) diff --git a/doc/development/packages/new_format_development.md b/doc/development/packages/new_format_development.md index 73a2b2f1f81..e9decea3094 100644 --- a/doc/development/packages/new_format_development.md +++ b/doc/development/packages/new_format_development.md @@ -1,7 +1,7 @@ --- stage: Package group: Package -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Developing support for a new package format diff --git a/doc/development/packages/settings.md b/doc/development/packages/settings.md index 37961c0504c..d591f708954 100644 --- a/doc/development/packages/settings.md +++ b/doc/development/packages/settings.md @@ -1,7 +1,7 @@ --- stage: Package group: Package -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Package Settings diff --git a/doc/development/packages/structure.md b/doc/development/packages/structure.md index f8d9da2cc73..e5426f8e2b8 100644 --- a/doc/development/packages/structure.md +++ b/doc/development/packages/structure.md @@ -1,7 +1,7 @@ --- stage: Package group: Package -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Package Structure diff --git a/doc/development/pages/index.md b/doc/development/pages/index.md index af9d4d33683..6ee8a9ac433 100644 --- a/doc/development/pages/index.md +++ b/doc/development/pages/index.md @@ -2,7 +2,7 @@ type: reference, dev stage: Create group: Editor -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: "GitLab's development guidelines for GitLab Pages" --- diff --git a/doc/development/performance.md b/doc/development/performance.md index 479782e0ccf..4f22d9ceb03 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Performance Guidelines @@ -79,7 +79,7 @@ GitLab provides built-in tools to help improve performance and availability: - [Service measurement](service_measurement.md) for measuring and logging service execution. GitLab team members can use [GitLab.com's performance monitoring systems](https://about.gitlab.com/handbook/engineering/monitoring/) located at -[`dashboards.gitlab.net`](https://dashboards.gitlab.net), this requires you to log in using your +[`dashboards.gitlab.net`](https://dashboards.gitlab.net), this requires you to sign in using your `@gitlab.com` email address. Non-GitLab team-members are advised to set up their own Prometheus and Grafana stack. @@ -391,7 +391,7 @@ We store these results also when running nightly scheduled CI jobs on the default branch on `gitlab.com`. Statistics of these profiling data are [available online](https://gitlab-org.gitlab.io/rspec_profiling_stats/). For example, you can find which tests take longest to run or which execute the most -queries. This can be handy for optimizing our tests or identifying performance +queries. Use this to optimize our tests or identify performance issues in our code. ## Memory optimization @@ -925,7 +925,7 @@ SOME_CONSTANT = 'bar' ## How to seed a database with millions of rows You might want millions of project rows in your local database, for example, -in order to compare relative query performance, or to reproduce a bug. You could +to compare relative query performance, or to reproduce a bug. You could do this by hand with SQL commands or using [Mass Inserting Rails Models](mass_insert.md) functionality. Assuming you are working with ActiveRecord models, you might also find these links helpful: diff --git a/doc/development/permissions.md b/doc/development/permissions.md index d348d659cad..cc7c1229e50 100644 --- a/doc/development/permissions.md +++ b/doc/development/permissions.md @@ -1,7 +1,7 @@ --- stage: Manage group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Implementing permissions @@ -37,7 +37,7 @@ Groups and projects can have the following visibility levels: - private (`0`) - an entity is visible only to the approved members of the entity By default, subgroups can **not** have higher visibility levels. -For example, if you create a new private group, it can not include a public subgroup. +For example, if you create a new private group, it cannot include a public subgroup. The visibility level of a group can be changed only if all subgroups and sub-projects have the same or lower visibility level. For example, a group can be set diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 648a2aac339..01e42fda2c9 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -1,7 +1,7 @@ --- stage: none group: Engineering Productivity -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Pipelines for the GitLab project @@ -62,11 +62,14 @@ The test mappings contain a map of each source files to a list of test files whi In the `detect-tests` job, we use this mapping to identify the minimal tests needed for the current merge request. +Later on in [the `rspec fail-fast` job](#fail-fast-job-in-merge-request-pipelines), we run the minimal tests needed for the current merge request. + #### Exceptional cases In addition, there are a few circumstances where we would always run the full RSpec tests: - when the `pipeline:run-all-rspec` label is set on the merge request +- when the `pipeline:run-full-rspec` label is set on the merge request, this label is assigned by triage automation when the merge request is approved by any reviewer - when the merge request is created by an automation (for example, Gitaly update or MR targeting a stable branch) - when the merge request is created in a security mirror - when any CI configuration file is changed (for example, `.gitlab-ci.yml` or `.gitlab/ci/**/*`) @@ -92,23 +95,11 @@ In addition, there are a few circumstances where we would always run the full Je ### Fork pipelines -We only run the minimal RSpec & Jest jobs for fork pipelines unless the `pipeline:run-all-rspec` +We run only the minimal RSpec & Jest jobs for fork pipelines, unless the `pipeline:run-all-rspec` label is set on the MR. The goal is to reduce the CI/CD minutes consumed by fork pipelines. See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1170). -## Faster feedback when reverting merge requests - -When you need to revert a merge request, to get accelerated feedback, you can add the `~pipeline:revert` label to your merge request. - -When this label is assigned, the following steps of the CI/CD pipeline are skipped: - -- The `e2e:package-and-test` job. -- The `rspec:undercoverage` job. -- The entire [Review Apps process](testing_guide/review_apps.md). - -Apply the label to the merge request, and run a new pipeline for the MR. - ## Fail-fast job in merge request pipelines To provide faster feedback when a merge request breaks existing tests, we are experimenting with a @@ -155,6 +146,18 @@ merge request. This prevents `rspec fail-fast` duration from exceeding the avera This number can be overridden by setting a CI/CD variable named `RSPEC_FAIL_FAST_TEST_FILE_COUNT_THRESHOLD`. +## Faster feedback when reverting merge requests + +When you need to revert a merge request, to get accelerated feedback, you can add the `~pipeline:revert` label to your merge request. + +When this label is assigned, the following steps of the CI/CD pipeline are skipped: + +- The `e2e:package-and-test` job. +- The `rspec:undercoverage` job. +- The entire [Review Apps process](testing_guide/review_apps.md). + +Apply the label to the merge request, and run a new pipeline for the MR. + ## Test jobs We have dedicated jobs for each [testing level](testing_guide/testing_levels.md) and each job runs depending on the @@ -356,6 +359,11 @@ with latest `master`, and then it triggers a regular branch pipeline for never be merged back to `master`. Any other Ruby 3 changes should go into `master` directly, which should be compatible with Ruby 2.7. +Previously, `ruby3-sync` was using a project token stored in `RUBY3_SYNC_TOKEN` +(now backed up in `RUBY3_SYNC_TOKEN_NOT_USED`), however due to various +permissions issues, we ended up using an access token from `gitlab-bot` so now +`RUBY3_SYNC_TOKEN` is actually an access token from `gitlab-bot`. + ### Long-term plan We follow the [PostgreSQL versions shipped with Omnibus GitLab](../administration/package_information/postgresql_versions.md): @@ -392,8 +400,7 @@ In general, pipelines for an MR fall into one of the following types (from short A "pipeline type" is an abstract term that mostly describes the "critical path" (for example, the chain of jobs for which the sum of individual duration equals the pipeline's duration). -We use these "pipeline types" in [metrics dashboards](https://app.periscopedata.com/app/gitlab/858266/GitLab-Pipeline-Durations) -in order to detect what types and jobs need to be optimized first. +We use these "pipeline types" in [metrics dashboards](https://app.periscopedata.com/app/gitlab/858266/GitLab-Pipeline-Durations) to detect what types and jobs need to be optimized first. An MR that touches multiple areas would be associated with the longest type applicable. For instance, an MR that touches backend and frontend would fall into the "Frontend" pipeline type since this type takes longer to finish than the "Backend" pipeline type. @@ -581,8 +588,9 @@ The current stages are: ### Dependency Proxy Some of the jobs are using images from Docker Hub, where we also use -`${GITLAB_DEPENDENCY_PROXY}` as a prefix to the image path, so that we pull +`${GITLAB_DEPENDENCY_PROXY_ADDRESS}` as a prefix to the image path, so that we pull images from our [Dependency Proxy](../user/packages/dependency_proxy/index.md). +By default, this variable is set from the value of `${GITLAB_DEPENDENCY_PROXY}`. `${GITLAB_DEPENDENCY_PROXY}` is a group CI/CD variable defined in [`gitlab-org`](https://gitlab.com/gitlab-org) as @@ -590,13 +598,32 @@ images from our [Dependency Proxy](../user/packages/dependency_proxy/index.md). defined as: ```yaml -image: ${GITLAB_DEPENDENCY_PROXY}alpine:edge +image: ${GITLAB_DEPENDENCY_PROXY_ADDRESS}alpine:edge ``` Projects in the `gitlab-org` group pull from the Dependency Proxy, while forks that reside on any other personal namespaces or groups fall back to Docker Hub unless `${GITLAB_DEPENDENCY_PROXY}` is also defined there. +#### Work around for when a pipeline is started by a Project access token user + +When a pipeline is started by a Project access token user (e.g. the `release-tools approver bot` user which +automatically updates the Gitaly version used in the main project), +[the Dependency proxy isn't accessible](https://gitlab.com/gitlab-org/gitlab/-/issues/332411#note_1130388163) +and the job fails at the `Preparing the "docker+machine" executor` step. +To work around that, we have a special workflow rule, that overrides the +`${GITLAB_DEPENDENCY_PROXY_ADDRESS}` variable so that Depdendency proxy isn't used in that case: + +```yaml +- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $GITLAB_USER_LOGIN =~ /project_\d+_bot\d*/' + variables: + GITLAB_DEPENDENCY_PROXY_ADDRESS: "" +``` + +NOTE: +We don't directly override the `${GITLAB_DEPENDENCY_PROXY}` variable because group-level +variables have higher precedence over `.gitlab-ci.yml` variables. + ### Common job definitions Most of the jobs [extend from a few CI definitions](../ci/yaml/index.md#extends) @@ -731,7 +758,7 @@ This works well for the following reasons: - `.static-analysis-cache` - `.rubocop-cache` - `.coverage-cache` - - `.danger-review-cache` + - `.ruby-node-cache` - `.qa-cache` - `.yarn-cache` - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). @@ -749,34 +776,87 @@ This works well for the following reasons: ### Artifacts strategy -We limit the artifacts that are saved and retrieved by jobs to the minimum in order to reduce the upload/download time and costs, as well as the artifacts storage. +We limit the artifacts that are saved and retrieved by jobs to the minimum to reduce the upload/download time and costs, as well as the artifacts storage. ### Components caching -Some external components (currently only GitLab Workhorse) of GitLab need to be built from source as a preliminary step for running tests. +Some external components (GitLab Workhorse and frontend assets) of GitLab need to be built from source as a preliminary step for running tests. + +#### `cache-workhorse` -In [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79766), we introduced a new `build-components` job that: +In [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79766), and then +[this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96297), +we introduced a new `cache-workhorse` job that: - runs automatically for all GitLab.com `gitlab-org/gitlab` scheduled pipelines - runs automatically for any `master` commit that touches the `workhorse/` folder -- is manual for GitLab.com's `gitlab-org`'s MRs +- is manual for GitLab.com's `gitlab-org`'s MRs that touches caching-related files This job tries to download a generic package that contains GitLab Workhorse binaries needed in the GitLab test suite (under `tmp/tests/gitlab-workhorse`). - If the package URL returns a 404: 1. It runs `scripts/setup-test-env`, so that the GitLab Workhorse binaries are built. 1. It then creates an archive which contains the binaries and upload it [as a generic package](https://gitlab.com/gitlab-org/gitlab/-/packages/). -- Otherwise, if the package already exists, it exit the job successfully. +- Otherwise, if the package already exists, it exits the job successfully. We also changed the `setup-test-env` job to: -1. First download the GitLab Workhorse generic package build and uploaded by `build-components`. +1. First download the GitLab Workhorse generic package build and uploaded by `cache-workhorse`. 1. If the package is retrieved successfully, its content is placed in the right folder (for example, `tmp/tests/gitlab-workhorse`), preventing the building of the binaries when `scripts/setup-test-env` is run later on. 1. If the package URL returns a 404, the behavior doesn't change compared to the current one: the GitLab Workhorse binaries are built as part of `scripts/setup-test-env`. NOTE: The version of the package is the workhorse tree SHA (for example, `git rev-parse HEAD:workhorse`). +#### `cache-assets` + +In [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96297), +we introduced three new `cache-assets:test`, `cache-assets:test as-if-foss`, +and `cache-assets:production` jobs that: + +- never run unless `$CACHE_ASSETS_AS_PACKAGE == "true"` +- runs automatically for all GitLab.com `gitlab-org/gitlab` scheduled pipelines +- runs automatically for any `master` commit that touches the assets-related folders +- is manual for GitLab.com's `gitlab-org`'s MRs that touches caching-related files + +This job tries to download a generic package that contains GitLab compiled assets +needed in the GitLab test suite (under `app/assets/javascripts/locale/**/app.js`, +and `public/assets`). + +- If the package URL returns a 404: + 1. It runs `bin/rake gitlab:assets:compile`, so that the GitLab assets are compiled. + 1. It then creates an archive which contains the assets and uploads it [as a generic package](https://gitlab.com/gitlab-org/gitlab/-/packages/). + The package version is set to the assets folders' hash sum. +- Otherwise, if the package already exists, it exits the job successfully. + +#### `compile-*-assets` + +We also changed the `compile-test-assets`, `compile-test-assets as-if-foss`, +and `compile-production-assets` jobs to: + +1. First download the "native" cache assets, which contain: + - The [compiled assets](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/global.gitlab-ci.yml#L86-87). + - A [`cached-assets-hash.txt` file](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/global.gitlab-ci.yml#L85) + containing the `SHA256` hexdigest of all the source files on which the assets depend on. + This list of files is a pessimistic list and the assets might not depend on + some of these files. At worst we compile the assets more often, which is better than + using outdated assets. + + The file is [created after assets are compiled](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/frontend.gitlab-ci.yml#L83). +1. We then we compute the `SHA256` hexdigest of all the source files the assets depend on, **for the current checked out branch**. We [store the hexdigest in the `GITLAB_ASSETS_HASH` variable](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/frontend.gitlab-ci.yml#L27). +1. If `$CACHE_ASSETS_AS_PACKAGE == "true"`, we download the generic package built and uploaded by [`cache-assets:*`](#cache-assets). + - If the cache is up-to-date for the checked out branch, we download the native cache + **and** the cache package. We could optimize that by not downloading + the genetic package but the native cache is actually very often outdated because it's + rebuilt only every 2 hours. +1. We [run the `assets_compile_script` function](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/frontend.gitlab-ci.yml#L35), + which [itself runs](https://gitlab.com/gitlab-org/gitlab/-/blob/c023191ef412e868ae957f3341208a41ca678403/scripts/utils.sh#L76) + the [`assets:compile` Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/c023191ef412e868ae957f3341208a41ca678403/lib/tasks/gitlab/assets.rake#L80-109). + + This task is responsible for deciding if assets need to be compiled or not. + It [compares the `HEAD` `SHA256` hexdigest from `$GITLAB_ASSETS_HASH` with the `master` hexdigest from `cached-assets-hash.txt`](https://gitlab.com/gitlab-org/gitlab/-/blob/c023191ef412e868ae957f3341208a41ca678403/lib/tasks/gitlab/assets.rake#L86). +1. If the hashes are the same, we don't compile anything. If they're different, we compile the assets. + ### Pre-clone step NOTE: diff --git a/doc/development/policies.md b/doc/development/policies.md index ddd6b8e9bce..b50d654ed28 100644 --- a/doc/development/policies.md +++ b/doc/development/policies.md @@ -1,7 +1,7 @@ --- stage: Manage group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # `DeclarativePolicy` framework diff --git a/doc/development/polling.md b/doc/development/polling.md index f854891a528..ff239effb80 100644 --- a/doc/development/polling.md +++ b/doc/development/polling.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Polling with ETag caching @@ -61,5 +61,5 @@ route matching easier. For more information see: - [`Poll-Interval` header](fe_guide/performance.md#real-time-components) -- [RFC 7232](https://tools.ietf.org/html/rfc7232) +- [RFC 7232](https://www.rfc-editor.org/rfc/rfc7232) - [ETag proposal](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/26926) diff --git a/doc/development/post_deployment_migrations.md b/doc/development/post_deployment_migrations.md deleted file mode 100644 index c3922718e77..00000000000 --- a/doc/development/post_deployment_migrations.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/post_deployment_migrations.md' -remove_date: '2022-07-08' ---- - -This document was moved to [another location](database/post_deployment_migrations.md). - -<!-- This redirect file can be deleted after <2022-07-08>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/product_qualified_lead_guide/index.md b/doc/development/product_qualified_lead_guide/index.md index 90b8d905264..89a05d59fc9 100644 --- a/doc/development/product_qualified_lead_guide/index.md +++ b/doc/development/product_qualified_lead_guide/index.md @@ -1,7 +1,7 @@ --- stage: Growth group: Acquisition -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Product Qualified Lead (PQL) development guide diff --git a/doc/development/profiling.md b/doc/development/profiling.md index a3142b06e12..3eb2c7c9144 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Profiling @@ -20,6 +20,9 @@ The first argument to the profiler is either a full URL (including the instance hostname) or an absolute path, including the leading slash. +By default the report dump will be stored in a temporary file, which can be +interacted with using the [stackprof API](#reading-a-gitlabprofiler-report). + When using the script, command-line documentation is available by passing no arguments. @@ -31,10 +34,11 @@ For example: ```ruby Gitlab::Profiler.profile('/my-user') -# Returns a RubyProf::Profile for the regular operation of this request +# Returns the location of the temp file where the report dump is stored class UsersController; def show; sleep 100; end; end Gitlab::Profiler.profile('/my-user') -# Returns a RubyProf::Profile where 100 seconds is spent in UsersController#show +# Returns the location of the temp file where the report dump is stored +# where 100 seconds is spent in UsersController#show ``` For routes that require authorization you must provide a user to @@ -52,57 +56,14 @@ documented with the method source. Gitlab::Profiler.profile('/gitlab-org/gitlab-test', user: User.first, logger: Logger.new($stdout)) ``` -There is also a RubyProf printer available: -`Gitlab::Profiler::TotalTimeFlatPrinter`. This acts like -`RubyProf::FlatPrinter`, but its `min_percent` option works on the method's -total time, not its self time. (This is because we often spend most of our time -in library code, but this comes from calls in our application.) It also offers a -`max_percent` option to help filter out outer calls that aren't useful (like -`ActionDispatch::Integration::Session#process`). - -There is a convenience method for using this, -`Gitlab::Profiler.print_by_total_time`: - -```ruby -result = Gitlab::Profiler.profile('/my-user') -Gitlab::Profiler.print_by_total_time(result, max_percent: 60, min_percent: 2) -# Measure Mode: wall_time -# Thread ID: 70005223698240 -# Fiber ID: 70004894952580 -# Total: 1.768912 -# Sort by: total_time -# -# %self total self wait child calls name -# 0.00 1.017 0.000 0.000 1.017 14 *ActionView::Helpers::RenderingHelper#render -# 0.00 1.017 0.000 0.000 1.017 14 *ActionView::Renderer#render_partial -# 0.00 1.017 0.000 0.000 1.017 14 *ActionView::PartialRenderer#render -# 0.00 1.007 0.000 0.000 1.007 14 *ActionView::PartialRenderer#render_partial -# 0.00 0.930 0.000 0.000 0.930 14 Hamlit::TemplateHandler#call -# 0.00 0.928 0.000 0.000 0.928 14 Temple::Engine#call -# 0.02 0.865 0.000 0.000 0.864 638 *Enumerable#inject -``` - -To print the profile in HTML format, use the following example: - -```ruby -result = Gitlab::Profiler.profile('/my-user') - -printer = RubyProf::CallStackPrinter.new(result) -printer.print(File.open('/tmp/profile.html', 'w')) -``` - -### Stackprof support - -By default, `Gitlab::Profiler.profile` uses a tracing profiler called [`ruby-prof`](https://ruby-prof.github.io/). However, sampling profilers -[run faster and use less memory](https://jvns.ca/blog/2017/12/17/how-do-ruby---python-profilers-work-/), so they might be preferred. - -You can switch to [Stackprof](https://github.com/tmm1/stackprof) (a sampling profiler) to generate a profile by passing `sampling_mode: true`. Pass in a `profiler_options` hash to configure the output file (`out`) of the sampling data. For example: ```ruby -Gitlab::Profiler.profile('/gitlab-org/gitlab-test', user: User.first, sampling_mode: true, profiler_options: { out: 'tmp/profile.dump' }) +Gitlab::Profiler.profile('/gitlab-org/gitlab-test', user: User.first, profiler_options: { out: 'tmp/profile.dump' }) ``` +## Reading a GitLab::Profiler report + You can get a summary of where time was spent by running Stackprof against the sampling data. For example: ```shell @@ -244,7 +205,7 @@ Example output: ``` NOTE: -This endpoint is only available for Rails web workers. Sidekiq workers can not be inspected this way. +This endpoint is only available for Rails web workers. Sidekiq workers cannot be inspected this way. ## Settings that impact performance @@ -286,7 +247,7 @@ The following table lists these variables along with their default values. ([Source](https://github.com/ruby/ruby/blob/45b29754cfba8435bc4980a87cd0d32c648f8a2e/gc.c#L254-L308)) -GitLab may decide to change these settings in order to speed up application performance, lower memory requirements, or both. +GitLab may decide to change these settings to speed up application performance, lower memory requirements, or both. You can see how each of these settings affect GC performance, memory use and application start-up time for an idle instance of GitLab by running the `scripts/perf/gc/collect_gc_stats.rb` script. It will output GC stats and general timing data to standard diff --git a/doc/development/project_templates.md b/doc/development/project_templates.md index f688d54ad4f..2f1ded23e38 100644 --- a/doc/development/project_templates.md +++ b/doc/development/project_templates.md @@ -1,7 +1,7 @@ --- stage: Manage group: Workspace -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" --- # Contribute a built-in project template @@ -50,7 +50,7 @@ Before GitLab can implement the project template, you must [create a merge reque 1. [Export the project](../user/project/settings/import_export.md#export-a-project-and-its-data) and save the file as `<name>.tar.gz`, where `<name>` is the short name of the project. Move this file to the root directory of `gitlab-org/gitlab`. -1. In `gitlab-org/gitlab`, create and checkout a new branch. +1. In `gitlab-org/gitlab`, create and check out a new branch. 1. Edit the following files to include the project template: - For **non-Enterprise** project templates: - In `lib/gitlab/project_template.rb`, add details about the template diff --git a/doc/development/projections.md b/doc/development/projections.md index e1dc37a7ce8..e45a16c267d 100644 --- a/doc/development/projections.md +++ b/doc/development/projections.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Projections diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index b3f259efc3d..c2caa354567 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -1,7 +1,7 @@ --- stage: Monitor group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Working with Prometheus Metrics **(FREE)** diff --git a/doc/development/pry_debugging.md b/doc/development/pry_debugging.md index 6751559b2ef..8ad584a8edc 100644 --- a/doc/development/pry_debugging.md +++ b/doc/development/pry_debugging.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Pry debugging diff --git a/doc/development/python_guide/index.md b/doc/development/python_guide/index.md index 77dd328b513..08d2f46c1cd 100644 --- a/doc/development/python_guide/index.md +++ b/doc/development/python_guide/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Python Development Guidelines diff --git a/doc/development/rails_initializers.md b/doc/development/rails_initializers.md index fca24cf8c01..24647429a57 100644 --- a/doc/development/rails_initializers.md +++ b/doc/development/rails_initializers.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Rails initializers diff --git a/doc/development/rails_update.md b/doc/development/rails_update.md index bda21860eae..a541de020fb 100644 --- a/doc/development/rails_update.md +++ b/doc/development/rails_update.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Rails update guidelines diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index f300904fc19..505f480c410 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Rake tasks for developers diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index b4deffe162a..3239748193f 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # `ReactiveCaching` diff --git a/doc/development/real_time.md b/doc/development/real_time.md index f113d4dd3b7..2aa48fed8eb 100644 --- a/doc/development/real_time.md +++ b/doc/development/real_time.md @@ -1,7 +1,7 @@ --- stage: Plan group: Project Management -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Real-Time Features @@ -19,7 +19,7 @@ out using the instructions below. ## Reuse an existing WebSocket connection Features reusing an existing connection incur minimal risk. Feature flag rollout -is recommended in order to give more control to self-hosting customers. However, +is recommended to give more control to self-hosting customers. However, it is not necessary to roll out in percentages, or to estimate new connections for GitLab.com. diff --git a/doc/development/redis.md b/doc/development/redis.md index e48048be624..75c7df0737b 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Redis guidelines diff --git a/doc/development/redis/new_redis_instance.md b/doc/development/redis/new_redis_instance.md index 24885b40eb9..d9d4bb6a9f8 100644 --- a/doc/development/redis/new_redis_instance.md +++ b/doc/development/redis/new_redis_instance.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Add a new Redis instance diff --git a/doc/development/refactoring_guide/index.md b/doc/development/refactoring_guide/index.md index 9793db3bb85..5dd505ff5c6 100644 --- a/doc/development/refactoring_guide/index.md +++ b/doc/development/refactoring_guide/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Refactoring guide @@ -18,14 +18,14 @@ Pinning tests help you ensure that you don't unintentionally change the output o 1. For each possible input, identify the significant possible values. 1. Create a test to save a full detailed snapshot for each helpful combination values per input. This should guarantee that we have "pinned down" the current behavior. The snapshot could be literally a screenshot, a dump of HTML, or even an ordered list of debugging statements. 1. Run all the pinning tests against the code, before you start refactoring (Oracle) -1. Perform the refactor (or checkout the commit with the work done) +1. Perform the refactor (or check out the commit with the work done) 1. Run again all the pinning test against the post refactor code (Pin) 1. Compare the Oracle with the Pin. If the Pin is different, you know the refactoring doesn't preserve existing behavior. 1. Repeat the previous three steps as necessary until the refactoring is complete. ### Example commit history -Leaving in the commits for adding and removing pins helps others checkout and verify the result of the test. +Leaving in the commits for adding and removing pins helps others check out and verify the result of the test. ```shell AAAAAA Add pinning tests to funky_foo @@ -77,7 +77,7 @@ expect(cleanForSnapshot(wrapper.element)).toMatchSnapshot(); - [Pinning test in a Haml to Vue refactor](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/27691#pinning-tests) - [Pinning test in isolating a bug](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/32198#note_212736225) -- [Pinning test in refactoring dropdown](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28173) +- [Pinning test in refactoring dropdown list](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28173) - [Pinning test in refactoring vulnerability_details.vue](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25830/commits) - [Pinning test in refactoring notes_award_list.vue](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29528#pinning-test) - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Video of pair programming session on pinning tests](https://youtu.be/LrakPcspBK4) diff --git a/doc/development/reference_processing.md b/doc/development/reference_processing.md index 1dfe6496e79..5a026e0a2a0 100644 --- a/doc/development/reference_processing.md +++ b/doc/development/reference_processing.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: 'An introduction to reference parsers and reference filters, and a guide to their implementation.' --- diff --git a/doc/development/renaming_features.md b/doc/development/renaming_features.md index bd25fa1377e..ee759efd7e2 100644 --- a/doc/development/renaming_features.md +++ b/doc/development/renaming_features.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Renaming features diff --git a/doc/development/repository_mirroring.md b/doc/development/repository_mirroring.md index 573ffaccaf9..a9164398afb 100644 --- a/doc/development/repository_mirroring.md +++ b/doc/development/repository_mirroring.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Repository mirroring diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index 826782d7036..2701192137c 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Guidelines for reusing abstractions @@ -63,7 +63,7 @@ do so, but then we'd need as many options as we have features. Every option adds two code paths, which means that for four features we have to cover 8 different code paths. -A much more reliable (and pleasant) way of dealing with this, is to simply use +A much more reliable (and pleasant) way of dealing with this, is to use the underlying bits that make up `GroupProjectsFinder` directly. This means we may need a little bit more code in `IssuableFinder`, but it also gives us much more control and certainty. This means we might end up with something like this: @@ -96,7 +96,7 @@ This is just a sketch, but it shows the general idea: we would use whatever the ## End goal The guidelines in this document are meant to foster _better_ code reuse, by -clearly defining what can be reused where, and what to do when you can not reuse +clearly defining what can be reused where, and what to do when you cannot reuse something. Clearly separating abstractions makes it harder to use the wrong one, makes it easier to debug the code, and (hopefully) results in fewer performance problems. @@ -122,7 +122,7 @@ the various abstractions and what they can (not) reuse: Everything in `app/controllers`. -Controllers should not do much work on their own, instead they simply pass input +Controllers should not do much work on their own, instead they pass input to other classes and present the results. ### API endpoints @@ -135,40 +135,70 @@ API endpoints have the same abstraction level as controllers. Everything that resides in `app/services`. -Services should consider inheriting from: - -- `BaseContainerService` for services scoped by container (project or group) -- `BaseProjectService` for services scoped to projects -- `BaseGroupService` for services scoped to groups - -or, create a new base class and update the list above. - -Legacy classes inherited from `BaseService` for historical reasons. - -In Service classes the use of `execute` and `#execute` is preferred over `call` and `#call`. - -Model properties should be passed to the constructor in a `params` hash, and will be assigned directly. - -To pass extra parameters (which need to be processed, and are not model properties), -include an `options` hash in the constructor and store it in an instance variable: - -```ruby -# container: Project, or Group -# current_user: Current user -# params: Model properties from the controller, already allowlisted with strong parameters -# options: Configuration for this service, can be any of the following: -# notify: Whether to send a notifcation to the current user -# cc: Email address to copy when sending a notification -def initialize(container:, current_user: nil, params: {}, options: {}) - super(container, current_user, params) - @options = options -end -``` - -View the [initial discussion](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90008#note_988744060) -and [further discussion](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90853#note_1053425083). - -Classes that are not service objects should be [created elsewhere](directory_structure.md#use-namespaces-to-define-bounded-contexts), such as in `lib`. +Service classes represent operations that coordinates changes between models +(such as entities and value objects). Changes impact the state of the application. + +1. When an object makes no changes to the state of the application, then it's not a service. + It may be a [finder](#finders) or a value object. +1. When there is no operation, there is no need to execute a service. The class would + probably be better designed as an entity, a value object, or a policy. + +When implementing a service class, consider: + +1. A service class initializer should contain in its arguments: + 1. A [model](#models) instance that is being acted upon. Should be first positional + argument of the initializer. The argument name of the argument is left to the + developer's discretion, such as: `issue`, `project`, `merge_request`. + 1. When service represents an action initiated by a user or executed in the + context of a user, the initializer must have the `current_user:` keyword argument. + Services with `current_user:` argument run high-level business logic. + 1. When service does not have a user context and it's not directly initiated + by a user (like background service or side-effects), the `current_user:` + argument is not needed. This describes low-level domain logic or instance-wide logic. + 1. For all additional data required by a service, the explicit keyword arguments are recommended. + When a service requires too long of a list of arguments, consider splitting them into: + - `params`: A hash with model properties that will be assigned directly. + - `options`: A hash with extra parameters (which need to be processed, + and are not model properties). The `options` hash should be stored in an instance variable. + + ```ruby + # merge_request: A model instance that is being acted upon. + # assignee: new MR assignee that will be assigned to the MR + # after the service is executed. + def initialize(merge_request, assignee:) + @merge_request = merge_request + @assignee = assignee + end + ``` + + ```ruby + # issue: A model instance that is being acted upon. + # current_user: Current user. + # params: Model properties. + # options: Configuration for this service. Can be any of the following: + # - notify: Whether to send a notification to the current user. + # - cc: Email address to copy when sending a notification. + def initialize(issue:, current_user:, params: {}, options: {}) + @issue = issue + @current_user = current_user + @params = params + @options = options + end + ``` + +1. It implements a single public instance method `#execute`, which invokes service class behavior: + - The `#execute` method takes no arguments. All required data is passed into initializer. + - Optional. If needed, the `#execute` method returns its result via [`ServiceResponse`](#serviceresponse). + +Several base classes implement the service classes convention. You may consider inheriting from: + +- `BaseContainerService` for services scoped by container (project or group). +- `BaseProjectService` for services scoped to projects. +- `BaseGroupService` for services scoped to groups. + +Classes that are not service objects should be +[created elsewhere](directory_structure.md#use-namespaces-to-define-bounded-contexts), +such as in `lib`. #### ServiceResponse @@ -235,7 +265,7 @@ For example: `:job_not_retriable`, `:duplicate_package`, `:merge_request_not_mer Everything in `app/finders`, typically used for retrieving data from a database. -Finders can not reuse other finders in an attempt to better control the SQL +Finders cannot reuse other finders in an attempt to better control the SQL queries they produce. Finders' `execute` method should return `ActiveRecord::Relation`. Exceptions diff --git a/doc/development/routing.md b/doc/development/routing.md index 54a531730f9..16ed15fdcc6 100644 --- a/doc/development/routing.md +++ b/doc/development/routing.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Routing diff --git a/doc/development/ruby3_gotchas.md b/doc/development/ruby3_gotchas.md index db328b0b1a5..4768d6e1182 100644 --- a/doc/development/ruby3_gotchas.md +++ b/doc/development/ruby3_gotchas.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Ruby 3 gotchas diff --git a/doc/development/ruby_upgrade.md b/doc/development/ruby_upgrade.md index 3b89a6fd1ea..ccd65b4e7e9 100644 --- a/doc/development/ruby_upgrade.md +++ b/doc/development/ruby_upgrade.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Ruby upgrade guidelines diff --git a/doc/development/scalability.md b/doc/development/scalability.md index 66f436bd391..671086f33b2 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab scalability @@ -216,7 +216,7 @@ core. It does not support multi-threading. Dumb secondaries: Redis secondaries (also known as replicas) don't actually handle any load. Unlike PostgreSQL secondaries, they don't even serve -read queries. They simply replicate data from the primary and take over +read queries. They replicate data from the primary and take over only when the primary fails. ### Redis Sentinels diff --git a/doc/development/sec/analyzer_development_guide.md b/doc/development/sec/analyzer_development_guide.md new file mode 100644 index 00000000000..a35bc2b7237 --- /dev/null +++ b/doc/development/sec/analyzer_development_guide.md @@ -0,0 +1,185 @@ +--- +stage: Secure +group: Static Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Sec section analyzer development + +Analyzers are shipped as Docker images to execute within a CI pipeline context. This guide describes development and testing +practices across analyzers. + +## Shared modules + +There are a number of shared Go modules shared across analyzers for common behavior and interfaces: + +- The [`command`](https://gitlab.com/gitlab-org/security-products/analyzers/command#how-to-use-the-library) Go package implements a CLI interface. +- The [`common`](https://gitlab.com/gitlab-org/security-products/analyzers/common) project provides miscellaneous shared modules for logging, certificate handling, and directory search capabilities. +- The [`report`](https://gitlab.com/gitlab-org/security-products/analyzers/report) Go package's `Report` and `Finding` structs marshal JSON reports. +- The [`template`](https://gitlab.com/gitlab-org/security-products/analyzers/template) project scaffolds new analyzers. + +## How to use the analyzers + +Analyzers are shipped as Docker images. For example, to run the +[semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) Docker image to scan the working directory: + +1. `cd` into the directory of the source code you want to scan. +1. Run `docker login registry.gitlab.com` and provide username plus + [personal](../../user/profile/personal_access_tokens.md#create-a-personal-access-token) + or [project](../../user/project/settings/project_access_tokens.md#create-a-project-access-token) + access token with at least the `read_registry` scope. +1. Run the Docker image: + + ```shell + docker run \ + --interactive --tty --rm \ + --volume "$PWD":/tmp/app \ + --env CI_PROJECT_DIR=/tmp/app \ + -w /tmp/app \ + registry.gitlab.com/gitlab-org/security-products/analyzers/semgrep:latest /analyzer run + ``` + +1. The Docker container generates a report in the mounted project directory with a report filename corresponding to the analyzer category. For example, [SAST](../../user/application_security/sast) generates a file named `gl-sast-report.json`. + +## Analyzers development + +To update the analyzer: + +1. Modify the Go source code. +1. Build a new Docker image. +1. Run the analyzer against its test project. +1. Compare the generated report with what's expected. + +Here's how to create a Docker image named `analyzer`: + +```shell +docker build -t analyzer . +``` + +For example, to test Secret Detection run the following: + +```shell +wget https://gitlab.com/gitlab-org/security-products/ci-templates/-/raw/master/scripts/compare_reports.sh +sh ./compare_reports.sh sd test/fixtures/gl-secret-detection-report.json test/expect/gl-secret-detection-report.json \ +| patch -Np1 test/expect/gl-secret-detection-report.json && Git commit -m 'Update expectation' test/expect/gl-secret-detection-report.json +rm compare_reports.sh +``` + +You can also compile the binary for your own environment and run it locally +but `analyze` and `run` probably won't work +since the runtime dependencies of the analyzer are missing. + +Here's an example based on +[SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs): + +```shell +go build -o analyzer +./analyzer search test/fixtures +./analyzer convert test/fixtures/app/spotbugsXml.Xml > ./gl-sast-report.json +``` + +## How to test the analyzers + +Video walkthrough of how Dependency Scanning analyzers are using [downstream pipeline](../../ci/pipelines/downstream_pipelines.md) feature to test analyzers using test projects: + +[](http://www.youtube.com/watch?v=KauRBlfUbDE) + +### Testing local changes + +To test local changes in the shared modules (such as `command` or `report`) for an analyzer +you can use the +[`go mod replace`](https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive) +directive to load `command` with your local changes instead of using the version of command that has been +tagged remotely. For example: + +```shell +go mod edit -replace gitlab.com/gitlab-org/security-products/analyzers/command/v3=/local/path/to/command +``` + +Alternatively you can achieve the same result by manually updating the `go.mod` file: + +```plaintext +module gitlab.com/gitlab-org/security-products/analyzers/awesome-analyzer/v2 + +replace gitlab.com/gitlab-org/security-products/analyzers/command/v3 => /path/to/command + +require ( + ... + gitlab.com/gitlab-org/security-products/analyzers/command/v3 v2.19.0 +) +``` + +#### Testing local changes in Docker + +To use Docker with `replace` in the `go.mod` file: + +1. Copy the contents of `command` into the directory of the analyzer. `cp -r /path/to/command path/to/analyzer/command`. +1. Add a copy statement in the analyzer's `Dockerfile`: `COPY command /command`. +1. Update the `replace` statement to make sure it matches the destination of the `COPY` statement in the step above: +`replace gitlab.com/gitlab-org/security-products/analyzers/command/v3 => /command` + +## Versioning and release process + +Analyzers are independent projects that follow their own versioning. `Patch` version bumps tend to correspond to a `Minor` version bump of the underlying tools (i.e. [`bandit`](https://wiki.openstack.org/wiki/Security/Projects/Bandit)), allowing us greater flexibility in reserving `Minor` bumps for more significant changes to our scanners. In case of breaking changes imposed by the wrapped scanner, creating a new analyzer on a separate repository must be considered. + +The analyzers are released as Docker images following this scheme: + +- each push to the `master` branch will override the `edge` image tag +- each push to any `awesome-feature` branch will generate a matching `awesome-feature` image tag +- each Git tag will generate the corresponding `Major.Minor.Patch` image tag. A manual job allows to override the corresponding `Major` and the `latest` image tags to point to this `Major.Minor.Patch`. + +To release a new analyzer Docker image, there are two different options: + +- Manual release process +- Automatic release process + +### Manual release process + +1. Ensure that the `CHANGELOG.md` entry for the new analyzer is correct. +1. Ensure that the release source (typically the `master` or `main` branch) has a passing pipeline. +1. Create a new release for the analyzer project by selecting the **Deployments** menu on the left-hand side of the project window, then selecting the **Releases** sub-menu. +1. Select **New release** to open the **New Release** page. + 1. In the **Tag name** drop down, enter the same version used in the `CHANGELOG.md`, for example `v2.4.2`, and select the option to create the tag (`Create tag v2.4.2` here). + 1. In the **Release title** text box enter the same version used above, for example `v2.4.2`. + 1. In the `Release notes` text box, copy and paste the notes from the corresponding version in the `CHANGELOG.md`. + 1. Leave all other settings as the default values. + 1. Select **Create release**. + +After following the above process and creating a new release, a new Git tag is created with the `Tag name` provided above. This triggers a new pipeline with the given tag version and a new analyzer Docker image is built. + +If the analyzer uses the [`analyzer.yml` template](https://gitlab.com/gitlab-org/security-products/ci-templates/blob/b446fd3/includes-dev/analyzer.yml#L209-217), then the pipeline triggered as part of the **New release** process above automatically tags and deploys a new version of the analyzer Docker image. + +If the analyzer does not use the `analyzer.yml` template, you'll need to manually tag and deploy a new version of the analyzer Docker image: + +1. Select the **CI/CD** menu on the left-hand side of the project window, then select the **Pipelines** sub-menu. +1. A new pipeline should currently be running with the same tag used previously, for example `v2.4.2`. +1. After the pipeline has completed, it will be in a `blocked` state. +1. Select the `Manual job` play button on the right hand side of the window and select `tag version` to tag and deploy a new version of the analyzer Docker image. + +Use your best judgment to decide when to create a Git tag, which will then trigger the release job. If you +can't decide, then ask for other's input. + +### Automatic release process + +The following must be performed before the automatic release process can be used: + +1. Configure `CREATE_GIT_TAG: true` as a [`CI/CD` environment variable](../../ci/variables/index.md). +1. Check the `Variables` in the CI/CD project settings. Unless the project already inherits the `GITLAB_TOKEN` environment variable from the project group, create a [project access token](../../user/project/settings/project_access_tokens.md) with `complete read/write access to the API` and configure `GITLAB_TOKEN` as a [`CI/CD` environment variable](../../ci/variables/index.md) which refers to this token. + +After the above steps have been completed, the automatic release process executes as follows: + +1. A project maintainer merges an MR into the default branch. +1. The default pipeline is triggered, and the `upsert git tag` job is executed. + - If the most recent version in the `CHANGELOG.md` matches one of the Git tags, the job is a no-op. + - Else, this job automatically creates a new release and Git tag using the [releases API](../../api/releases/index.md#create-a-release). The version and message is obtained from the most recent entry in the `CHANGELOG.md` file for the project. +1. A pipeline is automatically triggered for the new Git tag. This pipeline releases the `latest`, `major`, `minor` and `patch` Docker images of the analyzer. + +### Steps to perform after releasing an analyzer + +1. After a new version of the analyzer Docker image has been tagged and deployed, please test it with the corresponding test project. +1. Announce the release on the relevant group Slack channel. Example message: + + > FYI I've just released `ANALYZER_NAME` `ANALYZER_VERSION`. `LINK_TO_RELEASE` + +**Never delete a Git tag that has been pushed** as there is a good +chance that the tag will be used and/or cached by the Go package registry. diff --git a/doc/development/sec/index.md b/doc/development/sec/index.md index 06c20cee0bb..c9805044e58 100644 --- a/doc/development/sec/index.md +++ b/doc/development/sec/index.md @@ -1,7 +1,7 @@ --- stage: Secure group: Static Analysis -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: index, concepts, howto --- @@ -18,6 +18,7 @@ See [Terminology](../../user/application_security/terminology) for an overview o - [Scanning](#scanning) - [Processing, visualization, and management](#processing-visualization-and-management) - [Severity Levels](../../user/application_security/vulnerabilities/severities.md) +- [Analyzer Development](analyzer_development_guide.md) ## Overview @@ -45,7 +46,7 @@ flowchart LR ### Scanning The scanning part is responsible for finding vulnerabilities in given resources, and exporting results. -The scans are executed in CI/CD jobs via several small projects called [Analyzers](../../user/application_security/terminology/index.md#analyzer), which can be found in our [Analyzers sub-group](https://gitlab.com/gitlab-org/security-products/analyzers). +The scans are executed in CI/CD jobs via several small projects called [Analyzers](../../user/application_security/terminology/index.md#analyzer), which can be found in our [Analyzers subgroup](https://gitlab.com/gitlab-org/security-products/analyzers). The Analyzers are wrappers around security tools called [Scanners](../../user/application_security/terminology/index.md#scanner), developed internally or externally, to integrate them into GitLab. The Analyzers are mainly written in Go. @@ -65,5 +66,5 @@ Depending on the context, the security reports may be stored either in the datab ## CI/CD template development -While CI/CD templates are the responsibiility of the Verify section, many are critical to the Sec Section's feature usage. +While CI/CD templates are the responsibility of the Verify section, many are critical to the Sec Section's feature usage. If you are working with CI/CD templates, please read the [development guide for GitLab CI/CD templates](../cicd/templates.md). diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 4c2f3118366..67f7c556055 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -2,7 +2,7 @@ type: reference, dev stage: none group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" --- # Secure Coding Guidelines @@ -53,6 +53,10 @@ Each time you implement a new feature/endpoint, whether it is at UI, API or Grap Be careful to **also test [visibility levels](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/doc/development/permissions.md#feature-specific-permissions)** and not only project access rights. +The HTTP status code returned when an authorization check fails should generally be `404 Not Found` to avoid revealing information +about whether or not the requested resource exists. `403 Forbidden` may be appropriate if you need to display a specific message to the user +about why they cannot access the resource. If you are displaying a generic message such as "access denied", consider returning `404 Not Found` instead. + Some example of well implemented access controls and tests: 1. [example1](https://dev.gitlab.org/gitlab/gitlab-ee/-/merge_requests/710/diffs?diff_id=13750#af40ef0eaae3c1e018809e1d88086e32bccaca40_43_43) @@ -1067,7 +1071,7 @@ Symlink attacks makes it possible for an attacker to read the contents of arbitr #### Ruby -For zip files, the [`rubyzip`](https://rubygems.org/gems/rubyzip) Ruby gem is already patched against symlink attacks as it simply ignores symbolic links, so for this vulnerable example we will extract a `tar.gz` file with `Gem::Package::TarReader`: +For zip files, the [`rubyzip`](https://rubygems.org/gems/rubyzip) Ruby gem is already patched against symlink attacks as it ignores symbolic links, so for this vulnerable example we will extract a `tar.gz` file with `Gem::Package::TarReader`: ```ruby # Vulnerable tar.gz extraction example! diff --git a/doc/development/service_measurement.md b/doc/development/service_measurement.md index 17e509672f2..9d22e9c376c 100644 --- a/doc/development/service_measurement.md +++ b/doc/development/service_measurement.md @@ -1,12 +1,12 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab Developers Guide to service measurement -You can enable service measurement in order to debug any slow service's execution time, number of SQL calls, garbage collection stats, memory usage, etc. +You can enable service measurement to debug any slow service's execution time, number of SQL calls, garbage collection stats, memory usage, etc. ## Measuring module @@ -43,7 +43,7 @@ DummyService.prepend(Measurable) In case when you are prepending a module from the `EE` namespace with EE features, you need to prepend Measurable after prepending the `EE` module. -This way, `Measurable` is at the bottom of the ancestor chain, in order to measure execution of `EE` features as well: +This way, `Measurable` is at the bottom of the ancestor chain, to measure execution of `EE` features as well: ```ruby class DummyService diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 5448bbb4293..88a1454393f 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Implement Service Ping @@ -179,7 +179,7 @@ As the HyperLogLog algorithm is probabilistic, the **results always include erro The highest encountered error rate is 4.9%. When correctly used, the `estimate_batch_distinct_count` method enables efficient counting over -columns that contain non-unique values, which can not be assured by other counters. +columns that contain non-unique values, which cannot be assured by other counters. ##### estimate_batch_distinct_count method @@ -605,7 +605,7 @@ alt_usage_data(value = nil, fallback: -1, &block) Arguments: -- `value`: a simple static value in which case the value is simply returned. +- `value`: a static value in which case the value is returned. - or a `block`: which is evaluated - `fallback: -1`: the common value used for any metrics that are failing. @@ -712,10 +712,10 @@ We also use `#database-lab` and [explain.depesz.com](https://explain.depesz.com/ - Use specialized indexes. For examples, see these merge requests: - [Example 1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26871) - [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26445) -- Use defined `start` and `finish`, and simple queries. - These values can be memoized and reused, as in this [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37155). -- Avoid joins and write the queries as simply as possible, - as in this [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36316). +- Use defined `start` and `finish`. These values can be memoized and reused, as in this + [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37155). +- Avoid joins and unnecessary complexity in your queries. See this + [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36316) as an example. - Set a custom `batch_size` for `distinct_count`, as in this [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38000). ## Add the metric definition @@ -839,6 +839,22 @@ However, it has the following limitations: WARNING: This feature is intended solely for internal GitLab use. +The aggregated metrics feature provides insight into the data attributes in a collection of Service Ping metrics. +This aggregation allows you to count data attributes in events without counting each occurrence of the same data attribute in multiple events. +For example, you can aggregate the number of users who perform several actions, such as creating a new issue and opening a new merge request. +You can then count each user that performed any combination of these actions. + +### Defining aggregated metric via metric YAML definition + +To add data for aggregated metrics to the Service Ping payload, +create metric YAML definition file following [Aggregated metric instrumentation guide](metrics_instrumentation.md#aggregated-metrics). + +### (DEPRECATED) Defining aggregated metric via aggregated metric YAML config file + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98206) in GitLab 15.5 +and is planned for removal in 15.5. Use [metrics definition YAMLs](https://gitlab.com/gitlab-org/gitlab/-/issues/370963) instead. + To add data for aggregated metrics to the Service Ping payload, add a corresponding definition to: - [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/) for metrics available in the Community Edition. diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index f252eb967aa..0a4e5998345 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Service Ping Guide **(FREE SELF)** diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index d063c4c7601..3439f581e7f 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Metrics Dictionary Guide @@ -136,6 +136,9 @@ We use the following categories to classify a metric: - `subscription`: Data related to licensing. - `standard`: Standard set of identifiers that are included when collecting data. +An aggregate metric is a metric that is the sum of two or more child metrics. Service Ping uses the data category of +the aggregate metric to determine whether or not the data is included in the reported Service Ping payload. + ### Metric name suggestion examples #### Metric with `data_source: database` diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md index debb3a68c04..7a3f291460c 100644 --- a/doc/development/service_ping/metrics_instrumentation.md +++ b/doc/development/service_ping/metrics_instrumentation.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Metrics instrumentation guide @@ -254,6 +254,86 @@ options: - i_quickactions_approve ``` +## Aggregated metrics + +The aggregated metrics feature provides insight into the number of data attributes, for example `pseudonymized_user_ids`, that occurred in a collection of events. For example, you can aggregate the number of users who perform multiple actions such as creating a new issue and opening +a new merge request. + +You can use a YAML file to define your aggregated metrics. The following arguments are required: + +- `options.events`: List of event names to aggregate into metric data. All events in this list must + use the same data source. Additional data source requirements are described in + [Database sourced aggregated metrics](implement.md#database-sourced-aggregated-metrics) and + [Redis sourced aggregated metrics](implement.md#redis-sourced-aggregated-metrics). +- `options.aggregate.operator`: Operator that defines how the aggregated metric data is counted. Available operators are: + - `OR`: Removes duplicates and counts all entries that triggered any of the listed events. + - `AND`: Removes duplicates and counts all elements that were observed triggering all of the following events. +- `options.aggregate.attribute`: Information pointing to the attribute that is being aggregated across events. +- `time_frame`: One or more valid time frames. Use these to limit the data included in aggregated metrics to events within a specific date-range. Valid time frames are: + - `7d`: The last 7 days of data. + - `28d`: The last 28 days of data. + - `all`: All historical data, only available for `database` sourced aggregated metrics. +- `data_source`: Data source used to collect all events data included in the aggregated metrics. Valid data sources are: + - [`database`](implement.md#database-sourced-aggregated-metrics) + - [`redis_hll`](implement.md#redis-sourced-aggregated-metrics) + +Refer to merge request [98206](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98206) for an example of a merge request that adds an `AggregatedMetric` metric. + +Count unique `user_ids` that occurred in at least one of the events: `incident_management_alert_status_changed`, +`incident_management_alert_assigned`, `incident_management_alert_todo`, `incident_management_alert_create_incident`. + +```yaml +time_frame: 28d +instrumentation_class: AggregatedMetric +data_source: redis_hll +options: + aggregate: + operator: OR + attribute: user_id + events: + - `incident_management_alert_status_changed` + - `incident_management_alert_assigned` + - `incident_management_alert_todo` + - `incident_management_alert_create_incident` +``` + +### Availability-restrained Aggregated metrics + +If the Aggregated metric should only be available in the report under specific conditions, then you must specify these conditions in a new class that is a child of the `AggregatedMetric` class. + +```ruby +# frozen_string_literal: true + +module Gitlab + module Usage + module Metrics + module Instrumentations + class MergeUsageCountAggregatedMetric < AggregatedMetric + available? { Feature.enabled?(:merge_usage_data_missing_key_paths) } + end + end + end + end +end +``` + +You must also use the class's name in the YAML setup. + +```yaml +time_frame: 28d +instrumentation_class: MergeUsageCountAggregatedMetric +data_source: redis_hll +options: + aggregate: + operator: OR + attribute: user_id + events: + - `incident_management_alert_status_changed` + - `incident_management_alert_assigned` + - `incident_management_alert_todo` + - `incident_management_alert_create_incident` +``` + ## Numbers metrics - `operation`: Operations for the given `data` block. Currently we only support `add` operation. diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md index 28f77b6f587..f13aebeb16e 100644 --- a/doc/development/service_ping/metrics_lifecycle.md +++ b/doc/development/service_ping/metrics_lifecycle.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Service Ping metric lifecycle @@ -120,7 +120,7 @@ To remove a metric: Do not remove the metric's YAML definition altogether. Some self-managed instances might not immediately update to the latest version of GitLab, and therefore continue to report the removed metric. The Product Intelligence team - requires a record of all removed metrics in order to identify and filter them. + requires a record of all removed metrics to identify and filter them. For example please take a look at this [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60149/diffs#b01f429a54843feb22265100c0e4fec1b7da1240_10_10). diff --git a/doc/development/service_ping/performance_indicator_metrics.md b/doc/development/service_ping/performance_indicator_metrics.md index d2abc597a22..4c1c61aa05b 100644 --- a/doc/development/service_ping/performance_indicator_metrics.md +++ b/doc/development/service_ping/performance_indicator_metrics.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Performance Indicator Metrics guide diff --git a/doc/development/service_ping/review_guidelines.md b/doc/development/service_ping/review_guidelines.md index 1b00858be7e..a1806551303 100644 --- a/doc/development/service_ping/review_guidelines.md +++ b/doc/development/service_ping/review_guidelines.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Service Ping review guidelines diff --git a/doc/development/service_ping/troubleshooting.md b/doc/development/service_ping/troubleshooting.md index 4431d5df3ff..201d6a385eb 100644 --- a/doc/development/service_ping/troubleshooting.md +++ b/doc/development/service_ping/troubleshooting.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Troubleshooting Service Ping diff --git a/doc/development/service_ping/usage_data.md b/doc/development/service_ping/usage_data.md index 4181bd90a02..2f2df14f56d 100644 --- a/doc/development/service_ping/usage_data.md +++ b/doc/development/service_ping/usage_data.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Usage Data Metrics guide diff --git a/doc/development/session.md b/doc/development/session.md index 61a130e3a53..c30364c27f8 100644 --- a/doc/development/session.md +++ b/doc/development/session.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Accessing session data diff --git a/doc/development/shared_files.md b/doc/development/shared_files.md index 4f13bb80761..3bf18a4d410 100644 --- a/doc/development/shared_files.md +++ b/doc/development/shared_files.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Shared files diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index d8a3f86685e..3935e98199a 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Guidelines for shell commands in the GitLab codebase diff --git a/doc/development/shell_scripting_guide/index.md b/doc/development/shell_scripting_guide/index.md index 0591d2c64d0..d3d9304446e 100644 --- a/doc/development/shell_scripting_guide/index.md +++ b/doc/development/shell_scripting_guide/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Shell scripting standards and style guidelines diff --git a/doc/development/sidekiq/compatibility_across_updates.md b/doc/development/sidekiq/compatibility_across_updates.md index ac34d099202..cfb709d9110 100644 --- a/doc/development/sidekiq/compatibility_across_updates.md +++ b/doc/development/sidekiq/compatibility_across_updates.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Sidekiq Compatibility across Updates diff --git a/doc/development/sidekiq/idempotent_jobs.md b/doc/development/sidekiq/idempotent_jobs.md index da36cdc72aa..80c6c403549 100644 --- a/doc/development/sidekiq/idempotent_jobs.md +++ b/doc/development/sidekiq/idempotent_jobs.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Sidekiq idempotent jobs diff --git a/doc/development/sidekiq/index.md b/doc/development/sidekiq/index.md index 003f54d48b5..e8c939571cf 100644 --- a/doc/development/sidekiq/index.md +++ b/doc/development/sidekiq/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Sidekiq guides @@ -186,3 +186,9 @@ default weight, which is 1. Each Sidekiq worker must be tested using RSpec, just like any other class. These tests should be placed in `spec/workers`. + +## Interacting with Sidekiq Redis and APIs + +The application should minimise interaction with of any `Sidekiq.redis` and Sidekiq [APIs](https://github.com/mperham/sidekiq/blob/main/lib/sidekiq/api.rb). Such interactions in generic application logic should be abstracted to a [Sidekiq middleware](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/sidekiq_middleware) for re-use across teams. By decoupling application logic from Sidekiq's datastore, it allows for greater freedom when horizontally scaling the GitLab background processing setup. + +Some exceptions to this rule would be migration-related logic or administration operations. diff --git a/doc/development/sidekiq/limited_capacity_worker.md b/doc/development/sidekiq/limited_capacity_worker.md index 5b86e01781c..5efb9b16725 100644 --- a/doc/development/sidekiq/limited_capacity_worker.md +++ b/doc/development/sidekiq/limited_capacity_worker.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Sidekiq limited capacity worker diff --git a/doc/development/sidekiq/logging.md b/doc/development/sidekiq/logging.md index b461047ea47..8a2a77b5226 100644 --- a/doc/development/sidekiq/logging.md +++ b/doc/development/sidekiq/logging.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Sidekiq logging diff --git a/doc/development/sidekiq/worker_attributes.md b/doc/development/sidekiq/worker_attributes.md index a1d24d0c392..48a222d65a0 100644 --- a/doc/development/sidekiq/worker_attributes.md +++ b/doc/development/sidekiq/worker_attributes.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Sidekiq worker attributes @@ -60,7 +60,7 @@ that branch, but be told in the UI that the branch does not exist. We deem these jobs to be `urgency :high`. Extra effort is made to ensure that these jobs are started within a very short -period of time after being scheduled. However, in order to ensure throughput, +period of time after being scheduled. However, to ensure throughput, these jobs also have very strict execution duration requirements: 1. The median job execution time should be less than 1 second. @@ -117,7 +117,7 @@ Most background jobs in the GitLab application communicate with other GitLab services. For example, PostgreSQL, Redis, Gitaly, and Object Storage. These are considered to be "internal" dependencies for a job. -However, some jobs are dependent on external services in order to complete +However, some jobs are dependent on external services to complete successfully. Some examples include: 1. Jobs which call web-hooks configured by a user. diff --git a/doc/development/snowplow/event_dictionary_guide.md b/doc/development/snowplow/event_dictionary_guide.md index 7980395b1a9..c5a21f5a081 100644 --- a/doc/development/snowplow/event_dictionary_guide.md +++ b/doc/development/snowplow/event_dictionary_guide.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Event dictionary guide diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md index 9a923b115a2..26016eeba84 100644 --- a/doc/development/snowplow/implementation.md +++ b/doc/development/snowplow/implementation.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Implement Snowplow tracking diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index 42a968b9df0..7327f18fcaf 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Snowplow @@ -90,7 +90,7 @@ Each click event provides attributes that describe the event. | Attribute | Type | Required | Description | | --------- | ------- | -------- | ----------- | | category | text | true | The page or backend section of the application. Unless infeasible, use the Rails page attribute by default in the frontend, and namespace + class name on the backend, for example, `Notes::CreateService`. | -| action | text | true | The action the user takes, or aspect that's being instrumented. The first word must describe the action or aspect. For example, clicks must be `click`, activations must be `activate`, creations must be `create`. Use underscores to describe what was acted on. For example, activating a form field is `activate_form_input`, an interface action like clicking on a dropdown is `click_dropdown`, a behavior like creating a project record from the backend is `create_project`. | +| action | text | true | The action the user takes, or aspect that's being instrumented. The first word must describe the action or aspect. For example, clicks must be `click`, activations must be `activate`, creations must be `create`. Use underscores to describe what was acted on. For example, activating a form field is `activate_form_input`, an interface action like clicking on a dropdown list is `click_dropdown`, a behavior like creating a project record from the backend is `create_project`. | | label | text | false | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. For Service Ping metrics adapted to Snowplow events, this should be the full metric [key path](../service_ping/metrics_dictionary.md#metric-key_path) taken from its definition file. | | property | text | false | Any additional property of the element, or object being acted on. For Service Ping metrics adapted to Snowplow events, this should be additional information or context that can help analyze the event. For example, in the case of `usage_activity_by_stage_monthly.create.merge_requests_users`, there are four different possible merge request actions: "create", "merge", "comment", and "close". Each of these would be a possible property value. | | value | decimal | false | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | diff --git a/doc/development/snowplow/infrastructure.md b/doc/development/snowplow/infrastructure.md index ea4653dc91d..ae416f40c98 100644 --- a/doc/development/snowplow/infrastructure.md +++ b/doc/development/snowplow/infrastructure.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Snowplow infrastructure diff --git a/doc/development/snowplow/review_guidelines.md b/doc/development/snowplow/review_guidelines.md index 44de849792c..756dbd0ca92 100644 --- a/doc/development/snowplow/review_guidelines.md +++ b/doc/development/snowplow/review_guidelines.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Snowplow review guidelines diff --git a/doc/development/snowplow/schemas.md b/doc/development/snowplow/schemas.md index 799f8335000..482c0e0105b 100644 --- a/doc/development/snowplow/schemas.md +++ b/doc/development/snowplow/schemas.md @@ -1,7 +1,7 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Snowplow schemas diff --git a/doc/development/snowplow/troubleshooting.md b/doc/development/snowplow/troubleshooting.md index 3ad4c6c9549..306040f8c9c 100644 --- a/doc/development/snowplow/troubleshooting.md +++ b/doc/development/snowplow/troubleshooting.md @@ -1,16 +1,16 @@ --- stage: Analytics group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Troubleshooting Snowplow ## Monitoring -This page covers dashboards and alerts coming from a number of internal tools. +This page covers dashboards and alerts coming from a number of internal tools. -For a brief video overview of the tools used to monitor Snowplow usage, please check out [this internal video](https://www.youtube.com/watch?v=NxPS0aKa_oU) (you must be logged into GitLab Unfiltered to view). +For a brief video overview of the tools used to monitor Snowplow usage, please check out [this internal video](https://www.youtube.com/watch?v=NxPS0aKa_oU) (you must be logged into GitLab Unfiltered to view). ## Good events drop @@ -74,7 +74,7 @@ Enrichers are not scalling well for the amount of events we receive. See the [dashboard](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#dashboards:name=SnowPlow). -Could we get assistance in order to fix the delay? +Could we get assistance to fix the delay? Thank you! ``` diff --git a/doc/development/spam_protection_and_captcha/exploratory_testing.md b/doc/development/spam_protection_and_captcha/exploratory_testing.md index e508265cf83..e17d25acb9d 100644 --- a/doc/development/spam_protection_and_captcha/exploratory_testing.md +++ b/doc/development/spam_protection_and_captcha/exploratory_testing.md @@ -1,7 +1,7 @@ --- stage: Manage group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Exploratory testing of CAPTCHAs diff --git a/doc/development/spam_protection_and_captcha/graphql_api.md b/doc/development/spam_protection_and_captcha/graphql_api.md index e3f4e9069e5..846165d35f1 100644 --- a/doc/development/spam_protection_and_captcha/graphql_api.md +++ b/doc/development/spam_protection_and_captcha/graphql_api.md @@ -1,7 +1,7 @@ --- stage: Manage group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GraphQL API spam protection and CAPTCHA support diff --git a/doc/development/spam_protection_and_captcha/index.md b/doc/development/spam_protection_and_captcha/index.md index dbe8c4aa4e9..1ba1c1678c4 100644 --- a/doc/development/spam_protection_and_captcha/index.md +++ b/doc/development/spam_protection_and_captcha/index.md @@ -1,7 +1,7 @@ --- stage: Manage group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Spam protection and CAPTCHA diff --git a/doc/development/spam_protection_and_captcha/model_and_services.md b/doc/development/spam_protection_and_captcha/model_and_services.md index d0519cba68f..d0ca6d350dc 100644 --- a/doc/development/spam_protection_and_captcha/model_and_services.md +++ b/doc/development/spam_protection_and_captcha/model_and_services.md @@ -1,7 +1,7 @@ --- stage: Manage group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Model and services spam protection and CAPTCHA support diff --git a/doc/development/spam_protection_and_captcha/rest_api.md b/doc/development/spam_protection_and_captcha/rest_api.md index ad74977eb67..7a9f8b5def1 100644 --- a/doc/development/spam_protection_and_captcha/rest_api.md +++ b/doc/development/spam_protection_and_captcha/rest_api.md @@ -1,7 +1,7 @@ --- stage: Manage group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # REST API spam protection and CAPTCHA support diff --git a/doc/development/spam_protection_and_captcha/web_ui.md b/doc/development/spam_protection_and_captcha/web_ui.md index 9aeb9e96d44..ff7049dd29f 100644 --- a/doc/development/spam_protection_and_captcha/web_ui.md +++ b/doc/development/spam_protection_and_captcha/web_ui.md @@ -1,7 +1,7 @@ --- stage: Manage group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Web UI spam protection and CAPTCHA support diff --git a/doc/development/sql.md b/doc/development/sql.md index 029874011c4..5829d27b8ee 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -1,7 +1,7 @@ --- stage: Data Stores group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # SQL Query Guidelines diff --git a/doc/development/stage_group_observability/dashboards/error_budget_detail.md b/doc/development/stage_group_observability/dashboards/error_budget_detail.md index 19f98d404e7..a8f932b78c0 100644 --- a/doc/development/stage_group_observability/dashboards/error_budget_detail.md +++ b/doc/development/stage_group_observability/dashboards/error_budget_detail.md @@ -1,7 +1,7 @@ --- stage: Platforms group: Scalability -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Error budget detail dashboard diff --git a/doc/development/stage_group_observability/dashboards/index.md b/doc/development/stage_group_observability/dashboards/index.md index f4e646c8634..9f9ba609271 100644 --- a/doc/development/stage_group_observability/dashboards/index.md +++ b/doc/development/stage_group_observability/dashboards/index.md @@ -1,7 +1,7 @@ --- stage: Platforms group: Scalability -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Dashboards for stage groups diff --git a/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md b/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md index c8c18b93a8f..fb5d5bbe379 100644 --- a/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md +++ b/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md @@ -1,7 +1,7 @@ --- stage: Platforms group: Scalability -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Stage group dashboard @@ -54,8 +54,8 @@ description, note the following: precision is 2. In some extremely low panels, you can see `0.00`, even though there is still some real traffic. -To inspect the raw data of the panel for further calculation, select **Inspect** from the dropdown -list of a panel. Queries, raw data, and panel JSON structure are available. +To inspect the raw data of the panel for further calculation, select **Inspect** from the dropdown list of a panel. +Queries, raw data, and panel JSON structure are available. Read more at [Grafana panel inspection](http://grafana.com/docs/grafana/next/panels/query-a-data-source/). All the dashboards are powered by [Grafana](https://grafana.com/), a frontend for displaying metrics. diff --git a/doc/development/stage_group_observability/index.md b/doc/development/stage_group_observability/index.md index 868e55735e8..08f751c508e 100644 --- a/doc/development/stage_group_observability/index.md +++ b/doc/development/stage_group_observability/index.md @@ -1,7 +1,7 @@ --- stage: Platforms group: Scalability -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Observability for stage groups diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 221d6b89b20..f1a23f06699 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -2,7 +2,7 @@ type: reference, dev stage: none group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" description: "GitLab development guidelines - testing best practices." --- diff --git a/doc/development/testing_guide/contract/consumer_tests.md b/doc/development/testing_guide/contract/consumer_tests.md index 46f4f446ad9..67bb7160749 100644 --- a/doc/development/testing_guide/contract/consumer_tests.md +++ b/doc/development/testing_guide/contract/consumer_tests.md @@ -1,7 +1,7 @@ --- stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Writing consumer tests diff --git a/doc/development/testing_guide/contract/index.md b/doc/development/testing_guide/contract/index.md index 30a4adaca44..8412a260c7d 100644 --- a/doc/development/testing_guide/contract/index.md +++ b/doc/development/testing_guide/contract/index.md @@ -1,7 +1,7 @@ --- stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Contract testing @@ -66,7 +66,7 @@ As mentioned in the [folder structure section](#consumer-tests), consumer tests #### Provider naming -These are the API endpoints that provides the data to the consumer so they are simply named according to the API endpoint they pertain to. Be mindful that this name is as descriptive as possible. For example, if we're writing a test for the `GET /groups/:id/projects` endpoint, we don't want to simply name it "Projects endpoint" as there is a `GET /projects` endpoint as well that also fetches a list of projects the user has access to across all of GitLab. An easy way to name them is by checking out our [API documentation](../../../api/api_resources.md) and naming it the same way it is named in there. So the [`GET /groups/:id/projects`](../../../api/groups.md#list-a-groups-projects) would be called `List a group’s projects` and [`GET /projects`](../../../api/projects.md#list-all-projects) would be called `List all projects`. Subsequently, the test files are named `list_a_groups_projects_helper.rb` and `list_all_projects_helper.rb` respectively. +These are the API endpoints that provides the data to the consumer so they are named according to the API endpoint they pertain to. Be mindful that this name is as descriptive as possible. For example, if we're writing a test for the `GET /groups/:id/projects` endpoint, we don't want to name it "Projects endpoint" as there is a `GET /projects` endpoint as well that also fetches a list of projects the user has access to across all of GitLab. An easy way to name them is by checking out our [API documentation](../../../api/api_resources.md) and naming it the same way it is named in there. So the [`GET /groups/:id/projects`](../../../api/groups.md#list-a-groups-projects) would be called `List a group’s projects` and [`GET /projects`](../../../api/projects.md#list-all-projects) would be called `List all projects`. Subsequently, the test files are named `list_a_groups_projects_helper.rb` and `list_all_projects_helper.rb` respectively. There are some cases where the provider being tested may not be documented so, in those cases, fall back to choosing a name that is as descriptive as possible to ensure it's easy to tell what the provider is for. diff --git a/doc/development/testing_guide/contract/provider_tests.md b/doc/development/testing_guide/contract/provider_tests.md index 92ac4c4ed71..1772ed9384e 100644 --- a/doc/development/testing_guide/contract/provider_tests.md +++ b/doc/development/testing_guide/contract/provider_tests.md @@ -1,7 +1,7 @@ --- stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Writing provider tests diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index a13011d0101..b81379d89b2 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Beginner's guide to writing end-to-end tests @@ -120,6 +120,22 @@ module QA end ``` +### The `product_group` metadata + +Assign `product_group` metadata and specify what product group this test belongs to. In this case, `authentication_and_authorization`. + +```ruby +# frozen_string_literal: true + +module QA + RSpec.describe 'Manage' do + describe 'Login', product_group: :authentication_and_authorization do + + end + end +end +``` + ### The `it` blocks (examples) Every test suite contains at least one `it` block (example). A good way to start @@ -128,7 +144,7 @@ writing end-to-end tests is to write test case descriptions as `it` blocks: ```ruby module QA RSpec.describe 'Manage' do - describe 'Login' do + describe 'Login', product_group: :authentication_and_authorization do it 'can login' do end @@ -152,7 +168,7 @@ Begin by logging in. module QA RSpec.describe 'Manage' do - describe 'Login' do + describe 'Login', product_group: :authentication_and_authorization do it 'can login' do Flow::Login.sign_in @@ -175,7 +191,7 @@ should answer the question "What do we test?" module QA RSpec.describe 'Manage' do - describe 'Login' do + describe 'Login', product_group: :authentication_and_authorization do it 'can login' do Flow::Login.sign_in @@ -222,7 +238,7 @@ a call to `sign_in`. module QA RSpec.describe 'Manage' do - describe 'Login' do + describe 'Login', product_group: :authentication_and_authorization do before do Flow::Login.sign_in end @@ -246,12 +262,12 @@ end ``` The `before` block is essentially a `before(:each)` and is run before each example, -ensuring we now log in at the beginning of each test. +ensuring we now sign in at the beginning of each test. ## Test setup using resources and page objects Next, let's test something other than Login. Let's test Issues, which are owned by the Plan -stage, so [create a file](#identify-the-devops-stage) in +stage and the Project Management Group, so [create a file](#identify-the-devops-stage) in `qa/specs/features/browser_ui/3_create/issues` called `issues_spec.rb`. ```ruby @@ -259,7 +275,7 @@ stage, so [create a file](#identify-the-devops-stage) in module QA RSpec.describe 'Plan' do - describe 'Issues' do + describe 'Issues', product_group: :project_management do let(:issue) do Resource::Issue.fabricate_via_api! do |issue| issue.title = 'My issue' diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index b17ca9e6f8f..37eda7f76f5 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -1,7 +1,7 @@ --- stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # End-to-end testing Best Practices @@ -10,7 +10,7 @@ This is a tailored extension of the Best Practices [found in the testing guide]( ## Class and module naming -The QA framework uses [Zeitwerk](https://github.com/fxn/zeitwerk) for class and module autoloading. The default Zeitwerk [inflector](https://github.com/fxn/zeitwerk#zeitwerkinflector) simply converts snake_cased file names to PascalCased module or class names. It is advised to stick to this pattern to avoid manual maintenance of inflections. +The QA framework uses [Zeitwerk](https://github.com/fxn/zeitwerk) for class and module autoloading. The default Zeitwerk [inflector](https://github.com/fxn/zeitwerk#zeitwerkinflector) converts snake_cased file names to PascalCased module or class names. It is advised to stick to this pattern to avoid manual maintenance of inflections. In case custom inflection logic is needed, custom inflectors are added in the [qa.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa.rb) file in the `loader.inflector.inflect` method invocation. @@ -299,7 +299,7 @@ point of failure and so the screenshot would not be captured at the right moment ## Ensure tests do not leave the browser logged in -All tests expect to be able to log in at the start of the test. +All tests expect to be able to sign in at the start of the test. For an example see [issue #34736](https://gitlab.com/gitlab-org/gitlab/-/issues/34736). @@ -358,7 +358,7 @@ using the Git CLI. ## Preferred method to blur elements -To blur an element, the preferred method is to click another element that does not alter the test state. +To blur an element, the preferred method is to select another element that does not alter the test state. If there's a mask that blocks the page elements, such as may occur with some dropdowns, use WebDriver's native mouse events to simulate a click event on the coordinates of an element. Use the following method: `click_element_coordinates`. diff --git a/doc/development/testing_guide/end_to_end/capybara_to_chemlab_migration_guide.md b/doc/development/testing_guide/end_to_end/capybara_to_chemlab_migration_guide.md index a71e076b57f..f3911176263 100644 --- a/doc/development/testing_guide/end_to_end/capybara_to_chemlab_migration_guide.md +++ b/doc/development/testing_guide/end_to_end/capybara_to_chemlab_migration_guide.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Migration Guide Capybara → Chemlab diff --git a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md index 8770a5d33cd..5cc82916cdd 100644 --- a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md +++ b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Dynamic Element Validation @@ -48,7 +48,7 @@ click_element(:my_element, Some::Page) First it is important to define what a "required element" is. -Simply put, a required element is a visible HTML element that appears on a UI component without any user input. +A required element is a visible HTML element that appears on a UI component without any user input. "Visible" can be defined as diff --git a/doc/development/testing_guide/end_to_end/execution_context_selection.md b/doc/development/testing_guide/end_to_end/execution_context_selection.md index 0a4c5fcf451..23b8ab7287b 100644 --- a/doc/development/testing_guide/end_to_end/execution_context_selection.md +++ b/doc/development/testing_guide/end_to_end/execution_context_selection.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Execution context selection diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 9a39161f1ad..8e25c817938 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Testing with feature flags diff --git a/doc/development/testing_guide/end_to_end/flows.md b/doc/development/testing_guide/end_to_end/flows.md index b99a1f9deb7..58702ceb861 100644 --- a/doc/development/testing_guide/end_to_end/flows.md +++ b/doc/development/testing_guide/end_to_end/flows.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Flows in GitLab QA diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index cc9c02a65ce..1853ad47779 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # End-to-end Testing @@ -125,7 +125,7 @@ For example, when we [dequarantine](https://about.gitlab.com/handbook/engineerin a flaky test we first want to make sure that it's no longer flaky. We can do that using the `ce:custom-parallel` and `ee:custom-parallel` jobs. Both are manual jobs that you can configure using custom variables. -When clicking the name (not the play icon) of one of the parallel jobs, +When selecting the name (not the play icon) of one of the parallel jobs, you are prompted to enter variables. You can use any of [the variables that can be used with `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables) as well as these: diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index c93e8c9d13f..5fbf2ffbcde 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Page objects in GitLab QA @@ -10,7 +10,7 @@ In GitLab QA we are using a known pattern, called _Page Objects_. This means that we have built an abstraction for all pages in GitLab that we use to drive GitLab QA scenarios. Whenever we do something on a page, like filling -in a form or clicking a button, we do that only through a page object +in a form or selecting a button, we do that only through a page object associated with this area of GitLab. For example, when GitLab QA test harness signs in into GitLab, it needs to fill diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index a519d1ecb47..6e29e9b9dff 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Resource classes in GitLab QA diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 1abaf3ef323..b12c6bd443a 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # RSpec metadata for end-to-end tests diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 322f108783f..81e1c7d5dc0 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Running tests that require special setup diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index a9571df1188..f380b24d7a5 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Style guide for writing end-to-end tests diff --git a/doc/development/testing_guide/end_to_end/troubleshooting.md b/doc/development/testing_guide/end_to_end/troubleshooting.md index 76d19dc0159..e0925cb71f4 100644 --- a/doc/development/testing_guide/end_to_end/troubleshooting.md +++ b/doc/development/testing_guide/end_to_end/troubleshooting.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Troubleshooting end-to-end tests diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 7409f15969d..45f1d0ddf7e 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Flaky tests @@ -45,15 +45,27 @@ Once a test is in quarantine, there are 3 choices: On our CI, we use [RSpec::Retry](https://github.com/NoRedInk/rspec-retry) to automatically retry a failing example a few times (see [`spec/spec_helper.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/spec_helper.rb) for the precise retries count). -We also use a home-made `RspecFlaky::Listener` listener which records flaky -examples in a JSON report file on `main` (`retrieve-tests-metadata` and -`update-tests-metadata` jobs). +We also use a custom [`RspecFlaky::Listener`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/tooling/rspec_flaky/listener.rb). +This listener runs in the `update-tests-metadata` job in `maintenance` scheduled pipelines +on the `master` branch, and saves flaky examples to `rspec/flaky/report-suite.json`. +The report file is then retrieved by the `retrieve-tests-metadata` job in all pipelines. This was originally implemented in: <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021>. If you want to enable retries locally, you can use the `RETRIES` environment variable. For instance `RETRIES=1 bin/rspec ...` would retry the failing examples once. +To generate the reports locally, use the `FLAKY_RSPEC_GENERATE_REPORT` environment variable. +For example, `FLAKY_RSPEC_GENERATE_REPORT=1 bin/rspec ...`. + +### Usage of the `rspec/flaky/report-suite.json` report + +The `rspec/flaky/report-suite.json` report is: + +- Used for [automatically skipping known flaky tests](../pipelines.md#automatic-skipping-of-flaky-tests). +- [Imported into Snowflake](https://gitlab.com/gitlab-data/analytics/-/blob/master/extract/gitlab_flaky_tests/upload.py) + once per day, for monitoring with the [internal dashboard](https://app.periscopedata.com/app/gitlab/888968/EP---Flaky-tests). + ## Problems we had in the past at GitLab - [`rspec-retry` is biting us when some API specs fail](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29242): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9825> diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 22a8792bac6..d7627e2064f 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Frontend testing standards and style guidelines @@ -200,7 +200,6 @@ import { shallowMountExtended } from 'helpers/vue_test_utils_helper' const wrapper = shallowMountExtended(ExampleComponent); -// In this example, `wrapper` is a `@vue/test-utils` wrapper returned from `mount` or `shallowMount`. it('exists', () => { // Best (especially for integration tests) wrapper.findByRole('link', { name: /Click Me/i }) @@ -291,7 +290,7 @@ it('tests a promise rejection', async () => { }); ``` -You can also simply return a promise from the test function. +You can also return a promise from the test function. Using the `done` and `done.fail` callbacks is discouraged when working with promises. They should not be used. diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index e50902c4995..753089fa673 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Testing standards and style guidelines diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index b272d23522e..d86d1c3c7b4 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Review Apps @@ -84,7 +84,7 @@ the GitLab handbook information for the [shared 1Password account](https://about ### Enable a feature flag for my Review App -1. Open your Review App and log in as documented above. +1. Open your Review App and sign in as documented above. 1. Create a personal access token. 1. Enable the feature flag using the [Feature flag API](../../api/features.md). @@ -101,7 +101,7 @@ the GitLab handbook information for the [shared 1Password account](https://about 1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`. 1. Find and open the `toolbox` Deployment. For example, `review-qa-raise-e-12chm0-toolbox`. 1. Select the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz`. -1. Select the `KUBECTL` dropdown, then `Exec` -> `toolbox`. +1. Select the `KUBECTL` dropdown list, then `Exec` -> `toolbox`. 1. Replace `-c toolbox -- ls` with `-it -- gitlab-rails console` from the default command or - Run `kubectl exec --namespace review-qa-raise-e-12chm0 review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz -it -- gitlab-rails console` and diff --git a/doc/development/testing_guide/smoke.md b/doc/development/testing_guide/smoke.md index 4fd2729a4d3..62a09ca864b 100644 --- a/doc/development/testing_guide/smoke.md +++ b/doc/development/testing_guide/smoke.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Smoke Tests diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index c1bf3609b53..c93a5fd539b 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Testing levels diff --git a/doc/development/testing_guide/testing_migrations_guide.md b/doc/development/testing_guide/testing_migrations_guide.md index 3006a2230ac..b276a7e2a3a 100644 --- a/doc/development/testing_guide/testing_migrations_guide.md +++ b/doc/development/testing_guide/testing_migrations_guide.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- @@ -22,7 +22,8 @@ a database schema. Adding a `:migration` tag to a test signature enables some custom RSpec `before` and `after` hooks in our [`spec/support/migration.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/f81fa6ab1dd788b70ef44b85aaba1f31ffafae7d/spec/support/migration.rb) -to run. +to run. If performing a migration against a database schema other than +`:gitlab_main` (for example `:gitlab_ci`), then you must specify it as well: `migration: :gitlab_ci`. See [spec/migrations/change_public_projects_cost_factor_spec.rb](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/migrations/change_public_projects_cost_factor_spec.rb#L6-6) for an example. A `before` hook reverts all migrations to the point that a migration under test is not yet migrated. diff --git a/doc/development/testing_guide/testing_rake_tasks.md b/doc/development/testing_guide/testing_rake_tasks.md index 30d193de2f2..3dfe1f9b725 100644 --- a/doc/development/testing_guide/testing_rake_tasks.md +++ b/doc/development/testing_guide/testing_rake_tasks.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Testing Rake tasks diff --git a/doc/development/transient/prevention-patterns.md b/doc/development/transient/prevention-patterns.md index 93712b104e6..98634df22e9 100644 --- a/doc/development/transient/prevention-patterns.md +++ b/doc/development/transient/prevention-patterns.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Preventing Transient Bugs diff --git a/doc/development/uploads/background.md b/doc/development/uploads/background.md deleted file mode 100644 index 1ad1aec23f2..00000000000 --- a/doc/development/uploads/background.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2022-07-25' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after <2022-07-25>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/uploads/implementation.md b/doc/development/uploads/implementation.md deleted file mode 100644 index 1ad1aec23f2..00000000000 --- a/doc/development/uploads/implementation.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2022-07-25' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after <2022-07-25>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/uploads/index.md b/doc/development/uploads/index.md index b8326489d40..41ec71451fb 100644 --- a/doc/development/uploads/index.md +++ b/doc/development/uploads/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Uploads development guide diff --git a/doc/development/uploads/working_with_uploads.md b/doc/development/uploads/working_with_uploads.md index 5a5f987c37c..c88762e6bd5 100644 --- a/doc/development/uploads/working_with_uploads.md +++ b/doc/development/uploads/working_with_uploads.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Uploads guide: Adding new uploads diff --git a/doc/development/utilities.md b/doc/development/utilities.md index 3f6187a4c2e..45a6b74f33a 100644 --- a/doc/development/utilities.md +++ b/doc/development/utilities.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab utilities diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md index bbea89d5645..0d321133705 100644 --- a/doc/development/value_stream_analytics.md +++ b/doc/development/value_stream_analytics.md @@ -1,7 +1,7 @@ --- -stage: Manage +stage: Plan group: Optimize -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Value stream analytics development guide @@ -326,3 +326,25 @@ in your rails console (`rails c`): ```ruby Analytics::CycleAnalytics::ReaggregationWorker.new.perform ``` + +### Seed data + +#### Value stream analytics + +Seed issues and merge requests for value stream analytics: + + ```shell + // Seed 10 issues for the project specified by <project-id> + $ VSA_SEED_PROJECT_ID=<project-id> VSA_ISSUE_COUNT=10 SEED_VSA=true FILTER=cycle_analytics rake db:seed_fu + ``` + +#### DORA metrics + +Seed DORA daily metrics for value stream, insights and CI/CD analytics: + +1. [Create an environment from the UI](../ci/environments/index.md#create-a-static-environment) named `production`. +1. Open the rails console: + + ```shell + rails c + ``` diff --git a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md index 6087d4bd8f7..a07998550bf 100644 --- a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md +++ b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md @@ -1,7 +1,7 @@ --- -stage: Manage +stage: Plan group: Optimize -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Aggregated Value Stream Analytics @@ -43,7 +43,7 @@ Benefits of the aggregated VSA backend: - Simpler database queries (fewer JOINs). - Faster aggregations, only a single table is accessed. - Possibility to introduce further aggregations for improving the first page load time. -- Better performance for large groups (with many sub-groups, projects, issues and, merge requests). +- Better performance for large groups (with many subgroups, projects, issues and, merge requests). - Ready for database decomposition. The VSA related database tables could live in a separate database with a minimal development effort. - Ready for keyset pagination which can be useful for exporting the data. @@ -165,7 +165,7 @@ Creation time always happens first, so this stage always reports negative durati The data collection scans and processes all issues and merge requests records in the group hierarchy, starting from the top-level group. This means that if a group only has one value stream -in a sub-group, we nevertheless collect data of all issues and merge requests in the hierarchy of +in a subgroup, we nevertheless collect data of all issues and merge requests in the hierarchy of this group. This aims to simplify the data collection mechanism. Moreover, data research shows that most group hierarchies have their stages configured on the top level. @@ -278,7 +278,7 @@ attributes. - `summary`, `time_summary` - Top-level aggregations, most of the metrics are using different APIs/ finders and not invoking the aggregated backend. -When clicking on a specific stage, the `records` endpoint is invoked, which returns the related +When selecting a specific stage, the `records` endpoint is invoked, which returns the related records (paginated) for the chosen stage in a specific order. ### Database decomposition diff --git a/doc/development/wikis.md b/doc/development/wikis.md index deea3d38470..67dc567cc5f 100644 --- a/doc/development/wikis.md +++ b/doc/development/wikis.md @@ -1,7 +1,7 @@ --- stage: Create group: Editor -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: "GitLab's development guidelines for Wikis" --- diff --git a/doc/development/windows.md b/doc/development/windows.md index 17dfaddef36..5f32848da79 100644 --- a/doc/development/windows.md +++ b/doc/development/windows.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference, howto --- @@ -76,7 +76,7 @@ Build a Google Cloud image with the above shared runners repository by doing the 1. Copy and save the password as it is not shown again. 1. Select **RDP** down arrow. 1. Select **Download the RDP file**. -1. Open the downloaded RDP file with the Windows remote desktop app (<https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/remote-desktop-clients>). +1. Open the downloaded RDP file with the Windows remote desktop app (<https://learn.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/remote-desktop-clients>). 1. Select **Continue** to accept the certificate. 1. Enter the password and select **Next**. diff --git a/doc/development/work_items.md b/doc/development/work_items.md index f15c66ae847..eabad175bf7 100644 --- a/doc/development/work_items.md +++ b/doc/development/work_items.md @@ -1,7 +1,7 @@ --- stage: Plan group: Project Management -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Work items and work item types @@ -88,8 +88,8 @@ so that in future we can allow users to define custom WITs, we will move the to `work_item_types` will involve creating the set of WITs for all root-level groups. NOTE: -At first, defining a WIT will only be possible at the root-level group, which would then be inherited by sub-groups. -We will investigate the possibility of defining new WITs at sub-group levels at a later iteration. +At first, defining a WIT will only be possible at the root-level group, which would then be inherited by subgroups. +We will investigate the possibility of defining new WITs at subgroup levels at a later iteration. ### Introducing work_item_types table @@ -150,8 +150,27 @@ of widgets. In order to customize each WIT with corresponding active widgets we will need a data structure to map each WIT to specific widgets. +The intent is for work item types to be highly configurable, both by GitLab for +implementing various work item schemes for customers (an opinionated GitLab +workflow, or SAFe 5, etc), and eventually for customers to customize their own +workflows. + +In this case, a work item scheme would be defined as a set of types with +certain characteristics (some widgets enabled, others not), such as an Epic, +Story, Bug, and Task, etc. + +As we're building a new work item architecture, we want to build the ability to +define these various types in a very flexible manner. Having GitLab use +this system first (without introducing customer customization) allows us to +better build out the initial system. + NOTE: -The exact structure of the WITs widgets metadata is still to be defined. +Currently work item's `base_type` is used to define static mapping of what +widgets are available for each type (current status), this definition should be +rather stored in database table. The exact structure of the WIT widgets +metadata is still to be defined. `base_type` was added to help converting other +types of resources (requirements and incidents) into work items. Eventually (when +these resources become regular work items), `base_type` will be removed. ### Custom work item types diff --git a/doc/development/work_items_widgets.md b/doc/development/work_items_widgets.md index 92919c10a9f..89602d969e6 100644 --- a/doc/development/work_items_widgets.md +++ b/doc/development/work_items_widgets.md @@ -1,7 +1,7 @@ --- stage: Plan group: Project Management -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Work items widgets @@ -88,7 +88,7 @@ To maximize component reusability, widgets should be field wrappers owning the work item query and mutation of the attribute it's responsible for. A field component is a generic and simple component. It has no knowledge of the -attribute or work item details, such as input field, date selector, or dropdown. +attribute or work item details, such as input field, date selector, or dropdown list. Widgets must be configurable to support various use cases, depending on work items. When building widgets, use slots to provide extra context while minimizing @@ -96,18 +96,18 @@ the use of props and injected attributes. ### Examples -We have a [dropdown field component](https://gitlab.com/gitlab-org/gitlab/-/blob/eea9ad536fa2d28ee6c09ed7d9207f803142eed7/app/assets/javascripts/vue_shared/components/dropdown/dropdown_widget/dropdown_widget.vue) +We have a [dropdown list component](https://gitlab.com/gitlab-org/gitlab/-/blob/eea9ad536fa2d28ee6c09ed7d9207f803142eed7/app/assets/javascripts/vue_shared/components/dropdown/dropdown_widget/dropdown_widget.vue) for use as reference. -Any work item widget can wrap the dropdown component. The widget has knowledge of +Any work item widget can wrap the dropdown list. The widget has knowledge of the attribute it mutates, and owns the mutation for it. Multiple widgets can use the same field component. For example: - Title and description widgets use the input field component. - Start and end date use the date selector component. -- Labels, milestones, and assignees selectors use the dropdown component. +- Labels, milestones, and assignees selectors use the dropdown list. -Some frontend widgets already use the dropdown component. Use them as a reference +Some frontend widgets already use the dropdown list. Use them as a reference for work items widgets development: - `ee/app/assets/javascripts/boards/components/assignee_select.vue` diff --git a/doc/development/workhorse/channel.md b/doc/development/workhorse/channel.md index 33d7cc63f00..f5693b57f7a 100644 --- a/doc/development/workhorse/channel.md +++ b/doc/development/workhorse/channel.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Websocket channel support for Workhorse diff --git a/doc/development/workhorse/configuration.md b/doc/development/workhorse/configuration.md index a94ba2b4fc6..82e44a6f995 100644 --- a/doc/development/workhorse/configuration.md +++ b/doc/development/workhorse/configuration.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Workhorse configuration diff --git a/doc/development/workhorse/gitlab_features.md b/doc/development/workhorse/gitlab_features.md index 3b240d4cbc6..8b899242935 100644 --- a/doc/development/workhorse/gitlab_features.md +++ b/doc/development/workhorse/gitlab_features.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Features that rely on Workhorse diff --git a/doc/development/workhorse/index.md b/doc/development/workhorse/index.md index f210f511954..0f4e55a002a 100644 --- a/doc/development/workhorse/index.md +++ b/doc/development/workhorse/index.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab Workhorse diff --git a/doc/development/workhorse/new_features.md b/doc/development/workhorse/new_features.md index 5c00903497a..dbde06ddee4 100644 --- a/doc/development/workhorse/new_features.md +++ b/doc/development/workhorse/new_features.md @@ -1,7 +1,7 @@ --- stage: Create group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Adding new features to Workhorse diff --git a/doc/development/workspace/index.md b/doc/development/workspace/index.md index b01a7826b3d..f4738e3fc31 100644 --- a/doc/development/workspace/index.md +++ b/doc/development/workspace/index.md @@ -3,7 +3,7 @@ comments: false type: index, dev stage: none group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" description: "Development Guidelines: learn about workspace when developing GitLab." --- |
