Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2022-06-23 03:09:13 +0300
committerGitLab Bot <gitlab-bot@gitlab.com>2022-06-23 03:09:13 +0300
commit0a8d7059a2db069c1193009c56ff3af1aa0791b5 (patch)
treee089d18fcde8e8da0a606f58d98e7448e320211e /doc/development
parentf5ef42977e86b3e87eac6dd6687ab865238fa305 (diff)
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/development')
-rw-r--r--doc/development/code_review.md2
-rw-r--r--doc/development/dangerbot.md2
-rw-r--r--doc/development/development_processes.md124
-rw-r--r--doc/development/feature_development.md198
-rw-r--r--doc/development/index.md304
-rw-r--r--doc/development/iterating_tables_in_batches.md35
6 files changed, 352 insertions, 313 deletions
diff --git a/doc/development/code_review.md b/doc/development/code_review.md
index d864f369945..8fcbd09c8da 100644
--- a/doc/development/code_review.md
+++ b/doc/development/code_review.md
@@ -129,7 +129,7 @@ with [domain expertise](#domain-experts).
1. If your merge request includes documentation changes, it must be **approved
by a [Technical writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments)**,
based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages).
-1. If your merge request includes changes to development guidelines, follow the [review process](index.md#development-guidelines-review) and get the approvals accordingly.
+1. If your merge request includes changes to development guidelines, follow the [review process](development_processes.md#development-guidelines-review) and get the approvals accordingly.
1. If your merge request includes end-to-end **and** non-end-to-end changes (*4*), it must be **approved
by a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors)**.
1. If your merge request only includes end-to-end changes (*4*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)**
diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md
index 34b61858995..d2b231ebc7c 100644
--- a/doc/development/dangerbot.md
+++ b/doc/development/dangerbot.md
@@ -58,7 +58,7 @@ itself, increasing visibility.
## Development guidelines
-Danger code is Ruby code, so all our [usual backend guidelines](index.md#backend-guides)
+Danger code is Ruby code, so all our [usual backend guidelines](feature_development.md#backend-guides)
continue to apply. However, there are a few things that deserve special emphasis.
### When to use Danger
diff --git a/doc/development/development_processes.md b/doc/development/development_processes.md
new file mode 100644
index 00000000000..e199aedd3f5
--- /dev/null
+++ b/doc/development/development_processes.md
@@ -0,0 +1,124 @@
+---
+stage: none
+group: Development
+info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines"
+---
+
+# Development processes
+
+Consult these topics for information on development processes for contributing to GitLab.
+
+## Processes
+
+Must-reads:
+
+- [Guide on adapting existing and introducing new components](architecture.md#adapting-existing-and-introducing-new-components)
+- [Code review guidelines](code_review.md) for reviewing code and having code
+ reviewed
+- [Database review guidelines](database_review.md) for reviewing
+ database-related changes and complex SQL queries, and having them reviewed
+- [Secure coding guidelines](secure_coding_guidelines.md)
+- [Pipelines for the GitLab project](pipelines.md)
+
+Complementary reads:
+
+- [GitLab core team & GitLab Inc. contribution process](https://gitlab.com/gitlab-org/gitlab/-/blob/master/PROCESS.md)
+- [Security process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#security-releases-critical-non-critical-as-a-developer)
+- [Patch release process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/patch/process.md#process-for-developers)
+- [Guidelines for implementing Enterprise Edition features](ee_features.md)
+- [Adding a new service component to GitLab](adding_service_component.md)
+- [Guidelines for changelogs](changelog.md)
+- [Dependencies](dependencies.md)
+- [Danger bot](dangerbot.md)
+- [Requesting access to ChatOps on GitLab.com](chatops_on_gitlabcom.md#requesting-access) (for GitLab team members)
+
+### Development guidelines review
+
+When you submit a change to the GitLab development guidelines, who
+you ask for reviews depends on the level of change.
+
+#### Wording, style, or link changes
+
+Not all changes require extensive review. For example, MRs that don't change the
+content's meaning or function can be reviewed, approved, and merged by any
+maintainer or Technical Writer. These can include:
+
+- Typo fixes.
+- Clarifying links, such as to external programming language documentation.
+- Changes to comply with the [Documentation Style Guide](documentation/index.md)
+ that don't change the intent of the documentation page.
+
+#### Specific changes
+
+If the MR proposes changes that are limited to a particular stage, group, or team,
+request a review and approval from an experienced GitLab Team Member in that
+group. For example, if you're documenting a new internal API used exclusively by
+a given group, request an engineering review from one of the group's members.
+
+After the engineering review is complete, assign the MR to the
+[Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments)
+in the modified documentation page's metadata.
+If the page is not assigned to a specific group, follow the
+[Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines).
+
+#### Broader changes
+
+Some changes affect more than one group. For example:
+
+- Changes to [code review guidelines](code_review.md).
+- Changes to [commit message guidelines](contributing/merge_request_workflow.md#commit-messages-guidelines).
+- Changes to guidelines in [feature flags in development of GitLab](feature_flags/).
+- Changes to [feature flags documentation guidelines](documentation/feature_flags.md).
+
+In these cases, use the following workflow:
+
+1. Request a peer review from a member of your team.
+1. Request a review and approval of an Engineering Manager (EM)
+ or Staff Engineer who's responsible for the area in question:
+
+ - [Frontend](https://about.gitlab.com/handbook/engineering/frontend/)
+ - [Backend](https://about.gitlab.com/handbook/engineering/)
+ - [Database](https://about.gitlab.com/handbook/engineering/development/database/)
+ - [User Experience (UX)](https://about.gitlab.com/handbook/engineering/ux/)
+ - [Security](https://about.gitlab.com/handbook/engineering/security/)
+ - [Quality](https://about.gitlab.com/handbook/engineering/quality/)
+ - [Engineering Productivity](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/)
+ - [Infrastructure](https://about.gitlab.com/handbook/engineering/infrastructure/)
+ - [Technical Writing](https://about.gitlab.com/handbook/engineering/ux/technical-writing/)
+
+ You can skip this step for MRs authored by EMs or Staff Engineers responsible
+ for their area.
+
+ If there are several affected groups, you may need approvals at the
+ EM/Staff Engineer level from each affected area.
+
+1. After completing the reviews, consult with the EM/Staff Engineer
+ author / approver of the MR.
+
+ If this is a significant change across multiple areas, request final review
+ and approval from the VP of Development, the DRI for Development Guidelines,
+ @clefelhocz1.
+
+1. After all approvals are complete, assign the MR to the
+ [Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments)
+ in the modified documentation page's metadata.
+ If the page is not assigned to a specific group, follow the
+ [Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines).
+ The Technical Writer may ask for additional approvals as previously suggested before merging the MR.
+
+### Reviewer values
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57293) in GitLab 14.1.
+
+As a reviewer or as a reviewee, make sure to familiarize yourself with
+the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/reviewer-values/) we strive for at GitLab.
+
+## Language-specific guides
+
+### Go guides
+
+- [Go Guidelines](go_guide/index.md)
+
+### Shell Scripting guides
+
+- [Shell scripting standards and style guidelines](shell_scripting_guide/index.md)
diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md
new file mode 100644
index 00000000000..539746f3e4e
--- /dev/null
+++ b/doc/development/feature_development.md
@@ -0,0 +1,198 @@
+---
+stage: none
+group: Development
+info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines"
+---
+
+# Feature development
+
+Consult these topics for information on contributing to specific GitLab features.
+
+## UX and Frontend guides
+
+- [GitLab Design System](https://design.gitlab.com/), for building GitLab with
+ existing CSS styles and elements
+- [Frontend guidelines](fe_guide/index.md)
+- [Emoji guide](fe_guide/emojis.md)
+
+## Backend guides
+
+### General
+
+- [Directory structure](directory_structure.md)
+- [GitLab EventStore](event_store.md) to publish/subscribe to domain events
+- [GitLab utilities](utilities.md)
+- [Newlines style guide](newlines_styleguide.md)
+- [Logging](logging.md)
+- [Dealing with email/mailers](emails.md)
+- [Kubernetes integration guidelines](kubernetes.md)
+- [Permissions](permissions.md)
+- [Code comments](code_comments.md)
+- [Windows Development on GCP](windows.md)
+- [FIPS compliance](fips_compliance.md)
+- [`Gemfile` guidelines](gemfile.md)
+- [Ruby upgrade guidelines](ruby_upgrade.md)
+
+### Things to be aware of
+
+- [Gotchas](gotchas.md) to avoid
+- [Avoid modules with instance variables](module_with_instance_variables.md), if
+ possible
+- [Guidelines for reusing abstractions](reusing_abstractions.md)
+- [Ruby 3 gotchas](ruby3_gotchas.md)
+
+### Rails Framework related
+
+- [Routing](routing.md)
+- [Rails initializers](rails_initializers.md)
+- [Mass Inserting Models](mass_insert.md)
+- [Issuable-like Rails models](issuable-like-models.md)
+- [Issue types vs first-class types](issue_types.md)
+- [DeclarativePolicy framework](policies.md)
+- [Rails update guidelines](rails_update.md)
+
+### Debugging
+
+- [Pry debugging](pry_debugging.md)
+- [Sidekiq debugging](../administration/troubleshooting/sidekiq.md)
+
+### Git specifics
+
+- [How Git object deduplication works in GitLab](git_object_deduplication.md)
+- [Git LFS](lfs.md)
+
+### API
+
+- [API style guide](api_styleguide.md) for contributing to the API
+- [GraphQL API style guide](api_graphql_styleguide.md) for contributing to the
+ [GraphQL API](../api/graphql/index.md)
+
+### GitLab components and features
+
+- [Developing against interacting components or features](interacting_components.md)
+- [Manage feature flags](feature_flags/index.md)
+- [Licensed feature availability](licensed_feature_availability.md)
+- [Accessing session data](session.md)
+- [How to dump production data to staging](db_dump.md)
+- [Geo development](geo.md)
+- [Redis guidelines](redis.md)
+ - [Adding a new Redis instance](redis/new_redis_instance.md)
+- [Sidekiq guidelines](sidekiq/index.md) for working with Sidekiq workers
+- [Working with Gitaly](gitaly.md)
+- [Elasticsearch integration docs](elasticsearch.md)
+- [Working with merge request diffs](diffs.md)
+- [Approval Rules](approval_rules.md)
+- [Repository mirroring](repository_mirroring.md)
+- [Uploads development guide](uploads/index.md)
+- [Auto DevOps development guide](auto_devops.md)
+- [Renaming features](renaming_features.md)
+- [Code Intelligence](code_intelligence/index.md)
+- [Feature categorization](feature_categorization/index.md)
+- [Wikis development guide](wikis.md)
+- [Image scaling guide](image_scaling.md)
+- [Cascading Settings](cascading_settings.md)
+- [Shell commands](shell_commands.md) in the GitLab codebase
+- [Value Stream Analytics development guide](value_stream_analytics.md)
+- [Application limits](application_limits.md)
+
+### Import and Export
+
+- [Working with the GitHub importer](github_importer.md)
+- [Import/Export development documentation](import_export.md)
+- [Test Import Project](import_project.md)
+- [Group migration](bulk_import.md)
+- [Export to CSV](export_csv.md)
+
+## Performance guides
+
+- [Performance guidelines](performance.md) for writing code, benchmarks, and
+ certain patterns to avoid.
+- [Caching guidelines](caching.md) for using caching in Rails under a GitLab environment.
+- [Merge request performance guidelines](merge_request_performance_guidelines.md)
+ for ensuring merge requests do not negatively impact GitLab performance
+- [Profiling](profiling.md) a URL or tracking down N+1 queries using Bullet.
+- [Cached queries guidelines](cached_queries.md), for tracking down N+1 queries
+ masked by query caching, memory profiling and why should we avoid cached
+ queries.
+
+## Database guides
+
+See [database guidelines](database/index.md).
+
+## Integration guides
+
+- [Integrations development guide](integrations/index.md)
+- [Jira Connect app](integrations/jira_connect.md)
+- [Security Scanners](integrations/secure.md)
+- [Secure Partner Integration](integrations/secure_partner_integration.md)
+- [How to run Jenkins in development environment](integrations/jenkins.md)
+- [How to run local `Codesandbox` integration for Web IDE Live Preview](integrations/codesandbox.md)
+
+## Testing guides
+
+- [Testing standards and style guidelines](testing_guide/index.md)
+- [Frontend testing standards and style guidelines](testing_guide/frontend_testing.md)
+
+## Refactoring guides
+
+- [Refactoring guidelines](refactoring_guide/index.md)
+
+## Deprecation guides
+
+- [Deprecation guidelines](deprecation_guidelines/index.md)
+
+## Documentation guides
+
+- [Writing documentation](documentation/index.md)
+- [Documentation style guide](documentation/styleguide/index.md)
+- [Markdown](../user/markdown.md)
+
+## Internationalization (i18n) guides
+
+- [Introduction](i18n/index.md)
+- [Externalization](i18n/externalization.md)
+- [Translation](i18n/translation.md)
+
+## Product Intelligence guides
+
+- [Product Intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/)
+- [Service Ping guide](service_ping/index.md)
+- [Snowplow guide](snowplow/index.md)
+
+## Experiment guide
+
+- [Introduction](experiment_guide/index.md)
+
+## Build guides
+
+- [Building a package for testing purposes](build_test_package.md)
+
+## Compliance
+
+- [Licensing](licensing.md) for ensuring license compliance
+
+## Domain-specific guides
+
+- [CI/CD development documentation](cicd/index.md)
+- [AppSec development documentation](appsec/index.md)
+
+## Technical Reference by Group
+
+- [Create: Source Code BE](backend/create_source_code_be/index.md)
+
+## Other development guides
+
+- [Defining relations between files using projections](projections.md)
+- [Reference processing](reference_processing.md)
+- [Compatibility with multiple versions of the application running at the same time](multi_version_compatibility.md)
+- [Features inside `.gitlab/`](features_inside_dot_gitlab.md)
+- [Dashboards for stage groups](stage_group_dashboards.md)
+- [Preventing transient bugs](transient/prevention-patterns.md)
+- [GitLab Application SLIs](application_slis/index.md)
+- [Spam protection and CAPTCHA development guide](spam_protection_and_captcha/index.md)
+
+## Other GitLab Development Kit (GDK) guides
+
+- [Run full Auto DevOps cycle in a GDK instance](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/auto_devops.md)
+- [Using GitLab Runner with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/runner.md)
+- [Using the Web IDE terminal with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/web_ide_terminal_gdk_setup.md)
diff --git a/doc/development/index.md b/doc/development/index.md
index 3d5ec24d3e2..1b897db5097 100644
--- a/doc/development/index.md
+++ b/doc/development/index.md
@@ -46,307 +46,3 @@ GitLab instance, see the [Administrator documentation](../administration/index.m
- [Implement design & UI elements](contributing/design.md)
- [GitLab Architecture Overview](architecture.md)
- [Rake tasks](rake_tasks.md) for development
-
-## Processes
-
-**Must-reads:**
-
-- [Guide on adapting existing and introducing new components](architecture.md#adapting-existing-and-introducing-new-components)
-- [Code review guidelines](code_review.md) for reviewing code and having code
- reviewed
-- [Database review guidelines](database_review.md) for reviewing
- database-related changes and complex SQL queries, and having them reviewed
-- [Secure coding guidelines](secure_coding_guidelines.md)
-- [Pipelines for the GitLab project](pipelines.md)
-
-Complementary reads:
-
-- [GitLab core team & GitLab Inc. contribution process](https://gitlab.com/gitlab-org/gitlab/-/blob/master/PROCESS.md)
-- [Security process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#security-releases-critical-non-critical-as-a-developer)
-- [Patch release process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/patch/process.md#process-for-developers)
-- [Guidelines for implementing Enterprise Edition features](ee_features.md)
-- [Adding a new service component to GitLab](adding_service_component.md)
-- [Guidelines for changelogs](changelog.md)
-- [Dependencies](dependencies.md)
-- [Danger bot](dangerbot.md)
-- [Requesting access to ChatOps on GitLab.com](chatops_on_gitlabcom.md#requesting-access) (for GitLab team members)
-
-### Development guidelines review
-
-When you submit a change to the GitLab development guidelines, who
-you ask for reviews depends on the level of change.
-
-#### Wording, style, or link changes
-
-Not all changes require extensive review. For example, MRs that don't change the
-content's meaning or function can be reviewed, approved, and merged by any
-maintainer or Technical Writer. These can include:
-
-- Typo fixes.
-- Clarifying links, such as to external programming language documentation.
-- Changes to comply with the [Documentation Style Guide](documentation/index.md)
- that don't change the intent of the documentation page.
-
-#### Specific changes
-
-If the MR proposes changes that are limited to a particular stage, group, or team,
-request a review and approval from an experienced GitLab Team Member in that
-group. For example, if you're documenting a new internal API used exclusively by
-a given group, request an engineering review from one of the group's members.
-
-After the engineering review is complete, assign the MR to the
-[Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments)
-in the modified documentation page's metadata.
-If the page is not assigned to a specific group, follow the
-[Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines).
-
-#### Broader changes
-
-Some changes affect more than one group. For example:
-
-- Changes to [code review guidelines](code_review.md).
-- Changes to [commit message guidelines](contributing/merge_request_workflow.md#commit-messages-guidelines).
-- Changes to guidelines in [feature flags in development of GitLab](feature_flags/).
-- Changes to [feature flags documentation guidelines](documentation/feature_flags.md).
-
-In these cases, use the following workflow:
-
-1. Request a peer review from a member of your team.
-1. Request a review and approval of an Engineering Manager (EM)
- or Staff Engineer who's responsible for the area in question:
-
- - [Frontend](https://about.gitlab.com/handbook/engineering/frontend/)
- - [Backend](https://about.gitlab.com/handbook/engineering/)
- - [Database](https://about.gitlab.com/handbook/engineering/development/database/)
- - [User Experience (UX)](https://about.gitlab.com/handbook/engineering/ux/)
- - [Security](https://about.gitlab.com/handbook/engineering/security/)
- - [Quality](https://about.gitlab.com/handbook/engineering/quality/)
- - [Engineering Productivity](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/)
- - [Infrastructure](https://about.gitlab.com/handbook/engineering/infrastructure/)
- - [Technical Writing](https://about.gitlab.com/handbook/engineering/ux/technical-writing/)
-
- You can skip this step for MRs authored by EMs or Staff Engineers responsible
- for their area.
-
- If there are several affected groups, you may need approvals at the
- EM/Staff Engineer level from each affected area.
-
-1. After completing the reviews, consult with the EM/Staff Engineer
- author / approver of the MR.
-
- If this is a significant change across multiple areas, request final review
- and approval from the VP of Development, the DRI for Development Guidelines,
- @clefelhocz1.
-
-1. After all approvals are complete, assign the MR to the
- [Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments)
- in the modified documentation page's metadata.
- If the page is not assigned to a specific group, follow the
- [Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines).
- The Technical Writer may ask for additional approvals as previously suggested before merging the MR.
-
-### Reviewer values
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57293) in GitLab 14.1.
-
-As a reviewer or as a reviewee, make sure to familiarize yourself with
-the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/reviewer-values/) we strive for at GitLab.
-
-## UX and Frontend guides
-
-- [GitLab Design System](https://design.gitlab.com/), for building GitLab with
- existing CSS styles and elements
-- [Frontend guidelines](fe_guide/index.md)
-- [Emoji guide](fe_guide/emojis.md)
-
-## Backend guides
-
-### General
-
-- [Directory structure](directory_structure.md)
-- [GitLab EventStore](event_store.md) to publish/subscribe to domain events
-- [GitLab utilities](utilities.md)
-- [Newlines style guide](newlines_styleguide.md)
-- [Logging](logging.md)
-- [Dealing with email/mailers](emails.md)
-- [Kubernetes integration guidelines](kubernetes.md)
-- [Permissions](permissions.md)
-- [Code comments](code_comments.md)
-- [Windows Development on GCP](windows.md)
-- [FIPS compliance](fips_compliance.md)
-- [`Gemfile` guidelines](gemfile.md)
-- [Ruby upgrade guidelines](ruby_upgrade.md)
-
-### Things to be aware of
-
-- [Gotchas](gotchas.md) to avoid
-- [Avoid modules with instance variables](module_with_instance_variables.md), if
- possible
-- [Guidelines for reusing abstractions](reusing_abstractions.md)
-- [Ruby 3 gotchas](ruby3_gotchas.md)
-
-### Rails Framework related
-
-- [Routing](routing.md)
-- [Rails initializers](rails_initializers.md)
-- [Mass Inserting Models](mass_insert.md)
-- [Issuable-like Rails models](issuable-like-models.md)
-- [Issue types vs first-class types](issue_types.md)
-- [DeclarativePolicy framework](policies.md)
-- [Rails update guidelines](rails_update.md)
-
-### Debugging
-
-- [Pry debugging](pry_debugging.md)
-- [Sidekiq debugging](../administration/troubleshooting/sidekiq.md)
-
-### Git specifics
-
-- [How Git object deduplication works in GitLab](git_object_deduplication.md)
-- [Git LFS](lfs.md)
-
-### API
-
-- [API style guide](api_styleguide.md) for contributing to the API
-- [GraphQL API style guide](api_graphql_styleguide.md) for contributing to the
- [GraphQL API](../api/graphql/index.md)
-
-### GitLab components and features
-
-- [Developing against interacting components or features](interacting_components.md)
-- [Manage feature flags](feature_flags/index.md)
-- [Licensed feature availability](licensed_feature_availability.md)
-- [Accessing session data](session.md)
-- [How to dump production data to staging](db_dump.md)
-- [Geo development](geo.md)
-- [Redis guidelines](redis.md)
- - [Adding a new Redis instance](redis/new_redis_instance.md)
-- [Sidekiq guidelines](sidekiq/index.md) for working with Sidekiq workers
-- [Working with Gitaly](gitaly.md)
-- [Elasticsearch integration docs](elasticsearch.md)
-- [Working with merge request diffs](diffs.md)
-- [Approval Rules](approval_rules.md)
-- [Repository mirroring](repository_mirroring.md)
-- [Uploads development guide](uploads/index.md)
-- [Auto DevOps development guide](auto_devops.md)
-- [Renaming features](renaming_features.md)
-- [Code Intelligence](code_intelligence/index.md)
-- [Feature categorization](feature_categorization/index.md)
-- [Wikis development guide](wikis.md)
-- [Image scaling guide](image_scaling.md)
-- [Cascading Settings](cascading_settings.md)
-- [Shell commands](shell_commands.md) in the GitLab codebase
-- [Value Stream Analytics development guide](value_stream_analytics.md)
-- [Application limits](application_limits.md)
-
-### Import/Export
-
-- [Working with the GitHub importer](github_importer.md)
-- [Import/Export development documentation](import_export.md)
-- [Test Import Project](import_project.md)
-- [Group migration](bulk_import.md)
-- [Export to CSV](export_csv.md)
-
-## Language-specific guides
-
-### Go guides
-
-- [Go Guidelines](go_guide/index.md)
-
-### Shell Scripting guides
-
-- [Shell scripting standards and style guidelines](shell_scripting_guide/index.md)
-
-## Performance guides
-
-- [Performance guidelines](performance.md) for writing code, benchmarks, and
- certain patterns to avoid.
-- [Caching guidelines](caching.md) for using caching in Rails under a GitLab environment.
-- [Merge request performance guidelines](merge_request_performance_guidelines.md)
- for ensuring merge requests do not negatively impact GitLab performance
-- [Profiling](profiling.md) a URL or tracking down N+1 queries using Bullet.
-- [Cached queries guidelines](cached_queries.md), for tracking down N+1 queries
- masked by query caching, memory profiling and why should we avoid cached
- queries.
-
-## Database guides
-
-See [database guidelines](database/index.md).
-
-## Integration guides
-
-- [Integrations development guide](integrations/index.md)
-- [Jira Connect app](integrations/jira_connect.md)
-- [Security Scanners](integrations/secure.md)
-- [Secure Partner Integration](integrations/secure_partner_integration.md)
-- [How to run Jenkins in development environment](integrations/jenkins.md)
-- [How to run local `Codesandbox` integration for Web IDE Live Preview](integrations/codesandbox.md)
-
-## Testing guides
-
-- [Testing standards and style guidelines](testing_guide/index.md)
-- [Frontend testing standards and style guidelines](testing_guide/frontend_testing.md)
-
-## Refactoring guides
-
-- [Refactoring guidelines](refactoring_guide/index.md)
-
-## Deprecation guides
-
-- [Deprecation guidelines](deprecation_guidelines/index.md)
-
-## Documentation guides
-
-- [Writing documentation](documentation/index.md)
-- [Documentation style guide](documentation/styleguide/index.md)
-- [Markdown](../user/markdown.md)
-
-## Internationalization (i18n) guides
-
-- [Introduction](i18n/index.md)
-- [Externalization](i18n/externalization.md)
-- [Translation](i18n/translation.md)
-
-## Product Intelligence guides
-
-- [Product Intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/)
-- [Service Ping guide](service_ping/index.md)
-- [Snowplow guide](snowplow/index.md)
-
-## Experiment guide
-
-- [Introduction](experiment_guide/index.md)
-
-## Build guides
-
-- [Building a package for testing purposes](build_test_package.md)
-
-## Compliance
-
-- [Licensing](licensing.md) for ensuring license compliance
-
-## Domain-specific guides
-
-- [CI/CD development documentation](cicd/index.md)
-- [AppSec development documentation](appsec/index.md)
-
-## Technical Reference by Group
-
-- [Create: Source Code BE](backend/create_source_code_be/index.md)
-
-## Other Development guides
-
-- [Defining relations between files using projections](projections.md)
-- [Reference processing](reference_processing.md)
-- [Compatibility with multiple versions of the application running at the same time](multi_version_compatibility.md)
-- [Features inside `.gitlab/`](features_inside_dot_gitlab.md)
-- [Dashboards for stage groups](stage_group_dashboards.md)
-- [Preventing transient bugs](transient/prevention-patterns.md)
-- [GitLab Application SLIs](application_slis/index.md)
-- [Spam protection and CAPTCHA development guide](spam_protection_and_captcha/index.md)
-
-## Other GitLab Development Kit (GDK) guides
-
-- [Run full Auto DevOps cycle in a GDK instance](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/auto_devops.md)
-- [Using GitLab Runner with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/runner.md)
-- [Using the Web IDE terminal with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/web_ide_terminal_gdk_setup.md)
diff --git a/doc/development/iterating_tables_in_batches.md b/doc/development/iterating_tables_in_batches.md
index b4459b53efa..1159e3755e5 100644
--- a/doc/development/iterating_tables_in_batches.md
+++ b/doc/development/iterating_tables_in_batches.md
@@ -42,20 +42,20 @@ The API of this method is similar to `in_batches`, though it doesn't support
all of the arguments that `in_batches` supports. You should always use
`each_batch` _unless_ you have a specific need for `in_batches`.
-## Avoid iterating over non-unique columns
+## Iterating over non-unique columns
-One should proceed with extra caution, and possibly avoid iterating over a column that can contain
-duplicate values. When you iterate over an attribute that is not unique, even with the applied max
-batch size, there is no guarantee that the resulting batches do not surpass it. The following
-snippet demonstrates this situation when one attempt to select `Ci::Build` entries for users with
-`id` between `1` and `10,000`, the database returns `1 215 178` matching rows.
+One should proceed with extra caution. When you iterate over an attribute that is not unique,
+even with the applied max batch size, there is no guarantee that the resulting batches do not
+surpass it. The following snippet demonstrates this situation when one attempt to select
+`Ci::Build` entries for users with `id` between `1` and `10,000`, the database returns
+`1 215 178` matching rows.
```ruby
[ gstg ] production> Ci::Build.where(user_id: (1..10_000)).size
=> 1215178
```
-This happens because built relation is translated into the following query
+This happens because the built relation is translated into the following query:
```ruby
[ gstg ] production> puts Ci::Build.where(user_id: (1..10_000)).to_sql
@@ -69,6 +69,27 @@ threshold does not translate to the size of the returned dataset. That happens b
`n` possible values of attributes, one can't tell for sure that the number of records that contains
them is less than `n`.
+### Loose-index scan with `distinct_each_batch`
+
+When iterating over a non-unique column is necessary, use the `distinct_each_batch` helper
+method. The helper uses the [loose-index scan technique](https://wiki.postgresql.org/wiki/Loose_indexscan)
+(skip-index scan) to skip duplicated values within a database index.
+
+Example: iterating over distinct `author_id` in the Issue model
+
+```ruby
+Issue.distinct_each_batch(column: :author_id, of: 1000) do |relation|
+ users = User.where(id: relation.select(:author_id)).to_a
+end
+```
+
+The technique provides stable performance between the batches regardless of the data distribution.
+The `relation` object returns an ActiveRecord scope where only the given `column` is available.
+Other columns are not loaded.
+
+The underlying database queries use recursive CTEs, which adds extra overhead. We therefore advise to use
+smaller batch sizes than those used for a standard `each_batch` iteration.
+
## Column definition
`EachBatch` uses the primary key of the model by default for the iteration. This works most of the