diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-06-23 03:09:13 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-06-23 03:09:13 +0300 |
commit | 0a8d7059a2db069c1193009c56ff3af1aa0791b5 (patch) | |
tree | e089d18fcde8e8da0a606f58d98e7448e320211e /doc/development | |
parent | f5ef42977e86b3e87eac6dd6687ab865238fa305 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/development')
-rw-r--r-- | doc/development/code_review.md | 2 | ||||
-rw-r--r-- | doc/development/dangerbot.md | 2 | ||||
-rw-r--r-- | doc/development/development_processes.md | 124 | ||||
-rw-r--r-- | doc/development/feature_development.md | 198 | ||||
-rw-r--r-- | doc/development/index.md | 304 | ||||
-rw-r--r-- | doc/development/iterating_tables_in_batches.md | 35 |
6 files changed, 352 insertions, 313 deletions
diff --git a/doc/development/code_review.md b/doc/development/code_review.md index d864f369945..8fcbd09c8da 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -129,7 +129,7 @@ with [domain expertise](#domain-experts). 1. If your merge request includes documentation changes, it must be **approved by a [Technical writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments)**, based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages). -1. If your merge request includes changes to development guidelines, follow the [review process](index.md#development-guidelines-review) and get the approvals accordingly. +1. If your merge request includes changes to development guidelines, follow the [review process](development_processes.md#development-guidelines-review) and get the approvals accordingly. 1. If your merge request includes end-to-end **and** non-end-to-end changes (*4*), it must be **approved by a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors)**. 1. If your merge request only includes end-to-end changes (*4*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)** diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 34b61858995..d2b231ebc7c 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -58,7 +58,7 @@ itself, increasing visibility. ## Development guidelines -Danger code is Ruby code, so all our [usual backend guidelines](index.md#backend-guides) +Danger code is Ruby code, so all our [usual backend guidelines](feature_development.md#backend-guides) continue to apply. However, there are a few things that deserve special emphasis. ### When to use Danger diff --git a/doc/development/development_processes.md b/doc/development/development_processes.md new file mode 100644 index 00000000000..e199aedd3f5 --- /dev/null +++ b/doc/development/development_processes.md @@ -0,0 +1,124 @@ +--- +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + +# Development processes + +Consult these topics for information on development processes for contributing to GitLab. + +## Processes + +Must-reads: + +- [Guide on adapting existing and introducing new components](architecture.md#adapting-existing-and-introducing-new-components) +- [Code review guidelines](code_review.md) for reviewing code and having code + reviewed +- [Database review guidelines](database_review.md) for reviewing + database-related changes and complex SQL queries, and having them reviewed +- [Secure coding guidelines](secure_coding_guidelines.md) +- [Pipelines for the GitLab project](pipelines.md) + +Complementary reads: + +- [GitLab core team & GitLab Inc. contribution process](https://gitlab.com/gitlab-org/gitlab/-/blob/master/PROCESS.md) +- [Security process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#security-releases-critical-non-critical-as-a-developer) +- [Patch release process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/patch/process.md#process-for-developers) +- [Guidelines for implementing Enterprise Edition features](ee_features.md) +- [Adding a new service component to GitLab](adding_service_component.md) +- [Guidelines for changelogs](changelog.md) +- [Dependencies](dependencies.md) +- [Danger bot](dangerbot.md) +- [Requesting access to ChatOps on GitLab.com](chatops_on_gitlabcom.md#requesting-access) (for GitLab team members) + +### Development guidelines review + +When you submit a change to the GitLab development guidelines, who +you ask for reviews depends on the level of change. + +#### Wording, style, or link changes + +Not all changes require extensive review. For example, MRs that don't change the +content's meaning or function can be reviewed, approved, and merged by any +maintainer or Technical Writer. These can include: + +- Typo fixes. +- Clarifying links, such as to external programming language documentation. +- Changes to comply with the [Documentation Style Guide](documentation/index.md) + that don't change the intent of the documentation page. + +#### Specific changes + +If the MR proposes changes that are limited to a particular stage, group, or team, +request a review and approval from an experienced GitLab Team Member in that +group. For example, if you're documenting a new internal API used exclusively by +a given group, request an engineering review from one of the group's members. + +After the engineering review is complete, assign the MR to the +[Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) +in the modified documentation page's metadata. +If the page is not assigned to a specific group, follow the +[Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). + +#### Broader changes + +Some changes affect more than one group. For example: + +- Changes to [code review guidelines](code_review.md). +- Changes to [commit message guidelines](contributing/merge_request_workflow.md#commit-messages-guidelines). +- Changes to guidelines in [feature flags in development of GitLab](feature_flags/). +- Changes to [feature flags documentation guidelines](documentation/feature_flags.md). + +In these cases, use the following workflow: + +1. Request a peer review from a member of your team. +1. Request a review and approval of an Engineering Manager (EM) + or Staff Engineer who's responsible for the area in question: + + - [Frontend](https://about.gitlab.com/handbook/engineering/frontend/) + - [Backend](https://about.gitlab.com/handbook/engineering/) + - [Database](https://about.gitlab.com/handbook/engineering/development/database/) + - [User Experience (UX)](https://about.gitlab.com/handbook/engineering/ux/) + - [Security](https://about.gitlab.com/handbook/engineering/security/) + - [Quality](https://about.gitlab.com/handbook/engineering/quality/) + - [Engineering Productivity](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/) + - [Infrastructure](https://about.gitlab.com/handbook/engineering/infrastructure/) + - [Technical Writing](https://about.gitlab.com/handbook/engineering/ux/technical-writing/) + + You can skip this step for MRs authored by EMs or Staff Engineers responsible + for their area. + + If there are several affected groups, you may need approvals at the + EM/Staff Engineer level from each affected area. + +1. After completing the reviews, consult with the EM/Staff Engineer + author / approver of the MR. + + If this is a significant change across multiple areas, request final review + and approval from the VP of Development, the DRI for Development Guidelines, + @clefelhocz1. + +1. After all approvals are complete, assign the MR to the + [Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) + in the modified documentation page's metadata. + If the page is not assigned to a specific group, follow the + [Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). + The Technical Writer may ask for additional approvals as previously suggested before merging the MR. + +### Reviewer values + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57293) in GitLab 14.1. + +As a reviewer or as a reviewee, make sure to familiarize yourself with +the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/reviewer-values/) we strive for at GitLab. + +## Language-specific guides + +### Go guides + +- [Go Guidelines](go_guide/index.md) + +### Shell Scripting guides + +- [Shell scripting standards and style guidelines](shell_scripting_guide/index.md) diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md new file mode 100644 index 00000000000..539746f3e4e --- /dev/null +++ b/doc/development/feature_development.md @@ -0,0 +1,198 @@ +--- +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + +# Feature development + +Consult these topics for information on contributing to specific GitLab features. + +## UX and Frontend guides + +- [GitLab Design System](https://design.gitlab.com/), for building GitLab with + existing CSS styles and elements +- [Frontend guidelines](fe_guide/index.md) +- [Emoji guide](fe_guide/emojis.md) + +## Backend guides + +### General + +- [Directory structure](directory_structure.md) +- [GitLab EventStore](event_store.md) to publish/subscribe to domain events +- [GitLab utilities](utilities.md) +- [Newlines style guide](newlines_styleguide.md) +- [Logging](logging.md) +- [Dealing with email/mailers](emails.md) +- [Kubernetes integration guidelines](kubernetes.md) +- [Permissions](permissions.md) +- [Code comments](code_comments.md) +- [Windows Development on GCP](windows.md) +- [FIPS compliance](fips_compliance.md) +- [`Gemfile` guidelines](gemfile.md) +- [Ruby upgrade guidelines](ruby_upgrade.md) + +### Things to be aware of + +- [Gotchas](gotchas.md) to avoid +- [Avoid modules with instance variables](module_with_instance_variables.md), if + possible +- [Guidelines for reusing abstractions](reusing_abstractions.md) +- [Ruby 3 gotchas](ruby3_gotchas.md) + +### Rails Framework related + +- [Routing](routing.md) +- [Rails initializers](rails_initializers.md) +- [Mass Inserting Models](mass_insert.md) +- [Issuable-like Rails models](issuable-like-models.md) +- [Issue types vs first-class types](issue_types.md) +- [DeclarativePolicy framework](policies.md) +- [Rails update guidelines](rails_update.md) + +### Debugging + +- [Pry debugging](pry_debugging.md) +- [Sidekiq debugging](../administration/troubleshooting/sidekiq.md) + +### Git specifics + +- [How Git object deduplication works in GitLab](git_object_deduplication.md) +- [Git LFS](lfs.md) + +### API + +- [API style guide](api_styleguide.md) for contributing to the API +- [GraphQL API style guide](api_graphql_styleguide.md) for contributing to the + [GraphQL API](../api/graphql/index.md) + +### GitLab components and features + +- [Developing against interacting components or features](interacting_components.md) +- [Manage feature flags](feature_flags/index.md) +- [Licensed feature availability](licensed_feature_availability.md) +- [Accessing session data](session.md) +- [How to dump production data to staging](db_dump.md) +- [Geo development](geo.md) +- [Redis guidelines](redis.md) + - [Adding a new Redis instance](redis/new_redis_instance.md) +- [Sidekiq guidelines](sidekiq/index.md) for working with Sidekiq workers +- [Working with Gitaly](gitaly.md) +- [Elasticsearch integration docs](elasticsearch.md) +- [Working with merge request diffs](diffs.md) +- [Approval Rules](approval_rules.md) +- [Repository mirroring](repository_mirroring.md) +- [Uploads development guide](uploads/index.md) +- [Auto DevOps development guide](auto_devops.md) +- [Renaming features](renaming_features.md) +- [Code Intelligence](code_intelligence/index.md) +- [Feature categorization](feature_categorization/index.md) +- [Wikis development guide](wikis.md) +- [Image scaling guide](image_scaling.md) +- [Cascading Settings](cascading_settings.md) +- [Shell commands](shell_commands.md) in the GitLab codebase +- [Value Stream Analytics development guide](value_stream_analytics.md) +- [Application limits](application_limits.md) + +### Import and Export + +- [Working with the GitHub importer](github_importer.md) +- [Import/Export development documentation](import_export.md) +- [Test Import Project](import_project.md) +- [Group migration](bulk_import.md) +- [Export to CSV](export_csv.md) + +## Performance guides + +- [Performance guidelines](performance.md) for writing code, benchmarks, and + certain patterns to avoid. +- [Caching guidelines](caching.md) for using caching in Rails under a GitLab environment. +- [Merge request performance guidelines](merge_request_performance_guidelines.md) + for ensuring merge requests do not negatively impact GitLab performance +- [Profiling](profiling.md) a URL or tracking down N+1 queries using Bullet. +- [Cached queries guidelines](cached_queries.md), for tracking down N+1 queries + masked by query caching, memory profiling and why should we avoid cached + queries. + +## Database guides + +See [database guidelines](database/index.md). + +## Integration guides + +- [Integrations development guide](integrations/index.md) +- [Jira Connect app](integrations/jira_connect.md) +- [Security Scanners](integrations/secure.md) +- [Secure Partner Integration](integrations/secure_partner_integration.md) +- [How to run Jenkins in development environment](integrations/jenkins.md) +- [How to run local `Codesandbox` integration for Web IDE Live Preview](integrations/codesandbox.md) + +## Testing guides + +- [Testing standards and style guidelines](testing_guide/index.md) +- [Frontend testing standards and style guidelines](testing_guide/frontend_testing.md) + +## Refactoring guides + +- [Refactoring guidelines](refactoring_guide/index.md) + +## Deprecation guides + +- [Deprecation guidelines](deprecation_guidelines/index.md) + +## Documentation guides + +- [Writing documentation](documentation/index.md) +- [Documentation style guide](documentation/styleguide/index.md) +- [Markdown](../user/markdown.md) + +## Internationalization (i18n) guides + +- [Introduction](i18n/index.md) +- [Externalization](i18n/externalization.md) +- [Translation](i18n/translation.md) + +## Product Intelligence guides + +- [Product Intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) +- [Service Ping guide](service_ping/index.md) +- [Snowplow guide](snowplow/index.md) + +## Experiment guide + +- [Introduction](experiment_guide/index.md) + +## Build guides + +- [Building a package for testing purposes](build_test_package.md) + +## Compliance + +- [Licensing](licensing.md) for ensuring license compliance + +## Domain-specific guides + +- [CI/CD development documentation](cicd/index.md) +- [AppSec development documentation](appsec/index.md) + +## Technical Reference by Group + +- [Create: Source Code BE](backend/create_source_code_be/index.md) + +## Other development guides + +- [Defining relations between files using projections](projections.md) +- [Reference processing](reference_processing.md) +- [Compatibility with multiple versions of the application running at the same time](multi_version_compatibility.md) +- [Features inside `.gitlab/`](features_inside_dot_gitlab.md) +- [Dashboards for stage groups](stage_group_dashboards.md) +- [Preventing transient bugs](transient/prevention-patterns.md) +- [GitLab Application SLIs](application_slis/index.md) +- [Spam protection and CAPTCHA development guide](spam_protection_and_captcha/index.md) + +## Other GitLab Development Kit (GDK) guides + +- [Run full Auto DevOps cycle in a GDK instance](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/auto_devops.md) +- [Using GitLab Runner with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/runner.md) +- [Using the Web IDE terminal with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/web_ide_terminal_gdk_setup.md) diff --git a/doc/development/index.md b/doc/development/index.md index 3d5ec24d3e2..1b897db5097 100644 --- a/doc/development/index.md +++ b/doc/development/index.md @@ -46,307 +46,3 @@ GitLab instance, see the [Administrator documentation](../administration/index.m - [Implement design & UI elements](contributing/design.md) - [GitLab Architecture Overview](architecture.md) - [Rake tasks](rake_tasks.md) for development - -## Processes - -**Must-reads:** - -- [Guide on adapting existing and introducing new components](architecture.md#adapting-existing-and-introducing-new-components) -- [Code review guidelines](code_review.md) for reviewing code and having code - reviewed -- [Database review guidelines](database_review.md) for reviewing - database-related changes and complex SQL queries, and having them reviewed -- [Secure coding guidelines](secure_coding_guidelines.md) -- [Pipelines for the GitLab project](pipelines.md) - -Complementary reads: - -- [GitLab core team & GitLab Inc. contribution process](https://gitlab.com/gitlab-org/gitlab/-/blob/master/PROCESS.md) -- [Security process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#security-releases-critical-non-critical-as-a-developer) -- [Patch release process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/patch/process.md#process-for-developers) -- [Guidelines for implementing Enterprise Edition features](ee_features.md) -- [Adding a new service component to GitLab](adding_service_component.md) -- [Guidelines for changelogs](changelog.md) -- [Dependencies](dependencies.md) -- [Danger bot](dangerbot.md) -- [Requesting access to ChatOps on GitLab.com](chatops_on_gitlabcom.md#requesting-access) (for GitLab team members) - -### Development guidelines review - -When you submit a change to the GitLab development guidelines, who -you ask for reviews depends on the level of change. - -#### Wording, style, or link changes - -Not all changes require extensive review. For example, MRs that don't change the -content's meaning or function can be reviewed, approved, and merged by any -maintainer or Technical Writer. These can include: - -- Typo fixes. -- Clarifying links, such as to external programming language documentation. -- Changes to comply with the [Documentation Style Guide](documentation/index.md) - that don't change the intent of the documentation page. - -#### Specific changes - -If the MR proposes changes that are limited to a particular stage, group, or team, -request a review and approval from an experienced GitLab Team Member in that -group. For example, if you're documenting a new internal API used exclusively by -a given group, request an engineering review from one of the group's members. - -After the engineering review is complete, assign the MR to the -[Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) -in the modified documentation page's metadata. -If the page is not assigned to a specific group, follow the -[Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). - -#### Broader changes - -Some changes affect more than one group. For example: - -- Changes to [code review guidelines](code_review.md). -- Changes to [commit message guidelines](contributing/merge_request_workflow.md#commit-messages-guidelines). -- Changes to guidelines in [feature flags in development of GitLab](feature_flags/). -- Changes to [feature flags documentation guidelines](documentation/feature_flags.md). - -In these cases, use the following workflow: - -1. Request a peer review from a member of your team. -1. Request a review and approval of an Engineering Manager (EM) - or Staff Engineer who's responsible for the area in question: - - - [Frontend](https://about.gitlab.com/handbook/engineering/frontend/) - - [Backend](https://about.gitlab.com/handbook/engineering/) - - [Database](https://about.gitlab.com/handbook/engineering/development/database/) - - [User Experience (UX)](https://about.gitlab.com/handbook/engineering/ux/) - - [Security](https://about.gitlab.com/handbook/engineering/security/) - - [Quality](https://about.gitlab.com/handbook/engineering/quality/) - - [Engineering Productivity](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/) - - [Infrastructure](https://about.gitlab.com/handbook/engineering/infrastructure/) - - [Technical Writing](https://about.gitlab.com/handbook/engineering/ux/technical-writing/) - - You can skip this step for MRs authored by EMs or Staff Engineers responsible - for their area. - - If there are several affected groups, you may need approvals at the - EM/Staff Engineer level from each affected area. - -1. After completing the reviews, consult with the EM/Staff Engineer - author / approver of the MR. - - If this is a significant change across multiple areas, request final review - and approval from the VP of Development, the DRI for Development Guidelines, - @clefelhocz1. - -1. After all approvals are complete, assign the MR to the - [Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) - in the modified documentation page's metadata. - If the page is not assigned to a specific group, follow the - [Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). - The Technical Writer may ask for additional approvals as previously suggested before merging the MR. - -### Reviewer values - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57293) in GitLab 14.1. - -As a reviewer or as a reviewee, make sure to familiarize yourself with -the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/reviewer-values/) we strive for at GitLab. - -## UX and Frontend guides - -- [GitLab Design System](https://design.gitlab.com/), for building GitLab with - existing CSS styles and elements -- [Frontend guidelines](fe_guide/index.md) -- [Emoji guide](fe_guide/emojis.md) - -## Backend guides - -### General - -- [Directory structure](directory_structure.md) -- [GitLab EventStore](event_store.md) to publish/subscribe to domain events -- [GitLab utilities](utilities.md) -- [Newlines style guide](newlines_styleguide.md) -- [Logging](logging.md) -- [Dealing with email/mailers](emails.md) -- [Kubernetes integration guidelines](kubernetes.md) -- [Permissions](permissions.md) -- [Code comments](code_comments.md) -- [Windows Development on GCP](windows.md) -- [FIPS compliance](fips_compliance.md) -- [`Gemfile` guidelines](gemfile.md) -- [Ruby upgrade guidelines](ruby_upgrade.md) - -### Things to be aware of - -- [Gotchas](gotchas.md) to avoid -- [Avoid modules with instance variables](module_with_instance_variables.md), if - possible -- [Guidelines for reusing abstractions](reusing_abstractions.md) -- [Ruby 3 gotchas](ruby3_gotchas.md) - -### Rails Framework related - -- [Routing](routing.md) -- [Rails initializers](rails_initializers.md) -- [Mass Inserting Models](mass_insert.md) -- [Issuable-like Rails models](issuable-like-models.md) -- [Issue types vs first-class types](issue_types.md) -- [DeclarativePolicy framework](policies.md) -- [Rails update guidelines](rails_update.md) - -### Debugging - -- [Pry debugging](pry_debugging.md) -- [Sidekiq debugging](../administration/troubleshooting/sidekiq.md) - -### Git specifics - -- [How Git object deduplication works in GitLab](git_object_deduplication.md) -- [Git LFS](lfs.md) - -### API - -- [API style guide](api_styleguide.md) for contributing to the API -- [GraphQL API style guide](api_graphql_styleguide.md) for contributing to the - [GraphQL API](../api/graphql/index.md) - -### GitLab components and features - -- [Developing against interacting components or features](interacting_components.md) -- [Manage feature flags](feature_flags/index.md) -- [Licensed feature availability](licensed_feature_availability.md) -- [Accessing session data](session.md) -- [How to dump production data to staging](db_dump.md) -- [Geo development](geo.md) -- [Redis guidelines](redis.md) - - [Adding a new Redis instance](redis/new_redis_instance.md) -- [Sidekiq guidelines](sidekiq/index.md) for working with Sidekiq workers -- [Working with Gitaly](gitaly.md) -- [Elasticsearch integration docs](elasticsearch.md) -- [Working with merge request diffs](diffs.md) -- [Approval Rules](approval_rules.md) -- [Repository mirroring](repository_mirroring.md) -- [Uploads development guide](uploads/index.md) -- [Auto DevOps development guide](auto_devops.md) -- [Renaming features](renaming_features.md) -- [Code Intelligence](code_intelligence/index.md) -- [Feature categorization](feature_categorization/index.md) -- [Wikis development guide](wikis.md) -- [Image scaling guide](image_scaling.md) -- [Cascading Settings](cascading_settings.md) -- [Shell commands](shell_commands.md) in the GitLab codebase -- [Value Stream Analytics development guide](value_stream_analytics.md) -- [Application limits](application_limits.md) - -### Import/Export - -- [Working with the GitHub importer](github_importer.md) -- [Import/Export development documentation](import_export.md) -- [Test Import Project](import_project.md) -- [Group migration](bulk_import.md) -- [Export to CSV](export_csv.md) - -## Language-specific guides - -### Go guides - -- [Go Guidelines](go_guide/index.md) - -### Shell Scripting guides - -- [Shell scripting standards and style guidelines](shell_scripting_guide/index.md) - -## Performance guides - -- [Performance guidelines](performance.md) for writing code, benchmarks, and - certain patterns to avoid. -- [Caching guidelines](caching.md) for using caching in Rails under a GitLab environment. -- [Merge request performance guidelines](merge_request_performance_guidelines.md) - for ensuring merge requests do not negatively impact GitLab performance -- [Profiling](profiling.md) a URL or tracking down N+1 queries using Bullet. -- [Cached queries guidelines](cached_queries.md), for tracking down N+1 queries - masked by query caching, memory profiling and why should we avoid cached - queries. - -## Database guides - -See [database guidelines](database/index.md). - -## Integration guides - -- [Integrations development guide](integrations/index.md) -- [Jira Connect app](integrations/jira_connect.md) -- [Security Scanners](integrations/secure.md) -- [Secure Partner Integration](integrations/secure_partner_integration.md) -- [How to run Jenkins in development environment](integrations/jenkins.md) -- [How to run local `Codesandbox` integration for Web IDE Live Preview](integrations/codesandbox.md) - -## Testing guides - -- [Testing standards and style guidelines](testing_guide/index.md) -- [Frontend testing standards and style guidelines](testing_guide/frontend_testing.md) - -## Refactoring guides - -- [Refactoring guidelines](refactoring_guide/index.md) - -## Deprecation guides - -- [Deprecation guidelines](deprecation_guidelines/index.md) - -## Documentation guides - -- [Writing documentation](documentation/index.md) -- [Documentation style guide](documentation/styleguide/index.md) -- [Markdown](../user/markdown.md) - -## Internationalization (i18n) guides - -- [Introduction](i18n/index.md) -- [Externalization](i18n/externalization.md) -- [Translation](i18n/translation.md) - -## Product Intelligence guides - -- [Product Intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) -- [Service Ping guide](service_ping/index.md) -- [Snowplow guide](snowplow/index.md) - -## Experiment guide - -- [Introduction](experiment_guide/index.md) - -## Build guides - -- [Building a package for testing purposes](build_test_package.md) - -## Compliance - -- [Licensing](licensing.md) for ensuring license compliance - -## Domain-specific guides - -- [CI/CD development documentation](cicd/index.md) -- [AppSec development documentation](appsec/index.md) - -## Technical Reference by Group - -- [Create: Source Code BE](backend/create_source_code_be/index.md) - -## Other Development guides - -- [Defining relations between files using projections](projections.md) -- [Reference processing](reference_processing.md) -- [Compatibility with multiple versions of the application running at the same time](multi_version_compatibility.md) -- [Features inside `.gitlab/`](features_inside_dot_gitlab.md) -- [Dashboards for stage groups](stage_group_dashboards.md) -- [Preventing transient bugs](transient/prevention-patterns.md) -- [GitLab Application SLIs](application_slis/index.md) -- [Spam protection and CAPTCHA development guide](spam_protection_and_captcha/index.md) - -## Other GitLab Development Kit (GDK) guides - -- [Run full Auto DevOps cycle in a GDK instance](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/auto_devops.md) -- [Using GitLab Runner with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/runner.md) -- [Using the Web IDE terminal with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/web_ide_terminal_gdk_setup.md) diff --git a/doc/development/iterating_tables_in_batches.md b/doc/development/iterating_tables_in_batches.md index b4459b53efa..1159e3755e5 100644 --- a/doc/development/iterating_tables_in_batches.md +++ b/doc/development/iterating_tables_in_batches.md @@ -42,20 +42,20 @@ The API of this method is similar to `in_batches`, though it doesn't support all of the arguments that `in_batches` supports. You should always use `each_batch` _unless_ you have a specific need for `in_batches`. -## Avoid iterating over non-unique columns +## Iterating over non-unique columns -One should proceed with extra caution, and possibly avoid iterating over a column that can contain -duplicate values. When you iterate over an attribute that is not unique, even with the applied max -batch size, there is no guarantee that the resulting batches do not surpass it. The following -snippet demonstrates this situation when one attempt to select `Ci::Build` entries for users with -`id` between `1` and `10,000`, the database returns `1 215 178` matching rows. +One should proceed with extra caution. When you iterate over an attribute that is not unique, +even with the applied max batch size, there is no guarantee that the resulting batches do not +surpass it. The following snippet demonstrates this situation when one attempt to select +`Ci::Build` entries for users with `id` between `1` and `10,000`, the database returns +`1 215 178` matching rows. ```ruby [ gstg ] production> Ci::Build.where(user_id: (1..10_000)).size => 1215178 ``` -This happens because built relation is translated into the following query +This happens because the built relation is translated into the following query: ```ruby [ gstg ] production> puts Ci::Build.where(user_id: (1..10_000)).to_sql @@ -69,6 +69,27 @@ threshold does not translate to the size of the returned dataset. That happens b `n` possible values of attributes, one can't tell for sure that the number of records that contains them is less than `n`. +### Loose-index scan with `distinct_each_batch` + +When iterating over a non-unique column is necessary, use the `distinct_each_batch` helper +method. The helper uses the [loose-index scan technique](https://wiki.postgresql.org/wiki/Loose_indexscan) +(skip-index scan) to skip duplicated values within a database index. + +Example: iterating over distinct `author_id` in the Issue model + +```ruby +Issue.distinct_each_batch(column: :author_id, of: 1000) do |relation| + users = User.where(id: relation.select(:author_id)).to_a +end +``` + +The technique provides stable performance between the batches regardless of the data distribution. +The `relation` object returns an ActiveRecord scope where only the given `column` is available. +Other columns are not loaded. + +The underlying database queries use recursive CTEs, which adds extra overhead. We therefore advise to use +smaller batch sizes than those used for a standard `each_batch` iteration. + ## Column definition `EachBatch` uses the primary key of the model by default for the iteration. This works most of the |