diff options
Diffstat (limited to 'doc/development')
166 files changed, 2176 insertions, 1484 deletions
diff --git a/doc/development/README.md b/doc/development/README.md index 0d3c1b3cbe9..d73acdd9dbc 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -151,6 +151,7 @@ In these cases, use the following workflow: ## Backend guides +- [Directory structure](directory_structure.md) - [GitLab utilities](utilities.md) - [Issuable-like Rails models](issuable-like-models.md) - [Logging](logging.md) @@ -229,7 +230,7 @@ See [database guidelines](database/index.md). - [Security Scanners](integrations/secure.md) - [Secure Partner Integration](integrations/secure_partner_integration.md) - [How to run Jenkins in development environment](integrations/jenkins.md) -- [How to run local Codesandbox integration for Web IDE Live Preview](integrations/codesandbox.md) +- [How to run local `Codesandbox` integration for Web IDE Live Preview](integrations/codesandbox.md) ## Testing guides diff --git a/doc/development/adding_service_component.md b/doc/development/adding_service_component.md index 7e2add2c91f..f3e23906ac6 100644 --- a/doc/development/adding_service_component.md +++ b/doc/development/adding_service_component.md @@ -65,7 +65,7 @@ Notify the [Distribution team](https://about.gitlab.com/handbook/engineering/dev New services to be bundled with GitLab need to be available in the following environments. -**Dev environment** +**Development environment** The first step of bundling a new service is to provide it in the development environment to engage in collaboration and feedback. diff --git a/doc/development/agent/gitops.md b/doc/development/agent/gitops.md index 8c8586326fa..f183ba86aa1 100644 --- a/doc/development/agent/gitops.md +++ b/doc/development/agent/gitops.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# GitOps with the Kubernetes Agent **(PREMIUM ONLY)** +# GitOps with the Kubernetes Agent **(PREMIUM SELF)** The [GitLab Kubernetes Agent](../../user/clusters/agent/index.md) supports the [pull-based version](https://www.gitops.tech/#pull-based-deployments) of diff --git a/doc/development/agent/identity.md b/doc/development/agent/identity.md index 884ce015a02..49d20d2fd87 100644 --- a/doc/development/agent/identity.md +++ b/doc/development/agent/identity.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Kubernetes Agent identity and authentication **(PREMIUM ONLY)** +# Kubernetes Agent identity and authentication **(PREMIUM SELF)** This page uses the word `agent` to describe the concept of the GitLab Kubernetes Agent. The program that implements the concept is called `agentk`. diff --git a/doc/development/agent/index.md b/doc/development/agent/index.md index 95661c8ddbd..112162f8f90 100644 --- a/doc/development/agent/index.md +++ b/doc/development/agent/index.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Kubernetes Agent development **(PREMIUM ONLY)** +# Kubernetes Agent development **(PREMIUM SELF)** This page contains developer-specific information about the GitLab Kubernetes Agent. [End-user documentation about the GitLab Kubernetes Agent](../../user/clusters/agent/index.md) diff --git a/doc/development/agent/local.md b/doc/development/agent/local.md index 47246a6a6d3..603364567bf 100644 --- a/doc/development/agent/local.md +++ b/doc/development/agent/local.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Run the Kubernetes Agent locally **(PREMIUM ONLY)** +# Run the Kubernetes Agent locally **(PREMIUM SELF)** You can run `kas` and `agentk` locally to test the [Kubernetes Agent](index.md) yourself. diff --git a/doc/development/agent/repository_overview.md b/doc/development/agent/repository_overview.md new file mode 100644 index 00000000000..b9eea286a3e --- /dev/null +++ b/doc/development/agent/repository_overview.md @@ -0,0 +1,98 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Kubernetes Agent repository overview **(PREMIUM SELF)** + +This page describes the subfolders of the Kubernetes Agent repository. +[Development information](index.md) and +[end-user documentation](../../user/clusters/agent/index.md) are both available. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For a video overview, see +[GitLab Kubernetes Agent repository overview](https://www.youtube.com/watch?v=j8CyaCWroUY). + +## `build` + +Various files for the build process. + +### `build/deployment` + +A [`kpt`](https://googlecontainertools.github.io/kpt/) package that bundles some +[Kustomize](https://kustomize.io/) layers and components. Can be used as-is, or +to create a custom package to install `agentk`. + +## `cmd` + +Commands are binaries that this repository produces. They are: + +- `kas` is the GitLab Kubernetes Agent Server binary. +- `agentk` is the GitLab Kubernetes Agent binary. + +Each of these directories contain application bootstrap code for: + +- Reading configuration. +- Applying defaults to it. +- Constructing the dependency graph of objects that constitute the program. +- Running it. + +### `cmd/agentk` + +- `agentk` initialization logic. +- Implementation of the agent modules API. + +### `cmd/kas` + +- `kas` initialization logic. +- Implementation of the server modules API. + +## `examples` + +Git submodules for the example projects. + +## `internal` + +The main code of both `gitlab-kas` and `agentk`, and various supporting building blocks. + +### `internal/api` + +Structs that represent some important pieces of data. + +### `internal/gitaly` + +Items to work with [Gitaly](../../administration/gitaly/index.md). + +### `internal/gitlab` + +GitLab REST client. + +### `internal/module` + +Modules that implement server and agent-side functionality. + +### `internal/tool` + +Various building blocks. `internal/tool/testing` contains mocks and helpers +for testing. Mocks are generated with [`gomock`](https://pkg.go.dev/github.com/golang/mock). + +## `it` + +Contains scaffolding for integration tests. Unused at the moment. + +## `pkg` + +Contains exported packages. + +### `pkg/agentcfg` + +Contains protobuf definitions of the `agentk` configuration file. Used to configure +the agent through a configuration repository. + +### `pkg/kascfg` + +Contains protobuf definitions of the `gitlab-kas` configuration file. Contains an +example of that configuration file along with the test for it. The test ensures +the configuration file example is in sync with the protobuf definitions of the +file and defaults, which are applied when the file is loaded. diff --git a/doc/development/agent/routing.md b/doc/development/agent/routing.md index 43cc78ccdfb..9a7d6422d47 100644 --- a/doc/development/agent/routing.md +++ b/doc/development/agent/routing.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Routing `kas` requests in the Kubernetes Agent **(PREMIUM ONLY)** +# Routing `kas` requests in the Kubernetes Agent **(PREMIUM SELF)** This document describes how `kas` routes requests to concrete `agentk` instances. GitLab must talk to GitLab Kubernetes Agent Server (`kas`) to: @@ -134,90 +134,97 @@ of the `kas` receiving the request from the _external_ endpoint to retry and re- requests. This method ensures a single central component for each request can determine how a request is routed, rather than distributing the decision across several `kas` instances. -### API definitions +### Reverse gRPC tunnel + +This section explains how the `agentk` -> `kas` reverse gRPC tunnel is implemented. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For a video overview of how some of the blocks map to code, see +[GitLab Kubernetes Agent reverse gRPC tunnel architecture and code overview +](https://www.youtube.com/watch?v=9pnQF76hyZc). + +#### High level schema + +In this example, `Server side of module A` exposes its API to get the `Pod` list +on the `Public API gRPC server`. When it receives a request, it must determine +the agent ID from it, then call the proxying code which forwards the request to +a suitable `agentk` that can handle it. + +The `Agent side of module A` exposes the same API on the `Internal gRPC server`. +When it receives the request, it needs to handle it (such as retrieving and returning +the `Pod` list). + +This schema describes how reverse tunneling is handled fully transparently +for modules, so you can add new features: + +```mermaid +graph TB + subgraph kas + server-internal-grpc-server[Internal gRPC server] + server-api-grpc-server[Public API gRPC server] + server-module-a[Server side of module A] + server-module-b[Server side of module B] + end + subgraph agentk + agent-internal-grpc-server[Internal gRPC server] + agent-module-a[Agent side of module A] + agent-module-b[Agent side of module B] + end + + agent-internal-grpc-server -- request --> agent-module-a + agent-internal-grpc-server -- request --> agent-module-b + + server-module-a-. expose API on .-> server-internal-grpc-server + server-module-b-. expose API on .-> server-api-grpc-server + + server-internal-grpc-server -- proxy request --> agent-internal-grpc-server + server-api-grpc-server -- proxy request --> agent-internal-grpc-server +``` + +#### Implementation schema + +`HandleTunnelConnection()` is called with the server-side interface of the reverse +tunnel. It registers the connection and blocks, waiting for a request to proxy +through the connection. + +`HandleIncomingConnection()` is called with the server-side interface of the incoming +connection. It registers the connection and blocks, waiting for a matching tunnel +to proxy the connection through. + +After it has two connections that match, `Connection registry` starts bi-directional +data streaming: -```proto -syntax = "proto3"; - -import "google/protobuf/timestamp.proto"; - -message KasAddress { - string ip = 1; - uint32 port = 2; -} - -message ConnectedAgentInfo { - // Agent id. - int64 id = 1; - // Identifies a particular agentk->kas connection. Randomly generated when agent connects. - int64 connection_id = 2; - string version = 3; - string commit = 4; - // Pod namespace. - string pod_namespace = 5; - // Pod name. - string pod_name = 6; - // When the connection was established. - google.protobuf.Timestamp connected_at = 7; - KasAddress kas_address = 8; - // What else do we need? -} - -message KasInstanceInfo { - string version = 1; - string commit = 2; - KasAddress address = 3; - // What else do we need? -} - -message ConnectedAgentsForProjectRequest { - int64 project_id = 1; -} - -message ConnectedAgentsForProjectResponse { - // There may 0 or more agents with the same id, depending on the number of running Pods. - repeated ConnectedAgentInfo agents = 1; -} - -message ConnectedAgentsByIdRequest { - int64 agent_id = 1; -} - -message ConnectedAgentsByIdResponse { - repeated ConnectedAgentInfo agents = 1; -} - -// API for use by GitLab. -service KasApi { - // Connected agents for a particular configuration project. - rpc ConnectedAgentsForProject (ConnectedAgentsForProjectRequest) returns (ConnectedAgentsForProjectResponse) { - } - // Connected agents for a particular agent id. - rpc ConnectedAgentsById (ConnectedAgentsByIdRequest) returns (ConnectedAgentsByIdResponse) { - } - // Depends on the need, but here is the call from the example above. - rpc GetPods (GetPodsRequest) returns (GetPodsResponse) { - } -} - -message Pod { - string namespace = 1; - string name = 2; -} - -message GetPodsRequest { - int64 agent_id = 1; - int64 connection_id = 2; -} - -message GetPodsResponse { - repeated Pod pods = 1; -} - -// Internal API for use by kas for kas -> kas calls. -service KasInternal { - // Depends on the need, but here is the call from the example above. - rpc GetPods (GetPodsRequest) returns (GetPodsResponse) { - } -} +```mermaid +graph TB + subgraph kas + server-tunnel-module[Server tunnel module] + connection-registry[Connection registry] + server-internal-grpc-server[Internal gRPC server] + server-api-grpc-server[Public API gRPC server] + server-module-a[Server side of module A] + server-module-b[Server side of module B] + end + subgraph agentk + agent-internal-grpc-server[Internal gRPC server] + agent-tunnel-module[Agent tunnel module] + agent-module-a[Agent side of module A] + agent-module-b[Agent side of module B] + end + + server-tunnel-module -- "HandleTunnelConnection()" --> connection-registry + server-internal-grpc-server -- "HandleIncomingConnection()" --> connection-registry + server-api-grpc-server -- "HandleIncomingConnection()" --> connection-registry + server-module-a-. expose API on .-> server-internal-grpc-server + server-module-b-. expose API on .-> server-api-grpc-server + + agent-tunnel-module -- "establish tunnel, receive request" --> server-tunnel-module + agent-tunnel-module -- make request --> agent-internal-grpc-server + agent-internal-grpc-server -- request --> agent-module-a + agent-internal-grpc-server -- request --> agent-module-b ``` + +### API definitions + +- [`agent_tracker/agent_tracker.proto`](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/internal/module/agent_tracker/agent_tracker.proto) +- [`agent_tracker/rpc/rpc.proto`](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/internal/module/agent_tracker/rpc/rpc.proto) +- [`reverse_tunnel/rpc/rpc.proto`](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/internal/module/reverse_tunnel/rpc/rpc.proto) diff --git a/doc/development/agent/user_stories.md b/doc/development/agent/user_stories.md index 2929573ffd3..609be47a3cb 100644 --- a/doc/development/agent/user_stories.md +++ b/doc/development/agent/user_stories.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Kubernetes Agent user stories **(PREMIUM ONLY)** +# Kubernetes Agent user stories **(PREMIUM SELF)** The [personas in action](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#user-personas) for the Kubernetes Agent are: diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index d73c3a8d6f6..c0317a7e73e 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -10,7 +10,9 @@ This document outlines the style guide for the GitLab [GraphQL API](../api/graph ## How GitLab implements GraphQL +<!-- vale gitlab.Spelling = NO --> We use the [GraphQL Ruby gem](https://graphql-ruby.org/) written by [Robert Mosolgo](https://github.com/rmosolgo/). +<!-- vale gitlab.Spelling = YES --> All GraphQL queries are directed to a single endpoint ([`app/controllers/graphql_controller.rb#execute`](https://gitlab.com/gitlab-org/gitlab/blob/master/app%2Fcontrollers%2Fgraphql_controller.rb)), @@ -21,6 +23,7 @@ which is exposed as an API endpoint at `/api/graphql`. In March 2019, Nick Thomas hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on the GitLab [GraphQL API](../api/graphql/index.md) to share his domain specific knowledge with anyone who may work in this part of the codebase in the future. You can find the +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=-9L_1MWrjkg), and the slides on [Google Slides](https://docs.google.com/presentation/d/1qOTxpkTdHIp1CRjuTvO-aXg0_rUtzE3ETfLUdnBB5uQ/edit) and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/8e78ea7f326b2ef649e7d7d569c26d56/GraphQL_Deep_Dive__Create_.pdf). @@ -42,6 +45,32 @@ can be shared. It's also possible to add a `private_token` to the query string, or add a `HTTP_PRIVATE_TOKEN` header. +## Limits + +Several limits apply to the GraphQL API and some of these can be overridden +by developers. + +### Max page size + +By default, [connections](#connection-types) can only return +at most a maximum number of records defined in +[`app/graphql/gitlab_schema.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/gitlab_schema.rb) +per page. + +Developers can [specify a custom max page size](#page-size-limit) when defining +a connection. + +### Max complexity + +Complexity is explained [on our client-facing API page](../api/graphql/index.md#max-query-complexity). + +Fields default to adding `1` to a query's complexity score, but developers can +[specify a custom complexity](#field-complexity) when defining a field. + +### Request timeout + +Requests time out at 30 seconds. + ## Global IDs The GitLab GraphQL API uses Global IDs (i.e: `"gid://gitlab/MyObject/123"`) @@ -281,6 +310,61 @@ Use the functionality the framework provides unless there is a compelling reason For example, instead of `latest_pipeline`, use `pipelines(last: 1)`. +#### Page size limit + +By default, the API returns at most a maximum number of records defined in +[`app/graphql/gitlab_schema.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/gitlab_schema.rb) +per page within a connection and this will also be the default number of records +returned per page if no limiting arguments (`first:` or `last:`) are provided by a client. + +The `max_page_size` argument can be used to specify a different page size limit +for a connection. + +WARNING: +It's better to change the frontend client, or product requirements, to not need large amounts of +records per page than it is to raise the `max_page_size`, as the default is set to ensure +the GraphQL API remains performant. + +For example: + +```ruby +field :tags, + Types::ContainerRepositoryTagType.connection_type, + null: true, + description: 'Tags of the container repository', + max_page_size: 20 +``` + +### Field complexity + +The GitLab GraphQL API uses a _complexity_ score to limit performing overly complex queries. +Complexity is described in [our client documentation](../api/graphql/index.md#max-query-complexity) on the topic. + +Complexity limits are defined in [`app/graphql/gitlab_schema.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/gitlab_schema.rb). + +By default, fields will add `1` to a query's complexity score. This can be overridden by +[providing a custom `complexity`](https://graphql-ruby.org/queries/complexity_and_depth.html) value for a field. + +Developers should specify higher complexity for fields that cause more _work_ to be performed +by the server in order to return data. Fields that represent data that can be returned +with little-to-no _work_, for example in most cases; `id` or `title`, can be given a complexity of `0`. + +### `calls_gitaly` + +Fields that have the potential to perform a [Gitaly](../administration/gitaly/index.md) call when resolving _must_ be marked as +such by passing `calls_gitaly: true` to `field` when defining it. + +For example: + +```ruby +field :blob, type: Types::Snippets::BlobType, + description: 'Snippet blob', + null: false, + calls_gitaly: true +``` + +This will increment the [`complexity` score](#field-complexity) of the field by `1`. + ### Exposing permissions for a type To expose permissions the current user has on a resource, you can call @@ -1599,7 +1683,7 @@ full stack: - An argument or scalar's [`prepare`](#validating-arguments) applies correctly. - Logic in a resolver or mutation's [`#ready?` method](#correct-use-of-resolverready) applies correctly. - An [argument's `default_value`](https://graphql-ruby.org/fields/arguments.html) applies correctly. -- Objects resolve performantly and there are no N+1 issues. +- Objects resolve successfully, and there are no N+1 issues. When adding a query, you can use the `a working graphql query` shared example to test if the query renders valid results. diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index b2c93f16770..dd43281da6d 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -28,7 +28,7 @@ See the [Documentation Style Guide RESTful API page](documentation/restful_api_s ## Methods and parameters description Every method must be described using the [Grape DSL](https://github.com/ruby-grape/grape#describing-methods) -(see <https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/environments.rb> +(see [`environments.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/environments.rb) for a good example): - `desc` for the method summary. You should pass it a block for additional diff --git a/doc/development/approval_rules.md b/doc/development/approval_rules.md index 542bce1cb97..eae221c6598 100644 --- a/doc/development/approval_rules.md +++ b/doc/development/approval_rules.md @@ -4,7 +4,7 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Approval Rules **(STARTER)** +# Approval Rules development guide This document explains the backend design and flow of all related functionality about [merge request approval rules](../user/project/merge_requests/merge_request_approvals.md). @@ -44,8 +44,8 @@ erDiagram ### `Project` and `MergeRequest` `Project` and `MergeRequest` models are defined in `ee/app/models/ee/project.rb` -and `ee/app/models/ee/merge_request.rb`. They extend the non-EE versions since -approval rules is an EE only feature. Associations and other related stuff to +and `ee/app/models/ee/merge_request.rb`. They extend the non-EE versions, because +approval rules are an EE-only feature. Associations and other related stuff to merge request approvals are defined here. ### `ApprovalState` @@ -265,8 +265,8 @@ graph LR ApprovalWrappedRule --> Approval ``` -This flow gets initiated by the frontend component. The data returned will -then be used to display information on the MR widget. +This flow gets initiated by the frontend component. The data returned is +used to display information on the MR widget. ### Approving a merge request @@ -282,5 +282,5 @@ is executed instead. ## TODO -1. Add information related to other rule types (e.g. `code_owner` and `report_approver`). +1. Add information related to other rule types, such as `code_owner` and `report_approver`. 1. Add information about side effects of approving/unapproving merge request. diff --git a/doc/development/architecture.md b/doc/development/architecture.md index f8ab97de848..3aab57d8e15 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -91,6 +91,8 @@ The simplest way to ensure this, is to add support for your feature or service t [the official GitLab Helm chart](https://docs.gitlab.com/charts/) or reach out to [the Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/#how-to-work-with-distribution). +Refer to the [process for adding new service components](adding_service_component.md) for more details. + ### Simplified component overview This is a simplified architecture diagram that can be used to @@ -934,7 +936,7 @@ ps aux | grep '^git' ``` GitLab has several components to operate. It requires a persistent database -(PostgreSQL) and Redis database, and uses Apache `httpd` or NGINX to proxypass +(PostgreSQL) and Redis database, and uses Apache `httpd` or NGINX to `proxypass` Puma. All these components should run as different system users to GitLab (for example, `postgres`, `redis`, and `www-data`, instead of `git`). diff --git a/doc/development/auto_devops.md b/doc/development/auto_devops.md index c457573b87a..6b54de55738 100644 --- a/doc/development/auto_devops.md +++ b/doc/development/auto_devops.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Auto DevOps development guide +# Auto DevOps development guide **(CORE)** This document provides a development guide for contributors to [Auto DevOps](../topics/autodevops/index.md). diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 8fad32ed163..88a0cf8dff0 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -45,7 +45,7 @@ the `author` field. GitLab team members **should not**. **must** have a changelog entry, without `merge_request` value and with `type` set to `security`. - Any user-facing change **must** have a changelog entry. This includes both visual changes (regardless of how minor), and changes to the rendered DOM which impact how a screen reader may announce the content. -- Any client-facing change to our REST and GraphQL APIs **must** have a changelog entry. +- Any client-facing change to our REST and GraphQL APIs **must** have a changelog entry. This includes modifying complexity of GraphQL fields. - Performance improvements **should** have a changelog entry. - Changes that need to be documented in the Product Intelligence [Event Dictionary](https://about.gitlab.com/handbook/product/product-intelligence-guide/#event-dictionary) also require a changelog entry. diff --git a/doc/development/chaos_endpoints.md b/doc/development/chaos_endpoints.md index 9104c01c980..85c93f521ac 100644 --- a/doc/development/chaos_endpoints.md +++ b/doc/development/chaos_endpoints.md @@ -6,8 +6,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Generating chaos in a test GitLab instance +<!-- vale gitlab.Spelling = NO --> + As [Werner Vogels](https://twitter.com/Werner), the CTO at Amazon Web Services, famously put it, **Everything fails, all the time**. +<!-- vale gitlab.Spelling = NO --> + As a developer, it's as important to consider the failure modes in which your software may operate as much as normal operation. Doing so can mean the difference between a minor hiccup leading to a scattering of `500` errors experienced by a tiny fraction of users, and a full site outage that affects all users for an extended period. To paraphrase [Tolstoy](https://en.wikipedia.org/wiki/Anna_Karenina_principle), _all happy servers are alike, but all failing servers are failing in their own way_. Luckily, there are ways we can attempt to simulate these failure modes, and the chaos endpoints are tools for assisting in this process. @@ -160,3 +164,58 @@ GET /-/chaos/kill?async=true curl "http://localhost:3000/-/chaos/kill" --header 'X-Chaos-Secret: secret' curl "http://localhost:3000/-/chaos/kill?token=secret" ``` + +## Run garbage collector + +This endpoint triggers a GC run on the worker handling the request and returns its worker ID +plus GC stats as JSON. This is mostly useful when running Puma in standalone mode, since +otherwise the worker handling the request will not be known upfront. + +Endpoint: + +```plaintext +POST /-/chaos/gc +``` + +Example request: + +```shell +curl --request POST "http://localhost:3000/-/chaos/gc" --header 'X-Chaos-Secret: secret' +curl --request POST "http://localhost:3000/-/chaos/gc?token=secret" +``` + +Example response: + +```json +{ + "worker_id": "puma_1", + "gc_stat": { + "count": 94, + "heap_allocated_pages": 9077, + "heap_sorted_length": 9077, + "heap_allocatable_pages": 0, + "heap_available_slots": 3699720, + "heap_live_slots": 2827510, + "heap_free_slots": 872210, + "heap_final_slots": 0, + "heap_marked_slots": 2827509, + "heap_eden_pages": 9077, + "heap_tomb_pages": 0, + "total_allocated_pages": 9077, + "total_freed_pages": 0, + "total_allocated_objects": 14229357, + "total_freed_objects": 11401847, + "malloc_increase_bytes": 8192, + "malloc_increase_bytes_limit": 30949538, + "minor_gc_count": 71, + "major_gc_count": 23, + "compact_count": 0, + "remembered_wb_unprotected_objects": 41685, + "remembered_wb_unprotected_objects_limit": 83370, + "old_objects": 2617806, + "old_objects_limit": 5235612, + "oldmalloc_increase_bytes": 8192, + "oldmalloc_increase_bytes_limit": 122713697 + } +} +``` diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index eede1d691a9..d79d681263d 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -143,3 +143,25 @@ Finally if the runner can only pick jobs that are tagged, all untagged jobs are At this point we loop through remaining `pending` jobs and we try to assign the first job that the runner "can pick" based on additional policies. For example, runners marked as `protected` can only pick jobs that run against protected branches (such as production deployments). As we increase the number of runners in the pool we also increase the chances of conflicts which would arise if assigning the same job to different runners. To prevent that we gracefully rescue conflict errors and assign the next job in the list. + +## The definition of "Job" in GitLab CI/CD + +"Job" in GitLab CI context refers a task to drive Continuous Integration, Delivery and Deployment. +Typically, a pipeline contains multiple stages, and a stage contains multiple jobs. + +In Active Record modeling, Job is defined as `CommitStatus` class. +On top of that, we have the following types of jobs: + +- `Ci::Build` ... The job to be executed by runners. +- `Ci::Bridge` ... The job to trigger a downstream pipeline. +- `GenericCommitStatus` ... The job to be executed in an external CI/CD system e.g. Jenkins. + +Please note that, when you use the "Job" terminology in codebase, readers would +assume that the class/object is any type of above. +If you specifically refer `Ci::Build` class, you should not name the object/class +as "job" as this could cause some confusions. In documentation, +we should use "Job" in general, instead of "Build". + +We have a few inconsistencies in our codebase that should be refactored. +For example, `CommitStatus` should be `Ci::Job` and `Ci::JobArtifact` should be `Ci::BuildArtifact`. +See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/16111) for the full refactoring plan. diff --git a/doc/development/code_intelligence/index.md b/doc/development/code_intelligence/index.md index c5673f6eee2..02f701645f4 100644 --- a/doc/development/code_intelligence/index.md +++ b/doc/development/code_intelligence/index.md @@ -45,7 +45,7 @@ sequenceDiagram GitLab Rails to authorize the upload. 1. GitLab Rails validates whether the artifact can be uploaded and sends - `ProcessLsif: true` header if the lsif artifact can be processed. + `ProcessLsif: true` header if the LSIF artifact can be processed. 1. Workhorse reads the LSIF document line by line and generates code intelligence data for each file in the project. The output is a zipped directory of JSON diff --git a/doc/development/code_review.md b/doc/development/code_review.md index fe395dc2304..d856541bd91 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -24,7 +24,7 @@ uncovered edge cases. The default approach is to choose a reviewer from your group or team for the first review. This is only a recommendation and the reviewer may be from a different team. However, it is recommended to pick someone who is a [domain expert](#domain-experts). -If your merge request touches more than one domain (for example, Dynamic Analysis and GraphQL), ask for reviews from an expert from each domain. +If your merge request touches more than one domain (for example, Dynamic Analysis and GraphQL), ask for reviews from an expert from each domain. You can read more about the importance of involving reviewer(s) in the section on the responsibility of the author below. @@ -107,11 +107,11 @@ with [domain expertise](#domain-experts). be **approved by a [frontend foundations member](https://about.gitlab.com/direction/create/ecosystem/frontend-ux-foundations/)**. - If the license used by the new library hasn't been approved for use in GitLab, the license must be **approved by a [legal department member](https://about.gitlab.com/handbook/legal/)**. - More information about license compatiblity can be found in our + More information about license compatibility can be found in our [GitLab Licensing and Compatibility documentation](licensing.md). 1. If your merge request includes adding a new UI/UX paradigm (*1*), it must be **approved by a [UX lead](https://about.gitlab.com/company/team/)**. -1. If your merge request includes a new dependency or a filesystem change, it must be +1. If your merge request includes a new dependency or a file system change, it must be **approved by a [Distribution team member](https://about.gitlab.com/company/team/)**. See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/#how-to-work-with-distribution) for more details. 1. If your merge request includes documentation changes, it must be **approved by a [Technical writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments)**, based on @@ -121,6 +121,8 @@ with [domain expertise](#domain-experts). 1. If your merge request only includes end-to-end changes (*3*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)** 1. If your merge request includes a new or updated [application limit](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits), it must be **approved by a [product manager](https://about.gitlab.com/company/team/)**. 1. If your merge request includes Product Intelligence (telemetry or analytics) changes, it should be reviewed and approved by a [Product Intelligence engineer](https://gitlab.com/gitlab-org/growth/product_intelligence/engineers). +1. If your merge request includes an addition of, or changes to a [Feature spec](testing_guide/testing_levels.md#frontend-feature-tests), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa) or [Quality reviewer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_qa)**. +1. If your merge request introduces a new service to GitLab (Puma, Sidekiq, Gitaly are examples), it must be **approved by a [product manager](https://about.gitlab.com/company/team/)**. See the [process for adding a service component to GitLab](adding_service_component.md) for details. - (*1*): Please note that specs other than JavaScript specs are considered backend code. - (*2*): We encourage you to seek guidance from a database maintainer if your merge @@ -177,8 +179,11 @@ warrant a comment could be: Avoid: -- Adding comments (referenced above, or TODO items) directly to the source code unless the reviewer requires you to do so. If the comments are added due to an actionable task, -a link to an issue must be included. +- Adding TODO comments (referenced above) directly to the source code unless the reviewer requires + you to do so. If TODO comments are added due to an actionable task, + [include a link to the relevant issue](code_comments.md). +- Adding comments which only explain what the code is doing. If non-TODO comments are added, they should + [_explain why, not what_](https://blog.codinghorror.com/code-tells-you-how-comments-tell-you-why/). - Assigning merge requests with failed tests to maintainers. If the tests are failing and you have to assign, ensure you leave a comment with an explanation. - Excessively mentioning maintainers through email or Slack (if the maintainer is reachable through Slack). If you can't assign a merge request, `@` mentioning a maintainer in a comment is acceptable and in all other cases assigning the merge request is sufficient. @@ -340,7 +345,7 @@ experience, refactors the existing code). Then: convey your intent. - For non-mandatory suggestions, decorate with (non-blocking) so the author knows they can optionally resolve within the merge request or follow-up at a later stage. - - There's a [Chrome/Firefox addon](https://gitlab.com/conventionalcomments/conventional-comments-button) which you can use to apply [Conventional Comment](https://conventionalcomments.org/) prefixes. + - There's a [Chrome/Firefox add-on](https://gitlab.com/conventionalcomments/conventional-comments-button) which you can use to apply [Conventional Comment](https://conventionalcomments.org/) prefixes. - After a round of line notes, it can be helpful to post a summary note such as "Looks good to me", or "Just a couple things to address." - Assign the merge request to the author if changes are required following your @@ -565,7 +570,7 @@ A good example of collaboration on an MR touching multiple parts of the codebase ### Credits -Largely based on the [thoughtbot code review guide](https://github.com/thoughtbot/guides/tree/master/code-review). +Largely based on the [`thoughtbot` code review guide](https://github.com/thoughtbot/guides/tree/master/code-review). --- diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 7859af4e88b..5c8068fdee7 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -212,7 +212,7 @@ the contribution acceptance criteria below: 1. Changes do not degrade performance: - Avoid repeated polling of endpoints that require a significant amount of overhead. - Check for N+1 queries via the SQL log or [`QueryRecorder`](../merge_request_performance_guidelines.md). - - Avoid repeated access of the filesystem. + - Avoid repeated access of the file system. - Use [polling with ETag caching](../polling.md) if needed to support real-time features. 1. If the merge request adds any new libraries (gems, JavaScript libraries, etc.), they should conform to our [Licensing guidelines](../licensing.md). See those diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index c316d50c88c..2a2cfebe964 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -15,52 +15,83 @@ settings automatically by default. If your editor/IDE does not automatically sup we suggest investigating to see if a plugin exists. For instance here is the [plugin for vim](https://github.com/editorconfig/editorconfig-vim). -## Pre-push static analysis +## Pre-push static analysis with Lefthook -We strongly recommend installing [Lefthook](https://github.com/Arkweid/lefthook) to automatically check -for static analysis offenses before pushing your changes. +[Lefthook](https://github.com/Arkweid/lefthook) is a Git hooks manager that allows +custom logic to be executed prior to Git committing or pushing. GitLab comes with +Lefthook configuration (`lefthook.yml`), but it must be installed. -To install `lefthook`, run the following in your GitLab source directory: +We have a `lefthook.yml` checked in but it is ignored until Lefthook is installed. -```shell -# 1. Make sure to uninstall Overcommit first -overcommit --uninstall +### Uninstall Overcommit + +We were using Overcommit prior to Lefthook, so you may want to uninstall it first with `overcommit --uninstall`. + +### Install Lefthook + +1. Install the `lefthook` Ruby gem: + + ```shell + bundle install + ``` + +1. Install Lefthook managed Git hooks: + + ```shell + bundle exec lefthook install + ``` + +1. Test Lefthook is working by running the Lefthook `prepare-commit-msg` Git hook: + + ```shell + bundle exec lefthook run prepare-commit-msg + ``` + +This should return a fully qualified path command with no other output. + +### Lefthook configuration -# If using rbenv, at this point you may need to do: rbenv rehash +The current Lefthook configuration can be found in [`lefthook.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lefthook.yml). -# 2. Install lefthook... +Before you push your changes, Lefthook automatically runs the following checks: -## With Homebrew (macOS) -brew install Arkweid/lefthook/lefthook +- Danger: Runs a subset of checks that `danger-review` runs on your merge requests. +- ES lint: Run `yarn eslint` checks (with the [`.eslintrc.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.eslintrc.yml) configuration) on the modified `*.{js,vue}` files. Tags: `frontend`, `style`. +- HAML lint: Run `bundle exec haml-lint` checks (with the [`.haml-lint.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.haml-lint.yml) configuration) on the modified `*.html.haml` files. Tags: `view`, `haml`, `style`. +- Markdown lint: Run `yarn markdownlint` checks on the modified `*.md` files. Tags: `documentation`, `style`. +- SCSS lint: Run `bundle exec scss-lint` checks (with the [`.scss-lint.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.scss-lint.yml) configuration) on the modified `*.scss{,.css}` files. Tags: `stylesheet`, `css`, `style`. +- RuboCop: Run `bundle exec rubocop` checks (with the [`.rubocop.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.rubocop.yml) configuration) on the modified `*.rb` files. Tags: `backend`, `style`. +- Vale: Run `vale` checks (with the [`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.vale.ini) configuration) on the modified `*.md` files. Tags: `documentation`, `style`. -## Or with Go -go get github.com/Arkweid/lefthook +In addition to the default configuration, you can define a [local configuration](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#local-config). -## Or with Rubygems -gem install lefthook +### Disable Lefthook temporarily -### You may need to run the following if you're using rbenv -rbenv rehash +To disable Lefthook temporarily, you can set the `LEFTHOOK` environment variable to `0`. For instance: -# 3. Install the Git hooks -lefthook install -f +```shell +LEFTHOOK=0 git push ... ``` -Before you push your changes, Lefthook then automatically run Danger checks, and other checks -for changed files. This saves you time as you don't have to wait for the same errors to be detected -by CI/CD. +### Run Lefthook hooks manually + +To run the `pre-push` Git hook, run: + +```shell +bundle exec lefthook run pre-push +``` -Lefthook relies on a pre-push hook to prevent commits that violate its ruleset. -To override this behavior, pass the environment variable `LEFTHOOK=0`. That is, -`LEFTHOOK=0 git push`. +For more information, check out [Lefthook documentation](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#run-githook-group-directly). -You can also: +### Skip Lefthook checks per tag -- Define [local configuration](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#local-config). -- Skip [checks per tag on the fly](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#skip-some-tags-on-the-fly). - For example, `LEFTHOOK_EXCLUDE=frontend git push origin`. -- Run [hooks manually](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#run-githook-group-directly). - For example, `lefthook run pre-push`. +To skip some checks based on tags when pushing, you can set the `LEFTHOOK_EXCLUDE` environment variable. For instance: + +```shell +LEFTHOOK_EXCLUDE=frontend,documentation git push ... +``` + +For more information, check out [Lefthook documentation](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#skip-some-tags-on-the-fly). ## Ruby, Rails, RSpec @@ -111,7 +142,7 @@ This ensures that our list isn't mistakenly removed by another auto generation o the `.rubocop_todo.yml`. This also allows us greater visibility into the exceptions which are currently being resolved. -One way to generate the initial list is to run the todo auto generation, +One way to generate the initial list is to run the `todo` auto generation, with `exclude limit` set to a high number. ```shell @@ -149,8 +180,12 @@ See the dedicated [Shell scripting standards and style guidelines](../shell_scri ## Markdown +<!-- vale gitlab.Spelling = NO --> + We're following [Ciro Santilli's Markdown Style Guide](https://cirosantilli.com/markdown-style-guide/). +<!-- vale gitlab.Spelling = YES --> + ## Documentation See the dedicated [Documentation Style Guide](../documentation/styleguide/index.md). diff --git a/doc/development/cycle_analytics.md b/doc/development/cycle_analytics.md deleted file mode 100644 index 1619f3dcb10..00000000000 --- a/doc/development/cycle_analytics.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'value_stream_analytics.md' ---- - -This document was moved to [another location](value_stream_analytics.md) - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 59b31437161..413c0a31eec 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -105,9 +105,9 @@ minimize the number of lines of code in `danger/`. A non-trivial `Dangerfile` should mostly call plugin code with arguments derived from the methods provided by Danger. The plugin code itself should have unit tests. -At present, we do this by putting the code in a module in `lib/gitlab/danger/...`, +At present, we do this by putting the code in a module in `tooling/danger/...`, and including it in the matching `danger/plugins/...` file. Specs can then be -added in `spec/lib/gitlab/danger/...`. +added in `spec/tooling/danger/...`. To determine if your `Dangerfile` works, push the branch that contains it to GitLab. This can be quite frustrating, as it significantly increases the cycle diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index 26083183d6d..a242a3d5fd0 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -50,17 +50,17 @@ European/US and APAC friendly hours. You can join the office hours call and brin that require a more in-depth discussion between the database reviewers and maintainers: - [Database Office Hours Agenda](https://docs.google.com/document/d/1wgfmVL30F8SdMg-9yY6Y8djPSxWNvKmhR5XmsvYX1EI/edit). -- [Youtube playlist with past recordings](https://www.youtube.com/playlist?list=PL05JrBw4t0Kp-kqXeiF7fF7cFYaKtdqXM). +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [YouTube playlist with past recordings](https://www.youtube.com/playlist?list=PL05JrBw4t0Kp-kqXeiF7fF7cFYaKtdqXM). You should also join the [#database-labs](../understanding_explain_plans.md#database-lab) -Slack channel and get familiar with how to use Joe, the slackbot that provides developers +Slack channel and get familiar with how to use Joe, the Slackbot that provides developers with their own clone of the production database. Understanding and efficiently using `EXPLAIN` plans is at the core of the database review process. The following guides provide a quick introduction and links to follow on more advanced topics: - Guide on [understanding EXPLAIN plans](../understanding_explain_plans.md). -- [Explaining the unexplainable series in depesz](https://www.depesz.com/tag/unexplainable/). +- [Explaining the unexplainable series in `depesz`](https://www.depesz.com/tag/unexplainable/). Finally, you can find various guides in the [Database guides](index.md) page that cover more specific topics and use cases. The most frequently required during database reviewing are the following: diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index 33a0fd2ebb7..f338520c6ca 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -34,6 +34,12 @@ but only for updating the declaration of the columns. We can then validate it at `VALIDATE CONSTRAINT`, which requires only a `SHARE UPDATE EXCLUSIVE LOCK` (only conflicts with other validations and index creation while it allows reads and writes). +### Exceptions + +Text columns used by `attr_encrypted` are not required to have a limit, because the length of the +text after encryption may be longer than the text itself. Instead, you can use an Active Record +length validation on the attribute. + ## Create a new table with text columns When adding a new table, the limits for all text columns should be added in the same migration as diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md index 358b9bb42b0..207d5a733ce 100644 --- a/doc/development/database/table_partitioning.md +++ b/doc/development/database/table_partitioning.md @@ -60,7 +60,7 @@ was the first table to be partitioned in the application database (scheduled for deployment with the GitLab 13.5 release). This table tracks audit entries of security events that happen in the application. In almost all cases, users want to see audit activity that -occurs in a certain timeframe. As a result, date-range partitioning +occurs in a certain time frame. As a result, date-range partitioning was a natural fit for how the data would be accessed. To look at this in more detail, imagine a simplified `audit_events` schema: diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index f4558189000..dc64b59018b 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -69,16 +69,14 @@ bundle exec rails db -e development Use these instructions for exploring the GitLab database while developing with the GDK: 1. Install or open [Visual Studio Code](https://code.visualstudio.com/download). -1. Install the [PostgreSQL VSCode Extension](https://marketplace.visualstudio.com/items?itemName=ckolkman.vscode-postgres) by Chris Kolkman. +1. Install the [PostgreSQL VSCode Extension](https://marketplace.visualstudio.com/items?itemName=ckolkman.vscode-postgres). 1. In Visual Studio Code click on the PostgreSQL Explorer button in the left toolbar. 1. In the top bar of the new window, click on the `+` to **Add Database Connection**, and follow the prompts to fill in the details: 1. **Hostname**: the path to the PostgreSQL folder in your GDK directory (for example `/dev/gitlab-development-kit/postgresql`). 1. **PostgreSQL user to authenticate as**: usually your local username, unless otherwise specified during PostgreSQL installation. 1. **Password of the PostgreSQL user**: the password you set when installing PostgreSQL. 1. **Port number to connect to**: `5432` (default). - 1. <!-- vale gitlab.Spelling = NO --> - **Use an ssl connection?** - <!-- vale gitlab.Spelling = YES --> This depends on your installation. Options are: + 1. **Use an SSL connection?** This depends on your installation. Options are: - **Use Secure Connection** - **Standard Connection** (default) 1. **(Optional) The database to connect to**: `gitlabhq_development`. diff --git a/doc/development/database_review.md b/doc/development/database_review.md index da2c93cc1fd..cc4acba100b 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -147,7 +147,7 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac - Provide the link to the plan at: [explain.depesz.com](https://explain.depesz.com). Paste both the plan and the query used in the form. - When providing query plans, make sure it hits enough data: - You can use a GitLab production replica to test your queries on a large scale, - through the `#database-lab` Slack channel or through [chatops](understanding_explain_plans.md#chatops). + through the `#database-lab` Slack channel or through [ChatOps](understanding_explain_plans.md#chatops). - Usually, the `gitlab-org` namespace (`namespace_id = 9970`) and the `gitlab-org/gitlab-foss` (`project_id = 13083`) or the `gitlab-org/gitlab` (`project_id = 278964`) projects provide enough data to serve as a good example. @@ -220,13 +220,13 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac - Check for any obviously complex queries and queries the author specifically points out for review (if any) - If not present yet, ask the author to provide SQL queries and query plans - (for example, by using [chatops](understanding_explain_plans.md#chatops) or direct + (for example, by using [ChatOps](understanding_explain_plans.md#chatops) or direct database access) - For given queries, review parameters regarding data distribution - [Check query plans](understanding_explain_plans.md) and suggest improvements to queries (changing the query, schema or adding indexes and similar) - General guideline is for queries to come in below [100ms execution time](query_performance.md#timing-guidelines-for-queries) - - Avoid N+1 problems and minimalize the [query count](merge_request_performance_guidelines.md#query-counts). + - Avoid N+1 problems and minimize the [query count](merge_request_performance_guidelines.md#query-counts). ### Timing guidelines for migrations diff --git a/doc/development/diffs.md b/doc/development/diffs.md index fba8eda0408..52ba89a4d6e 100644 --- a/doc/development/diffs.md +++ b/doc/development/diffs.md @@ -14,12 +14,20 @@ We rely on different sources to present diffs. These include: ## Deep Dive +<!-- vale gitlab.Spelling = NO --> + In January 2019, Oswaldo Ferreira hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on GitLab Diffs and Commenting on Diffs -functionality to share his domain specific knowledge with anyone who may work in this part of the -codebase in the future. You can find the [recording on YouTube](https://www.youtube.com/watch?v=K6G3gMcFyek), -and the slides on [Google Slides](https://docs.google.com/presentation/d/1bGutFH2AT3bxOPZuLMGl1ANWHqFnrxwQwjiwAZkF-TU/edit) -and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/b5ad2f336e0afcfe0f99db0af0ccc71a/). +functionality to share domain-specific knowledge with anyone who may work in this part of the +codebase in the future: + +<!-- vale gitlab.Spelling = YES --> + +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> + [Recording on YouTube](https://www.youtube.com/watch?v=K6G3gMcFyek) +- Slides on [Google Slides](https://docs.google.com/presentation/d/1bGutFH2AT3bxOPZuLMGl1ANWHqFnrxwQwjiwAZkF-TU/edit) +- [PDF slides](https://gitlab.com/gitlab-org/create-stage/uploads/b5ad2f336e0afcfe0f99db0af0ccc71a/) + Everything covered in this deep dive was accurate as of GitLab 11.7, and while specific details may have changed since then, it should still serve as a good introduction. @@ -125,7 +133,7 @@ Gitaly only returns `Diff.Collapsed` (RPC) when surpassing collection limits. #### Not expandable patches (too large) The patch not be rendered if it's larger than `ApplicationSettings#diff_max_patch_bytes`. -Users see a `This source diff could not be displayed because it is too large` message. +Users see a `Changes are too large to be shown.` message and a button to view only that file in that commit. ```ruby Commit::DIFF_SAFE_LINES = Gitlab::Git::DiffCollection::DEFAULT_LIMITS[:max_lines] = 5000 @@ -180,8 +188,8 @@ has been introduced. One of the key challenges to deal with when working on merge ref diffs are merge conflicts. If the target and source branch contains a merge conflict, the branches -cannot be automatically merged. The [recording on -YouTube](https://www.youtube.com/watch?v=GFXIFA4ZuZw&feature=youtu.be&ab_channel=GitLabUnfiltered) +cannot be automatically merged. The +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=GFXIFA4ZuZw&feature=youtu.be&ab_channel=GitLabUnfiltered) is a quick introduction to the problem and the motivation behind the [epic](https://gitlab.com/groups/gitlab-org/-/epics/854). In 13.5 a solution for both-modified merge diff --git a/doc/development/directory_structure.md b/doc/development/directory_structure.md new file mode 100644 index 00000000000..c2329feb941 --- /dev/null +++ b/doc/development/directory_structure.md @@ -0,0 +1,36 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Backend directory structure + +## Use namespaces to define bounded contexts + +A healthy application is divided into macro and sub components that represent the contexts at play, +whether they are related to business domain or infrastructure code. + +As GitLab code has so many features and components it's hard to see what contexts are involved. +We should expect any class to be defined inside a module/namespace that represents the contexts where it operates. + +When we namespace classes inside their domain: + +- Similar terminology becomes unambiguous as the domain clarifies the meaning: + For example, `MergeRequests::Diff` and `Notes::Diff`. +- Top-level namespaces could be associated to one or more groups identified as domain experts. +- We can better identify the interactions and coupling between components. + For example, several classes inside `MergeRequests::` domain interact more with `Ci::` + domain and less with `ImportExport::`. + +```ruby +# bad +class MyClass +end + +# good +module MyDomain + class MyClass + end +end +``` diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index 9228609aae9..eb20e721e46 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Distributed Tracing - development guidelines +# Distributed Tracing - development guidelines **(FREE)** GitLab is instrumented for distributed tracing. Distributed Tracing in GitLab is currently considered **experimental**, as it has not yet been tested at scale on GitLab.com. diff --git a/doc/development/doc_styleguide.md b/doc/development/doc_styleguide.md deleted file mode 100644 index 1595a841ade..00000000000 --- a/doc/development/doc_styleguide.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'documentation/styleguide.md' ---- - -This document was moved to [another location](documentation/styleguide.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/documentation/feature-change-workflow.md b/doc/development/documentation/feature-change-workflow.md deleted file mode 100644 index 78e5510ffca..00000000000 --- a/doc/development/documentation/feature-change-workflow.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'workflow.md' ---- - -This document was moved to [another location](workflow.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 7547ec59fb2..6aa1aed0236 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -30,7 +30,7 @@ See how to document them below, according to the state of the flag: - [Features that can be enabled or disabled for a single project](#features-enabled-by-project). - [Features with the feature flag removed](#features-with-flag-removed). -The [`**(CORE ONLY)**`](styleguide/index.md#product-tier-badges) badge or equivalent for +The [`**(FREE SELF)**`](styleguide/index.md#product-tier-badges) badge or equivalent for the feature's tier should be added to the line and heading that refers to enabling/disabling feature flags as Admin access is required to do so, therefore, it indicates that it cannot be done by regular users of GitLab.com. @@ -61,7 +61,7 @@ be enabled for a single project, and is not ready for production use: > - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), disabled by default. > - It's disabled on GitLab.com. > - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#anchor-to-section). **(CORE ONLY)** +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#anchor-to-section). **(FREE SELF)** WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -70,7 +70,7 @@ This feature might not be available to you. Check the **version history** note a <!-- Add this at the end of the file --> -### Enable or disable <Feature Name> **(CORE ONLY)** +### Enable or disable <Feature Name> **(FREE SELF)** <Feature Name> is under development and not ready for production use. It is deployed behind a feature flag that is **disabled by default**. @@ -91,7 +91,7 @@ Feature.disable(:<feature flag>) ```` Adjust the blurb according to the state of the feature you're documenting. -Replace `<Feature name>`, `**(CORE ONLY)**`, `<feature flag>`, and +Replace `<Feature name>`, `**(FREE SELF)**`, `<feature flag>`, and `<replace with path to>`, and `#anchor-to-section` accordingly. ### Features that became enabled by default @@ -120,7 +120,7 @@ use: > - [Became enabled by default](link-to-issue) on GitLab 12.1. > - It's enabled on GitLab.com. > - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(CORE ONLY)** +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(FREE SELF)** WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -129,7 +129,7 @@ This feature might not be available to you. Check the **version history** note a <!-- Add this at the end of the file --> -### Enable or disable <Feature Name> **(CORE ONLY)** +### Enable or disable <Feature Name> **(FREE SELF)** <Feature Name> is under development but ready for production use. It is deployed behind a feature flag that is **enabled by default**. @@ -150,7 +150,7 @@ Feature.disable(:<feature flag>) ```` Adjust the blurb according to the state of the feature you're documenting. -Replace `<Feature name>`, `**(CORE ONLY)**`, `<feature flag>`, +Replace `<Feature name>`, `**(FREE SELF)**`, `<feature flag>`, `<replace with path to>`, and `#anchor-to-section` accordingly. ### Features directly enabled by default @@ -176,7 +176,7 @@ cannot be enabled for a single project, and is ready for production use: > - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. > - It's enabled on GitLab.com. > - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(CORE ONLY)** +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(FREE SELF)** WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -185,7 +185,7 @@ This feature might not be available to you. Check the **version history** note a <!-- Add this at the end of the file --> -### Enable or disable <Feature Name> **(CORE ONLY)** +### Enable or disable <Feature Name> **(FREE SELF)** <Feature Name> is under development but ready for production use. It is deployed behind a feature flag that is **enabled by default**. @@ -206,7 +206,7 @@ Feature.disable(:<feature flag>) ```` Adjust the blurb according to the state of the feature you're documenting. -Replace `<Feature name>`, `**(CORE ONLY)**`, `<feature flag>`, +Replace `<Feature name>`, `**(FREE SELF)**`, `<feature flag>`, `<replace with path to>`, and `#anchor-to-section` accordingly. ### Features enabled by project @@ -249,7 +249,7 @@ be enabled by project, and is ready for production use: > - It's enabled on GitLab.com. > - It can be enabled or disabled for a single project. > - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(CORE ONLY)** +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(FREE SELF)** WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -258,7 +258,7 @@ This feature might not be available to you. Check the **version history** note a <!-- Add this at the end of the file --> -### Enable or disable <Feature Name> **(CORE ONLY)** +### Enable or disable <Feature Name> **(FREE SELF)** <Feature Name> is under development but ready for production use. It is deployed behind a feature flag that is **enabled by default**. @@ -285,7 +285,7 @@ Feature.disable(:<feature flag>, Project.find(<project id>)) ```` Adjust the blurb according to the state of the feature you're documenting. -Replace `<Feature name>`, `**(CORE ONLY)**`, `<feature flag>`, +Replace `<Feature name>`, `**(FREE SELF)**`, `<feature flag>`, `<replace with path to>`, and `#anchor-to-section` accordingly. ### Features with flag removed diff --git a/doc/development/documentation/improvement-workflow.md b/doc/development/documentation/improvement-workflow.md deleted file mode 100644 index 78e5510ffca..00000000000 --- a/doc/development/documentation/improvement-workflow.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'workflow.md' ---- - -This document was moved to [another location](workflow.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 55f5d43b175..2d865ec3a7e 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -12,7 +12,7 @@ The GitLab documentation is [intended as the single source of truth (SSOT)](http In addition to this page, the following resources can help you craft and contribute to documentation: - [Style Guide](styleguide/index.md) - What belongs in the docs, language guidelines, Markdown standards to follow, links, and more. -- [Structure and template](structure.md) - Learn the typical parts of a doc page and how to write each one. +- [Topic type template](structure.md) - Learn about the different types of topics. - [Documentation process](workflow.md). - [Markdown Guide](../../user/markdown.md) - A reference for all Markdown syntax supported by GitLab. - [Site architecture](site_architecture/index.md) - How <https://docs.gitlab.com> is built. @@ -514,7 +514,7 @@ To run the tool on an existing screenshot generator, take the following steps: 1. Set up the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/gitlab_docs.md). 1. Navigate to the subdirectory with your cloned GitLab repository, typically `gdk/gitlab`. 1. Make sure that your GDK database is fully migrated: `bin/rake db:migrate RAILS_ENV=development`. -1. Install pngquant, see the tool website for more information: [`pngquant`](https://pngquant.org/) +1. Install `pngquant`, see the tool website for more information: [`pngquant`](https://pngquant.org/) 1. Run `scripts/docs_screenshots.rb spec/docs_screenshots/<name_of_screenshot_generator>.rb <milestone-version>`. 1. Identify the location of the screenshots, based on the `gitlab/doc` location defined by the `it` parameter in your script. 1. Commit the newly created screenshots. diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index c8c48e71b48..e05f6760ff1 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -165,7 +165,7 @@ curl --request POST \ ### Post data using form-data -Instead of using JSON or urlencode you can use multipart/form-data which +Instead of using JSON or URL-encoding data, you can use `multipart/form-data` which properly handles data encoding: ```shell diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index fcf4662502f..f66b0543ad1 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -283,8 +283,8 @@ and the following syntax rules. - As convention, always wrap URLs in single quotes `'url'`. - Always use relative paths against the home of CE and EE. Examples: - For `https://docs.gitlab.com/ee/README.html`, the relative URL is `README.html`. - - For `https://docs.gitlab.com/ee/user/project/cycle_analytics.html`, the relative - URL is `user/project/cycle_analytics.html`. + - For `https://docs.gitlab.com/ee/user/analytics/value_stream_analytics.md`, the relative + URL is `user/analytics/value_stream_analytics.html`. - For `README.html` files, add the complete path `path/to/README.html`. - For `index.html` files, use the clean (canonical) URL: `path/to/`. - For EE-only docs, use the same relative path, but add the attribute `ee_only: true` below @@ -328,7 +328,7 @@ There are three main considerations on the logic built for the nav: - `https://docs.gitlab.com/*` - [EE-only](#ee-only-docs): documentation only available in `/ee/`, not on `/ce/`, e.g.: - `https://docs.gitlab.com/ee/user/group/epics/` - - `https://docs.gitlab.com/ee/user/project/security_dashboard.html` + - `https://docs.gitlab.com/ee/user/application_security/security_dashboard/index.html` - [Default URL](#default-url): between CE and EE docs, the default is `ee`, therefore, all docs should link to `/ee/` unless if on `/ce/` linking internally to `ce`. diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index be25a083948..92fd17f9d3e 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -114,7 +114,7 @@ located in the [Dockerfiles directory](https://gitlab.com/gitlab-org/gitlab-docs If you need to rebuild the Docker images immediately (must have maintainer level permissions): WARNING: -If you change the dockerfile configuration and rebuild the images, you can break the master +If you change the Dockerfile configuration and rebuild the images, you can break the master pipeline in the main `gitlab` repository as well as in `gitlab-docs`. Create an image with a different name first and test it to ensure you do not break the pipelines. diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index fd0c29f0fc1..65949d5b5f5 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -5,167 +5,165 @@ info: To determine the technical writer assigned to the Stage/Group associated w description: What to include in GitLab documentation pages. --- -# Documentation structure and template +# Documentation topic types -Use these standards to contribute content to the GitLab documentation. +At GitLab, we have not traditionally used topic types. However, we are starting to +move in this direction, and we now use four topic types: -Before getting started, familiarize yourself with [Documentation guidelines for GitLab](index.md) -and the [Documentation Style Guide](styleguide/index.md). +- [Concept](#concept) +- [Task](#task) +- [Reference](#reference) +- [Troubleshooting](#troubleshooting) -## Components of a documentation page +Each page contains multiple topic types. For example, +a page with the title `Pipelines`, which is generated from a file called `index.md`, +can include a concept and multiple task and reference topics. -Most pages are dedicated to a specific GitLab feature or to a use case that -involves one or more features, potentially in conjunction with third-party tools. +GitLab also uses high-level landing pages. -In general, each topic should include the following content, in this sequence: +## Landing pages -- *Metadata*: Information about the stage, group, and how to find the technical - writer for the topic. This information isn't visible in the published help. -- *Title*: A top-level heading with the feature or use case name. Choose a term - that defines the functionality and use the same term in all the resources - where the feature is mentioned. -- *Introduction*: In a few sentences beneath the title, describe what the - feature or topic is, what it does, and in what context it should be used. -- *Use cases*: Describe real user scenarios. -- *Prerequisites*: Describe the software, configuration, account, permissions, - or knowledge required to use this functionality. -- *Tasks*: Present detailed step-by-step instructions on how to use the feature. -- *Troubleshooting*: List errors and how to address them. Recommended but not - required. +Landing pages are topics that group other topics and help a user to navigate a section. -You can include additional subsections, as appropriate, such as *How it Works*, -or *Architecture*. You can also include other logical divisions, such as -pre-deployment and post-deployment tasks. +Users who are using the in-product help do not have a left nav, +and need these topics to navigate the documentation. -## Template for new docs +These topics can also help other users find the most important topics +in a section. -Follow the [folder structure and filename guidelines](styleguide/index.md#folder-structure-overview) -and create a new topic by using this template: +Landing page topics should be in this format: ```markdown -<!--Follow the Style Guide when working on this document. -https://docs.gitlab.com/ee/development/documentation/styleguide.html -When done, remove all of this commented-out text, except a commented-out -Troubleshooting section, which, if empty, can be left in place to encourage future use.--> ---- -description: "Short document description." # Up to ~200 chars long. This information is displayed -in Google Search snippets. It may help to write the page intro first, and then reuse it here. -stage: Add the stage name here -group: Add the group name here -info: To determine the technical writer assigned to the Stage/Group associated with this page, -see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments ---- +# Title (a noun, like "CI/CD or "Analytics") -# Feature or Use Case Name **[TIER]** (1) -<!--If you are writing about a use case, start with a verb, -for example, "Configure", "Implement", + the goal/scenario--> +Brief introduction to the concept or product area. +Include the reason why someone would use this thing. + +- Bulleted list of important related topics. +- These links are needed because users of in-product help do not have left navigation. +``` -<!--For pages on newly-introduced features, add the following line. -If only some aspects of the feature have been introduced, specify which parts of the feature.--> -> [Introduced](link_to_issue_or_mr) in GitLab (Tier) X.Y (2). +## Concept -Write a description of the feature or use case. This introduction should answer -these questions: +A concept topic introduces a single feature or concept. -- What is this feature or use case? -- Who is it for? -- What is the context in which it is used and are there any prerequisites or - requirements? -- What can the audience do with this? (Be sure to consider all applicable - audiences, such as GitLab admin and developer-user.) -- What are the benefits of using this over any existing alternatives? +A concept should answer the questions: -You can reuse this content, or part of it, for the front matter's `description` -at the top of this file. +- What is this? +- Why would I use it? -## Use cases +Think of everything someone might want to know if they’ve never heard of this topic before. -Describe common use cases, typically in bulleted form. Include real-life examples -for each. +Don’t tell them **how** to do this thing. Tell them **what it is**. -If the page itself is dedicated to a use case, this section usually includes more -specific scenarios for use (for example, variations on the main use case), but if -that's not applicable, you can omit this section. +If you start describing another topic, start a new concept and link to it. + +Concept topics should be in this format: + +```markdown +# Title (a noun, like "Widgets") -Examples of use cases on feature pages: +A paragraph that explains what this thing is. -- CE and EE: [Issues](../../user/project/issues/index.md#use-cases) -- CE and EE: [Merge Requests](../../user/project/merge_requests/index.md) -- EE-only: [Geo](../../administration/geo/index.md) -- EE-only: [Jenkins integration](../../integration/jenkins.md) +Another paragraph that explains what this thing is. -## Prerequisites +Remember, if you start to describe about another concept, stop yourself. +Each concept topic should be about one concept only. +``` -State any prerequisites for using the feature. These might include: +## Task -- Technical prereqs (for example, an account on a third-party service, an amount - of storage space, or prior configuration of another feature) -- Prerequisite knowledge (for example, familiarity with certain GitLab features - or other products and technologies). +A task topic gives instructions for how to complete a procedure. -Link each one to an appropriate place for more information. +Task topics should be in this format: -## Tasks +```markdown +# Title (starts with an active verb, like "Create a widget" or "Delete a widget") -Each topic should help users accomplish a specific task. +Do this task when you want to... -The heading should: +Prerequisites (optional): -- Describe the task and start with a verb. For example, `Create a package` or - `Configure a pipeline`. -- Be short and descriptive (up to ~50 chars). -- Start from an `h2` (`##`), then go over `h3`, `h4`, `h5`, and `h6` as needed. - Never skip a hierarchy level (like `h2` > `h4`). It breaks the table of - contents and can affect the breadcrumbs. +- Thing 1 +- Thing 2 +- Thing 3 -Bigger tasks can have subsections that explain specific phases of the process. +To do this task: -Include example code or configurations when needed. Use Markdown to wrap code -blocks with [syntax highlighting](../../user/markdown.md#colored-code-and-syntax-highlighting). +1. Location then action. (Go to this menu, then select this item.) +1. Another step. +1. Another step. -Example topic: +Task result (optional). Next steps (optional). +``` -## Create a teddy bear +Here is an example. -Create a teddy bear when you need something to hug. (Include the reason why you -might do the task.) +```markdown +# Create an issue -To create a teddy bear: +Create an issue when you want to track bugs or future work. -1. Go to **Settings > CI/CD**. -1. Expand **This** and click **This**. -1. Do another step. +Prerequisites: -The teddy bear is now in the kitchen, in the cupboard above the sink. _(This is the result.)_ +- A minimum of Contributor access to a project in GitLab. -You can retrieve the teddy bear and put it on the couch with the other animals. _(These are next steps.)_ +To create an issue: -Screenshots are not necessary. They are difficult to keep up-to-date and can -clutter the page. +1. Go to **Issues > List**. +1. In the top right, click **New issue**. +1. Complete the fields. (If you have a reference topic that lists each field, link to it here.) +1. Click **Submit issue**. -<!-- ## Troubleshooting +The issue is created. You can view it by going to **Issues > List**. +``` -Include any troubleshooting steps that you can foresee. If you know beforehand -what issues one might have when setting this up, or when something is changed, -or on upgrading, it's important to describe those, too. Think of things that may -go wrong and include them here. This is important to minimize requests for -Support, and to avoid documentation comments with questions that you know -someone might ask. +## Reference -Each scenario can be a third-level heading, for example, `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place but -commented out to help encourage others to add to it in the future. --> +A reference topic provides information in an easily-scannable format, +like a table or list. It's similar to a dictionary or encyclopedia entry. ---- +```markdown +# Title (a noun, like "Pipeline settings" or "Administrator options") -Notes: +Introductory sentence. -- (1): Apply the [tier badges](styleguide/index.md#product-badges) accordingly. -- (2): Apply the correct format for the - [GitLab version that introduces the feature](styleguide/index.md#gitlab-versions-and-tiers). +| Setting | Description | +|---------|-------------| +| **Name** | Descriptive sentence about the setting. | ``` -## Help and feedback section +## Troubleshooting + +Troubleshooting topics can be one of two categories: + +- **Troubleshooting task.** This topic is written the same as a [standard task topic](#task). + For example, "Run debug tools" or "Verify syntax." +- **Troubleshooting reference.** This topic has a specific format. + +Troubleshooting reference topics should be in this format: + +```markdown +# Title (the error message or a description of it) + +You might get an error that states <error message>. + +This issue occurs when... + +The workaround is... +``` + +## Other information on a topic + +Topics include other information. + +For example: + +- Each topic must have a [tier badge](styleguide/index.md#product-tier-badges). +- New topics must have information about the + [GitLab version where the feature was introduced](styleguide/index.md#where-to-put-version-text). + +### Help and feedback section This section ([introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/319) in GitLab 11.4) is displayed at the end of each document and can be omitted by adding a key into @@ -180,7 +178,7 @@ feedback: false The default is to leave it there. If you want to omit it from a document, you must check with a technical writer before doing so. -### Disqus +#### Disqus We also have integrated the docs site with Disqus (introduced by [!151](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/151)), @@ -206,7 +204,7 @@ The click events in the feedback section are tracked with Google Tag Manager. The conversions can be viewed on Google Analytics by navigating to **Behavior > Events > Top events > docs**. -## Guidelines for good practices +### Guidelines for good practices > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36576/) in GitLab 13.2 as GitLab Development documentation. diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index a12c2740083..ade0b89a92c 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -3,3 +3,6 @@ redirect_to: 'styleguide/index.md' --- This document was moved to [another location](styleguide/index.md). + +<!-- This redirect file can be deleted after <2022-02-13>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index bba94c7de7e..baee01b8245 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -7,34 +7,25 @@ description: 'Writing styles, markup, formatting, and other standards for GitLab # Documentation Style Guide -This document defines the standards for GitLab documentation. +This document defines the standards for GitLab documentation, including grammar, formatting, word use, and more. -For broader information about the documentation, see the [Documentation guidelines](../index.md). +For style questions, mention `@tw-style` in an issue or merge request. If you have access to the GitLab Slack workspace, +use the `#docs-processes` channel. -For guidelines specific to text in the GitLab interface, see the Pajamas [Content](https://design.gitlab.com/content/error-messages/) section. +In addition to this page, the following resources can help you craft and contribute to documentation: -For information on how to validate styles locally or by using GitLab CI/CD, see [Testing](../testing.md). - -Use this guide for standards on grammar, formatting, word usage, and more. - -You can also view a list of [recent updates to this guide](https://gitlab.com/dashboard/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=tw-style¬[label_name][]=docs%3A%3Afix). - -If you can't find what you need: - -- View the GitLab Handbook for [writing style guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines) that apply to all GitLab content. -- Refer to: - - - [Microsoft Style Guide](https://docs.microsoft.com/en-us/style-guide/welcome/). - - [Google Developer Documentation Style Guide](https://developers.google.com/style). - -If you have questions about style, mention `@tw-style` in an issue or merge request, or, if you have access to the GitLab Slack workspace, use `#docs-process`. +- [Doc contribution guidelines](../index.md) +- [Doc style and consistency testing](../testing.md) +- [UI text guidelines](https://design.gitlab.com/content/error-messages/) +- [GitLab Handbook style guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines) +- [Microsoft Style Guide](https://docs.microsoft.com/en-us/style-guide/welcome/) +- [Google Developer Documentation Style Guide](https://developers.google.com/style) +- [Recent updates to this guide](https://gitlab.com/dashboard/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=tw-style¬[label_name][]=docs%3A%3Afix) ## Documentation is the single source of truth (SSOT) -### Why a single source of truth - -The documentation of GitLab products and features is the SSOT for all -information related to implementation, usage, and troubleshooting. It evolves +The GitLab documentation is the SSOT for all +information related to GitLab implementation, usage, and troubleshooting. It evolves continuously, in keeping with new products and features, and with improvements for clarity, accuracy, and completeness. @@ -44,7 +35,7 @@ about GitLab products. It also informs decisions about the kinds of content we include in our documentation. -### All information +### The documentation includes all information Include problem-solving actions that may address rare cases or be considered _risky_, but provide proper context through fully-detailed @@ -54,10 +45,13 @@ If you think you have found an exception to this rule, contact the Technical Writing team. GitLab adds all troubleshooting information to the documentation, no matter how -unlikely a user is to encounter a situation. For the [Troubleshooting sections](#troubleshooting), -people in GitLab Support can merge additions themselves. +unlikely a user is to encounter a situation. -### All media types +GitLab Support maintains their own +[troubleshooting content](../../../administration/index.md#support-team-docs) +in the GitLab documentation. + +### The documentation includes all media types Include any media types/sources if the content is relevant to readers. You can freely include or link presentations, diagrams, and videos. No matter who @@ -71,48 +65,33 @@ include it. quotation with the source cited. Typically it is better to either rephrase relevant information in your own words or link out to the other source. -### No special types +### Topic types In the software industry, it is a best practice to organize documentation in -different types. For example, [Divio recommends](https://www.divio.com/blog/documentation/): - -- Tutorials -- How-to guides -- Explanation -- Reference (for example, a glossary) +different types. For example: -At GitLab, we have so many product changes in our monthly releases that we can't -afford to continuously update multiple types of information. If we have multiple -types, the information becomes outdated. Therefore, we have a -[single template](../structure.md) for documentation. +- Concepts +- Tasks +- Reference +- Troubleshooting -GitLab documentation does not distinguish specific document types. We are open to -reconsidering this policy after the documentation has reached a future stage of -maturity and quality. If you are reading this, then despite our continuous -improvement efforts, that point hasn't been reached. +At GitLab, we have not traditionally used topic types. However, we are starting to +move in this direction, so we can address these issues: -### Link instead of summarize +- **Content is hard to find.** Our docs are comprehensive and include a large amount of + useful information. Topic types create repeatable patterns that make our content easier + to scan and parse. +- **Content is often written from the contributor's point of view.** Our docs + are written by contributors. Topic types (tasks specifically) help put + information into a format that is geared toward helping others, rather than + documenting how a feature was implemented. -There is a temptation to summarize the information on another page, which -causes the information to live in two places. Instead, link to the single source -of truth and explain why it is important to consume the information. +GitLab uses these [topic type templates](../structure.md). -### Organize by topic, not by type +### Link instead of repeating text -We organize content by topic, not by type, so it can be located in the -single-source-of-truth (SSOT) section for the subject matter. Top-level audience-type -folders, like `administration`, are exceptions. - -For example, do not create groupings of similar media types. For example: - -- Glossaries. -- FAQs. -- Sets of all articles or videos. - -Such grouping of content by type makes it difficult to browse for the information -you need and difficult to maintain up-to-date content. Instead, organize content -by its subject (for example, everything related to CI goes together) and -cross-link between any related content. +Rather than repeating information from another topic, link to the single source +of truth and explain why it is important. ### Docs-first methodology @@ -127,14 +106,9 @@ of GitLab more efficient. should be to create a merge request (MR) to add this information to the documentation. You can then share the MR to communicate this information. -New information about the future usage or troubleshooting -of GitLab should not be written directly in a forum or other messaging system. -Instead, add it to a documentation merge request, then reference it. Note -that among any other documentation changes, you can either: - -- Add a [Troubleshooting section](#troubleshooting) to a doc if none exists. -- Un-comment and use the placeholder Troubleshooting section included as part of - our [documentation template](../structure.md#template-for-new-docs), if present. +New information that would be useful toward the future usage or troubleshooting +of GitLab should not be written directly in a forum or other messaging system, +but added to a documentation MR and then referenced, as described above. The more we reflexively add information to the documentation, the more the documentation helps others efficiently accomplish tasks and solve problems. @@ -217,8 +191,11 @@ included in backticks. For example: ## Structure -Because we want documentation to be a SSOT, we should [organize by topic, not by -type](#organize-by-topic-not-by-type). +We include concept and task topic types in the same larger topic. + +In general, we have one topic that's a [landing page](../structure.md#landing-pages). +Below that topic in the left nav are individual topics. Each of these include a concept +and multiple related tasks, reference, and troubleshooting topics. ### Folder structure overview @@ -299,7 +276,7 @@ place for it. ### Avoid duplication Do not include the same information in multiple places. -[Link to a single source of truth instead.](#link-instead-of-summarize) +[Link to a single source of truth instead.](#link-instead-of-repeating-text) ### References across documents @@ -389,7 +366,7 @@ by default. Capitalize names of: - GitLab [product tiers](https://about.gitlab.com/pricing/). For example, - GitLab Core and GitLab Ultimate. (Tested in [`BadgeCapitalization.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/BadgeCapitalization.yml).) + GitLab Free and GitLab Ultimate. (Tested in [`BadgeCapitalization.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/BadgeCapitalization.yml).) - Third-party organizations, software, and products. For example, Prometheus, Kubernetes, Git, and The Linux Foundation. - Methods or methodologies. For example, Continuous Integration, @@ -942,7 +919,7 @@ NOTE: [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39717) in GitLab 13.4, [product badges](#product-tier-badges) used in headings aren't included in the generated anchor links. For example, when you link to -`## This is an example **(CORE)**`, use the anchor `#this-is-an-example`. +`## This is an example **(FREE)**`, use the anchor `#this-is-an-example`. Keep in mind that the GitLab user interface links to many documentation pages and anchor links to take the user to the right spot. When you change @@ -966,8 +943,8 @@ this option. ## Links -Links are important in GitLab documentation. They allow you to [link instead of -summarizing](#link-instead-of-summarize) to help preserve a [single source of truth](#why-a-single-source-of-truth) +Links are important in GitLab documentation. Use links instead of +summarizing to help preserve a [single source of truth](#documentation-is-the-single-source-of-truth-ssot) in GitLab documentation. We include guidance for links in these categories: @@ -1020,7 +997,8 @@ To link to internal documentation: - Use relative links to Markdown files in the same repository. - Do not use absolute URLs or URLs from `docs.gitlab.com`. - Use `../` to navigate to higher-level directories. -- Don't prepend `./` to links to files or directories. +- Don't prepend `./` to links to files or directories. To link to a file in the + same directory or one of its sub-directories, use the syntax `path/to/file.md`. - Don't link relative to root. For example, `/ee/user/gitlab_com/index.md`. Don't: @@ -1045,6 +1023,7 @@ To link to internal documentation: - `../../merge_requests/index.md` - `../../issues/tags.md` - `../../issues/tags.md#stages` + - `issues/tags.md` NOTE: Using the Markdown extension is necessary for the [`/help`](../index.md#gitlab-help) @@ -1671,8 +1650,8 @@ the blockquote to use a bulleted list: If a feature is moved to another tier: ```markdown -> - [Moved](<link-to-issue>) from GitLab Premium to GitLab Starter in 11.8. -> - [Moved](<link-to-issue>) from GitLab Starter to GitLab Core in 12.0. +> - [Moved](<link-to-issue>) from GitLab Ultimate to GitLab Premium in 11.8. +> - [Moved](<link-to-issue>) from GitLab Premium to GitLab Free in 12.0. ``` If a feature is deprecated, include a link to a replacement (when available): @@ -1779,7 +1758,7 @@ To add a tier badge to a heading, add the relevant [tier badge](#available-produ after the heading text. For example: ```markdown -# Heading title `**(CORE)**` +# Heading title **(FREE)** ``` #### Product tier badges on other content @@ -1787,30 +1766,26 @@ after the heading text. For example: In paragraphs, list names, and table cells, an information icon displays when you add a tier badge. More verbose information displays when a user points to the icon: -- `**(STARTER)**` displays as **(STARTER)** -- `**(STARTER ONLY)**` displays as **(STARTER ONLY)** -- `**(SILVER ONLY)**` displays as **(SILVER ONLY)** +- `**(FREE)**` displays as **(FREE)** +- `**(FREE SELF)**` displays as **(FREE SELF)** +- `**(FREE SAAS)**` displays as **(FREE SAAS)** -The `**(STARTER)**` generates a `span` element to trigger the -badges and tooltips (`<span class="badge-trigger starter">`). When the keyword -_only_ is added, the corresponding GitLab.com badge isn't displayed. +The `**(FREE)**` generates a `span` element to trigger the +badges and tooltips (`<span class="badge-trigger free">`). #### Available product tier badges -| Tier in which feature is available | Tier badge | -|:-----------------------------------------------------------------------|:----------------------| -| GitLab Core and GitLab.com Free, and their higher tiers | `**(CORE)**` | -| GitLab Starter and GitLab.com Bronze, and their higher tiers | `**(STARTER)**` | -| GitLab Premium and GitLab.com Silver, and their higher tiers | `**(PREMIUM)**` | -| GitLab Ultimate and GitLab.com Gold | `**(ULTIMATE)**` | -| _Only_ GitLab Core and higher tiers (no GitLab.com-based tiers) | `**(CORE ONLY)**` | -| _Only_ GitLab Starter and higher tiers (no GitLab.com-based tiers) | `**(STARTER ONLY)**` | -| _Only_ GitLab Premium and higher tiers (no GitLab.com-based tiers) | `**(PREMIUM ONLY)**` | -| _Only_ GitLab Ultimate (no GitLab.com-based tiers) | `**(ULTIMATE ONLY)**` | -| _Only_ GitLab.com Free and higher tiers (no self-managed instances) | `**(FREE ONLY)**` | -| _Only_ GitLab.com Bronze and higher tiers (no self-managed instances) | `**(BRONZE ONLY)**` | -| _Only_ GitLab.com Silver and higher tiers (no self-managed instances) | `**(SILVER ONLY)**` | -| _Only_ GitLab.com Gold (no self-managed instances) | `**(GOLD ONLY)**` | +| Tier in which feature is available | Tier badge | +|:--------------------------------------------------------------------------|:----------------------| +| GitLab Free self-managed and SaaS, and higher tiers | `**(FREE)**` | +| GitLab Premium self-managed and SaaS, and their higher tiers | `**(PREMIUM)**` | +| GitLab Ultimate self-managed and SaaS | `**(ULTIMATE)**` | +| _Only_ GitLab Free self-managed and higher tiers (no SaaS-based tiers) | `**(FREE SELF)**` | +| _Only_ GitLab Premium self-managed and higher tiers (no SaaS-based tiers) | `**(PREMIUM SELF)**` | +| _Only_ GitLab Ultimate self-managed (no SaaS-based tiers) | `**(ULTIMATE SELF)**` | +| _Only_ GitLab Free SaaS and higher tiers (no self-managed instances) | `**(FREE SAAS)**` | +| _Only_ GitLab Premium SaaS and higher tiers (no self-managed instances) | `**(PREMIUM SAAS)**` | +| _Only_ GitLab Ultimate SaaS (no self-managed instances) | `**(ULTIMATE SAAS)**` | Topics that mention the `gitlab.rb` file are referring to self-managed instances of GitLab. To prevent confusion, include the relevant `TIER ONLY` @@ -1899,21 +1874,6 @@ In this case: - Use the [GitLab Restart](#gitlab-restart) section to explain any required restart or reconfigure of GitLab. -### Troubleshooting - -For troubleshooting sections, provide as much context as possible so -users can identify their problem and resolve it on their own. You -can facilitate this by making sure the troubleshooting content addresses: - -1. The problem the user needs to solve. -1. How the user can confirm they have the problem. -1. Steps the user can take towards resolution of the problem. - -If the contents of each category can be summarized in one line and a list of -steps aren't required, consider setting up a [table](#tables). Create headers of -_Problem_ \| _Cause_ \| _Solution_ (or _Workaround_ if the fix is temporary), or -_Error message_ \| _Solution_. - ## Feature flags Learn how to [document features deployed behind flags](../feature_flags.md). For diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index 561727648f0..f3d6e0a5c71 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -93,8 +93,8 @@ To execute documentation link tests locally: ### UI link tests -The `ui-docs-links lint` job uses `haml-lint` to test that all links to docs from -UI elements (`app/views` files, for example) are linking to valid docs and anchors. +The `ui-docs-links lint` job uses `haml-lint` to test that all documentation links from +UI elements (`app/views` files, for example) are linking to valid pages and anchors. To run the `ui-docs-links` test locally: @@ -156,12 +156,12 @@ markdownlint configuration is found in the following projects: - [`charts`](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/.markdownlint.json) - [`gitlab-development-kit`](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/.markdownlint.json) -This configuration is also used within build pipelines. +This configuration is also used in build pipelines. You can use markdownlint: - [On the command line](https://github.com/igorshubovych/markdownlint-cli#markdownlint-cli--). -- [Within a code editor](#configure-editors). +- [In a code editor](#configure-editors). - [In a `pre-push` hook](#configure-pre-push-hooks). ### Vale @@ -172,10 +172,10 @@ English language. Vale's configuration is stored in the directory of projects. Vale supports creating [custom tests](https://errata-ai.github.io/vale/styles/) that extend any of -several types of checks, which we store in the `.linting/vale/styles/gitlab` directory within the +several types of checks, which we store in the `.linting/vale/styles/gitlab` directory in the documentation directory of projects. -Vale configuration is found in the following projects: +You can find Vale configuration in the following projects: - [`gitlab`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/.vale/gitlab) - [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/master/docs/.vale/gitlab) @@ -183,13 +183,13 @@ Vale configuration is found in the following projects: - [`charts`](https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/doc/.vale/gitlab) - [`gitlab-development-kit`](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/master/doc/.vale/gitlab) -This configuration is also used within build pipelines, where +This configuration is also used in build pipelines, where [error-level rules](#vale-result-types) are enforced. You can use Vale: - [On the command line](https://errata-ai.gitbook.io/vale/getting-started/usage). -- [Within a code editor](#configure-editors). +- [In a code editor](#configure-editors). - [In a Git hook](#configure-pre-push-hooks). Vale only reports errors in the Git hook (the same configuration as the CI/CD pipelines), and does not report suggestions or warnings. @@ -243,32 +243,40 @@ To match the versions of `markdownlint-cli` and `vale` used in the GitLab projec [versions used](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/.gitlab-ci.yml#L447) when building the `image:docs-lint-markdown` Docker image containing these tools for CI/CD. -| Tool | Version | Command | Additional information | -|--------------------|----------|-------------------------------------------|------------------------| -| `markdownlint-cli` | Latest | `yarn global add markdownlint-cli` | n/a | -| `markdownlint-cli` | Specfic | `yarn global add markdownlint-cli@0.23.2` | The `@` indicates a specific version, and this example updates the tool to version `0.23.2`. | -| Vale | Latest | `brew update && brew upgrade vale` | This command is for macOS only. | -| Vale | Specific | n/a | Not possible using `brew`, but can be [directly downloaded](https://github.com/errata-ai/vale/releases). | +| Tool | Version | Command | Additional information | +|--------------------|-----------|-------------------------------------------|------------------------| +| `markdownlint-cli` | Latest | `yarn global add markdownlint-cli` | n/a | +| `markdownlint-cli` | Specific | `yarn global add markdownlint-cli@0.23.2` | The `@` indicates a specific version, and this example updates the tool to version `0.23.2`. | +| Vale | Latest | `brew update && brew upgrade vale` | This command is for macOS only. | +| Vale | Specific | n/a | Not possible using `brew`, but can be [directly downloaded](https://github.com/errata-ai/vale/releases). | ### Configure editors Using linters in your editor is more convenient than having to run the commands from the command line. -To configure markdownlint within your editor, install one of the following as appropriate: +To configure markdownlint in your editor, install one of the following as appropriate: -- [Sublime Text](https://packagecontrol.io/packages/SublimeLinter-contrib-markdownlint) -- [Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint) -- [Atom](https://atom.io/packages/linter-node-markdownlint) -- [Vim](https://github.com/dense-analysis/ale) +- Sublime Text [`SublimeLinter-contrib-markdownlint` package](https://packagecontrol.io/packages/SublimeLinter-contrib-markdownlint). +- Visual Studio Code [`DavidAnson.vscode-markdownlint` extension](https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint). +- Atom [`linter-node-markdownlint` package](https://atom.io/packages/linter-node-markdownlint). +- Vim [ALE plugin](https://github.com/dense-analysis/ale). -To configure Vale within your editor, install one of the following as appropriate: +To configure Vale in your editor, install one of the following as appropriate: -- The Sublime Text [`SublimeLinter-contrib-vale` plugin](https://packagecontrol.io/packages/SublimeLinter-contrib-vale). -- The Visual Studio Code [`errata-ai.vale-server` extension](https://marketplace.visualstudio.com/items?itemName=errata-ai.vale-server). - You don't need Vale Server to use the plugin. You can configure the plugin to +- Sublime Text [`SublimeLinter-contrib-vale` package](https://packagecontrol.io/packages/SublimeLinter-contrib-vale). +- Visual Studio Code [`errata-ai.vale-server` extension](https://marketplace.visualstudio.com/items?itemName=errata-ai.vale-server). + You can configure the plugin to [display only a subset of alerts](#show-subset-of-vale-alerts). -- [Vim](https://github.com/dense-analysis/ale). + + In the extension's settings: + + - Select the **Use CLI** checkbox. + - In the **Config** setting, enter an absolute path to [`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/blob/master/.vale.ini) in one of the cloned GitLab repositories on your computer. + - In the **Path** setting, enter the absolute path to the Vale binary. In most + cases, `vale` should work. To find the location, run `which vale` in a terminal. + +- Vim [ALE plugin](https://github.com/dense-analysis/ale). We don't use [Vale Server](https://errata-ai.github.io/vale/#using-vale-with-a-text-editor-or-another-third-party-application). @@ -286,7 +294,7 @@ Configuration for `lefthook` is available in the [`lefthook.yml`](https://gitlab file for the [`gitlab`](https://gitlab.com/gitlab-org/gitlab) project. To set up `lefthook` for documentation linting, see -[Pre-push static analysis](../contributing/style_guides.md#pre-push-static-analysis). +[Pre-push static analysis](../contributing/style_guides.md#pre-push-static-analysis-with-lefthook). ### Show subset of Vale alerts diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 5be601187ca..81014b7624c 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -92,12 +92,36 @@ class User < ActiveRecord::Base # ... lots of code here ... end -User.prepend_if_ee('EE::User') +User.prepend_ee_mod ``` Do not use methods such as `prepend`, `extend`, and `include`. Instead, use -`prepend_if_ee`, `extend_if_ee`, or `include_if_ee`. These methods take a -_String_ containing the full module name as the argument, not the module itself. +`prepend_ee_mod`, `extend_ee_mod`, or `include_ee_mod`. These methods will try to +find the relevant EE module by the name of the receiver module, for example; + +```ruby +module Vulnerabilities + class Finding + #... + end +end + +Vulnerabilities::Finding.prepend_ee_mod +``` + +will prepend the module named `::EE::Vulnerabilities::Finding`. + +If the extending module does not follow this naming convention, you can also provide the module name +by using `prepend_if_ee`, `extend_if_ee`, or `include_if_ee`. These methods take a +_String_ containing the full module name as the argument, not the module itself, like so; + +```ruby +class User + #... +end + +User.prepend_if_ee('::EE::UserExtension') +``` Since the module would require an `EE` namespace, the file should also be put in an `ee/` sub-directory. For example, we want to extend the user model @@ -142,9 +166,9 @@ still having access the class's implementation with `super`. There are a few gotchas with it: - you should always [`extend ::Gitlab::Utils::Override`](utilities.md#override) and use `override` to - guard the "overrider" method to ensure that if the method gets renamed in + guard the `overrider` method to ensure that if the method gets renamed in CE, the EE override isn't silently forgotten. -- when the "overrider" would add a line in the middle of the CE +- when the `overrider` would add a line in the middle of the CE implementation, you should refactor the CE method and split it in smaller methods. Or create a "hook" method that is empty in CE, and with the EE-specific implementation in EE. @@ -947,7 +971,7 @@ information on managing page-specific JavaScript within EE. #### Child Component only used in EE -To separate Vue template differences we should [async import the components](https://vuejs.org/v2/guide/components-dynamic-async.html#Async-Components). +To separate Vue template differences we should [import the components asynchronously](https://vuejs.org/v2/guide/components-dynamic-async.html#Async-Components). Doing this allows for us to load the correct component in EE while in CE we can load a empty component that renders nothing. This code **should** @@ -1044,7 +1068,7 @@ export default { **For EE components that need different results for the same computed values, we can pass in props to the CE wrapper as seen in the example.** - **EE Child components** - - Since we are using the async loading to check which component to load, we'd still use the component's name, check [this example](#child-component-only-used-in-ee). + - Since we are using the asynchronous loading to check which component to load, we'd still use the component's name, check [this example](#child-component-only-used-in-ee). - **EE extra HTML** - For the templates that have extra HTML in EE we should move it into a new component and use the `ee_else_ce` dynamic import diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 8bf8a5fccb8..10fa331e1dd 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -4,7 +4,7 @@ group: Global Search info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Elasticsearch knowledge **(STARTER ONLY)** +# Elasticsearch knowledge **(PREMIUM SELF)** This area is to maintain a compendium of useful information when working with Elasticsearch. @@ -13,9 +13,9 @@ the [Elasticsearch integration documentation](../integration/elasticsearch.md#en ## Deep Dive -In June 2019, Mario de la Ossa hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on the GitLab [Elasticsearch integration](../integration/elasticsearch.md) to share his domain specific knowledge with anyone who may work in this part of the codebase in the future. You can find the [recording on YouTube](https://www.youtube.com/watch?v=vrvl-tN2EaA), and the slides on [Google Slides](https://docs.google.com/presentation/d/1H-pCzI_LNrgrL5pJAIQgvLX8Ji0-jIKOg1QeJQzChug/edit) and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/c5aa32b6b07476fa8b597004899ec538/Elasticsearch_Deep_Dive.pdf). Everything covered in this deep dive was accurate as of GitLab 12.0, and while specific details may have changed since then, it should still serve as a good introduction. +In June 2019, Mario de la Ossa hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on the GitLab [Elasticsearch integration](../integration/elasticsearch.md) to share his domain specific knowledge with anyone who may work in this part of the codebase in the future. You can find the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=vrvl-tN2EaA), and the slides on [Google Slides](https://docs.google.com/presentation/d/1H-pCzI_LNrgrL5pJAIQgvLX8Ji0-jIKOg1QeJQzChug/edit) and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/c5aa32b6b07476fa8b597004899ec538/Elasticsearch_Deep_Dive.pdf). Everything covered in this deep dive was accurate as of GitLab 12.0, and while specific details may have changed since then, it should still serve as a good introduction. -In August 2020, a second Deep Dive was hosted, focusing on [GitLab-specific architecture for multi-indices support](#zero-downtime-reindexing-with-multiple-indices). The [recording on YouTube](https://www.youtube.com/watch?v=0WdPR9oB2fg) and the [slides](https://lulalala.gitlab.io/gitlab-elasticsearch-deepdive/) are available. Everything covered in this deep dive was accurate as of GitLab 13.3. +In August 2020, a second Deep Dive was hosted, focusing on [GitLab-specific architecture for multi-indices support](#zero-downtime-reindexing-with-multiple-indices). The <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=0WdPR9oB2fg) and the [slides](https://lulalala.gitlab.io/gitlab-elasticsearch-deepdive/) are available. Everything covered in this deep dive was accurate as of GitLab 13.3. ## Supported Versions @@ -210,7 +210,7 @@ class MigrationName < Elastic::Migration end ``` -Applied migrations are stored in `gitlab-#{RAILS_ENV}-migrations` index. All unexecuted migrations +Applied migrations are stored in `gitlab-#{RAILS_ENV}-migrations` index. All migrations not executed are applied by the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/elastic/migration_worker.rb) cron worker sequentially. @@ -337,7 +337,7 @@ cluster.routing.allocation.disk.watermark.low: 15gb cluster.routing.allocation.disk.watermark.high: 10gb ``` -Restart Elasticsearch, and the `read_only_allow_delete` will clear on it's own. +Restart Elasticsearch, and the `read_only_allow_delete` will clear on its own. _from "Disk-based Shard Allocation | Elasticsearch Reference" [5.6](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/disk-allocator.html#disk-allocator) and [6.x](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/disk-allocator.html)_ @@ -351,7 +351,7 @@ simply reindex everything from scratch. If your Elasticsearch index is incredibly large it may be too time consuming or cause too much downtime to reindex from scratch. There aren't any built in -mechanisms for automatically finding discrepencies and resyncing an +mechanisms for automatically finding discrepancies and resyncing an Elasticsearch index if it gets out of sync but one tool that may be useful is looking at the logs for all the updates that occurred in a time range you believe may have been missed. This information is very low level and only diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index a1899ab5f18..5d77c0ca0e9 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -12,7 +12,7 @@ Experiments are run as an A/B test and are behind a feature flag to turn the tes ## Experiment tracking issue -Each experiment should have an [Experiment tracking](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=growth%20experiment&search=%22Experiment+tracking%22) issue to track the experiment from roll-out through to cleanup/removal. Immediately after an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future). +Each experiment should have an [Experiment tracking](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=growth%20experiment&search=%22Experiment+tracking%22) issue to track the experiment from roll-out through to cleanup/removal. The tracking issue is similar to a feature flag rollout issue, and is also used to track the status of an experiment. Immediately after an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future). After the deadline, the issue needs to be resolved and either: - It was successful and the experiment becomes the new default. @@ -36,7 +36,17 @@ and link to the issue that resolves the experiment. If the experiment is successful and becomes part of the product, any follow up issues should be addressed. -## How to create an A/B test +## Experiments using `gitlab-experiment` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300383) in GitLab 13.7. +> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - It's enabled on GitLab.com. +> - It is not yet intended for use in GitLab self-managed instances. + +[GitLab Experiment](https://gitlab.com/gitlab-org/gitlab-experiment/) is a gem included +in GitLab that can be used for running experiments. + +## How to create an A/B test using `experimentation.rb` ### Implement the experiment @@ -358,7 +368,7 @@ Use a comma to list more than one experiment to be forced: document.cookie = "force_experiment=<EXPERIMENT_KEY>,<ANOTHER_EXPERIMENT_KEY>; path=/"; ``` -Clear the experiments by unsetting the `force_experiment` cookie: +To clear the experiments, unset the `force_experiment` cookie: ```javascript document.cookie = "force_experiment=; path=/"; diff --git a/doc/development/export_csv.md b/doc/development/export_csv.md index 99b6062736d..0bf12149779 100644 --- a/doc/development/export_csv.md +++ b/doc/development/export_csv.md @@ -14,7 +14,7 @@ This document lists the different implementations of CSV export in GitLab codeba | Downloading | - Query and write data in batches to a temporary file.<br>- Loads the file into memory.<br>- Sends the file to the client. | - Report available immediately. | - Large amount of data might cause request timeout.<br>- Memory intensive.<br>- Request expires when user navigates to a different page. | [Export Chain of Custody Report](../user/compliance/compliance_dashboard/#chain-of-custody-report) | | As email attachment | - Asynchronously process the query with background job.<br>- Email uses the export as an attachment. | - Asynchronous processing. | - Requires users use a different app (email) to download the CSV.<br>- Email providers may limit attachment size. | - [Export Issues](../user/project/issues/csv_export.md)<br>- [Export Merge Requests](../user/project/merge_requests/csv_export.md) | | As downloadable link in email (*) | - Asynchronously process the query with background job.<br>- Email uses an export link. | - Asynchronous processing.<br>- Bypasses email provider attachment size limit. | - Requires users use a different app (email).<br>- Requires additional storage and cleanup. | [Export User Permissions](https://gitlab.com/gitlab-org/gitlab/-/issues/1772) | -| Polling (non-persistent state) | - Asynchronously processes the query with the background job.<br>- Frontend(FE) polls every few seconds to check if CSV file is ready. | - Asynchronous processing.<br>- Automatically downloads to local machine on completion.<br>- In-app solution. | - Non-persistable request - request expires when user navigates to a different page.<br>- API is processed for each polling request. | [Export Vulnerabilities](../user/application_security/security_dashboard/#export-vulnerabilities) | +| Polling (non-persistent state) | - Asynchronously processes the query with the background job.<br>- Frontend(FE) polls every few seconds to check if CSV file is ready. | - Asynchronous processing.<br>- Automatically downloads to local machine on completion.<br>- In-app solution. | - Non-persistable request - request expires when user navigates to a different page.<br>- API is processed for each polling request. | [Export Vulnerabilities](../user/application_security/vulnerability_report/#export-vulnerabilities) | | Polling (persistent state) (*) | - Asynchronously processes the query with background job.<br>- Backend (BE) maintains the export state<br>- FE polls every few seconds to check status.<br>- FE shows 'Download link' when export is ready.<br>- User can download or regenerate a new report. | - Asynchronous processing.<br>- No database calls made during the polling requests (HTTP 304 status is returned until export status changes).<br>- Does not require user to stay on page until export is complete.<br>- In-app solution.<br>- Can be expanded into a generic CSV feature (such as dashboard / CSV API). | - Requires to maintain export states in DB.<br>- Does not automatically download the CSV export to local machine, requires users to click 'Download' button. | [Export Merge Commits Report](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43055) | NOTE: diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index 92730e8139f..5ad1a701fac 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -9,11 +9,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Resources [Chrome Accessibility Developer Tools](https://github.com/GoogleChrome/accessibility-developer-tools) -are useful for testing for potential accessibility problems in GitLab. +assist with testing for potential accessibility problems in GitLab. -The [axe](https://www.deque.com/axe/) browser extension (available for [Firefox](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/) and [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd)) is -also a handy tool for running audits and getting feedback on markup, CSS and even potentially problematic color usages. +The [axe](https://www.deque.com/axe/) browser extension (available for [Firefox](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/) and [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd)) provides running audits and feedback on markup, CSS, and even potentially problematic color usages. Accessibility best-practices and more in-depth information are available on -[the Audit Rules page](https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules) for the Chrome Accessibility Developer Tools. The [Awesome Accessibility](https://github.com/brunopulis/awesome-a11y) list is also a -useful compilation of accessibility-related material. +[the Audit Rules page](https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules) for the Chrome Accessibility Developer Tools. The [Awesome Accessibility](https://github.com/brunopulis/awesome-a11y) list is a compilation of accessibility-related material. diff --git a/doc/development/fe_guide/architecture.md b/doc/development/fe_guide/architecture.md index 964837dc5f7..c51f99ca9d2 100644 --- a/doc/development/fe_guide/architecture.md +++ b/doc/development/fe_guide/architecture.md @@ -6,9 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Architecture -When you are developing a new feature that requires architectural design, or if -you are changing the fundamental design of an existing feature, make sure it is -discussed with one of the Frontend Architecture Experts. +When developing a feature that requires architectural design, or changing the fundamental design of an existing feature, discuss it with a Frontend Architecture Expert. A Frontend Architect is an expert who makes high-level Frontend design decisions and decides on technical standards, including coding standards and frameworks. diff --git a/doc/development/fe_guide/axios.md b/doc/development/fe_guide/axios.md index cf5a4970c04..2d699b305ce 100644 --- a/doc/development/fe_guide/axios.md +++ b/doc/development/fe_guide/axios.md @@ -44,7 +44,7 @@ Advantages over [`spyOn()`](https://jasmine.github.io/api/edge/global.html#spyOn - no need to create response objects - does not allow call through (which we want to avoid) -- simple API to test error cases +- clear API to test error cases - provides `replyOnce()` to allow for different responses We have also decided against using [Axios interceptors](https://github.com/axios/axios#interceptors) because they are not suitable for mocking. diff --git a/doc/development/fe_guide/components.md b/doc/development/fe_guide/components.md deleted file mode 100644 index 6b6274a6480..00000000000 --- a/doc/development/fe_guide/components.md +++ /dev/null @@ -1,3 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/components/status/' ---- diff --git a/doc/development/fe_guide/design_patterns.md b/doc/development/fe_guide/design_patterns.md index 784612682f8..d6b1f38b417 100644 --- a/doc/development/fe_guide/design_patterns.md +++ b/doc/development/fe_guide/design_patterns.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w When exactly one object is needed for a given task, prefer to define it as a `class` rather than as an object literal. Prefer also to explicitly restrict -instantiation, unless flexibility is important (e.g. for testing). +instantiation, unless flexibility is important (such as for testing). ```javascript // bad @@ -56,7 +56,7 @@ export default class MyThing { ## Manipulating the DOM in a JS Class When writing a class that needs to manipulate the DOM guarantee a container option is provided. -This is useful when we need that class to be instantiated more than once in the same page. +This can be used when we need that class to be instantiated more than once in the same page. Bad: diff --git a/doc/development/fe_guide/development_process.md b/doc/development/fe_guide/development_process.md index d122459f51c..b85ed4da442 100644 --- a/doc/development/fe_guide/development_process.md +++ b/doc/development/fe_guide/development_process.md @@ -12,9 +12,9 @@ You can find more about the organization of the frontend team in the [handbook]( The idea is to remind us about specific topics during the time we build a new feature or start something. This is a common practice in other industries (like pilots) that also use standardized checklists to reduce problems early on. -Copy the content over to your issue or merge request and if something doesn't apply simply remove it from your current list. +Copy the content over to your issue or merge request and if something doesn't apply, remove it from your current list. -This checklist is intended to help us during development of bigger features/refactorings, it's not a "use it always and every point always matches" list. +This checklist is intended to help us during development of bigger features/refactorings. It is not a "use it always and every point always matches" list. Please use your best judgment when to use it and please contribute new points through merge requests if something comes to your mind. @@ -77,7 +77,7 @@ With the purpose of being [respectful of others' time](https://about.gitlab.com/ - includes tests - includes a changelog entry (when necessary) - Before assigning to a maintainer, assign to a reviewer. -- If you assigned a merge request or pinged someone directly, be patient because we work in different timezones and asynchronously. Unless the merge request is urgent (like fixing a broken master), please don't DM or reassign the merge request before waiting for a 24-hour window. +- If you assigned a merge request or pinged someone directly, be patient because we work in different timezones and asynchronously. Unless the merge request is urgent (like fixing a broken default branch), please don't DM or reassign the merge request before waiting for a 24-hour window. - If you have a question regarding your merge request/issue, make it on the merge request/issue. When we DM each other, we no longer have a SSOT and [no one else is able to contribute](https://about.gitlab.com/handbook/values/#public-by-default). - When you have a big **Draft** merge request with many changes, you're advised to get the review started before adding/removing significant code. Make sure it is assigned well before the release cut-off, as the reviewer(s)/maintainer(s) would always prioritize reviewing finished MRs before the **Draft** ones. - Make sure to remove the `Draft:` title before the last round of review. diff --git a/doc/development/fe_guide/dropdowns.md b/doc/development/fe_guide/dropdowns.md deleted file mode 100644 index bd2dae13c5b..00000000000 --- a/doc/development/fe_guide/dropdowns.md +++ /dev/null @@ -1,3 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/components/dropdown/' ---- diff --git a/doc/development/fe_guide/editor_lite.md b/doc/development/fe_guide/editor_lite.md index 47ef85d8737..f783a97fbd3 100644 --- a/doc/development/fe_guide/editor_lite.md +++ b/doc/development/fe_guide/editor_lite.md @@ -8,13 +8,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Background -**Editor Lite** is a technological product driving [Web Editor](../../user/project/repository/web_editor.md), [Snippets](../../user/snippets.md), [CI Linter](../../ci/lint.md), etc. Editor Lite is the driving technology for any single-file editing experience across the product. +**Editor Lite** is a technological product driving features like [Web Editor](../../user/project/repository/web_editor.md), [Snippets](../../user/snippets.md), and [CI Linter](../../ci/lint.md). Editor Lite is the driving technology for any single-file editing experience across the product. Editor Lite is a thin wrapper around [the Monaco editor](https://microsoft.github.io/monaco-editor/index.html) that provides the necessary helpers and abstractions and extends Monaco using extensions. ## How to use Editor Lite -Editor Lite is framework-agnostic and can be used in any application, whether it's Rails or Vue. For the convenience of integration, we have [the dedicated `<editor-lite>` Vue component](#vue-component), but in general, the integration of Editor Lite is pretty straightforward: +Editor Lite is framework-agnostic and can be used in any application, whether it's Rails or Vue. For the convenience of integration, we have the dedicated `<editor-lite>` Vue component, but in general, the integration of Editor Lite is pretty straightforward: 1. Import Editor Lite: @@ -65,7 +65,7 @@ The editor follows the same public API as [provided by Monaco editor](https://mi 1. Editor's loading state. -Editor Lite comes with the loading state built-in, making spinners and loaders rarely needed in HTML. To benefit the built-in loading state, set the `data-editor-loading` property on the HTML element that is supposed to contain the editor. Editor Lite will show the loader automatically while it's bootstrapping. +Editor Lite comes with the loading state built-in, making spinners and loaders rarely needed in HTML. To benefit the built-in loading state, set the `data-editor-loading` property on the HTML element that is supposed to contain the editor. Editor Lite shows the loader automatically while it's bootstrapping.  1. Update syntax highlighting if the filename changes. @@ -89,7 +89,7 @@ form.addEventListener('submit', () => { 1. Performance -Even though Editor Lite itself is extremely slim, it still depends on Monaco editor. Monaco is not an easily tree-shakeable module. Hence, every time you add Editor Lite to a view, the JavaScript bundle's size significantly increases, affecting your view's loading performance. To avoid that, it is recommended to import the editor on demand on those views where it is not 100% certain that the editor will be used. Or if the editor is a secondary element of the view. Loading Editor Lite on demand is no different from loading any other module: +Even though Editor Lite itself is extremely slim, it still depends on Monaco editor. Monaco is not an easily tree-shakeable module. Hence, every time you add Editor Lite to a view, the JavaScript bundle's size significantly increases, affecting your view's loading performance. It is recommended to import the editor on demand on those views where it is not 100% certain that the editor is needed. Or if the editor is a secondary element of the view. Loading Editor Lite on demand is no different from loading any other module: ```javascript someActionFunction() { @@ -109,8 +109,8 @@ which would not depend on any particular group. Even though the Editor Lite's co [Create::Editor FE Team](https://about.gitlab.com/handbook/engineering/development/dev/create-editor/), the main functional elements — extensions — can be owned by any group. Editor Lite extensions' main idea is that the core of the editor remains very slim and stable. At the same time, whatever new functionality -is needed can be added as an extension to this core, without touching the core itself. It allows any group -to build and own any new editing functionality without being afraid of it being broken or overridden with +is needed can be added as an extension to this core, without touching the core itself. Any group is allowed +to build and own new editing functionality without being afraid of it being broken or overridden with the Editor Lite changes. Structurally, the complete implementation of Editor Lite could be presented as the following diagram: @@ -145,7 +145,7 @@ Important things to note here: ### Using an existing extension -Adding an extension to Editor Lite's instance is simple: +Adding an extension to Editor Lite's instance requires the following steps: ```javascript import EditorLite from '~/editor/editor_lite'; @@ -159,7 +159,7 @@ editor.use(MyExtension); ### Creating an extension -Let's create our first Editor Lite extension. As aforementioned, extensions are ES6 modules exporting the simple `Object` that is used to extend Editor Lite's functionality. As the most straightforward test, let's create an extension that extends Editor Lite with a new function that, when called, will output editor's content in `alert`. +Let's create our first Editor Lite extension. Extensions are ES6 modules exporting a basic `Object` that is used to extend Editor Lite's functionality. As a test, let's create an extension that extends Editor Lite with a new function that, when called, outputs editor's content in `alert`. `~/my_folder/my_fancy_extension.js:` @@ -225,7 +225,3 @@ Just pass the array of extensions to your `use` method: ```javascript editor.use([FileTemplateExtension, MyFancyExtension]); ``` - -## <a id="vue-component"></a>`<editor-lite>` Vue component - -TBD diff --git a/doc/development/fe_guide/emojis.md b/doc/development/fe_guide/emojis.md index 7151e2ffeee..2dedbc8f19d 100644 --- a/doc/development/fe_guide/emojis.md +++ b/doc/development/fe_guide/emojis.md @@ -25,7 +25,7 @@ when your platform does not support it. - `app/assets/images/emoji.png` - `app/assets/images/emoji@2x.png` 1. Ensure you see new individual images copied into `app/assets/images/emoji/` - 1. Ensure you can see the new emojis and their aliases in the GFM Autocomplete + 1. Ensure you can see the new emojis and their aliases in the GitLab Flavored Markdown (GFM) Autocomplete 1. Ensure you can see the new emojis and their aliases in the award emoji menu 1. You might need to add new emoji Unicode support checks and rules for platforms that do not support a certain emoji and we need to fallback to an image. diff --git a/doc/development/fe_guide/frontend_faq.md b/doc/development/fe_guide/frontend_faq.md index 9612f604b56..772f1672293 100644 --- a/doc/development/fe_guide/frontend_faq.md +++ b/doc/development/fe_guide/frontend_faq.md @@ -21,7 +21,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## FAQ -### 1. How do I find the Rails route for a page? +### 1. How does one find the Rails route for a page? #### Check the 'page' data attribute @@ -36,7 +36,7 @@ Find here the [source code setting the attribute](https://gitlab.com/gitlab-org/ #### Rails routes -The `rake routes` command can be used to list all the routes available in the application, piping the output into `grep`, we can perform a search through the list of available routes. +The `rake routes` command can be used to list all the routes available in the application. Piping the output into `grep`, we can perform a search through the list of available routes. The output includes the request types available, route parameters and the relevant controller. ```shell @@ -46,13 +46,13 @@ bundle exec rake routes | grep "issues" ### 2. `modal_copy_button` vs `clipboard_button` The `clipboard_button` uses the `copy_to_clipboard.js` behavior, which is -initialized on page load, so if there are vue-based clipboard buttons that -don't exist at page load (such as ones in a `GlModal`), they do not have the +initialized on page load. Vue clipboard buttons that +don't exist at page load (such as ones in a `GlModal`) do not have click handlers associated with the clipboard package. -`modal_copy_button` was added that manages an instance of the +`modal_copy_button` manages an instance of the [`clipboard` plugin](https://www.npmjs.com/package/clipboard) specific to -the instance of that component, which means that clipboard events are +the instance of that component. This means that clipboard events are bound on mounting and destroyed when the button is, mitigating the above issue. It also has bindings to a particular container or modal ID available, to work with the focus trap created by our GlModal. @@ -60,7 +60,7 @@ available, to work with the focus trap created by our GlModal. ### 3. A `gitlab-ui` component not conforming to [Pajamas Design System](https://design.gitlab.com/) Some [Pajamas Design System](https://design.gitlab.com/) components implemented in -`gitlab-ui` do not conform with the design system specs because they lack some +`gitlab-ui` do not conform with the design system specs. This is because they lack some planned features or are not correctly styled yet. In the Pajamas website, a banner on top of the component examples indicates that: @@ -77,18 +77,17 @@ It makes codebase unified and more comfortable to maintain/refactor in the futur Ensure a [Product Designer](https://about.gitlab.com/company/team/?department=ux-department) reviews the use of the non-conforming component as part of the MR review. Make a -follow up issue and attach it to the component implementation epic found within the +follow up issue and attach it to the component implementation epic found in the [Components of Pajamas Design System epic](https://gitlab.com/groups/gitlab-org/-/epics/973). ### 4. My submit form button becomes disabled after submitting -If you are using a submit button inside a form and you attach an `onSubmit` event listener on the form element, [this piece of code](https://gitlab.com/gitlab-org/gitlab/blob/794c247a910e2759ce9b401356432a38a4535d49/app/assets/javascripts/main.js#L225) adds a `disabled` class selector to the submit button when the form is submitted. -To avoid this behavior, add the class `js-no-auto-disable` to the button. +A Submit button inside of a form attaches an `onSubmit` event listener on the form element. [This code](https://gitlab.com/gitlab-org/gitlab/blob/794c247a910e2759ce9b401356432a38a4535d49/app/assets/javascripts/main.js#L225) adds a `disabled` class selector to the submit button when the form is submitted. To avoid this behavior, add the class `js-no-auto-disable` to the button. -### 5. Should I use a full URL (i.e. `gon.gitlab_url`) or a full path (i.e. `gon.relative_url_root`) when referencing backend endpoints? +### 5. Should one use a full URL (for example `gon.gitlab_url`) or a full path (for example `gon.relative_url_root`) when referencing backend endpoints? -It's preferred to use a **full path** over a **full URL** because the URL uses the hostname configured with -GitLab which may not match the request. This causes [CORS issues like this Web IDE one](https://gitlab.com/gitlab-org/gitlab/-/issues/36810). +It's preferred to use a **full path** over a **full URL**. This is because the URL uses the hostname configured with +GitLab which may not match the request. This causes [cross-origin resource sharing issues like this Web IDE example](https://gitlab.com/gitlab-org/gitlab/-/issues/36810). Example: @@ -117,7 +116,7 @@ Example: ### 6. How should the Frontend reference Backend paths? -We prefer not to add extra coupling by hardcoding paths. If possible, +We prefer not to add extra coupling by hard-coding paths. If possible, add these paths as data attributes to the DOM element being referenced in the JavaScript. Example: @@ -153,7 +152,7 @@ export const fetchFoos = ({ state }) => { }; ``` -### 7. How can I test the production build locally? +### 7. How can one test the production build locally? Sometimes it's necessary to test locally what the frontend production build would produce, to do so the steps are: @@ -161,7 +160,7 @@ Sometimes it's necessary to test locally what the frontend production build woul 1. Open `gitlab.yaml` located in your `gitlab` installation folder, scroll down to the `webpack` section and change `dev_server` to `enabled: false`. 1. Run `yarn webpack-prod && gdk restart rails-web`. -The production build takes a few minutes to be completed; any code changes at this point are +The production build takes a few minutes to be completed. Any code changes at this point are displayed only after executing the item 3 above again. To return to the normal development mode: @@ -176,8 +175,8 @@ To return to the normal development mode: > [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/28837) in GitLab 12.8. GitLab has enabled the Babel `preset-env` option -[`useBuiltIns: 'usage'`](https://babeljs.io/docs/en/babel-preset-env#usebuiltins-usage), -which adds the appropriate `core-js` polyfills once for each JavaScript feature +[`useBuiltIns: 'usage'`](https://babeljs.io/docs/en/babel-preset-env#usebuiltins-usage). +This adds the appropriate `core-js` polyfills once for each JavaScript feature we're using that our target browsers don't support. You don't need to add `core-js` polyfills manually. diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index cbaa648570c..a53d9fee029 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -18,34 +18,46 @@ info: "See the Technical Writers assigned to Development Guidelines: https://abo **GraphQL at GitLab**: -- [🎬 GitLab Unfiltered GraphQL playlist](https://www.youtube.com/watch?v=wHPKZBDMfxE&list=PL05JrBw4t0KpcjeHjaRMB7IGB2oDWyJzv) -- [🎬 GraphQL at GitLab: Deep Dive](../api_graphql_styleguide.md#deep-dive) (video) by Nick Thomas +<!-- vale gitlab.Spelling = NO --> + +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [GitLab Unfiltered GraphQL playlist](https://www.youtube.com/watch?v=wHPKZBDMfxE&list=PL05JrBw4t0KpcjeHjaRMB7IGB2oDWyJzv) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [GraphQL at GitLab: Deep Dive](../api_graphql_styleguide.md#deep-dive) (video) by Nick Thomas - An overview of the history of GraphQL at GitLab (not frontend-specific) -- [🎬 GitLab Feature Walkthrough with GraphQL and Vue Apollo](https://www.youtube.com/watch?v=6yYp2zB7FrM) (video) by Natalia Tepluhina +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [GitLab Feature Walkthrough with GraphQL and Vue Apollo](https://www.youtube.com/watch?v=6yYp2zB7FrM) (video) by Natalia Tepluhina - A real-life example of implementing a frontend feature in GitLab using GraphQL -- [🎬 History of client-side GraphQL at GitLab](https://www.youtube.com/watch?v=mCKRJxvMnf0) (video) Illya Klymov and Natalia Tepluhina -- [🎬 From Vuex to Apollo](https://www.youtube.com/watch?v=9knwu87IfU8) (video) by Natalia Tepluhina - - A useful overview of when Apollo might be a better choice than Vuex, and how one could go about the transition +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [History of client-side GraphQL at GitLab](https://www.youtube.com/watch?v=mCKRJxvMnf0) (video) Illya Klymov and Natalia Tepluhina +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [From Vuex to Apollo](https://www.youtube.com/watch?v=9knwu87IfU8) (video) by Natalia Tepluhina + - An overview of when Apollo might be a better choice than Vuex, and how one could go about the transition - [🛠Vuex -> Apollo Migration: a proof-of-concept project](https://gitlab.com/ntepluhina/vuex-to-apollo/blob/master/README.md) - A collection of examples that show the possible approaches for state management with Vue+GraphQL+(Vuex or Apollo) apps +<!-- vale gitlab.Spelling = YES --> + ### Libraries We use [Apollo](https://www.apollographql.com/) (specifically [Apollo Client](https://www.apollographql.com/docs/react/)) and [Vue Apollo](https://github.com/vuejs/vue-apollo) when using GraphQL for frontend development. -If you are using GraphQL within a Vue application, the [Usage in Vue](#usage-in-vue) section +If you are using GraphQL in a Vue application, the [Usage in Vue](#usage-in-vue) section can help you learn how to integrate Vue Apollo. For other use cases, check out the [Usage outside of Vue](#usage-outside-of-vue) section. +<!-- vale gitlab.Spelling = NO --> + We use [Immer](https://immerjs.github.io/immer/docs/introduction) for immutable cache updates; see [Immutability and cache updates](#immutability-and-cache-updates) for more information. +<!-- vale gitlab.Spelling = YES --> + ### Tooling +<!-- vale gitlab.Spelling = NO --> + - [Apollo Client Devtools](https://github.com/apollographql/apollo-client-devtools) +<!-- vale gitlab.Spelling = YES --> + #### [Apollo GraphQL VS Code extension](https://marketplace.visualstudio.com/items?itemName=apollographql.vscode-apollo) If you use VS Code, the Apollo GraphQL extension supports autocompletion in `.graphql` files. To set up @@ -76,7 +88,7 @@ Our GraphQL API can be explored via GraphiQL at your instance's where needed. You can check all existing queries and mutations on the right side -of GraphiQL in its **Documentation explorer**. It's also possible to +of GraphiQL in its **Documentation explorer**. You can also write queries and mutations directly on the left tab and check their execution by clicking **Execute query** button on the top left: @@ -93,8 +105,8 @@ Default client accepts two parameters: `resolvers` and `config`. - `resolvers` parameter is created to accept an object of resolvers for [local state management](#local-state-with-apollo) queries and mutations - `config` parameter takes an object of configuration settings: - `cacheConfig` field accepts an optional object of settings to [customize Apollo cache](https://www.apollographql.com/docs/react/caching/cache-configuration/#configuring-the-cache) - - `baseUrl` allows us to pass a URL for GraphQL endpoint different from our main endpoint (i.e.`${gon.relative_url_root}/api/graphql`) - - `assumeImmutableResults` (set to `false` by default) - this setting, when set to `true`, will assume that every single operation on updating Apollo Cache is immutable. It also sets `freezeResults` to `true`, so any attempt on mutating Apollo Cache will throw a console warning in development environment. Please ensure you're following the immutability pattern on cache update operations before setting this option to `true`. + - `baseUrl` allows us to pass a URL for GraphQL endpoint different from our main endpoint (for example, `${gon.relative_url_root}/api/graphql`) + - `assumeImmutableResults` (set to `false` by default) - this setting, when set to `true`, assumes that every single operation on updating Apollo Cache is immutable. It also sets `freezeResults` to `true`, so any attempt on mutating Apollo Cache throws a console warning in development environment. Please ensure you're following the immutability pattern on cache update operations before setting this option to `true`. - `fetchPolicy` determines how you want your component to interact with the Apollo cache. Defaults to "cache-first". ## GraphQL Queries @@ -139,7 +151,7 @@ fragment DesignItem on Design { ``` More about fragments: -[GraphQL Docs](https://graphql.org/learn/queries/#fragments) +[GraphQL documentation](https://graphql.org/learn/queries/#fragments) ## Global IDs @@ -157,12 +169,17 @@ const primaryKeyId = getIdFromGraphQLId(data.id); ## Immutability and cache updates -From Apollo version 3.0.0 all the cache updates need to be immutable; it needs to be replaced entirely +From Apollo version 3.0.0 all the cache updates need to be immutable. It needs to be replaced entirely with a **new and updated** object. -To facilitate the process of updating the cache and returning the new object we use the library [Immer](https://immerjs.github.io/immer/docs/introduction). +<!-- vale gitlab.Spelling = NO --> + +To facilitate the process of updating the cache and returning the new object we +use the library [Immer](https://immerjs.github.io/immer/docs/introduction). When possible, follow these conventions: +<!-- vale gitlab.Spelling = YES --> + - The updated cache is named `data`. - The original cache data is named `sourceData`. @@ -184,10 +201,10 @@ client.writeQuery({ ``` As shown in the code example by using `produce`, we can perform any kind of direct manipulation of the -`draftState`. Besides, `immer` guarantees that a new state which includes the changes to `draftState` will be generated. +`draftState`. Besides, `immer` guarantees that a new state which includes the changes to `draftState` is generated. Finally, to verify whether the immutable cache update is working properly, we need to change -`assumeImmutableResults` to `true` in the default client configuration (see [Apollo Client](#apollo-client) for more information). +`assumeImmutableResults` to `true` in the default client configuration. See [Apollo Client](#apollo-client) for more information. If everything is working properly `assumeImmutableResults` should remain set to `true`. @@ -259,11 +276,11 @@ query User { } ``` -Along with creating local data, we can also extend existing GraphQL types with `@client` fields. This is extremely useful when we need to mock an API responses for fields not yet added to our GraphQL API. +Along with creating local data, we can also extend existing GraphQL types with `@client` fields. This is extremely helpful when we need to mock an API response for fields not yet added to our GraphQL API. #### Mocking API response with local Apollo cache -Using local Apollo Cache is handy when we have a need to mock some GraphQL API responses, queries or mutations locally (e.g. when they're still not added to our actual API). +Using local Apollo Cache is helpful when we have a need to mock some GraphQL API responses, queries, or mutations locally (such as when they're still not added to our actual API). For example, we have a [fragment](#fragments) on `DesignVersion` used in our queries: @@ -274,7 +291,7 @@ fragment VersionListItem on DesignVersion { } ``` -We need to fetch also version author and the 'created at' property to display them in the versions dropdown but these changes are still not implemented in our API. We can change the existing fragment to get a mocked response for these new fields: +We also need to fetch the version author and the `created at` property to display in the versions dropdown. But, these changes are still not implemented in our API. We can change the existing fragment to get a mocked response for these new fields: ```javascript fragment VersionListItem on DesignVersion { @@ -288,7 +305,7 @@ fragment VersionListItem on DesignVersion { } ``` -Now Apollo will try to find a _resolver_ for every field marked with `@client` directive. Let's create a resolver for `DesignVersion` type (why `DesignVersion`? because our fragment was created on this type). +Now Apollo tries to find a _resolver_ for every field marked with `@client` directive. Let's create a resolver for `DesignVersion` type (why `DesignVersion`? because our fragment was created on this type). ```javascript // resolvers.js @@ -319,13 +336,13 @@ import resolvers from './graphql/resolvers'; const defaultClient = createDefaultClient(resolvers); ``` -For each attempt to fetch a version, our client will fetch `id` and `sha` from the remote API endpoint and will assign our hardcoded values to the `author` and `createdAt` version properties. With this data, frontend developers are able to work on their UI without being blocked by backend. When the actual response is added to the API, our custom local resolver can be removed and the only change to the query/fragment is to remove the `@client` directive. +For each attempt to fetch a version, our client fetches `id` and `sha` from the remote API endpoint. It then assigns our hardcoded values to the `author` and `createdAt` version properties. With this data, frontend developers are able to work on their UI without being blocked by backend. When the response is added to the API, our custom local resolver can be removed. The only change to the query/fragment is to remove the `@client` directive. Read more about local state management with Apollo in the [Vue Apollo documentation](https://vue-apollo.netlify.app/guide/local-state.html#local-state). ### Using with Vuex -When Apollo Client is used within Vuex and fetched data is stored in the Vuex store, there is no need to keep Apollo Client cache enabled. Otherwise we would have data from the API stored in two places - Vuex store and Apollo Client cache. With Apollo's default settings, a subsequent fetch from the GraphQL API could result in fetching data from Apollo cache (in the case where we have the same query and variables). To prevent this behavior, we need to disable Apollo Client cache by passing a valid `fetchPolicy` option to its constructor: +When the Apollo Client is used in Vuex and fetched data is stored in the Vuex store, the Apollo Client cache does not need to be enabled. Otherwise we would have data from the API stored in two places - Vuex store and Apollo Client cache. With Apollo's default settings, a subsequent fetch from the GraphQL API could result in fetching data from Apollo cache (in the case where we have the same query and variables). To prevent this behavior, we need to disable Apollo Client cache by passing a valid `fetchPolicy` option to its constructor: ```javascript import fetchPolicies from '~/graphql_shared/fetch_policy_constants'; @@ -338,11 +355,61 @@ export const gqClient = createGqClient( ); ``` -### Feature flags in queries +### Working on GraphQL-based features when frontend and backend are not in sync + +Any feature that requires GraphQL queries/mutations to be created or updated should be carefully +planned. Frontend and backend counterparts should agree on a schema that satisfies both client-side and +server-side requirements. This enables both departments to start implementing their parts without +blocking each other. + +Ideally, the backend implementation should be done prior to the frontend so that the client can +immediately start querying the API with minimal back and forth between departments. However, we +recognize that priorities don't always align. For the sake of iteration and +delivering work we're committed to, it might be necessary for the frontend to be implemented ahead +of the backend. + +#### Implementing frontend queries and mutations ahead of the backend -Sometimes it may be useful to have an entity in the GraphQL query behind a feature flag. -For example, when working on a feature where the backend has already been merged but the frontend -hasn't you might want to put the GraphQL entity behind a feature flag to allow for smaller +In such case, the frontend will define GraphQL schemas or fields that do not correspond to any +backend resolver yet. This is fine as long as the implementation is properly feature-flagged so it +does not translate to public-facing errors in the product. However, we do validate client-side +queries/mutations against the backend GraphQL schema with the `graphql-verify` CI job. +You must confirm your changes pass the validation if they are to be merged before the +backend actually supports them. Below are a few suggestions to go about this. + +##### Using the `@client` directive + +The preferred approach is to use the `@client` directive on any new query, mutation, or field that +isn't yet supported by the backend. Any entity with the directive is skipped by the +`graphql-verify` validation job. + +Additionally Apollo will attempt to resolve them client-side, which can be used in conjunction with +[Mocking API response with local Apollo cache](#mocking-api-response-with-local-apollo-cache). This +provides a convenient way of testing your feature with fake data defined client-side. +When opening a merge request for your changes, it can be a good idea to provide local resolvers as a +patch that reviewers can apply in their GDK to easily smoke-test your work. + +Make sure to track the removal of the directive in a follow-up issue, or as part of the backend +implementation plan. + +##### Adding an exception to the list of known failures + +GraphQL queries/mutations validation can be completely turned off for specific files by adding their +paths to the +[`config/known_invalid_graphql_queries.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/known_invalid_graphql_queries.yml) +file, much like you would disable ESLint for some files via an `.eslintignore` file. +Bear in mind that any file listed in here will not be validated at all. So if you're only adding +fields to an existing query, use the `@client` directive approach so that the rest of the query +is still validated. + +Again, make sure that those overrides are as short-lived as possible by tracking their removal in +the appropriate issue. + +#### Feature flags in queries + +Sometimes it may be helpful to have an entity in the GraphQL query behind a feature flag. +One example is working on a feature where the backend has already been merged but the frontend +has not. In this case, you may consider putting the GraphQL entity behind a feature flag to allow smaller merge requests to be created and merged. To do this we can use the `@include` directive to exclude an entity if the `if` statement passes. @@ -355,7 +422,7 @@ query getAuthorData($authorNameEnabled: Boolean = false) { ``` Then in the Vue (or JavaScript) call to the query we can pass in our feature flag. This feature -flag will need to be already setup correctly. See the [feature flag documentation](../feature_flags/development.md) +flag needs to be already set up correctly. See the [feature flag documentation](../feature_flags/development.md) for the correct way to do this. ```javascript @@ -469,7 +536,7 @@ Note that we are using the [`pageInfo.fragment.graphql`](https://gitlab.com/gitl #### Using `fetchMore` method in components -This approach makes sense to use with user-handled pagination (e.g. when the scrolls to fetch more data or explicitly clicks a "Next Page"-button). +This approach makes sense to use with user-handled pagination. For example, when the scrolling to fetch more data or explicitly clicking a **Next Page** button. When we need to fetch all the data initially, it is recommended to use [a (non-smart) query, instead](#using-a-recursive-query-in-components). When making an initial fetch, we usually want to start a pagination from the beginning. @@ -479,7 +546,7 @@ In this case, we can either: - Pass `null` explicitly to `after`. After data is fetched, we can use the `update`-hook as an opportunity [to customize -the data that is set in the Vue component property](https://apollo.vuejs.org/api/smart-query.html#options), getting a hold of the `pageInfo` object among other data. +the data that is set in the Vue component property](https://apollo.vuejs.org/api/smart-query.html#options). This allows us to get a hold of the `pageInfo` object among other data. In the `result`-hook, we can inspect the `pageInfo` object to see if we need to fetch the next page. Note that we also keep a `requestCount` to ensure that the application @@ -561,8 +628,8 @@ fetchNextPage(endCursor) { When it is necessary to fetch all paginated data initially an Apollo query can do the trick for us. If we need to fetch the next page based on user interactions, it is recommend to use a [`smartQuery`](https://apollo.vuejs.org/api/smart-query.html) along with the [`fetchMore`-hook](#using-fetchmore-method-in-components). -When the query resolves we can update the component data and inspect the `pageInfo` object -to see if we need to fetch the next page, i.e. call the method recursively. +When the query resolves we can update the component data and inspect the `pageInfo` object. This allows us +to see if we need to fetch the next page, calling the method recursively. Note that we also keep a `requestCount` to ensure that the application does not keep requesting the next page, indefinitely. @@ -634,7 +701,7 @@ or [`.writeQuery()`](https://www.apollographql.com/docs/react/v2/api/apollo-clie This can be tedious and counter-intuitive. To make it easier to deal with cached paginated queries, Apollo provides the `@connection` directive. -The directive accepts a `key` parameter that will be used as a static key when caching the data. +The directive accepts a `key` parameter that is used as a static key when caching the data. You'd then be able to retrieve the data without providing any pagination-specific variables. Here's an example of a query using the `@connection` directive: @@ -661,7 +728,7 @@ query DastSiteProfiles($fullPath: ID!, $after: String, $before: String, $first: } ``` -In this example, Apollo will store the data with the stable `dastSiteProfiles` cache key. +In this example, Apollo stores the data with the stable `dastSiteProfiles` cache key. To retrieve that data from the cache, you'd then only need to provide the `$fullPath` variable, omitting pagination-specific variables like `after` or `before`: @@ -679,9 +746,9 @@ Read more about the `@connection` directive in [Apollo's documentation](https:// ### Managing performance -The Apollo client will batch queries by default. This means that if you have 3 queries defined, -Apollo will group them into one request, send the single request off to the server and only -respond once all 3 queries have completed. +The Apollo client batches queries by default. Given 3 deferred queries, +Apollo groups them into one request, sends the single request to the server, and +responds after all 3 queries have completed. If you need to have queries sent as individual requests, additional context can be provided to tell Apollo to do this. @@ -703,9 +770,13 @@ export default { #### Mocking response as component data -With [Vue test utils](https://vue-test-utils.vuejs.org/) it is easy to quickly test components that +<!-- vale gitlab.Spelling = NO --> + +With [Vue Test Utils](https://vue-test-utils.vuejs.org/) one can quickly test components that fetch GraphQL queries. The simplest way is to use `shallowMount` and then set -the data on the component +the data on the component: + +<!-- vale gitlab.Spelling = YES --> ```javascript it('tests apollo component', () => { @@ -719,7 +790,7 @@ it('tests apollo component', () => { #### Testing loading state -If we need to test how our component renders when results from the GraphQL API are still loading, we can mock a loading state into respective Apollo queries/mutations: +To test how a component renders when results from the GraphQL API are still loading, mock a loading state into respective Apollo queries/mutations: ```javascript function createComponent({ @@ -817,9 +888,9 @@ it('calls mutation on submitting form ', () => { To test the logic of Apollo cache updates, we might want to mock an Apollo Client in our unit tests. We use [`mock-apollo-client`](https://www.npmjs.com/package/mock-apollo-client) library to mock Apollo client and [`createMockApollo` helper](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/frontend/__helpers__/mock_apollo_helper.js) we created on top of it. -To separate tests with mocked client from 'usual' unit tests, it's recommended to create an additional factory and pass the created `mockApollo` as an option to the `createComponent`-factory. This way we only create Apollo Client instance when it's necessary. +To separate tests with mocked client from 'usual' unit tests, create an additional factory and pass the created `mockApollo` as an option to the `createComponent`-factory. This way we only create Apollo Client instance when it's necessary. -We need to inject `VueApollo` to the Vue local instance and, likewise, it is recommended to call `localVue.use()` within `createMockApolloProvider()` to only load it when it is necessary. +We need to inject `VueApollo` to the Vue local instance and, likewise, it is recommended to call `localVue.use()` in `createMockApolloProvider()` to only load it when it is necessary. ```javascript import VueApollo from 'vue-apollo'; @@ -861,7 +932,7 @@ describe('Some component', () => { }); ``` -Within `createMockApolloProvider`-factory, we need to define an array of _handlers_ for every query or mutation: +In the `createMockApolloProvider`-factory, we need to define an array of _handlers_ for every query or mutation: ```javascript import getDesignListQuery from '~/design_management/graphql/queries/get_design_list.query.graphql'; @@ -1251,9 +1322,9 @@ describe('My Index test with `createMockApollo`', () => { ## Handling errors -The GitLab GraphQL mutations currently have two distinct error modes: [Top-level](#top-level-errors) and [errors-as-data](#errors-as-data). +The GitLab GraphQL mutations have two distinct error modes: [Top-level](#top-level-errors) and [errors-as-data](#errors-as-data). -When utilising a GraphQL mutation, we must consider handling **both of these error modes** to ensure that the user receives the appropriate feedback when an error occurs. +When utilising a GraphQL mutation, consider handling **both of these error modes** to ensure that the user receives the appropriate feedback when an error occurs. ### Top-level errors @@ -1261,13 +1332,13 @@ These errors are located at the "top level" of a GraphQL response. These are non #### Handling top-level errors -Apollo is aware of top-level errors, so we are able to leverage Apollo's various error-handling mechanisms to handle these errors (e.g. handling Promise rejections after invoking the [`mutate`](https://www.apollographql.com/docs/react/api/core/ApolloClient/#ApolloClient.mutate) method, or handling the `error` event emitted from the [`ApolloMutation`](https://apollo.vuejs.org/api/apollo-mutation.html#events) component). +Apollo is aware of top-level errors, so we are able to leverage Apollo's various error-handling mechanisms to handle these errors. For example, handling Promise rejections after invoking the [`mutate`](https://www.apollographql.com/docs/react/api/core/ApolloClient/#ApolloClient.mutate) method, or handling the `error` event emitted from the [`ApolloMutation`](https://apollo.vuejs.org/api/apollo-mutation.html#events) component. Because these errors are not intended for users, error messages for top-level errors should be defined client-side. ### Errors-as-data -These errors are nested within the `data` object of a GraphQL response. These are recoverable errors that, ideally, can be presented directly to the user. +These errors are nested in the `data` object of a GraphQL response. These are recoverable errors that, ideally, can be presented directly to the user. #### Handling errors-as-data @@ -1283,7 +1354,7 @@ mutation createNoteMutation($input: String!) { } ``` -Now, when we commit this mutation and errors occur, the response will include `errors` for us to handle: +Now, when we commit this mutation and errors occur, the response includes `errors` for us to handle: ```javascript { @@ -1316,7 +1387,7 @@ When [using Vuex](#using-with-vuex), disable the cache when: - The data is being cached elsewhere - The use case does not need caching -if the data is being cached elsewhere, or if there is simply no need for it for the given use case. +if the data is being cached elsewhere, or if there is no need for it for the given use case. ```javascript import createDefaultClient from '~/lib/graphql'; @@ -1416,7 +1487,7 @@ for your application. To add GraphQL startup calls, we use `add_page_startup_graphql_call` helper where the first parameter is a path to the query, the second one is an object containing query variables. Path to the query is relative to `app/graphql/queries` folder: for example, if we need a -`app/graphql/queries/repository/files.query.graphql` query, the path will be +`app/graphql/queries/repository/files.query.graphql` query, the path is `repository/files`. ```yaml diff --git a/doc/development/fe_guide/icons.md b/doc/development/fe_guide/icons.md index af587a31bbb..821334e3008 100644 --- a/doc/development/fe_guide/icons.md +++ b/doc/development/fe_guide/icons.md @@ -26,12 +26,16 @@ To use a sprite Icon in HAML or Rails we use a specific helper function: sprite_icon(icon_name, size: nil, css_class: '') ``` +<!-- vale gitlab.Spelling = NO --> + - **icon_name**: Use the icon_name for the SVG sprite in the list of ([GitLab SVGs](https://gitlab-org.gitlab.io/gitlab-svgs)). - **size (optional)**: Use one of the following sizes : 16, 24, 32, 48, 72 (this is translated into a `s16` class) - **css_class (optional)**: If you want to add additional CSS classes. +<!-- vale gitlab.Spelling = YES --> + **Example** ```haml @@ -100,7 +104,7 @@ by the examples that follow: - `container` (optional): wraps the loading icon in a container, which centers the loading icon using the `text-center` CSS property. - `color` (optional): either `orange` (default), `light`, or `dark`. - `size` (optional): either `sm` (default), `md`, `lg`, or `xl`. -- `css_class` (optional): defaults to an empty string, but can be useful for utility classes to fine-tune alignment or spacing. +- `css_class` (optional): defaults to an empty string, but can be used for utility classes to fine-tune alignment or spacing. **Example 1:** @@ -164,8 +168,8 @@ export default { ## SVG Illustrations -Please use from now on for any SVG based illustrations simple `img` tags to show an illustration by simply using either `image_tag` or `image_path` helpers. -Please use the class `svg-content` around it to ensure nice rendering. +From now on, use `img` tags to display any SVG based illustrations with either `image_tag` or `image_path` helpers. +Using the class `svg-content` around it ensures nice rendering. ### Usage in HAML/Rails diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 84c1623f8c0..f590473c3c3 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -11,7 +11,7 @@ across the GitLab frontend team. ## Overview -GitLab is built on top of [Ruby on Rails](https://rubyonrails.org) using [Haml](https://haml.info/) and also a JavaScript based Frontend with [Vue.js](https://vuejs.org). +GitLab is built on top of [Ruby on Rails](https://rubyonrails.org). It uses [Haml](https://haml.info/) and a JavaScript0based frontend with [Vue.js](https://vuejs.org). Be wary of [the limitations that come with using Hamlit](https://github.com/k0kubun/hamlit/blob/master/REFERENCE.md#limitations). We also use [SCSS](https://sass-lang.com) and plain JavaScript with modern ECMAScript standards supported through [Babel](https://babeljs.io/) and ES module support through [webpack](https://webpack.js.org/). @@ -21,7 +21,7 @@ Working with our frontend assets requires Node (v10.13.0 or greater) and Yarn ### Browser Support -For our currently-supported browsers, see our [requirements](../../install/requirements.md#supported-web-browsers). +For supported browsers, see our [requirements](../../install/requirements.md#supported-web-browsers). Use [BrowserStack](https://www.browserstack.com/) to test with our supported browsers. Sign in to BrowserStack with the credentials saved in the **Engineering** vault of the GitLab @@ -121,3 +121,7 @@ Our accessibility standards and resources. Frontend internationalization support is described in [this document](../i18n/). The [externalization part of the guide](../i18n/externalization.md) explains the helpers/methods available. + +## [Troubleshooting](troubleshooting.md) + +Running into a Frontend development problem? Check out [this guide](troubleshooting.md) to help resolve your issue. diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index aac2258f3a3..2a1c3daa6b8 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -202,15 +202,15 @@ help identify marks and measures coming from the different apps on the same page ## Best Practices -### Realtime Components +### Real-time Components -When writing code for realtime features we have to keep a couple of things in mind: +When writing code for real-time features we have to keep a couple of things in mind: 1. Do not overload the server with requests. -1. It should feel realtime. +1. It should feel real-time. -Thus, we must strike a balance between sending requests and the feeling of realtime. -Use the following rules when creating realtime solutions. +Thus, we must strike a balance between sending requests and the feeling of real-time. +Use the following rules when creating real-time solutions. 1. The server tells you how much to poll by sending `Poll-Interval` in the header. Use that as your polling interval. This enables system administrators to change the @@ -221,7 +221,7 @@ Use the following rules when creating realtime solutions. 1. Poll on active tabs only. Please use [Visibility](https://github.com/ai/visibilityjs). 1. Use regular polling intervals, do not use backoff polling, or jitter, as the interval is controlled by the server. -1. The backend code is likely to be using etags. You do not and should not check for status +1. The backend code is likely to be using ETags. You do not and should not check for status `304 Not Modified`. The browser transforms it for you. ### Lazy Loading Images @@ -230,12 +230,12 @@ To improve the time to first render we are using lazy loading for images. This w the actual image source on the `data-src` attribute. After the HTML is rendered and JavaScript is loaded, the value of `data-src` is moved to `src` automatically if the image is in the current viewport. -- Prepare images in HTML for lazy loading by renaming the `src` attribute to `data-src` AND adding the class `lazy`. +- Prepare images in HTML for lazy loading by renaming the `src` attribute to `data-src` and adding the class `lazy`. - If you are using the Rails `image_tag` helper, all images are lazy-loaded by default unless `lazy: false` is provided. -If you are asynchronously adding content which contains lazy images then you need to call the function +When asynchronously adding content which contains lazy images, call the function `gl.lazyLoader.searchLazyImages()` which searches for lazy images and loads them if needed. -But in general it should be handled automatically through a `MutationObserver` in the lazy loading function. +In general, it should be handled automatically through a `MutationObserver` in the lazy loading function. ### Animations @@ -243,7 +243,7 @@ Only animate `opacity` & `transform` properties. Other properties (such as `top` Layout to be recalculated, which is much more expensive. For details on this, see "Styles that Affect Layout" in [High Performance Animations](https://www.html5rocks.com/en/tutorials/speed/high-performance-animations/). -If you _do_ need to change layout (for example, a sidebar that pushes main content over), prefer [FLIP](https://aerotwist.com/blog/flip-your-animations/) to change expensive +If you _do_ need to change layout (for example, a sidebar that pushes main content over), prefer [FLIP](https://aerotwist.com/blog/flip-your-animations/). FLIP allows you to change expensive properties once, and handle the actual animation with transforms. ## Reducing Asset Footprint @@ -251,7 +251,7 @@ properties once, and handle the actual animation with transforms. ### Universal code Code that is contained in `main.js` and `commons/index.js` is loaded and -run on _all_ pages. **DO NOT ADD** anything to these files unless it is truly +run on _all_ pages. **Do not add** anything to these files unless it is truly needed _everywhere_. These bundles include ubiquitous libraries like `vue`, `axios`, and `jQuery`, as well as code for the main navigation and sidebar. Where possible we should aim to remove modules from these bundles to reduce our @@ -277,9 +277,9 @@ manually generated webpack bundles. However under this new system you should not ever need to manually add an entry point to the `webpack.config.js` file. NOTE: -If you are unsure what controller and action corresponds to a given page, you -can find this out by inspecting `document.body.dataset.page` in your -browser's developer console while on any page in GitLab. +When unsure what controller and action corresponds to a page, +inspect `document.body.dataset.page` in your +browser's developer console from any page in GitLab. #### Important Considerations @@ -294,7 +294,7 @@ browser's developer console while on any page in GitLab. All GitLab JavaScript files are added with the `defer` attribute. According to the [Mozilla documentation](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/script#attr-defer), this implies that "the script is meant to be executed after the document has - been parsed, but before firing `DOMContentLoaded`". Since the document is already + been parsed, but before firing `DOMContentLoaded`". Because the document is already parsed, `DOMContentLoaded` is not needed to bootstrap applications because all the DOM nodes are already at our disposal. @@ -333,7 +333,7 @@ browser's developer console while on any page in GitLab. action(); ``` - For example, see how we use this in [app/assets/javascripts/pages/projects/graphs/charts/index.js](https://gitlab.com/gitlab-org/gitlab/-/commit/5e90885d6afd4497002df55bf015b338efcfc3c5#02e81de37f5b1716a3ef3222fa7f7edf22c40969_9_8): + For example, see how we use this in [`app/assets/javascripts/pages/projects/graphs/charts/index.js`](https://gitlab.com/gitlab-org/gitlab/-/commit/5e90885d6afd4497002df55bf015b338efcfc3c5#02e81de37f5b1716a3ef3222fa7f7edf22c40969_9_8): ```javascript waitForCSSLoaded(() => { @@ -366,9 +366,9 @@ browser's developer console while on any page in GitLab. ### Code Splitting -For any code that does not need to be run immediately upon page load, (for example, -modals, dropdowns, and other behaviors that can be lazy-loaded), you can split -your module into asynchronous chunks with dynamic import statements. These +Code that does not need to be run immediately upon page load (for example, +modals, dropdowns, and other behaviors that can be lazy-loaded) should be split +into asynchronous chunks with dynamic import statements. These imports return a Promise which is resolved after the script has loaded: ```javascript @@ -377,16 +377,16 @@ import(/* webpackChunkName: 'emoji' */ '~/emoji') .catch(/* report error */) ``` -Please try to use `webpackChunkName` when generating these dynamic imports as +Use `webpackChunkName` when generating dynamic imports as it provides a deterministic filename for the chunk which can then be cached -the browser across GitLab versions. +in the browser across GitLab versions. More information is available in [webpack's code splitting documentation](https://webpack.js.org/guides/code-splitting/#dynamic-imports). ### Minimizing page size -A smaller page size means the page loads faster (especially important on mobile -and poor connections), the page is parsed more quickly by the browser, and less +A smaller page size means the page loads faster, especially on mobile +and poor connections. The page is parsed more quickly by the browser, and less data is used for users with capped data plans. General tips: diff --git a/doc/development/fe_guide/security.md b/doc/development/fe_guide/security.md index 627c5f4d12f..df4613d521d 100644 --- a/doc/development/fe_guide/security.md +++ b/doc/development/fe_guide/security.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Resources -[Mozilla’s HTTP Observatory CLI](https://github.com/mozilla/http-observatory-cli) and the +[Mozilla’s HTTP Observatory CLI](https://github.com/mozilla/http-observatory-cli) and [Qualys SSL Labs Server Test](https://www.ssllabs.com/ssltest/analyze.html) are good resources for finding potential problems and ensuring compliance with security best practices. @@ -76,7 +76,7 @@ such as with reCAPTCHA, which cannot be used without an `iframe`. In order to protect users from [XSS vulnerabilities](https://en.wikipedia.org/wiki/Cross-site_scripting), we intend to disable inline scripts in the future using Content Security Policy. -While inline scripts can be useful, they're also a security concern. If +While inline scripts can make something easier, they're also a security concern. If user-supplied content is unintentionally left un-sanitized, malicious users can inject scripts into the web app. diff --git a/doc/development/fe_guide/style_guide_js.md b/doc/development/fe_guide/style_guide_js.md deleted file mode 100644 index 73a5bea189d..00000000000 --- a/doc/development/fe_guide/style_guide_js.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'style/javascript.md' ---- - -This document was moved to [another location](style/javascript.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/fe_guide/style_guide_scss.md b/doc/development/fe_guide/style_guide_scss.md deleted file mode 100644 index eaa6d33fb43..00000000000 --- a/doc/development/fe_guide/style_guide_scss.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'style/scss.md' ---- - -This document was moved to [another location](style/scss.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/fe_guide/testing.md b/doc/development/fe_guide/testing.md deleted file mode 100644 index 457d15235ae..00000000000 --- a/doc/development/fe_guide/testing.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../testing_guide/frontend_testing.md' ---- - -This document was moved to [another location](../testing_guide/frontend_testing.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/fe_guide/tooling.md b/doc/development/fe_guide/tooling.md index d33022b9355..7a2d8fccdbf 100644 --- a/doc/development/fe_guide/tooling.md +++ b/doc/development/fe_guide/tooling.md @@ -111,7 +111,7 @@ preferred editor (all major editors are supported) accordingly. We suggest setting up Prettier to run when each file is saved. For instructions about using Prettier in your preferred editor, see the [Prettier documentation](https://prettier.io/docs/en/editors.html). -Please take care that you only let Prettier format the same file types as the global Yarn script does (`.js`, `.vue`, `.graphql`, and `.scss`). In VSCode by example you can easily exclude file formats in your settings file: +Please take care that you only let Prettier format the same file types as the global Yarn script does (`.js`, `.vue`, `.graphql`, and `.scss`). For example, you can exclude file formats in your Visual Studio Code settings file: ```json "prettier.disableLanguages": [ @@ -128,13 +128,13 @@ The following yarn scripts are available to do global formatting: yarn prettier-staged-save ``` -Updates all currently staged files (based on `git diff`) with Prettier and saves the needed changes. +Updates all staged files (based on `git diff`) with Prettier and saves the needed changes. ```shell yarn prettier-staged ``` -Checks all currently staged files (based on `git diff`) with Prettier and log which files would need manual updating to the console. +Checks all staged files (based on `git diff`) with Prettier and log which files would need manual updating to the console. ```shell yarn prettier-all diff --git a/doc/development/fe_guide/troubleshooting.md b/doc/development/fe_guide/troubleshooting.md new file mode 100644 index 00000000000..abaf9cd68c7 --- /dev/null +++ b/doc/development/fe_guide/troubleshooting.md @@ -0,0 +1,41 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Troubleshooting + +Running into a problem? Maybe this will help ¯\＿(ツ)＿/¯. + +## Troubleshooting issues + +### This guide doesn't contain the issue I ran into + +If you run into a Frontend development issue that is not in this guide, please consider updating this guide with your issue and possible remedies. This way future adventurers can face these dragons with more success, being armed with your experience and knowledge. + +## Testing issues + +### ``Property or method `nodeType` is not defined`` but I'm not using `nodeType` anywhere + +This issue can happen in Vue component tests, when an expectation fails, but there is an error thrown when +Jest tries to pretty print the diff in the console. It's been noted that using `toEqual` with an array as a +property might also be a contributing factor. + +See [this video](https://youtu.be/-BkEhghP-kM) for an in-depth overview and investigation. + +**Remedy - Try cloning the object that has Vue watchers** + +```patch +- expect(wrapper.find(ChildComponent).props()).toEqual(...); ++ expect(cloneDeep(wrapper.find(ChildComponent).props())).toEqual(...) +``` + +**Remedy - Try using `toMatchObject` instead of `toEqual`** + +```patch +- expect(wrapper.find(ChildComponent).props()).toEqual(...); ++ expect(wrapper.find(ChildComponent).props()).toMatchObject(...); +``` + +Please note that `toMatchObject` actually changes the nature of the assertion and won't fail if some items are **missing** from the expectation. diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index b3fbb9556a9..79ccd966bca 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -22,7 +22,7 @@ All new features built with Vue.js must follow a [Flux architecture](https://fac The main goal we are trying to achieve is to have only one data flow and only one data entry. In order to achieve this goal we use [vuex](#vuex). -You can also read about this architecture in Vue docs about +You can also read about this architecture in Vue documentation about [state management](https://vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch) and about [one way data flow](https://vuejs.org/v2/guide/components.html#One-Way-Data-Flow). @@ -70,7 +70,7 @@ The advantage of providing data from the DOM to the Vue instance through `props` function instead of querying the DOM inside the main Vue component is avoiding the need to create a fixture or an HTML element in the unit test, which makes the tests easier. -See the following example, also, please refer to our [Vue style guide](style/vue.md#basic-rules) for +See the following example. Also, please refer to our [Vue style guide](style/vue.md#basic-rules) for additional information on why we explicitly declare the data being passed into the Vue app; ```javascript @@ -101,8 +101,8 @@ across the codebase. #### Accessing the `gl` object -When we need to query the `gl` object for data that doesn't change during the application's life -cycle, we should do it in the same place where we query the DOM. By following this practice, we can +We query the `gl` object for data that doesn't change during the application's life +cycle in the same place we query the DOM. By following this practice, we can avoid the need to mock the `gl` object, which makes tests easier. It should be done while initializing our Vue instance, and the data should be provided as `props` to the main component: @@ -148,8 +148,8 @@ This approach has a few benefits: - Arbitrarily deeply nested components can opt-in and access the flag without intermediate components being aware of it (c.f. passing the flag down via props). -- Good testability, since the flag can be provided to `mount`/`shallowMount` - from `vue-test-utils` simply as a prop. +- Good testability, because the flag can be provided to `mount`/`shallowMount` + from `vue-test-utils` as a prop. ```javascript import { shallowMount } from '@vue/test-utils'; @@ -207,20 +207,20 @@ Based on the Vue guidance: such as `user: new User()`. - **Do not** add new JavaScript class implementations. - **Do** use [GraphQL](../api_graphql_styleguide.md), [Vuex](vuex.md) or a set of components if -cannot use simple primitives or objects. +cannot use primitives or objects. - **Do** maintain existing implementations using such approaches. - **Do** Migrate components to a pure object model when there are substantial changes to it. -- **Do** add business logic to helpers or utils, so you can test them separately from your component. +- **Do** add business logic to helpers or utilities, so you can test them separately from your component. #### Why There are additional reasons why having a JavaScript class presents maintainability issues on a huge codebase: -- Once a class is created, it is easy to extend it in a way that can infringe Vue reactivity and best practices. +- After a class is created, it can be extended in a way that can infringe Vue reactivity and best practices. - A class adds a layer of abstraction, which makes the component API and its inner workings less clear. -- It makes it harder to test. Since the class is instantiated by the component data function, it is +- It makes it harder to test. Because the class is instantiated by the component data function, it is harder to 'manage' component and class separately. -- Adding OOP to a functional codebase adds yet another way of writing code, reducing consistency and clarity. +- Adding Object Oriented Principles (OOP) to a functional codebase adds yet another way of writing code, reducing consistency and clarity. ## Style guide @@ -234,8 +234,8 @@ for guidelines and best practices for testing your Vue components. Each Vue component has a unique output. This output is always present in the render function. -Although we can test each method of a Vue component individually, our goal must be to test the output -of the render/template function, which represents the state at all times. +Although each method of a Vue component can be tested individually, our goal is to test the output +of the render function, which represents the state at all times. Here's an example of a well structured unit test for [this Vue component](#appendix---vue-component-subject-under-test): @@ -366,7 +366,7 @@ component under test, with the `computed` property, for example). Remember to us ### Events -We should test for events emitted in response to an action within our component, this is useful to +We should test for events emitted in response to an action in our component. This is used to verify the correct events are being fired with the correct arguments. For any DOM events we should use [`trigger`](https://vue-test-utils.vuejs.org/api/wrapper/#trigger) @@ -416,7 +416,7 @@ You should only apply to be a Vue.js expert when your own merge requests and you > This section is added temporarily to support the efforts to migrate the codebase from Vue 2.x to Vue 3.x -Currently, we recommend to minimize adding certain features to the codebase to prevent increasing +We recommend to minimize adding certain features to the codebase to prevent increasing the tech debt for the eventual migration: - filters; diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index 9d2f3b27968..90bf46d9ad9 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -26,7 +26,7 @@ Component's computed properties / methods or external helpers. **What to use instead** -Vue docs recommend using [mitt](https://github.com/developit/mitt) library. It's relatively small (200 bytes gzipped) and has a simple API: +Vue documentation recommends using the [mitt](https://github.com/developit/mitt) library. It's relatively small (200 bytes gzipped) and has a clear API: ```javascript import mitt from 'mitt' @@ -51,9 +51,9 @@ emitter.off('foo', onFoo) // unlisten **Event hub factory** -To make it easier for you to migrate existing event hubs to the new recommended approach, or simply -to create new ones, we have created a factory that you can use to instantiate a new mitt-based -event hub. +We have created a factory that you can use to instantiate a new mitt-based event hub. +This makes it easier to migrate existing event hubs to the new recommended approach, or +to create new ones. ```javascript import createEventHub from '~/helpers/event_hub_factory'; @@ -88,7 +88,7 @@ It is not recommended to replace stateful components with functional components **Why?** -In Vue 2.6 `slot` attribute was already deprecated in favor of `v-slot` directive but its usage is still allowed and sometimes we prefer using them because it simplifies unit tests (with old syntax, slots are rendered on `shallowMount`). However, in Vue 3 we can't use old syntax anymore. +In Vue 2.6 `slot` attribute was already deprecated in favor of `v-slot` directive. The `slot` attribute usage is still allowed and sometimes we prefer using it because it simplifies unit tests (with old syntax, slots are rendered on `shallowMount`). However, in Vue 3 we can't use old syntax anymore. **What to use instead** diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 1d83335291a..cc1d9ccab77 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -146,7 +146,7 @@ The only way to change state in a Vuex store is by committing a mutation. Most mutations are committed from an action using `commit`. If you don't have any asynchronous operations, you can call mutations from a component using the `mapMutations` helper. -See the Vuex docs for examples of [committing mutations from components](https://vuex.vuejs.org/guide/mutations.html#committing-mutations-in-components). +See the Vuex documentation for examples of [committing mutations from components](https://vuex.vuejs.org/guide/mutations.html#committing-mutations-in-components). #### Naming Pattern: `REQUEST` and `RECEIVE` namespaces @@ -271,7 +271,7 @@ import { mapGetters } from 'vuex'; ### `mutation_types.js` -From [vuex mutations docs](https://vuex.vuejs.org/guide/mutations.html): +From [Vuex mutations documentation](https://vuex.vuejs.org/guide/mutations.html): > It is a commonly seen pattern to use constants for mutation types in various Flux implementations. > This allows the code to take advantage of tooling like linters, and putting all constants in a > single file allows your collaborators to get an at-a-glance view of what mutations are possible @@ -429,7 +429,7 @@ export default { #### Testing Vuex concerns -Refer to [Vuex docs](https://vuex.vuejs.org/guide/testing.html) regarding testing Actions, Getters and Mutations. +Refer to [Vuex documentation](https://vuex.vuejs.org/guide/testing.html) regarding testing Actions, Getters and Mutations. #### Testing components that need a store diff --git a/doc/development/feature_flags.md b/doc/development/feature_flags.md deleted file mode 100644 index 7456e8df8d9..00000000000 --- a/doc/development/feature_flags.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'feature_flags/index.md' ---- - -This document was moved to [another location](feature_flags/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index adcf3175c45..cda4f4c905c 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -11,12 +11,12 @@ info: "See the Technical Writers assigned to Development Guidelines: https://abo To be able to turn on/off features behind feature flags in any of the GitLab Inc. provided environments such as staging and production, you need to -have access to the [Chatops](../chatops_on_gitlabcom.md) bot. The Chatops bot +have access to the [ChatOps](../chatops_on_gitlabcom.md) bot. The ChatOps bot is currently running on the ops instance, which is different from <https://gitlab.com> or <https://dev.gitlab.org>. -Follow the Chatops document to [request access](../chatops_on_gitlabcom.md#requesting-access). +Follow the ChatOps document to [request access](../chatops_on_gitlabcom.md#requesting-access). -Once you are added to the project test if your access propagated, +After you are added to the project test if your access propagated, run: ```shell @@ -37,7 +37,7 @@ easier to measure the impact of both separately. The GitLab feature library (using [Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature Flags process](process.md) guide) supports rolling out changes to a percentage of -time to users. This in turn can be controlled using [GitLab Chatops](../../ci/chatops/README.md). +time to users. This in turn can be controlled using [GitLab ChatOps](../../ci/chatops/README.md). For an up to date list of feature flag commands please see [the source code](https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/commands/feature.rb). @@ -48,7 +48,7 @@ If you get an error "Whoops! This action is not allowed. This incident will be reported." that means your Slack account is not allowed to change feature flags or you do not [have access](#access). -### Enabling a feature for preproduction testing +### Enabling a feature for pre-production testing As a first step in a feature rollout, you should enable the feature on <https://about.staging.gitlab.com> and <https://dev.gitlab.org>. @@ -62,7 +62,7 @@ a (very) rough estimate of how your feature will look/behave on GitLab.com. Both of these instances are connected to Sentry so make sure you check the projects there for any exceptions while testing your feature after enabling the feature flag. -For these preproduction environments, the commands should be run in a +For these pre-production environments, the commands should be run in a Slack channel for the stage the feature is relevant to. For example, use the `#s_monitor` channel for features developed by the Monitor stage, Health group. @@ -77,7 +77,7 @@ To enable a feature for 25% of all users, run the following in Slack: ### Enabling a feature for GitLab.com When a feature has successfully been -[enabled on a preproduction](#enabling-a-feature-for-preproduction-testing) +[enabled on a pre-production](#enabling-a-feature-for-pre-production-testing) environment and verified as safe and working, you can roll out the change to GitLab.com (production). @@ -226,11 +226,27 @@ you should fully roll out the feature by enabling the flag **globally** by runni This changes the feature flag state to be **enabled** always, which overrides the existing gates (e.g. `--group=gitlab-org`) in the above processes. +##### Disabling feature flags + +To disable a feature flag that has been globally enabled you can run: + +```shell +/chatops run feature set some_feature false +``` + +To disable a feature flag that has been enabled for a specific project you can run: + +```shell +/chatops run feature set --group=gitlab-org some_feature false +``` + +You cannot selectively disable feature flags for a specific project/group/user without applying a [specific method of implementing](development.md#selectively-disable-by-actor) the feature flags. + ### Feature flag change logging -#### Chatops level +#### ChatOps level -Any feature flag change that affects GitLab.com (production) via [Chatops](https://gitlab.com/gitlab-com/chatops) +Any feature flag change that affects GitLab.com (production) via [ChatOps](https://gitlab.com/gitlab-com/chatops) is automatically logged in an issue. The issue is created in the @@ -243,7 +259,7 @@ The issue is then also posted to the GitLab internal marker to make the change even more visible. Changes to the issue format can be submitted in the -[Chatops project](https://gitlab.com/gitlab-com/chatops). +[ChatOps project](https://gitlab.com/gitlab-com/chatops). #### Instance level diff --git a/doc/development/feature_flags/development.md b/doc/development/feature_flags/development.md index dd732a08c72..0cdfa3e68d7 100644 --- a/doc/development/feature_flags/development.md +++ b/doc/development/feature_flags/development.md @@ -66,6 +66,13 @@ Feature.disabled?(:my_ops_flag, project, type: :ops) push_frontend_feature_flag(:my_ops_flag, project, type: :ops) ``` +### `experiment` type + +`experiment` feature flags are used for A/B testing on GitLab.com. + +An `experiment` feature flag should conform to the same standards as a `development` feature flag, +although the interface has some differences. More information can be found in the [experiment guide](../experiment_guide/index.md). + ## Feature flag definition and validation > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229161) in GitLab 13.3. @@ -321,6 +328,28 @@ Feature.enabled?(:feature_flag, group) Feature.enabled?(:feature_flag, user) ``` +#### Selectively disable by actor + +By default you cannot selectively disable a feature flag by actor. + +```shell +# This will not work how you would expect. +/chatops run feature set some_feature true +/chatops run feature set --project=gitlab-org/gitlab some_feature false +``` + +However, if you add two feature flags, you can write your conditional statement in such a way that the equivalent selective disable is possible. + +```ruby +Feature.enabled?(:a_feature, project) && Feature.disabled?(:a_feature_override, project) +``` + +```shell +# This will enable a feature flag globally, except for gitlab-org/gitlab +/chatops run feature set a_feature true +/chatops run feature set --project=gitlab-org/gitlab a_feature_override true +``` + ### Enable additional objects as actors To use feature gates based on actors, the model needs to respond to @@ -398,6 +427,9 @@ to ensure the feature works properly. When using the testing environment, all feature flags are enabled by default. +WARNING: +This does not apply to end-to-end (QA) tests, which [do not disable feature flags by default](#end-to-end-qa-tests). There is a different [process for using feature flags in end-to-end tests](../testing_guide/end_to_end/feature_flags.md). + To disable a feature flag in a test, use the `stub_feature_flags` helper. For example, to globally disable the `ci_live_trace` feature flag in a test: @@ -545,3 +577,17 @@ a mode that is used by `production` and `development`. You should use this mode only when you really want to tests aspects of Flipper with how it interacts with `ActiveRecord`. + +### End-to-end (QA) tests + +Toggling feature flags works differently in end-to-end (QA) tests. The end-to-end test framework does not have direct access to +Rails or the database, so it can't use Flipper. Instead, it uses [the public API](../../api/features.md#set-or-create-a-feature). Each end-to-end test can [enable or disable a feature flag during the test](../testing_guide/end_to_end/feature_flags.md). Alternatively, you can enable or disable a feature flag before one or more tests when you [run them from your GitLab repository's `qa` directory](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled-or-disabled), or if you [run the tests via GitLab QA](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#running-tests-with-a-feature-flag-enabled). + +[As noted above, feature flags are not enabled by default in end-to-end tests.](#feature-flags-in-tests) +This means that end-to-end tests will run with feature flags in the default state implemented in the source +code, or with the feature flag in its current state on the GitLab instance under test, unless the +test is written to enable/disable a feature flag explicitly. + +When a feature flag is changed on Staging or on GitLab.com, a Slack message will be posted to the `#qa-staging` or `#qa-production` channels to inform +the pipeline triage DRI so that they can more easily determine if any failures are related to a feature flag change. However, if you are working on a change you can +help to avoid unexpected failures by [confirming that the end-to-end tests pass with a feature flag enabled.](../testing_guide/end_to_end/feature_flags.md#confirming-that-end-to-end-tests-pass-with-a-feature-flag-enabled) diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index e93a5b3de1b..4890cc5da35 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -31,7 +31,8 @@ In all cases, those working on the changes should ask themselves: > Why do I need to add a feature flag? If I don't add one, what options do I have to control the impact on application reliability, and user experience? -For perspective on why we limit our use of feature flags please see the following [video](https://www.youtube.com/watch?v=DQaGqyolOd8). +For perspective on why we limit our use of feature flags please see +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Feature flags only when needed](https://www.youtube.com/watch?v=DQaGqyolOd8). In case you are uncertain if a feature flag is necessary, simply ask about this in an early merge request, and those reviewing the changes will likely provide you with an answer. diff --git a/doc/development/feature_flags/process.md b/doc/development/feature_flags/process.md index 7e6299c193c..65a6d01e0a2 100644 --- a/doc/development/feature_flags/process.md +++ b/doc/development/feature_flags/process.md @@ -155,7 +155,7 @@ It may seem like feature flags are configuration, which goes against our [conven principle. However, configuration is by definition something that is user-manageable. Feature flags are not intended to be user-editable. Instead, they are intended as a tool for Engineers and Site Reliability Engineers to use to de-risk their changes. Feature flags are the shim that gets us -to Continuous Delivery with our mono repo and without having to deploy the entire codebase on every change. +to Continuous Delivery with our monorepo and without having to deploy the entire codebase on every change. Feature flags are created to ensure that we can safely rollout our work on our terms. If we use Feature Flags as a configuration, we are doing it wrong and are indeed in violation of our principles. If something needs to be configured, we should intentionally make it configuration from the @@ -173,5 +173,5 @@ Some of the benefits of using development-type feature flags are: 1. Improved throughput: when a change is less risky because a flag exists, theoretical tests about scalability can potentially become unnecessary or less important. This allows an engineer to potentially test a feature on a small project, monitor the impact, and proceed. The alternative might - be to build complex benchmarks locally, or on staging, or on another GitLab deployment, which has an - outsized impact on the time it can take to build and release a feature. + be to build complex benchmarks locally, or on staging, or on another GitLab deployment, which has a + large impact on the time it can take to build and release a feature. diff --git a/doc/development/filtering_by_label.md b/doc/development/filtering_by_label.md index b10e2f6e082..2b9c7efc087 100644 --- a/doc/development/filtering_by_label.md +++ b/doc/development/filtering_by_label.md @@ -137,7 +137,7 @@ object, so the number of combinations would balloon further. ### Attempt B2: store label titles for each object -From the perspective of updating the labelable object, this is the worst +From the perspective of updating the object, this is the worst option. We have to bulk update the objects when: 1. The objects are moved from one project to another. @@ -159,7 +159,7 @@ However, at present, the disadvantages outweigh the advantages. ## Conclusion -We have yet to find a method that is demonstratably better than the current +We have yet to find a method that is demonstrably better than the current method, when considering: 1. Query performance. diff --git a/doc/development/foreign_keys.md b/doc/development/foreign_keys.md index 37764a12f97..abe191ace5e 100644 --- a/doc/development/foreign_keys.md +++ b/doc/development/foreign_keys.md @@ -75,7 +75,7 @@ your models _unless_ absolutely required and only when approved by database specialists. For example, if each row in a table has a corresponding file on a file system it may be tempting to add a `after_destroy` hook. This however introduces non database logic to a model, and means we can no longer rely on -foreign keys to remove the data as this would result in the filesystem data +foreign keys to remove the data as this would result in the file system data being left behind. In such a case you should use a service class instead that takes care of removing non database data. diff --git a/doc/development/frontend.md b/doc/development/frontend.md deleted file mode 100644 index 040004a6917..00000000000 --- a/doc/development/frontend.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'fe_guide/index.md' ---- - -This document was moved to [another location](fe_guide/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md index c4a9ec4c454..1615c96309c 100644 --- a/doc/development/gemfile.md +++ b/doc/development/gemfile.md @@ -18,3 +18,16 @@ dependencies and build times. ## License compliance Refer to [licensing guidelines](licensing.md) for ensuring license compliance. + +## Upgrade Rails + +When upgrading the Rails gem and its dependencies, you also should update the following: + +- The [`Gemfile` in the `qa` directory](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/Gemfile). +- The [`Gemfile` in Gitaly Ruby](https://gitlab.com/gitlab-org/gitaly/-/blob/master/ruby/Gemfile), + to ensure that we ship only one version of these gems. + +You should also update NPM packages that follow the current version of Rails: + +- `@rails/ujs` +- `@rails/actioncable` diff --git a/doc/development/geo.md b/doc/development/geo.md index ed1837f9564..05fadcad08a 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -4,7 +4,7 @@ group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Geo (development) **(PREMIUM ONLY)** +# Geo (development) **(PREMIUM SELF)** Geo connects GitLab instances together. One GitLab instance is designated as a **primary** node and can be run with multiple @@ -305,7 +305,7 @@ events include: - Repositories Changed event - Repository Created event - Hashed Storage Migrated event -- Lfs Object Deleted event +- LFS Object Deleted event - Hashed Storage Attachments event - Job Artifact Deleted event - Upload Deleted event diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index 148953dc418..c2f267f0786 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -279,17 +279,32 @@ For example, to add support for files referenced by a `Widget` model with a unless table_exists?(:widget_registry) ActiveRecord::Base.transaction do create_table :widget_registry, id: :bigserial, force: :cascade do |t| - t.integer :widget_id, null: false + t.bigint :widget_id, null: false + t.datetime_with_timezone :created_at, null: false + t.datetime_with_timezone :last_synced_at + t.datetime_with_timezone :retry_at + t.datetime_with_timezone :verified_at + t.datetime_with_timezone :verification_started_at + t.datetime_with_timezone :verification_retry_at t.integer :state, default: 0, null: false, limit: 2 + t.integer :verification_state, default: 0, null: false, limit: 2 t.integer :retry_count, default: 0, limit: 2 - t.datetime_with_timezone :retry_at - t.datetime_with_timezone :last_synced_at - t.datetime_with_timezone :created_at, null: false + t.integer :verification_retry_count, default: 0, limit: 2 + t.boolean :checksum_mismatch + t.binary :verification_checksum + t.binary :verification_checksum_mismatched + t.string :verification_failure, limit: 255 t.text :last_sync_failure t.index :widget_id, name: :index_widget_registry_on_widget_id, unique: true t.index :retry_at t.index :state + # To optimize performance of WidgetRegistry.verification_failed_batch + t.index :verification_retry_at, name: :widget_registry_failed_verification, order: "NULLS FIRST", where: "((state = 2) AND (verification_state = 3))" + # To optimize performance of WidgetRegistry.needs_verification_count + t.index :verification_state, name: :widget_registry_needs_verification, where: "((state = 2) AND (verification_state = ANY (ARRAY[0, 3])))" + # To optimize performance of WidgetRegistry.verification_pending_batch + t.index :verified_at, name: :widget_registry_pending_verification, order: "NULLS FIRST", where: "((state = 2) AND (verification_state = 0))" end end end @@ -310,6 +325,10 @@ For example, to add support for files referenced by a `Widget` model with a class Geo::WidgetRegistry < Geo::BaseRegistry include Geo::ReplicableRegistry + # TODO: Include VerificationState in VerifiableRegistry + # https://gitlab.com/gitlab-org/gitlab/-/issues/298811 + include ::Gitlab::Geo::VerificationState + include ::Geo::VerifiableRegistry MODEL_CLASS = ::Widget MODEL_FOREIGN_KEY = :widget_id @@ -366,6 +385,7 @@ For example, to add support for files referenced by a `Widget` model with a end include_examples 'a Geo framework registry' + include_examples 'a Geo verifiable registry' describe '.find_registry_differences' do ... # To be implemented diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index 5d062d7404e..98649f08dea 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -12,17 +12,18 @@ Workhorse and GitLab Shell. ## Deep Dive -In May 2019, Bob Van Landuyt hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) -on the [Gitaly project](https://gitlab.com/gitlab-org/gitaly) and how to contribute to it as a -Ruby developer, to share his domain specific knowledge with anyone who may work in this part of the +In May 2019, <!-- vale gitlab.Spelling = NO --> Bob Van Landuyt <!-- vale gitlab.Spelling = YES --> +hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) +on the [Gitaly project](https://gitlab.com/gitlab-org/gitaly). It included how to contribute to it as a +Ruby developer, and shared domain-specific knowledge with anyone who may work in this part of the codebase in the future. -You can find the [recording on YouTube](https://www.youtube.com/watch?v=BmlEWFS8ORo), and the slides +You can find the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=BmlEWFS8ORo), and the slides on [Google Slides](https://docs.google.com/presentation/d/1VgRbiYih9ODhcPnL8dS0W98EwFYpJ7GXMPpX-1TM6YE/edit) and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/a4fdb1026278bda5c1c5bb574379cf80/Create_Deep_Dive__Gitaly_for_Create_Ruby_Devs.pdf). Everything covered in this deep dive was accurate as of GitLab 11.11, and while specific details may -have changed since then, it should still serve as a good introduction. +have changed, it should still serve as a good introduction. ## Beginner's guide @@ -43,7 +44,7 @@ needs direct access to the Git repository *must* be implemented in Gitaly, and exposed via an RPC. It's often easier to develop a new feature in Gitaly if you make the changes to -GitLab that will use the new feature in a separate merge request, to be merged +GitLab that intends to use the new feature in a separate merge request, to be merged immediately after the Gitaly one. This allows you to test your changes before they are merged. @@ -107,9 +108,9 @@ bundle exec rake gitlab:features:disable_rugged Most of this code exists in the `lib/gitlab/git/rugged_impl` directory. NOTE: -You should NOT need to add or modify code related to -Rugged unless explicitly discussed with the [Gitaly -Team](https://gitlab.com/groups/gl-gitaly/group_members). This code does +You should *not* need to add or modify code related to +Rugged unless explicitly discussed with the +[Gitaly Team](https://gitlab.com/groups/gl-gitaly/group_members). This code does NOT work on GitLab.com or other GitLab instances that do not use NFS. ## `TooManyInvocationsError` errors @@ -129,7 +130,7 @@ Isolate the source of the n+1 problem. This is normally a loop that results in G element in an array. If you are unable to isolate the problem, please contact a member of the [Gitaly Team](https://gitlab.com/groups/gl-gitaly/group_members) for assistance. -Once the source has been found, wrap it in an `allow_n_plus_1_calls` block, as follows: +After the source has been found, wrap it in an `allow_n_plus_1_calls` block, as follows: ```ruby # n+1: link to n+1 issue @@ -139,7 +140,7 @@ Gitlab::GitalyClient.allow_n_plus_1_calls do end ``` -Once the code is wrapped in this block, this code path is excluded from n+1 detection. +After the code is wrapped in this block, this code path is excluded from n+1 detection. ## Request counts @@ -164,13 +165,13 @@ end Normally, GitLab CE/EE tests use a local clone of Gitaly in `tmp/tests/gitaly` pinned at the version specified in `GITALY_SERVER_VERSION`. The `GITALY_SERVER_VERSION` file supports also -branches and SHA to use a custom commit in <https://gitlab.com/gitlab-org/gitaly>. +branches and SHA to use a custom commit in [the repository](https://gitlab.com/gitlab-org/gitaly). NOTE: With the introduction of auto-deploy for Gitaly, the format of `GITALY_SERVER_VERSION` was aligned with Omnibus syntax. It no longer supports `=revision`, it evaluates the file content as a Git -reference (branch or SHA). Only if it matches a semver does it prepend a `v`. +reference (branch or SHA). Only if it matches a semantic version does it prepend a `v`. If you want to run tests locally against a modified version of Gitaly you can replace `tmp/tests/gitaly` with a symlink. This is much faster @@ -195,7 +196,7 @@ Note that CI tests do not use your locally modified version of Gitaly. To use a custom Gitaly version in CI you need to update GITALY_SERVER_VERSION as described at the beginning of this section. -To use a different Gitaly repository, e.g., if your changes are present +To use a different Gitaly repository, such as if your changes are present on a fork, you can specify a `GITALY_REPO_URL` environment variable when running tests: @@ -218,7 +219,7 @@ as a [CI environment variable](../ci/variables/README.md#gitlab-cicd-environment If you are making changes to the RPC client, such as adding a new endpoint or adding a new parameter to an existing endpoint, follow the guide for -[Gitaly proto](https://gitlab.com/gitlab-org/gitaly/blob/master/proto/README.md). After pushing +[Gitaly protobuf specifications](https://gitlab.com/gitlab-org/gitaly/blob/master/proto/README.md). After pushing the branch with the changes (`new-feature-branch`, for example): 1. Change the `gitaly` line in the Rails' `Gemfile` to: @@ -328,15 +329,15 @@ the integration by using GDK: 1. Uncomment `prometheus_listen_addr` in the configuration file and run `gdk restart gitaly`. 1. Make sure that the flag is not enabled yet: - 1. Perform whatever action is required to trigger your changes (project creation, - submitting commit, observing history, etc.). + 1. Perform whatever action is required to trigger your changes, such as project creation, + submitting commit, or observing history. 1. Check that the list of current metrics has the new counter for the feature flag: ```shell curl --silent "http://localhost:9236/metrics" | grep go_find_all_tags ``` -1. Once you observe the metrics for the new feature flag and it increments, you +1. After you observe the metrics for the new feature flag and it increments, you can enable the new feature: 1. Navigate to GDK's root directory. 1. Start a Rails console: @@ -359,7 +360,7 @@ the integration by using GDK: ``` 1. Exit the Rails console and perform whatever action is required to trigger - your changes (project creation, submitting commit, observing history, etc.). + your changes, such as project creation, submitting commit, or observing history. 1. Verify the feature is on by observing the metrics for it: ```shell diff --git a/doc/development/go_guide/dependencies.md b/doc/development/go_guide/dependencies.md index 72b3f82d86f..c5af21d0772 100644 --- a/doc/development/go_guide/dependencies.md +++ b/doc/development/go_guide/dependencies.md @@ -178,4 +178,4 @@ authenticate requests made over HTTP. Go rejects HTTP-only entries in `GOPROXY` that have embedded credentials. In a future version, Go may add support for arbitrary authentication headers. -Follow [golang/go#26232](https://github.com/golang/go/issues/26232) for details. +Follow [`golang/go#26232`](https://github.com/golang/go/issues/26232) for details. diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 68210c08a00..f352db918ed 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -63,10 +63,9 @@ of possible security breaches in our code: Remember to run [SAST](../../user/application_security/sast/index.md) and [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) -**(ULTIMATE)** on your project (or at least the [gosec -analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/gosec)), -and to follow our [Security -requirements](../code_review.md#security-requirements). +**(ULTIMATE)** on your project (or at least the +[`gosec` analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/gosec)), +and to follow our [Security requirements](../code_review.md#security-requirements). Web servers can take advantages of middlewares like [Secure](https://github.com/unrolled/secure). @@ -92,9 +91,9 @@ projects: - Avoid global variables, even in packages. By doing so you introduce side effects if the package is included multiple times. - Use `goimports` before committing. - [goimports](https://godoc.org/golang.org/x/tools/cmd/goimports) + [`goimports`](https://godoc.org/golang.org/x/tools/cmd/goimports) is a tool that automatically formats Go source code using - [Gofmt](https://golang.org/cmd/gofmt/), in addition to formatting import lines, + [`Gofmt`](https://golang.org/cmd/gofmt/), in addition to formatting import lines, adding missing ones and removing unreferenced ones. Most editors/IDEs allow you to run commands before/after saving a file, you can set it @@ -162,8 +161,8 @@ be downloaded repeatedly, which can lead to intermittent problems due to rate limiting or network failures. In these circumstances, you should [cache the downloaded code between](../../ci/caching/index.md#caching-go-dependencies). -There was a [bug on modules -checksums](https://github.com/golang/go/issues/29278) in Go < v1.11.4, so make +There was a +[bug on modules checksums](https://github.com/golang/go/issues/29278) in Go versions earlier than v1.11.4, so make sure to use at least this version to avoid `checksum mismatch` errors. ### ORM @@ -171,7 +170,7 @@ sure to use at least this version to avoid `checksum mismatch` errors. We don't use object-relational mapping libraries (ORMs) at GitLab (except [ActiveRecord](https://guides.rubyonrails.org/active_record_basics.html) in Ruby on Rails). Projects can be structured with services to avoid them. -[pgx](https://github.com/jackc/pgx) should be enough to interact with PostgreSQL +[`pgx`](https://github.com/jackc/pgx) should be enough to interact with PostgreSQL databases. ### Migrations @@ -193,7 +192,7 @@ external dependencies might be worth considering in case we decide to use a spec library or framework: - [Testify](https://github.com/stretchr/testify) -- [httpexpect](https://github.com/gavv/httpexpect) +- [`httpexpect`](https://github.com/gavv/httpexpect) ### Subtests @@ -329,8 +328,8 @@ A few things to keep in mind when adding context: ## CLIs Every Go program is launched from the command line. -[cli](https://github.com/urfave/cli) is a convenient package to create command -line apps. It should be used whether the project is a daemon or a simple cli +[`cli`](https://github.com/urfave/cli) is a convenient package to create command +line apps. It should be used whether the project is a daemon or a simple CLI tool. Flags can be mapped to [environment variables](https://github.com/urfave/cli#values-from-the-environment) directly, which documents and centralizes at the same time all the possible command line @@ -389,7 +388,7 @@ functionality: This gives us a thin abstraction over underlying implementations that is consistent across Workhorse, Gitaly, and, in future, other Go servers. For example, in the case of `gitlab.com/gitlab-org/labkit/tracing` we can switch -from using Opentracing directly to using Zipkin or Gokit's own tracing wrapper +from using `Opentracing` directly to using `Zipkin` or Gokit's own tracing wrapper without changes to the application code, while still keeping the same consistent configuration mechanism (i.e. the `GITLAB_TRACING` environment variable). @@ -489,9 +488,9 @@ The following are some style guidelines that are specific to the Secure Team. ### Code style and format Use `goimports -local gitlab.com/gitlab-org` before committing. -[goimports](https://godoc.org/golang.org/x/tools/cmd/goimports) +[`goimports`](https://godoc.org/golang.org/x/tools/cmd/goimports) is a tool that automatically formats Go source code using -[Gofmt](https://golang.org/cmd/gofmt/), in addition to formatting import lines, +[`Gofmt`](https://golang.org/cmd/gofmt/), in addition to formatting import lines, adding missing ones and removing unreferenced ones. By using the `-local gitlab.com/gitlab-org` option, `goimports` groups locally referenced packages separately from external ones. See diff --git a/doc/development/graphql_guide/batchloader.md b/doc/development/graphql_guide/batchloader.md index e965e678aba..1869c6e6f80 100644 --- a/doc/development/graphql_guide/batchloader.md +++ b/doc/development/graphql_guide/batchloader.md @@ -20,7 +20,7 @@ When implementing a new endpoint we should aim to minimise the number of SQL que Batch loading is useful when a series of queries for inputs `Qα, Qβ, ... Qω` can be combined to a single query for `Q[α, β, ... ω]`. An example of this is lookups by ID, where we can find two users by usernames as cheaply as one, but real-world examples can be more complex. -Batchloading is not suitable when the result sets have different sort-orders, grouping, aggregation or other non-composable features. +Batch loading is not suitable when the result sets have different sort-orders, grouping, aggregation or other non-composable features. There are two ways to use the batch-loader in your code. For simple ID lookups, use `::Gitlab::Graphql::Loaders::BatchModelLoader.new(model, id).find`. For more complex cases, you can use the batch API directly. diff --git a/doc/development/i18n/merging_translations.md b/doc/development/i18n/merging_translations.md index 582c1428bdd..e7d25942143 100644 --- a/doc/development/i18n/merging_translations.md +++ b/doc/development/i18n/merging_translations.md @@ -66,12 +66,14 @@ have been fixed on master. NOTE: These instructions work only for GitLab Team Members. -If for some reason the GitLab integration in CrowdIn does not exist, it can be -recreated by the following steps: - -1. Sign in to GitLab as `gitlab-crowdin-bot` (If you're a GitLab Team Member, find credentials in the GitLab shared [1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams) -1. Sign in to Crowdin with the GitLab integration -1. Navigate to Settings > Integrations > GitLab > Set Up Integration -1. Select `gitlab-org/gitlab` repository -1. On `Select Branches for Translation`, select `master` -1. Ensure the `Service Branch Name` is `master-i18n` +If for some reason the GitLab integration in CrowdIn doesn't exist, you can +recreate it with the following steps: + +1. Sign in to GitLab as `gitlab-crowdin-bot`. (If you're a GitLab Team Member, + find credentials in the GitLab shared + [1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams).) +1. Sign in to CrowdIn with the GitLab integration. +1. Go to **Settings > Integrations > GitLab > Set Up Integration**. +1. Select the `gitlab-org/gitlab` repository. +1. In `Select Branches for Translation`, select `master`. +1. Ensure the `Service Branch Name` is `master-i18n`. diff --git a/doc/development/i18n_guide.md b/doc/development/i18n_guide.md deleted file mode 100644 index ddc91f9308e..00000000000 --- a/doc/development/i18n_guide.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'i18n/index.md' ---- - -This document was moved to [another location](i18n/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/import_project.md b/doc/development/import_project.md index dfe6153ad45..a4917cc0c3d 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -170,7 +170,7 @@ The last option is to import a project using a Rails console: Gitlab::ImportExport::RepoRestorer.new(path_to_bundle: repo_path, shared: shared, - project: project).restore + importable: project).restore ``` We are storing all import failures in the `import_failures` data table. diff --git a/doc/development/instrumentation.md b/doc/development/instrumentation.md index 94b56e10d9e..338ad76d414 100644 --- a/doc/development/instrumentation.md +++ b/doc/development/instrumentation.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Instrumenting Ruby code +# Instrumenting Ruby code **(FREE)** [GitLab Performance Monitoring](../administration/monitoring/performance/index.md) allows instrumenting of both methods and custom blocks of Ruby code. Method instrumentation is the primary form of diff --git a/doc/development/integrations/codesandbox.md b/doc/development/integrations/codesandbox.md index faa1ec0ee3f..3909991eed5 100644 --- a/doc/development/integrations/codesandbox.md +++ b/doc/development/integrations/codesandbox.md @@ -1,14 +1,20 @@ -# Set up local Codesandbox development environment +--- +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- -This guide walks through setting up a local [Codesandbox repository](https://github.com/codesandbox/codesandbox-client) and integrating it with a local GitLab instance. Codesandbox -is used to power the Web IDE's [Live Preview feature](../../user/project/web_ide/index.md#live-preview). Having a local Codesandbox setup is useful for debugging upstream issues or +# Set up local CodeSandbox development environment + +This guide walks through setting up a local [CodeSandbox repository](https://github.com/codesandbox/codesandbox-client) and integrating it with a local GitLab instance. CodeSandbox +is used to power the Web IDE's [Live Preview feature](../../user/project/web_ide/index.md#live-preview). Having a local CodeSandbox setup is useful for debugging upstream issues or creating upstream contributions like [this one](https://github.com/codesandbox/codesandbox-client/pull/5137). ## Initial setup -Before using Codesandbox with your local GitLab instance, you must: +Before using CodeSandbox with your local GitLab instance, you must: -1. Enable HTTPS on your GDK. Codesandbox uses Service Workers that require `https`. +1. Enable HTTPS on your GDK. CodeSandbox uses Service Workers that require `https`. Follow the GDK [NGINX configuration instructions](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/nginx.md) to enable HTTPS for GDK. 1. Clone the [`codesandbox-client` project](https://github.com/codesandbox/codesandbox-client) locally. If you plan on contributing upstream, you might want to fork and clone first. @@ -32,16 +38,16 @@ Before using Codesandbox with your local GitLab instance, you must: You can run `yarn build:clean` to clean up the build assets. -## Use local GitLab instance with local Codesandbox +## Use local GitLab instance with local CodeSandbox -GitLab integrates with two parts of Codesandbox: +GitLab integrates with two parts of CodeSandbox: - An NPM package called `smooshpack` (called `sandpack` in the `codesandbox-client` project). This exposes an entrypoint for us to kick off Codesandbox's bundler. -- A server that houses Codesandbox assets for bundling and previewing. This is hosted +- A server that houses CodeSandbox assets for bundling and previewing. This is hosted on a separate server for security. -Each time you want to run GitLab and Codesandbox together, you need to perform the +Each time you want to run GitLab and CodeSandbox together, you need to perform the steps in the following sections. ### Use local `smooshpack` for GitLab @@ -88,7 +94,7 @@ mkdir node_modules ln -s $PATH_TO_LOCAL_GITLAB/node_modules/core-js ./node_modules/core-js ``` -### Start building codesandbox app assets +### Start building CodeSandbox app assets In the `codesandbox-client` project directory: @@ -98,7 +104,7 @@ cd packages/app yarn start:sandpack-sandbox ``` -### Create HTTPS proxy for Codesandbox `sandpack` assets +### Create HTTPS proxy for CodeSandbox `sandpack` assets Because we need `https`, we need to create a proxy to the webpack server. We can use [`http-server`](https://www.npmjs.com/package/http-server), which can do this proxying @@ -111,7 +117,7 @@ npx http-server --proxy http://localhost:3000 -S -C $PATH_TO_CERT_PEM -K $PATH_T ### Update `bundler_url` setting in GitLab We need to update our `application_setting_implementation.rb` to point to the server that hosts the -Codesandbox `sandpack` assets. For instance, if these assets are hosted by a server at `https://sandpack.local:8044`: +CodeSandbox `sandpack` assets. For instance, if these assets are hosted by a server at `https://sandpack.local:8044`: ```patch diff --git a/app/models/application_setting_implementation.rb b/app/models/application_setting_implementation.rb @@ -133,7 +139,7 @@ index 6eed627b502..1824669e881 100644 NOTE: You can apply this patch by copying it to your clipboard and running `pbpaste | git apply`. -You'll might want to restart the GitLab Rails server after making this change: +You may want to restart the GitLab Rails server after making this change: ```shell gdk restart rails-web diff --git a/doc/development/integrations/jenkins.md b/doc/development/integrations/jenkins.md index a9a1026f1a8..16aba023fab 100644 --- a/doc/development/integrations/jenkins.md +++ b/doc/development/integrations/jenkins.md @@ -21,7 +21,7 @@ brew services start jenkins GitLab does not allow requests to localhost or the local network by default. When running Jenkins on your local machine, you need to enable local access. -1. Log into your GitLab instance as an admin. +1. Log into your GitLab instance as an administrator. 1. Go to **Admin Area > Settings > Network**. 1. Expand **Outbound requests** and check the following checkboxes: @@ -32,7 +32,7 @@ GitLab does not allow requests to localhost or the local network by default. Whe Jenkins uses the GitLab API and needs an access token. -1. Log in to your GitLab instance. +1. Sign in to your GitLab instance. 1. Click on your profile picture, then click **Settings**. 1. Click **Access Tokens**. 1. Create a new Access Token with the **API** scope enabled. Note the value of the token. diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index 48beb526774..bfe523ee390 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -19,10 +19,12 @@ The following are required to install and test the app: - [GDK with Gitpod](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/gitpod.md) documentation. - You **must not** use tunneling tools such as Serveo or `ngrok`. These are + You **must not** use tunneling tools such as + <!-- vale gitlab.Spelling = NO --> Serveo <!-- vale gitlab.Spelling = YES --> + or `ngrok`. These are security risks, and must not be run on developer laptops. - Jira requires all connections to the app host to be over SSL, so if you set up + Jira requires all connections to the app host to be over SSL. If you set up your own environment, remember to enable SSL and an appropriate certificate. ## Install the app in Jira diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index fb9d894d203..1d5aced5869 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -260,6 +260,8 @@ When executing command lines, scanners should use the `debug` level to log the c For instance, the [bundler-audit](https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit) scanner uses the `debug` level to log the command line `bundle audit check --quiet`, and what `bundle audit` writes to the standard output. +If the command line fails, then it should be logged with the `error` log level; +this makes it possible to debug the problem without having to change the log level to `debug` and rerun the scanning job. #### common logutil package diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index 80f632639ca..364e18ad015 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -19,7 +19,7 @@ integration as well as linking to more detailed resources for how to do so. ## Integration Tiers -The security offerings in GitLab are designed for GitLab Gold and GitLab Ultimate users, and the +The security offerings in GitLab are designed for GitLab Ultimate users, and the [DevSecOps](https://about.gitlab.com/handbook/use-cases/#4-devsecops-shift-left-security) use case. All the features are in those tiers. This includes the APIs and standard reporting framework needed to provide a consistent experience for users to easily bring their preferred @@ -75,7 +75,7 @@ and complete an integration with the Secure stage. 1. [Create an issue](https://gitlab.com/gitlab-com/alliances/alliances/-/issues/new?issuable_template=new_partner) using our new partner issue template to begin the discussion. 1. Get a test account to begin developing your integration. You can - request a [GitLab.com Gold Subscription Sandbox](https://about.gitlab.com/partners/integrate/#gitlabcom-gold-subscription-sandbox-request) + request a [GitLab.com Subscription Sandbox](https://about.gitlab.com/partners/integrate/#gitlabcom-subscription-sandbox-request) or an [EE Developer License](https://about.gitlab.com/partners/integrate/#requesting-ee-dev-license-for-rd). 1. Provide a [pipeline job](../../development/pipelines.md) template that users could integrate into their own GitLab pipelines. @@ -114,7 +114,7 @@ and complete an integration with the Secure stage. doing an [Unfiltered blog post](https://about.gitlab.com/handbook/marketing/blog/unfiltered/), doing a co-branded webinar, or producing a co-branded white paper. -We have a [video playlist](https://www.youtube.com/playlist?list=PL05JrBw4t0KpMqYxJiOLz-uBIr5w-yP4A) +We have a <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [video playlist](https://www.youtube.com/playlist?list=PL05JrBw4t0KpMqYxJiOLz-uBIr5w-yP4A) that may be helpful as part of this process. This covers various topics related to integrating your tool. diff --git a/doc/development/internal_api.md b/doc/development/internal_api.md index 43655c37048..b0e8dc0d22f 100644 --- a/doc/development/internal_api.md +++ b/doc/development/internal_api.md @@ -26,7 +26,7 @@ file, and include the token Base64 encoded in a `secret_token` parameter or in the `Gitlab-Shared-Secret` header. NOTE: -The internal API used by GitLab Pages, and GitLab Kubernetes Agent Server (kas) uses JSON Web Token (JWT) +The internal API used by GitLab Pages, and GitLab Kubernetes Agent Server (`kas`) uses JSON Web Token (JWT) authentication, which is different from GitLab Shell. ## Git Authentication @@ -51,7 +51,7 @@ POST /internal/allowed | `key_id` | string | no | ID of the SSH-key used to connect to GitLab Shell | | `username` | string | no | Username from the certificate used to connect to GitLab Shell | | `project` | string | no (if `gl_repository` is passed) | Path to the project | -| `gl_repository` | string | no (if `project` is passed) | Repository identifier (e.g. `project-7`) | +| `gl_repository` | string | no (if `project` is passed) | Repository identifier, such as `project-7` | | `protocol` | string | yes | SSH when called from GitLab Shell, HTTP or SSH when called from Gitaly | | `action` | string | yes | Git command being run (`git-upload-pack`, `git-receive-pack`, `git-upload-archive`) | | `changes` | string | yes | `<oldrev> <newrev> <refname>` when called from Gitaly, the magic string `_any` when called from GitLab Shell | @@ -378,7 +378,7 @@ Example response: > - This feature is not deployed on GitLab.com > - It's not recommended for production use. -The following endpoints are used by the GitLab Kubernetes Agent Server (kas) +The following endpoints are used by the GitLab Kubernetes Agent Server (`kas`) for various purposes. These endpoints are all authenticated using JWT. The JWT secret is stored in a file @@ -390,9 +390,9 @@ The Kubernetes agent is under development and is not recommended for production ### Kubernetes agent information -Called from GitLab Kubernetes Agent Server (kas) to retrieve agent +Called from GitLab Kubernetes Agent Server (`kas`) to retrieve agent information for the given agent token. This returns the Gitaly connection -information for the agent's project in order for kas to fetch and update +information for the agent's project in order for `kas` to fetch and update the agent's configuration. ```plaintext @@ -407,13 +407,13 @@ curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Auth ### Kubernetes agent project information -Called from GitLab Kubernetes Agent Server (kas) to retrieve project +Called from GitLab Kubernetes Agent Server (`kas`) to retrieve project information for the given agent token. This returns the Gitaly -connection for the requested project. GitLab kas uses this to configure +connection for the requested project. GitLab `kas` uses this to configure the agent to fetch Kubernetes resources from the project repository to sync. -Only public projects are currently supported. For private projects, the ability for the +Only public projects are supported. For private projects, the ability for the agent to be authorized is [not yet implemented](https://gitlab.com/gitlab-org/gitlab/-/issues/220912). | Attribute | Type | Required | Description | @@ -432,7 +432,7 @@ curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Auth ### Kubernetes agent usage metrics -Called from GitLab Kubernetes Agent Server (kas) to increase the usage +Called from GitLab Kubernetes Agent Server (`kas`) to increase the usage metric counters. | Attribute | Type | Required | Description | @@ -456,7 +456,7 @@ Cluster. | Attribute | Type | Required | Description | |:----------|:-------|:---------|:------------| -| `alert` | Hash | yes | Alerts detail. Currently same format as [3rd party alert](../operations/incident_management/alert_integrations.md#customize-the-alert-payload-outside-of-gitlab). | +| `alert` | Hash | yes | Alerts detail. Same format as [3rd party alert](../operations/incident_management/integrations.md#customize-the-alert-payload-outside-of-gitlab). | ```plaintext POST internal/kubernetes/modules/cilium_alert diff --git a/doc/development/iterating_tables_in_batches.md b/doc/development/iterating_tables_in_batches.md index 43d7f32ad7f..bae55bccf7c 100644 --- a/doc/development/iterating_tables_in_batches.md +++ b/doc/development/iterating_tables_in_batches.md @@ -46,7 +46,7 @@ all of the arguments that `in_batches` supports. You should always use One should proceed with extra caution, and possibly avoid iterating over a column that can contain duplicate values. When you iterate over an attribute that is not unique, even with the applied max batch size, there is no guarantee that the resulting batches will not surpass it. -The following snippet demonstrates this situation, whe one attempt to select `Ci::Build` entries for users with `id` between `1` and `10,s000`, database returns `1 215 178` +The following snippet demonstrates this situation, when one attempt to select `Ci::Build` entries for users with `id` between `1` and `10,s000`, database returns `1 215 178` matching rows ```ruby diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md index a56a0bd48be..6976dda433e 100644 --- a/doc/development/kubernetes.md +++ b/doc/development/kubernetes.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Kubernetes integration - development guidelines +# Kubernetes integration - development guidelines **(CORE)** This document provides various guidelines when developing for the GitLab [Kubernetes integration](../user/project/clusters/index.md). diff --git a/doc/development/lfs.md b/doc/development/lfs.md index 9df1f659654..eeeebd52ecb 100644 --- a/doc/development/lfs.md +++ b/doc/development/lfs.md @@ -9,13 +9,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Deep Dive In April 2019, Francisco Javier López hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) -on the GitLab [Git LFS](../topics/git/lfs/index.md) implementation to share his domain -specific knowledge with anyone who may work in this part of the codebase in the future. -You can find the [recording on YouTube](https://www.youtube.com/watch?v=Yyxwcksr0Qc), +on the GitLab [Git LFS](../topics/git/lfs/index.md) implementation to share domain-specific +knowledge with anyone who may work in this part of the codebase in the future. +You can find the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=Yyxwcksr0Qc), and the slides on [Google Slides](https://docs.google.com/presentation/d/1E-aw6-z0rYd0346YhIWE7E9A65zISL9iIMAOq2zaw9E/edit) and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/07a89257a140db067bdfb484aecd35e1/Git_LFS_Deep_Dive__Create_.pdf). -Everything covered in this deep dive was accurate as of GitLab 11.10, and while specific -details may have changed since then, it should still serve as a good introduction. +This deep dive was accurate as of GitLab 11.10, and while specific +details may have changed, it should still serve as a good introduction. ## Including LFS blobs in project archives @@ -52,7 +52,7 @@ JSON payload prefaced with `git-archive`. This payload includes the archive already exists in the archive cache, Workhorse sends that file. Otherwise, Workhorse sends the `SendArchiveRequest` to the appropriate Gitaly server. -1. The Gitaly server will call `git archive <ref>` to begin generating +1. The Gitaly server calls `git archive <ref>` to begin generating the Git archive on-the-fly. If the `include_lfs_blobs` flag is enabled, Gitaly enables a custom LFS smudge filter via the `-c filter.lfs.smudge=/path/to/gitaly-lfs-smudge` Git option. @@ -76,7 +76,7 @@ process, which writes the contents to the standard output. 1. The archive data is sent back to the client. In step 7, the `gitaly-lfs-smudge` filter must talk to Workhorse, not to -Rails, or an invalid LFS blob will be saved. To support this, GitLab +Rails, or an invalid LFS blob is saved. To support this, GitLab 13.5 [changed the default Omnibus configuration to have Gitaly talk to the Workhorse](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4592) instead of Rails. @@ -84,6 +84,6 @@ instead of Rails. One side effect of this change: the correlation ID of the original request is not preserved for the internal API requests made by Gitaly (or `gitaly-lfs-smudge`), such as the one made in step 8. The -correlation IDs for those API requests will be random values until [this +correlation IDs for those API requests are random values until [this Workhorse issue](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/309) is resolved. diff --git a/doc/development/logging.md b/doc/development/logging.md index 07ec9d53145..30398eb87a1 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Developers Guide to Logging +# GitLab Developers Guide to Logging **(FREE)** [GitLab Logs](../administration/logs.md) play a critical role for both administrators and GitLab team members to diagnose problems in the field. @@ -291,7 +291,7 @@ method or variable shouldn't be evaluated right away) - Change `Gitlab::ApplicationContext` to accept these new values - Make sure the new attributes are accepted at [`Labkit::Context`](https://gitlab.com/gitlab-org/labkit-ruby/blob/master/lib/labkit/context.rb) -See our [HOWTO: Use Sidekiq metadata logs](https://www.youtube.com/watch?v=_wDllvO_IY0) for further knowledge on +See our <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [HOWTO: Use Sidekiq metadata logs](https://www.youtube.com/watch?v=_wDllvO_IY0) for further knowledge on creating visualizations in Kibana. The fields of the context are currently only logged for Sidekiq jobs triggered @@ -299,6 +299,28 @@ through web requests. See the [follow-up work](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/68) for more information. +### Logging context metadata (through workers) + +Additional metadata can be attached to a worker through the use of the [`ApplicationWorker#log_extra_metadata_on_done`](https://gitlab.com/gitlab-org/gitlab/-/blob/16ecc33341a3f6b6bebdf78d863c5bce76b040d3/app/workers/concerns/application_worker.rb#L31-34) +method. Using this method adds metadata that is later logged to Kibana with the done job payload. + +```ruby +class MyExampleWorker + include ApplicationWorker + + def perform(*args) + # Worker performs work + # ... + + # The contents of value will appear in Kibana under `json.extra.my_example_worker.my_key` + log_extra_metadata_on_done(:my_key, value) + end +end +``` + +Please see [this example](https://gitlab.com/gitlab-org/gitlab/-/blob/16ecc33341a3f6b6bebdf78d863c5bce76b040d3/app/workers/ci/pipeline_artifacts/expire_artifacts_worker.rb#L20-21) +which logs a count of how many artifacts are destroyed per run of the `ExpireArtifactsWorker`. + ## Exception Handling It often happens that you catch the exception and want to track it. diff --git a/doc/development/module_with_instance_variables.md b/doc/development/module_with_instance_variables.md index 75575105178..16c7a807053 100644 --- a/doc/development/module_with_instance_variables.md +++ b/doc/development/module_with_instance_variables.md @@ -121,71 +121,7 @@ module Gitlab end ``` -Now the cop doesn't complain. Here's a bad example which we could rewrite: - -``` ruby -module SpamCheckService - def filter_spam_check_params - @request = params.delete(:request) - @api = params.delete(:api) - @recaptcha_verified = params.delete(:recaptcha_verified) - @spam_log_id = params.delete(:spam_log_id) - end - - def spam_check(spammable, user) - spam_service = SpamService.new(spammable, @request) - - spam_service.when_recaptcha_verified(@recaptcha_verified, @api) do - user.spam_logs.find_by(id: @spam_log_id)&.update!(recaptcha_verified: true) - end - end -end -``` - -There are several implicit dependencies here. First, `params` should be -defined before use. Second, `filter_spam_check_params` should be called -before `spam_check`. These are all implicit and the includer could be using -those instance variables without awareness. - -This should be rewritten like: - -``` ruby -class SpamCheckService - def initialize(request:, api:, recaptcha_verified:, spam_log_id:) - @request = request - @api = api - @recaptcha_verified = recaptcha_verified - @spam_log_id = spam_log_id - end - - def spam_check(spammable, user) - spam_service = SpamService.new(spammable, @request) - - spam_service.when_recaptcha_verified(@recaptcha_verified, @api) do - user.spam_logs.find_by(id: @spam_log_id)&.update!(recaptcha_verified: true) - end - end -end -``` - -And use it like: - -``` ruby -class UpdateSnippetService < BaseService - def execute - # ... - spam = SpamCheckService.new(params.slice!(:request, :api, :recaptcha_verified, :spam_log_id)) - - spam.check(snippet, current_user) - # ... - end -end -``` - -This way, all those instance variables are isolated in `SpamCheckService` -rather than whatever includes the module, and those modules which were also -included, making it much easier to track down any issues, -and reducing the chance of having name conflicts. +Now the cop doesn't complain. ## How to disable this cop diff --git a/doc/development/new_fe_guide/development/testing.md b/doc/development/new_fe_guide/development/testing.md deleted file mode 100644 index 034324989ef..00000000000 --- a/doc/development/new_fe_guide/development/testing.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../../testing_guide/frontend_testing.md' ---- - -This document was moved to [another location](../../testing_guide/frontend_testing.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/new_fe_guide/modules/widget_extensions.md b/doc/development/new_fe_guide/modules/widget_extensions.md index ffcd736511a..375baea0444 100644 --- a/doc/development/new_fe_guide/modules/widget_extensions.md +++ b/doc/development/new_fe_guide/modules/widget_extensions.md @@ -10,13 +10,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Summary -Extensions in the merge request widget allow for others team to quickly and easily add new features -into the widget that will match the existing design and interaction as other extensions. +Extensions in the merge request widget enable you to add new features +into the widget that match the existing design and interaction as other extensions. ## Usage -To use extensions you need to first create a new extension object that will be used to fetch the -data that will be rendered in the extension. See the example file in +To use extensions you need to first create a new extension object to fetch the +data to render in the extension. See the example file in `app/assets/javascripts/vue_merge_request_widget/extensions/issues.js` for a working example. The basic object structure is as below: @@ -36,8 +36,8 @@ export default { }; ``` -Following the same data structure allows each extension to follow the same registering structure -but allows for each extension to manage where it gets its own data from. +By following the same data structure, each extension can follow the same registering structure, +but each extension can manage its data sources. After creating this structure you need to register it. Registering the extension can happen at any point _after_ the widget has been created. diff --git a/doc/development/new_fe_guide/style/html.md b/doc/development/new_fe_guide/style/html.md deleted file mode 100644 index afbfbdcf834..00000000000 --- a/doc/development/new_fe_guide/style/html.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../../fe_guide/style/html.md' ---- - -This document was moved to [another location](../../fe_guide/style/html.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/new_fe_guide/style/index.md b/doc/development/new_fe_guide/style/index.md deleted file mode 100644 index 77700441aa3..00000000000 --- a/doc/development/new_fe_guide/style/index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../../fe_guide/style/index.md' ---- - -This document was moved to [another location](../../fe_guide/style/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/new_fe_guide/style/javascript.md b/doc/development/new_fe_guide/style/javascript.md deleted file mode 100644 index 17722d2c41b..00000000000 --- a/doc/development/new_fe_guide/style/javascript.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../../fe_guide/style/javascript.md' ---- - -This document was moved to [another location](../../fe_guide/style/javascript.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/new_fe_guide/style/prettier.md b/doc/development/new_fe_guide/style/prettier.md deleted file mode 100644 index 6c0f3b77505..00000000000 --- a/doc/development/new_fe_guide/style/prettier.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../../fe_guide/tooling.md#formatting-with-prettier' ---- - -This document was moved to [another location](../../fe_guide/tooling.md#formatting-with-prettier). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/ordering_table_columns.md b/doc/development/ordering_table_columns.md index a0a38acd816..00ce15fcc10 100644 --- a/doc/development/ordering_table_columns.md +++ b/doc/development/ordering_table_columns.md @@ -63,17 +63,17 @@ bits platform and 8 bytes for a 64 bits platform. | Type | Size | Alignment needed | |:-----------------|:-------------------------------------|:-----------| -| smallint | 2 bytes | 1 word | -| integer | 4 bytes | 1 word | -| bigint | 8 bytes | 8 bytes | -| real | 4 bytes | 1 word | -| double precision | 8 bytes | 8 bytes | -| boolean | 1 byte | not needed | -| text / string | variable, 1 byte plus the data | 1 word | -| bytea | variable, 1 or 4 bytes plus the data | 1 word | -| timestamp | 8 bytes | 8 bytes | -| timestamptz | 8 bytes | 8 bytes | -| date | 4 bytes | 1 word | +| `smallint` | 2 bytes | 1 word | +| `integer` | 4 bytes | 1 word | +| `bigint` | 8 bytes | 8 bytes | +| `real` | 4 bytes | 1 word | +| `double precision` | 8 bytes | 8 bytes | +| `boolean` | 1 byte | not needed | +| `text` / `string` | variable, 1 byte plus the data | 1 word | +| `bytea` | variable, 1 or 4 bytes plus the data | 1 word | +| `timestamp` | 8 bytes | 8 bytes | +| `timestamptz` | 8 bytes | 8 bytes | +| `date` | 4 bytes | 1 word | A "variable" size means the actual size depends on the value being stored. If PostgreSQL determines this can be embedded directly into a row it may do so, but diff --git a/doc/development/packages.md b/doc/development/packages.md index aadd71c9ffa..e1fa48cc350 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -70,7 +70,7 @@ The current state of existing package registries availability is: | Maven | Yes | Yes | Yes | | Conan | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/11679) | Yes | | NPM | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36853) | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36853) | -| NuGet | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36423) | No | +| NuGet | Yes | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36425) | | PyPI | Yes | No | No | | Go | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/213900) | No - [open-issue](https://gitlab.com/gitlab-org/gitlab/-/issues/213902) | | Composer | Yes | Yes | No | @@ -191,7 +191,7 @@ support is done by overriding a specific function in the API helpers, like For this authentication mechanism, keep in mind that some clients can send an unauthenticated request first, wait for the 401 Unauthorized response with the [`WWW-Authenticate`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/WWW-Authenticate) field, then send an updated (authenticated) request. This case is more involved as -GitLab needs to handle the 401 Unauthorized response. The [Nuget API](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/nuget_packages.rb) +GitLab needs to handle the 401 Unauthorized response. The [NuGet API](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/nuget_packages.rb) supports this case. #### Authorization @@ -245,8 +245,8 @@ in your local development environment. #### Rate Limits on GitLab.com Package manager clients can make rapid requests that exceed the -[GitLab.com standard API rate limits](../user/gitlab_com/index.md#gitlabcom-specific-rate-limits). -This results in a `429 Too Many Requests` error. +[GitLab.com standard API rate limits](../user/gitlab_com/index.md#gitlabcom-specific-rate-limits). +This results in a `429 Too Many Requests` error. We have opened a set of paths to allow higher rate limits. Unless it is not possible, new package managers should follow these conventions so they can take advantage of the diff --git a/doc/development/performance.md b/doc/development/performance.md index f3ce924de38..3b4525dc8ee 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -50,7 +50,7 @@ GitLab provides built-in tools to help improve performance and availability: - [Service measurement](service_measurement.md) for measuring and logging service execution. GitLab team members can use [GitLab.com's performance monitoring systems](https://about.gitlab.com/handbook/engineering/monitoring/) located at -<https://dashboards.gitlab.net>, this requires you to log in using your +[`dashboards.gitlab.net`](https://dashboards.gitlab.net), this requires you to log in using your `@gitlab.com` email address. Non-GitLab team-members are advised to set up their own Prometheus and Grafana stack. @@ -176,7 +176,7 @@ stackprof tmp/project_policy_spec.rb.dump --graphviz > project_policy_spec.dot dot -Tsvg project_policy_spec.dot > project_policy_spec.svg ``` -To load the profile in [kcachegrind](https://kcachegrind.github.io/): +To load the profile in [KCachegrind](https://kcachegrind.github.io/): ```shell stackprof tmp/project_policy_spec.rb.dump --callgrind > project_policy_spec.callgrind @@ -184,7 +184,7 @@ kcachegrind project_policy_spec.callgrind # Linux qcachegrind project_policy_spec.callgrind # Mac ``` -For flamegraphs, enable raw collection first. Note that raw +For flame graphs, enable raw collection first. Note that raw collection can generate a very large file, so increase the `INTERVAL`, or run on a smaller number of specs for smaller file size: @@ -192,7 +192,7 @@ run on a smaller number of specs for smaller file size: RAW=true bin/rspec-stackprof spec/policies/group_member_policy_spec.rb ``` -You can then generate, and view the resultant flamegraph. It might take a +You can then generate, and view the resultant flame graph. It might take a while to generate based on the output file size: ```shell @@ -251,7 +251,7 @@ In order to enable production profiling for Ruby processes, you can set the `STA The following configuration options can be configured: -- `STACKPROF_ENABLED`: Enables stackprof signal handler on SIGUSR2 signal. +- `STACKPROF_ENABLED`: Enables Stackprof signal handler on SIGUSR2 signal. Defaults to `false`. - `STACKPROF_MODE`: See [sampling modes](https://github.com/tmm1/stackprof#sampling). Defaults to `cpu`. @@ -264,7 +264,7 @@ The following configuration options can be configured: - `STACKPROF_TIMEOUT_S`: Profiling timeout in seconds. Profiling will automatically stop after this time has elapsed. Defaults to `30`. - `STACKPROF_RAW`: Whether to collect raw samples or only aggregates. Raw - samples are needed to generate flamegraphs, but they do have a higher memory + samples are needed to generate flame graphs, but they do have a higher memory and disk overhead. Defaults to `true`. Once enabled, profiling can be triggered by sending a `SIGUSR2` signal to the @@ -287,7 +287,7 @@ The Puma master process is not supported. Neither is Unicorn. Sending SIGUSR2 to either of those triggers restarts. In the case of Puma, take care to only send the signal to Puma workers. -This can be done via `pkill -USR2 puma:`. The `:` disambiguates between `puma +This can be done via `pkill -USR2 puma:`. The `:` distinguishes between `puma 4.3.3.gitlab.2 ...` (the master process) from `puma: cluster worker 0: ...` (the worker processes), selecting the latter. @@ -296,7 +296,7 @@ For Sidekiq, the signal can be sent to the `sidekiq-cluster` process via `pkill children. Alternatively, you can also select a specific pid of interest. Production profiles can be especially noisy. It can be helpful to visualize them -as a [flamegraph](https://github.com/brendangregg/FlameGraph). This can be done +as a [flame graph](https://github.com/brendangregg/FlameGraph). This can be done via: ```shell @@ -543,6 +543,88 @@ test += " world" When adding new Ruby files, please check that you can add the above header, as omitting it may lead to style check failures. +## Banzai pipelines and filters + +When writing or updating [Banzai filters and pipelines](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/banzai), +it can be difficult to understand what the performance of the filter is, and what effect it might +have on the overall pipeline performance. + +To perform benchmarks run: + +```shell +bin/rake benchmark:banzai +``` + +This command generates output like this: + +```plaintext +--> Benchmarking Full, Wiki, and Plain pipelines +Calculating ------------------------------------- + Full pipeline 1.000 i/100ms + Wiki pipeline 1.000 i/100ms + Plain pipeline 1.000 i/100ms +------------------------------------------------- + Full pipeline 3.357 (±29.8%) i/s - 31.000 + Wiki pipeline 2.893 (±34.6%) i/s - 25.000 in 10.677014s + Plain pipeline 15.447 (±32.4%) i/s - 119.000 + +Comparison: + Plain pipeline: 15.4 i/s + Full pipeline: 3.4 i/s - 4.60x slower + Wiki pipeline: 2.9 i/s - 5.34x slower + +. +--> Benchmarking FullPipeline filters +Calculating ------------------------------------- + Markdown 24.000 i/100ms + Plantuml 8.000 i/100ms + SpacedLink 22.000 i/100ms + +... + + TaskList 49.000 i/100ms + InlineDiff 9.000 i/100ms + SetDirection 369.000 i/100ms +------------------------------------------------- + Markdown 237.796 (±16.4%) i/s - 2.304k + Plantuml 80.415 (±36.1%) i/s - 520.000 + SpacedLink 168.188 (±10.1%) i/s - 1.672k + +... + + TaskList 101.145 (± 6.9%) i/s - 1.029k + InlineDiff 52.925 (±15.1%) i/s - 522.000 + SetDirection 3.728k (±17.2%) i/s - 34.317k in 10.617882s + +Comparison: + Suggestion: 739616.9 i/s + Kroki: 306449.0 i/s - 2.41x slower +InlineGrafanaMetrics: 156535.6 i/s - 4.72x slower + SetDirection: 3728.3 i/s - 198.38x slower + +... + + UserReference: 2.1 i/s - 360365.80x slower + ExternalLink: 1.6 i/s - 470400.67x slower + ProjectReference: 0.7 i/s - 1128756.09x slower + +. +--> Benchmarking PlainMarkdownPipeline filters +Calculating ------------------------------------- + Markdown 19.000 i/100ms +------------------------------------------------- + Markdown 241.476 (±15.3%) i/s - 2.356k + +``` + +This can give you an idea how various filters perform, and which ones might be performing the slowest. + +The test data has a lot to do with how well a filter performs. If there is nothing in the test data +that specifically triggers the filter, it might look like it's running incredibly fast. +Make sure that you have relevant test data for your filter in the +[`spec/fixtures/markdown.md.erb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/fixtures/markdown.md.erb) +file. + ## Reading from files and other data sources Ruby offers several convenience functions that deal with file contents specifically @@ -665,5 +747,4 @@ Assuming you are working with ActiveRecord models, you might also find these lin ### Examples -You may find some useful examples in this snippet: -<https://gitlab.com/gitlab-org/gitlab-foss/snippets/33946> +You may find some useful examples in [this snippet](https://gitlab.com/gitlab-org/gitlab-foss/snippets/33946). diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 0354e703357..be725a702e9 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -1,12 +1,12 @@ --- stage: none -group: unassigned +group: Engineering Productivity info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Pipelines for the GitLab project -Pipelines for <https://gitlab.com/gitlab-org/gitlab> and <https://gitlab.com/gitlab-org/gitlab-foss> (as well as the +Pipelines for [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) and [`gitlab-org/gitlab-foss`](https://gitlab.com/gitlab-org/gitlab-foss) (as well as the `dev` instance's mirrors) are configured in the usual [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml) which itself includes files under @@ -37,14 +37,14 @@ Pipeline creation is also affected by the following CI variables: No pipeline is created in any other cases (for example, when pushing a branch with no MR for it). -The source of truth for these workflow rules is defined in <https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml>. +The source of truth for these workflow rules is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml). ### Pipelines for Merge Requests In general, pipelines for an MR fall into one or more of the following types, depending on the changes made in the MR: -- [Docs-only MR pipeline](#docs-only-mr-pipeline): This is typically created for an MR that only changes documentation. +- [Documentation only MR pipeline](#documentation-only-mr-pipeline): This is typically created for an MR that only changes documentation. - [Code-only MR pipeline](#code-only-mr-pipeline): This is typically created for an MR that only changes code, either backend or frontend. - [Frontend-only MR pipeline](#frontend-only-mr-pipeline): This is typically created for an MR that only changes frontend code. - [QA-only MR pipeline](#qa-only-mr-pipeline): This is typically created for an MR that only changes end to end tests related code. @@ -53,23 +53,27 @@ We use the [`rules:`](../ci/yaml/README.md#rules) and [`needs:`](../ci/yaml/READ to determine the jobs that need to be run in a pipeline. Note that an MR that includes multiple types of changes would have a pipelines that include jobs from multiple types (e.g. a combination of docs-only and code-only pipelines). -#### Docs-only MR pipeline +#### Documentation only MR pipeline -Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/135236627> +[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/250546928): ```mermaid graph LR subgraph "No needed jobs"; 1-1["danger-review (2.3 minutes)"]; click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" - 1-50["docs lint (9 minutes)"]; - click 1-50 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356757&udv=0" - end + 1-2["docs-lint markdown (1.5 minutes)"]; + click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=10224335&udv=0" + 1-3["docs-lint links (6 minutes)"]; + click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356757&udv=0" + 1-4["ui-docs-links lint (2.5 minutes)"]; + click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=10823717&udv=1020379" + end ``` #### Code-only MR pipeline -Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/136295694> +[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/136295694) ```mermaid graph RL; @@ -107,7 +111,7 @@ graph RL; class 1-6 criticalPath; end - 2_1-1["graphql-reference-verify (5 minutes)"]; + 2_1-1["graphql-verify (4 minutes)"]; click 2_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356715&udv=0" 2_1-2["memory-static (4.75 minutes)"]; click 2_1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356721&udv=0" @@ -173,7 +177,7 @@ graph RL; #### Frontend-only MR pipeline -Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/134661039> +[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/134661039): ```mermaid graph RL; @@ -212,7 +216,7 @@ graph RL; class 1-6 criticalPath; end - 2_1-1["graphql-reference-verify (5 minutes)"]; + 2_1-1["graphql-verify (4 minutes)"]; click 2_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356715&udv=0" 2_1-2["memory-static (4.75 minutes)"]; click 2_1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356721&udv=0" @@ -304,7 +308,7 @@ graph RL; #### QA-only MR pipeline -Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/134645109> +[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/134645109): ```mermaid graph RL; @@ -341,7 +345,7 @@ graph RL; class 1-5 criticalPath; end - 2_1-1["graphql-reference-verify (5 minutes)"]; + 2_1-1["graphql-verify (4 minutes)"]; click 2_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356715&udv=0" subgraph "Needs `setup-test-env`"; 2_1-1 --> 1-6; @@ -493,6 +497,7 @@ request, be sure to start the `dont-interrupt-me` job before pushing. 1. We currently have several different caches defined in [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml), with fixed keys: + - `.setup-test-env-cache`. - `.rails-cache`. - `.static-analysis-cache`. - `.coverage-cache` @@ -500,6 +505,7 @@ request, be sure to start the `dont-interrupt-me` job before pushing. - `.yarn-cache`. - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). 1. Only 6 specific jobs, running in 2-hourly scheduled pipelines, are pushing (i.e. updating) to the caches: + - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-rails-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-static-analysis-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-coverage-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). @@ -584,8 +590,8 @@ runner, or [you can incur network egress charges](https://cloud.google.com/stora The current stages are: -- `sync`: This stage is used to synchronize changes from <https://gitlab.com/gitlab-org/gitlab> to - <https://gitlab.com/gitlab-org/gitlab-foss>. +- `sync`: This stage is used to synchronize changes from [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) to + [`gitlab-org/gitlab-foss`](https://gitlab.com/gitlab-org/gitlab-foss). - `prepare`: This stage includes jobs that prepare artifacts that are needed by jobs in subsequent stages. - `build-images`: This stage includes jobs that prepare Docker images @@ -614,7 +620,9 @@ that is deployed in stage `review`. The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml). +<!-- vale gitlab.Spelling = NO --> It includes Ruby, Go, Git, Git LFS, Chrome, Node, Yarn, PostgreSQL, and Graphics Magick. +<!-- vale gitlab.Spelling = YES --> The images used in our pipelines are configured in the [`gitlab-org/gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) @@ -628,7 +636,7 @@ The current version of the build images can be found in the In addition to the [predefined variables](../ci/variables/predefined_variables.md), each pipeline includes default variables defined in -<https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml>. +[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml). ### Common job definitions @@ -640,6 +648,7 @@ that are scoped to a single [configuration keyword](../ci/yaml/README.md#job-key |------------------|-------------| | `.default-retry` | Allows a job to [retry](../ci/yaml/README.md#retry) upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. | | `.default-before_script` | Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (e.g. tests). | +| `.setup-test-env-cache` | Allows a job to use a default `cache` definition suitable for setting up test environment for subsequent Ruby/Rails tasks. | | `.rails-cache` | Allows a job to use a default `cache` definition suitable for Ruby/Rails tasks. | | `.static-analysis-cache` | Allows a job to use a default `cache` definition suitable for static analysis tasks. | | `.coverage-cache` | Allows a job to use a default `cache` definition suitable for coverage tasks. | @@ -659,7 +668,7 @@ that are scoped to a single [configuration keyword](../ci/yaml/README.md#job-key We're using the [`rules` keyword](../ci/yaml/README.md#rules) extensively. All `rules` definitions are defined in -<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml>, +[`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml), then included in individual jobs via [`extends`](../ci/yaml/README.md#extends). The `rules` definitions are composed of `if:` conditions and `changes:` patterns, diff --git a/doc/development/polling.md b/doc/development/polling.md index 18f9fb954dd..f854891a528 100644 --- a/doc/development/polling.md +++ b/doc/development/polling.md @@ -60,6 +60,6 @@ route matching easier. For more information see: -- [`Poll-Interval` header](fe_guide/performance.md#realtime-components) +- [`Poll-Interval` header](fe_guide/performance.md#real-time-components) - [RFC 7232](https://tools.ietf.org/html/rfc7232) - [ETag proposal](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/26926) diff --git a/doc/development/product_analytics/index.md b/doc/development/product_analytics/index.md index 4d2168cf304..e8b8e0c4885 100644 --- a/doc/development/product_analytics/index.md +++ b/doc/development/product_analytics/index.md @@ -4,10 +4,5 @@ redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-gui This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). -<!-- Needed by the Product Intelligence group - -Since our new standard for redirects otherwise lies within the gitlab-docs repo, -as long as we need a redirect to the handbook, we need to retain this file. - --> <!-- This redirect file can be deleted after December 1, 2021. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/product_analytics/snowplow.md b/doc/development/product_analytics/snowplow.md index bb056ffddfe..ff91e450e1e 100644 --- a/doc/development/product_analytics/snowplow.md +++ b/doc/development/product_analytics/snowplow.md @@ -4,5 +4,5 @@ redirect_to: '../snowplow.md' This document was moved to [another location](../snowplow.md). -<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- This redirect file can be deleted after April 1, 2021. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/product_analytics/usage_ping.md b/doc/development/product_analytics/usage_ping.md index 5fbdb508bb1..79075157ce4 100644 --- a/doc/development/product_analytics/usage_ping.md +++ b/doc/development/product_analytics/usage_ping.md @@ -4,5 +4,5 @@ redirect_to: '../usage_ping.md' This document was moved to [another location](../usage_ping.md). -<!-- This redirect file can be deleted after February 1, 2021. --> +<!-- This redirect file can be deleted after April 1, 2021. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/profiling.md b/doc/development/profiling.md index ce9c1191648..5714d25cedd 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -227,5 +227,5 @@ The following table lists these variables along with their default values. GitLab may decide to change these settings in order to speed up application performance, lower memory requirements, or both. You can see how each of these settings affect GC performance, memory use and application start-up time for an idle instance of -GitLab by runnning the `scripts/perf/gc/collect_gc_stats.rb` script. It will output GC stats and general timing data to standard +GitLab by running the `scripts/perf/gc/collect_gc_stats.rb` script. It will output GC stats and general timing data to standard out as CSV. diff --git a/doc/development/projections.md b/doc/development/projections.md index e62f13981c9..c112ec6f119 100644 --- a/doc/development/projections.md +++ b/doc/development/projections.md @@ -27,7 +27,7 @@ You can find a basic list of projection options in - VSCode - [Alternate File](https://marketplace.visualstudio.com/items?itemName=will-wow.vscode-alternate-file) - [projectionist](https://github.com/jarsen/projectionist) - - [jumpto](https://github.com/gmdayley/jumpto) + - [`jumpto`](https://github.com/gmdayley/jumpto) - Atom - [projectionist-atom](https://atom.io/packages/projectionist-atom) - Command-line @@ -35,6 +35,8 @@ You can find a basic list of projection options in ## History +<!-- vale gitlab.Spelling = NO --> This started as a [plugin for vim by tpope](https://github.com/tpope/vim-projectionist) It has since become editor-agnostic and ported to most modern editors. +<!-- vale gitlab.Spelling = YES -->
\ No newline at end of file diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index df13cb7d56c..7296ea726ca 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -4,7 +4,7 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Working with Prometheus Metrics +# Working with Prometheus Metrics **(FREE)** ## Adding to the library diff --git a/doc/development/pry_debugging.md b/doc/development/pry_debugging.md index f29e0d403cd..7182cf33ccc 100644 --- a/doc/development/pry_debugging.md +++ b/doc/development/pry_debugging.md @@ -79,7 +79,7 @@ Similar to source browsing, is [Documentation browsing](https://github.com/pry/p ### Command history -With **Ctrl+R** you can search your [command history](https://github.com/pry/pry/wiki/History). +With <kbd>Control</kbd> + <kbd>R</kbd> you can search your [command history](https://github.com/pry/pry/wiki/History). ## Stepping diff --git a/doc/development/python_guide/index.md b/doc/development/python_guide/index.md index 3291b6c31a9..fe5492c3bd8 100644 --- a/doc/development/python_guide/index.md +++ b/doc/development/python_guide/index.md @@ -14,8 +14,8 @@ As of GitLab 11.10, we require Python 3. ## Installation There are several ways of installing Python on your system. To be able to use the same version we use in production, -we suggest you use [pyenv](https://github.com/pyenv/pyenv). It works and behaves similarly to its counterpart in the -Ruby world: [rbenv](https://github.com/rbenv/rbenv). +we suggest you use [`pyenv`](https://github.com/pyenv/pyenv). It works and behaves similarly to its counterpart in the +Ruby world: [`rbenv`](https://github.com/rbenv/rbenv). ### macOS @@ -35,11 +35,11 @@ curl "https://pyenv.run" | bash Alternatively, you may find `pyenv` available as a system package via your distribution's package manager. -You can read more about it in: <https://github.com/pyenv/pyenv-installer#prerequisites>. +You can read more about it in [the `pyenv` prerequisites](https://github.com/pyenv/pyenv-installer#prerequisites). ### Shell integration -Pyenv installation adds required changes to Bash. If you use a different shell, +`Pyenv` installation adds required changes to Bash. If you use a different shell, check for any additional steps required for it. For Fish, you can install a plugin for [Fisher](https://github.com/jorgebucaran/fisher): diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index 5d514ffbfc9..0223f5d91d6 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # `ReactiveCaching` -> This doc refers to <https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/reactive_caching.rb>. +> This doc refers to [`reactive_caching.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/reactive_caching.rb). The `ReactiveCaching` concern is used for fetching some data in the background and storing it in the Rails cache, keeping it up-to-date for as long as it is being requested. If the @@ -103,7 +103,7 @@ not wait until the background worker completes. - An API that calls a model or service method that uses `ReactiveCaching` should return `202 accepted` when the cache is being calculated (when `#with_reactive_cache` returns `nil`). - It should also - [set the polling interval header](fe_guide/performance.md#realtime-components) with + [set the polling interval header](fe_guide/performance.md#real-time-components) with `Gitlab::PollingInterval.set_header`. - The consumer of the API is expected to poll the API. - You can also consider implementing [ETag caching](polling.md) to reduce the server diff --git a/doc/development/redis.md b/doc/development/redis.md index bb725e3c321..0dca415b77c 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -74,8 +74,10 @@ which is enabled for the `cache` and `shared_state` ## Redis in structured logging -For GitLab Team Members: There are [basic](https://www.youtube.com/watch?v=Uhdj19Dc6vU) and -[advanced](https://youtu.be/jw1Wv2IJxzs) videos that show how you can work with the Redis +For GitLab Team Members: There are <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +[basic](https://www.youtube.com/watch?v=Uhdj19Dc6vU) and +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [advanced](https://youtu.be/jw1Wv2IJxzs) +videos that show how you can work with the Redis structured logging fields on GitLab.com. Our [structured logging](logging.md#use-structured-json-logging) for web diff --git a/doc/development/refactoring_guide/index.md b/doc/development/refactoring_guide/index.md index 224b6bf9b38..a25000589c0 100644 --- a/doc/development/refactoring_guide/index.md +++ b/doc/development/refactoring_guide/index.md @@ -80,4 +80,4 @@ expect(cleanForSnapshot(wrapper.element)).toMatchSnapshot(); - [Pinning test in refactoring dropdown](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28173) - [Pinning test in refactoring vulnerability_details.vue](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25830/commits) - [Pinning test in refactoring notes_award_list.vue](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29528#pinning-test) -- [Video of pair programming session on pinning tests](https://youtu.be/LrakPcspBK4) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Video of pair programming session on pinning tests](https://youtu.be/LrakPcspBK4) diff --git a/doc/development/repository_mirroring.md b/doc/development/repository_mirroring.md index 4153bcf77a5..61157c88618 100644 --- a/doc/development/repository_mirroring.md +++ b/doc/development/repository_mirroring.md @@ -8,10 +8,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Deep Dive +<!-- vale gitlab.Spelling = NO --> + In December 2018, Tiago Botelho hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on the GitLab [Pull Repository Mirroring functionality](../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository) to share his domain specific knowledge with anyone who may work in this part of the -codebase in the future. You can find the [recording on YouTube](https://www.youtube.com/watch?v=sSZq0fpdY-Y), +codebase in the future. You can find the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=sSZq0fpdY-Y), and the slides in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/8693404888a941fd851f8a8ecdec9675/Gitlab_Create_-_Pull_Mirroring_Deep_Dive.pdf). Everything covered in this deep dive was accurate as of GitLab 11.6, and while specific details may have changed since then, it should still serve as a good introduction. + +<!-- vale gitlab.Spelling = YES -->
\ No newline at end of file diff --git a/doc/development/rolling_out_changes_using_feature_flags.md b/doc/development/rolling_out_changes_using_feature_flags.md deleted file mode 100644 index 7456e8df8d9..00000000000 --- a/doc/development/rolling_out_changes_using_feature_flags.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'feature_flags/index.md' ---- - -This document was moved to [another location](feature_flags/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index bd98ea170e5..41a7defbc26 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -180,7 +180,7 @@ For other regular expressions, here are a few guidelines: - If there's a clean non-regex solution, such as `String#start_with?`, consider using it - Ruby supports some advanced regex features like [atomic groups](https://www.regular-expressions.info/atomic.html) -and [possessive quantifiers](https://www.regular-expressions.info/possessive.html) that eleminate backtracking +and [possessive quantifiers](https://www.regular-expressions.info/possessive.html) that eliminate backtracking - Avoid nested quantifiers if possible (for example `(a+)+`) - Try to be as precise as possible in your regex and avoid the `.` if there's an alternative - For example, Use `_[^_]+_` instead of `_.*_` to match `_text here_` @@ -288,9 +288,9 @@ XSS issues are commonly classified in three categories, by their delivery method The injected client-side code is executed on the victim's browser in the context of their current session. This means the attacker could perform any same action the victim would normally be able to do through a browser. The attacker would also have the ability to: -- [log victim keystrokes](https://youtu.be/2VFavqfDS6w?t=1367) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [log victim keystrokes](https://youtu.be/2VFavqfDS6w?t=1367) - launch a network scan from the victim's browser -- potentially [obtain the victim's session tokens](https://youtu.be/2VFavqfDS6w?t=739) +- potentially <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [obtain the victim's session tokens](https://youtu.be/2VFavqfDS6w?t=739) - perform actions that lead to data loss/theft or account takeover Much of the impact is contingent upon the function of the application and the capabilities of the victim's session. For further impact possibilities, please check out [the beef project](https://beefproject.com/). @@ -309,14 +309,14 @@ In most situations, a two-step solution can be used: input validation and output ##### Setting expectations -For any and all input fields, ensure to define expectations on the type/format of input, the contents, [size limits](https://youtu.be/2VFavqfDS6w?t=7582), the context in which it will be output. It's important to work with both security and product teams to determine what is considered acceptable input. +For any and all input fields, ensure to define expectations on the type/format of input, the contents, <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [size limits](https://youtu.be/2VFavqfDS6w?t=7582), the context in which it will be output. It's important to work with both security and product teams to determine what is considered acceptable input. ##### Validate input - Treat all user input as untrusted. - Based on the expectations you [defined above](#setting-expectations): - - Validate the [input size limits](https://youtu.be/2VFavqfDS6w?t=7582). - - Validate the input using an [allowlist approach](https://youtu.be/2VFavqfDS6w?t=7816) to only allow characters through which you are expecting to receive for the field. + - Validate the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [input size limits](https://youtu.be/2VFavqfDS6w?t=7582). + - Validate the input using an <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [allowlist approach](https://youtu.be/2VFavqfDS6w?t=7816) to only allow characters through which you are expecting to receive for the field. - Input which fails validation should be **rejected**, and not sanitized. - When adding redirects or links to a user-controlled URL, ensure that the scheme is HTTP or HTTPS. Allowing other schemes like `javascript://` can lead to XSS and other security issues. @@ -328,8 +328,8 @@ Once you've [determined when and where](#setting-expectations) the user submitte - Content placed inside HTML elements need to be [HTML entity encoded](https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html#rule-1---html-escape-before-inserting-untrusted-data-into-html-element-content). - Content placed into a JSON response needs to be [JSON encoded](https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html#rule-31---html-escape-json-values-in-an-html-context-and-read-the-data-with-jsonparse). -- Content placed inside [HTML URL GET parameters](https://youtu.be/2VFavqfDS6w?t=3494) need to be [URL-encoded](https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html#rule-5---url-escape-before-inserting-untrusted-data-into-html-url-parameter-values) -- [Additional contexts may require context-specific encoding](https://youtu.be/2VFavqfDS6w?t=2341). +- Content placed inside <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [HTML URL GET parameters](https://youtu.be/2VFavqfDS6w?t=3494) need to be [URL-encoded](https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html#rule-5---url-escape-before-inserting-untrusted-data-into-html-url-parameter-values) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Additional contexts may require context-specific encoding](https://youtu.be/2VFavqfDS6w?t=2341). ### Additional information @@ -352,10 +352,10 @@ Do also sanitize and validate URL schemes. References: -- [XSS Defense in Rails](https://youtu.be/2VFavqfDS6w?t=2442) -- [XSS Defense with HAML](https://youtu.be/2VFavqfDS6w?t=2796) -- [Validating Untrusted URLs in Ruby](https://youtu.be/2VFavqfDS6w?t=3936) -- [RoR Model Validators](https://youtu.be/2VFavqfDS6w?t=7636) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [XSS Defense in Rails](https://youtu.be/2VFavqfDS6w?t=2442) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [XSS Defense with HAML](https://youtu.be/2VFavqfDS6w?t=2796) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Validating Untrusted URLs in Ruby](https://youtu.be/2VFavqfDS6w?t=3936) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [RoR Model Validators](https://youtu.be/2VFavqfDS6w?t=7636) #### XSS mitigation and prevention in JavaScript and Vue @@ -376,7 +376,7 @@ References: #### Content Security Policy -- [Content Security Policy](https://www.youtube.com/watch?v=2VFavqfDS6w&t=12991s) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Content Security Policy](https://www.youtube.com/watch?v=2VFavqfDS6w&t=12991s) - [Use nonce-based Content Security Policy for inline JavaScript](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/65330) #### Free form input field @@ -390,26 +390,26 @@ References: ### Internal Developer Training -- [Introduction to XSS](https://www.youtube.com/watch?v=PXR8PTojHmc&t=7785s) -- [Reflected XSS](https://youtu.be/2VFavqfDS6w?t=603s) -- [Persistent XSS](https://youtu.be/2VFavqfDS6w?t=643) -- [DOM XSS](https://youtu.be/2VFavqfDS6w?t=5871) -- [XSS in depth](https://www.youtube.com/watch?v=2VFavqfDS6w&t=111s) -- [XSS Defense](https://youtu.be/2VFavqfDS6w?t=1685) -- [XSS Defense in Rails](https://youtu.be/2VFavqfDS6w?t=2442) -- [XSS Defense with HAML](https://youtu.be/2VFavqfDS6w?t=2796) -- [JavaScript URLs](https://youtu.be/2VFavqfDS6w?t=3274) -- [URL encoding context](https://youtu.be/2VFavqfDS6w?t=3494) -- [Validating Untrusted URLs in Ruby](https://youtu.be/2VFavqfDS6w?t=3936) -- [HTML Sanitization](https://youtu.be/2VFavqfDS6w?t=5075) -- [DOMPurify](https://youtu.be/2VFavqfDS6w?t=5381) -- [Safe Client-side JSON Handling](https://youtu.be/2VFavqfDS6w?t=6334) -- [iframe sandboxing](https://youtu.be/2VFavqfDS6w?t=7043) -- [Input Validation](https://youtu.be/2VFavqfDS6w?t=7489) -- [Validate size limits](https://youtu.be/2VFavqfDS6w?t=7582) -- [RoR model validators](https://youtu.be/2VFavqfDS6w?t=7636) -- [Allowlist input validation](https://youtu.be/2VFavqfDS6w?t=7816) -- [Content Security Policy](https://www.youtube.com/watch?v=2VFavqfDS6w&t=12991s) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Introduction to XSS](https://www.youtube.com/watch?v=PXR8PTojHmc&t=7785s) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Reflected XSS](https://youtu.be/2VFavqfDS6w?t=603s) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Persistent XSS](https://youtu.be/2VFavqfDS6w?t=643) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [DOM XSS](https://youtu.be/2VFavqfDS6w?t=5871) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [XSS in depth](https://www.youtube.com/watch?v=2VFavqfDS6w&t=111s) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [XSS Defense](https://youtu.be/2VFavqfDS6w?t=1685) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [XSS Defense in Rails](https://youtu.be/2VFavqfDS6w?t=2442) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [XSS Defense with HAML](https://youtu.be/2VFavqfDS6w?t=2796) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [JavaScript URLs](https://youtu.be/2VFavqfDS6w?t=3274) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [URL encoding context](https://youtu.be/2VFavqfDS6w?t=3494) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Validating Untrusted URLs in Ruby](https://youtu.be/2VFavqfDS6w?t=3936) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [HTML Sanitization](https://youtu.be/2VFavqfDS6w?t=5075) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [DOMPurify](https://youtu.be/2VFavqfDS6w?t=5381) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Safe Client-side JSON Handling](https://youtu.be/2VFavqfDS6w?t=6334) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [iframe sandboxing](https://youtu.be/2VFavqfDS6w?t=7043) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Input Validation](https://youtu.be/2VFavqfDS6w?t=7489) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Validate size limits](https://youtu.be/2VFavqfDS6w?t=7582) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [RoR model validators](https://youtu.be/2VFavqfDS6w?t=7636) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Allowlist input validation](https://youtu.be/2VFavqfDS6w?t=7816) +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Content Security Policy](https://www.youtube.com/watch?v=2VFavqfDS6w&t=12991s) ## Path Traversal guidelines @@ -423,14 +423,14 @@ Path Traversal attacks can lead to multiple critical and high severity issues, l ### When to consider -When working with user-controlled filenames/paths and filesystem APIs. +When working with user-controlled filenames/paths and file system APIs. ### Mitigation and prevention In order to prevent Path Traversal vulnerabilities, user-controlled filenames or paths should be validated before being processed. - Comparing user input against an allowlist of allowed values or verifying that it only contains allowed characters. -- After validating the user supplied input, it should be appended to the base directory and the path should be canonicalized using the filesystem API. +- After validating the user supplied input, it should be appended to the base directory and the path should be canonicalized using the file system API. #### GitLab specific validations diff --git a/doc/development/shared_files.md b/doc/development/shared_files.md index 859650c2e6c..6c273f2899d 100644 --- a/doc/development/shared_files.md +++ b/doc/development/shared_files.md @@ -32,7 +32,7 @@ disk in a temporary file so you can perform some checks on it. When the checks pass, you make the file official. In scenarios like this please follow these rules: -- Store the temporary file under `shared/tmp`, i.e. on the same filesystem you +- Store the temporary file under `shared/tmp`, i.e. on the same file system you want the official file to be on. - Use move/rename operations when operating on the file instead of copy operations where possible, because renaming a file is much faster than diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index db72454b482..f28828c2e4e 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -202,7 +202,7 @@ When using regular expressions to validate user input that is passed as an argum If you don't, an attacker could use this to execute commands with potentially harmful effect. -For example, when a project's `import_url` is validated like below, the user could trick GitLab into cloning from a Git repository on the local filesystem. +For example, when a project's `import_url` is validated like below, the user could trick GitLab into cloning from a Git repository on the local file system. ```ruby validates :import_url, format: { with: URI.regexp(%w(ssh git http https)) } diff --git a/doc/development/sidekiq_debugging.md b/doc/development/sidekiq_debugging.md deleted file mode 100644 index a9b6e246861..00000000000 --- a/doc/development/sidekiq_debugging.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../administration/troubleshooting/sidekiq.md' ---- - -This document was moved to [another location](../administration/troubleshooting/sidekiq.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index e290eaee7c2..ce00934b35c 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -550,7 +550,7 @@ in the default execution mode - using does not account for weights. As we are [moving towards using `sidekiq-cluster` in -Core](https://gitlab.com/gitlab-org/gitlab/-/issues/34396), newly-added +Free](https://gitlab.com/gitlab-org/gitlab/-/issues/34396), newly-added workers do not need to have weights specified. They can simply use the default weight, which is 1. diff --git a/doc/development/snowplow.md b/doc/development/snowplow.md index 6b37936cd93..fa912f8f41f 100644 --- a/doc/development/snowplow.md +++ b/doc/development/snowplow.md @@ -104,9 +104,9 @@ The current method provides several attributes that are sent on each click event | attribute | type | required | description | | --------- | ------- | -------- | ----------- | -| category | text | true | The page or backend area of the application. Unless infeasible, please use the Rails page attribute by default in the frontend, and namespace + classname on the backend. | +| category | text | true | The page or backend area of the application. Unless infeasible, please use the Rails page attribute by default in the frontend, and namespace + class name on the backend. | | action | text | true | The action the user is taking, or aspect that's being instrumented. The first word should always describe the action or aspect: clicks should be `click`, activations should be `activate`, creations should be `create`, etc. Use underscores to describe what was acted on; for example, activating a form field would be `activate_form_input`. An interface action like clicking on a dropdown would be `click_dropdown`, while a behavior like creating a project record from the backend would be `create_project` | -| label | text | false | The specific element, or object that's being acted on. This is either the label of the element (e.g. a tab labeled 'Create from template' may be `create_from_template`) or a unique identifier if no text is available (e.g. closing the Groups dropdown in the top navbar might be `groups_dropdown_close`), or it could be the name or title attribute of a record being created. | +| label | text | false | The specific element, or object that's being acted on. This is either the label of the element (e.g. a tab labeled 'Create from template' may be `create_from_template`) or a unique identifier if no text is available (e.g. closing the Groups dropdown in the top navigation bar might be `groups_dropdown_close`), or it could be the name or title attribute of a record being created. | | property | text | false | Any additional property of the element, or object being acted on. | | value | decimal | false | Describes a numeric value or something directly related to the event. This could be the value of an input (e.g. `10` when clicking `internal` visibility). | @@ -120,9 +120,9 @@ GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tra | field | type | default value | description | |:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `category` | string | document.body.dataset.page | Page or subsection of a page that events are being captured within. | +| `category` | string | `document.body.dataset.page` | Page or subsection of a page that events are being captured within. | | `action` | string | 'generic' | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | -| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | +| `data` | object | `{}` | Additional data such as `label`, `property`, `value`, and `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | ### Tracking in HAML (or Vue Templates) @@ -303,11 +303,14 @@ GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby T Custom event tracking and instrumentation can be added by directly calling the `GitLab::Tracking.event` class method, which accepts the following arguments: -| argument | type | default value | description | -|:-----------|:-------|:--------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `category` | string | 'application' | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. | -| `action` | string | 'generic' | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | -| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described in [Structured event taxonomy](#structured-event-taxonomy). These are set as empty strings if you don't provide them. | +| argument | type | default value | description | +|------------|---------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------| +| `category` | String | | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. | +| `action` | String | | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | +| `label` | String | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | +| `property` | String | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | +| `value` | Numeric | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | +| `context` | Array\[SelfDescribingJSON\] | nil | An array of custom contexts to send with this event. Most events should not have any custom contexts. | Tracking can be viewed as either tracking user behavior, or can be used for instrumentation to monitor and visualize performance over time in an area or aspect of code. @@ -379,7 +382,7 @@ Snowplow Micro is a Docker-based solution for testing frontend and backend event - Read [Introducing Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) - Look at the [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro) -- Watch our [installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) +- Watch our <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) 1. Ensure Docker is installed and running. @@ -478,146 +481,152 @@ For GitLab.com, we're setting up a [QA and Testing environment](https://gitlab.c ## Snowplow Schemas -### [gitlab_standard](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/1-0-0) Schema +### `gitlab_standard` -| Field Name | Required | Type | Description | -|--------------|---------------------|---------|--------------------------------| -| project_id | **{dotted-circle}** | integer | ID of the associated project | -| namespace_id | **{dotted-circle}** | integer | ID of the associated namespace | +We are currently working towards including the [`gitlab_standard` schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/) with every event. See [Standardize Snowplow Schema](https://gitlab.com/groups/gitlab-org/-/epics/5218) for details. + +The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/tracking/standard_context.rb) class represents this schema in the application. + +| Field Name | Required | Type | Description | +|----------------|---------------------|-----------------------|---------------------------------------------------------------------------------------------| +| `project_id` | **{dotted-circle}** | integer | | +| `namespace_id` | **{dotted-circle}** | integer | | +| `environment` | **{check-circle}** | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | +| `source` | **{check-circle}** | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | ### Default Schema | Field Name | Required | Type | Description | |--------------------------|---------------------|-----------|----------------------------------------------------------------------------------------------------------------------------------| -| app_id | **{check-circle}** | string | Unique identifier for website / application | -| base_currency | **{dotted-circle}** | string | Reporting currency | -| br_colordepth | **{dotted-circle}** | integer | Browser color depth | -| br_cookies | **{dotted-circle}** | boolean | Does the browser permit cookies? | -| br_family | **{dotted-circle}** | string | Browser family | -| br_features_director | **{dotted-circle}** | boolean | Director plugin installed? | -| br_features_flash | **{dotted-circle}** | boolean | Flash plugin installed? | -| br_features_gears | **{dotted-circle}** | boolean | Google gears installed? | -| br_features_java | **{dotted-circle}** | boolean | Java plugin installed? | -| br_features_pdf | **{dotted-circle}** | boolean | Adobe PDF plugin installed? | -| br_features_quicktime | **{dotted-circle}** | boolean | Quicktime plugin installed? | -| br_features_realplayer | **{dotted-circle}** | boolean | Realplayer plugin installed? | -| br_features_silverlight | **{dotted-circle}** | boolean | Silverlight plugin installed? | -| br_features_windowsmedia | **{dotted-circle}** | boolean | Windows media plugin installed? | -| br_lang | **{dotted-circle}** | string | Language the browser is set to | -| br_name | **{dotted-circle}** | string | Browser name | -| br_renderengine | **{dotted-circle}** | string | Browser rendering engine | -| br_type | **{dotted-circle}** | string | Browser type | -| br_version | **{dotted-circle}** | string | Browser version | -| br_viewheight | **{dotted-circle}** | string | Browser viewport height | -| br_viewwidth | **{dotted-circle}** | string | Browser viewport width | -| collector_tstamp | **{dotted-circle}** | timestamp | Time stamp for the event recorded by the collector | -| contexts | **{dotted-circle}** | | | -| derived_contexts | **{dotted-circle}** | | Contexts derived in the Enrich process | -| derived_tstamp | **{dotted-circle}** | timestamp | Timestamp making allowance for innaccurate device clock | -| doc_charset | **{dotted-circle}** | string | Web page’s character encoding | -| doc_height | **{dotted-circle}** | string | Web page height | -| doc_width | **{dotted-circle}** | string | Web page width | -| domain_sessionid | **{dotted-circle}** | string | Unique identifier (UUID) for this visit of this user_id to this domain | -| domain_sessionidx | **{dotted-circle}** | integer | Index of number of visits that this user_id has made to this domain (The first visit is `1`) | -| domain_userid | **{dotted-circle}** | string | Unique identifier for a user, based on a first party cookie (so domain specific) | -| dvce_created_tstamp | **{dotted-circle}** | timestamp | Timestamp when event occurred, as recorded by client device | -| dvce_ismobile | **{dotted-circle}** | boolean | Indicates whether device is mobile | -| dvce_screenheight | **{dotted-circle}** | string | Screen / monitor resolution | -| dvce_screenwidth | **{dotted-circle}** | string | Screen / monitor resolution | -| dvce_sent_tstamp | **{dotted-circle}** | timestamp | Timestamp when event was sent by client device to collector | -| dvce_type | **{dotted-circle}** | string | Type of device | -| etl_tags | **{dotted-circle}** | string | JSON of tags for this ETL run | -| etl_tstamp | **{dotted-circle}** | timestamp | Timestamp event began ETL | -| event | **{dotted-circle}** | string | Event type | -| event_fingerprint | **{dotted-circle}** | string | Hash client-set event fields | -| event_format | **{dotted-circle}** | string | Format for event | -| event_id | **{dotted-circle}** | string | Event UUID | -| event_name | **{dotted-circle}** | string | Event name | -| event_vendor | **{dotted-circle}** | string | The company who developed the event model | -| event_version | **{dotted-circle}** | string | Version of event schema | -| geo_city | **{dotted-circle}** | string | City of IP origin | -| geo_country | **{dotted-circle}** | string | Country of IP origin | -| geo_latitude | **{dotted-circle}** | string | An approximate latitude | -| geo_longitude | **{dotted-circle}** | string | An approximate longitude | -| geo_region | **{dotted-circle}** | string | Region of IP origin | -| geo_region_name | **{dotted-circle}** | string | Region of IP origin | -| geo_timezone | **{dotted-circle}** | string | Timezone of IP origin | -| geo_zipcode | **{dotted-circle}** | string | Zip (postal) code of IP origin | -| ip_domain | **{dotted-circle}** | string | Second level domain name associated with the visitor’s IP address | -| ip_isp | **{dotted-circle}** | string | Visitor’s ISP | -| ip_netspeed | **{dotted-circle}** | string | Visitor’s connection type | -| ip_organization | **{dotted-circle}** | string | Organization associated with the visitor’s IP address – defaults to ISP name if none is found | -| mkt_campaign | **{dotted-circle}** | string | The campaign ID | -| mkt_clickid | **{dotted-circle}** | string | The click ID | -| mkt_content | **{dotted-circle}** | string | The content or ID of the ad. | -| mkt_medium | **{dotted-circle}** | string | Type of traffic source | -| mkt_network | **{dotted-circle}** | string | The ad network to which the click ID belongs | -| mkt_source | **{dotted-circle}** | string | The company / website where the traffic came from | -| mkt_term | **{dotted-circle}** | string | Keywords associated with the referrer | -| name_tracker | **{dotted-circle}** | string | The tracker namespace | -| network_userid | **{dotted-circle}** | string | Unique identifier for a user, based on a cookie from the collector (so set at a network level and shouldn’t be set by a tracker) | -| os_family | **{dotted-circle}** | string | Operating system family | -| os_manufacturer | **{dotted-circle}** | string | Manufacturers of operating system | -| os_name | **{dotted-circle}** | string | Name of operating system | -| os_timezone | **{dotted-circle}** | string | Client operating system timezone | -| page_referrer | **{dotted-circle}** | string | Referrer URL | -| page_title | **{dotted-circle}** | string | Page title | -| page_url | **{dotted-circle}** | string | Page URL | -| page_urlfragment | **{dotted-circle}** | string | Fragment aka anchor | -| page_urlhost | **{dotted-circle}** | string | Host aka domain | -| page_urlpath | **{dotted-circle}** | string | Path to page | -| page_urlport | **{dotted-circle}** | integer | Port if specified, 80 if not | -| page_urlquery | **{dotted-circle}** | string | Query string | -| page_urlscheme | **{dotted-circle}** | string | Scheme (protocol name) | -| platform | **{dotted-circle}** | string | The platform the app runs on | -| pp_xoffset_max | **{dotted-circle}** | integer | Maximum page x offset seen in the last ping period | -| pp_xoffset_min | **{dotted-circle}** | integer | Minimum page x offset seen in the last ping period | -| pp_yoffset_max | **{dotted-circle}** | integer | Maximum page y offset seen in the last ping period | -| pp_yoffset_min | **{dotted-circle}** | integer | Minimum page y offset seen in the last ping period | -| refr_domain_userid | **{dotted-circle}** | string | The Snowplow domain_userid of the referring website | -| refr_dvce_tstamp | **{dotted-circle}** | timestamp | The time of attaching the domain_userid to the inbound link | -| refr_medium | **{dotted-circle}** | string | Type of referer | -| refr_source | **{dotted-circle}** | string | Name of referer if recognised | -| refr_term | **{dotted-circle}** | string | Keywords if source is a search engine | -| refr_urlfragment | **{dotted-circle}** | string | Referer URL fragment | -| refr_urlhost | **{dotted-circle}** | string | Referer host | -| refr_urlpath | **{dotted-circle}** | string | Referer page path | -| refr_urlport | **{dotted-circle}** | integer | Referer port | -| refr_urlquery | **{dotted-circle}** | string | Referer URL querystring | -| refr_urlscheme | **{dotted-circle}** | string | Referer scheme | -| se_action | **{dotted-circle}** | string | The action / event itself | -| se_category | **{dotted-circle}** | string | The category of event | -| se_label | **{dotted-circle}** | string | A label often used to refer to the ‘object’ the action is performed on | -| se_property | **{dotted-circle}** | string | A property associated with either the action or the object | -| se_value | **{dotted-circle}** | decimal | A value associated with the user action | -| ti_category | **{dotted-circle}** | string | Item category | -| ti_currency | **{dotted-circle}** | string | Currency | -| ti_name | **{dotted-circle}** | string | Item name | -| ti_orderid | **{dotted-circle}** | string | Order ID | -| ti_price | **{dotted-circle}** | decimal | Item price | -| ti_price_base | **{dotted-circle}** | decimal | Item price in base currency | -| ti_quantity | **{dotted-circle}** | integer | Item quantity | -| ti_sku | **{dotted-circle}** | string | Item SKU | -| tr_affiliation | **{dotted-circle}** | string | Transaction affiliation (such as channel) | -| tr_city | **{dotted-circle}** | string | Delivery address: city | -| tr_country | **{dotted-circle}** | string | Delivery address: country | -| tr_currency | **{dotted-circle}** | string | Transaction Currency | -| tr_orderid | **{dotted-circle}** | string | Order ID | -| tr_shipping | **{dotted-circle}** | decimal | Delivery cost charged | -| tr_shipping_base | **{dotted-circle}** | decimal | Shipping cost in base currency | -| tr_state | **{dotted-circle}** | string | Delivery address: state | -| tr_tax | **{dotted-circle}** | decimal | Transaction tax value (such as amount of VAT included) | -| tr_tax_base | **{dotted-circle}** | decimal | Tax applied in base currency | -| tr_total | **{dotted-circle}** | decimal | Transaction total value | -| tr_total_base | **{dotted-circle}** | decimal | Total amount of transaction in base currency | -| true_tstamp | **{dotted-circle}** | timestamp | User-set exact timestamp | -| txn_id | **{dotted-circle}** | string | Transaction ID | -| unstruct_event | **{dotted-circle}** | JSON | The properties of the event | -| uploaded_at | **{dotted-circle}** | | | -| user_fingerprint | **{dotted-circle}** | integer | User identifier based on (hopefully unique) browser features | -| user_id | **{dotted-circle}** | string | Unique identifier for user, set by the business using setUserId | -| user_ipaddress | **{dotted-circle}** | string | IP address | -| useragent | **{dotted-circle}** | string | User agent (expressed as a browser string) | -| v_collector | **{dotted-circle}** | string | Collector version | -| v_etl | **{dotted-circle}** | string | ETL version | -| v_tracker | **{dotted-circle}** | string | Identifier for Snowplow tracker | +| `app_id` | **{check-circle}** | string | Unique identifier for website / application | +| `base_currency` | **{dotted-circle}** | string | Reporting currency | +| `br_colordepth` | **{dotted-circle}** | integer | Browser color depth | +| `br_cookies` | **{dotted-circle}** | boolean | Does the browser permit cookies? | +| `br_family` | **{dotted-circle}** | string | Browser family | +| `br_features_director` | **{dotted-circle}** | boolean | Director plugin installed? | +| `br_features_flash` | **{dotted-circle}** | boolean | Flash plugin installed? | +| `br_features_gears` | **{dotted-circle}** | boolean | Google gears installed? | +| `br_features_java` | **{dotted-circle}** | boolean | Java plugin installed? | +| `br_features_pdf` | **{dotted-circle}** | boolean | Adobe PDF plugin installed? | +| `br_features_quicktime` | **{dotted-circle}** | boolean | Quicktime plugin installed? | +| `br_features_realplayer` | **{dotted-circle}** | boolean | RealPlayer plugin installed? | +| `br_features_silverlight` | **{dotted-circle}** | boolean | Silverlight plugin installed? | +| `br_features_windowsmedia` | **{dotted-circle}** | boolean | Windows media plugin installed? | +| `br_lang` | **{dotted-circle}** | string | Language the browser is set to | +| `br_name` | **{dotted-circle}** | string | Browser name | +| `br_renderengine` | **{dotted-circle}** | string | Browser rendering engine | +| `br_type` | **{dotted-circle}** | string | Browser type | +| `br_version` | **{dotted-circle}** | string | Browser version | +| `br_viewheight` | **{dotted-circle}** | string | Browser viewport height | +| `br_viewwidth` | **{dotted-circle}** | string | Browser viewport width | +| `collector_tstamp` | **{dotted-circle}** | timestamp | Time stamp for the event recorded by the collector | +| `contexts` | **{dotted-circle}** | | | +| `derived_contexts` | **{dotted-circle}** | | Contexts derived in the Enrich process | +| `derived_tstamp` | **{dotted-circle}** | timestamp | Timestamp making allowance for inaccurate device clock | +| `doc_charset` | **{dotted-circle}** | string | Web page’s character encoding | +| `doc_height` | **{dotted-circle}** | string | Web page height | +| `doc_width` | **{dotted-circle}** | string | Web page width | +| `domain_sessionid` | **{dotted-circle}** | string | Unique identifier (UUID) for this visit of this user_id to this domain | +| `domain_sessionidx` | **{dotted-circle}** | integer | Index of number of visits that this user_id has made to this domain (The first visit is `1`) | +| `domain_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a first party cookie (so domain specific) | +| `dvce_created_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event occurred, as recorded by client device | +| `dvce_ismobile` | **{dotted-circle}** | boolean | Indicates whether device is mobile | +| `dvce_screenheight` | **{dotted-circle}** | string | Screen / monitor resolution | +| `dvce_screenwidth` | **{dotted-circle}** | string | Screen / monitor resolution | +| `dvce_sent_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event was sent by client device to collector | +| `dvce_type` | **{dotted-circle}** | string | Type of device | +| `etl_tags` | **{dotted-circle}** | string | JSON of tags for this ETL run | +| `etl_tstamp` | **{dotted-circle}** | timestamp | Timestamp event began ETL | +| `event` | **{dotted-circle}** | string | Event type | +| `event_fingerprint` | **{dotted-circle}** | string | Hash client-set event fields | +| `event_format` | **{dotted-circle}** | string | Format for event | +| `event_id` | **{dotted-circle}** | string | Event UUID | +| `event_name` | **{dotted-circle}** | string | Event name | +| `event_vendor` | **{dotted-circle}** | string | The company who developed the event model | +| `event_version` | **{dotted-circle}** | string | Version of event schema | +| `geo_city` | **{dotted-circle}** | string | City of IP origin | +| `geo_country` | **{dotted-circle}** | string | Country of IP origin | +| `geo_latitude` | **{dotted-circle}** | string | An approximate latitude | +| `geo_longitude` | **{dotted-circle}** | string | An approximate longitude | +| `geo_region` | **{dotted-circle}** | string | Region of IP origin | +| `geo_region_name` | **{dotted-circle}** | string | Region of IP origin | +| `geo_timezone` | **{dotted-circle}** | string | Timezone of IP origin | +| `geo_zipcode` | **{dotted-circle}** | string | Zip (postal) code of IP origin | +| `ip_domain` | **{dotted-circle}** | string | Second level domain name associated with the visitor’s IP address | +| `ip_isp` | **{dotted-circle}** | string | Visitor’s ISP | +| `ip_netspeed` | **{dotted-circle}** | string | Visitor’s connection type | +| `ip_organization` | **{dotted-circle}** | string | Organization associated with the visitor’s IP address – defaults to ISP name if none is found | +| `mkt_campaign` | **{dotted-circle}** | string | The campaign ID | +| `mkt_clickid` | **{dotted-circle}** | string | The click ID | +| `mkt_content` | **{dotted-circle}** | string | The content or ID of the ad. | +| `mkt_medium` | **{dotted-circle}** | string | Type of traffic source | +| `mkt_network` | **{dotted-circle}** | string | The ad network to which the click ID belongs | +| `mkt_source` | **{dotted-circle}** | string | The company / website where the traffic came from | +| `mkt_term` | **{dotted-circle}** | string | Keywords associated with the referrer | +| `name_tracker` | **{dotted-circle}** | string | The tracker namespace | +| `network_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a cookie from the collector (so set at a network level and shouldn’t be set by a tracker) | +| `os_family` | **{dotted-circle}** | string | Operating system family | +| `os_manufacturer` | **{dotted-circle}** | string | Manufacturers of operating system | +| `os_name` | **{dotted-circle}** | string | Name of operating system | +| `os_timezone` | **{dotted-circle}** | string | Client operating system timezone | +| `page_referrer` | **{dotted-circle}** | string | Referrer URL | +| `page_title` | **{dotted-circle}** | string | Page title | +| `page_url` | **{dotted-circle}** | string | Page URL | +| `page_urlfragment` | **{dotted-circle}** | string | Fragment aka anchor | +| `page_urlhost` | **{dotted-circle}** | string | Host aka domain | +| `page_urlpath` | **{dotted-circle}** | string | Path to page | +| `page_urlport` | **{dotted-circle}** | integer | Port if specified, 80 if not | +| `page_urlquery` | **{dotted-circle}** | string | Query string | +| `page_urlscheme` | **{dotted-circle}** | string | Scheme (protocol name) | +| `platform` | **{dotted-circle}** | string | The platform the app runs on | +| `pp_xoffset_max` | **{dotted-circle}** | integer | Maximum page x offset seen in the last ping period | +| `pp_xoffset_min` | **{dotted-circle}** | integer | Minimum page x offset seen in the last ping period | +| `pp_yoffset_max` | **{dotted-circle}** | integer | Maximum page y offset seen in the last ping period | +| `pp_yoffset_min` | **{dotted-circle}** | integer | Minimum page y offset seen in the last ping period | +| `refr_domain_userid` | **{dotted-circle}** | string | The Snowplow `domain_userid` of the referring website | +| `refr_dvce_tstamp` | **{dotted-circle}** | timestamp | The time of attaching the `domain_userid` to the inbound link | +| `refr_medium` | **{dotted-circle}** | string | Type of referer | +| `refr_source` | **{dotted-circle}** | string | Name of referer if recognised | +| `refr_term` | **{dotted-circle}** | string | Keywords if source is a search engine | +| `refr_urlfragment` | **{dotted-circle}** | string | Referer URL fragment | +| `refr_urlhost` | **{dotted-circle}** | string | Referer host | +| `refr_urlpath` | **{dotted-circle}** | string | Referer page path | +| `refr_urlport` | **{dotted-circle}** | integer | Referer port | +| `refr_urlquery` | **{dotted-circle}** | string | Referer URL query string | +| `refr_urlscheme` | **{dotted-circle}** | string | Referer scheme | +| `se_action` | **{dotted-circle}** | string | The action / event itself | +| `se_category` | **{dotted-circle}** | string | The category of event | +| `se_label` | **{dotted-circle}** | string | A label often used to refer to the ‘object’ the action is performed on | +| `se_property` | **{dotted-circle}** | string | A property associated with either the action or the object | +| `se_value` | **{dotted-circle}** | decimal | A value associated with the user action | +| `ti_category` | **{dotted-circle}** | string | Item category | +| `ti_currency` | **{dotted-circle}** | string | Currency | +| `ti_name` | **{dotted-circle}** | string | Item name | +| `ti_orderid` | **{dotted-circle}** | string | Order ID | +| `ti_price` | **{dotted-circle}** | decimal | Item price | +| `ti_price_base` | **{dotted-circle}** | decimal | Item price in base currency | +| `ti_quantity` | **{dotted-circle}** | integer | Item quantity | +| `ti_sku` | **{dotted-circle}** | string | Item SKU | +| `tr_affiliation` | **{dotted-circle}** | string | Transaction affiliation (such as channel) | +| `tr_city` | **{dotted-circle}** | string | Delivery address: city | +| `tr_country` | **{dotted-circle}** | string | Delivery address: country | +| `tr_currency` | **{dotted-circle}** | string | Transaction Currency | +| `tr_orderid` | **{dotted-circle}** | string | Order ID | +| `tr_shipping` | **{dotted-circle}** | decimal | Delivery cost charged | +| `tr_shipping_base` | **{dotted-circle}** | decimal | Shipping cost in base currency | +| `tr_state` | **{dotted-circle}** | string | Delivery address: state | +| `tr_tax` | **{dotted-circle}** | decimal | Transaction tax value (such as amount of VAT included) | +| `tr_tax_base` | **{dotted-circle}** | decimal | Tax applied in base currency | +| `tr_total` | **{dotted-circle}** | decimal | Transaction total value | +| `tr_total_base` | **{dotted-circle}** | decimal | Total amount of transaction in base currency | +| `true_tstamp` | **{dotted-circle}** | timestamp | User-set exact timestamp | +| `txn_id` | **{dotted-circle}** | string | Transaction ID | +| `unstruct_event` | **{dotted-circle}** | JSON | The properties of the event | +| `uploaded_at` | **{dotted-circle}** | | | +| `user_fingerprint` | **{dotted-circle}** | integer | User identifier based on (hopefully unique) browser features | +| `user_id` | **{dotted-circle}** | string | Unique identifier for user, set by the business using setUserId | +| `user_ipaddress` | **{dotted-circle}** | string | IP address | +| `useragent` | **{dotted-circle}** | string | User agent (expressed as a browser string) | +| `v_collector` | **{dotted-circle}** | string | Collector version | +| `v_etl` | **{dotted-circle}** | string | ETL version | +| `v_tracker` | **{dotted-circle}** | string | Identifier for Snowplow tracker | diff --git a/doc/development/stage_group_dashboards.md b/doc/development/stage_group_dashboards.md index 453d71411c3..e75237869ba 100644 --- a/doc/development/stage_group_dashboards.md +++ b/doc/development/stage_group_dashboards.md @@ -61,7 +61,7 @@ Although most of the metrics displayed in the panels are self-explanatory in the To inspect the raw data of the panel for further calculation, click on the Inspect button from the dropdown menu of a panel. Queries, raw data, and panel JSON structure are available. Read more at [Grafana panel inspection](https://grafana.com/docs/grafana/latest/panels/inspect-panel/). -All the dashboards are powered by [Grafana](https://grafana.com/), a frontend for displaying metrics. Grafana consumes the data returned from queries to backend Prometheus data source, then presents them under different visualizations. The stage group dashboards are built to serve the most common use cases with a limited set of filters, and pre-built queries. Grafana provides a way to explore and visualize the metrics data with [Grafana Explore](https://grafana.com/docs/grafana/latest/explore/). This would require some knowledge about [Prometheus Promql query language](https://prometheus.io/docs/prometheus/latest/querying/basics/). +All the dashboards are powered by [Grafana](https://grafana.com/), a frontend for displaying metrics. Grafana consumes the data returned from queries to backend Prometheus data source, then presents them under different visualizations. The stage group dashboards are built to serve the most common use cases with a limited set of filters, and pre-built queries. Grafana provides a way to explore and visualize the metrics data with [Grafana Explore](https://grafana.com/docs/grafana/latest/explore/). This would require some knowledge about [Prometheus PromQL query language](https://prometheus.io/docs/prometheus/latest/querying/basics/). ## How to debug with the dashboards @@ -83,7 +83,7 @@ All the dashboards are powered by [Grafana](https://grafana.com/), a frontend fo ## How to customize the dashboard -All Grafana dashboards at GitLab are generated from the [Jsonnet files](https://github.com/grafana/grafonnet-lib) stored in [the runbook project](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards). Particularly, the stage group dashboards definitions are stored in [/dashboards/stage-groups](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards/stage-groups) subfolder in the Runbook. By convention, each group has a corresponding jsonnet file. The dashboards are synced with GitLab [stage group data](https://gitlab.com/gitlab-com/www-gitlab-com/-/raw/master/data/stages.yml) every month. Expansion and customization are one of the key principles used when we designed this system. To customize your group's dashboard, you need to edit the corresponding file and follow the [Runbook workflow](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards#dashboard-source). The dashboard is updated after the MR is merged. Looking at an autogenerated file, for example, [`product_planning.dashboard.jsonnet`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/dashboards/stage-groups/product_planning.dashboard.jsonnet): +All Grafana dashboards at GitLab are generated from the [Jsonnet files](https://github.com/grafana/grafonnet-lib) stored in [the runbook project](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards). Particularly, the stage group dashboards definitions are stored in [/dashboards/stage-groups](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards/stage-groups) subfolder in the Runbook. By convention, each group has a corresponding Jsonnet file. The dashboards are synced with GitLab [stage group data](https://gitlab.com/gitlab-com/www-gitlab-com/-/raw/master/data/stages.yml) every month. Expansion and customization are one of the key principles used when we designed this system. To customize your group's dashboard, you need to edit the corresponding file and follow the [Runbook workflow](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards#dashboard-source). The dashboard is updated after the MR is merged. Looking at an autogenerated file, for example, [`product_planning.dashboard.jsonnet`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/dashboards/stage-groups/product_planning.dashboard.jsonnet): ```jsonnet // This file is autogenerated using scripts/update_stage_groups_dashboards.rb @@ -145,4 +145,8 @@ stageGroupDashboards.dashboard('source_code')  +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +If you want to see the workflow in action, we've recorded a pairing session on customizing a dashboard, +available on [GitLab Unfiltered](https://youtu.be/shEd_eiUjdI). + For deeper customization and more complicated metrics, visit the [Grafonnet lib](https://github.com/grafana/grafonnet-lib) project and the [GitLab Prometheus Metrics](../administration/monitoring/prometheus/gitlab_metrics.md#gitlab-prometheus-metrics) documentation. diff --git a/doc/development/telemetry/event_dictionary.md b/doc/development/telemetry/event_dictionary.md deleted file mode 100644 index b3b3b0b4fdd..00000000000 --- a/doc/development/telemetry/event_dictionary.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/telemetry/index.md b/doc/development/telemetry/index.md deleted file mode 100644 index b3b3b0b4fdd..00000000000 --- a/doc/development/telemetry/index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/telemetry/snowplow.md b/doc/development/telemetry/snowplow.md deleted file mode 100644 index bb056ffddfe..00000000000 --- a/doc/development/telemetry/snowplow.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../snowplow.md' ---- - -This document was moved to [another location](../snowplow.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/telemetry/usage_ping.md b/doc/development/telemetry/usage_ping.md deleted file mode 100644 index 5fbdb508bb1..00000000000 --- a/doc/development/telemetry/usage_ping.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../usage_ping.md' ---- - -This document was moved to [another location](../usage_ping.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/testing.md b/doc/development/testing.md deleted file mode 100644 index a0130bfb4ac..00000000000 --- a/doc/development/testing.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'testing_guide/index.md' ---- - -This document was moved to [another location](testing_guide/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index ac5f1a47f9b..0ccf513f5d8 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -13,13 +13,13 @@ description: "GitLab development guidelines - testing best practices." Testing at GitLab is a first class citizen, not an afterthought. It's important we consider the design of our tests as we do the design of our features. -When implementing a feature, we think about developing the right capabilities the right way, which helps us +When implementing a feature, we think about developing the right capabilities the right way. This helps us narrow our scope to a manageable level. When implementing tests for a feature, we must think about developing -the right tests, but then cover _all_ the important ways the test may fail, which can quickly widen our scope to +the right tests, but then cover _all_ the important ways the test may fail. This can quickly widen our scope to a level that is difficult to manage. Test heuristics can help solve this problem. They concisely address many of the common ways bugs -manifest themselves within our code. When designing our tests, take time to review known test heuristics to inform +manifest themselves in our code. When designing our tests, take time to review known test heuristics to inform our test design. We can find some helpful heuristics documented in the Handbook in the [Test Engineering](https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-heuristics) section. @@ -90,7 +90,7 @@ Obviously we should reduce test dependencies, and avoiding capabilities also reduces the amount of set-up needed. `:js` is particularly important to avoid. This must only be used if the feature -test requires JavaScript reactivity in the browser, since using a headless +test requires JavaScript reactivity in the browser. Using a headless browser is much slower than parsing the HTML response from the app. #### Optimize factory usage @@ -108,8 +108,8 @@ To avoid creation, it is worth bearing in mind that: - `instance_double` and `spy` are faster than `FactoryBot.build(...)`. - `FactoryBot.build(...)` and `.build_stubbed` are faster than `.create`. -- Don't `create` an object when `build`, `build_stubbed`, `attributes_for`, - `spy`, or `instance_double` will do. Database persistence is slow! +- Don't `create` an object when you can use `build`, `build_stubbed`, `attributes_for`, + `spy`, or `instance_double`. Database persistence is slow! Use [Factory Doctor](https://test-prof.evilmartians.io/#/profilers/factory_doctor) to find cases where database persistence is not needed in a given test. @@ -171,14 +171,14 @@ RSpec.describe API::Search, factory_default: :keep do let_it_be(:namespace) { create_default(:namespace) } ``` -Then every project we create will use this `namespace`, without us having to pass +Then every project we create uses this `namespace`, without us having to pass it as `namespace: namespace`. In order to make it work along with `let_it_be`, `factory_default: :keep` -must be explicitly specified. That will keep the default factory for every example in a suite instead of +must be explicitly specified. That keeps the default factory for every example in a suite instead of recreating it for each example. Maybe we don't need to create 208 different projects - we can create one and reuse it. In addition, we can see that only about 1/3 of the -projects we create are ones we ask for (76/208), so there is benefit in setting +projects we create are ones we ask for (76/208). There is benefit in setting a default value for projects as well: ```ruby @@ -233,8 +233,8 @@ Finished in 2 minutes 19 seconds (files took 1 minute 4.42 seconds to load) ``` From this result, we can see the most expensive examples in our spec, giving us -a place to start. The fact that the most expensive examples here are in -shared examples means that any reductions are likely to have a larger impact as +a place to start. The most expensive examples here are in +shared examples; any reductions generally have a larger impact as they are called in multiple places. #### Avoid repeating expensive actions @@ -287,7 +287,7 @@ results are available, and not just the first failure. - Use `.method` to describe class methods and `#method` to describe instance methods. - Use `context` to test branching logic. -- Try to match the ordering of tests to the ordering within the class. +- Try to match the ordering of tests to the ordering in the class. - Try to follow the [Four-Phase Test](https://thoughtbot.com/blog/four-phase-test) pattern, using newlines to separate phases. - Use `Gitlab.config.gitlab.host` rather than hard coding `'localhost'` @@ -295,10 +295,10 @@ results are available, and not just the first failure. [Gotchas](../gotchas.md#do-not-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). - Avoid using `expect_any_instance_of` or `allow_any_instance_of` (see [Gotchas](../gotchas.md#do-not-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). -- Don't supply the `:each` argument to hooks since it's the default. +- Don't supply the `:each` argument to hooks because it's the default. - On `before` and `after` hooks, prefer it scoped to `:context` over `:all` - When using `evaluate_script("$('.js-foo').testSomething()")` (or `execute_script`) which acts on a given element, - use a Capybara matcher beforehand (e.g. `find('.js-foo')`) to ensure the element actually exists. + use a Capybara matcher beforehand (such as `find('.js-foo')`) to ensure the element actually exists. - Use `focus: true` to isolate parts of the specs you want to run. - Use [`:aggregate_failures`](https://relishapp.com/rspec/rspec-core/docs/expectation-framework-integration/aggregating-failures) when there is more than one expectation in a test. - For [empty test description blocks](https://github.com/rubocop-hq/rspec-style-guide#it-and-specify), use `specify` rather than `it do` if the test is self-explanatory. @@ -343,7 +343,7 @@ writing one](testing_levels.md#consider-not-writing-a-system-test)! For instance, if you want to verify that a record was created, add expectations that its attributes are displayed on the page, not that `Model.count` increased by one. -- It's ok to look for DOM elements but don't abuse it since it makes the tests +- It's ok to look for DOM elements, but don't abuse it, because it makes the tests more brittle #### Debugging Capybara @@ -353,7 +353,7 @@ Sometimes you may need to debug Capybara tests by observing browser behavior. #### Live debug You can pause Capybara and view the website on the browser by using the -`live_debug` method in your spec. The current page will be automatically opened +`live_debug` method in your spec. The current page is automatically opened in your default browser. You may need to sign in first (the current user's credentials are displayed in the terminal). @@ -381,13 +381,13 @@ Finished in 34.51 seconds (files took 0.76702 seconds to load) #### Run `:js` spec in a visible browser -Run the spec with `CHROME_HEADLESS=0`, e.g.: +Run the spec with `CHROME_HEADLESS=0`, like this: ```shell CHROME_HEADLESS=0 bin/rspec some_spec.rb ``` -The test will go by quickly, but this will give you an idea of what's happening. +The test completes quickly, but this gives you an idea of what's happening. Using `live_debug` with `CHROME_HEADLESS=0` pauses the open browser, and does not open the page again. This can be used to debug and inspect elements. @@ -416,20 +416,20 @@ There is a [small hack](https://gitlab.com/gitlab-org/gitlab-foss/snippets/17184 ### Fast unit tests -Some classes are well-isolated from Rails and you should be able to test them +Some classes are well-isolated from Rails. You should be able to test them without the overhead added by the Rails environment and Bundler's `:default` group's gem loading. In these cases, you can `require 'fast_spec_helper'` instead of `require 'spec_helper'` in your test file, and your test should run -really fast since: +really fast because: -- Gems loading is skipped +- Gem loading is skipped - Rails app boot is skipped - GitLab Shell and Gitaly setup are skipped - Test repositories setup are skipped `fast_spec_helper` also support autoloading classes that are located inside the -`lib/` directory. It means that as long as your class / module is using only -code from the `lib/` directory you will not need to explicitly load any +`lib/` directory. If your class or module is using only +code from the `lib/` directory, you don't need to explicitly load any dependencies. `fast_spec_helper` also loads all ActiveSupport extensions, including core extensions that are commonly used in the Rails environment. @@ -439,9 +439,11 @@ in `lib/`. For example, if you want to test your code that is calling the `Gitlab::UntrustedRegexp` class, which under the hood uses `re2` library, you -should either add `require_dependency 're2'` to files in your library that -need `re2` gem, to make this requirement explicit, or you can add it to the -spec itself, but the former is preferred. +should either: + +- Add `require_dependency 're2'` to files in your library that need `re2` gem, + to make this requirement explicit. This approach is preferred. +- Add it to the spec itself. It takes around one second to load tests that are using `fast_spec_helper` instead of 30+ seconds in case of a regular `spec_helper`. @@ -465,7 +467,7 @@ so we need to set some guidelines for their use going forward: - Don't define a `let` variable that's only used by the definition of another. Use a helper method instead. - `let!` variables should be used only in case if strict evaluation with defined - order is required, otherwise `let` will suffice. Remember that `let` is lazy and won't + order is required, otherwise `let` suffices. Remember that `let` is lazy and won't be evaluated until it is referenced. - Avoid referencing `subject` in examples. Use a named subject `subject(:name)`, or a `let` variable instead, so the variable has a contextual name. @@ -475,7 +477,7 @@ so we need to set some guidelines for their use going forward: In some cases, there is no need to recreate the same object for tests again for each example. For example, a project and a guest of that project -is needed to test issues on the same project, one project and user will do for the entire file. +are needed to test issues on the same project, so one project and user are enough for the entire file. As much as possible, do not implement this using `before(:all)` or `before(:context)`. If you do, you would need to manually clean up the data as those hooks run outside a database transaction. @@ -494,9 +496,9 @@ before_all do end ``` -This will result in only one `Project`, `User`, and `ProjectMember` created for this context. +This results in only one `Project`, `User`, and `ProjectMember` created for this context. -`let_it_be` and `before_all` are also available within nested contexts. Cleanup after the context +`let_it_be` and `before_all` are also available in nested contexts. Cleanup after the context is handled automatically using a transaction rollback. Note that if you modify an object defined inside a `let_it_be` block, @@ -519,6 +521,35 @@ let_it_be_with_refind(:project) { create(:project) } let_it_be(:project, refind: true) { create(:project) } ``` +### License stubbing with `let_it_be` + +`let_it_be_with_refind` is also useful when using `stub_licensed_features` in your tests: + +```ruby +let_it_be_with_refind(:project) { create(:project) } +# Project#licensed_feature_available? is memoized, and so we need to refind +# the project for license updates to be applied. +# An alternative is `project.clear_memoization(:licensed_feature_available)`. + +subject { project.allows_multiple_assignees? } + +context 'with license multiple_issue_assignees disabled' do + before do + stub_licensed_features(multiple_issue_assignees: true) + end + + it { is_expected.to eq(true) } +end + +context 'with license multiple_issue_assignees disabled' do + before do + stub_licensed_features(multiple_issue_assignees: false) + end + + it { is_expected.to eq(false) } +end +``` + ### Time-sensitive tests [`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) @@ -545,14 +576,14 @@ This section was moved to [developing with feature flags](../feature_flags/devel The code exercised by a single GitLab test may access and modify many items of data. Without careful preparation before a test runs, and cleanup afterward, -data can be changed by a test in such a way that it affects the behavior of +a test can change data in a way that affects the behavior of following tests. This should be avoided at all costs! Fortunately, the existing test framework handles most cases already. When the test environment does get polluted, a common outcome is -[flaky tests](flaky_tests.md). Pollution will often manifest as an order -dependency: running spec A followed by spec B will reliably fail, but running -spec B followed by spec A will reliably succeed. In these cases, you can use +[flaky tests](flaky_tests.md). Pollution often manifests as an order +dependency: running spec A followed by spec B reliably fails, but running +spec B followed by spec A reliably succeeds. In these cases, you can use `rspec --bisect` (or a manual pairwise bisect of spec files) to determine which spec is at fault. Fixing the problem requires some understanding of how the test suite ensures the environment is pristine. Read on to discover more about each @@ -561,15 +592,15 @@ data store! #### SQL database This is managed for us by the `database_cleaner` gem. Each spec is surrounded in -a transaction, which is rolled back once the test completes. Certain specs will -instead issue `DELETE FROM` queries against every table after completion; this +a transaction, which is rolled back after the test completes. Certain specs +instead issue `DELETE FROM` queries against every table after completion. This allows the created rows to be viewed from multiple database connections, which is important for specs that run in a browser, or migration specs, among others. One consequence of using these strategies, instead of the well-known `TRUNCATE TABLES` approach, is that primary keys and other sequences are **not** reset across specs. So if you create a project in spec A, then create a project -in spec B, the first will have `id=1`, while the second will have `id=2`. +in spec B, the first has `id=1`, while the second has `id=2`. This means that specs should **never** rely on the value of an ID, or any other sequence-generated column. To avoid accidental conflicts, specs should also @@ -610,7 +641,7 @@ DNS requests are stubbed universally in the test suite (as of [!22368](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22368)), as DNS can cause issues depending on the developer's local network. There are RSpec labels available in `spec/support/dns.rb` which you can apply to tests if you need to -bypass the DNS stubbing, e.g.: +bypass the DNS stubbing, like this: ```ruby it "really connects to Prometheus", :permit_dns do @@ -625,8 +656,8 @@ In the situations where you need to [stub](https://relishapp.com/rspec/rspec-mocks/v/3-9/docs/basics/allowing-messages) methods such as `File.read`, make sure to: -1. Stub `File.read` for only the filepath you are interested in. -1. Call the original implementation for other filepaths. +1. Stub `File.read` for only the file path you are interested in. +1. Call the original implementation for other file paths. Otherwise `File.read` calls from other parts of the codebase get stubbed incorrectly. You should use the `stub_file_read`, and @@ -645,19 +676,19 @@ allow(File).to receive(:read).and_call_original allow(File).to receive(:read).with(my_filepath) ``` -#### Filesystem +#### File system -Filesystem data can be roughly split into "repositories", and "everything else". +File system data can be roughly split into "repositories", and "everything else". Repositories are stored in `tmp/tests/repositories`. This directory is emptied before a test run starts, and after the test run ends. It is not emptied between -specs, so created repositories accumulate within this directory over the +specs, so created repositories accumulate in this directory over the lifetime of the process. Deleting them is expensive, but this could lead to pollution unless carefully managed. To avoid this, [hashed storage](../../administration/repository_storage_types.md) is enabled in the test suite. This means that repositories are given a unique -path that depends on their project's ID. Since the project IDs are not reset -between specs, this guarantees that each spec gets its own repository on disk, +path that depends on their project's ID. Because the project IDs are not reset +between specs, each spec gets its own repository on disk, and prevents changes from being visible between specs. If a spec manually specifies a project ID, or inspects the state of the @@ -671,9 +702,9 @@ written to disk in locations determined by ID, so conflicts should not occur. Some specs disable hashed storage by passing the `:legacy_storage` trait to the `projects` factory. Specs that do this must **never** override the `path` of the -project, or any of its groups. The default path includes the project ID, so will -not conflict; but if two specs create a `:legacy_storage` project with the same -path, they will use the same repository on disk and lead to test environment +project, or any of its groups. The default path includes the project ID, so it +does not conflict. If two specs create a `:legacy_storage` project with the same +path, they use the same repository on disk and lead to test environment pollution. Other files must be managed manually by the spec. If you run code that creates a @@ -712,21 +743,20 @@ If you need to modify the contents of the `ENV` constant, you can use the While most Ruby **instances** are not shared between specs, **classes** and **modules** generally are. Class and module instance variables, accessors, class variables, and other stateful idioms, should be treated in the same way as -global variables - don't modify them unless you have to! In particular, prefer +global variables. Don't modify them unless you have to! In particular, prefer using expectations, or dependency injection along with stubs, to avoid the need -for modifications. If you have no other choice, an `around` block similar to the -example for global variables, above, can be used, but this should be avoided if -at all possible. +for modifications. If you have no other choice, an `around` block like the global +variables example can be used, but avoid this if at all possible. #### Test Snowplow events WARNING: Snowplow performs **runtime type checks** by using the [contracts gem](https://rubygems.org/gems/contracts). -Since Snowplow is **by default disabled in tests and development**, it can be hard to +Because Snowplow is **by default disabled in tests and development**, it can be hard to **catch exceptions** when mocking `Gitlab::Tracking`. -To catch runtime errors due to type checks, you can enable Snowplow in tests by marking the spec with -`:snowplow` and use the `expect_snowplow_event` helper which will check for +To catch runtime errors due to type checks, you can enable Snowplow in tests. Mark the spec with +`:snowplow` and use the `expect_snowplow_event` helper, which checks for calls to `Gitlab::Tracking#event`. ```ruby @@ -737,12 +767,14 @@ describe '#show', :snowplow do expect_snowplow_event( category: 'Experiment', action: 'start', + standard_context: { namespace: group, project: project } ) expect_snowplow_event( category: 'Experiment', action: 'sent', property: 'property', - label: 'label' + label: 'label', + standard_context: { namespace: group, project: project } ) end end @@ -794,7 +826,7 @@ end WARNING: Only use simple values as input in the `where` block. Using procs, stateful -objects, FactoryBot-created objects etc. can lead to +objects, FactoryBot-created objects, and similar items can lead to [unexpected results](https://github.com/tomykaira/rspec-parameterized/issues/8). ### Prometheus tests @@ -807,7 +839,7 @@ reset before each example, add the `:prometheus` tag to the RSpec test. Custom matchers should be created to clarify the intent and/or hide the complexity of RSpec expectations. They should be placed under `spec/support/matchers/`. Matchers can be placed in subfolder if they apply to -a certain type of specs only (e.g. features, requests etc.) but shouldn't be if +a certain type of specs only (such as features or requests) but shouldn't be if they apply to multiple type of specs. #### `be_like_time` @@ -881,13 +913,13 @@ expect(json_string).to be_valid_json.and match_schema(schema) Testing query performance allows us to: -- Assert that N+1 problems do not exist within a block of code. -- Ensure that the number of queries within a block of code does not increase unnoticed. +- Assert that N+1 problems do not exist in a block of code. +- Ensure that the number of queries in a block of code does not increase unnoticed. #### QueryRecorder `QueryRecorder` allows profiling and testing of the number of database queries -performed within a given block of code. +performed in a given block of code. See the [`QueryRecorder`](../query_recorder.md) section for more details. @@ -905,9 +937,9 @@ Any shared contexts used by more than one spec file: - Should be placed under `spec/support/shared_contexts/`. - Can be placed in subfolder if they apply to a certain type of specs only - (e.g. features, requests etc.) but shouldn't be if they apply to multiple type of specs. + (such as features or requests) but shouldn't be if they apply to multiple type of specs. -Each file should include only one context and have a descriptive name, e.g. +Each file should include only one context and have a descriptive name, such as `spec/support/shared_contexts/controllers/githubish_import_controller_shared_context.rb`. ### Shared examples @@ -917,9 +949,9 @@ Any shared examples used by more than one spec file: - Should be placed under `spec/support/shared_examples/`. - Can be placed in subfolder if they apply to a certain type of specs only - (e.g. features, requests etc.) but shouldn't be if they apply to multiple type of specs. + (such as features or requests) but shouldn't be if they apply to multiple type of specs. -Each file should include only one context and have a descriptive name, e.g. +Each file should include only one context and have a descriptive name, such as `spec/support/shared_examples/controllers/githubish_import_controller_shared_example.rb`. ### Helpers @@ -927,8 +959,8 @@ Each file should include only one context and have a descriptive name, e.g. Helpers are usually modules that provide some methods to hide the complexity of specific RSpec examples. You can define helpers in RSpec files if they're not intended to be shared with other specs. Otherwise, they should be placed -under `spec/support/helpers/`. Helpers can be placed in subfolder if they apply -to a certain type of specs only (e.g. features, requests etc.) but shouldn't be +under `spec/support/helpers/`. Helpers can be placed in a subfolder if they apply +to a certain type of specs only (such as features or requests) but shouldn't be if they apply to multiple type of specs. Helpers should follow the Rails naming / namespacing convention. For instance @@ -985,7 +1017,7 @@ All fixtures should be placed under `spec/fixtures/`. ### Repositories -Testing some functionality, e.g., merging a merge request, requires a Git +Testing some functionality, such as merging a merge request, requires a Git repository with a certain state to be present in the test environment. GitLab maintains the [`gitlab-test`](https://gitlab.com/gitlab-org/gitlab-test) repository for certain common cases - you can ensure a copy of the repository is @@ -996,7 +1028,7 @@ let(:project) { create(:project, :repository) } ``` Where you can, consider using the `:custom_repo` trait instead of `:repository`. -This allows you to specify exactly what files will appear in the `master` branch +This allows you to specify exactly what files appear in the `master` branch of the project's repository. For example: ```ruby @@ -1011,17 +1043,17 @@ let(:project) do end ``` -This will create a repository containing two files, with default permissions and +This creates a repository containing two files, with default permissions and the specified content. ### Configuration -RSpec configuration files are files that change the RSpec configuration (i.e. +RSpec configuration files are files that change the RSpec configuration (like `RSpec.configure do |config|` blocks). They should be placed under `spec/support/`. -Each file should be related to a specific domain, e.g. -`spec/support/capybara.rb`, `spec/support/carrierwave.rb`, etc. +Each file should be related to a specific domain, such as +`spec/support/capybara.rb` or `spec/support/carrierwave.rb`. If a helpers module applies only to a certain kind of specs, it should add modifiers to the `config.include` call. For instance if @@ -1047,7 +1079,7 @@ file which is used by the `spec/fast_spec_helper.rb` file. See Services for the test environment are automatically configured and started when tests are run, including Gitaly, Workhorse, Elasticsearch, and Capybara. When run in CI, or -if the service needs to be installed, the test environment will log information +if the service needs to be installed, the test environment logs information about set-up time, producing log messages like the following: ```plaintext diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index b761e33367f..a5a2d2a1113 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -10,7 +10,7 @@ This is a tailored extension of the Best Practices [found in the testing guide]( ## Link a test to its test-case issue -Every test should have a corresponding issue in the [Quality Testcases project](https://gitlab.com/gitlab-org/quality/testcases/). +Every test should have a corresponding issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/). It's recommended that you reuse the issue created to plan the test. If one does not already exist you can create the issue yourself. Alternatively, you can run the test in a pipeline that has reporting enabled and the test-case issue reporter will automatically create a new issue. @@ -244,7 +244,7 @@ point of failure and so the screenshot would not be captured at the right moment All tests expect to be able to log in at the start of the test. -For an example see: <https://gitlab.com/gitlab-org/gitlab/-/issues/34736> +For an example see [issue #34736](https://gitlab.com/gitlab-org/gitlab/-/issues/34736). Ideally, actions performed in an `after(:context)` (or [`before(:context)`](#limit-the-use-of-the-ui-in-beforecontext-and-after-hooks)) diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 1e0eda6491a..1bc33b79c7c 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -74,3 +74,16 @@ existing tests or write new ones. Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) for details. + +## Confirming that end-to-end tests pass with a feature flag enabled + +End-to-end tests should pass with a feature flag enabled before it is enabled on Staging or on GitLab.com. Tests that need to be updated should be identified as part of [quad-planning](https://about.gitlab.com/handbook/engineering/quality/quad-planning/). The relevant [counterpart Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors) is responsible for updating the tests or assisting another engineer to do so. However, if a change does not go through quad-planning and a required test update is not made, test failures could block deployment. + +If a test enables a feature flag as describe above, it is sufficient to run the `package-and-qa` job in a merge request containing the relevant changes. +Or, if the feature flag and relevant changes have already been merged, you can confirm that the tests +pass on `master`. The end-to-end tests run on `master` every two hours, and the results are posted to a [Test +Session Report, which is available in the testcase-sessions project](https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues?label_name%5B%5D=found%3Amaster). + +If the relevant tests do not enable the feature flag themselves, you can check if the tests will need +to be updated by opening a draft merge request that enables the flag by default and then running the `package-and-qa` job. +The merge request can be closed once the tests pass. If you need assistance to update the tests, please contact the relevant [stable counterpart in the Quality department](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), or any Software Engineer in Test if there is no stable counterpart for your group. diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 939e44cedd9..d9309f74e0e 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -145,7 +145,7 @@ for each element defined. In our case, `data-qa-selector="login_field"`, `data-qa-selector="password_field"` and `data-qa-selector="sign_in_button"` -**app/views/my/view.html.haml** +`app/views/my/view.html.haml` ```haml = f.text_field :login, class: "form-control top", autofocus: "autofocus", autocapitalize: "off", autocorrect: "off", required: true, title: "This field is required.", data: { qa_selector: 'login_field' } diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 6d6f7fbcf8d..8a929737ebe 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -14,31 +14,32 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | Tag | Description | |-----|-------------| | `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. | +| `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. | | `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | +| `:github` | The test requires a GitHub personal access token. | +| `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | +| `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | | `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. | `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | +| `:ldap_no_server` | The test requires a GitLab instance to be configured to use LDAP. To be used with the `:orchestrated` tag. It does not spin up an LDAP server at orchestration time. Instead, it creates the LDAP server at runtime. | +| `:ldap_no_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS not enabled. | +| `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. | +| `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. | +| `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. | | `:only` | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. | | `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). | +| `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. | | `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). | +| `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). | | `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | +| `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. | | `:requires_admin` | The test requires an admin account. Tests with the tag are excluded when run against Canary and Production environments. | +| `:requires_git_protocol_v2` | The test requires that Git protocol version 2 is enabled on the server. It's assumed to be enabled by default but if not the test can be skipped by setting `QA_CAN_TEST_GIT_PROTOCOL_V2` to `false`. | +| `:requires_praefect` | The test requires that the GitLab instance uses [Gitaly Cluster](../../../administration/gitaly/praefect.md) (a.k.a. Praefect) as the repository storage . It's assumed to be used by default but if not the test can be skipped by setting `QA_CAN_TEST_PRAEFECT` to `false`. | | `:runner` | The test depends on and sets up a GitLab Runner instance, typically to run a pipeline. | | `:skip_live_env` | The test is excluded when run against live deployed environments such as Staging, Canary, and Production. | -| `:testcase` | The link to the test case issue in the [Quality Testcases project](https://gitlab.com/gitlab-org/quality/testcases/). | -| `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. | -| `:ldap_no_server` | The test requires a GitLab instance to be configured to use LDAP. To be used with the `:orchestrated` tag. It does not spin up an LDAP server at orchestration time. Instead, it creates the LDAP server at runtime. | -| `:ldap_no_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS not enabled. | -| `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. | -| `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. | -| `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | -| `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | -| `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | | `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. | | `:smoke` | The test belongs to the test suite which verifies basic functionality of a GitLab instance.| -| `:github` | The test requires a GitHub personal access token. | -| `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. | -| `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. | -| `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). | -| `:requires_git_protocol_v2` | The test requires that Git protocol version 2 is enabled on the server. It's assumed to be enabled by default but if not the test can be skipped by setting `QA_CAN_TEST_GIT_PROTOCOL_V2` to `false`. | -| `:requires_praefect` | The test requires that the GitLab instance uses [Gitaly Cluster](../../../administration/gitaly/praefect.md) (a.k.a. Praefect) as the repository storage . It's assumed to be used by default but if not the test can be skipped by setting `QA_CAN_TEST_PRAEFECT` to `false`. | -| `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. | +| `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | +| `:testcase` | The link to the test case issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/). | +| `:transient` | The test tests transient bugs. It is excluded by default. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index cd429a74a2a..b6293ec41b8 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -75,17 +75,17 @@ When all the containers are running, the output of the `docker ps` command shows ```plaintext CONTAINER ID ... PORTS NAMES -d15d3386a0a8 ... 22/tcp, 443/tcp, 0.0.0.0:32772->80/tcp gitlab-gitaly-ha +d15d3386a0a8 ... 22/tcp, 443/tcp, 0.0.0.0:32772->80/tcp gitlab-gitaly-cluster ``` -That shows that the GitLab instance running in the `gitlab-gitaly-ha` container can be reached via `http://localhost:32772`. However, Git operations like cloning and pushing are performed against the URL revealed via the UI as the clone URL. It uses the hostname configured for the GitLab instance, which in this case matches the Docker container name and network, `gitlab-gitaly-ha.test`. Before you can run the tests you need to configure your computer to access the container via that address. One option is to [use caddyserver as described for running tests against GDK](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/run_qa_against_gdk.md#workarounds). +That shows that the GitLab instance running in the `gitlab-gitaly-cluster` container can be reached via `http://localhost:32772`. However, Git operations like cloning and pushing are performed against the URL revealed via the UI as the clone URL. It uses the hostname configured for the GitLab instance, which in this case matches the Docker container name and network, `gitlab-gitaly-cluster.test`. Before you can run the tests you need to configure your computer to access the container via that address. One option is to [use Caddy server as described for running tests against GDK](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/run_qa_against_gdk.md#workarounds). Another option is to use NGINX. -In both cases you must configure your machine to translate `gitlab-gitlab-ha.test` into an appropriate IP address: +In both cases you must configure your machine to translate `gitlab-gitaly-cluster.test` into an appropriate IP address: ```shell -echo '127.0.0.1 gitlab-gitaly-ha.test' | sudo tee -a /etc/hosts +echo '127.0.0.1 gitlab-gitaly-cluster.test' | sudo tee -a /etc/hosts ``` Then install NGINX: @@ -101,19 +101,19 @@ apt install nginx yum install nginx ``` -Finally, configure NGINX to pass requests for `gitlab-gitaly-ha.test` to the GitLab instance: +Finally, configure NGINX to pass requests for `gitlab-gitaly-cluster.test` to the GitLab instance: ```plaintext # On Debian/Ubuntu, in /etc/nginx/sites-enabled/gitlab-cluster # On macOS, in /usr/local/etc/nginx/nginx.conf server { - server_name gitlab-gitaly-ha.test; + server_name gitlab-gitaly-cluster.test; client_max_body_size 500m; location / { proxy_pass http://127.0.0.1:32772; - proxy_set_header Host gitlab-gitaly-ha.test; + proxy_set_header Host gitlab-gitaly-cluster.test; } } ``` @@ -131,14 +131,14 @@ sudo nginx -s reload You could then run the tests from the `/qa` directory: ```shell -CHROME_HEADLESS=false bin/qa Test::Instance::All http://gitlab-gitaly-ha.test -- --tag gitaly_ha +CHROME_HEADLESS=false bin/qa Test::Instance::All http://gitlab-gitaly-cluster.test -- --tag gitaly_cluster ``` Once you have finished testing you can stop and remove the Docker containers: ```shell -docker stop gitlab-gitaly-ha praefect postgres gitaly3 gitaly2 gitaly1 -docker rm gitlab-gitaly-ha praefect postgres gitaly3 gitaly2 gitaly1 +docker stop gitlab-gitaly-cluster praefect postgres gitaly3 gitaly2 gitaly1 +docker rm gitlab-gitaly-cluster praefect postgres gitaly3 gitaly2 gitaly1 ``` ## Guide to run and debug Monitor tests @@ -177,7 +177,7 @@ The following includes more information on the command: At the moment of this writing, there are two specs which run monitor tests: --`qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - has the specs of features in GitLab Core +-`qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - has the specs of features in GitLab Free -`qa/specs/features/ee/browser_ui/8_monitor/all_monitor_features_spec.rb` - has the specs of features for paid GitLab (Enterprise Edition) ### How to debug @@ -408,15 +408,15 @@ Geo requires an EE license. To visit the Geo sites in your browser, you need a r Tests that are tagged with `:ldap_tls` and `:ldap_no_tls` meta are orchestrated tests where the sign-in happens via LDAP. -These tests spin up a Docker container [(osixia/openldap)](https://hub.docker.com/r/osixia/openldap) running an instance of [OpenLDAP](https://www.openldap.org/). -The container uses fixtures [checked into the GitLab-QA repo](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap) to create +These tests spin up a Docker container [(`osixia/openldap`)](https://hub.docker.com/r/osixia/openldap) running an instance of [OpenLDAP](https://www.openldap.org/). +The container uses fixtures [checked into the GitLab-QA repository](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap) to create base data such as users and groups including the admin group. The password for [all users](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap/2_add_users.ldif) including [the `tanuki` user](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap/tanuki.ldif) is `password`. A GitLab instance is also created in a Docker container based on our [General LDAP setup](../../../administration/auth/ldap/index.md#general-ldap-setup) documentation. -Tests that are tagged `:ldap_tls` enable TLS on GitLab using the certificate [checked into the GitLab-QA repo](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/9ffb9ad3be847a9054967d792d6772a74220fb42/tls_certificates/gitlab). +Tests that are tagged `:ldap_tls` enable TLS on GitLab using the certificate [checked into the GitLab-QA repository](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/9ffb9ad3be847a9054967d792d6772a74220fb42/tls_certificates/gitlab). -The certificate was generated with openssl using this command: +The certificate was generated with OpenSSL using this command: ```shell openssl req -x509 -newkey rsa:4096 -keyout gitlab.test.key -out gitlab.test.crt -days 3650 -nodes -subj "/C=US/ST=CA/L=San Francisco/O=GitLab/OU=Org/CN=gitlab.test" @@ -432,7 +432,7 @@ To run the LDAP tests on your local with TLS enabled, follow these steps: `127.0.0.1 gitlab.test` - You can then run tests against GitLab in a Docker container on `https://gitlab.test`. Please note that the TLS certificate [checked into the GitLab-QA repo](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/9ffb9ad3be847a9054967d792d6772a74220fb42/tls_certificates/gitlab) is configured for this domain. + You can then run tests against GitLab in a Docker container on `https://gitlab.test`. Please note that the TLS certificate [checked into the GitLab-QA repository](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/9ffb9ad3be847a9054967d792d6772a74220fb42/tls_certificates/gitlab) is configured for this domain. 1. Run the OpenLDAP container with TLS enabled. Change the path to [`gitlab-qa/fixtures/ldap`](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap) directory to your local checkout path: ```shell diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 94bc80abcdb..7e7f62e41dd 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -31,7 +31,7 @@ Jest tests can be found in `/spec/frontend` and `/ee/spec/frontend` in EE. ## Karma test suite While GitLab has switched over to [Jest](https://jestjs.io), Karma tests still exist in our -application because some of our specs require a browser and can't be easiliy migrated to Jest. +application because some of our specs require a browser and can't be easily migrated to Jest. Those specs intend to eventually drop Karma in favor of either Jest or RSpec. You can track this migration in the [related epic](https://gitlab.com/groups/gitlab-org/-/epics/4900). @@ -258,7 +258,7 @@ it('exists', () => { ### Naming unit tests When writing describe test blocks to test specific functions/methods, -please use the method name as the describe block name. +use the method name as the describe block name. **Bad**: @@ -439,7 +439,7 @@ it('waits for an Ajax call', done => { }); ``` -If you are not able to register handlers to the `Promise`, for example because it is executed in a synchronous Vue life cycle hook, please take a look at the [waitFor](#wait-until-axios-requests-finish) helpers or you can flush all pending `Promise`s: +If you are not able to register handlers to the `Promise`, for example because it is executed in a synchronous Vue life cycle hook, take a look at the [waitFor](#wait-until-axios-requests-finish) helpers or you can flush all pending `Promise`s: **in Jest:** @@ -702,10 +702,10 @@ unit testing by mocking out modules which cannot be easily consumed in our test Jest supports [manual module mocks](https://jestjs.io/docs/en/manual-mocks) by placing a mock in a `__mocks__/` directory next to the source module (e.g. `app/assets/javascripts/ide/__mocks__`). **Don't do this.** We want to keep all of our test-related code in one place (the `spec/` folder). -If a manual mock is needed for a `node_modules` package, please use the `spec/frontend/__mocks__` folder. Here's an example of +If a manual mock is needed for a `node_modules` package, use the `spec/frontend/__mocks__` folder. Here's an example of a [Jest mock for the package `monaco-editor`](https://gitlab.com/gitlab-org/gitlab/blob/b7f914cddec9fc5971238cdf12766e79fa1629d7/spec/frontend/__mocks__/monaco-editor/index.js#L1). -If a manual mock is needed for a CE module, please place it in `spec/frontend/mocks/ce`. +If a manual mock is needed for a CE module, place it in `spec/frontend/mocks/ce`. - Files in `spec/frontend/mocks/ce` mocks the corresponding CE module from `app/assets/javascripts`, mirroring the source module's path. - Example: `spec/frontend/mocks/ce/lib/utils/axios_utils` mocks the module `~/lib/utils/axios_utils`. @@ -728,11 +728,11 @@ If a manual mock is needed for a CE module, please place it in `spec/frontend/mo Global mocks introduce magic and technically can reduce test coverage. When mocking is deemed profitable: - Keep the mock short and focused. -- Please leave a top-level comment in the mock on why it is necessary. +- Leave a top-level comment in the mock on why it is necessary. ### Additional mocking techniques -Please consult the [official Jest docs](https://jestjs.io/docs/en/jest-object#mock-modules) for a full overview of the available mocking features. +Consult the [official Jest docs](https://jestjs.io/docs/en/jest-object#mock-modules) for a full overview of the available mocking features. ## Running Frontend Tests @@ -865,7 +865,7 @@ end This will create a new fixture located at `tmp/tests/frontend/fixtures-ee/graphql/releases/queries/all_releases.query.graphql.json`. -Note that you will need to provide the paths to all fragments used by the query. +You will need to provide the paths to all fragments used by the query. `get_graphql_query_as_string` reads all of the provided file paths and returns the result as a single, concatenated string. @@ -929,7 +929,8 @@ it.each([ ); ``` -**Note**: only use template literal block if pretty print is **not** needed for spec output. For example, empty strings, nested objects etc. +NOTE: +Only use template literal block if pretty print is not needed for spec output. For example, empty strings, nested objects etc. For example, when testing the difference between an empty search string and a non-empty search string, the use of the array block syntax with the pretty print option would be preferred. That way the differences between an empty string e.g. `''` and a non-empty string e.g. `'search string'` would be visible in the spec output. Whereas with a template literal block, the empty string would be shown as a space, which could lead to a confusing developer experience @@ -1038,7 +1039,8 @@ import Subject from '~/feature/the_subject.vue'; import _Thing from '~/feature/path/to/thing.vue'; ``` -**PLEASE NOTE:** Do not simply disregard test timeouts. This could be a sign that there's +NOTE: +Do not disregard test timeouts. This could be a sign that there's actually a production problem. Use this opportunity to analyze the production webpack bundles and chunks and confirm that there is not a production issue with the async imports. @@ -1063,7 +1065,7 @@ See also [Notes on testing Vue components](../fe_guide/vue.md#testing-vue-compon ## Test helpers Test helpers can be found in [`spec/frontend/__helpers__`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/frontend/__helpers__). -If you introduce new helpers, please place them in that directory. +If you introduce new helpers, place them in that directory. ### Vuex Helper: `testAction` diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 68326879dd0..c22a4e0b3ad 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -9,9 +9,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w This document describes various guidelines and best practices for automated testing of the GitLab project. -It is meant to be an _extension_ of the [thoughtbot testing +It is meant to be an _extension_ of the [Thoughtbot testing style guide](https://github.com/thoughtbot/guides/tree/master/testing-rspec). If -this guide defines a rule that contradicts the thoughtbot guide, this guide +this guide defines a rule that contradicts the Thoughtbot guide, this guide takes precedence. Some guidelines may be repeated verbatim to stress their importance. diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index a5294be40a9..f1c74f990cb 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -186,9 +186,9 @@ the GitLab handbook information for the [shared 1Password account](https://about ### Find my Review App slug 1. Open the `review-deploy` job. -1. Look for `Checking for previous deployment of review-*`. -1. For instance for `Checking for previous deployment of review-qa-raise-e-12chm0`, - your Review App slug would be `review-qa-raise-e-12chm0` in this case. +1. Look for `** Deploying review-*`. +1. For instance for `** Deploying review-1234-abc-defg... **`, + your Review App slug would be `review-1234-abc-defg` in this case. ### Run a Rails console @@ -357,7 +357,7 @@ using `v232`. For the record, the debugging steps to find out this issue were: -1. Switch kubectl context to review-apps-ce (we recommend using [kubectx](https://github.com/ahmetb/kubectx/)) +1. Switch kubectl context to `review-apps-ce` (we recommend using [`kubectx`](https://github.com/ahmetb/kubectx/)) 1. `kubectl get pods | grep dns` 1. `kubectl describe pod <pod name>` & confirm exact error message 1. Web search for exact error message, following rabbit hole to [a relevant Kubernetes bug report](https://github.com/kubernetes/kubernetes/issues/57345) diff --git a/doc/development/testing_guide/testing_migrations_guide.md b/doc/development/testing_guide/testing_migrations_guide.md index 31054d0ffb2..d54ca0d3c64 100644 --- a/doc/development/testing_guide/testing_migrations_guide.md +++ b/doc/development/testing_guide/testing_migrations_guide.md @@ -32,7 +32,7 @@ migrate the database **down** to the previous migration version. With this approach you can test a migration against a database schema. -An `after` hook migrates the database **up** and reinstitutes the latest +An `after` hook migrates the database **up** and restores the latest schema version, so that the process does not affect subsequent specs and ensures proper isolation. diff --git a/doc/development/ui_guide.md b/doc/development/ui_guide.md deleted file mode 100644 index 1e84bf608f4..00000000000 --- a/doc/development/ui_guide.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/' ---- - -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/). diff --git a/doc/development/usage_ping.md b/doc/development/usage_ping.md index 10c3de2f0a1..02a070bc3ad 100644 --- a/doc/development/usage_ping.md +++ b/doc/development/usage_ping.md @@ -6,10 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Usage Ping Guide -> - Introduced in GitLab Enterprise Edition 8.10. -> - More statistics were added in GitLab Enterprise Edition 8.12. -> - Moved to GitLab Core in 9.1. -> - More statistics were added in GitLab Ultimate 11.2. +> Introduced in GitLab Ultimate 11.2, more statistics. This guide describes Usage Ping's purpose and how it's implemented. @@ -122,6 +119,60 @@ sequenceDiagram the hostname is `version.gitlab.com`, the protocol is `TCP`, and the port number is `443`, the required URL is <https://version.gitlab.com/>. +## Usage Ping Metric Life cycle + +### 1. New metrics addition + +Please follow the [Implementing Usage Ping](#implementing-usage-ping) guide. + +### 2. Existing metric change + +Because we do not control when customers update their self-managed instances of GitLab, +we **STRONGLY DISCOURAGE** changes to the logic used to calculate any metric. +Any such changes lead to inconsistent reports from multiple GitLab instances. +If there is a problem with an existing metric, it's best to deprecate the existing metric, +and use it, side by side, with the desired new metric. + +Example: +Consider following change. Before GitLab 12.6, the `example_metric` was implemented as: + +```ruby +{ + ... + example_metric: distinct_count(Project, :creator_id) +} +``` + +For GitLab 12.6, the metric was changed to filter out archived projects: + +```ruby +{ + ... + example_metric: distinct_count(Project.non_archived, :creator_id) +} +``` + +In this scenario all instances running up to GitLab 12.5 continue to report `example_metric`, +including all archived projects, while all instances running GitLab 12.6 and higher filters +out such projects. As Usage Ping data is collected from all reporting instances, the +resulting dataset includes mixed data, which distorts any following business analysis. + +The correct approach is to add a new metric for GitLab 12.6 release with updated logic: + +```ruby +{ + ... + example_metric_without_archived: distinct_count(Project.non_archived, :creator_id) +} +``` + +and update existing business analysis artefacts to use `example_metric_without_archived` instead of `example_metric` + +### 3. Metrics deprecation and removal + +The process for deprecating and removing metrics is currently under development. For +more information, see the following [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/284637). + ## Implementing Usage Ping Usage Ping consists of two kinds of data, counters and observations. Counters track how often a certain event @@ -175,9 +226,9 @@ Arguments: - `relation` the ActiveRecord_Relation to perform the count - `column` the column to perform the count on, by default is the primary key -- `batch`: default `true` in order to use batch counting -- `start`: custom start of the batch counting in order to avoid complex min calculations -- `end`: custom end of the batch counting in order to avoid complex min calculations +- `batch`: default `true` to use batch counting +- `start`: custom start of the batch counting to avoid complex min calculations +- `end`: custom end of the batch counting to avoid complex min calculations Examples: @@ -199,10 +250,10 @@ Arguments: - `relation` the ActiveRecord_Relation to perform the count - `column` the column to perform the distinct count, by default is the primary key -- `batch`: default `true` in order to use batch counting +- `batch`: default `true` to use batch counting - `batch_size`: if none set it uses default value 10000 from `Gitlab::Database::BatchCounter` -- `start`: custom start of the batch counting in order to avoid complex min calculations -- `end`: custom end of the batch counting in order to avoid complex min calculations +- `start`: custom start of the batch counting to avoid complex min calculations +- `end`: custom end of the batch counting to avoid complex min calculations WARNING: Counting over non-unique columns can lead to performance issues. Take a look at the [iterating tables in batches](iterating_tables_in_batches.md) guide for more details. @@ -228,8 +279,8 @@ Arguments: - `relation` the ActiveRecord_Relation to perform the operation - `column` the column to sum on - `batch_size`: if none set it uses default value 1000 from `Gitlab::Database::BatchCounter` -- `start`: custom start of the batch counting in order to avoid complex min calculations -- `end`: custom end of the batch counting in order to avoid complex min calculations +- `start`: custom start of the batch counting to avoid complex min calculations +- `end`: custom end of the batch counting to avoid complex min calculations Examples: @@ -281,7 +332,7 @@ The method includes the following arguments: - `column`: The column to perform the distinct count. The default is the primary key. - `batch_size`: The default is 10,000, from `Gitlab::Database::PostgresHll::BatchDistinctCounter::DEFAULT_BATCH_SIZE`. - `start`: The custom start of the batch count, to avoid complex minimum calculations. -- `finish`: The custom end of the batch count in order to avoid complex maximum calculations. +- `finish`: The custom end of the batch count to avoid complex maximum calculations. The method includes the following prerequisites: @@ -364,7 +415,7 @@ Examples of implementation: API requests are protected by checking for a valid CSRF token. - In order to be able to increment the values the related feature `usage_data_<event_name>` should be enabled. + To be able to increment the values, the related feature `usage_data_<event_name>` should be enabled. ```plaintext POST /usage_data/increment_counter @@ -383,7 +434,7 @@ Examples of implementation: 1. Track events using JavaScript/Vue API helper which calls the API above - Note that `usage_data_api` and `usage_data_#{event_name}` should be enabled in order to be able to track events + Note that `usage_data_api` and `usage_data_#{event_name}` should be enabled to be able to track events ```javascript import api from '~/api'; @@ -429,7 +480,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF - `a_` for events encompassing all `g_`, `p_`, `i_`. - `o_` for other. - Consider including in the event's name the Redis slot in order to be able to count totals for a specific category. + Consider including in the event's name the Redis slot to be able to count totals for a specific category. Example names: `i_compliance_credential_inventory`, `g_analytics_contribution`. @@ -444,7 +495,9 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF aggregation. - `aggregation`: may be set to a `:daily` or `:weekly` key. Defines how counting data is stored in Redis. Aggregation on a `daily` basis does not pull more fine grained data. - - `feature_flag`: optional. For details, see our [GitLab internal Feature flags](feature_flags/) documentation. + - `feature_flag`: optional. For details, see our [GitLab internal Feature flags](feature_flags/) documentation. The feature flags are owned by the group adding the event tracking. + +Use one of the following methods to track events: 1. Track event in controller using `RedisTracking` module with `track_redis_hll_event(*controller_actions, name:, feature:, feature_default_enabled: false)`. @@ -481,7 +534,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF 1. Track event in API using `increment_unique_values(event_name, values)` helper method. - In order to be able to track the event, Usage Ping must be enabled and the event feature `usage_data_<event_name>` must be enabled. + To be able to track the event, Usage Ping must be enabled and the event feature `usage_data_<event_name>` must be enabled. Arguments: @@ -526,7 +579,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF API requests are protected by checking for a valid CSRF token. - In order to increment the values, the related feature `usage_data_<event_name>` should be + To increment the values, the related feature `usage_data_<event_name>` should be set to `default_enabled: true`. For more information, see [Feature flags in development of GitLab](feature_flags/index.md). @@ -562,21 +615,6 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF api.trackRedisHllUserEvent('my_already_defined_event_name'), ``` -1. Track event using base module `Gitlab::UsageDataCounters::HLLRedisCounter.track_event(event_name, values:)`. - - Arguments: - - - `event_name`: event name. - - `values`: One value or array of values we count. For example: user_id, visitor_id, user_ids. - -1. Track event on context level using base module `Gitlab::UsageDataCounters::HLLRedisCounter.track_event_in_context(event_name, values:, context:)`. - - Arguments: - - - `event_name`: event name. - - `values`: values we count. For example: user_id, visitor_id. - - `context`: context value. Allowed values are `default`, `free`, `bronze`, `silver`, `gold`, `starter`, `premium`, `ultimate` - 1. Get event data using `Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names:, start_date:, end_date:, context: '')`. Arguments: @@ -780,7 +818,7 @@ Paste the SQL query into `#database-lab` to see how the query performs at scale. - Any single query must stay below [1 second execution time](query_performance.md#timing-guidelines-for-queries) with cold caches. - Add a specialized index on columns involved to reduce the execution time. -In order to have an understanding of the query's execution we add in the MR description the following information: +To have an understanding of the query's execution we add in the MR description the following information: - For counters that have a `time_period` test we add information for both cases: - `time_period = {}` for all time periods @@ -839,16 +877,16 @@ There are currently three kinds of components that may export data to Prometheus This is the recommended approach to test Prometheus based Usage Ping. -The easiest way to verify your changes is to build a new Omnibus image from your code branch via CI, then download the image +The easiest way to verify your changes is to build a new Omnibus image from your code branch by using CI, then download the image and run a local container instance: 1. From your merge request, click on the `qa` stage, then trigger the `package-and-qa` job. This job triggers an Omnibus build in a [downstream pipeline of the `omnibus-gitlab-mirror` project](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/pipelines). 1. In the downstream pipeline, wait for the `gitlab-docker` job to finish. 1. Open the job logs and locate the full container name including the version. It takes the following form: `registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>`. -1. On your local machine, make sure you are logged in to the GitLab Docker registry. You can find the instructions for this in +1. On your local machine, make sure you are signed in to the GitLab Docker registry. You can find the instructions for this in [Authenticate to the GitLab Container Registry](../user/packages/container_registry/index.md#authenticate-with-the-container-registry). -1. Once logged in, download the new image via `docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>` +1. Once signed in, download the new image by using `docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>` 1. For more information about working with and running Omnibus GitLab containers in Docker, please refer to [GitLab Docker images](https://docs.gitlab.com/omnibus/docker/README.html) in the Omnibus documentation. #### Test with GitLab development toolkits @@ -875,7 +913,7 @@ appear to be associated to any of the services running, since they all appear to WARNING: This feature is intended solely for internal GitLab use. -In order to add data for aggregated metrics into Usage Ping payload you should add corresponding definition in [`aggregated_metrics`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/aggregated_metrics/). Each aggregate definition includes following parts: +To add data for aggregated metrics into Usage Ping payload you should add corresponding definition in [`aggregated_metrics`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/aggregated_metrics/). Each aggregate definition includes following parts: - name: unique name under which aggregate metric is added to Usage Ping payload - operator: operator that defines how aggregated metric data is counted. Available operators are: diff --git a/doc/development/usage_ping/dictionary.md b/doc/development/usage_ping/dictionary.md new file mode 100644 index 00000000000..27a3ac42af6 --- /dev/null +++ b/doc/development/usage_ping/dictionary.md @@ -0,0 +1,158 @@ +--- +stage: Growth +group: Product Intelligence +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +<!--- + This documentation is auto generated by a script. + + Please do not edit this file directly, check generate_metrics_dictionary task on lib/tasks/gitlab/usage_data.rake. +---> + +<!-- vale gitlab.Spelling = NO --> + +# Metrics Dictionary + +This file is autogenerated, please do not edit directly. + +To generate these files from the GitLab repository, run: + +```shell +bundle exec rake gitlab:usage_data:generate_metrics_dictionary +``` + +The Metrics Dictionary is based on the following metrics definition YAML files: + +- [`config/metrics`]('https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/metrics') +- [`ee/config/metrics`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/config/metrics) +Each table includes a `milestone`, which corresponds to the GitLab version when the metric +was released. + +## counts.deployments + +Total deployments count + +| field | value | +| --- | --- | +| `key_path` | **counts.deployments** | +| `value_type` | integer | +| `stage` | release | +| `status` | data_available | +| `milestone` | 8.12 | +| `introduced_by_url` | [Introduced by](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/735) | +| `group` | `group::ops release` | +| `time_frame` | all | +| `data_source` | Database | +| `distribution` | ee, ce | +| `tier` | free, starter, premium, ultimate, bronze, silver, gold | + +## counts.geo_nodes + +Total number of sites in a Geo deployment + +| field | value | +| --- | --- | +| `key_path` | **counts.geo_nodes** | +| `value_type` | integer | +| `product_category` | disaster_recovery | +| `stage` | enablement | +| `status` | data_available | +| `milestone` | 11.2 | +| `group` | `group::geo` | +| `time_frame` | all | +| `data_source` | Database | +| `distribution` | ee | +| `tier` | premium, ultimate | + +## counts_monthly.deployments + +Total deployments count for recent 28 days + +| field | value | +| --- | --- | +| `key_path` | **counts_monthly.deployments** | +| `value_type` | integer | +| `stage` | release | +| `status` | data_available | +| `milestone` | 13.2 | +| `introduced_by_url` | [Introduced by](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35493) | +| `group` | `group::ops release` | +| `time_frame` | 28d | +| `data_source` | Database | +| `distribution` | ee, ce | +| `tier` | free, starter, premium, ultimate, bronze, silver, gold | + +## database.adapter + +This metric only returns a value of PostgreSQL in supported versions of GitLab. It could be removed from the usage ping. Historically MySQL was also supported. + +| field | value | +| --- | --- | +| `key_path` | **database.adapter** | +| `value_type` | string | +| `product_category` | collection | +| `stage` | growth | +| `status` | data_available | +| `group` | `group::enablement distribution` | +| `time_frame` | none | +| `data_source` | Database | +| `distribution` | ee, ce | +| `tier` | free, starter, premium, ultimate, bronze, silver, gold | + +## recorded_at + +When the Usage Ping computation was started + +| field | value | +| --- | --- | +| `key_path` | **recorded_at** | +| `value_type` | string | +| `product_category` | collection | +| `stage` | growth | +| `status` | data_available | +| `milestone` | 8.1 | +| `introduced_by_url` | [Introduced by](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/557) | +| `group` | `group::product analytics` | +| `time_frame` | none | +| `data_source` | Ruby | +| `distribution` | ee, ce | +| `tier` | free, starter, premium, ultimate, bronze, silver, gold | + +## redis_hll_counters.issues_edit.g_project_management_issue_title_changed_weekly + +Distinct users count that changed issue title in a group for last recent week + +| field | value | +| --- | --- | +| `key_path` | **redis_hll_counters.issues_edit.g_project_management_issue_title_changed_weekly** | +| `value_type` | integer | +| `product_category` | issue_tracking | +| `stage` | plan | +| `status` | data_available | +| `milestone` | 13.6 | +| `introduced_by_url` | [Introduced by](https://gitlab.com/gitlab-org/gitlab/-/issues/229918) | +| `group` | `group::project management` | +| `time_frame` | 7d | +| `data_source` | Redis_hll | +| `distribution` | ee, ce | +| `tier` | free, starter, premium, ultimate, bronze, silver, gold | + +## uuid + +GitLab instance unique identifier + +| field | value | +| --- | --- | +| `key_path` | **uuid** | +| `value_type` | string | +| `product_category` | collection | +| `stage` | growth | +| `status` | data_available | +| `milestone` | 9.1 | +| `introduced_by_url` | [Introduced by](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1521) | +| `group` | `group::product analytics` | +| `time_frame` | none | +| `data_source` | Database | +| `distribution` | ee, ce | +| `tier` | free, starter, premium, ultimate, bronze, silver, gold | diff --git a/doc/development/usage_ping/metrics_dictionary.md b/doc/development/usage_ping/metrics_dictionary.md index bae79689f3b..7c41c331e88 100644 --- a/doc/development/usage_ping/metrics_dictionary.md +++ b/doc/development/usage_ping/metrics_dictionary.md @@ -14,24 +14,22 @@ We are using [JSON Schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/co This process is meant to ensure consistent and valid metrics defined for Usage Ping. All metrics *must*: -- Comply with the definied [JSON schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/schema.json). -- Have a unique `full_path` . +- Comply with the defined [JSON schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/schema.json). +- Have a unique `key_path` . - Have an owner. All metrics are stored in YAML files: - [`config/metrics`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/metrics) -Each metric is definied in a separate YAML file consisting of a number of fields: +Each metric is defined in a separate YAML file consisting of a number of fields: | Field | Required | Additional information | |---------------------|----------|----------------------------------------------------------------| -| `name` | yes | | +| `key_path` | yes | JSON key path for the metric, location in Usage Ping payload. | | `description` | yes | | | `value_type` | yes | | | `status` | yes | | -| `default_generation`| yes | Default generation path of the metric. One full_path value. (1) | -| `full_path` | yes | Full path of the metric for one or multiple generations. Path of the metric in Usage Ping payload. (1) | | `group` | yes | The [group](https://about.gitlab.com/handbook/product/categories/#devops-stages) that owns the metric. | | `time_frame` | yes | `string`; may be set to a value like "7d" | | `data_source` | yes | `string`: may be set to a value like `database` or `redis_hll`. | @@ -43,9 +41,6 @@ Each metric is definied in a separate YAML file consisting of a number of fields | `milestone_removed` | no | The milestone when the metric is removed. | | `introduced_by_url` | no | The URL to the Merge Request that introduced the metric. | -1. The default generation path is the location of the metric in the Usage Ping payload. - The `full_path` is the list locations for multiple Usage Ping generaations. - ### Example metric definition The linked [`uuid`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/license/uuid.yml) @@ -53,16 +48,12 @@ YAML file includes an example metric definition, where the `uuid` metric is the instance unique identifier. ```yaml -name: uuid +key_path: uuid description: GitLab instance unique identifier value_type: string product_category: collection stage: growth status: data_available -default_generation: generation_1 -full_path: - generation_1: uuid - generation_2: license.uuid milestone: 9.1 introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1521 group: group::product intelligence @@ -71,3 +62,25 @@ data_source: database distribution: [ee, ce] tier: ['free', 'starter', 'premium', 'ultimate', 'bronze', 'silver', 'gold'] ``` + +## Create a new metric definition + +The GitLab codebase provides a dedicated [generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/usage_metric_definition_generator.rb) to create new metric definitions. + +The generator takes the key path argument and 2 options and creates the metric YAML definition in corresponding location: + +- `--ee`, `--no-ee` Indicates if metric is for EE. +- `--dir=DIR` indicates the metric directory. It must be one of: `counts_7d`, `7d`, `counts_28d`, `28d`, `counts_all`, `all`, `settings`, `license`. + +```shell +bundle exec rails generate gitlab:usage_metric_definition counts.issues --dir=7d +create config/metrics/counts_7d/issues.yml +``` + +NOTE: +To create a metric definition used in EE, add the `--ee` flag. + +```shell +bundle exec rails generate gitlab:usage_metric_definition counts.issues --ee --dir=7d +create ee/config/metrics/counts_7d/issues.yml +``` diff --git a/doc/development/utilities.md b/doc/development/utilities.md index 2d347df0559..d7baa6b23a5 100644 --- a/doc/development/utilities.md +++ b/doc/development/utilities.md @@ -10,7 +10,7 @@ We have developed a number of utilities to help ease development: ## `MergeHash` -Refer to: <https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/utils/merge_hash.rb>: +Refer to [`merge_hash.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/utils/merge_hash.rb): - Deep merges an array of hashes: @@ -109,9 +109,50 @@ Refer to [`override.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gi Because only a class or prepended module can actually override a method. Including or extending a module into another cannot override anything. +### Interactions with `ActiveSupport::Concern`, `prepend`, and `class_methods` + +When you use `ActiveSupport::Concern` that includes class methods, you do not +get expected results because `ActiveSupport::Concern` doesn't work like a +regular Ruby module. + +Since we already have `Prependable` as a patch for `ActiveSupport::Concern` +to enable `prepend`, it has consequences with how it would interact with +`override` and `class_methods`. As a workaround, `extend` `ClassMethods` +into the defining `Prependable` module. + +This allows us to use `override` to verify `class_methods` used in the +context mentioned above. This workaround only applies when we run the +verification, not when running the application itself. + +Here are example code blocks that demonstrate the effect of this workaround: +following codes: + +```ruby +module Base + extend ActiveSupport::Concern + + class_methods do + def f + end + end +end + +module Derived + include Base +end + +# Without the workaround +Base.f # => NoMethodError +Derived.f # => nil + +# With the workaround +Base.f # => nil +Derived.f # => nil +``` + ## `StrongMemoize` -Refer to <https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/utils/strong_memoize.rb>: +Refer to [`strong_memoize.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/utils/strong_memoize.rb): - Memoize the value even if it is `nil` or `false`. diff --git a/doc/development/ux_guide/animation.md b/doc/development/ux_guide/animation.md deleted file mode 100644 index 0f7a24042bb..00000000000 --- a/doc/development/ux_guide/animation.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/product-foundations/motion/' ---- - -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/product-foundations/motion/). diff --git a/doc/development/ux_guide/basics.md b/doc/development/ux_guide/basics.md deleted file mode 100644 index 1e84bf608f4..00000000000 --- a/doc/development/ux_guide/basics.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/' ---- - -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/). diff --git a/doc/development/ux_guide/components.md b/doc/development/ux_guide/components.md deleted file mode 100644 index 1e84bf608f4..00000000000 --- a/doc/development/ux_guide/components.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/' ---- - -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/). diff --git a/doc/development/ux_guide/copy.md b/doc/development/ux_guide/copy.md deleted file mode 100644 index 1e84bf608f4..00000000000 --- a/doc/development/ux_guide/copy.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/' ---- - -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/). diff --git a/doc/development/ux_guide/features.md b/doc/development/ux_guide/features.md deleted file mode 100644 index 1e84bf608f4..00000000000 --- a/doc/development/ux_guide/features.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/' ---- - -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/). diff --git a/doc/development/ux_guide/illustrations.md b/doc/development/ux_guide/illustrations.md deleted file mode 100644 index 815f870f8c5..00000000000 --- a/doc/development/ux_guide/illustrations.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/product-foundations/illustration/' ---- - -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/product-foundations/illustration/). diff --git a/doc/development/ux_guide/index.md b/doc/development/ux_guide/index.md deleted file mode 100644 index 1e84bf608f4..00000000000 --- a/doc/development/ux_guide/index.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/' ---- - -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/). diff --git a/doc/development/ux_guide/principles.md b/doc/development/ux_guide/principles.md deleted file mode 100644 index 1e84bf608f4..00000000000 --- a/doc/development/ux_guide/principles.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/' ---- - -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/). diff --git a/doc/development/ux_guide/resources.md b/doc/development/ux_guide/resources.md deleted file mode 100644 index ae092246d05..00000000000 --- a/doc/development/ux_guide/resources.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/resources/design-resources/' ---- - -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/resources/design-resources/). diff --git a/doc/development/ux_guide/surfaces.md b/doc/development/ux_guide/surfaces.md deleted file mode 100644 index 1e84bf608f4..00000000000 --- a/doc/development/ux_guide/surfaces.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/' ---- - -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/). diff --git a/doc/development/ux_guide/tips.md b/doc/development/ux_guide/tips.md deleted file mode 100644 index 1e84bf608f4..00000000000 --- a/doc/development/ux_guide/tips.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'https://design.gitlab.com/' ---- - -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/). diff --git a/doc/development/ux_guide/users.md b/doc/development/ux_guide/users.md deleted file mode 100644 index a01aa4bcf35..00000000000 --- a/doc/development/ux_guide/users.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/wikis.md b/doc/development/wikis.md index a2da4843eac..7995afb1e17 100644 --- a/doc/development/wikis.md +++ b/doc/development/wikis.md @@ -1,7 +1,7 @@ --- type: reference, dev stage: Create -group: Knowledge +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: "GitLab's development guidelines for Wikis" --- @@ -12,8 +12,8 @@ description: "GitLab's development guidelines for Wikis" ## Overview -The wiki functionality in GitLab is based on [Gollum 4.x](https://github.com/gollum/gollum/), -which is used in [Gitaly's](gitaly.md) Ruby service and accessed from the Rails app through Gitaly RPC calls. +The wiki functionality in GitLab is based on [Gollum 4.x](https://github.com/gollum/gollum/). +It's used in [Gitaly's](gitaly.md) Ruby service, and accessed from the Rails app through Gitaly RPC calls. Wikis use Git repositories as storage backend, and can be accessed through: @@ -43,7 +43,7 @@ When rendering wiki pages, we don't use Gollum at all and instead go through a [custom Banzai pipeline](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/banzai/pipeline/wiki_pipeline.rb). This adds some [wiki-specific markup](../user/markdown.md#wiki-specific-markdown), such as Gollum's `[[link]]` syntax. -Since we do not make use of most of Gollum's features, we plan to move away from it entirely at some point. +Because we do not make use of most of Gollum's features, we plan to move away from it entirely at some point. [See this epic](https://gitlab.com/groups/gitlab-org/-/epics/2381) for reference. ## Model classes diff --git a/doc/development/windows.md b/doc/development/windows.md index 08ff29a4e58..07f8a80e95f 100644 --- a/doc/development/windows.md +++ b/doc/development/windows.md @@ -63,7 +63,7 @@ Build a Google Cloud image with the above shared runners repository by doing the ## How to use a Windows image in GCP -1. In a web browser, go to <https://console.cloud.google.com/compute/images>. +1. In a web browser, go to the [Google Cloud Platform console](https://console.cloud.google.com/compute/images). 1. Filter images by the name you used when creating image, `windows` is likely all you need to filter by. 1. Click the image's name. 1. Click the **CREATE INSTANCE** link. @@ -81,7 +81,7 @@ Build a Google Cloud image with the above shared runners repository by doing the 1. Click **Continue** to accept the certificate. 1. Enter the password and click **Next**. -You should now be remoted into a Windows machine with a command prompt. +You should now be connected into a Windows machine with a command prompt. ### Optional: Use GCP VM Instance as a runner diff --git a/doc/development/writing_documentation.md b/doc/development/writing_documentation.md deleted file mode 100644 index 038a4b1e6ea..00000000000 --- a/doc/development/writing_documentation.md +++ /dev/null @@ -1,3 +0,0 @@ ---- -redirect_to: 'documentation/index.md' ---- |