Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2022-04-09 00:09:52 +0300
committerGitLab Bot <gitlab-bot@gitlab.com>2022-04-09 00:09:52 +0300
commit81c0f29ad962733b0750bdab2d3250e2c796a578 (patch)
tree42d31d9c48d156849c87aee7f4c3c494d2ca6260 /doc/development/workhorse
parent842ac3526cba09feb4b9ccefd0aeeb6edc02035d (diff)
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/development/workhorse')
-rw-r--r--doc/development/workhorse/channel.md244
-rw-r--r--doc/development/workhorse/configuration.md129
-rw-r--r--doc/development/workhorse/gitlab_features.md2
-rw-r--r--doc/development/workhorse/new_features.md106
4 files changed, 257 insertions, 224 deletions
diff --git a/doc/development/workhorse/channel.md b/doc/development/workhorse/channel.md
index 9d7dd524c2f..33d7cc63f00 100644
--- a/doc/development/workhorse/channel.md
+++ b/doc/development/workhorse/channel.md
@@ -4,29 +4,29 @@ group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
-# Websocket channel support
+# Websocket channel support for Workhorse
-In some cases, GitLab can provide in-browser terminal access to an
-environment (which is a running server or container, onto which a
-project has been deployed), or even access to services running in CI
-through a WebSocket. Workhorse manages the WebSocket upgrade and
-long-lived connection to the websocket connection, which frees
-up GitLab to process other requests.
+In some cases, GitLab can provide the following through a WebSocket:
-This document outlines the architecture of these connections.
+- In-browser terminal access to an environment: a running server or container,
+ onto which a project has been deployed.
+- Access to services running in CI.
+
+Workhorse manages the WebSocket upgrade and long-lived connection to the websocket
+connection, which frees up GitLab to process other requests. This document outlines
+the architecture of these connections.
## Introduction to WebSockets
-A websocket is an "upgraded" HTTP/1.1 request. Their purpose is to
-permit bidirectional communication between a client and a server.
-**Websockets are not HTTP**. Clients can send messages (known as
-frames) to the server at any time, and vice-versa. Client messages
-are not necessarily requests, and server messages are not necessarily
-responses. WebSocket URLs have schemes like `ws://` (unencrypted) or
+Websockets are an "upgraded" `HTTP/1.1` request. They permit bidirectional
+communication between a client and a server. **Websockets are not HTTP**.
+Clients can send messages (known as frames) to the server at any time, and
+vice versa. Client messages are not necessarily requests, and server messages are
+not necessarily responses. WebSocket URLs have schemes like `ws://` (unencrypted) or
`wss://` (TLS-secured).
-When requesting an upgrade to WebSocket, the browser sends a HTTP/1.1
-request that looks like this:
+When requesting an upgrade to WebSocket, the browser sends a `HTTP/1.1`
+request like this:
```plaintext
GET /path.ws HTTP/1.1
@@ -36,164 +36,166 @@ Sec-WebSocket-Protocol: terminal.gitlab.com
# More headers, including security measures
```
-At this point, the connection is still HTTP, so this is a request and
-the server can send a normal HTTP response, including `404 Not Found`,
-`500 Internal Server Error`, etc.
+At this point, the connection is still HTTP, so this is a request.
+The server can send a normal HTTP response, such as `404 Not Found` or
+`500 Internal Server Error`.
+
+If the server decides to permit the upgrade, it sends a HTTP
+`101 Switching Protocols` response. From this point, the connection is no longer
+HTTP. It is now a WebSocket and frames, not HTTP requests, flow over it. The connection
+persists until the client or server closes the connection.
+
+In addition to the sub-protocol, individual websocket frames may
+also specify a message type, such as:
-If the server decides to permit the upgrade, it will send a HTTP
-`101 Switching Protocols` response. From this point, the connection
-is no longer HTTP. It is a WebSocket and frames, not HTTP requests,
-will flow over it. The connection will persist until the client or
-server closes the connection.
+- `BinaryMessage`
+- `TextMessage`
+- `Ping`
+- `Pong`
+- `Close`
-In addition to the subprotocol, individual websocket frames may
-also specify a message type - examples include `BinaryMessage`,
-`TextMessage`, `Ping`, `Pong` or `Close`. Only binary frames can
-contain arbitrary data - other frames are expected to be valid
-UTF-8 strings, in addition to any subprotocol expectations.
+Only binary frames can contain arbitrary data. The frames are expected to be valid
+UTF-8 strings, in addition to any sub-protocol expectations.
## Browser to Workhorse
-Using the terminal as an example, GitLab serves a JavaScript terminal
-emulator to the browser on a URL like
-`https://gitlab.com/group/project/-/environments/1/terminal`.
-This opens a websocket connection to, e.g.,
-`wss://gitlab.com/group/project/-/environments/1/terminal.ws`,
-This endpoint doesn't exist in GitLab - only in Workhorse.
-
-When receiving the connection, Workhorse first checks that the
-client is authorized to access the requested terminal. It does
-this by performing a "preauthentication" request to GitLab.
-
-If the client has the appropriate permissions and the terminal
-exists, GitLab responds with a successful response that includes
-details of the terminal that the client should be connected to.
-Otherwise, it returns an appropriate HTTP error response.
-
-Errors are passed back to the client as HTTP responses, but if
-GitLab returns valid terminal details to Workhorse, it will
-connect to the specified terminal, upgrade the browser to a
-WebSocket, and proxy between the two connections for as long
-as the browser's credentials are valid. Workhorse will also
-send regular `PingMessage` control frames to the browser, to
-keep intervening proxies from terminating the connection
-while the browser is present.
-
-The browser must request an upgrade with a specific subprotocol:
+Using the terminal as an example:
+
+1. GitLab serves a JavaScript terminal emulator to the browser on a URL like
+ `https://gitlab.com/group/project/-/environments/1/terminal`.
+1. This URL opens a websocket connection to
+ `wss://gitlab.com/group/project/-/environments/1/terminal.ws`.
+ This endpoint exists only in Workhorse, and doesn't exist in GitLab.
+1. When receiving the connection, Workhorse first performs a `preauthentication`
+ request to GitLab to confirm the client is authorized to access the requested terminal:
+ - If the client has the appropriate permissions and the terminal exists, GitLab
+ responds with a successful response that includes details of the terminal
+ the client should be connected to.
+ - Otherwise, Workhorse returns an appropriate HTTP error response.
+1. If GitLab returns valid terminal details to Workhorse, it:
+ 1. Connects to the specified terminal.
+ 1. Upgrades the browser to a WebSocket.
+ 1. Proxies between the two connections for as long as the browser's credentials are valid.
+ 1. Send regular `PingMessage` control frames to the browser, to prevent intervening
+ proxies from terminating the connection while the browser is present.
+
+The browser must request an upgrade with a specific sub-protocol:
+
+- [`terminal.gitlab.com`](#terminalgitlabcom)
+- [`base64.terminal.gitlab.com`](#base64terminalgitlabcom)
### `terminal.gitlab.com`
-This subprotocol considers `TextMessage` frames to be invalid.
-Control frames, such as `PingMessage` or `CloseMessage`, have
-their usual meanings.
+This sub-protocol considers `TextMessage` frames to be invalid. Control frames,
+such as `PingMessage` or `CloseMessage`, have their usual meanings.
-`BinaryMessage` frames sent from the browser to the server are
-arbitrary text input.
-
-`BinaryMessage` frames sent from the server to the browser are
-arbitrary text output.
+- `BinaryMessage` frames sent from the browser to the server are
+ arbitrary text input.
+- `BinaryMessage` frames sent from the server to the browser are
+ arbitrary text output.
These frames are expected to contain ANSI text control codes
and may be in any encoding.
### `base64.terminal.gitlab.com`
-This subprotocol considers `BinaryMessage` frames to be invalid.
+This sub-protocol considers `BinaryMessage` frames to be invalid.
Control frames, such as `PingMessage` or `CloseMessage`, have
their usual meanings.
-`TextMessage` frames sent from the browser to the server are
-base64-encoded arbitrary text input (so the server must
-base64-decode them before inputting them).
-
-`TextMessage` frames sent from the server to the browser are
-base64-encoded arbitrary text output (so the browser must
-base64-decode them before outputting them).
+- `TextMessage` frames sent from the browser to the server are
+ base64-encoded arbitrary text input. The server must
+ base64-decode them before inputting them.
+- `TextMessage` frames sent from the server to the browser are
+ base64-encoded arbitrary text output. The browser must
+ base64-decode them before outputting them.
In their base64-encoded form, these frames are expected to
contain ANSI terminal control codes, and may be in any encoding.
## Workhorse to GitLab
-Using again the terminal as an example, before upgrading the browser,
+Using the terminal as an example, before upgrading the browser,
Workhorse sends a normal HTTP request to GitLab on a URL like
`https://gitlab.com/group/project/environments/1/terminal.ws/authorize`.
This returns a JSON response containing details of where the
terminal can be found, and how to connect it. In particular,
the following details are returned in case of success:
-- WebSocket URL to **connect** to, e.g.: `wss://example.com/terminals/1.ws?tty=1`
-- WebSocket subprotocols to support, e.g.: `["channel.k8s.io"]`
-- Headers to send, e.g.: `Authorization: Token xxyyz..`
-- Certificate authority to verify `wss` connections with (optional)
+- WebSocket URL to connect** to, such as `wss://example.com/terminals/1.ws?tty=1`.
+- WebSocket sub-protocols to support, such as `["channel.k8s.io"]`.
+- Headers to send, such as `Authorization: Token xxyyz`.
+- Optional. Certificate authority to verify `wss` connections with.
-Workhorse periodically re-checks this endpoint, and if it gets an
-error response, or the details of the terminal change, it will
-terminate the websocket session.
+Workhorse periodically rechecks this endpoint. If it receives an error response,
+or the details of the terminal change, it terminates the websocket session.
## Workhorse to the WebSocket server
-In GitLab, environments or CI jobs may have a deployment service (e.g.,
+In GitLab, environments or CI jobs may have a deployment service (like
`KubernetesService`) associated with them. This service knows
-where the terminals or the service for an environment may be found, and these
-details are returned to Workhorse by GitLab.
-
-These URLs are *also* WebSocket URLs, and GitLab tells Workhorse
-which subprotocols to speak over the connection, along with any
-authentication details required by the remote end.
-
-Before upgrading the browser's connection to a websocket,
-Workhorse opens a HTTP client connection, according to the
-details given to it by Workhorse, and attempts to upgrade
-that connection to a websocket. If it fails, an error
-response is sent to the browser; otherwise, the browser is
-also upgraded.
-
-Workhorse now has two websocket connections, albeit with
-differing subprotocols. It decodes incoming frames from the
-browser, re-encodes them to the channel's subprotocol, and
-sends them to the channel. Similarly, it decodes incoming
-frames from the channel, re-encodes them to the browser's
-subprotocol, and sends them to the browser.
-
-When either connection closes or enters an error state,
-Workhorse detects the error and closes the other connection,
-terminating the channel session. If the browser is the
-connection that has disconnected, Workhorse will send an ANSI
-`End of Transmission` control code (the `0x04` byte) to the
-channel, encoded according to the appropriate subprotocol.
-Workhorse will automatically reply to any websocket ping frame
-sent by the channel, to avoid being disconnected.
-
-Currently, Workhorse only supports the following subprotocols.
-Supporting new deployment services will require new subprotocols
-to be supported:
+where the terminals or the service for an environment may be found, and GitLab
+returns these details to Workhorse.
+
+These URLs are also WebSocket URLs. GitLab tells Workhorse which sub-protocols to
+speak over the connection, along with any authentication details required by the
+remote end.
+
+Before upgrading the browser's connection to a websocket, Workhorse:
+
+1. Opens a HTTP client connection, according to the details given to it by Workhorse.
+1. Attempts to upgrade that connection to a websocket.
+ - If it fails, an error response is sent to the browser.
+ - If it succeeds, the browser is also upgraded.
+
+Workhorse now has two websocket connections, albeit with differing sub-protocols,
+and then:
+
+- Decodes incoming frames from the browser, re-encodes them to the channel's
+ sub-protocol, and sends them to the channel.
+- Decodes incoming frames from the channel, re-encodes them to the browser's
+ sub-protocol, and sends them to the browser.
+
+When either connection closes or enters an error state, Workhorse detects the error
+and closes the other connection, terminating the channel session. If the browser
+is the connection that has disconnected, Workhorse sends an ANSI `End of Transmission`
+control code (the `0x04` byte) to the channel, encoded according to the appropriate
+sub-protocol. To avoid being disconnected, Workhorse replies to any websocket ping
+frame sent by the channel.
+
+Workhorse only supports the following sub-protocols:
+
+- [`channel.k8s.io`](#channelk8sio)
+- [`base64.channel.k8s.io`](#base64channelk8sio)
+
+Supporting new deployment services requires new sub-protocols to be supported.
### `channel.k8s.io`
-Used by Kubernetes, this subprotocol defines a simple multiplexed
-channel.
+Used by Kubernetes, this sub-protocol defines a simple multiplexed channel.
Control frames have their usual meanings. `TextMessage` frames are
invalid. `BinaryMessage` frames represent I/O to a specific file
descriptor.
The first byte of each `BinaryMessage` frame represents the file
-descriptor (fd) number, as a `uint8` (so the value `0x00` corresponds
-to fd 0, `STDIN`, while `0x01` corresponds to fd 1, `STDOUT`).
+descriptor (`fd`) number, as a `uint8`. For example:
+
+- `0x00` corresponds to `fd 0`, `STDIN`.
+- `0x01` corresponds to `fd 1`, `STDOUT`.
The remaining bytes represent arbitrary data. For frames received
from the server, they are bytes that have been received from that
-fd. For frames sent to the server, they are bytes that should be
-written to that fd.
+`fd`. For frames sent to the server, they are bytes that should be
+written to that `fd`.
### `base64.channel.k8s.io`
-Also used by Kubernetes, this subprotocol defines a similar multiplexed
+Also used by Kubernetes, this sub-protocol defines a similar multiplexed
channel to `channel.k8s.io`. The main differences are:
- `TextMessage` frames are valid, rather than `BinaryMessage` frames.
- The first byte of each `TextMessage` frame represents the file
descriptor as a numeric UTF-8 character, so the character `U+0030`,
- or "0", is fd 0, STDIN).
+ or "0", is `fd 0`, `STDIN`.
- The remaining bytes represent base64-encoded arbitrary data.
diff --git a/doc/development/workhorse/configuration.md b/doc/development/workhorse/configuration.md
index f715cf442eb..7f9331e6f1e 100644
--- a/doc/development/workhorse/configuration.md
+++ b/doc/development/workhorse/configuration.md
@@ -6,9 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Workhorse configuration
-For historical reasons Workhorse uses both command line flags, a configuration file and environment variables.
+For historical reasons, Workhorse uses:
-All new configuration options that get added to Workhorse should go into the configuration file.
+- Command line flags.
+- A configuration file.
+- Environment variables.
+
+Add any new Workhorse configuration options into the configuration file.
## CLI options
@@ -61,35 +65,32 @@ Options:
```
The 'auth backend' refers to the GitLab Rails application. The name is
-a holdover from when GitLab Workhorse only handled Git push/pull over
+a holdover from when GitLab Workhorse only handled `git push` and `git pull` over
HTTP.
GitLab Workhorse can listen on either a TCP or a Unix domain socket. It
can also open a second listening TCP listening socket with the Go
[`net/http/pprof` profiler server](http://golang.org/pkg/net/http/pprof/).
-GitLab Workhorse can listen on Redis events (currently only builds/register
-for runners). This requires you to pass a valid TOML configuration file via
-`-config` flag.
-For regular setups it only requires the following (replacing the string
+GitLab Workhorse can listen on Redis build and runner registration events if you
+pass a valid TOML configuration file through the `-config` flag.
+A regular setup it only requires the following (replacing the string
with the actual socket)
## Redis
GitLab Workhorse integrates with Redis to do long polling for CI build
-requests. This is configured via two things:
+requests. To configure it:
-- Redis settings in the TOML configuration file
-- The `-apiCiLongPollingDuration` command line flag to control polling
- behavior for CI build requests
+- Configure Redis settings in the TOML configuration file.
+- Control polling behavior for CI build requests with the `-apiCiLongPollingDuration`
+ command-line flag.
-It is OK to enable Redis in the configuration file but to leave CI polling
-disabled; this just results in an idle Redis pubsub connection. The
-opposite is not possible: CI long polling requires a correct Redis
-configuration.
+You can enable Redis in the configuration file while leaving CI polling
+disabled. This configuration results in an idle Redis Pub/Sub connection. The
+opposite is not possible: CI long polling requires a correct Redis configuration.
-Below we discuss the options for the `[redis]` section in the configuration
-file.
+For example, the `[redis]` section in the configuration file could contain:
```plaintext
[redis]
@@ -99,15 +100,13 @@ Sentinel = [ "tcp://sentinel1:23456", "tcp://sentinel2:23456" ]
SentinelMaster = "mymaster"
```
-- `URL` takes a string in the format `unix://path/to/redis.sock` or
-`tcp://host:port`.
-- `Password` is only required if your Redis instance is password-protected
-- `Sentinel` is used if you are using Sentinel.
+- `URL` - A string in the format `unix://path/to/redis.sock` or `tcp://host:port`.
+- `Password` - Required only if your Redis instance is password-protected.
+- `Sentinel` - Required if you use Sentinel.
- NOTE:
- If both `Sentinel` and `URL` are given, only `Sentinel` will be used.
+If both `Sentinel` and `URL` are given, only `Sentinel` is used.
-Optional fields are as follows:
+Optional fields:
```plaintext
[redis]
@@ -116,15 +115,14 @@ MaxIdle = 1
MaxActive = 1
```
-- `DB` is the Database to connect to. Defaults to `0`
-- `MaxIdle` is how many idle connections can be in the Redis pool at once. Defaults to 1
-- `MaxActive` is how many connections the pool can keep. Defaults to 1
+- `DB` - The database to connect to. Defaults to `0`.
+- `MaxIdle` - How many idle connections can be in the Redis pool at once. Defaults to `1`.
+- `MaxActive` - How many connections the pool can keep. Defaults to `1`.
## Relative URL support
-If you are mounting GitLab at a relative URL, e.g.
-`example.com/gitlab`, then you should also use this relative URL in
-the `authBackend` setting:
+If you mount GitLab at a relative URL, like `example.com/gitlab`), use this
+relative URL in the `authBackend` setting:
```plaintext
gitlab-workhorse -authBackend http://localhost:8080/gitlab
@@ -132,33 +130,32 @@ gitlab-workhorse -authBackend http://localhost:8080/gitlab
## Interaction of authBackend and authSocket
-The interaction between `authBackend` and `authSocket` can be a bit
-confusing. It comes down to: if `authSocket` is set it overrides the
-_host_ part of `authBackend` but not the relative path.
+The interaction between `authBackend` and `authSocket` can be confusing.
+If `authSocket` is set, it overrides the host portion of `authBackend`, but not
+the relative path.
In table form:
-|authBackend|authSocket|Workhorse connects to?|Rails relative URL|
-|---|---|---|---|
-|unset|unset|`localhost:8080`|`/`|
-|`http://localhost:3000`|unset|`localhost:3000`|`/`|
-|`http://localhost:3000/gitlab`|unset|`localhost:3000`|`/gitlab`|
-|unset|`/path/to/socket`|`/path/to/socket`|`/`|
-|`http://localhost:3000`|`/path/to/socket`|`/path/to/socket`|`/`|
-|`http://localhost:3000/gitlab`|`/path/to/socket`|`/path/to/socket`|`/gitlab`|
+| authBackend | authSocket | Workhorse connects to | Rails relative URL |
+|--------------------------------|-------------------|-----------------------|--------------------|
+| unset | unset | `localhost:8080` | `/` |
+| `http://localhost:3000` | unset | `localhost:3000` | `/` |
+| `http://localhost:3000/gitlab` | unset | `localhost:3000` | `/gitlab` |
+| unset | `/path/to/socket` | `/path/to/socket` | `/` |
+| `http://localhost:3000` | `/path/to/socket` | `/path/to/socket` | `/` |
+| `http://localhost:3000/gitlab` | `/path/to/socket` | `/path/to/socket` | `/gitlab` |
The same applies to `cableBackend` and `cableSocket`.
## Error tracking
-GitLab-Workhorse supports remote error tracking with
-[Sentry](https://sentry.io). To enable this feature set the
-`GITLAB_WORKHORSE_SENTRY_DSN` environment variable.
+GitLab-Workhorse supports remote error tracking with [Sentry](https://sentry.io).
+To enable this feature, set the `GITLAB_WORKHORSE_SENTRY_DSN` environment variable.
You can also set the `GITLAB_WORKHORSE_SENTRY_ENVIRONMENT` environment variable to
-use the Sentry environment functionality to separate staging, production and
+use the Sentry environment feature to separate staging, production and
development.
-Omnibus (`/etc/gitlab/gitlab.rb`):
+Omnibus GitLab (`/etc/gitlab/gitlab.rb`):
```ruby
gitlab_workhorse['env'] = {
@@ -174,46 +171,48 @@ export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar'
export GITLAB_WORKHORSE_SENTRY_ENVIRONMENT='production'
```
-## Distributed Tracing
+## Distributed tracing
-Workhorse supports distributed tracing through [LabKit](https://gitlab.com/gitlab-org/labkit/) using [OpenTracing APIs](https://opentracing.io).
+Workhorse supports distributed tracing through [LabKit](https://gitlab.com/gitlab-org/labkit/)
+using [OpenTracing APIs](https://opentracing.io).
-By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints) or build constraints. This can be done by setting the `BUILD_TAGS` make variable.
+By default, no tracing implementation is linked into the binary. You can link in
+different OpenTracing providers with [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints)
+or build constraints by setting the `BUILD_TAGS` make variable.
-For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"`.
+For more details of the supported providers, refer to LabKit. For an example of
+Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"` like this:
```shell
make BUILD_TAGS="tracer_static tracer_static_jaeger"
```
-Once Workhorse is compiled with an opentracing provider, the tracing configuration is configured via the `GITLAB_TRACING` environment variable.
-
-For example:
+After you compile Workhorse with an OpenTracing provider, configure the tracing
+configuration with the `GITLAB_TRACING` environment variable, like this:
```shell
GITLAB_TRACING=opentracing://jaeger ./gitlab-workhorse
```
-## Continuous Profiling
-
-Workhorse supports continuous profiling through [LabKit](https://gitlab.com/gitlab-org/labkit/) using [Stackdriver Profiler](https://cloud.google.com/profiler).
+## Continuous profiling
-By default, the Stackdriver Profiler implementation is linked in the binary using [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints), though it's not
-required and can be skipped.
-
-For example:
+Workhorse supports continuous profiling through [LabKit](https://gitlab.com/gitlab-org/labkit/)
+using [Stackdriver Profiler](https://cloud.google.com/profiler). By default, the
+Stackdriver Profiler implementation is linked in the binary using
+[build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints), though it's not
+required and can be skipped. For example:
```shell
make BUILD_TAGS=""
```
-Once Workhorse is compiled with Continuous Profiling, the profiler configuration can be set via `GITLAB_CONTINUOUS_PROFILING`
-environment variable.
-
-For example:
+After you compile Workhorse with continuous profiling, set the profiler configuration
+with the `GITLAB_CONTINUOUS_PROFILING` environment variable. For example:
```shell
GITLAB_CONTINUOUS_PROFILING="stackdriver?service=workhorse&service_version=1.0.1&project_id=test-123 ./gitlab-workhorse"
```
-More information about see the [LabKit monitoring documentation](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).
+## Related topics
+
+- [LabKit monitoring documentation](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).
diff --git a/doc/development/workhorse/gitlab_features.md b/doc/development/workhorse/gitlab_features.md
index cdec2a8f0c1..2aa8d9d2399 100644
--- a/doc/development/workhorse/gitlab_features.md
+++ b/doc/development/workhorse/gitlab_features.md
@@ -10,7 +10,7 @@ Workhorse itself is not a feature, but there are several features in
GitLab that would not work efficiently without Workhorse.
To put the efficiency benefit in context, consider that in 2020Q3 on
-GitLab.com [we see][https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1]
+GitLab.com [we see](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1)
Rails application threads using on average
about 200MB of RSS vs about 200KB for Workhorse goroutines.
diff --git a/doc/development/workhorse/new_features.md b/doc/development/workhorse/new_features.md
index 6b17af0fc34..3ad15c1de16 100644
--- a/doc/development/workhorse/new_features.md
+++ b/doc/development/workhorse/new_features.md
@@ -7,40 +7,72 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Adding new features to Workhorse
GitLab Workhorse is a smart reverse proxy for GitLab. It handles
-"long" HTTP requests such as file downloads, file uploads, Git
-push/pull and Git archive downloads.
-
-Workhorse itself is not a feature, but there are [several features in GitLab](gitlab_features.md) that would not work efficiently without Workhorse.
-
-At a first glance, it may look like Workhorse is just a pipeline for processing HTTP streams so that you can reduce the amount of logic in your Ruby on Rails controller, but there are good reasons to avoid treating it like that.
-
-Engineers embarking on the quest of offloading a feature to Workhorse often find that the endeavor is much higher than what originally anticipated. In part because of the new programming language (only a few engineers at GitLab are Go developers), in part because of the demanding requirements for Workhorse. Workhorse is stateless, memory and disk usage must be kept under tight control, and the request should not be slowed down in the process.
-
-## Can I add a new feature to Workhorse?
-
-We suggest to follow this route only if absolutely necessary and no other options are available.
-
-Splitting a feature between the Rails code-base and Workhorse is deliberately choosing to introduce technical debt. It adds complexity to the system and coupling between the two components.
-
-- Building features using Workhorse has a considerable complexity cost, so you should prefer designs based on Rails requests and Sidekiq jobs.
-- Even when using Rails+Sidekiq is "more work" than using Rails+Workhorse, Rails+Sidekiq is easier to maintain in the long term because Workhorse is unique to GitLab while Rails+Sidekiq is an industry standard.
-- For "global" behaviors around web requests consider using a Rack middleware instead of Workhorse.
-- Generally speaking, we should only use Rails+Workhorse if the HTTP client expects behavior that is not reasonable to implement in Rails, like "long" requests.
-
-## What is a "long" request?
-
-There is one order of magnitude between Workhorse and Puma RAM usage. Having connection open for a period longer than milliseconds is a problem because of the amount of RAM it monopolizes once it reaches the Ruby on Rails controller.
-
-So far we identified two classes of "long" requests: data transfers and HTTP long polling.
-
-`git push`, `git pull`, uploading or downloading an artifact, the CI runner waiting for a new job are all good examples of long requests.
-
-With the rise of cloud-native installations, Workhorse's feature-set was extended to add object storage direct-upload, to get rid of the shared Network File System (NFS) drives.
-
-In 2020 @nolith presented at FOSDEM a talk titled [_Speed up the monolith. Building a smart reverse proxy in Go_](https://archive.fosdem.org/2020/schedule/event/speedupmonolith/).
-You can watch the recording for more details on the history of Workhorse and the NFS removal.
-
-[Uploads development documentation](../uploads.md)
-contains the most common use-cases for adding a new type of upload and may answer all of your questions.
-
-If you still think we should add a new feature to Workhorse, please open an issue explaining **what you want to implement** and **why it can't be implemented in our Ruby code-base**. Workhorse maintainers will be happy to help you assessing the situation.
+[long HTTP requests](#what-are-long-requests), such as:
+
+- File downloads.
+- File uploads.
+- Git pushes and pulls.
+- Git archive downloads.
+
+Workhorse itself is not a feature, but [several features in GitLab](gitlab_features.md)
+would not work efficiently without Workhorse.
+
+At a first glance, Workhorse appears to be just a pipeline for processing HTTP
+streams to reduce the amount of logic in your Ruby on Rails controller. However,
+don't treat it that way. Engineers trying to offload a feature to Workhorse often
+find it takes more work than originally anticipated:
+
+- It's a new programming language, and only a few engineers at GitLab are Go developers.
+- Workhorse has demanding requirements:
+ - It's stateless.
+ - Memory and disk usage must be kept under tight control.
+ - The request should not be slowed down in the process.
+
+## Avoid adding new features
+
+We suggest adding new features only if absolutely necessary and no other options exist.
+Splitting a feature between the Rails codebase and Workhorse is a deliberate choice
+to introduce technical debt. It adds complexity to the system, and coupling between
+the two components:
+
+- Building features using Workhorse has a considerable complexity cost, so you should
+ prefer designs based on Rails requests and Sidekiq jobs.
+- Even when using Rails-and-Sidekiq is more work than using Rails-and-Workhorse,
+ Rails-and-Sidekiq is easier to maintain in the long term. Workhorse is unique
+ to GitLab, while Rails-and-Sidekiq is an industry standard.
+- For global behaviors around web requests, consider using a Rack middleware
+ instead of Workhorse.
+- Generally speaking, use Rails-and-Workhorse only if the HTTP client expects
+ behavior reasonable to implement in Rails, like long requests.
+
+## What are long requests?
+
+One order of magnitude exists between Workhorse and Puma RAM usage. Having a connection
+open for longer than milliseconds is problematic due to the amount of RAM
+it monopolizes after it reaches the Ruby on Rails controller. We've identified two classes
+of long requests: data transfers and HTTP long polling. Some examples:
+
+- `git push`.
+- `git pull`.
+- Uploading or downloading an artifact.
+- A CI runner waiting for a new job.
+
+With the rise of cloud-native installations, Workhorse's feature set was extended
+to add object storage direct-upload. This change removed the need for the shared
+Network File System (NFS) drives.
+
+If you still think we should add a new feature to Workhorse, open an issue for the
+Workhorse maintainers and explain:
+
+1. What you want to implement.
+1. Why it can't be implemented in our Ruby codebase.
+
+The Workhorse maintainers can help you assess the situation.
+
+## Related topics
+
+- In 2020, `@nolith` presented the talk
+ ["Speed up the monolith. Building a smart reverse proxy in Go"](https://archive.fosdem.org/2020/schedule/event/speedupmonolith/)
+ at FOSDEM. The talk includes more details on the history of Workhorse and the NFS removal.
+- The [uploads development documentation](../uploads.md) contains the most common
+ use cases for adding a new type of upload.