diff options
Diffstat (limited to 'workhorse/doc/architecture')
| -rw-r--r-- | workhorse/doc/architecture/channel.md | 194 | ||||
| -rw-r--r-- | workhorse/doc/architecture/gitlab_features.md | 69 |
2 files changed, 263 insertions, 0 deletions
diff --git a/workhorse/doc/architecture/channel.md b/workhorse/doc/architecture/channel.md new file mode 100644 index 00000000000..8423405a9cf --- /dev/null +++ b/workhorse/doc/architecture/channel.md @@ -0,0 +1,194 @@ +# Websocket channel support + +In some cases, GitLab can provide in-browser terminal access to an +environment (which is a running server or container, onto which a +project has been deployed), or even access to services running in CI +through a WebSocket. Workhorse manages the WebSocket upgrade and +long-lived connection to the websocket connection, which frees +up GitLab to process other requests. + +This document outlines the architecture of these connections. + +## Introduction to WebSockets + +A websocket is an "upgraded" HTTP/1.1 request. Their purpose is to +permit bidirectional communication between a client and a server. +**Websockets are not HTTP**. Clients can send messages (known as +frames) to the server at any time, and vice-versa. Client messages +are not necessarily requests, and server messages are not necessarily +responses. WebSocket URLs have schemes like `ws://` (unencrypted) or +`wss://` (TLS-secured). + +When requesting an upgrade to WebSocket, the browser sends a HTTP/1.1 +request that looks like this: + +``` +GET /path.ws HTTP/1.1 +Connection: upgrade +Upgrade: websocket +Sec-WebSocket-Protocol: terminal.gitlab.com +# More headers, including security measures +``` + +At this point, the connection is still HTTP, so this is a request and +the server can send a normal HTTP response, including `404 Not Found`, +`500 Internal Server Error`, etc. + +If the server decides to permit the upgrade, it will send a HTTP +`101 Switching Protocols` response. From this point, the connection +is no longer HTTP. It is a WebSocket and frames, not HTTP requests, +will flow over it. The connection will persist until the client or +server closes the connection. + +In addition to the subprotocol, individual websocket frames may +also specify a message type - examples include `BinaryMessage`, +`TextMessage`, `Ping`, `Pong` or `Close`. Only binary frames can +contain arbitrary data - other frames are expected to be valid +UTF-8 strings, in addition to any subprotocol expectations. + +## Browser to Workhorse + +Using the terminal as an example, GitLab serves a JavaScript terminal +emulator to the browser on a URL like +`https://gitlab.com/group/project/-/environments/1/terminal`. +This opens a websocket connection to, e.g., +`wss://gitlab.com/group/project/-/environments/1/terminal.ws`, +This endpoint doesn't exist in GitLab - only in Workhorse. + +When receiving the connection, Workhorse first checks that the +client is authorized to access the requested terminal. It does +this by performing a "preauthentication" request to GitLab. + +If the client has the appropriate permissions and the terminal +exists, GitLab responds with a successful response that includes +details of the terminal that the client should be connected to. +Otherwise, it returns an appropriate HTTP error response. + +Errors are passed back to the client as HTTP responses, but if +GitLab returns valid terminal details to Workhorse, it will +connect to the specified terminal, upgrade the browser to a +WebSocket, and proxy between the two connections for as long +as the browser's credentials are valid. Workhorse will also +send regular `PingMessage` control frames to the browser, to +keep intervening proxies from terminating the connection +while the browser is present. + +The browser must request an upgrade with a specific subprotocol: + +### `terminal.gitlab.com` + +This subprotocol considers `TextMessage` frames to be invalid. +Control frames, such as `PingMessage` or `CloseMessage`, have +their usual meanings. + +`BinaryMessage` frames sent from the browser to the server are +arbitrary text input. + +`BinaryMessage` frames sent from the server to the browser are +arbitrary text output. + +These frames are expected to contain ANSI text control codes +and may be in any encoding. + +### `base64.terminal.gitlab.com` + +This subprotocol considers `BinaryMessage` frames to be invalid. +Control frames, such as `PingMessage` or `CloseMessage`, have +their usual meanings. + +`TextMessage` frames sent from the browser to the server are +base64-encoded arbitrary text input (so the server must +base64-decode them before inputting them). + +`TextMessage` frames sent from the server to the browser are +base64-encoded arbitrary text output (so the browser must +base64-decode them before outputting them). + +In their base64-encoded form, these frames are expected to +contain ANSI terminal control codes, and may be in any encoding. + +## Workhorse to GitLab + +Using again the terminal as an example, before upgrading the browser, +Workhorse sends a normal HTTP request to GitLab on a URL like +`https://gitlab.com/group/project/environments/1/terminal.ws/authorize`. +This returns a JSON response containing details of where the +terminal can be found, and how to connect it. In particular, +the following details are returned in case of success: + +* WebSocket URL to **connect** to, e.g.: `wss://example.com/terminals/1.ws?tty=1` +* WebSocket subprotocols to support, e.g.: `["channel.k8s.io"]` +* Headers to send, e.g.: `Authorization: Token xxyyz..` +* Certificate authority to verify `wss` connections with (optional) + +Workhorse periodically re-checks this endpoint, and if it gets an +error response, or the details of the terminal change, it will +terminate the websocket session. + +## Workhorse to the WebSocket server + +In GitLab, environments or CI jobs may have a deployment service (e.g., +`KubernetesService`) associated with them. This service knows +where the terminals or the service for an environment may be found, and these +details are returned to Workhorse by GitLab. + +These URLs are *also* WebSocket URLs, and GitLab tells Workhorse +which subprotocols to speak over the connection, along with any +authentication details required by the remote end. + +Before upgrading the browser's connection to a websocket, +Workhorse opens a HTTP client connection, according to the +details given to it by Workhorse, and attempts to upgrade +that connection to a websocket. If it fails, an error +response is sent to the browser; otherwise, the browser is +also upgraded. + +Workhorse now has two websocket connections, albeit with +differing subprotocols. It decodes incoming frames from the +browser, re-encodes them to the the channel's subprotocol, and +sends them to the channel. Similarly, it decodes incoming +frames from the channel, re-encodes them to the browser's +subprotocol, and sends them to the browser. + +When either connection closes or enters an error state, +Workhorse detects the error and closes the other connection, +terminating the channel session. If the browser is the +connection that has disconnected, Workhorse will send an ANSI +`End of Transmission` control code (the `0x04` byte) to the +channel, encoded according to the appropriate subprotocol. +Workhorse will automatically reply to any websocket ping frame +sent by the channel, to avoid being disconnected. + +Currently, Workhorse only supports the following subprotocols. +Supporting new deployment services will require new subprotocols +to be supported: + +### `channel.k8s.io` + +Used by Kubernetes, this subprotocol defines a simple multiplexed +channel. + +Control frames have their usual meanings. `TextMessage` frames are +invalid. `BinaryMessage` frames represent I/O to a specific file +descriptor. + +The first byte of each `BinaryMessage` frame represents the file +descriptor (fd) number, as a `uint8` (so the value `0x00` corresponds +to fd 0, `STDIN`, while `0x01` corresponds to fd 1, `STDOUT`). + +The remaining bytes represent arbitrary data. For frames received +from the server, they are bytes that have been received from that +fd. For frames sent to the server, they are bytes that should be +written to that fd. + +### `base64.channel.k8s.io` + +Also used by Kubernetes, this subprotocol defines a similar multiplexed +channel to `channel.k8s.io`. The main differences are: + +* `TextMessage` frames are valid, rather than `BinaryMessage` frames. +* The first byte of each `TextMessage` frame represents the file + descriptor as a numeric UTF-8 character, so the character `U+0030`, + or "0", is fd 0, STDIN). +* The remaining bytes represent base64-encoded arbitrary data. + diff --git a/workhorse/doc/architecture/gitlab_features.md b/workhorse/doc/architecture/gitlab_features.md new file mode 100644 index 00000000000..2d3ca78f5e8 --- /dev/null +++ b/workhorse/doc/architecture/gitlab_features.md @@ -0,0 +1,69 @@ +# Features that rely on Workhorse + +Workhorse itself is not a feature, but there are several features in +GitLab that would not work efficiently without Workhorse. + +To put the efficiency benefit in context, consider that in 2020Q3 on +GitLab.com [we see][thanos] Rails application threads using on average +about 200MB of RSS vs about 200KB for Workhorse goroutines. + +Examples of features that rely on Workhorse: + +## 1. `git clone` and `git push` over HTTP + +Git clone, pull and push are slow because they transfer large amounts +of data and because each is CPU intensive on the GitLab side. Without +Workhorse, HTTP access to Git repositories would compete with regular +web access to the application, requiring us to run way more Rails +application servers. + +## 2. CI runner long polling + +GitLab CI runners fetch new CI jobs by polling the GitLab server. +Workhorse acts as a kind of "waiting room" where CI runners can sit +and wait for new CI jobs. Because of Go's efficiency we can fit a lot +of runners in the waiting room at little cost. Without this waiting +room mechanism we would have to add a lot more Rails server capacity. + +## 3. File uploads and downloads + +File uploads and downloads may be slow either because the file is +large or because the user's connection is slow. Workhorse can handle +the slow part for Rails. This improves the efficiency of features such +as CI artifacts, package repositories, LFS objects, etc. + +## 4. Websocket proxying + +Features such as the web terminal require a long lived connection +between the user's web browser and a container inside GitLab that is +not directly accessible from the internet. Dedicating a Rails +application thread to proxying such a connection would cost much more +memory than it costs to have Workhorse look after it. + +## Quick facts (how does Workhorse work) + +- Workhorse can handle some requests without involving Rails at all: + for example, JavaScript files and CSS files are served straight + from disk. +- Workhorse can modify responses sent by Rails: for example if you use + `send_file` in Rails then GitLab Workhorse will open the file on + disk and send its contents as the response body to the client. +- Workhorse can take over requests after asking permission from Rails. + Example: handling `git clone`. +- Workhorse can modify requests before passing them to Rails. Example: + when handling a Git LFS upload Workhorse first asks permission from + Rails, then it stores the request body in a tempfile, then it sends + a modified request containing the tempfile path to Rails. +- Workhorse can manage long-lived WebSocket connections for Rails. + Example: handling the terminal websocket for environments. +- Workhorse does not connect to PostgreSQL, only to Rails and (optionally) Redis. +- We assume that all requests that reach Workhorse pass through an + upstream proxy such as NGINX or Apache first. +- Workhorse does not accept HTTPS connections. +- Workhorse does not clean up idle client connections. +- We assume that all requests to Rails pass through Workhorse. + +For more information see ['A brief history of GitLab Workhorse'][brief-history-blog]. + +[thanos]: https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1 +[brief-history-blog]: https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/ |