diff options
author | Evan Read <eread@gitlab.com> | 2022-10-19 04:22:33 +0300 |
---|---|---|
committer | Evan Read <eread@gitlab.com> | 2022-10-19 04:22:33 +0300 |
commit | a9a6a610517985099b32cd4d8e0520838ed3f614 (patch) | |
tree | 47018ea2a1003fe7e3c5e7dab849a569190ac9a8 /doc | |
parent | 334a620a54df6bbb1563c440514e06d7068255e7 (diff) |
Add Markdownlint configuration file and tidy up some Markdown files
Diffstat (limited to 'doc')
-rw-r--r-- | doc/configuration/README.md | 96 | ||||
-rw-r--r-- | doc/configuration/logging.md | 2 | ||||
-rw-r--r-- | doc/configuration/praefect.md | 10 | ||||
-rw-r--r-- | doc/rfcs/README.md | 3 | ||||
-rw-r--r-- | doc/rfcs/praefect-queue-storage.md | 68 | ||||
-rw-r--r-- | doc/rfcs/snapshot-storage.md | 20 | ||||
-rw-r--r-- | doc/rfcs/template.md | 1 |
7 files changed, 99 insertions, 101 deletions
diff --git a/doc/configuration/README.md b/doc/configuration/README.md index 4f29c3e69..4ece83e2a 100644 --- a/doc/configuration/README.md +++ b/doc/configuration/README.md @@ -6,13 +6,13 @@ application. Gitaly is configured via a [TOML](https://github.com/toml-lang/toml) configuration file. Where this TOML file is located and how you should edit it depend on how you installed GitLab. See: -https://docs.gitlab.com/ce/administration/gitaly/ +<https://docs.gitlab.com/ee/administration/gitaly/> The configuration file is passed as an argument to the `gitaly` -executable. This is usually done by either omnibus-gitlab or your init +executable. This is usually done by either `omnibus-gitlab` or your init script. -``` +```shell gitaly /path/to/config.toml ``` @@ -38,14 +38,14 @@ name = "my_shard" #path = "/path/to/other/repositories" ``` -|name|type|required|notes| -|----|----|--------|-----| -|socket_path|string|see notes|A path which gitaly should open a Unix socket. Required unless listen_addr is set| -|listen_addr|string|see notes|TCP address for Gitaly to listen on (See #GITALY_LISTEN_ADDR). Required unless socket_path is set| -|internal_socket_dir|string|yes|Path where Gitaly will create sockets for internal Gitaly calls to connect to| -|bin_dir|string|yes|Directory containing Gitaly's executables| -|prometheus_listen_addr|string|no|TCP listen address for Prometheus metrics. If not set, no Prometheus listener is started| -|storage|array|yes|An array of storage shards| +| Name | Type | Required | Notes | +|:-------------------------|:-------|:----------|:----------------------------------------------------------------------------------------------------| +| `socket_path` | string | see notes | A path which Gitaly should open a Unix socket. Required unless listen_addr is set | +| `listen_addr` | string | see notes | TCP address for Gitaly to listen on (See #GITALY_LISTEN_ADDR). Required unless `socket_path` is set | +| `internal_socket_dir` | string | yes | Path where Gitaly will create sockets for internal Gitaly calls to connect to | +| `bin_dir` | string | yes | Directory containing Gitaly's executables | +| `prometheus_listen_addr` | string | no | TCP listen address for Prometheus metrics. If not set, no Prometheus listener is started | +| `storage` | array | yes | An array of storage shards | ### Authentication @@ -53,7 +53,7 @@ Gitaly can be configured to reject requests that do not contain a specific bearer token in their headers. This is a security measure to be used when serving requests over TCP. -Authentication is disabled when the token setting in config.toml is absent or the empty string. +Authentication is disabled when the token setting in `config.toml` is absent or the empty string. ```toml [auth] @@ -85,31 +85,31 @@ GitLab repositories are grouped into 'storages'. These are directories by GitLab , with names (e.g. `default`). These names and paths are also defined in the `gitlab.yml` -configuration file of gitlab-ce (or gitlab-ee). When you run Gitaly on -the same machine as gitlab-ce, which is the default and recommended -configuration, storage paths defined in Gitaly's config.toml must -match those in gitlab.yml. +configuration file of `gitlab-foss` (or `gitlab`). When you run Gitaly on +the same machine as `gitlab-foss`, which is the default and recommended +configuration, storage paths defined in Gitaly's `config.toml` must +match those in `gitlab.yml`. -|name|type|required|notes| -|----|----|--------|-----| -|path|string|yes|Path to storage shard| -|name|string|yes|Name of storage shard| +| Name | Type | Required | Notes | +|:-------|:-------|:---------|:----------------------| +| `path` | string | yes | Path to storage shard | +| `name` | string | yes | Name of storage shard | ### Git The following values can be set in the `[git]` section of the configuration file: -|name|type|required|notes| -|----|----|--------|-----| -|bin_path|string|no|Path to git binary. If not set, will be resolved using PATH.| -|catfile_cache_size|integer|no|Maximum number of cached cat-file processes (see below). Default 100.| +| Name | Type | Required | Notes | +|:---------------------|:--------|:---------|:----------------------------------------------------------------------| +| `bin_path` | string | no | Path to Git binary. If not set, will be resolved using PATH. | +| `catfile_cache_size` | integer | no | Maximum number of cached cat-file processes (see below). Default 100. | #### cat-file cache A lot of Gitaly RPC's need to look up Git objects from repositories. Most of the time we use `git cat-file --batch` processes for that. For the sake of performance, Gitaly can re-use thse `git cat-file` processes -across RPC calls. Previously used processes are kept around in a "git +across RPC calls. Previously used processes are kept around in a "Git cat-file cache". In order to control how much system resources this uses we have a maximum number of cat-file processes that can go into the cache. @@ -123,29 +123,29 @@ Ideally the number should be large enough to handle normal (peak) traffic. If you raise the limit you should measure the cache hit ratio before and after. If the hit ratio does not improve, the higher limit is probably not making a meaningful difference. Here is an example -prometheus query to see the hit rate: +Prometheus query to see the hit rate: -``` +```prometheus sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m])) ``` -### gitaly-ruby +### `gitaly-ruby` -A Gitaly process uses one or more gitaly-ruby helper processes to +A Gitaly process uses one or more `gitaly-ruby` helper processes to execute RPC's implemented in Ruby instead of Go. The `[gitaly-ruby]` section of the config file contains settings for these helper processes. These processes are known to occasionally suffer from memory leaks. -Gitaly restarts its gitaly-ruby helpers when their memory exceeds the +Gitaly restarts its `gitaly-ruby` helpers when their memory exceeds the max\_rss limit. -|name|type|required|notes| -|----|----|--------|-----| -|dir|string|yes|Path to where gitaly-ruby is installed (needed to boot the process).| -|max_rss|integer|no|Resident set size limit that triggers a gitaly-ruby restart, in bytes. Default 300MB.| -|graceful_restart_timeout|string|no|Grace period to allow a gitaly-ruby process to finish ongoing requests. Default 10 minutes ("10m").| -|restart_delay|string|no|Time memory must be high before a restart is triggered, in seconds. Default 5 minutes ("5m").| -|num_workers|integer|no|Number of gitaly-ruby worker processes. Try increasing this number in case of ResourceExhausted errors. Default 2, minimum 2.| +| Name | Type | Required | Notes | +|:---------------------------|:--------|:---------|:--------------------------------------------------------------------------------------------------------------------------------| +| `dir` | string | yes | Path to where `gitaly-ruby` is installed (needed to boot the process). | +| `max_rss` | integer | no | Resident set size limit that triggers a `gitaly-ruby` restart, in bytes. Default 300MB. | +| `graceful_restart_timeout` | string | no | Grace period to allow a `gitaly-ruby` process to finish ongoing requests. Default 10 minutes ("10m"). | +| `restart_delay` | string | no | Time memory must be high before a restart is triggered, in seconds. Default 5 minutes ("5m"). | +| `num_workers` | integer | no | Number of `gitaly-ruby` worker processes. Try increasing this number in case of ResourceExhausted errors. Default 2, minimum 2. | ### Logging @@ -156,20 +156,20 @@ Example: level = "warn" ``` -|name|type|required|notes| -|----|----|--------|-----| -|format|string|no|Log format: "text" or "json". Default: "text"| -|level|string|no| Log level: "debug", "info", "warn", "error", "fatal", or "panic". Default: "info"| -|sentry_dsn|string|no|Sentry DSN for exception monitoring| -|sentry_environment|string|no|Sentry Environment for exception monitoring| -|ruby_sentry_dsn|string|no|Sentry DSN for gitaly-ruby exception monitoring| +| Name | Type | Required | Notes | +|:---------------------|:-------|:---------|:----------------------------------------------------------------------------------| +| `format` | string | no | Log format: `text` or `json`. Default: `text` | +| `level` | string | no | Log level: `debug`, `info`, `warn`, `error`, `fatal`, or `panic`. Default: `info` | +| `sentry_dsn` | string | no | Sentry DSN for exception monitoring | +| `sentry_environment` | string | no | Sentry Environment for exception monitoring | +| `ruby_sentry_dsn` | string | no | Sentry DSN for `gitaly-ruby` exception monitoring | #### Environment variables -|name|default|notes| -|----|-------|-----| -|GITALY_LOG_REQUEST_METHOD_ALLOW_PATTERN||Regular expression that controls which gRPC methods should be logged| -|GITALY_LOG_REQUEST_METHOD_DENY_PATTERN|`^/grpc.health.v1.Health/Check$`|Regular expression that controls which gRPC methods should be filtered out| +| Name | Default | Notes | +|:------------------------------------------|:---------------------------------|:---------------------------------------------------------------------------| +| `GITALY_LOG_REQUEST_METHOD_ALLOW_PATTERN` | | Regular expression that controls which gRPC methods should be logged | +| `GITALY_LOG_REQUEST_METHOD_DENY_PATTERN` | `^/grpc.health.v1.Health/Check$` | Regular expression that controls which gRPC methods should be filtered out | Note that `GITALY_LOG_REQUEST_METHOD_ALLOW_PATTERN` takes precedence over `GITALY_LOG_REQUEST_METHOD_DENY_PATTERN`. If a pattern matches in diff --git a/doc/configuration/logging.md b/doc/configuration/logging.md index e5eb7a74a..87f1bf968 100644 --- a/doc/configuration/logging.md +++ b/doc/configuration/logging.md @@ -1 +1 @@ -This documentation was moved to https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/administration/gitaly/reference.md#logging. +This documentation was moved to <https://docs.gitlab.com/ee/administration/gitaly/reference.html#logging>. diff --git a/doc/configuration/praefect.md b/doc/configuration/praefect.md index 57c1ed79e..0401ec7b0 100644 --- a/doc/configuration/praefect.md +++ b/doc/configuration/praefect.md @@ -1,16 +1,16 @@ # Configuring Praefect -This document describes how to configure the praefect server. +This document describes how to configure the Praefect server. Praefect is configured via a [TOML](https://github.com/toml-lang/toml) configuration file. The TOML file contents and location depends on how you -installed GitLab. See: https://docs.gitlab.com/ce/administration/gitaly/ +installed GitLab. See: <https://docs.gitlab.com/ee/administration/gitaly/> The configuration file is passed as an argument to the `praefect` -executable. This is usually done by either omnibus-gitlab or your init +executable. This is usually done by either `omnibus-gitlab` or your init script. -``` +```shell praefect -config /path/to/config.toml ``` @@ -38,4 +38,4 @@ name = 'praefect' token = 'secret_token' ``` -An example [config toml](../../config.praefect.toml.example) is stored in this repository. +An example [config TOML](../../config.praefect.toml.example) is stored in this repository. diff --git a/doc/rfcs/README.md b/doc/rfcs/README.md index a7d2a0aa0..d044829e6 100644 --- a/doc/rfcs/README.md +++ b/doc/rfcs/README.md @@ -20,7 +20,7 @@ In order to preserve architectural integrity, we sometimes require thought frame - Use an RFC to vet a complex concept with your peers and reach consensus on an approach. - Use an RFC to retroactively capture the thinking behind design decisions. -Note: this is not an exhaustive list. Be creative :) +Note: this is not an exhaustive list. Be creative `:)` ## When NOT to use an RFC? @@ -39,4 +39,3 @@ Note: this is not an exhaustive list. Be creative :) ### Updating an existing RFC When updating an RFC, use discretion when deciding between using the normal merge request process, or using the more process heavy RFC process. The RFC process should be used for more substantial revisions of an RFC. - diff --git a/doc/rfcs/praefect-queue-storage.md b/doc/rfcs/praefect-queue-storage.md index 992f18ca8..05aa7ab40 100644 --- a/doc/rfcs/praefect-queue-storage.md +++ b/doc/rfcs/praefect-queue-storage.md @@ -16,14 +16,14 @@ is implemented in-memory in the Praefect process. While useful for prototyping, this is unsuitable for real HA Gitaly for two reasons: -1. The job queue must be **persistent**. Currently, the queue is - emptied if a Praefect process restarts. This can lead to data loss - in case we fail over away from a repository that is ahead of its - replicas. -2. The job queue must be **shared**. We expect multiple Praefect - processes to be serving up the same Gitaly storage cluster. This is - so that Praefect itself is not a single point of failure. These - Praefect processes must all see and use the same job queue. +1. The job queue must be **persistent**. Currently, the queue is + emptied if a Praefect process restarts. This can lead to data loss + in case we fail over away from a repository that is ahead of its + replicas. +1. The job queue must be **shared**. We expect multiple Praefect + processes to be serving up the same Gitaly storage cluster. This is + so that Praefect itself is not a single point of failure. These + Praefect processes must all see and use the same job queue. ## Does it have to be a queue? @@ -46,7 +46,7 @@ If a message gets lost, that can lead to data loss. ## What sort of throughput do we expect? -Currently (November 2019), gitlab.com has about 5000 Gitaly calls per +Currently (November 2019), `gitlab.com` has about 5000 Gitaly calls per second. About 300 of those [are labeled as "mutators"](https://prometheus.gprd.gitlab.net/graph?g0.range_input=7d&g0.expr=sum(rate(gitaly_cacheinvalidator_optype_total%5B5m%5D))%20by%20(type)&g0.tab=0), which suggests that today we'd see about 300 replication jobs per @@ -80,13 +80,13 @@ queue database to be highly available itself. As always at GitLab, we want to choose solutions that are suitable for self-managed GitLab installations. -- Should be open source -- Don't pick an open core solution, and rely on features that are not - in the core -- Don't assume that "the cloud" makes problems go away; assume there - is no cloud -- Running the queue database should require as little expertise as - possible, or it should be a commodity component +- Should be open source +- Don't pick an open core solution, and rely on features that are not + in the core +- Don't assume that "the cloud" makes problems go away; assume there + is no cloud +- Running the queue database should require as little expertise as + possible, or it should be a commodity component ## Do we have other database needs in Praefect? @@ -112,43 +112,43 @@ nodes, and those could serve as the source of truth. Pro: -- Already used in GitLab -- Has queue primitives +- Already used in GitLab +- Has queue primitives Con: -- Deployed with snapshot persistence (RDB dump) in GitLab, which is - not the durability I think we want +- Deployed with snapshot persistence (RDB dump) in GitLab, which is + not the durability I think we want ### Postgres Pro: -- Already used in GitLab -- Gold standard for persistence -- General purpose database: likely to be able to grow with us as we - develop other needs +- Already used in GitLab +- Gold standard for persistence +- General purpose database: likely to be able to grow with us as we + develop other needs Con: -- Can be used for queues, but not meant for it -- Need to find queueing library, or develop SQL-backed queue ourselves - (hard, subtle) -- Because not meant to be a queue, may have a lower ceiling where we - are forced to scale horizontally. When we hit the ceiling we would - have to run multiple Praefect clusters each with their own HA - Postgres cluster behind it) +- Can be used for queues, but not meant for it +- Need to find queueing library, or develop SQL-backed queue ourselves + (hard, subtle) +- Because not meant to be a queue, may have a lower ceiling where we + are forced to scale horizontally. When we hit the ceiling we would + have to run multiple Praefect clusters each with their own HA + Postgres cluster behind it) ### Kafka Pro: -- Closely matches description of "durable queue" +- Closely matches description of "durable queue" Con: -- Would be new to GitLab: no development experience nor operational - experience +- Would be new to GitLab: no development experience nor operational + experience ### SQLite or BoltDB diff --git a/doc/rfcs/snapshot-storage.md b/doc/rfcs/snapshot-storage.md index a504c9877..513800bf8 100644 --- a/doc/rfcs/snapshot-storage.md +++ b/doc/rfcs/snapshot-storage.md @@ -9,18 +9,18 @@ repository snapshots in SQL and object storage (e.g. S3). Key properties: -- Use a SQL database as a catalogue of the repositories in snapshot storage -- Git data (objects and refs) is stored as cold "snapshots" in object - storage -- snapshots can have a "parent", so a repository is stored as a linked - list of snapshots. The linked list relation is stored in SQL. -- to use the repository it must first be copied down to a local - filesystem +- Use a SQL database as a catalogue of the repositories in snapshot storage +- Git data (objects and refs) is stored as cold "snapshots" in object + storage +- snapshots can have a "parent", so a repository is stored as a linked + list of snapshots. The linked list relation is stored in SQL. +- to use the repository it must first be copied down to a local + filesystem Possible applications: -- incremental repository backups -- cold storage for repositories +- incremental repository backups +- cold storage for repositories ## Primitives @@ -90,7 +90,7 @@ repositories, and that Praefect is aware of when repositories change and a new snapshot is warranted, it would be a candidate for managing snapshots. -However, we could also build this in gitlab-rails. That should work fine +However, we could also build this in GitLab Rails. That should work fine for periodic snapshots, where we take snapshots regardless of whether we know/think there was a change in the repository. diff --git a/doc/rfcs/template.md b/doc/rfcs/template.md index e58a17fa4..4da510890 100644 --- a/doc/rfcs/template.md +++ b/doc/rfcs/template.md @@ -5,4 +5,3 @@ <REPLACE ABSTRACT> <!--- Replace this line and start writing your RFC. Good luck! --> - |