Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitaly.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorEvan Read <eread@gitlab.com>2022-10-26 01:30:08 +0300
committerPatrick Steinhardt <psteinhardt@gitlab.com>2022-10-27 09:43:53 +0300
commit1824a953bce1268269937a7a66684b642ebee505 (patch)
treefebdc9dbec10bb67797d8a5f7a85650380876101 /README.md
parent1977fb4f1fc1e379688e13b58050111f79b33e84 (diff)
Add docs testing Make target and CI/CD joberead/add-docs-testing-make-target-and-ci_cd-job
Also expand reach of Markdown linting to more files
Diffstat (limited to 'README.md')
-rw-r--r--README.md167
1 files changed, 82 insertions, 85 deletions
diff --git a/README.md b/README.md
index c3e16f2cc..17e7993ad 100644
--- a/README.md
+++ b/README.md
@@ -6,21 +6,19 @@
[GitLab Gitaly Issues](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&state=opened&label_name%5B%5D=Gitaly) |
[GitLab Gitaly Merge Requests](https://gitlab.com/groups/gitlab-org/-/merge_requests?label_name%5B%5D=Gitaly) |
---------------------------------------------
-
Gitaly is a Git [RPC](https://en.wikipedia.org/wiki/Remote_procedure_call)
-service for handling all the git calls made by GitLab.
+service for handling all the Git calls made by GitLab.
To see where it fits in please look at [GitLab's architecture](https://docs.gitlab.com/ce/development/architecture.html#system-layout).
## Project Goals
-Fault-tolerant horizontal scaling of Git storage in GitLab, and particularly, on [gitlab.com](https://gitlab.com).
+Fault-tolerant horizontal scaling of Git storage in GitLab, and particularly, on [GitLab.com](https://gitlab.com).
This will be achieved by focusing on two areas (in this order):
- 1. **Migrate from repository access via NFS to gitaly-proto, GitLab's new Git RPC protocol**
- 1. **Evolve from large Gitaly servers managed as "pets" to smaller Gitaly servers that are "cattle"**
+1. **Migrate from repository access via NFS to `gitaly-proto`, GitLab's new Git RPC protocol**
+1. **Evolve from large Gitaly servers managed as "pets" to smaller Gitaly servers that are "cattle"**
## Current Status
@@ -33,21 +31,20 @@ removed](https://about.gitlab.com/2018/09/12/the-road-to-gitaly-1-0/).
For performance reasons some RPCs can be performed through NFS still. An
effort is made to mitigate performance issues by removing [Gitaly N+1](https://gitlab.com/groups/gitlab-org/-/epics/827).
Once that is no longer necessary we can conclude the migration project by
-[removing the Git repository storage paths from gitlab-rails's
-configuration](https://gitlab.com/gitlab-org/gitaly/issues/1282).
+[removing the Git repository storage paths from GitLab Rails configuration](https://gitlab.com/gitlab-org/gitaly/issues/1282).
In the meantime we are building features according to our [roadmap][roadmap].
If you're interested in seeing how well Gitaly is performing on
GitLab.com, read about our [observability story](doc/observability.md)!
-##### Overall
+### Overall
![image](https://gitlab.com/gitlab-org/gitaly/uploads/c3aa987884d5e78c3567a3a7469ea6c2/overview.png)
[Dashboard](https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview) (The link can be accessed by GitLab team members.)
-##### By Feature
+### By Feature
![image](https://gitlab.com/gitlab-org/gitaly/uploads/3e8a5616863fa17c5bf08cb67c1bb385/feature.png)
@@ -73,10 +70,10 @@ See [CONTRIBUTING.md](CONTRIBUTING.md).
## Name
-Gitaly is a tribute to git and the town of [Aly](https://en.wikipedia.org/wiki/Aly). Where the town of
+Gitaly is a tribute to Git and the town of [Aly](https://en.wikipedia.org/wiki/Aly). Where the town of
Aly has zero inhabitants most of the year we would like to reduce the number of
disk operations to zero for most actions. It doesn't hurt that it sounds like
-Italy, the capital of which is [the destination of all roads](https://en.wikipedia.org/wiki/All_roads_lead_to_Rome). All git actions in
+Italy, the capital of which is [the destination of all roads](https://en.wikipedia.org/wiki/All_roads_lead_to_Rome). All Git actions in
GitLab end up in Gitaly.
## Design
@@ -91,23 +88,23 @@ High-level architecture overview:
As of Q4 2018, the following GitLab components act as Gitaly clients:
-- [gitlab-rails](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/gitaly_client.rb):
- the main GitLab Rails application.
-- [gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell/tree/master):
- for `git clone`, `git push` etc. via SSH.
-- [gitlab-workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/internal/gitaly/gitaly.go):
- for `git clone` via HTTPS and for slow requests that serve raw Git
- data.
- ([example](https://gitlab.com/gitlab-org/gitaly/raw/master/README.md))
-- [gitaly-ssh](https://gitlab.com/gitlab-org/gitaly/tree/master/cmd/gitaly-ssh):
- for internal Git data transfers between Gitaly servers.
-- [gitaly-ruby](https://gitlab.com/gitlab-org/gitaly/blob/master/ruby/lib/gitlab/git/gitaly_remote_repository.rb):
- for RPC's that interact with more than one repository, such as
- merging a branch.
-
-The clients written in Go (gitlab-shell, gitlab-workhorse, gitaly-ssh)
+- [`gitlab`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/gitaly_client.rb):
+ the main GitLab Rails application.
+- [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell/tree/main):
+ for `git clone`, `git push` etc. via SSH.
+- [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/workhorse/internal/gitaly/gitaly.go):
+ for `git clone` via HTTPS and for slow requests that serve raw Git
+ data.
+ ([example](https://gitlab.com/gitlab-org/gitaly/raw/master/README.md))
+- [`gitaly-ssh`](https://gitlab.com/gitlab-org/gitaly/tree/master/cmd/gitaly-ssh):
+ for internal Git data transfers between Gitaly servers.
+- [`gitaly-ruby`](https://gitlab.com/gitlab-org/gitaly/blob/master/ruby/lib/gitlab/git/gitaly_remote_repository.rb):
+ for RPC's that interact with more than one repository, such as
+ merging a branch.
+
+The clients written in Go (`gitlab-shell`, `gitlab-workhorse`, `gitaly-ssh`)
use library code from the
-[gitlab.com/gitlab-org/gitaly/client](https://gitlab.com/gitlab-org/gitaly/tree/master/client)
+[`gitlab.com/gitlab-org/gitaly/client`](https://gitlab.com/gitlab-org/gitaly/tree/master/client)
package.
## High Availability
@@ -123,13 +120,13 @@ Praefect, which is hosted alongside the rest of Gitaly in this
repository. As we currently envision it, Praefect will have four
responsibilities:
-- route RPC traffic to the primary Gitaly server
-- inspect RPC traffic and mark repositories as dirty if the RPC is a
- "mutator"
-- ensure dirty repositories have their changes replicated to the
- secondary Gitaly servers
-- in the event of a failure on the primary, demote it to secondary and
- elect a new primary
+- route RPC traffic to the primary Gitaly server
+- inspect RPC traffic and mark repositories as dirty if the RPC is a
+ "mutator"
+- ensure dirty repositories have their changes replicated to the
+ secondary Gitaly servers
+- in the event of a failure on the primary, demote it to secondary and
+ elect a new primary
Praefect has internal state: it needs to be able to "remember" which
repositories are in need of replication, and which Gitaly server is the
@@ -152,7 +149,7 @@ By default, no tracing implementation is linked into the binary, but different O
For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"`.
```shell
-$ make BUILD_TAGS="tracer_static tracer_static_jaeger"
+make BUILD_TAGS="tracer_static tracer_static_jaeger"
```
Once Gitaly is compiled with an opentracing provider, the tracing configuration is configured via the `GITLAB_TRACING` environment variable.
@@ -170,75 +167,75 @@ Gitaly supports Continuous Profiling through [LabKit][] using [Stackdriver Profi
For more information on how to set it up, see the [LabKit monitoring docs](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).
## Presentations
+
- [Praefect code walkthrough](https://youtu.be/w2R4ptDLPH4)
- A walkthrough of the Praefect codebase.
+ A walkthrough of the Praefect codebase.
- [How to configure backpressure in Gitaly](https://youtu.be/wX9CtFdLYxE)
- An overview of the knobs in the Gitaly config to set limits on incoming traffic.
- There is also [written documentation](doc/backpressure.md).
+ An overview of the knobs in the Gitaly config to set limits on incoming traffic.
+ There is also [written documentation](doc/backpressure.md).
- [How Gitaly fits into GitLab (Youtube)](https://www.youtube.com/playlist?list=PL05JrBw4t0KqoFUiX42JG7BAc7pipMBAy) - a series of 1-hour training videos for contributors new to GitLab and Gitaly.
- - [Part 1: the Gitaly client in gitlab-ce, 2019-02-21](https://www.youtube.com/watch?v=j0HNiKCnLTI&list=PL05JrBw4t0KqoFUiX42JG7BAc7pipMBAy)
+ - [Part 1: the Gitaly client in `gitlab-ce`, 2019-02-21](https://www.youtube.com/watch?v=j0HNiKCnLTI&list=PL05JrBw4t0KqoFUiX42JG7BAc7pipMBAy)
- Overview of GitLab backend processes, gitlab-rails deep dive: Gitaly
- config in gitlab-rails, SQL data model, overview of how Gitaly calls get
- made via GitalyClient.call.
+ Overview of GitLab backend processes, GitLab Rails deep dive: Gitaly
+ config in GitLab Rails, SQL data model, overview of how Gitaly calls get
+ made via GitalyClient.call.
- [Part 2: Git SSH, 2019-02-28](https://www.youtube.com/watch?v=0kY0HPFn25o&list=PL05JrBw4t0KqoFUiX42JG7BAc7pipMBAy)
- What is in a gitaly-proto Repository message, legacy vs
- hashed storage (repository directories), `git clone` via SSH,
- gitlab-shell, `authorized_keys` and forced commands, what happens
- during `git push`.
+ What is in a `gitaly-proto` Repository message, legacy vs
+ hashed storage (repository directories), `git clone` via SSH,
+ `gitlab-shell`, `authorized_keys` and forced commands, what happens
+ during `git push`.
- [Part 3: Git push, 2019-03-07](https://www.youtube.com/watch?v=-kXYycFYDzo&list=PL05JrBw4t0KqoFUiX42JG7BAc7pipMBAy)
- A closer look at the final stage of `git push` where the git hooks run
- and the refs get updated. Interaction between the git hooks and GitLab
- internal API. The Git
- [object quarantine mechanism](https://git-scm.com/docs/git-receive-pack#_quarantine_environment).
- Preview of Git HTTP (to be discussed next time).
+ A closer look at the final stage of `git push` where the Git hooks run
+ and the refs get updated. Interaction between the Git hooks and GitLab
+ internal API. The Git
+ [object quarantine mechanism](https://git-scm.com/docs/git-receive-pack#_quarantine_environment).
+ Preview of Git HTTP (to be discussed next time).
- [Part 4: Git HTTP, 2019-03-14](https://www.youtube.com/watch?v=lM13p8lCu8A&list=PL05JrBw4t0KqoFUiX42JG7BAc7pipMBAy)
- Intercepting Git HTTP traffic with mitmproxy, overview of
- Git HTTP clone steps, code walk in gitlab-workhorse and gitlab-ce,
- investigating internal workhorse API messages used for Git HTTP.
+ Intercepting Git HTTP traffic with mitmproxy, overview of
+ Git HTTP clone steps, code walk in `gitlab-workhorse` and `gitlab-ce`,
+ investigating internal workhorse API messages used for Git HTTP.
- [Part 5: Merge Requests across Forks, 2019-03-21](https://www.youtube.com/watch?v=yGSuOz0XOHQ&list=PL05JrBw4t0KqoFUiX42JG7BAc7pipMBAy)
- Fixing a locally broken Ruby gem C
- extension by recompiling, demo of how creating a MR across forks
- causes new commits to suddenly appear in the fork parent repository,
- deep dive into the FetchSourceBranch RPC, adding debug code to see
- how address and authentication metadata is passed down to
- gitaly-ruby, failed attempt to log gitaly-ssh arguments, comparison
- of gitaly-ssh and gitlab-shell, a Gitaly server can end up making RPC calls to itself.
-
- - [Part 6: Creating Git commits on behalf of Git users, 2019-03-21](https://www.youtube.com/watch?v=Rbe0KGTLkxY&list=PL05JrBw4t0KqoFUiX42JG7BAc7pipMBAy)
-
- Demonstrate how usually Git hooks are run by
- `git-receive-pack`, but sometimes by `gitaly-ruby`. Deep dive into
- UserCommitFiles: where do those hooks actually get run? A look at
- UserMerge. How does Gitaly make merge commits. A look at the
- implementation of the special feature where users are not allowed
- push to a branch, but are allowed to merge into it.
-
- - [Part 7: How Gitaly uses Prometheus monitoring, 2019-07-09](https://youtu.be/R6F674Nj3wI)
-
- What is [Prometheus](https://prometheus.io/). Reconstructing a
- [Grafana](https://dashboards.gitlab.net) dashboard panel
- with
- [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/).
- Adding a new counter to Gitaly. Querying Prometheus in Gitaly
- during development. Comparing latency calculation with
- [ELK](https://log.gitlab.net). [GRPC Prometheus
- middleware](https://github.com/grpc-ecosystem/go-grpc-prometheus)
- in Gitaly.
-
-- [TheConf talk on Scaling GitLab git storage with Gitaly, 2019-08-16](https://speakerdeck.com/olsfer/how-gitlab-scaled-git-access-with-a-go-service)
+ Fixing a locally broken Ruby gem C
+ extension by recompiling, demo of how creating a MR across forks
+ causes new commits to suddenly appear in the fork parent repository,
+ deep dive into the FetchSourceBranch RPC, adding debug code to see
+ how address and authentication metadata is passed down to
+ `gitaly-ruby`, failed attempt to log `gitaly-ssh` arguments, comparison
+ of `gitaly-ssh` and `gitlab-shell`, a Gitaly server can end up making RPC calls to itself.
+
+ - [Part 6: Creating Git commits on behalf of Git users, 2019-03-21](https://www.youtube.com/watch?v=Rbe0KGTLkxY&list=PL05JrBw4t0KqoFUiX42JG7BAc7pipMBAy)
+
+ Demonstrate how usually Git hooks are run by
+ `git-receive-pack`, but sometimes by `gitaly-ruby`. Deep dive into
+ UserCommitFiles: where do those hooks actually get run? A look at
+ UserMerge. How does Gitaly make merge commits. A look at the
+ implementation of the special feature where users are not allowed
+ push to a branch, but are allowed to merge into it.
+
+ - [Part 7: How Gitaly uses Prometheus monitoring, 2019-07-09](https://youtu.be/R6F674Nj3wI)
+
+ What is [Prometheus](https://prometheus.io/). Reconstructing a
+ [Grafana](https://dashboards.gitlab.net) dashboard panel
+ with [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/).
+ Adding a new counter to Gitaly. Querying Prometheus in Gitaly
+ during development. Comparing latency calculation with
+ [ELK](https://log.gitlab.net). [GRPC Prometheus
+ middleware](https://github.com/grpc-ecosystem/go-grpc-prometheus)
+ in Gitaly.
+
+- [TheConf talk on Scaling GitLab Git storage with Gitaly, 2019-08-16](https://speakerdeck.com/olsfer/how-gitlab-scaled-git-access-with-a-go-service)
- [Infrastructure Team Update 2017-05-11](https://about.gitlab.com/2017/05/11/functional-group-updates/#infrastructure-team)
- [Gitaly Basics, 2017-05-01](https://docs.google.com/presentation/d/1cLslUbXVkniOaeJ-r3s5AYF0kQep8VeNfvs0XSGrpA0/edit#slide=id.g1c73db867d_0_0)
- [Git Paris meetup, 2017-02-22](https://docs.google.com/presentation/d/19OZUalFMIDM8WujXrrIyCuVb_oVeaUzpb-UdGThOvAo/edit?usp=sharing) a high-level overview of what our plans are and where we are.