Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitaly.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorEvan Read <eread@gitlab.com>2022-10-20 03:28:44 +0300
committerEvan Read <eread@gitlab.com>2022-10-20 03:29:35 +0300
commite67bd87f2f93e0927ddb71e78609206ccd888ac8 (patch)
tree51052ab910caf549df41b002060fcff71204d6e9
parentebcf168f6704fbfede1ce6512d3390eb285a6912 (diff)
More tidy up of violations of new Markdownlint configuration
-rw-r--r--doc/beginners_guide.md85
-rw-r--r--doc/cgroups.md10
-rw-r--r--doc/dashboards.md10
-rw-r--r--doc/delta_islands.md10
4 files changed, 55 insertions, 60 deletions
diff --git a/doc/beginners_guide.md b/doc/beginners_guide.md
index ab81e1c80..502df1052 100644
--- a/doc/beginners_guide.md
+++ b/doc/beginners_guide.md
@@ -1,15 +1,15 @@
-## Beginner's guide to Gitaly contributions
+# Beginner's guide to Gitaly contributions
-### Setup
+## Setup
-#### GitLab
+### GitLab
Before you can develop on Gitaly, it's required to have the
[GitLab Development Kit][gdk] properly installed. After installing GitLab, verify
it to be working by starting the required servers and visiting GitLab at
`http://localhost:3000`.
-#### Gitaly Proto
+### Gitaly Proto
Data is shared between GitLab Rails and Gitaly using the [Google Protocol Buffers](https://developers.google.com/protocol-buffers) to provide a shared format for serializing the exchanged data. They are also referred to as _protobuf_.
@@ -17,7 +17,7 @@ Protocol buffers define which requests can be made and what data the requester m
The protocol definitions can be found in `proto/*.proto`.
-#### Gitaly
+### Gitaly
Gitaly provides high-level RPC access to Git repositories. It controls access to the `git` binary and is used by GitLab to read and write Git data. Gitaly is present in every GitLab installation and coordinates Git repository storage and retrieval.
@@ -35,11 +35,11 @@ gitaly:
auto_update: false
```
-### Development
+## Development
-#### General advice
+### General advice
-##### Using the Makefile
+#### Using the Makefile
Gitaly uses [Make](https://en.wikipedia.org/wiki/Make_(software)) to manage its build process, and all targets are defined in
our top-level [Makefile](../Makefile). By default, simply running `make` will
@@ -64,7 +64,7 @@ will be used for generating code.
If you wish to persist your configuration, you may create a `config.mak` file
next to the Makefile and put all variables you wish to override in there.
-##### Experimenting with editing code
+#### Experimenting with editing code
If you're used to Ruby on Rails development you may be used to a "edit
code and reload" cycle where you keep editing and reloading until you
@@ -75,7 +75,7 @@ At some point you will know which Gitaly RPC you need to work on. This
is where you probably want to stop using `localhost:3000` and zoom in on
the RPC instead.
-###### A suggested workflow
+##### A suggested workflow
To experiment with changing an RPC you should use the Gitaly service
tests. The RPC you want to work on will have tests somewhere in
@@ -83,7 +83,7 @@ tests. The RPC you want to work on will have tests somewhere in
Before you edit any code, make sure the tests pass when you run them:
-```
+```shell
TEST_PACKAGES=./internal/gitaly/service/foobar TEST_OPTIONS="-count=1 -run=MyRPC" make test
```
@@ -92,7 +92,7 @@ In this command, `MyRPC` is a regex that will match functions like
Once you have found your tests and your test command, you can start tweaking the implementation or adding test cases and re-running the tests.
-This approach is many times faster than "edit gitaly, reinstall Gitaly into GDK,
+This approach is many times faster than "edit Gitaly, reinstall Gitaly into GDK,
restart, reload `localhost:3000`".
To see the changes, run the following commands in
@@ -103,41 +103,41 @@ make gitaly-setup
gdk restart gitaly
```
-#### Development Process
+### Development Process
The general approach is:
+
1. Add a request/response combination to [Gitaly Proto][gitaly-proto], or edit
an existing one
1. Change [Gitaly][gitaly] accordingly
1. Use the endpoint in other GitLab components (CE/EE, GitLab Workhorse, etc.)
-
-##### Configuration changes
+#### Configuration changes
When modifying the Gitaly or Praefect configuration, the changes should be propagated to other GitLab projects that rely on them:
-1. [gitlab/omnibus-gitlab](https://gitlab.com/gitlab-org/omnibus-gitlab) contains template files that are used to generate Gitaly's and Praefect's configuration.
-2. [gitlab/CNG](https://gitlab.com/gitlab-org/build/CNG) contains configuration required to run Gitaly in a container.
+1. [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) contains template files that are used to generate Gitaly's and Praefect's configuration.
+1. [CNG](https://gitlab.com/gitlab-org/build/CNG) contains configuration required to run Gitaly in a container.
-##### Gitaly Proto
+#### Gitaly Proto
The [Protocol buffer documentation][proto-docs] combined with the `*.proto` files in the `proto/` directory should be enough to get you started. A service needs to be picked that can receive the procedure call. A general rule of thumb is that the service is named either after the Git CLI command, or after the Git object type.
If either your request or response data can exceed 100KB you need to use the `stream` keyword. To generate the server and client code, run `make proto`.
-##### Gitaly
+#### Gitaly
If proto is updated, run `make`. This should compile successfully.
-##### Gitaly-ruby
+#### `gitaly-ruby`
Gitaly is mostly written in Go but it also uses a pool of Ruby helper
-processes. This helper application is called gitaly-ruby and its code
-is in the `ruby` subdirectory of Gitaly. Gitaly-ruby is a gRPC server,
+processes. This helper application is called `gitaly-ruby` and its code
+is in the `ruby` subdirectory of Gitaly. `gitaly-ruby` is a gRPC server,
just like its Go parent process. The Go parent proxies certain
-requests to gitaly-ruby.
+requests to `gitaly-ruby`.
-It is our experience that gitaly-ruby is unsuitable for RPC's that are slow, or that are called with a high frequency. It should only be used for:
+It is our experience that `gitaly-ruby` is unsuitable for RPC's that are slow, or that are called with a high frequency. It should only be used for:
- legacy GitLab application code that is too complex or subtle to rewrite in Go
- prototyping (if you the contributor are uncomfortable writing Go)
@@ -145,28 +145,29 @@ It is our experience that gitaly-ruby is unsuitable for RPC's that are slow, or
Note that for any changes to `gitaly-ruby` to be used by GDK, you need to
run `make gitaly-setup` in your GDK root and restart your processes.
-###### Gitaly-ruby boilerplate
+##### `gitaly-ruby` boilerplate
To create the Ruby endpoint, some Go is required as the go code receives the
requests and proxies it to the Go server. In general this is boilerplate code
where only method and variable names are different.
Examples:
+
- Simple: [Simple request in, simple response out](https://gitlab.com/gitlab-org/gitaly/blob/6841327adea214666417ee339ca37b58b20c649c/internal/service/wiki/delete_page.go)
- Client Streamed: [Stream in, simple response out](https://gitlab.com/gitlab-org/gitaly/blob/6841327adea214666417ee339ca37b58b20c649c/internal/service/wiki/write_page.go)
- Server Streamed: [Simple request in, streamed response out](https://gitlab.com/gitlab-org/gitaly/blob/6841327adea214666417ee339ca37b58b20c649c/internal/service/wiki/find_page.go)
- Bidirectional: No example at this time
-###### Ruby
+##### Ruby
The Ruby code needs to be added to `ruby/lib/gitaly_server/<service-name>_service.rb`.
The method name should match the name defined by the `gitaly` gem. To be sure
run `bundle open gitaly`. The return value of the method should be an
instance of the response object.
-There is no autoloader in gitaly-ruby. If you add new ruby files, you need to manually add a `require` statement in `ruby/lib/gitlab/git.rb` or `ruby/lib/gitaly_server.rb.`
+There is no autoloader in `gitaly-ruby`. If you add new Ruby files, you need to manually add a `require` statement in `ruby/lib/gitlab/git.rb` or `ruby/lib/gitaly_server.rb.`
-### Testing
+## Testing
Gitaly's tests are mostly written in Go but it is possible to write RSpec tests too.
@@ -188,12 +189,12 @@ docker run --name praefect-pg -p 5432:5432 -e POSTGRES_HOST_AUTH_METHOD=trust -d
To run the full test suite, use `make test`.
You'll need some [test repositories](test_repos.md), you can set these up with `make prepare-tests`.
-#### Go tests
+### Go tests
- each RPC must have end-to-end tests at the service level
- optionally, you can add unit tests for functions that need more coverage
-##### Integration Tests
+#### Integration Tests
A typical set of Go tests for an RPC consists of two or three test
functions:
@@ -206,7 +207,7 @@ Our Go RPC tests use in-process test servers that only implement the service the
For example, if you are working on an RPC in the 'RepositoryService', your tests would go in `internal/gitaly/service/repository/your_rpc_test.go`.
-##### Running a specific test
+#### Running a specific test
When you are trying to fix a specific test failure it is inefficient
to run `make test` all the time. To run just one test you need to know
@@ -216,7 +217,7 @@ test name (e.g. `TestRepositoryExists`).
To run the test you need a terminal window with working directory
`/path/to/gdk/gitaly`. To run just the one test you're interested in:
-```
+```shell
TEST_PACKAGES=./internal/gitaly/service/repository TEST_OPTIONS="-count=1 -run=TestRepositoryExists" make test-go
```
@@ -228,28 +229,28 @@ called on `testing.T`.
[require]: https://github.com/stretchr/testify/tree/master/require
[assert]: https://github.com/stretchr/testify/tree/master/assert
-##### Using Delve to debug a test
+#### Using Delve to debug a test
The process to debug a test in your terminal using
[Delve](https://github.com/go-delve/delve) is almost the same as
[running a single test](#running-a-specific-test), just change the
target to `debug-test-go`:
-```
+```shell
TEST_PACKAGES=./internal/gitaly/service/repository TEST_OPTIONS="-count=1 -run=TestRepositoryExists" make debug-test-go
```
-##### Useful snippets for creating a test
+#### Useful snippets for creating a test
-###### testhelper package
+##### testhelper package
-The `testhelper` package provides functions to create configurations to run gitaly and helpers to run a Gitaly gRPC server:
+The `testhelper` package provides functions to create configurations to run Gitaly and helpers to run a Gitaly gRPC server:
- [Create test configuration](https://gitlab.com/gitlab-org/gitaly/-/blob/aa098de7b267e3d6cb8a05e7862a1ad34f8f2ab5/internal/gitaly/service/ref/testhelper_test.go#L43)
- [Run Gitaly](https://gitlab.com/gitlab-org/gitaly/-/blob/aa098de7b267e3d6cb8a05e7862a1ad34f8f2ab5/internal/gitaly/service/ref/testhelper_test.go#L57)
- [Clone test repository](https://gitlab.com/gitlab-org/gitaly/-/blob/aa098de7b267e3d6cb8a05e7862a1ad34f8f2ab5/internal/gitaly/service/ref/find_all_tags_test.go#L30)
-#### RSpec tests
+### RSpec tests
It is possible to write end-to-end RSpec tests that run against a full
Gitaly server. This is more or less equivalent to the service-level
@@ -261,12 +262,12 @@ Gitaly every time you change the Go code. Run `make` to recompile.
Then, you can run RSpec tests in the `ruby` subdirectory.
-```
+```shell
cd ruby
bundle exec rspec
```
-### Rails tests
+## Rails tests
To use your custom Gitaly when running Rails tests in GDK, go to the
`gitlab` directory in your GDK and follow the instructions at
@@ -274,10 +275,6 @@ To use your custom Gitaly when running Rails tests in GDK, go to the
[custom-gitaly]: https://docs.gitlab.com/ee/development/gitaly.html#running-tests-with-a-locally-modified-version-of-gitaly
[gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/#getting-started
-[git-remote]: https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes
[gitaly]: https://gitlab.com/gitlab-org/gitaly
[gitaly-proto]: https://gitlab.com/gitlab-org/gitaly/tree/master/proto
-[gitaly-issue]: https://gitlab.com/gitlab-org/gitaly/issues
-[gitlab]: https://gitlab.com
-[go-workspace]: https://golang.org/doc/code.html#Workspaces
[proto-docs]: https://developers.google.com/protocol-buffers/docs/overview
diff --git a/doc/cgroups.md b/doc/cgroups.md
index 74603352a..a6c265565 100644
--- a/doc/cgroups.md
+++ b/doc/cgroups.md
@@ -1,8 +1,8 @@
-# Cgroups in Gitaly
+# Cgroups in Gitaly
## High Level
-Gitaly can be configured to run its git processes inside of cgroups, which prevent
+Gitaly can be configured to run its Git processes inside of cgroups, which prevent
shelled-out processes from hogging too much CPU and memory. See [these docs](https://man7.org/linux/man-pages/man7/cgroups.7.html) for a full specification on cgroups.
## Configuration
@@ -61,11 +61,11 @@ and [here](https://lwn.net/Kernel/Index/#OOM_killer).
This provides baseline protection from processes running haywire, such as a
memory leak. However, this only provides limited protection because there are
-some legitimate git processes that can take a huge amount of memory such as
+some legitimate Git processes that can take a huge amount of memory such as
`git-repack(1)`, or `git-upload-pack(1)`. For a large repository, these
operations can easily take 12gb of ram as seen in production systems.
-Hence, we also need finer grained controls to allow certain expensive git
+Hence, we also need finer grained controls to allow certain expensive Git
operations to have their own cgroups.
## CPU Limits
@@ -76,7 +76,7 @@ that will be a fraction of the total CPU resources a machine has access to.
## Cgroup Hierarchy
-```
+```plaintext
/sys/fs/cgroup
|
|--memory
diff --git a/doc/dashboards.md b/doc/dashboards.md
index fa4aa5d47..c6df05bbc 100644
--- a/doc/dashboards.md
+++ b/doc/dashboards.md
@@ -1,4 +1,4 @@
-# Observibility
+# Observability
## Logging
@@ -12,14 +12,14 @@ Messages can be queried leveraging our [Kibana instance](https://log.gitlab.net)
## Prometheus
Gitaly emits low cardinality metrics through Prometheus. Most of these are added
-by [go-grpc-prometheus](https://github.com/grpc-ecosystem/go-grpc-prometheus).
+by [`go-grpc-prometheus`](https://github.com/grpc-ecosystem/go-grpc-prometheus).
Many custom metrics were also added.
### Grafana
-To display the prometheus metrics, GitLab leverages Grafana. The instance is
+To display the Prometheus metrics, GitLab leverages Grafana. The instance is
available only for GitLab team members to view the dashboards. The dashboards can be found at:
-[dashboards.gitlab.net](https://dashboards.gitlab.net).
+<https://dashboards.gitlab.net>.
#### Editing Gitaly dashboards
@@ -51,5 +51,5 @@ Grafana-specific modifiers.
## Sentry
-Errors are tracked in our sentry instance, and due to their sensitive nature only viewable
+Errors are tracked in our Sentry instance, and due to their sensitive nature only viewable
by developers at GitLab at [the error tracking page](https://gitlab.com/gitlab-org/gitaly/error_tracking).
diff --git a/doc/delta_islands.md b/doc/delta_islands.md
index d085b3164..8a39729d0 100644
--- a/doc/delta_islands.md
+++ b/doc/delta_islands.md
@@ -1,4 +1,4 @@
-## Delta Islands
+# Delta Islands
Most of the time, an object in a Git repository is stored as a delta against
another object. This means that if a blob is stored once, and only one line is
@@ -13,7 +13,7 @@ blobs now form a delta chain. Git stores these delta chains in [pack files][git-
and when a `repack` is executed these chains might be recalculated if one of the
blobs isn't required anymore, or if a better delta base is discovered.
-In the case of a git fetch, it might happen that the client doesn't have a set
+In the case of a Git fetch, it might happen that the client doesn't have a set
of branches that, during repack, were detected to share many of the same contents
and thus form a delta chain. Git can then decide to send the full delta chain.
@@ -33,7 +33,7 @@ full objects, which improves the load on the server and latency for the fetch.
The drawback of this feature is that the packs on disk are potentially
larger as it's not always the case the optimal object can be used as delta base.
-### In GitLab
+## In GitLab
Delta Island relies on Git version 2.20 or later, which GitLab is expected to
use from version 11.11 onwards. The change on the Gitaly side is limited to
@@ -43,13 +43,11 @@ runtime to prevent having to write the configuration file for all repositories.
User impact of this feature includes faster fetches, as Git on the server does
less work and reuses previous work better.
-#### Further reading
+## Further reading
As is usually the case, the [tests of Git][git-delta-test] provide a good overview
of how the feature works.
[git-delta-test]: https://github.com/git/git/blob/041f5ea1cf987a4068ef5f39ba0a09be85952064/t/t5320-delta-islands.sh
[git-pack]: https://git-scm.com/docs/git-pack-objects
-[delta-mr]: https://gitlab.com/gitlab-org/gitaly/merge_requests/1110
[delta-config]: https://gitlab.com/gitlab-org/gitaly/merge_requests/1110/diffs#e01aecd9d7ee43aee1959795092f852d07a1e7ed_55_78
-