| Age | Commit message (Collapse) | Author |
|---|
|
gitlab has been using the pointer layout since 15.3.
See https://gitlab.com/gitlab-org/gitlab/-/issues/355945
Changelog: changed
|
|
The concurrency limiter has two different mechanisms:
1. Callers first get put into a queue that is global across all
limiting keys. This queue is depth- and time-limited, which means
that callers will get rejected if it is full and will be evicted
when they took too long to acquire the concurrency token.
2. The concurrency limit itself that will only allow a set number of
callers at the same time. This functionality is per limiting key.
While the intent is usually to concurrency-limit either by user or by
repository, this is sabotaged by the fact that the queue is key-agnostic
and thus global. So even if almost all calls would apply to a single
repository only, if the queue is enabled and full then it would become
impossible to invoke the function for any other repository now. As a
result the queue is rather useless as a concurrency-limiting primitive.
But there is another big design flaw: as callers need to be in the queue
to obtain the concurrency-limiting token, and the number of callers in
the queue is strictly limited, we essentially have a global concurrency
limit to obtain the concurrency-limiting tokens. Suppose you have two
calls to a function that has a maximum queue depth of 1 and two calls to
this function at the same time. Even if the concurrency limit would now
theoretically allow for both functions to run at the same time, there is
a race window where both callers might try to enter the queue at the
same point in time. If this race is lost, then one of both callers will
be rejected due to the queue being full while the other one is trying to
obtain the concurrency token. This issue in fact surfaces in our tests,
where the `TestStreamLimitHandler` test is frequently failing because
the race is often lost.
This second design flaw cannot easily be fixed while the queue remains
global: we need to remain in the queue when trying to acquire the
concurrency token, or otherwise it wouldn't really be limiting anything
at all.
Convert the queue to be per-key so that each resource identified by a
key essentially has its own queue. This fixes both issues:
- If the queue is full then we only limit access to the specific
resource for which it is full. This makes the queueing mechanism
useful again.
- We can now change the queueing logic to allow as many callers into
the queue as the concurrency limit would allow _plus_ the allowed
depth of the queue itself. This fixes the race described aboved.
Changelog: fixed
|
|
Remove infrastructure to clone the "gitlab-test-mirror.git" and
"gitlab-git-test.git" seed repositories. They are not used anymore.
|
|
While we already have some advice in our beginners guide about how to
use our Makefile, it doesn't hint at the available `make help` target.
Mention it so that users can discover by themselves what other targets
exist.
|
|
After https://gitlab.com/groups/gitlab-org/-/epics/8005 yields some
early results, the way we use feature flag changes. Now Gitaly fully
supports feature flag actors. Although the set of supported actors are
the same, the way we intercept them is a bit different from GitLab Rails
This commit updates the feature flag documentation:
* Add overall architecture and explains how a flag is propagated around
* Add FF Development guideline
* Add new rollout strategies
Changelog: other
Co-authored-by: John Cai <jcai@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>
|
|
Document Gitaly Git distribution method
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4864
Merged-by: Christian Couder <chriscool@tuxfamily.org>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: Siddharth Asthana <siddharthasthana31@gmail.com>
|
|
When we roll out feature flags, we always want to roll out gradually
rather than all at once. Additionally, when turning on feature flags, we
should use certain Slack channels for maximum visibility. Explicitly
include these instructions in the PROCESS.md doc.
|
|
Changelog: other
|
|
The Ruby sidecar is slowly nearing its end: right now, it only
implements a single RPC that is in the process of being migrated to Go.
This means we'll start to soon remove all the Ruby infrastructure in the
Gitaly project.
Part of the infrastructure is the Ruby code generated from our Protobuf
definitions. And while we won't need that code ourselves anymore, it is
important for clients written in Ruby that want to interface with us. As
such, the Gitaly project still has to release the `gitaly` Gem. In the
past, we built the Gem by simply soaking up the code in `ruby/proto`.
But as it does not make any sense for us to keep the Ruby code around
anymore this strategy will fail once we have removed it.
Merge the `generate-proto-ruby` any `publish-gem` scripts into a single
script that:
1. Generates all required Ruby sources into a temporary directory.
2. Writes the `gitaly.gemspec` file into the same directory.
3. Builds the Gitaly Gem from those temporary files.
Like this, all code is generated ad-hoc and doesn't need to be committed
to the repository anymore.
Change the Makefile's `proto` target to not generate Ruby code anymore.
Instead, there are two new Makefile targets `build-proto-gem` and
`publish-proto-gem` that are responsible for building and publishing the
Gem, respecfively.
The generated Ruby code is thus stale now and will be removed in a
subsequent commit.
|
|
We've got multiple scripts that are required to generate Ruby code from
our Protobuf definitions in the `_support` directory. This has multiple
smells:
- It's out-of-line with all the other tools, which nowadays are
located in the `tools` directory.
- It's hard to discover and find out which parts logically form a
unit.
- We are reusing the Gemfile of the Ruby sidecar to pin the
`grpc-tools` dependency to a specific version.
Move the tooling into its own `tools/protogem` directory that's got its
own Gemfile to fix these points. This also allows us to auto-update
dependencies via the Renovate bot like we do for our other tools.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Now we're no longer loading language colors from a json file, it also
does not make sense to make the path of this json file configurable. So
remove it from the config.
|
|
In the past it wasn't always clear where exactly we put the boundary for
RPCs that should be implemented in Gitaly versus those that should be in
Rails. This was recently clarified though: Gitaly should always avoid to
implement business-specific logic and policy. Instead, its provided RPCs
should focus on the problem-domains specific to Gitaly and Praefect.
Clarify this in our documentation to make it more readily discoverable
by engineers.
|
|
Also performance. is replaced with dashboards.
Signed-off-by: Takuya Noguchi <takninnovationresearch@gmail.com>
Changelog: other
|
|
The dashboard URLs are pointing to the old gitlab.com URLs, we now use
gitlab.net for dashboards. Lets change the URLs in README.md and
observability.md.
Signed-off-by: Karthik Nayak <knayak@gitlab.com>
|
|
The `check` subcommand has been relocated from `gitaly-hooks` to the
main `gitaly` binary. References to the subcommand were updated to
reflect this change.
|
|
|
|
Makefile: Add target to debug go tests in Delve
See merge request gitlab-org/gitaly!4662
|
|
Signed-off-by: blanet <moweng.xx@alibaba-inc.com>
|
|
Add a section to the beginner's guide on how to use `dlv` to debug
tests via the Makefile.
|
|
Clarify the story of how `gitaly-hooks` is invoked through being
symlinked from `post-receive`, `pre-receive`, `update`,
`reference-transaction`.
Add section on `reference-transaction` hook.
Update some outdated code references.
|
|
This commit changes the major version in the package name from v14 to
v15
Updating go.mod & go.sum with new module name v15
Update Makefile to bump major version to v15
Update the gitaly package name in the Makefile. Also update
gitaly-git2go-v14 -> gitaly-git2go-v15. We need to keep
gitaly-git2go-v14 for a release however, for zero downtime upgrades.
This pulls directly from a sha that is v14.
Update package name from v14->v15 for auth, client, cmd, internal packages
This commit changes the package name from v14 to v15 in go and proto
files in the internal, auth, client, cmd packages.
proto: Update major package number in package name
tools: Change major version number in package name from v14 to v15
gitaly-git2go: Change the package name from v14 to v15
update module updater for v15
Update the documentation for the module updater to reflect v15
|
|
The 'Contributing' section of the Protobuf doc still references the old
`gitaly-proto` project and incorrectly suggests that the source files
generated by `protoc` may vary with Go version.
This removes the outdated and incorrect information and adds a new
section with details on the `make` targets for linting and validating
protos.
|
|
proto: Update documentation and modernize our style guide
See merge request gitlab-org/gitaly!4539
|
|
docs: Cgroups per git process
See merge request gitlab-org/gitaly!4461
|
|
Modernize our Protobuf documentation to clearly document our current
architecture and changed style guidelines.
|
|
In ancient times, Gitaly's Protobuf definitions had been living in a
separate project. This is still mentioned in some of our documentation,
even though it doesn't really play much of a role nowadays anymore.
Stop mentioning this old project: it's a thing of the past, and I'd be
surprised if anybody still needs information about it.
|
|
The Protobuf-related docs are currently hosted in the `proto/` directory
due to historic reasons when `gitaly-proto` still had its own separate
repository. Let's modernize this a bit:
- `README.md` is moved into `doc/protobuf.md`.
- `CONTRIBUTING.md` is removed. We already have such a file in the
root directory.
- `DEPRECATION.md` is merged into `doc/PROCESS.md`.
This should hopefully help discoverability of this documentation.
|
|
|
|
With the ability to do feature-flag-based rollouts of the Git version
our process to upgrade Git versions has become a bit more complex.
Document it so that it's easy to follow.
|
|
|
|
There are a number of knobs in Gitaly to tune backpressure Gitaly can impose on services that call it. This commit documents these.
|
|
|
|
This is a design doc that describes how cgroups work in Gitaly.
|
|
Document the different ways to access Git installations supported by
Gitaly. Most importantly, this also documents the way our new bundled
Git binaries work and why they were introduced.
|
|
git: Remove support for the Ruby hooks directory
Closes #4006
See merge request gitlab-org/gitaly!4356
|
|
gRPC HealthCheck messages are quite noisy, dominate the log volume in
Gitaly, and usually are not that useful. We now disable them by
default and add documentation on how to enable them.
Closes https://gitlab.com/gitlab-org/gitaly/-/issues/3428
Changelog: changed
|
|
The Ruby hooks directory does not exist anymore, and instead we're using
symlinks to the gitaly-hooks binary. Update our documentation to reflect
that change.
|
|
The docs mentions that hooks belongs to Gitlab Shell but it was changed:
https://gitlab.com/gitlab-org/gitaly/issues/1226
|
|
chore: Add ssh signature test case
See merge request gitlab-org/gitaly!4277
|
|
7b20a6045e8d2a25c86633461c03b13353915643 introduced support for SSH
signatures to the `catfile` package. The GetCommitSignatures RPC uses a
different implementation. This implementation is already compatible with
SSH sigantures, but is not tested with them. This change adds tests
which use SSH signatures.
Changelog: other
|
|
|
|
Update the hooks documentation to reflect the new hooks setup which uses
a temporary directory.
|
|
|
|
|
