| Age | Commit message (Collapse) | Author |
|---|
|
|
|
Fix typos found by typos-cli(https://github.com/crate-ci/typos). Some
affected tests are adjusted.
There are a bunch of other typos are ignored, including
* CHANGELOG.md
* NOTICE
* internal/.../migrations/20201208163237_cleanup_notifications_payload.go
* other intended typos or false positives
Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
|
|
Enhance developing workflow with Ruby gem in Gitlab Rails
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6593
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Approved-by: Eric Ju <eju@gitlab.com>
Reviewed-by: Toon Claes <toon@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Eric Ju <eju@gitlab.com>
|
|
|
|
Sometimes, we need to work on both Gitaly and GitLab Rails
implementations in parallel. Ideally, we can define and publish all
Protobuf changes beforehand. In practice, a complicated change might
require modifying the Protobuf multiple times until reaching a stable
state. It's more convenient to let GitLab Rails point the gem to the the
developing branch in Gitaly.
Unfortunately, Gitaly source code doesn't include the gemspec or any
autogenerated Ruby codes. All of them are generated on the flight while
running `make build-proto-gem`. Thus, we cannot point the gem to the
developing branch directly on CI environment.
This commit tweaks the gem generation script:
- Use current HEAD hash as the pre-release version.
- Persist gem working directory after building. This allows local GitLab
Rails point to that directory directly.
- Add support for gem name customization. This allows the developer
publish the gem under a different name during development.
|
|
|
|
|
|
|
|
|
|
We don't use either Git2go nor libgit2 anymore, so let's stop mentioning
it.
|
|
Remove the now-unused gitaly-git2o command as well as its supporting
infrastructure.
|
|
Now that restore is supported by gitaly-backup we can add a more
complete section on server-side backups.
|
|
Add server-side flag to gitaly-backup create
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6026
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Co-authored-by: James Fargher <jfargher@gitlab.com>
|
|
|
|
|
|
This option triggers gitaly-backup to use the new BackupRepository RPC
to create backups. These backups are streamed from the gitaly node
directly to the object-storage configured in config.backup.go_cloud_url.
Tests added for the command are smoke tests as the backup logic itself
is tested on backup.Manager.
Changelog: added
|
|
Restore to given backup ID
Closes #5356
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5930
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Evan Read <eread@gitlab.com>
Co-authored-by: James Fargher <jfargher@gitlab.com>
|
|
Allow the backup-id to be specified by the user. The manager backup ID
used to be arbitrarily set to a timestamp. This matches what the create
command does. It never used to matter because the backup ID was ignored
on restore.
Changelog: changed
|
|
|
|
Create help text style guide for project
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5899
Merged-by: Andras Horvath <ahorvath@gitlab.com>
Approved-by: Russell Dickenson <rdickenson@gitlab.com>
Approved-by: Andras Horvath <ahorvath@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>
|
|
|
|
|
|
|
|
The feature flag logic is currently hosted inside of
`internal/metadata/featureflag`. And while they do have something to do
with metadata, they really are a standalone concept.
Move the package to `internal/featureflag` to reflect this.
|
|
|
|
Add a link to our gitlab pages site that has generated HTML docs for
each of our RPCs.
|
|
|
|
Remove fine-grained repository housekeeping RPCs
Closes #4039
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5654
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
|
|
The RepackFull RPC has been deprecated in favor of OptimizeRepository.
Remove its Protobuf declaration and implementation.
Changelog: removed
|
|
|
|
Trace2 was integrated into Gitaly. It runs in production and offers
great benefits for observability. This commit adds a technical
documentation about this tool, especially how to implement a custom
trace2 hook.
|
|
|
|
doc: Add a doc about using Jaeger for local development
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5628
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Evan Read <eread@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Evan Read <eread@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
|
|
Distributed tracing and Jaeger are powerful tool. They are integrated
deep into Gitaly and other services at GitLab. Unfortunately, the
adoption at GitLab is not high. This commit adds a development doc to
get started with using Jaeger for Gitaly local development.
|
|
docs: Remove rspec from the beginner's guide
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5617
Merged-by: James Fargher <proglottis@gmail.com>
Approved-by: Evan Read <eread@gitlab.com>
Approved-by: James Fargher <proglottis@gmail.com>
Reviewed-by: Evan Read <eread@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
|
|
|
|
When using GDK it's even easier to set up a Postgres database, so
document this.
|
|
We no longer have rspec tests (in use), so remove documentation about
those. While at it, restructure the text a bit.
|
|
|
|
There is no sidecar no more, so no need to have docs about it.
|
|
|
|
We've gone back and forth about whether or not to remove the major
version in Gitaly. We decided that we will not. Here's a brief history
and background as to why.
|
|
What
---
- Add a new configuration under `cgroups` called `cpu_quota_us` to
configure `cfs_quota_us` for the parent cgroup
https://docs.kernel.org/scheduler/sched-bwc.html?highlight=cfs_quota_us
- Add a new configuration under `cgroups.repositories` called
`cpu_quota_us` to configure `cfs_quota_us` for the repository cgroup
https://docs.kernel.org/scheduler/sched-bwc.html?highlight=cfs_quota_us
- Add metrics
- `gitaly_cgroup_cpu_cfs_periods_total`: Read from `cpu.stat` nr_periods https://docs.kernel.org/scheduler/sched-bwc.html#statistics
- `gitaly_cgroup_cpu_cfs_throttled_periods_total`: Read from `cpu.stat` nr_throttled https://docs.kernel.org/scheduler/sched-bwc.html#statistics
- `gitaly_cgroup_cpu_cfs_throttled_seconds_total`: Read from `cpu.stat` throttled_time https://docs.kernel.org/scheduler/sched-bwc.html#statistics
- Add more test coverage when only specific values are set.
Why
---
At the moment we limit memory and CPU via
[`cpu.shares`](https://kernel.googlesource.com/pub/scm/linux/kernel/git/glommer/memcg/+/cpu_stat/Documentation/cgroups/cpu.txt)
which will only throttle a cgroup when there is contention on the CPU.
This means that potentially a single repository can still hog all of the
CPU on a gitaly node. We've seen a case of this in
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8318, a
single repository saturated the CPU, and the scheduler couldn't balance
the CPU for other tasks/requests to be scheduled.
We hoped CPU shares would be enough, but we need an upper CPU quota for
gitaly cgroups so no single repository can fully saturate the CPU.
There are a few concerns that are addressed
Concern 1: cfs_period_us
`cfs_period_us` is used to calculate the `cfs_quota_us` (what we are
setting now), the default value seems to be
[hardcoded](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/kernel/sched/fair.c?h=v5.15.92#n5492)
but the Linux kernel but this can be updated, so Gitaly is explicitly
settings this to 100ms (default value)
Concern 2: not using `cfs_burst_us`
This could allow for CPU bursts, even when they exceed the
`cfs_quota_us`, we don't set this because it's available on the newer kernel
versions (5.15). The way users can avoid throttling is by
oversubscribing `cfs_quota_us`
Concern 3: Wasting available resources
When the user sets these we'll be artificially limiting the CPU that they
consume, this can leave performance on the table when a repository is
using all its quota, and no other process is using the CPU. This is the
only drawback and one we are willing to take since it adds more
reliability in the long run. We can reduce the effect of this by oversubscribing.
Concern 4: Observability
The kernel already exports
[stats](https://docs.kernel.org/scheduler/sched-bwc.html#statistics)
which Gitaly exposes as, and also
[cadvisor](https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md#prometheus-container-metrics)
Reference: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/17332
Changelog: added
Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>
|
|
|
|
gitlab has been using the pointer layout since 15.3.
See https://gitlab.com/gitlab-org/gitlab/-/issues/355945
Changelog: changed
|
|
The concurrency limiter has two different mechanisms:
1. Callers first get put into a queue that is global across all
limiting keys. This queue is depth- and time-limited, which means
that callers will get rejected if it is full and will be evicted
when they took too long to acquire the concurrency token.
2. The concurrency limit itself that will only allow a set number of
callers at the same time. This functionality is per limiting key.
While the intent is usually to concurrency-limit either by user or by
repository, this is sabotaged by the fact that the queue is key-agnostic
and thus global. So even if almost all calls would apply to a single
repository only, if the queue is enabled and full then it would become
impossible to invoke the function for any other repository now. As a
result the queue is rather useless as a concurrency-limiting primitive.
But there is another big design flaw: as callers need to be in the queue
to obtain the concurrency-limiting token, and the number of callers in
the queue is strictly limited, we essentially have a global concurrency
limit to obtain the concurrency-limiting tokens. Suppose you have two
calls to a function that has a maximum queue depth of 1 and two calls to
this function at the same time. Even if the concurrency limit would now
theoretically allow for both functions to run at the same time, there is
a race window where both callers might try to enter the queue at the
same point in time. If this race is lost, then one of both callers will
be rejected due to the queue being full while the other one is trying to
obtain the concurrency token. This issue in fact surfaces in our tests,
where the `TestStreamLimitHandler` test is frequently failing because
the race is often lost.
This second design flaw cannot easily be fixed while the queue remains
global: we need to remain in the queue when trying to acquire the
concurrency token, or otherwise it wouldn't really be limiting anything
at all.
Convert the queue to be per-key so that each resource identified by a
key essentially has its own queue. This fixes both issues:
- If the queue is full then we only limit access to the specific
resource for which it is full. This makes the queueing mechanism
useful again.
- We can now change the queueing logic to allow as many callers into
the queue as the concurrency limit would allow _plus_ the allowed
depth of the queue itself. This fixes the race described aboved.
Changelog: fixed
|
|
Remove infrastructure to clone the "gitlab-test-mirror.git" and
"gitlab-git-test.git" seed repositories. They are not used anymore.
|
|
While we already have some advice in our beginners guide about how to
use our Makefile, it doesn't hint at the available `make help` target.
Mention it so that users can discover by themselves what other targets
exist.
|
|
After https://gitlab.com/groups/gitlab-org/-/epics/8005 yields some
early results, the way we use feature flag changes. Now Gitaly fully
supports feature flag actors. Although the set of supported actors are
the same, the way we intercept them is a bit different from GitLab Rails
This commit updates the feature flag documentation:
* Add overall architecture and explains how a flag is propagated around
* Add FF Development guideline
* Add new rollout strategies
Changelog: other
Co-authored-by: John Cai <jcai@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>
|
|
Document Gitaly Git distribution method
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4864
Merged-by: Christian Couder <chriscool@tuxfamily.org>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: Siddharth Asthana <siddharthasthana31@gmail.com>
|
