| Age | Commit message (Collapse) | Author |
|---|
|
As GetInfoAttributes are being deprecated, we will be using
GetFileAttributes RCP call to replace it.
|
|
commit: Allow path scoped revisions for `CheckObjectsExist` RPC
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6539
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Emily Chui <echui@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
Currently the `CheckObjectsExist` RPC fails validation when the request
provides path scoped revisions. Pass the `git.AllowPathScopedRevision()`
option to `git.ValidateRevision()` to enable support.
|
|
gitaly-backup: Migrate command to `urfave/cli` framework
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6514
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Gerardo <ggutierrez@gitlab.com>
|
|
|
|
'qmnguyen0711/cgroup-allow-spawning-command-use-up-to-m-cgroups' into 'master'
cgroup: Allow a repository to use up to M repository cgroups instead of one
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6537
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Christian Couder <chriscool@tuxfamily.org>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
By default, a repository can spawn processes in at most one cgroups. If
the number of repositories is more than the number of cgroups (likely),
multiple repositories share the same one. This model works very well if
the repositories under the management of Cgroup are equivalent in size
or traffic. If a node has some enormous repositories (mono-repo, for
example), the scoping cgroups become excessively large compared to the
rest. This imbalance situation might force the operators to lift the
repository-level cgroups. As a result, the isolation effect is not
effective.
This commit allows a repository to use up to M repository cgroups
instead of one. This feature is designed to balance resource usage
between cgroups, mitigate competition for resources within a single
cgroup, and enhance memory usage efficiency and isolation. The value can
be adjusted based on the specific workload and number of repository
cgroups on the node. A Git process uses its target repository's relative
path as the hash key to find the corresponding cgroup. It is allocated
randomly to any of the consequent AllocationCount cgroups. It wraps
around if needed.
|
|
gitaly: Don't cleanup cgroups on exit
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6532
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: Will Chandler <wchandler@gitlab.com>
|
|
repository: Add resolved address to `CreateRepositoryFromURL` error
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6542
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: James Fargher <jfargher@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
The `CreateRepositoryFromURL` RPC may have a resolved address set in its
request to mitigate DNS rebinding. In the event the underlying
`git-clone(1)` process returns an error, append the resolved address as
structured error metadata for debugging.
|
|
Add links to Gitaly Training youtube channel
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6538
Merged-by: Andras Horvath <ahorvath@gitlab.com>
Approved-by: Andras Horvath <ahorvath@gitlab.com>
Co-authored-by: John Cai <jcai@gitlab.com>
|
|
proto: Add documentation for `RemoteService`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6536
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Evan Read <eread@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
|
|
We have several background tasks spawned as part Gitaly's startup
process. Currently we start many of these towards the beginning of
startup, but on systems with limited available parallelism these may
slow the progression of the main goroutine.
Move these to the end of startup so that critical tasks can complete as
quickly as possible.
On a 2-core host this improves startup time by roughly 15%:
Benchmark 1: ./async-prune serve config.toml
Time (mean ± σ): 535.3 ms ± 106.7 ms [User: 462.0 ms, System: 228.5 ms]
Range (min … max): 436.9 ms … 727.9 ms 10 runs
Benchmark 2: ./tasks-end serve config.toml
Time (mean ± σ): 468.0 ms ± 23.5 ms [User: 438.6 ms, System: 220.6 ms]
Range (min … max): 430.8 ms … 517.3 ms 10 runs
Summary
./tasks-end serve config.toml ran
1.14 ± 0.24 times faster than ./async-prune serve config.toml
Changelog: performance
|
|
Currently we block on startup while pruning cgroups from old Gitaly
processes. However, stale cgroups have no impact on Gitaly's other
startup tasks as we namespace them by pid. We can safely make this a
background task so that critical startup tasks can move ahead unimpeded.
On a host with 1000 repo cgroups and teardown is ~1ms per cgroup, this
improves startup time by roughly 15%. On our hosts where cgroup teardown
is closer to 20ms this will have a much larger impact.
Benchmark 1: ./gitaly-5b092369 serve config.toml
Time (mean ± σ): 632.7 ms ± 150.7 ms [User: 473.7 ms, System: 226.7 ms]
Range (min … max): 461.9 ms … 868.1 ms 10 runs
Benchmark 2: ./async-prune serve config.toml
Time (mean ± σ): 549.3 ms ± 127.9 ms [User: 464.6 ms, System: 223.4 ms]
Range (min … max): 427.1 ms … 754.6 ms 10 runs
Summary
./async-prune serve config.toml ran
1.15 ± 0.38 times faster than ./gitaly-5b092369 serve config.toml
|
|
To mitigate the impact of removing `tableflip` we need both startup and
shutdown of Gitaly processes to be as fast as possible. With 105f6dd816
(cgroups: Create repository cgroups on-demand, 2023-10-26) we improved
startup times by creating cgroups individually when needed, rather than
as a blocking task during startup.
We currently block shutdown while removing our existing
cgroups. This is as slow as cgroup creation, taking up to 20 seconds to
remove 1000 cgroups on hosts with large number cgroups.
`Cleanup` is equivalent to an eager `PruneOldCgroups` scoped to the
current processes's cgroups. However, there is no urgent need to remove
cgroups immediately, so long as we ensure they don't build up
excessively. We can rely on the eventual cleanup from `PruneOldCgroups`
run by the next Gitaly process, which allows us to avoid delaying
shutdown.
Stop using `Cleanup` and remove it from the `cgroup.Manager` interface.
|
|
ssh & smarthttp: Advertise server-side backups as bundle-URI
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6472
Merged-by: James Fargher <jfargher@gitlab.com>
Approved-by: James Fargher <jfargher@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: James Fargher <jfargher@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
|
|
Record reference transaction ordering in Transaction
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6524
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
|
|
Remove NamespaceService
Closes #3803
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6292
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
The Gitaly Training and resources playlist will house all of our Gitaly
videos. Replace the current bullet point with a link to the playlist.
|
|
Now that we have more Gitaly training videos, we've conslidated these
all into one youtube unfiltered playlist. Let's replace the existing
block of text with a link to the playlist so that we don't have to keep
updating this README when we do more training videos.
|
|
Some of the RPC definitions and protobuf message types in the
`RemoteService` lack documentation. Add the missing documentation.
|
|
Allow snapshotting multiple repositories
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6522
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
limiter: Ignore `total_inactive_file` instead of `inactive_file`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6529
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
Link security tracking issue in the security issue template
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6535
Merged-by: John Cai <jcai@gitlab.com>
Co-authored-by: Steve Abrams <sabrams@gitlab.com>
|
|
In previous commits, we've added server-side backups for bundle-URI in
the smarthttp server. Now we can do the same in the ssh server and
advertise server-side bundles to be used for bundle-URI. Inject all the
git configuration into the git-upload-pack(1) command to advertise these
when possible.
Label: feature::addition
|
|
Now all is set up to make it possible to advertise server-side bundles
to be used for bundle-URI. Inject all the git configuration into the
git-upload-pack(1) command to advertise these when possible.
Label: feature::addition
|
|
Expose the SignedURL function of the bucket through the Sink interface.
Some blob backends will support this.
It's hard to test signed URLs with actual servers, so for now we can
only test some backends return a "not implemented" error.
|
|
To use server-side backups for bundle-URI we'll need the backup.Locator
& backup.Sink in the smarthttp server. Extract these from the
dependencies.
|
|
The first step toward using bundle-URI is making the server advertise it
supports this capability.
Insert uploadpack.advertiseBundleURIs in the git config so the info/refs
response contains the bundle-uri capability.
With this we also introduce the feature flag that allows us to toggle
the use of bundle-URIs.
|
|
We're about to make some changes to this function and callers should not
be aware of it's implementation details. That's why we're making this
function non-public and make ResolveSink() the only entry point to
initialize a backup.Sink.
|
|
requestinfohandler: Restore missing log fields in SSHReceivePack
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6534
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Stan Hu <stanhu@gmail.com>
|
|
proto: Add documentation for `RefService`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6527
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
|
|
In https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6508, we
implemented a fix to ignore highly evictable Page Caches from cgroup
memory threshold. The used metric was parent cgroup's `inactive_file`.
Unfortunately, that metric reflects the inactive Page Caches of direct
processes inside the parent cgroup. It doesn't account for the indirect
processes in children repository cgroups. We don't spawn any process in
the parent cgroup. So, the fix was useless.
In Cgroup V1, `memory.stat` has `total_inactive_file`. It's exactly what
we are looking for. In Cgroup V2, `inactive_file` includes all of its
substree consumptions. So, we can keep using that field.
|
|
cgroups: Unify v1 and v2 tests
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6519
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Will Chandler <wchandler@gitlab.com>
|
|
a497cfc8 replaced the v1 grpc-go-middleware `Tags` with the v2
`logging.Fields`, but this caused fields such as `correlation_id` and
`user_id` to be dropped for calls such as `SSHReceivePack`. This
occurred because the logger still uses the v1
`StreamServerInterceptor` and `UnaryServerInterceptor`, which extracts
logging fields from a different context key than the one used by v2
interceptors.
To restore these logging fields until the v2 middlewares are used,
restore the v1 `Tags`. This means we have a bit of redundancy since
the fields are stored for both v1 and v2 formats. However, once
https://gitlab.com/gitlab-org/gitaly/-/work_items/5661 is complete,
the v1 middleware and `Tags` can be dropped.
Changelog: fixed
|
|
Some of the RPC definitions and protobuf message types in the
`RefService` lack documentation. Add the missing documentation.
|
|
praefect: Reconcile non-matching object pools during replication
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6526
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
Makefile: Update test-wal to call test-go
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6530
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: Will Chandler <wchandler@gitlab.com>
|
|
Currently the `test-wal` Makefile target will call the `test` target
internally. This will run both Gitaly's Go tests and our linter tests.
If overriding `TEST_PACKAGES` this may result in a spurious failure on
the linter tests:
make test-wal TEST_PACKAGES=./internal/gitaly/storage/storagemgr
✓ internal/gitaly/storage/storagemgr (39.673s)
DONE 237 tests in 42.262s
stat /Users/wchandler/devel/gdk/gitaly/tools/golangci-lint/gitaly/internal/gitaly/storage/storagemgr: directory not found
=== Errors
stat /Users/wchandler/devel/gdk/gitaly/tools/golangci-lint/gitaly/internal/gitaly/storage/storagemgr: directory not found
DONE 0 tests, 1 error in 0.008s
make: *** [test-gitaly-linters] Error 1
Switch `test-wal` to use the `test-go` target which will skip the
linter.
|
|
featureflag: Remove `TransactionalLinkRepository`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6528
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
'5559-task5653-update-module-github-com-grpc-ecosystem-go-grpc-middleware-to-v2' into 'master'
package: Upgrade to go-grpc-middleware/v2 (requestinfohandler, sentryhandler)
Closes #5653
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6495
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Eric Ju <eju@gitlab.com>
Co-authored-by: Eric Ju <eju@gitlab.com>
|
|
repository: Remove RenameRepository RPC
Closes #4277
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6506
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
|
[ci skip]
|
|
The `TransactionalLinkRepository` feature flag, when enabled, makes the
`LinkRepositoryToObjectPool` RPC transactional. Remove the feature flag
so that the RPC is always transactional for clients.
|
|
The `CreateBranch` and `DeleteBranch` RPCs no longer exist but still
have their request and response protobuf messages defined. Remove these
definitions as they are no longer needed.
|
|
The `CreateBranch` RPC has been removed and there is no longer a need
for its associated request and response messages. Update call sites to
now use `UserCreateBranch` messages.
|
|
When a Praefect replica becomes out-of-date with the primary repository,
a replication job is scheduled to sync repository state. If the primary
and secondary replica are linked to non-matching object pools, the
replication job will always fail.
In this scenario, to reconcile the different object pools the target
repository should first disconnect from its object pool and then be
linked to the matching source repository object pool. Update Praefect
replication jobs to support synchronizing repositories with non-matching
object pools.
|
|
[ci skip]
|
|
This commit upgrades sentryhandler to use "github.com/grpc-ecosystem/go-grpc-middleware/v2"
|
