| Age | Commit message (Collapse) | Author |
|---|
|
1. add reporter, option, decider
2. change interceptor interface
3. fixing tests.
|
|
gitaly: Don't cleanup cgroups on exit
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6532
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: Will Chandler <wchandler@gitlab.com>
|
|
repository: Add resolved address to `CreateRepositoryFromURL` error
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6542
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: James Fargher <jfargher@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
The `CreateRepositoryFromURL` RPC may have a resolved address set in its
request to mitigate DNS rebinding. In the event the underlying
`git-clone(1)` process returns an error, append the resolved address as
structured error metadata for debugging.
|
|
Add links to Gitaly Training youtube channel
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6538
Merged-by: Andras Horvath <ahorvath@gitlab.com>
Approved-by: Andras Horvath <ahorvath@gitlab.com>
Co-authored-by: John Cai <jcai@gitlab.com>
|
|
proto: Add documentation for `RemoteService`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6536
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Evan Read <eread@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
|
|
We have several background tasks spawned as part Gitaly's startup
process. Currently we start many of these towards the beginning of
startup, but on systems with limited available parallelism these may
slow the progression of the main goroutine.
Move these to the end of startup so that critical tasks can complete as
quickly as possible.
On a 2-core host this improves startup time by roughly 15%:
Benchmark 1: ./async-prune serve config.toml
Time (mean ± σ): 535.3 ms ± 106.7 ms [User: 462.0 ms, System: 228.5 ms]
Range (min … max): 436.9 ms … 727.9 ms 10 runs
Benchmark 2: ./tasks-end serve config.toml
Time (mean ± σ): 468.0 ms ± 23.5 ms [User: 438.6 ms, System: 220.6 ms]
Range (min … max): 430.8 ms … 517.3 ms 10 runs
Summary
./tasks-end serve config.toml ran
1.14 ± 0.24 times faster than ./async-prune serve config.toml
Changelog: performance
|
|
Currently we block on startup while pruning cgroups from old Gitaly
processes. However, stale cgroups have no impact on Gitaly's other
startup tasks as we namespace them by pid. We can safely make this a
background task so that critical startup tasks can move ahead unimpeded.
On a host with 1000 repo cgroups and teardown is ~1ms per cgroup, this
improves startup time by roughly 15%. On our hosts where cgroup teardown
is closer to 20ms this will have a much larger impact.
Benchmark 1: ./gitaly-5b092369 serve config.toml
Time (mean ± σ): 632.7 ms ± 150.7 ms [User: 473.7 ms, System: 226.7 ms]
Range (min … max): 461.9 ms … 868.1 ms 10 runs
Benchmark 2: ./async-prune serve config.toml
Time (mean ± σ): 549.3 ms ± 127.9 ms [User: 464.6 ms, System: 223.4 ms]
Range (min … max): 427.1 ms … 754.6 ms 10 runs
Summary
./async-prune serve config.toml ran
1.15 ± 0.38 times faster than ./gitaly-5b092369 serve config.toml
|
|
To mitigate the impact of removing `tableflip` we need both startup and
shutdown of Gitaly processes to be as fast as possible. With 105f6dd816
(cgroups: Create repository cgroups on-demand, 2023-10-26) we improved
startup times by creating cgroups individually when needed, rather than
as a blocking task during startup.
We currently block shutdown while removing our existing
cgroups. This is as slow as cgroup creation, taking up to 20 seconds to
remove 1000 cgroups on hosts with large number cgroups.
`Cleanup` is equivalent to an eager `PruneOldCgroups` scoped to the
current processes's cgroups. However, there is no urgent need to remove
cgroups immediately, so long as we ensure they don't build up
excessively. We can rely on the eventual cleanup from `PruneOldCgroups`
run by the next Gitaly process, which allows us to avoid delaying
shutdown.
Stop using `Cleanup` and remove it from the `cgroup.Manager` interface.
|
|
ssh & smarthttp: Advertise server-side backups as bundle-URI
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6472
Merged-by: James Fargher <jfargher@gitlab.com>
Approved-by: James Fargher <jfargher@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: James Fargher <jfargher@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
|
|
Record reference transaction ordering in Transaction
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6524
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
|
|
Remove NamespaceService
Closes #3803
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6292
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
The Gitaly Training and resources playlist will house all of our Gitaly
videos. Replace the current bullet point with a link to the playlist.
|
|
Now that we have more Gitaly training videos, we've conslidated these
all into one youtube unfiltered playlist. Let's replace the existing
block of text with a link to the playlist so that we don't have to keep
updating this README when we do more training videos.
|
|
Some of the RPC definitions and protobuf message types in the
`RemoteService` lack documentation. Add the missing documentation.
|
|
Allow snapshotting multiple repositories
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6522
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
limiter: Ignore `total_inactive_file` instead of `inactive_file`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6529
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
Link security tracking issue in the security issue template
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6535
Merged-by: John Cai <jcai@gitlab.com>
Co-authored-by: Steve Abrams <sabrams@gitlab.com>
|
|
In previous commits, we've added server-side backups for bundle-URI in
the smarthttp server. Now we can do the same in the ssh server and
advertise server-side bundles to be used for bundle-URI. Inject all the
git configuration into the git-upload-pack(1) command to advertise these
when possible.
Label: feature::addition
|
|
Now all is set up to make it possible to advertise server-side bundles
to be used for bundle-URI. Inject all the git configuration into the
git-upload-pack(1) command to advertise these when possible.
Label: feature::addition
|
|
Expose the SignedURL function of the bucket through the Sink interface.
Some blob backends will support this.
It's hard to test signed URLs with actual servers, so for now we can
only test some backends return a "not implemented" error.
|
|
To use server-side backups for bundle-URI we'll need the backup.Locator
& backup.Sink in the smarthttp server. Extract these from the
dependencies.
|
|
The first step toward using bundle-URI is making the server advertise it
supports this capability.
Insert uploadpack.advertiseBundleURIs in the git config so the info/refs
response contains the bundle-uri capability.
With this we also introduce the feature flag that allows us to toggle
the use of bundle-URIs.
|
|
We're about to make some changes to this function and callers should not
be aware of it's implementation details. That's why we're making this
function non-public and make ResolveSink() the only entry point to
initialize a backup.Sink.
|
|
requestinfohandler: Restore missing log fields in SSHReceivePack
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6534
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Stan Hu <stanhu@gmail.com>
|
|
proto: Add documentation for `RefService`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6527
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
|
|
In https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6508, we
implemented a fix to ignore highly evictable Page Caches from cgroup
memory threshold. The used metric was parent cgroup's `inactive_file`.
Unfortunately, that metric reflects the inactive Page Caches of direct
processes inside the parent cgroup. It doesn't account for the indirect
processes in children repository cgroups. We don't spawn any process in
the parent cgroup. So, the fix was useless.
In Cgroup V1, `memory.stat` has `total_inactive_file`. It's exactly what
we are looking for. In Cgroup V2, `inactive_file` includes all of its
substree consumptions. So, we can keep using that field.
|
|
cgroups: Unify v1 and v2 tests
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6519
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Will Chandler <wchandler@gitlab.com>
|
|
a497cfc8 replaced the v1 grpc-go-middleware `Tags` with the v2
`logging.Fields`, but this caused fields such as `correlation_id` and
`user_id` to be dropped for calls such as `SSHReceivePack`. This
occurred because the logger still uses the v1
`StreamServerInterceptor` and `UnaryServerInterceptor`, which extracts
logging fields from a different context key than the one used by v2
interceptors.
To restore these logging fields until the v2 middlewares are used,
restore the v1 `Tags`. This means we have a bit of redundancy since
the fields are stored for both v1 and v2 formats. However, once
https://gitlab.com/gitlab-org/gitaly/-/work_items/5661 is complete,
the v1 middleware and `Tags` can be dropped.
Changelog: fixed
|
|
Some of the RPC definitions and protobuf message types in the
`RefService` lack documentation. Add the missing documentation.
|
|
praefect: Reconcile non-matching object pools during replication
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6526
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
Makefile: Update test-wal to call test-go
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6530
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: Will Chandler <wchandler@gitlab.com>
|
|
Currently the `test-wal` Makefile target will call the `test` target
internally. This will run both Gitaly's Go tests and our linter tests.
If overriding `TEST_PACKAGES` this may result in a spurious failure on
the linter tests:
make test-wal TEST_PACKAGES=./internal/gitaly/storage/storagemgr
✓ internal/gitaly/storage/storagemgr (39.673s)
DONE 237 tests in 42.262s
stat /Users/wchandler/devel/gdk/gitaly/tools/golangci-lint/gitaly/internal/gitaly/storage/storagemgr: directory not found
=== Errors
stat /Users/wchandler/devel/gdk/gitaly/tools/golangci-lint/gitaly/internal/gitaly/storage/storagemgr: directory not found
DONE 0 tests, 1 error in 0.008s
make: *** [test-gitaly-linters] Error 1
Switch `test-wal` to use the `test-go` target which will skip the
linter.
|
|
featureflag: Remove `TransactionalLinkRepository`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6528
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
'5559-task5653-update-module-github-com-grpc-ecosystem-go-grpc-middleware-to-v2' into 'master'
package: Upgrade to go-grpc-middleware/v2 (requestinfohandler, sentryhandler)
Closes #5653
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6495
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Eric Ju <eju@gitlab.com>
Co-authored-by: Eric Ju <eju@gitlab.com>
|
|
repository: Remove RenameRepository RPC
Closes #4277
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6506
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
|
[ci skip]
|
|
The `TransactionalLinkRepository` feature flag, when enabled, makes the
`LinkRepositoryToObjectPool` RPC transactional. Remove the feature flag
so that the RPC is always transactional for clients.
|
|
The `CreateBranch` and `DeleteBranch` RPCs no longer exist but still
have their request and response protobuf messages defined. Remove these
definitions as they are no longer needed.
|
|
The `CreateBranch` RPC has been removed and there is no longer a need
for its associated request and response messages. Update call sites to
now use `UserCreateBranch` messages.
|
|
When a Praefect replica becomes out-of-date with the primary repository,
a replication job is scheduled to sync repository state. If the primary
and secondary replica are linked to non-matching object pools, the
replication job will always fail.
In this scenario, to reconcile the different object pools the target
repository should first disconnect from its object pool and then be
linked to the matching source repository object pool. Update Praefect
replication jobs to support synchronizing repositories with non-matching
object pools.
|
|
[ci skip]
|
|
This commit upgrades sentryhandler to use "github.com/grpc-ecosystem/go-grpc-middleware/v2"
|
|
This commit is part of the process of upgrading to
github.com/grpc-ecosystem/go-grpc-middleware v2.
Since Tags struct is removed in v2, this commit uses logging.Fields
to replace the old Tags struct.
Each tag will become a feild, injected by `logging.InjectLogField`,
extracted by `logging.ExtractFields`.
This commit also let the LogrusLogger has the capbility to print entry
with v2's logging.Fields.
|
|
Transaction is currently recording all reference updates in a single
map, and verifying all reference updates in a single reference
transaction. This is fine for most operations but some RPC handlers,
mainly fetches, may stage updates that would lead to a directory-file
conflict by for example staging:
1. delete 'refs/heads/parent'
2. create 'refs/heads/parent/child'
To accommodate for such transactions, this commit changes the Transaction
to record the individual reference transactions made. This retains the
ordering knowledge between the reference updates performed and allows for
saying which of the conflicting reference should exist at the end of the
transaction.
Most of the logic around references stays the same. The notable differences
are in reference verification and logging:
1. References are now verified by applying the reference transactions against
a snapshot of the target repository. This ensures the verification process
takes into account the correct ordering of the reference transactions, and
that the directory-file inducing 'refs/heads/parent' is deleted before the
child reference is verified.
2. The log entries now record reference transactions, and they are applied in
order to the repository when the log entry is being applied.
With the above changes, it's now possible to perform updates in a transaction
which would have before resulted a directory-file conflict.
|
|
TransactionManager is currently verifying references against the
main repository. The verification process prepares a reference
transaction and considers the verification successful if no errors
come up. Preparing a reference transaction in the repository may
leave locks if there is a crash while the locks are held. As the
verification happens prior to logging the write, there's no record
of the fact that locks may have been left in the repository. As it
is, such locks would be left in the repository, and never cleaned
up. This commit runs verifies the references against a snapshot of
the target repository. This has few benefits:
1. The locks target the snapshot. If there is a crash while the locks
are held, they'll be removed along with the snapshot when Gitaly
restarts.
2. As the verification process no longer modifies the repository,
transactions can take their snapshots concurrently with the
reference verification running.
3. This paves the way for verifying more complex transactions by using
multiple reference transactions. This is useful for example when a
transaction resolves a directory-file conflict. Such updates can't
be done in a single reference transaction.
In future we can also verify transactions and apply log to the repository
concurrently as the verification process no longer writes the locks to
the repository. This will especially improve throughput with Raft as
log entries don't have to be committed one by one anymore.
|
|
The snapshotting logic is currently tied to the Transaction. This was
fine until now as we always created a single snapshot in context of a
transaction. We're soon about to verify transactions against a
snapshot of the target repository. In order to reuse the code to do so,
this commit extracts out the snapshotting logic into a separate type.
The logic remains the same but we no longer access fields directly
from TransactionManager nor Transaction, and the snapshot type helps
with rewriting the relative paths to point to the snapshot.
|
|
Transactions currently snapshot the target repository and the alternate
repository if it has one. This allows transactions to operate on
repositories that already have an alternate but doesn't yet make it
possible add new alternate links. To do so, we'd need to be able to
start a transaction covering both the target repository, and the
alternate repository it is about to be linked to.
This commit makes it possible to start a transaction covering multiple
repositories. The transaction has a single target repository where
changes are recorded. In addition, multiple other repositories can be
included in the snapshot. Changes into these repositories are not
recorded or committed as we don't have a need to do so yet.
With the pool and the new member in the snapshot, the RPC handler
managing alternate links can create the link and verify it in its own
snapshot, and stage the change for committing. This will be implemented
later.
|
|
Now that all version-specific tests have been converted to unified tests
in `handler_linux_test.go`, move the remaining helper functions into
that file as well. Remove the version-specific test files as they are
now empty.
|
