| Age | Commit message (Collapse) | Author |
|---|
|
Remove the now unused v1 and v2 handler implementations, copying
still-used freestanding functions into `handler_linux.go`.
|
|
Start using `genericHandler` now that it fully implements the
`cgroupHandler` interface.
|
|
Add the `collect()` method to `genericHandler`. As with `stats()`, the
types we deal with here vary between v1 and v2, so we keep separate
functions for them.
|
|
Add the `stats()` method to `genericHandler`. We use reflection here as
the stats collected vary significantly between v1 and v2.
|
|
Add the methods to `genericHandler` that contain no version-specific
logic.
|
|
Create a new `genericHandler` to unify most of the logic for cgroups-v1
and v2. This is made generic over two parameters:
T: The cgroup type, `cgroup1.Cgroup` or `*cgroup2.Manager`.
H: The hierarchy or mountpoint passed when creating a cgroup. For
`cgroup1.New` this is a `cgroup1.Hierarchy`, for `cgroup2.New` this
is a `string` with the path to the cgroup's mountpoint.
Four generic functions are added for the operations that are easily
abstracted over. Metrics and stats collection are not included as they
involve significant version-specific logic.
|
|
Now that all version-specific tests have been converted to unified tests
in `handler_linux_test.go`, move the remaining helper functions into
that file as well. Remove the version-specific test files as they are
now empty.
|
|
Combine the v1 and v2 versions of `TestStats()` to a single test.
Here we have the opposite of `TestPruneOldCgroups()`, the test cases
are unique to v1 or v2, but the assertions are version agnostic. Add a
`version` field to the test cases, rather than iterating over each case
with each version as we did with the other tests.
|
|
Combine the v1 and v2 copies of `TestPruneOldCgroups` into a single
test. We are unable to deduplicate most of the actual assertions as they
rely heavily on the subcomponent paths. This is still a worthwhile
change as the test cases can still be shared.
|
|
Convert the separate v1 and v2 `TestMetrics` versions to a single test.
|
|
Convert the v1 and v2 versions of TestCleanup to a single unified test.
|
|
Convert the v1 and v2 versions of `TestAddCommand` to a single shared
test.
|
|
Convert the v1 and v2 copies of `TestSetup_RepoCgroups` to a single test
that covers both. We drop the `Setup` from the test name as repository
cgroups are no longer created during setup.
|
|
Coalesce `TestSetup_ParentCgroups()` and `TestSetup_ParentCgroupsV2()`
into a single test that handles both v1 and v2 cgroups.
Note that the `requireCgroupComponents()` helper introduced slightly
changes the conditions checked versus the v1 test. The v2 manager
requires that `setupMockCgroupFiles()` be executed before running
`Setup()`. The v1 test used `requireCgroup()`, which would verify that a
file was _not_ present, which is no longer be the case now that we have
setup the cgroup files. We switch to `requireCgroupWithInt()` to avoid
this failure.
|
|
Rename `TestNewManager` to `TestNewManagerNoop` so that we can use the
more generic name for our combined test. Unify the separate manager
tests for v1 and v2 cgroups into a single `TestNewManager` test.
|
|
Introduce a `mockCgroup` interface to allow tests to create a v1 or v2
mock depending on a test parameter, rather than duplicating the entire
test.
This requires accessing the `root` path via the new `rootPath()` method,
and a convenience method for calculating the mock's repository cgroup
paths is also added.
|
|
We will shortly be creating an interface to cover the mock cgroups for
v1 and v2. Rename `mockCgroup` to `mockCgroupV1` to clear the way.
|
|
On systems with very large numbers of existing cgroups, initializing
Gitaly's repository cgroups may take upwards of 20 seconds. With
`tableflip` this delay has no impact as the existing process is still
present to take traffic until the new one finishes bootstrapping.
However, we need to remove `tableflip` as the WAL does not allow
concurrent Gitaly processes accessing the same repository.
Create a given repository cgroup on-demand the first time a
process needs to be added to it.
Changelog: performance
|
|
Update the `CloneIntoCgroup` method to check if the target cgroup is
present and create it if not.
This also updates the method to be clear that the file handle we open is
for the target cgroup directory, not a file. From clone(2):
This file descriptor can be obtained by opening a cgroup v2
directory using either the O_RDONLY or the O_PATH flag.
|
|
We will shortly start creating cgroups on-demand, rather than up-front
as part of Gitaly's start up process.
Hoist the loop over repository cgroups into the manager's `Collect`
method, passing the path into the version-specific handlers. This allows
us to avoid leaking the `cgroupLock` abstraction outside of the manager.
|
|
We will shortly start creating repository cgroups on-demand, rather than
at startup. This will require synchronization as we may have concurrent
requests hitting the same cgroup. Racing requests may cause spurious
failures and waste system resources.
Add the `setupRepoCgroup` and `setupRepoCgroupAndAdd` methods to create
individual repo cgroups and take a `cgroupLock`. These provides
exclusive access when creating the cgroup. Also add a map which is
pre-populated with initialized `cgroupLock`s for every available value.
|
|
Currently we generate the `reposResources` value via the
`configRepositoryResources()` function on the `CGroupManager`. This
works well when we create all of the repository cgroups in one
batch, but we will shortly move to creating them on-demand.
To avoid regenerating this struct for each of the N repository cgroups
defined, create it once during construction of the `CgroupManger` and
store it in the struct.
|
|
featureflag: Default enable `TransactionalAlternatesDisconnect`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6511
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
The `TransactionalAlternatesDisconnect` feature flag enables
transactions for the `DisconnectGitAlternates` RPC. Set this feature
flag to be enabled by default.
|
|
Exclude `inactive_file` from adaptive limiter's memory watcher
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6508
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
protocol: Add support for git protocol version 1
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6504
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Xing Xin <xingxin.xx@bytedance.com>
|
|
When the resource watchers report the backoff events to the adaptive
calculator, they attach a reason for the backoff (threshold exceeded for
example). The reason consists of some stringified numbers for
clarification. This commit moves those numbers to a hashmap for better
readability and explorability. In some later commits, we'll attach more
stats to that hashmap.
|
|
This commit adds some more essential fields collected from `memory.stat`
file. There are two differences between Cgroup V1 and V2:
- `rss` in V1 is named `anon` in V2.
- `cache` in V1 is named `file` in V2.
The names in V2 reflect the meanings of underlying metrics better. Thus,
the reported stats use V2's names.
|
|
Support running Gitaly with transactions enabled
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6496
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
Protocol version 1 is a valid protocol that is supported by git.
However, we currently only have support for versions 0 and 2. This
commit expands our support to include protocol v1 as well, as there is
no particular reason to disable it.
Furthermore, there are indeed some differences when pushing using v1,
as compared to the output when using v0 or v2.
When pushing using v1, we got
$ GIT_TRACE_PACKET=1 git -c protocol.version=1 push git@github.com:blanet/demo.git
14:38:23.953382 pkt-line.c:85 packet: push< version 1
14:38:23.954345 pkt-line.c:85 packet: push< 47528871788c78cc2182219b6f3333cb401ce180 refs/heads/main\0report-status report-status-v2 delete-refs side-band-64k ofs-delta atomic object-format=sha1 quiet agent=github/spokes-receive-pack-302ef7e34db14cc0c1823fcb16b9517d4c29a87b session-id=fdba6af034c2f1e0ef3d1c5fca233a54 push-options
14:38:23.954446 pkt-line.c:85 packet: push< 0000
14:38:23.954570 pkt-line.c:85 packet: push> 0000
Everything up-to-date
When pushing using v0 or v2, we got
$ GIT_TRACE_PACKET=1 git -c protocol.version=2 push git@github.com:blanet/demo.git
14:37:54.596645 pkt-line.c:85 packet: push< 47528871788c78cc2182219b6f3333cb401ce180 refs/heads/main\0report-status report-status-v2 delete-refs side-band-64k ofs-delta atomic object-format=sha1 quiet agent=github/spokes-receive-pack-302ef7e34db14cc0c1823fcb16b9517d4c29a87b session-id=7509d3b188f163aa43593dab6d0c45e6 push-options
14:37:54.598059 pkt-line.c:85 packet: push< 0000
14:37:54.598170 pkt-line.c:85 packet: push> 0000
Everything up-to-date
Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
Changelog: added
|
|
Disable the confusing reftxn setup for InfoRefsReceivePack, which is an
ACCESSOR that does not involve reference updates.
Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
|
|
Begin transaction against a given repository
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6501
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
|
|
|
|
cgroups: Deduplicate repeated code
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6500
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Eric Ju <eju@gitlab.com>
|
|
This commit adds a configuration option for running Gitaly with
transaction management enabled. While transactions are not yet ready
for production use, this allows them to be configured for development
and testing purposes.
Transactions are integrated with rest of Gitaly by configuring an
interceptor in GitalyServerFactory that handles beginning,
committing and rolling back transactions for RPCs.
PartitionManager is initialized on Gitaly's start up, and closed when
Gitaly is terminating. Initializing and stopping the partition manager
opens and closes the database handles for each storage where Gitaly
stores some transaction processing related state.
|
|
BadgerDB stores the keys in the LSM tree and the in a log. The LSM
tree is compacted in the background automatically. The client is
responsible for triggering the value log's garbage collection to prune
out unneeded values. As we'll soon wire transactions to Gitaly, we
should start garbage collecting the value log as well.
This commit makes PartitionManager run a garbage collection goroutine
for each storage's database. The goroutine is started when the database
is opened and stopped when the PartitionManager is closed. Garbage
collection is ran for each database once per minute. The garbage collection
works by rewriting the value log while omitting the pruned values. While
the garbage collection runs once a minute, it's configured to only rewrite
the log file if it can free more than 50% of the space. This aims to strike
a balance between the load from rewriting the log files and reclaiming space.
Each RunValueLogGC run garbage collects only a single file. If a file was
garbage collected, we run the function again immediately to check for further
files that may need garbage collection
The garbage collection interval and the discard ratio may need to be changed
in the future. We'll make them configurable in later changes.
|
|
Gitaly's code is using tickers in multiple places. They make it easier
to test code that runs on an interval. The tickers are generally
constructed using a factory method injected into the type using the
ticker. This makes it easier to mock out the timer based tickers in
tests. We'll soon need this in the PartitionManager tests. This commit
implements factory methods for timer ticker and a ticker that never
ticks. The timer tickers can be used to configure the production code,
and the null tickers can be used to disable the ticking for example in
tests. If the tests need to manually control the ticking, they can
do so by passing a custom TickerFactoryFunc into the type under test.
|
|
PartitionManager will soon gain a goroutine that runs garbage
collection on the Badger database. In order to test that change,
we need to be able to mock out badger in the test. This commit
injects a database opener into PartitionManager so we can mock out
the database in the tests.
|
|
We're currently taking in a concrete *badger.DB instance into most
of the types. We're soon about to introduce a garbage collecting
goroutine that would be best tested by mocking the database. Convert
the types to take in the interface instead.
|
|
The Database interface doesn't currently cover all of the methods
used by the various components. Add the missing methods on it so we
can inject the interface into all of the components.
|
|
This commit exports the currently unexported database interface. This
makes it easier to run tests which require mocking the database as part
of the test.
|
|
This commit adds a test for ensuring we return a proper error when a
transaction is begun without a relative path provided. This would not
happen in production as we validate relative paths already in the
transaction middleware but is here for thoroughness and to ease
troubleshooting.
|
|
TransactionManager tests are currently defaulting to the pre-existing
repository's relative path if one is not specified in the test. This
takes away the empty relative path as a valid parameter in the test
setup. Configure each test case explicitly with the relative path they
want to access instead. This ensures:
1. We can explicitly test the case where transaction is begun with
an empty relative path.
2. Lays ground for future refactorings to remove the pre-existing
repository from the tests. It was needed before as we couldn't
create repositories through transactions. Now we can, so we should
exercise that code where we need a repository. This is left for
the future though.
|
|
This commit extends the Begin method to take in a relative path and
start a transaction against that repository. With this change it becomes
possible to operate on multiple repositories within given partition.
As it is now possible to specify which repository the transaction targets,
the relativePath field of the TransactionManager is removed.
|
|
This commit removes the TransactionOptions type places the ReadOnly
field as a parameter of Begin. Having a separate type seems somewhat
unnecessary for now given the low number of parameters going into
Begin.
|
|
PartitionManager is currently taking in a storage.Repository to determine
the target repository of a transaction. This worked reasonably for a single
repository. It's not as convenient as an interface when a transaction may
span multiple repositories. Each of the repositories must be in the same
storage, so storage should be given as a parameter only once. The quarantine
parameters may also be set in Repository yet they are completely ignored
by the transaction code. Change the interface to instead take in a storage
name and the relative path of the repository directly. This removes the
unnecessary fields inside Repository from the interface. When we start
supporting multi-repo transactions, we'll only take in the storage name
once and can take in the other relative paths a simple slice.
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
