| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
backup: Only delete repositories missing from a restore
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6579
Merged-by: James Liu <jliu@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
|
|
Report stat failures correctly when creating a repository
Closes #5751
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6597
Merged-by: James Liu <jliu@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
Default enable CLONE_INTO_CGROUP support
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6598
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
Adds a handler to Praefect to intercept calls to the WalkRepos RPC. The
handler provides an alternate implementation of listing repositories in
a storage, which queries the Praefect DB rather than walking the
filesystem on disk. This is required so when the RPC is invoked via
Praefect, the DB is used as the source of truth rather than a random
Gitaly node.
The only user-facing difference between this and the original
implementation is that the `modification_time` attribute of the response
message is left empty, as this cannot be determined via the DB.
|
|
Now that we've adjusted the restore mechanism to delete individual
repos as needed, this RPC is no longer required. See the following
issues for more context:
- https://gitlab.com/gitlab-org/gitaly/-/issues/5357
- https://gitlab.com/gitlab-org/gitaly/-/issues/5269
Changelog: deprecated
|
|
This is now deprecated in favour of removing individual repos.
|
|
Instead of invoking the RemoveAll() RPC prior to the restore, we instead
compare the set of existing repositories to the set of repositories
contained in the backup being restored. The difference between the two
sets -- the set of "dangling" repositories -- are removed individually.
This is done to ensure the state of repos in Gitaly matches the
worldview held by the Rails DB after a GitLab instance restore.
|
|
Modifies the Restore CLI tests so we stop sending an invalid restore
command as the final JSON object into stdin. This is required as a
subsequent commit will move the repository removal logic to execute
after the Pipeline completes successfully. If the tests purposely cause
the pipeline to fail, the restore logic will never execute.
Coverage of the Pipeline's error messaging is covered separately in the
Pipeline's unit tests.
|
|
Adds a new method to the Strategy interface used by regular and
server-side backups for performing repository backups and restores. This
new method calls the internal WalkRepos() RPC to fetch a list of repos
in a given storage.
|
|
Adds a new method to the Strategy interface used by regular and
server-side backups for performing repository backups and restores. This
new method removes a single repository from its storage, and will
eventually replace the existing RemoveAllRepositories method.
|
|
Enhance developing workflow with Ruby gem in Gitlab Rails
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6593
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Approved-by: Eric Ju <eju@gitlab.com>
Reviewed-by: Toon Claes <toon@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Eric Ju <eju@gitlab.com>
|
|
protoc-gen-doc: Update github.com/gogo/protobuf
Closes #5746
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6600
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
|
|
Updating link because I removed Pajamas page
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6599
Merged-by: John Cai <jcai@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Co-authored-by: Suzanne Selhorn <sselhorn@gitlab.com>
|
|
|
|
There is a known vulnerability in protoc-gen-doc. Fix this by updating
to v1.3.2.
|
|
Sometimes, we need to work on both Gitaly and GitLab Rails
implementations in parallel. Ideally, we can define and publish all
Protobuf changes beforehand. In practice, a complicated change might
require modifying the Protobuf multiple times until reaching a stable
state. It's more convenient to let GitLab Rails point the gem to the the
developing branch in Gitaly.
Unfortunately, Gitaly source code doesn't include the gemspec or any
autogenerated Ruby codes. All of them are generated on the flight while
running `make build-proto-gem`. Thus, we cannot point the gem to the
developing branch directly on CI environment.
This commit tweaks the gem generation script:
- Use current HEAD hash as the pre-release version.
- Persist gem working directory after building. This allows local GitLab
Rails point to that directory directly.
- Add support for gem name customization. This allows the developer
publish the gem under a different name during development.
|
|
operations: Return structured errors in UserRevert
Closes #5630
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6533
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>
Co-authored-by: wang yadong <wangyadong1108@foxmail.com>
|
|
|
|
protolint: Update golang.org/x/text
Closes #5747 and #5748
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6595
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
|
|
There are known vulnerabilities in golang.org/x/text v0.3.* used by
protolint. Update to the most recent version.
|
|
CLONE_INTO_CGROUP support is currently default disabled. We can't test
it on production as we're not running cgroup v2. Cgroup v2 is also
broken when used through Omnibus as Gitaly doesn't run in the same
hierarchy as the child cgroups and can't thus move move the processes
it spawns into those cgroups.
To remove the feature flag before we support cgroup v2 ourselves, let's
just enable the flag by default and remove it a couple of releases after
if there are no issues reported. Manual testing has shown it to work.
Default enabling this would only change behavior for setups that manually
configure the cgroups and avoid the issue with moving processes across
hierarchies.
Changelog: fixed
|
|
Gitaly currently reports always a 'repository exists already' when
statting a repository during creation fails. This is wrong and
confusing as we hide the actual cause of the error. This commit
fixes the error check to report failures of the stat call correctly.
Changelog: fixed
|
|
[ci skip]
|
|
Adds a map to the Pipeline to track repos that have been restored or
backed up. A mutex is used to synchronise access to the map, as entries
are appended by goroutines operating in the workers.
The signature of Done() is modified to return the map, and is
intentionally ignored in the actual backup and restore logic for now. A
subsequent commit will utilise the map for restore operations.
|
|
git: Add bundled Git version 2.43
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6587
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
|
|
Add Git version 2.43 to the installed bundled Git versions. Use this new
version of Git when feature flag `git_v243` is enabled.
Roll-out-issue: https://gitlab.com/gitlab-org/gitaly/-/issues/5739
Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/5713
Label: maintenance::dependency
|
|
backup: Refactor Pipeline struct
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6576
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: James Fargher <jfargher@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>
|
|
hook: Add the `ProcReceiveHook` server
Closes #4515
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6577
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
|
|
With the creation of `ProcReceiveHook` under the `HookManager`, we can
now create the `ProcReceiveHook` server. The server simply propagates
the data to/from the `HookMananger.ProcReceiveHook` to the client.
|
|
For the `ProcReceiveHook` to work we need to create a RPC which is
called by the `gitaly-hooks` binary. Before we create this RPC, let's
note the definition of the RPC in the `hook.proto` file.
In the following commits we'll define the client and the service
following this definition.
|
|
The `Transmit()` function provided by `ProcReceiveRegistry` is a
blocking function, which ensures that a handler is transmitted to any
waiters.
The problem is, that there is a potential scenario where the waiter
might have exited early. Hence blocking this call forever. So to avoid
that, we expand the function to receive a context. With this, if the
main RPC (waiter) exits early, the context will be canceled and we wont
be blocked forever.
|
|
Currently the `Close()` function of the `ProcReceiveHandler` only
provides information on whether the handler's job is completed or not.
But it would be useful for the handler to also propagate errors, this
could be used by the ProcReceive server to return the appropriate exit
code to git-receive-pack(1).
So let's convert the `Close()` function to accept an error and propagate
this error via the `doneCh`.
|
|
We currently have `ProtoEqual`, which compares two proto structs. This
function uses the `require` package. The require package though is not
goroutine safe since it uses `t.FailNow()`.
So introduce a similar `ProtoEqualAssert()` which uses the `assert`
package. This is safe to use in goroutines.
|
|
praefect: Enable grpcui compatibility
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6586
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Toon Claes <toon@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>
|
|
Registers the gRPC reflection service with the Praefect server to enable
tools like grpcui to detect the server's endpoints. With this change, it
is possible to test Praefect's endpoints locally with the following:
grpcui -unix -plaintext <path/to/praefect.socket>
|
|
gitattributes: Get rid of endOfAttributes
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6490
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
|
|
When the user passes an empty list of attributes, we cannot properly
parse the output from git-check-attr(1). To protect against that, return
an error when the list of names passed to `gitattributes.CheckAttr()` is
empty.
|
|
`gitattributes.CheckAttr` appended "end-of-attributes-----" to the names
of the attributes it wants to read. When the output of git-check-attr(1)
is parsed, Gitaly reads until this attribute is found.
Instead, memorize the number of attributes the user requested and keep
reading until that number of attributes is read. This works because the
user should not do concurrent calls to `Check()`
Label: maintenance::refactor
|
|
Renames pipelineErrors to commandErrors and encapsulates the mutex so
locking can be managed internally.
|
|
Modifies the parallelism tests for the Pipeline so the number of
concurrent jobs can be tracked per storage. This allows us to ensure
that both the `parallel` and `parallelStorage` thresholds are being
respected, and will come in handy for a future optimisation that will
attempt to balance operations across storages.
|
|
|
|
Uses the option function pattern to set the concurrency of a pipeline
during initialisation. This makes for a more readable flow, and allows
us to set the default state of nil-concurrency internally.
|
|
It's not used outside of the backup package.
|
|
Now that we have a single Pipeline implementation, we can drop the
Parallel prefix from the Pipeline name. This also removes the need for
the separate Pipeline interface, which can now be deleted too.
|
|
Adjust housekeeping's grace period of stale file cleaning
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6584
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
'eread/move-rest-of-content-from-gitlab-docs-to-gitaly-example-configuration-file' into 'master'
Move rest of content from GitLab docs to Gitaly example config file
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6580
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>
|
|
|
|
Our housekeeping manager has a task to clean up stale files. It uses
object's modification time to determine if a file needs to be cleaned.
The current grace period is 7 days.
The original purpose of this grace period was to avoid a temp dir being
removed when some commands still need it. Scanning through Git code
bases, I found some similar usage of temp dirs/files inside the objects
directory:
* tmp_objdir-incoming.
* tmp_objdir-remerge-diff.
* tmp_objdir-bulk-fsync.
* tmp_obj_XXXXX.
* pack/tmp_pack_XXXXXX
* pack/tmp_idx_XXXXXX
* pack/tmp_rev_XXXXXX
* pack/tmp_mtimes_XXXXXX
* pack/tmp_bitmap_XXXXXX
* And many more less common files
In most cases, these temp files are either removed or migrated right
after a command is done. A majority of them are extremely short-lived.
So, the current 7-day window seems to be too relaxed. There mustn't be a
command running for days.
We can cut it down to a couple hours. However:
- Rails' default long timeout is 6 hours. Gitaly should take that into
account to avoid cleaning up data by accident while a request is
runnning.
- Some housekeeping commands might run for hours. We cannot control all
the factors leading to that slowness. Adding some headroom is a safer
choice.
As a result, this commit reduces the grace period down to 24 hours.
That should be good enough for most cases.
|
