| Age | Commit message (Collapse) | Author |
|---|
|
Write latest manifest file
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6613
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: James Fargher <jfargher@gitlab.com>
|
|
Partition fork with source repository in CreateFork
Closes #5762
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6612
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
fix: Fix a collection of typos found by typos-cli
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6594
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Evan Read <eread@gitlab.com>
Co-authored-by: Xing Xin <xingxin.xx@bytedance.com>
|
|
Fix typos found by typos-cli(https://github.com/crate-ci/typos). Some
affected tests are adjusted.
There are a bunch of other typos are ignored, including
* CHANGELOG.md
* NOTICE
* internal/.../migrations/20201208163237_cleanup_notifications_payload.go
* other intended typos or false positives
Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
|
|
[ci skip]
|
|
backup: Track repos that have been processed (re-attempt)
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6614
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>
|
|
Now that backups no longer invoke the RemoveAll RPC, we can remove the
exemption for these tests when the WAL is enabled.
|
|
Adds a handler to Praefect to intercept calls to the WalkRepos RPC. The
handler provides an alternate implementation of listing repositories in
a storage, which queries the Praefect DB rather than walking the
filesystem on disk. This is required so when the RPC is invoked via
Praefect, the DB is used as the source of truth rather than a random
Gitaly node.
The only user-facing difference between this and the original
implementation is that the `modification_time` attribute of the response
message is left empty, as this cannot be determined via the DB.
|
|
Now that we've adjusted the restore mechanism to delete individual
repos as needed, this RPC is no longer required. See the following
issues for more context:
- https://gitlab.com/gitlab-org/gitaly/-/issues/5357
- https://gitlab.com/gitlab-org/gitaly/-/issues/5269
Changelog: deprecated
|
|
This is now deprecated in favour of removing individual repos.
|
|
Instead of invoking the RemoveAll() RPC prior to the restore, we instead
compare the set of existing repositories to the set of repositories
contained in the backup being restored. The difference between the two
sets -- the set of "dangling" repositories -- are removed individually.
This is done to ensure the state of repos in Gitaly matches the
worldview held by the Rails DB after a GitLab instance restore.
Also fixes the existing tests so that all repositories are created with
`gittest.CreateRepository` and are thus visible when we later query
`ListRepositories` in the restore logic.
|
|
Do not pass Gitaly variables to Rails pipeline
Closes #5743
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6617
Merged-by: John Cai <jcai@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Co-authored-by: Lin Jen-Shin <jen-shin@gitlab.com>
|
|
|
|
Now that a latest manifest is written, we can use this manifest when
loading the "latest" backup.
|
|
Manifests currently cannot restore the "latest" backup because it would
require an expensive object-storage directory traversal. So instead it
defers to the pointer layout. The problem with this is that you then
loose manifest only features like setting the default branch properly.
Here we write two manifests: the normal backup manifest as before and an
additional "latest" manifest file. This latest manifest is overwritten
on each backup taken.
For WORM we would ideally not overwrite any files, but until this is
implemented we need something to fill the latest restore gap.
Changelog: changed
|
|
Modifies the Restore CLI tests so we stop sending an invalid restore
command as the final JSON object into stdin. This is required as a
subsequent commit will move the repository removal logic to execute
after the Pipeline completes successfully. If the tests purposely cause
the pipeline to fail, the restore logic will never execute.
Coverage of the Pipeline's error messaging is covered separately in the
Pipeline's unit tests.
|
|
Adds a new method to the Strategy interface used by regular and
server-side backups for performing repository backups and restores. This
new method calls the internal WalkRepos() RPC to fetch a list of repos
in a given storage.
|
|
Adds a new method to the Strategy interface used by regular and
server-side backups for performing repository backups and restores. This
new method removes a single repository from its storage, and will
eventually replace the existing RemoveAllRepositories method.
|
|
Adds a map to the Pipeline to track repos that have been restored or
backed up. A mutex is used to synchronise access to the map, as entries
are appended by goroutines operating in the workers.
The signature of Done() is modified to return the map, and is
intentionally ignored in the actual backup and restore logic for now. A
subsequent commit will utilise the map for restore operations.
|
|
Gitaly's TransactionManager requires all object pool members to be
in the same partition. Currently we're ensuring doing that generally
by extracting the additional repository from the request and ensuring
the target repository of the RPC gets partitioned with it. Additional
repository is used in the ObjectPoolService's RPCs to tag the other
repository being accessed in the pool related operations, and it may
be either the object pool or one of the member repositories depending
on the RPC.
When a repository is first accessed, we also check whether it has an
existing alternate link on the disk, and place the repository in the
same partition with its alternate if so.
These two methods are enough to ensure in general the pools and their
members get partitioned together. One execption to this is CreateFork.
The created fork should be placed in the same partition as the origin
repository as they'll both eventually be connected to the same pool.
CreateFork does not tag the source repository as an additional
repository so the general handling does not apply for it. Tagging it
as the additional repository won't work with Praefect as Praefect
rewrites the paths of additional repositories. The additional repository
is fetched through the API so it needs to have its original relative
path intact.
This commit introduces special handling for CreateFork. The transaction
middleware checks whether the request is a CreateForkRequest. If so,
the source repository is extracted and the newly created fork gets
partitioned with it.
Along with the behavior changes, we add a test that exercises the
entire fork creation flow as typically done. Turns out Gitaly didn't
have a test covering the scenario at all.
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
|
|
|
|
Unify transaction manager's references assertion
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6592
Merged-by: James Liu <jliu@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
In some prior commits, we added pack-refs housekeeping task support to
the transaction manager. We introduced a new assertion helper to verify
the content of packed-refs file. That helper reads the content and
compares with the expected packed-refs text. That approach makes the
test setup hard because the expected packed-refs must match the actual
content of packed-refs on disk, line by line.
This commit enhances that helper. It now parses the content of
packed-refs file. The expected value becomes a map of ref name to its
object ID.
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
|
|
|
|
|
|
'master'
proto: Add structured error proto message definitions for FindCommits grpc
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6589
Merged-by: Eric Ju <eju@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Eric Ju <eju@gitlab.com>
|
|
|
|
smarthttp: Fix upload-pack using bundle-URI with missing backup
Closes #5740
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6591
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>
|
|
|
|
featureflag: Remove `AtomicFetchRemote`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6605
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
|
|
When `bundleuri.UploadPackGitConfig()` returns an empty slice, we see
the `PostUploadPackWithSidechannel` RPC fail with `FailedPrecondition`.
The git-upload-pack(1) process started here quits with the error:
fatal: invalid command 'bundle-uri'
Looking at the source code of Git, we see when git-upload-pack(1)
receives the 'bundle-uri' command, it checks if the that capability
should be advertised. To query for this, Git checks if the config
`uploadpack.advertiseBundleURIs` is set to `true`. When Gitaly could not
build a bundle URI to pass to the git-upload-pack(1), this config was
not set, and Git did not accept this command.
For the SmartHTTP protocol Gitaly uses separate processes to handle
'GET info/refs' and 'POST upload-pack', using option `--stateless-rpc`.
Due to this, the server advertised to the client it supports
'bundle-uri' in the 'GET info/refs' response, but in the consecutive
requests we might spawn git-upload-pack(1) without this config, and Git
will not accept the 'bundle-uri' command.
Inject the config `uploadpack.advertiseBundleURIs=true` into all calls
to git-upload-pack(1), even when the bundle-URI cannot be built.
Label: bug::availability
|
|
Add new proto definition for the structured errors of FindCommits.
|
|
Revert "Merge branch 'jliu-track-restored-repos-2' into 'master'"
See merge request https://gitlab.com/gitlab-org/security/gitaly/-/merge_requests/95
Merged-by: Ahmad Tolba <atolba@gitlab.com>
Approved-by: James Fargher <jfargher@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>
|
|
The `AtomicFetchRemote` feature flag enables atomic fetches for a
repository. Since the feature flag is default enabled, remove the flag.
|
|
This reverts commit 7de65fabe6eaf803771cf0dd5d53dbaf1e628d56, reversing
changes made to 0ffc495be9bb5a8f4ff60764b545443d54210e56.
|
|
Revert "Merge branch 'fix-user-revert-return-structured-error' into 'master'"
See merge request https://gitlab.com/gitlab-org/security/gitaly/-/merge_requests/94
Merged-by: Jenny Kim <yjeankim@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
This reverts commit 7f019337fb51305215538a793cf80171111b6e29, reversing
changes made to 82dcc6a7a5692244932c4f6d4d92b3fa615aeb85.
|
|
commit: Fix bug in commit signature parsing
See merge request https://gitlab.com/gitlab-org/security/gitaly/-/merge_requests/92
Merged-by: Jenny Kim <yjeankim@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
