Age | Commit message (Collapse) | Author |
|
Instead of recreating the repository from scratch during a restore, try
resetting the refs in the repository to the same state as they are in
the backup. If this doesn't work (for example if objects get pruned), we
proceed with recreating the repository from the bundle.
|
|
Adds a new LoggerOption to the logger provided by the `testhelper`
package that allows the caller to configure the sink of the logger. This
may be useful in test assertions to ensure that a particular message was
logged by the system under test.
|
|
Modifies the Repository interface to add a function called `ResetRefs`
which resets the references in the repository to mirror a provided list
of references.
This will be used to optimise repository restores, as we can simply
apply the list of refs from the backup to the existing repo rather than
recreating the repo from the backed-up bundle.
Additionally, the `NewRemoteRepository` and `NewLocalRepository`
constructors have exported to facilitate unit testing. I dislike this
approach, but we get an import cycle in `repository_test.go` otherwise,
since it'll need to be defined within the `backup` package instead of
`backup_test`, and it imports the `testserver` package which in turn
imports the `backup` package.
|
|
Extracts the loop responsible for restoring each step of a backup into
a separate function, along with the call to remove the repository. As a
consequence of the latter change, the tests have been updated to expect
the repository to exist even if the backup does not.
This change is done to facilitate a subsequent MR which will introduce
another function that optimises the restore process by resetting the ref
list rather than using bundles.
|
|
Moves the call to restore custom hooks out of the loop that iterates
through each incremental backup. This is done for two reasons:
- custom hooks are not backed up incrementally, so restoring them at
each iteration of the loop isn't required.
- a subsequent commit will alter how the Git data are restored, so
separating custom hooks out allows for a cleaner approach.
|
|
Adds a logger field to be used by a subsequent commit for logging
non-critical errors.
|
|
backup: Track repos that have been processed (re-attempt)
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6614
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>
|
|
Now that backups no longer invoke the RemoveAll RPC, we can remove the
exemption for these tests when the WAL is enabled.
|
|
Adds a handler to Praefect to intercept calls to the WalkRepos RPC. The
handler provides an alternate implementation of listing repositories in
a storage, which queries the Praefect DB rather than walking the
filesystem on disk. This is required so when the RPC is invoked via
Praefect, the DB is used as the source of truth rather than a random
Gitaly node.
The only user-facing difference between this and the original
implementation is that the `modification_time` attribute of the response
message is left empty, as this cannot be determined via the DB.
|
|
Now that we've adjusted the restore mechanism to delete individual
repos as needed, this RPC is no longer required. See the following
issues for more context:
- https://gitlab.com/gitlab-org/gitaly/-/issues/5357
- https://gitlab.com/gitlab-org/gitaly/-/issues/5269
Changelog: deprecated
|
|
This is now deprecated in favour of removing individual repos.
|
|
Instead of invoking the RemoveAll() RPC prior to the restore, we instead
compare the set of existing repositories to the set of repositories
contained in the backup being restored. The difference between the two
sets -- the set of "dangling" repositories -- are removed individually.
This is done to ensure the state of repos in Gitaly matches the
worldview held by the Rails DB after a GitLab instance restore.
Also fixes the existing tests so that all repositories are created with
`gittest.CreateRepository` and are thus visible when we later query
`ListRepositories` in the restore logic.
|
|
Do not pass Gitaly variables to Rails pipeline
Closes #5743
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6617
Merged-by: John Cai <jcai@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Co-authored-by: Lin Jen-Shin <jen-shin@gitlab.com>
|
|
|
|
Modifies the Restore CLI tests so we stop sending an invalid restore
command as the final JSON object into stdin. This is required as a
subsequent commit will move the repository removal logic to execute
after the Pipeline completes successfully. If the tests purposely cause
the pipeline to fail, the restore logic will never execute.
Coverage of the Pipeline's error messaging is covered separately in the
Pipeline's unit tests.
|
|
Adds a new method to the Strategy interface used by regular and
server-side backups for performing repository backups and restores. This
new method calls the internal WalkRepos() RPC to fetch a list of repos
in a given storage.
|
|
Adds a new method to the Strategy interface used by regular and
server-side backups for performing repository backups and restores. This
new method removes a single repository from its storage, and will
eventually replace the existing RemoveAllRepositories method.
|
|
Adds a map to the Pipeline to track repos that have been restored or
backed up. A mutex is used to synchronise access to the map, as entries
are appended by goroutines operating in the workers.
The signature of Done() is modified to return the map, and is
intentionally ignored in the actual backup and restore logic for now. A
subsequent commit will utilise the map for restore operations.
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
|
|
|
|
Unify transaction manager's references assertion
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6592
Merged-by: James Liu <jliu@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
In some prior commits, we added pack-refs housekeeping task support to
the transaction manager. We introduced a new assertion helper to verify
the content of packed-refs file. That helper reads the content and
compares with the expected packed-refs text. That approach makes the
test setup hard because the expected packed-refs must match the actual
content of packed-refs on disk, line by line.
This commit enhances that helper. It now parses the content of
packed-refs file. The expected value becomes a map of ref name to its
object ID.
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
|
|
|
|
|
|
'master'
proto: Add structured error proto message definitions for FindCommits grpc
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6589
Merged-by: Eric Ju <eju@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Eric Ju <eju@gitlab.com>
|
|
|
|
smarthttp: Fix upload-pack using bundle-URI with missing backup
Closes #5740
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6591
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>
|
|
|
|
featureflag: Remove `AtomicFetchRemote`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6605
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
|
|
When `bundleuri.UploadPackGitConfig()` returns an empty slice, we see
the `PostUploadPackWithSidechannel` RPC fail with `FailedPrecondition`.
The git-upload-pack(1) process started here quits with the error:
fatal: invalid command 'bundle-uri'
Looking at the source code of Git, we see when git-upload-pack(1)
receives the 'bundle-uri' command, it checks if the that capability
should be advertised. To query for this, Git checks if the config
`uploadpack.advertiseBundleURIs` is set to `true`. When Gitaly could not
build a bundle URI to pass to the git-upload-pack(1), this config was
not set, and Git did not accept this command.
For the SmartHTTP protocol Gitaly uses separate processes to handle
'GET info/refs' and 'POST upload-pack', using option `--stateless-rpc`.
Due to this, the server advertised to the client it supports
'bundle-uri' in the 'GET info/refs' response, but in the consecutive
requests we might spawn git-upload-pack(1) without this config, and Git
will not accept the 'bundle-uri' command.
Inject the config `uploadpack.advertiseBundleURIs=true` into all calls
to git-upload-pack(1), even when the bundle-URI cannot be built.
Label: bug::availability
|
|
Add new proto definition for the structured errors of FindCommits.
|
|
Revert "Merge branch 'jliu-track-restored-repos-2' into 'master'"
See merge request https://gitlab.com/gitlab-org/security/gitaly/-/merge_requests/95
Merged-by: Ahmad Tolba <atolba@gitlab.com>
Approved-by: James Fargher <jfargher@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>
|
|
The `AtomicFetchRemote` feature flag enables atomic fetches for a
repository. Since the feature flag is default enabled, remove the flag.
|
|
This reverts commit 7de65fabe6eaf803771cf0dd5d53dbaf1e628d56, reversing
changes made to 0ffc495be9bb5a8f4ff60764b545443d54210e56.
|
|
Revert "Merge branch 'fix-user-revert-return-structured-error' into 'master'"
See merge request https://gitlab.com/gitlab-org/security/gitaly/-/merge_requests/94
Merged-by: Jenny Kim <yjeankim@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
This reverts commit 7f019337fb51305215538a793cf80171111b6e29, reversing
changes made to 82dcc6a7a5692244932c4f6d4d92b3fa615aeb85.
|
|
commit: Fix bug in commit signature parsing
See merge request https://gitlab.com/gitlab-org/security/gitaly/-/merge_requests/92
Merged-by: Jenny Kim <yjeankim@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
|
|
|
'master'
Add pack-refs housekeeping task support to the transaction manager
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6551
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|