| Age | Commit message (Collapse) | Author |
|---|
|
Since the gpg signature is time-dependent, aligning the timestamp with the
author's date makes the gpg signature consistent across all gitaly nodes.
|
|
cleanup: Add RewriteHistory RPC
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6615
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
|
|
git: Use Git version 2.43 only
Closes #5739
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6624
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Eric Ju <eju@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
Enable transactions for the new RewriteHistory RPC.
|
|
On large repositories git-filter-repo(1) make take a significant amount
of time to run. Should a write occur after the git-fast-export(1)
portion of the task has completed, it is possible that the repository
history will not be fully rewritten.
To guard against this condition, we checksum the repository before and
after running filter-repo. If the checksums do not match we abort and do
not fetch the updated history into the repository.
|
|
git-filter-repo(1) uses git-fast-import(1) to import the rewritten
repository history. This will unpack the new objects, then iterate over
references serially and update them using reference transactions. This
does not atomically update the references[0], so an interruption during
this final stage will result in partially applied changes.
To mitigate this risk, create a temporary staging repository to write
the updated history into, then atomically force fetch that into the
original repo.
This has the downside of being slower than modifying the repository
in-place[1], but improving safety for a high-risk operation like this
is a greater priority.
[0] https://gitlab.com/gitlab-org/git/-/blob/d4dbce1db5cd227a57074bcfc7ec9f0655961bba/builtin/fast-import.c#L1659-1668
[1] https://github.com/newren/git-filter-repo/issues/66#issuecomment-602100316
|
|
Historically we have advised users who need to rewrite history to do so
locally and force push their change to Gitlab. However, upcoming changes
may prevent a user from pushing in scenarios where they need to remove a
large blob from their repository's history.
To handle this scenario, we introduce a new `RewriteHistory` RPC which
will invoke git-filer-repo(1) on the target repository. filter-repo
has a large number of options, but we will support only two:
--strip-blogs-with-ids
Given a file containing a list of newline-delimited object ids,
rewrite history to remove them from all commits.
--replace-text
Given a file of literals and patterns, replace all matching
instances in history with '***REMOVED***'.
filter-repo works by fetching the repository contents via
git-fast-export(1), making the requested changes, and writing the
changes back via git-fast-import(1). As filter-repo uses the '--force'
flag[0] the repository must be made read-only before calling this RPC.
filter-repo is currently incompatible with SHA256 repositories.
[0] https://git-scm.com/docs/git-fast-import#_parallel_operation
Changelog: added
|
|
Add git-filter-repo(1) as a recognized subcommand.
|
|
backup: Check for backup before deleting repo
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6620
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>
|
|
Remove the feature flag to toggle the use of v2.43, enable it by
default, and remove the use of v2.42.
Label: maintenance::dependency
|
|
|
|
Reorders the repository restore logic so that we do not remove the repo
until we've checked that it has an existing backup. If the repo has no
backup, it's skipped from the restore and will be removed later by the
caller.
The previous order of operations caused issues when performing a full
restore that included a dangling repo (a repo created after the backup
was taken). When the `Restore` function was executed against this repo,
it was removed immediately. Then, the remainder of the restore was
skipped since no backup existed for that repo. Since the repo was not
marked as restored, the caller of the `Restore` function tried to remove
it once again, leading to a "repository not found" error since it had
already been erased.
The "missing backup" test case for the restore of a specific backup has
been updated to expect that the repo exists to align with this logic
|
|
Write latest manifest file
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6613
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: James Fargher <jfargher@gitlab.com>
|
|
Partition fork with source repository in CreateFork
Closes #5762
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6612
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
Fix typos found by typos-cli(https://github.com/crate-ci/typos). Some
affected tests are adjusted.
There are a bunch of other typos are ignored, including
* CHANGELOG.md
* NOTICE
* internal/.../migrations/20201208163237_cleanup_notifications_payload.go
* other intended typos or false positives
Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
|
|
Now that backups no longer invoke the RemoveAll RPC, we can remove the
exemption for these tests when the WAL is enabled.
|
|
Adds a handler to Praefect to intercept calls to the WalkRepos RPC. The
handler provides an alternate implementation of listing repositories in
a storage, which queries the Praefect DB rather than walking the
filesystem on disk. This is required so when the RPC is invoked via
Praefect, the DB is used as the source of truth rather than a random
Gitaly node.
The only user-facing difference between this and the original
implementation is that the `modification_time` attribute of the response
message is left empty, as this cannot be determined via the DB.
|
|
Now that we've adjusted the restore mechanism to delete individual
repos as needed, this RPC is no longer required. See the following
issues for more context:
- https://gitlab.com/gitlab-org/gitaly/-/issues/5357
- https://gitlab.com/gitlab-org/gitaly/-/issues/5269
Changelog: deprecated
|
|
This is now deprecated in favour of removing individual repos.
|
|
Instead of invoking the RemoveAll() RPC prior to the restore, we instead
compare the set of existing repositories to the set of repositories
contained in the backup being restored. The difference between the two
sets -- the set of "dangling" repositories -- are removed individually.
This is done to ensure the state of repos in Gitaly matches the
worldview held by the Rails DB after a GitLab instance restore.
Also fixes the existing tests so that all repositories are created with
`gittest.CreateRepository` and are thus visible when we later query
`ListRepositories` in the restore logic.
|
|
Now that a latest manifest is written, we can use this manifest when
loading the "latest" backup.
|
|
Manifests currently cannot restore the "latest" backup because it would
require an expensive object-storage directory traversal. So instead it
defers to the pointer layout. The problem with this is that you then
loose manifest only features like setting the default branch properly.
Here we write two manifests: the normal backup manifest as before and an
additional "latest" manifest file. This latest manifest is overwritten
on each backup taken.
For WORM we would ideally not overwrite any files, but until this is
implemented we need something to fill the latest restore gap.
Changelog: changed
|
|
Modifies the Restore CLI tests so we stop sending an invalid restore
command as the final JSON object into stdin. This is required as a
subsequent commit will move the repository removal logic to execute
after the Pipeline completes successfully. If the tests purposely cause
the pipeline to fail, the restore logic will never execute.
Coverage of the Pipeline's error messaging is covered separately in the
Pipeline's unit tests.
|
|
Adds a new method to the Strategy interface used by regular and
server-side backups for performing repository backups and restores. This
new method calls the internal WalkRepos() RPC to fetch a list of repos
in a given storage.
|
|
Adds a new method to the Strategy interface used by regular and
server-side backups for performing repository backups and restores. This
new method removes a single repository from its storage, and will
eventually replace the existing RemoveAllRepositories method.
|
|
Adds a map to the Pipeline to track repos that have been restored or
backed up. A mutex is used to synchronise access to the map, as entries
are appended by goroutines operating in the workers.
The signature of Done() is modified to return the map, and is
intentionally ignored in the actual backup and restore logic for now. A
subsequent commit will utilise the map for restore operations.
|
|
Gitaly's TransactionManager requires all object pool members to be
in the same partition. Currently we're ensuring doing that generally
by extracting the additional repository from the request and ensuring
the target repository of the RPC gets partitioned with it. Additional
repository is used in the ObjectPoolService's RPCs to tag the other
repository being accessed in the pool related operations, and it may
be either the object pool or one of the member repositories depending
on the RPC.
When a repository is first accessed, we also check whether it has an
existing alternate link on the disk, and place the repository in the
same partition with its alternate if so.
These two methods are enough to ensure in general the pools and their
members get partitioned together. One execption to this is CreateFork.
The created fork should be placed in the same partition as the origin
repository as they'll both eventually be connected to the same pool.
CreateFork does not tag the source repository as an additional
repository so the general handling does not apply for it. Tagging it
as the additional repository won't work with Praefect as Praefect
rewrites the paths of additional repositories. The additional repository
is fetched through the API so it needs to have its original relative
path intact.
This commit introduces special handling for CreateFork. The transaction
middleware checks whether the request is a CreateForkRequest. If so,
the source repository is extracted and the newly created fork gets
partitioned with it.
Along with the behavior changes, we add a test that exercises the
entire fork creation flow as typically done. Turns out Gitaly didn't
have a test covering the scenario at all.
|
|
|
|
Unify transaction manager's references assertion
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6592
Merged-by: James Liu <jliu@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
In some prior commits, we added pack-refs housekeeping task support to
the transaction manager. We introduced a new assertion helper to verify
the content of packed-refs file. That helper reads the content and
compares with the expected packed-refs text. That approach makes the
test setup hard because the expected packed-refs must match the actual
content of packed-refs on disk, line by line.
This commit enhances that helper. It now parses the content of
packed-refs file. The expected value becomes a map of ref name to its
object ID.
|
|
|
|
smarthttp: Fix upload-pack using bundle-URI with missing backup
Closes #5740
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6591
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>
|
|
|
|
featureflag: Remove `AtomicFetchRemote`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6605
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
|
|
When `bundleuri.UploadPackGitConfig()` returns an empty slice, we see
the `PostUploadPackWithSidechannel` RPC fail with `FailedPrecondition`.
The git-upload-pack(1) process started here quits with the error:
fatal: invalid command 'bundle-uri'
Looking at the source code of Git, we see when git-upload-pack(1)
receives the 'bundle-uri' command, it checks if the that capability
should be advertised. To query for this, Git checks if the config
`uploadpack.advertiseBundleURIs` is set to `true`. When Gitaly could not
build a bundle URI to pass to the git-upload-pack(1), this config was
not set, and Git did not accept this command.
For the SmartHTTP protocol Gitaly uses separate processes to handle
'GET info/refs' and 'POST upload-pack', using option `--stateless-rpc`.
Due to this, the server advertised to the client it supports
'bundle-uri' in the 'GET info/refs' response, but in the consecutive
requests we might spawn git-upload-pack(1) without this config, and Git
will not accept the 'bundle-uri' command.
Inject the config `uploadpack.advertiseBundleURIs=true` into all calls
to git-upload-pack(1), even when the bundle-URI cannot be built.
Label: bug::availability
|
|
The `AtomicFetchRemote` feature flag enables atomic fetches for a
repository. Since the feature flag is default enabled, remove the flag.
|
|
This reverts commit 7de65fabe6eaf803771cf0dd5d53dbaf1e628d56, reversing
changes made to 0ffc495be9bb5a8f4ff60764b545443d54210e56.
|
|
This reverts commit 7f019337fb51305215538a793cf80171111b6e29, reversing
changes made to 82dcc6a7a5692244932c4f6d4d92b3fa615aeb85.
|
|
commit: Fix bug in commit signature parsing
See merge request https://gitlab.com/gitlab-org/security/gitaly/-/merge_requests/92
Merged-by: Jenny Kim <yjeankim@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
|
'master'
Add pack-refs housekeeping task support to the transaction manager
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6551
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
backup: Only delete repositories missing from a restore
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6579
Merged-by: James Liu <jliu@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
|
|
Report stat failures correctly when creating a repository
Closes #5751
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6597
Merged-by: James Liu <jliu@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
Default enable CLONE_INTO_CGROUP support
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6598
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
Adds a handler to Praefect to intercept calls to the WalkRepos RPC. The
handler provides an alternate implementation of listing repositories in
a storage, which queries the Praefect DB rather than walking the
filesystem on disk. This is required so when the RPC is invoked via
Praefect, the DB is used as the source of truth rather than a random
Gitaly node.
The only user-facing difference between this and the original
implementation is that the `modification_time` attribute of the response
message is left empty, as this cannot be determined via the DB.
|
|
Now that we've adjusted the restore mechanism to delete individual
repos as needed, this RPC is no longer required. See the following
issues for more context:
- https://gitlab.com/gitlab-org/gitaly/-/issues/5357
- https://gitlab.com/gitlab-org/gitaly/-/issues/5269
Changelog: deprecated
|
|
This is now deprecated in favour of removing individual repos.
|
|
Instead of invoking the RemoveAll() RPC prior to the restore, we instead
compare the set of existing repositories to the set of repositories
contained in the backup being restored. The difference between the two
sets -- the set of "dangling" repositories -- are removed individually.
This is done to ensure the state of repos in Gitaly matches the
worldview held by the Rails DB after a GitLab instance restore.
|
|
Modifies the Restore CLI tests so we stop sending an invalid restore
command as the final JSON object into stdin. This is required as a
subsequent commit will move the repository removal logic to execute
after the Pipeline completes successfully. If the tests purposely cause
the pipeline to fail, the restore logic will never execute.
Coverage of the Pipeline's error messaging is covered separately in the
Pipeline's unit tests.
|
|
Adds a new method to the Strategy interface used by regular and
server-side backups for performing repository backups and restores. This
new method calls the internal WalkRepos() RPC to fetch a list of repos
in a given storage.
|
|
Adds a new method to the Strategy interface used by regular and
server-side backups for performing repository backups and restores. This
new method removes a single repository from its storage, and will
eventually replace the existing RemoveAllRepositories method.
|
