| Age | Commit message (Collapse) | Author |
|---|
|
tools/protolint: update package to latest
Closes #5749
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6632
Merged-by: John Cai <jcai@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
|
fix: deterministic gpg signature
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6621
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Adrien <adrien@huggingface.co>
|
|
Since the gpg signature is time-dependent, aligning the timestamp with the
author's date makes the gpg signature consistent across all gitaly nodes.
|
|
Update the "protolint" package from v0.46.1 to v0.47.5. This updates most
of the dependencies, including "gopkg.in/yaml". The older
"gopkg.in/yaml" package, below v2.2.4 [1], contained a vulnerability [2]
which is now removed.
[1]: https://github.com/go-yaml/yaml/releases/tag/v2.2.4
[2]: https://pkg.go.dev/vuln/GO-2022-0956
|
|
|
|
cleanup: Add RewriteHistory RPC
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6615
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
|
|
git: Use Git version 2.43 only
Closes #5739
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6624
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Eric Ju <eju@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
Enable transactions for the new RewriteHistory RPC.
|
|
On large repositories git-filter-repo(1) make take a significant amount
of time to run. Should a write occur after the git-fast-export(1)
portion of the task has completed, it is possible that the repository
history will not be fully rewritten.
To guard against this condition, we checksum the repository before and
after running filter-repo. If the checksums do not match we abort and do
not fetch the updated history into the repository.
|
|
git-filter-repo(1) uses git-fast-import(1) to import the rewritten
repository history. This will unpack the new objects, then iterate over
references serially and update them using reference transactions. This
does not atomically update the references[0], so an interruption during
this final stage will result in partially applied changes.
To mitigate this risk, create a temporary staging repository to write
the updated history into, then atomically force fetch that into the
original repo.
This has the downside of being slower than modifying the repository
in-place[1], but improving safety for a high-risk operation like this
is a greater priority.
[0] https://gitlab.com/gitlab-org/git/-/blob/d4dbce1db5cd227a57074bcfc7ec9f0655961bba/builtin/fast-import.c#L1659-1668
[1] https://github.com/newren/git-filter-repo/issues/66#issuecomment-602100316
|
|
Historically we have advised users who need to rewrite history to do so
locally and force push their change to Gitlab. However, upcoming changes
may prevent a user from pushing in scenarios where they need to remove a
large blob from their repository's history.
To handle this scenario, we introduce a new `RewriteHistory` RPC which
will invoke git-filer-repo(1) on the target repository. filter-repo
has a large number of options, but we will support only two:
--strip-blogs-with-ids
Given a file containing a list of newline-delimited object ids,
rewrite history to remove them from all commits.
--replace-text
Given a file of literals and patterns, replace all matching
instances in history with '***REMOVED***'.
filter-repo works by fetching the repository contents via
git-fast-export(1), making the requested changes, and writing the
changes back via git-fast-import(1). As filter-repo uses the '--force'
flag[0] the repository must be made read-only before calling this RPC.
filter-repo is currently incompatible with SHA256 repositories.
[0] https://git-scm.com/docs/git-fast-import#_parallel_operation
Changelog: added
|
|
Add git-filter-repo(1) as a recognized subcommand.
|
|
The ubi image used for FIPS testing does not have Python3 installed by
default. Install it as part of the `before_script` steps.
|
|
We will shortly begin using git-filter-repo(1) to rewrite repository
history. This is a Python script that requires Python 3.5+, but it has
no external dependencies and can be downloaded as a single file.
We have added Python3 and installed git-filter-repo in Omnibus and CNG
GitLab, but still need it to be present for local testing and CI.
Add a target to clone filter-repo as a dependency and copy the script
into ${BUILD_DIR}/bin which is added to PATH for our tests.
We add a '.version' file for filter-repo to be consistent with the other
dependencies, but this isn't really required as we're not running a
build process in its source directory.
|
|
backup: Check for backup before deleting repo
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6620
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>
|
|
In previous commit we've dropped all use of Git v2.42, so now we can
stop building it.
|
|
Remove the feature flag to toggle the use of v2.43, enable it by
default, and remove the use of v2.42.
Label: maintenance::dependency
|
|
.gitattributes: Mark .pb.go files as generated
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6623
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
|
|
To make code reviews easier, mark "*.pb.go" files as generated, which
makes them collapsed by default in GitLab. See:
https://docs.gitlab.com/ee/user/project/merge_requests/changes.html#collapse-generated-files
|
|
'master'
Create new hook to export git trace 2 events in gitaly logs
Closes #5700
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6570
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Emily Chui <echui@gitlab.com>
Co-authored-by: Emily Chui <echui@gitlab.com>
|
|
|
|
.gitlab-ci.yml: start rails spec and cleanup automatically
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6618
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Lin Jen-Shin <jen-shin@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: John Cai <jcai@gitlab.com>
|
|
|
|
Reorders the repository restore logic so that we do not remove the repo
until we've checked that it has an existing backup. If the repo has no
backup, it's skipped from the restore and will be removed later by the
caller.
The previous order of operations caused issues when performing a full
restore that included a dangling repo (a repo created after the backup
was taken). When the `Restore` function was executed against this repo,
it was removed immediately. Then, the remainder of the restore was
skipped since no backup existed for that repo. Since the repo was not
marked as restored, the caller of the `Restore` function tried to remove
it once again, leading to a "repository not found" error since it had
already been erased.
The "missing backup" test case for the restore of a specific backup has
been updated to expect that the repo exists to align with this logic
|
|
Write latest manifest file
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6613
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: James Fargher <jfargher@gitlab.com>
|
|
Partition fork with source repository in CreateFork
Closes #5762
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6612
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
fix: Fix a collection of typos found by typos-cli
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6594
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Evan Read <eread@gitlab.com>
Co-authored-by: Xing Xin <xingxin.xx@bytedance.com>
|
|
Fix typos found by typos-cli(https://github.com/crate-ci/typos). Some
affected tests are adjusted.
There are a bunch of other typos are ignored, including
* CHANGELOG.md
* NOTICE
* internal/.../migrations/20201208163237_cleanup_notifications_payload.go
* other intended typos or false positives
Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
|
|
[ci skip]
|
|
backup: Track repos that have been processed (re-attempt)
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6614
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>
|
|
Now that backups no longer invoke the RemoveAll RPC, we can remove the
exemption for these tests when the WAL is enabled.
|
|
Adds a handler to Praefect to intercept calls to the WalkRepos RPC. The
handler provides an alternate implementation of listing repositories in
a storage, which queries the Praefect DB rather than walking the
filesystem on disk. This is required so when the RPC is invoked via
Praefect, the DB is used as the source of truth rather than a random
Gitaly node.
The only user-facing difference between this and the original
implementation is that the `modification_time` attribute of the response
message is left empty, as this cannot be determined via the DB.
|
|
Now that we've adjusted the restore mechanism to delete individual
repos as needed, this RPC is no longer required. See the following
issues for more context:
- https://gitlab.com/gitlab-org/gitaly/-/issues/5357
- https://gitlab.com/gitlab-org/gitaly/-/issues/5269
Changelog: deprecated
|
|
This is now deprecated in favour of removing individual repos.
|
|
Instead of invoking the RemoveAll() RPC prior to the restore, we instead
compare the set of existing repositories to the set of repositories
contained in the backup being restored. The difference between the two
sets -- the set of "dangling" repositories -- are removed individually.
This is done to ensure the state of repos in Gitaly matches the
worldview held by the Rails DB after a GitLab instance restore.
Also fixes the existing tests so that all repositories are created with
`gittest.CreateRepository` and are thus visible when we later query
`ListRepositories` in the restore logic.
|
|
Do not pass Gitaly variables to Rails pipeline
Closes #5743
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6617
Merged-by: John Cai <jcai@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Co-authored-by: Lin Jen-Shin <jen-shin@gitlab.com>
|
|
|
|
Now that a latest manifest is written, we can use this manifest when
loading the "latest" backup.
|
|
Manifests currently cannot restore the "latest" backup because it would
require an expensive object-storage directory traversal. So instead it
defers to the pointer layout. The problem with this is that you then
loose manifest only features like setting the default branch properly.
Here we write two manifests: the normal backup manifest as before and an
additional "latest" manifest file. This latest manifest is overwritten
on each backup taken.
For WORM we would ideally not overwrite any files, but until this is
implemented we need something to fill the latest restore gap.
Changelog: changed
|
|
Modifies the Restore CLI tests so we stop sending an invalid restore
command as the final JSON object into stdin. This is required as a
subsequent commit will move the repository removal logic to execute
after the Pipeline completes successfully. If the tests purposely cause
the pipeline to fail, the restore logic will never execute.
Coverage of the Pipeline's error messaging is covered separately in the
Pipeline's unit tests.
|
|
Adds a new method to the Strategy interface used by regular and
server-side backups for performing repository backups and restores. This
new method calls the internal WalkRepos() RPC to fetch a list of repos
in a given storage.
|
|
Adds a new method to the Strategy interface used by regular and
server-side backups for performing repository backups and restores. This
new method removes a single repository from its storage, and will
eventually replace the existing RemoveAllRepositories method.
|
|
Adds a map to the Pipeline to track repos that have been restored or
backed up. A mutex is used to synchronise access to the map, as entries
are appended by goroutines operating in the workers.
The signature of Done() is modified to return the map, and is
intentionally ignored in the actual backup and restore logic for now. A
subsequent commit will utilise the map for restore operations.
|
|
Gitaly's TransactionManager requires all object pool members to be
in the same partition. Currently we're ensuring doing that generally
by extracting the additional repository from the request and ensuring
the target repository of the RPC gets partitioned with it. Additional
repository is used in the ObjectPoolService's RPCs to tag the other
repository being accessed in the pool related operations, and it may
be either the object pool or one of the member repositories depending
on the RPC.
When a repository is first accessed, we also check whether it has an
existing alternate link on the disk, and place the repository in the
same partition with its alternate if so.
These two methods are enough to ensure in general the pools and their
members get partitioned together. One execption to this is CreateFork.
The created fork should be placed in the same partition as the origin
repository as they'll both eventually be connected to the same pool.
CreateFork does not tag the source repository as an additional
repository so the general handling does not apply for it. Tagging it
as the additional repository won't work with Praefect as Praefect
rewrites the paths of additional repositories. The additional repository
is fetched through the API so it needs to have its original relative
path intact.
This commit introduces special handling for CreateFork. The transaction
middleware checks whether the request is a CreateForkRequest. If so,
the source repository is extracted and the newly created fork gets
partitioned with it.
Along with the behavior changes, we add a test that exercises the
entire fork creation flow as typically done. Turns out Gitaly didn't
have a test covering the scenario at all.
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
|
|
|
|
Unify transaction manager's references assertion
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6592
Merged-by: James Liu <jliu@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
