| Age | Commit message (Collapse) | Author |
|---|
|
The object pool service RPCs carry an additional repository along
with the target repository. This repository lies on the same storage
as the target repository. The additional repository's relative path
needs to also be rewritten by Praefect with the replica path as
it would have been when the additional repository was originally
created.
Praefect currently does so for other mutator RPCs than 'create' type
as there are no 'create' type RPCs yet that contain an additional
repository. CreateObjectPool is incorrectly not marked as 'create'
type call though. In preparation for fixing that, this commit already
introduces relative path rewriting for additional repositories in
'create' type RPCs.
|
|
Disable Praefect in hook service tests
See merge request gitlab-org/gitaly!4319
|
|
|
|
[ci skip]
|
|
[ci skip]
|
|
[ci skip]
|
|
Disable metadata creation hack in commit service tests
See merge request gitlab-org/gitaly!4307
|
|
HookService is called by gitaly-hooks to execute hook logic. Right
now, the tests are ran with Praefect in front of the Gitaly. In
production, Praefect never proxies HookService calls as HookService
is strictly for Gitaly's internal use. This commit disables Praefect
in the tests of HookService.
|
|
setupHookService helper is currently using a TODO context. This commit
propagates the test context down from the tests to the setup helper.
|
|
Hook service is currently creating repositories using test helpers
and not via the API. This commit converts majority of the tests to
create the repos through the API so their state matches better how
the repos will actually be in production.
TestPreReceiveHook_GitlabAPIAccess was not converted to create the
repository through the API as it relies on some setup prior to the
repository being created.
|
|
Disable metadata creation hack in repository service tests
See merge request gitlab-org/gitaly!4275
|
|
setupRepositoryService is currently using a TODO context. This commit
propagates the test context down from the tests to the setup helper.
|
|
This commit disables metadata force creation in repository service
tests when it is ran with Praefect. This ensures any problems in
Praefect are not hidden by the hack. Disabling the hack surfaces few
behavior deviations with differing error messages.
The RenameRepository tests do not work with Praefect without the
metadata creation hack. This is due to a bug in Praefect where the
rename still gets applied even if the Gitalys failed the rename.
As this is an existing bug, the test is skipped for now with a link
to the tracking issue.
|
|
This commits converst CreateFork_successful to create the repository
via API. The test has some custom server setup code so this was left
in a separate commit for clarity.
|
|
Some tests perform custom connection setup, for example when TLS is
used. To avoid complicating the general case of CreateRepository, this
commit adds an option to provide a pre-established ClientConn. This way
the test can perform whatever it needs to do to setup a connection and
CreateRepository doesn't need to get more complicated.
|
|
This commits converts FetchRemote to also create repositories via
API. The tests were somewhat simplified by removing a method to copy
the test repository and instead initiate another copy of the test repo
using the existing helpers. As TestFetchRemote_sshCommand is overriding
the git command, it's not possible to create repositories via the API.
Thus, the test is skipped when Praefect is enabled.
|
|
This commit updates the FindLicense's test to create the repository
via API. As a special note, the branch had to be changed to main as
Gitaly nowadays creates repositories with main as default branch.
|
|
This commit creates the repositories in tests of repsository service
via API. This enables the tests to work with Praefect without the
metadata creation hack as Praefect can create the metadata when is
proxies the creation calls. The tests converted in this commit are the
ones where it was straighforward replacing, the rest will be done in
more detailed follow up commits.
|
|
This commit sets up the test repository via API in repository service
tests. This paves the way for disabling the Praefect's metadata creating
hack in the tests.
TestRemoveRepository_locking was updated to delete the lock after
the test so the repository creation helpers can delete the created
repository after the test.
|
|
Some tests modify the repository that is returned from CreateRepository.
To allow for the clean up function to still succeed, this commit returns
a copy of the repository instead so the clean up still targets the correct
repository.
|
|
Disable metadata creation hack in ref service tests
See merge request gitlab-org/gitaly!4281
|
|
Create repositories via API in conflicts service tests
See merge request gitlab-org/gitaly!4276
|
|
fetch: Stop writing FETCH_HEAD
See merge request gitlab-org/gitaly!4305
|
|
Add stderr to errors returned by SSHUploadPack
See merge request gitlab-org/gitaly!4290
|
|
limithandler: enable max queue wait time
See merge request gitlab-org/gitaly!4271
|
|
config: Exclude DB metrics by default
See merge request gitlab-org/gitaly!4300
|
|
Test setup helpers are currently using a TODO context. This commit
propagates the test context down from the tests to the setup helper.
|
|
This commit amends the diff service's tests to not rely on Praefect's
metadata force creation hack. The hack creates metadata for repositories
when Praefect first sees a request for the repository. This has potential
to hide problems in tests and doesn't match the production scenario. To
prepare for removing the hack entirely, this commit refactors the tests
to create the test repositories through the API rather than placing them
directly in the storage.
In some tests, the tested branch was changed from 'master' to 'main' as
Gitaly creates repositories with 'main' as the default branch.
|
|
createRepoWithDivergentBranches is using a worktree to build its
test state. This is unnecessarily complex and makes it slightly more
difficult to create the repository through the API in the helper.
This commit refactors the helper to setup the test state in a bare
repository. createCommits is extracted from the function as an
independent helper so it can be reused across tests. createCommits is
then used also in TestSuccessfulCountComitsRequest to replace its custom
test setup.
|
|
Commit service's package contains helpers for working with tree
entries. These helpers are used also in the repository service.
This creates a cyclic dependency when repository service is setup
in commit service's tests to create repositories via the API. This
commit moves the tree entry helpers out of the commit package so the
cyclic dependency is broken.
|
|
setupRefService is currently using a TODO context. This commit
propagates the test context down from the tests to the setup helper.
|
|
This commit amends the ref service's tests to not rely on Praefect's
metadata force creation hack. The hack creates metadata for repositories
when Praefect first sees a request for the repository. This has potential
to hide problems in tests and doesn't match the production scenario. To
prepare for removing the hack entirely, this commit refactors the tests
to create the test repositories through the API rather than placing them
directly in the storage.
RepositoryService is now set up and registered in the test setup as it's
the service that is responsible for repository creations.
The tests themselves are mostly just updated to use the helpers that use
the API. In TestServer_ListRefs, the branch name was changed from master
to main as Gitaly creates repositories with main as the default branch.
|
|
SetupConflictsService is currently using a TODO context. This commit
propagates the test context down from the tests to the setup helper.
|
|
This commit changes the conflicts service to create its test repositories
via the API rather than placing them directly to the storage. This allows
Praefect to create the repository metadata records when proxying the calls
if the test are ran with Praefect enabled. With this fixed, the metadata
force creation hack is also disabled for the tests.
|
|
The other tests in the conflicts package use the same setup function to create the
test service. TestSuccessfulResolveConflictsRequestHelper however uses different
setup as it needs to know commit IDs before it can setup the mock hook manager. This
commit refactors the test to use the common setup functions. This allows for later
creating the test repository via the API.
|
|
repository: Optimize reference negotiation for large repositories
See merge request gitlab-org/gitaly!4287
|
|
When fetching into a repository, git-fetch(1) by default writes the
FETCH_HEAD file containing all references which have been updated. While
this isn't much of a problem in "normal-sized" repositories, the file
may easily be hundreds of megabytes in size when mirror-fetching from
repositories with millions of references. Given that we ultimately don't
use that file anywhere this is a big waste of resources.
In "www-gitlab-com", the FETCH_HEAD file is about 379MB in size after
performing a mirror-fetch against another replica. The following shows
the difference with and without writing FETCH_HEAD:
Benchmark 1: git fetch replica refs/*:refs/*
Time (mean ± σ): 28.292 s ± 0.122 s [User: 25.561 s, System: 3.787 s]
Range (min … max): 28.159 s … 28.398 s 3 runs
Benchmark 2: git fetch --no-write-fetch-head replica refs/*:refs/*
Time (mean ± σ): 27.820 s ± 0.064 s [User: 25.068 s, System: 3.650 s]
Range (min … max): 27.758 s … 27.885 s 3 runs
Summary
'git fetch --no-write-fetch-head replica refs/*:refs/*' ran
1.02 ± 0.00 times faster than 'git fetch replica refs/*:refs/*'
While the speed improvement is surprisingly small, it is still the right
thing to do: we don't read FETCH_HEAD anything, and Git itself doesn't
use its information for anything either without us asking for it.
Changelog: changed
|
|
Extend tests which use git-fetch(1) to demonstrate that we're writing
the FETCH_HEAD file.
|
|
Some of our remote tests use the FETCH_HEAD file to determine whether
expected references have been fetched. We're about to stop writing that
file though, which will break these tests.
Refactor them to use `localrepo.GetReferences()` to determine whether
refs match after the fetch.
|
|
The testhelper function `MustReadFile()` is not marked as test helper,
and as a result test failures point to that function instead of the call
site.
Fix this by calling `t.Helper()`.
|
|
localrepo: Use protocol v2 for internal fetches
See merge request gitlab-org/gitaly!4303
|
|
The extremely costly query executed by the read-only repository
Prometheus metric regularly causes customer outages after upgrades
to v14.3+. The addition of
`PrometheusExcludeDatabaseFromDefaultMetrics` allows admins to avoid
this problem, but is not enabled by default.
This commit enables `PrometheusExcludeDatabaseFromDefaultMetrics` by
default to help prevent future outages.
Changelog: changed
|
|
Remove obsolete info about hooks from docs
See merge request gitlab-org/gitaly!4302
|
|
Disable metadata creation hack in operation service tests
See merge request gitlab-org/gitaly!4272
|
|
demo: Fix deletion of Praefect Cloud SQL instance
See merge request gitlab-org/gitaly!4299
|
|
When spawning git-upload-pack(1) we need to splice through GIT_PROTOCOL
such that we can tell which Git protocol shall be used. Right now, we
support both protocol v0, which is the default protocol, and protocol
v2, which is a newer iteration of this protocol. Wherever possible we
want to use v2, which provides some benefits for us like being able to
skip the initial reference advertisement phase in case we're only
fetching a subset of references.
While we have the setup in place to pass through the protocol version
for the SSH-based transport, we do not currently use it when doing
internal fetches. As a result, we fall back to v0 of the protocol and
thus miss out on any potentially new features.
Fix this by requesting protocol v2 for internal fetches.
Changelog: performance
|
|
Praefect piggybacks on Gitalys tests for testing the various RPCs.
The tests are simply run with Praefect in front of the Gitaly. This
worked fine before Praefect became stricter about consistency of the
operations and began requiring database records of the repositories.
The database records are updated when Praefect proxies the various
RPCs to the repository. Gitalys tests generally do not create the test
repositories via the API but place the directly in the storage. This
does not work with Praefect as Praefect won't see the changes and won't
be able to update its metadata. A hack was put in place in Praefect to
create the metadata in the tests when Praefect receives a request for
a repository. This is not ideal as it can hide bugs in Praefect.
As all of the relevant operation service tests have now been amended to
create their test repositories via the API, this commit disables the
metadata creation hack for operation service tests.
|
|
This commit creates the test repository via API in TestUserApplyPatch.
This change uncovered a small behavior difference in Praefect and Gitaly.
Now that Praefect is not using the metadata creation hack anymore, we can
see that Praefect's error message for non-existent repositories deviates
from Gitaly's. This is left as is since its been like this for a while
already with no apparent problems.
|
|
This commit amends most of the operations service's test to create the
test state through the API. This better matches the production scenario
and allows for Praefect to create the metadata it would create in real
setups. Most of the replacements are straight forward but some tests rely
on working trees. As Gitaly does not support creating worktrees through the
API, the worktrees are created after creating the repo through the API.
TestUserCommitFiles was simplified a bit and its branch was changed as new
repositories nowadays have main as their default branch.
|
|
This commit changes setupOperationsServiceWithCfg to create the test
repository through the API. This ensures Praefect has the chance to
create the relevant metadata for the test repository.
TestUserApplyPatchTransactional had to be amended as it counts votes
and creating the repo in the setup creates some votes.
|
