# Gitaly Team Process ## Feature flags Gitaly uses feature flags to safely roll out features in production. Feature flags are part of the `context.Context` of each RPC. The `featureflag` package will help you with flow control. Most of this documentation assumes operations on `gitlab.com`. For customers, an [HTTP API is available][ff-api]. In order to roll out feature flags to `gitlab.com`, you should follow the documented rollout process below. Once you have [developed your feature][#development-with-feature-flags] you [start by creating an issue for the rollout][issue-for-feature-rollout]. The "Feature Flag Roll Out" [template for the issue][feature-issue-template] has a checklist for the rest of the steps. [ff-api]: https://docs.gitlab.com/ee/api/features.html#features-flags-api [issue-for-feature-rollout]: https://gitlab.com/gitlab-org/gitaly/-/issues/new?issuable_template=Feature%20Flag%20Roll%20Out [feature-issue-template]: https://gitlab.com/gitlab-org/gitaly/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md ### Development with feature flags It's quite common that a change in Gitaly needs some changes in the clients, especially GitLab Rails. There is a clear distinction between using feature flags in GitLab Rails and Gitaly. #### Feature flags in GitLab Rails GitLab defines a very clear and strict [workflow for development][rails-feature-development]. Because the flag evaluation is invoked in Rails, we have the flexibility of picking the desired actors for a feature flag. [rails-feature-development]: https://docs.gitlab.com/ee/development/feature_flags/index.html #### Feature flags in Gitaly Feature flag definitions and their feature gates are managed by GitLab Rails. Gitaly doesn't have direct access to feature flags. Instead, Rails is responsible for pre-evaluating feature flags relevant to Gitaly and propagating the evaluation result to Gitaly through gRPC metadata. Other Gitaly consumers, such as GitLab Shell or Workhorse, follow the same mechanism. They propagate the flag evaluation extracted from responses of internal authentication APIs. After Gitaly receives the flags, it continues to pass feature flags between its internal components using either: - Environment variables. - Its gRPC calls. ```mermaid graph TD GL[GitLab Rails] GS[GitLab Shell] W[Workhorse] G[Gitaly]:::Gitaly GH[Gitaly Hooks] W -- extract features from auth APIs --> GL GS -- extract features from auth APIs --> GL GS -- "RPC(gitaly-feature-a: true)" --> G W -- "RPC(gitaly-feature-a: true)" --> G GL -- "RPC(gitaly-feature-a: true)" --> G G -- Encode flags to env --> GH GH -- "RPC(gitaly-feature-a: true)" --> G G -- "RPC(gitaly-feature-a: true)" --> G classDef Gitaly fill:#f96; ``` > WARNING: This architecture imposes some limitations on how Gitaly uses feature flags > > - All Gitaly feature flags must start with the `gitaly_` prefix. Otherwise, they are > not propagated to Gitaly. This prefix is stripped when used in Gitaly internally. > - Feature flags are not available if an operation does not contact Rails. It > means Gitaly background jobs, such as repository maintenance, cannot use feature flags. To check feature flags in the source code, that flag must be defined in [internal/metadatta/featureflag][gitaly-featureflag-folder] folder. For example: ```go package featureflag // GoFindLicense enables Go implementation of FindLicense var GoFindLicense = NewFeatureFlag( // Snake-cased name of the flag. This is the name used to control the flag // via chatops or admin API. The `gitaly_` prefix is stripped. "go_find_license", // Target version of the flag "v14.3.0", // Rollout Issue "https://gitlab.com/gitlab-org/gitaly/-/issues/3759", // Default value false, ) ``` The flag definition is exported and used to evaluate the flag. The evaluation is then extracted from the context (essentially the gRPC context). One important note here. As feature flags may be turned on/off randomly, the code should work well in both states. The flag should not introduce one-way data transformation that makes it impossible to switch back. In addition, Gitaly should be backward-compatible with the consumers regardless of the current flag state. ```go func (s *server) FindLicense(ctx context.Context, req *gitalypb.FindLicenseRequest) (*gitalypb.FindLicenseResponse, error) { // Blah blah if featureflag.GoFindLicense.IsEnabled(ctx) { } } ``` In the test suite, it's recommended to use the [NewFeatureSets][test-featureset] helper. This helper repeats the test twice for each on/off state of the flag. ```go func TestSuccessfulFindLicenseRequest(t *testing.T) { testhelper.NewFeatureSets(featureflag.GoFindLicense).Run(t, func (t *testing.T, ctx context.Context) { if featureflag.GoFindLicense.IsEnabled(ctx) { // Test the enabled case } else { // Test the disabled case } }) } ``` After the code associated with the flag is deployed, we now can turn on the flag using chatops. Please note the `gitaly_` prefix. The rollout process is described in detail in the following sections. ```shell /chatops run feature set gitaly_go_find_license true --staging ``` [gitaly-featureflag-folder]: https://gitlab.com/gitlab-org/gitaly/-/tree/master/internal/featureflag [test-featureset]: https://gitlab.com/gitlab-org/gitaly/blob/c6fb49b9c6e854c0a2803f53106af501d6006cb8/internal/testhelper/featureset.go#L55-55 ### Flag management with chatops Feature flags are [enabled through chatops][enable-flags] (which is just a consumer [of the API][ff-api]). In [`#chat-ops-test`][chan-chat-ops-test] try: ```shell /chatops run feature list --match gitaly_ ``` If you get a permission error you need to request access first. That can be done [in the `#production` channel][production-request-acl]. For Gitaly, you have to prepend `gitaly_` to your feature flag when enabling or disabling. For example: to check if [`gitaly_go_user_delete_tag`][chan-production] is enabled on staging run: ```shell /chatops run feature get gitaly_go_user_delete_tag --staging ``` [enable-flags]: https://docs.gitlab.com/ee/development/feature_flags/controls.html [chan-chat-ops-test]: https://gitlab.slack.com/archives/CB2S7NNDP [production-request-acl]: https://gitlab.slack.com/archives/C101F3796 [chan-production]: https://gitlab.com/gitlab-org/gitaly/-/issues/3371 ## Feature flags issue checklist The rest of this section is help for the individual checklist steps in [the issue template][feature-issue-template]. If this is your first time doing this you might want to first skip ahead to the help below, you'll likely need to file some access requests. ### Feature flag labels The lifecycle of feature flags is monitored via issue labels. When the issue is created from a template it'll be created with [`featureflag::disabled`][featureflag-disabled]. Then as part of the checklist the person rolling it out will add [`featureflag::staging`][featureflag-staging] and [`featureflag::production`][featureflag-production] flags to it. [featureflag-disabled]: https://gitlab.com/gitlab-org/gitaly/-/issues?label_name[]=featureflag%3A%3Adisabled [featureflag-staging]: https://gitlab.com/gitlab-org/gitaly/-/issues?label_name[]=featureflag%3A%3Astaging [featureflag-production]: https://gitlab.com/gitlab-org/gitaly/-/issues?label_name[]=featureflag%3A%3Aproduction ### Is the required code deployed? A quick way to see if your MR is deployed is to check if [the release bot][release-bot] has deployed it to staging, canary or production by checking if the MR has [a `workflow::staging`][deployed-staging], [`workflow::canary`][deployed-canary] or [`workflow::production`][deployed-production] label. The [/help action on GitLab.com][help-action] shows the currently deployed hash. Copy that `HASH` and look at `GITALY_SERVER_VERSION` in [`gitlab-org/gitlab.git`][gitlab-git] to see what the embedded Gitaly version is. Or in [a `gitaly.git` checkout][gitaly-git] run this to see what commits aren't deployed yet: ```shell git fetch git shortlog $(curl -s https://gitlab.com/gitlab-org/gitlab/-/raw/HASH/GITALY_SERVER_VERSION)..origin/master ``` See the [documentation on releases below](#gitaly-releases) for more details on the tagging and release process. [release-bot]: https://gitlab.com/gitlab-release-tools-bot [deployed-staging]: https://gitlab.com/gitlab-org/gitaly/-/merge_requests?state=merged&label_name=workflow%3A%3Aproduction [deployed-canary]: https://gitlab.com/gitlab-org/gitaly/-/merge_requests?state=merged&label_name=workflow%3A%3Aproduction [deployed-production]: https://gitlab.com/gitlab-org/gitaly/-/merge_requests?state=merged&label_name=workflow%3A%3Aproduction [help-action]: https://gitlab.com/help [gitlab-git]: https://gitlab.com/gitlab-org/gitlab/ [gitaly-git]: https://gitlab.com/gitlab-org/gitaly/ ### Enable on staging #### Prerequisites You'll need chatops access. See [ChatOps on GitLab.com][chatops-access]. [chatops-access]: https://docs.gitlab.com/ee/development/chatops_on_gitlabcom.html#requesting-access #### Steps Run: `/chatops run feature set gitaly_X true --staging` Where `X` is the name of your feature. ### Test on staging #### Prerequisites Access to is not the same as on GitLab.com (or signing in with Google on the `@gitlab.com` account). You must [request access to it][staging-access-request]. As of December 2020 clicking "Sign in" on will redirect to , so make sure to use the `/users` link. As of writing signing in at [that link][staging-users-link] will land you on the `/users` 404 page once you're logged in. You should then typically manually modify the URL `https://staging.gitlab.com/YOURUSER` (e.g. ) or another way to get at a test repository, and manually test from there. [staging-access-request]: https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/new?issuable_template=Individual_Bulk_Access_Request [staging-users-link]: https://staging.gitlab.com/users #### Steps Manually use the feature in whatever way exercises the code paths being enabled. Then enable `X` on staging, run the following in the `#staging` Slack channel: ```shell /chatops run feature set gitaly_X --staging ``` #### Discussion It's a good idea to run the feature for a full day on staging, this is because there are daily smoke tests that run daily in that environment. These are handled by [`gitlab-org/gitlab-qa.git`][gitlab-qa-git] [gitlab-qa-git]: https://gitlab.com/gitlab-org/gitlab-qa#how-do-we-use-it ### Enable in production #### Prerequisites Have you waited enough time with the feature running in the staging environment? Good! #### Steps Feature flags must be rolled out in production systems gradually to reduce risk of incidents. Use percentage rollout instead of enabling a feature flag fully. The concrete percentages depend on the scope of the feature flag as well as its inherent risk. In general, add more fine-grained steps the higher the risk and the broader the scope of the gated feature. The following chatops commands must run in the `#production` channel. After the [Improve feature flags in Gitaly][improve-flag-epic] epic is addressed, Gitaly will support various strategies for us to pick from. [improve-flag-epic]: https://gitlab.com/groups/gitlab-org/-/epics/8005 #### Rollout to a set of users It's very common that we want to perform testing on production without affecting our customers. Gitaly supports enabling a flag for a set of users by running the following command in chatops channels. ```shell /chatops run feature set --user=qmnguyen0711,toon gitaly_X ``` After enabling, the flag is turned on for the users listed when performing an action. One caveat of this strategy is that actions without authentication, such as cloning a public repository, are not affected. #### Rollout to a set of repositories Typically, Git repositories are the main objects that Gitaly takes care of. Nearly all operations in Gitaly is done inside the scope of Git repositories. Some repositories have a huge impact on a particular Gitaly node. So, it's natural to roll out the flags for a set of repositories, typically internal ones. ```shell /chatops run feature set --repository=gitlab-org/gitlab.git,gitlab-com/www-gitlab-com.git gitaly_X ``` The `--repository` flag supports all repository types, such as Wiki or Snippet. ```shell /chatops run feature set --repository=snippets/2427310.git gitaly_X /chatops run feature set --repository=gitlab-org.wiki.git gitaly_X ``` #### Rollout to a set of groups `gitlab-org` and `gitlab-com` are some of the largest groups on GitLab.com. It also makes a lot of sense for a feature to be visible to us for verification. Therefore, to enable a flag for a set of groups, run the following command in chatops channels: ```shell /chatops run feature set --group=gitlab-org,gitlab-com gitaly_X ``` #### Rollout to a percentage of repositories To enable your `X` feature at 5/25/50 percentage of repositories, run the following command in chatops channels. ```shell /chatops run feature set gitaly_X 5 --actors /chatops run feature set gitaly_X 25 --actors /chatops run feature set gitaly_X 50 --actors ``` Due to the pre-evaluation architecture, we could not choose the actor types for percentage-based rollout. The result of this strategy is persistent between evaluations of the same repository. That brings the predictability and ease of debugging if something goes wrong. #### Rollout to a percentage of time In some less common cases, we may need a fully-randomized percentage-base rollout. You need to use `--random` tag in chatops commands: ```shell /chatops run feature set gitaly_X 5 --random /chatops run feature set gitaly_X 25 --random /chatops run feature set gitaly_X 50 --random ``` #### Final steps And then finally when you're happy it works properly do: ```shell /chatops run feature set gitaly_X 100 ``` Followed by: ```shell /chatops run feature set gitaly_X true ``` Note that you need both the `100` and `true` as separate commands. See [the documentation on actor gates][actor-gates] If the feature is left at `50%` but is also set to `true` by default the `50%` will win, even if `OnByDefault: true` is [set for it](#feature-lifecycle-after-it-is-live). It'll only be 100% live once the feature flag code is deleted. So make sure you don't skip the `100%` step. [actor-gates]: https://docs.gitlab.com/ee/development/feature_flags/controls.html#process #### Discussion What percentages should you pick and how long should you wait? It makes sense to be aggressive about getting to 50% and then 100% as soon as possible. You should use lower percentages only as a paranoia check to make sure that it e.g. doesn't spew errors at users unexpectedly at a high rate, or (e.g. if it invokes a new expensive `git` command) doesn't create runaway load on our servers. But say running at 5% for hours after we've already had sufficient data to demonstrate that we won't be spewing errors or taking down the site just means you're delaying getting more data to be certain that it works properly. Nobody's better off if you wait 10 hours at 1% to get error data you could have waited 1 hour at 10% to get, or just over 10 minutes with close monitoring at 50%. ### Feature lifecycle after it is live #### Discussion After a feature is running at `100%` for what ever's deemed to be a safe amount of time we should change it to be `OnByDefault: true`. See [this MR for an example][example-on-by-default-mr]. We should add a changelog entry when `OnByDefault: true` is flipped. That should then be followed up by another MR to remove the pre-feature code from the codebase, and we should add another changelog entry when doing that. This is because even after setting `OnByDefault: true` users might still have opted to disable the new feature. See [the discussion below](#two-phase-ruby-to-go-rollouts) for possibly needing to do such changes over multiple releases. #### Two phase Ruby to Go rollouts Depending on what the feature does it may be bad to remove the `else` branch where we have the feature disabled at this point. E.g. if it's a rewrite of Ruby code in Go. As we deploy the Ruby code might be in the middle of auto-restarting, so we could remove its code before the Go code has a chance to update with its default, and would still want to call it. So therefore you need to do any such removal in two GitLab release cycles. See the example of [MR !3033][example-on-by-default-mr] and [MR !3056][example-post-go-ruby-code-removal-mr] for how to do such a two-phase removal. [example-on-by-default-mr]: https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3033 [example-post-go-ruby-code-removal-mr]: https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3056 #### Remove the feature flag via chatops After completing the above steps the feature flag should be deleted from the database of available features via `chatops`. If you don't do this others will continue to see the features with e.g.: ```shell /chatops run feature list --match=gitaly_ ``` It also incrementally adds to data that needs to be fetched & populated on every request. To remove the flag first sanity check that it's the feature you want, that it's at [`100%` and is `true`](#enable-in-production): ```shell /chatops run feature get gitaly_X ``` Then delete it if that's the data you're expecting: ```shell /chatops run feature delete gitaly_X ``` ## Git Version Upgrades With the introduction of [bundled Git][bundled-git] we have gained the ability to do feature-flag-based rollouts of new Git versions, and using feature flags for these upgrades has since then become mandatory. In the past, upgrades of the Git distribution simply happened in-place, where we just overwrote existing binaries with a new set of binaries. This has the potential to cause problems during zero-downtime upgrades: an old Gitaly process may be expecting a different Git version, or it may end up executing Git in an environment where some executables are the old version, where some other have already been replaced with the newer version. Naturally, this can lead to all kinds of problems. With the new process we thus do the upgrade in three separate steps to ensure that we have no such issues with zero-downtime upgrades: 1. We roll out the new bundled Git binaries in parallel to the old bundled Git binaries. The new version is guarded behind a feature flag at this point in time. 1. We roll out the feature flag and eventually remove it. 1. We remove the old bundled Git binaries. Note that because we cannot remove the old Git binaries at the same time when we add the new ones. We must ensure that both sets exist in parallel for at least one release. [bundled-git]: git-execution-environments.md#bundled-git-recommended ### Detailed Process The following detailed steps need to be done to upgrade to a new Git version: 1. Add the new bundled Git distribution to the `Makefile`. See c0d05650be681c2accb4cec5aac74a6dd77a2fa6. 1. Add a new bundled Git execution environment with a feature flag. See b547b368c8f584e9aabe8eef9342f99440b0c248. Please note that execution environments are ordered by decreasing priority: the first environment whose feature flags are all turned on will be picked. You thus have to add your new environment to the top. 1. Roll out the feature flag by following our feature flag process. You may decide to remove the feature flag before the feature flag is removed in case it is a low-risk upgrade of the Git version (e.g. when you perform a patch-release upgrade, only). 1. Remove the feature flag. See 888e6233fd85691f0852ae6c4a3656da9bf3d8e4. 1. Remove the execution environment of the old bundled Git version. See af1a3fe7b536d22a6db9ba6591d222b23d01d83f. 1. Remove the old set of bundled Git binaries from the `Makefile`. See 9c700ea473d781eea50eab685d643d95e9c4ffee. Note that this must only happen _after_ both old and new bundled Git binaries have been installed in parallel in a release already. ## Gitaly Releases Gitaly releases are tagged automatically by [`release-tools`][release-tools] when a Release Manager tags a GitLab version. [release-tools]: https://gitlab.com/gitlab-org/release-tools ### Major or minor releases Once we release GitLab X.Y.0, we also release Gitaly X.Y.0 based on the content of `GITALY_SERVER_VERSION`. This version file is automatically updated by `release-tools` during auto-deploy picking. Because Gitaly master is moving we need to take extra care of what we tag. Let's imagine a situation like this on `master` ```mermaid graph LR; A-->b0; A-->B; b0:::branch-->b1; b1:::branch-->B; B-->C; B-->c0; c0:::branch-->C; classDef branch fill:#f96; classDef tag fill:yellow; ``` Commit `C` is picked into auto-deploy and the build is successfully deployed to production We are ready to tag `v12.9.0` but there is a new merge commit, `D`, on Gitaly `master`. ```mermaid graph LR; A-->b0; A-->B; b0:::branch-->b1; b1:::branch-->B; B-->C; B-->c0; c0:::branch-->C; C-->D; C-->d0; d0:::branch-->D classDef branch fill:#f96; classDef tag fill:yellow; ``` We cannot tag on `D` as it never reached production. `release-tools` follows this algorithm: 1. create a stable branch from `GITALY_SERVER_VERSION` (commit `C`), 1. bump the version and 1. prepare the changelog (commit `C'`). Then we tag this commit and we merge back to `master` ```mermaid graph LR; A-->b0; A-->B; b0:::branch-->b1; b1:::branch-->B; B-->C; B-->c0; c0:::branch-->C; C-->D; C-->d0; d0:::branch-->D C-->C'; id1>v12.9.0]:::tag-->C'; D-->E; C':::stable-->E; classDef branch fill:#f96; classDef tag fill:yellow; classDef stable fill:green; ``` Legend ```mermaid graph TD; A["master commit"]; b0["feature branch commit"]:::branch; id1>tag]:::tag; C["stable branch commit"]:::stable; classDef branch fill:#f96; classDef tag fill:yellow; classDef stable fill:green; ``` With this solution, the team can autonomously tag any RC they like, but the other releases are handled by the GitLab tagging process. ### Patch releases The Gitaly team usually works on patch releases in the context of a security release. The release automation creates the stable branches, tagging the stable branch is automated in `release-tools` as well. A Gitaly maintainer will only take care of merging the fixes on the stable branch. For patch releases, we don't merge back to master. But `release-tools` will commit a changelog update to both the patch release, and the master branch. ### Creating a release candidate Release candidate (RC) can be created with a chatops command. This is the only type of release that a developer can build autonomously. When working on a GitLab feature that requires a minimum Gitaly version, tagging a RC is a good way to make sure the `gitlab` feature branch has the proper Gitaly version. - Pick the current milestone (i.e. 12.9) - Pick a release candidate number, you can check `VERSION` to see if we have one already (12.9.0-rc1) - run `/chatops run gitaly tag 12.9.0-rc1` - The release will be published - The [pipeline of a tag](https://gitlab.com/gitlab-org/gitaly/pipelines?scope=tags&page=1) has a **manual** job, `update-downstream-server-version`, that will create a merge request on the GitLab codebase to bump the Gitaly server version, and this will be assigned to you. Once the build has completed successfully, assign it to a maintainer for review. ## Publishing the Ruby gem If an updated version of the Ruby proto gem is needed, it can be published to rubygems.org with the `make publish-proto-gem` Makefile target. If the changes needed are not yet released, [create a release candidate](#creating-a-release-candidate) first. - Checkout the tag to publish (vX.Y.Z). - Run `make publish-proto-gem`, which automatically builds any publishes the Protobuf Gem. ## Using an unpublished Gem in GitLab Rails Sometimes, we need to work on both Gitaly and GitLab Rails implementations in parallel. Ideally, we can define and publish all Protobuf changes beforehand. In practice, a complicated change might require modifying the Protobuf multiple times until reaching a stable state. It's more convenient to let GitLab Rails point the gem to the the developing branch in Gitaly. In the local environment, we can point it to the developing gem using following steps: - In Gitaly directory, run `BUILD_GEM_OPTIONS="--skip-verify-tag" make build-proto-gem`. This command builds gem at `_built/gitaly.gem` and the gem contents to the `_build/gitaly-gem` directory. - Modify GitLab Rails' Gemfile: ```ruby gem 'gitaly', path: '/path/to/gitaly/_build/gitaly-gem' ``` - Re-run bundler to update the gem. Unfortunately, the Gitaly repository does not include the gemspec or any auto-generated Ruby code. All of it is generated on the fly while running `make build-proto-gem`. Thus, we cannot point the `Gemfile` to the developing branch directly on CI environment. As a workaround, we can build and publish the gem under a different name for development purpose only. Here's [an example](https://rubygems.org/gems/gitaly-qmnguyen). - Run `BUILD_GEM_OPTIONS="--skip-verify-tag" BUILD_GEM_NAME="gitaly-yourname" make build-proto-gem`. This command builds the gem under a different name. The resulting gem is located at `_build/gitaly-yourname.gem`. - Run `gem push _build/gitaly-yourname.gem` to publish that custom gem to rubygems.org or similar package registry. - Modify `gitaly` gem in GitLab Rails' `Gemfile`: ```ruby gem 'gitaly-yourname' ``` Or, we can specify a particular version. ```ruby gem 'gitaly-yourname', '~> 16.7.0.pre.0f0db5710' ``` After the development phase, please remember to remove this workaround and restore the `Gemfile` to the official Gitaly gem release. Note: Instead of setting `BUILD_GEM_OPTIONS` and `BUILD_GEM_NAME` in every command, we can also set them in `config.mak`. ## Publishing the go module If an [updated version](https://golang.org/doc/modules/release-workflow) of the go module is needed, it can be [published](https://golang.org/doc/modules/publishing) by tag creation. If a new [major module version update](https://golang.org/doc/modules/major-version) is needed, it can be changed by running `upgrade-module` `make` task with desired parameters: ```bash make upgrade-module FROM_MODULE=v15 TO_MODULE=v16 ``` It replaces old imports with the new version in the go source files, updates `*.proto` files and modifies `go.mod` file to use a new target version of the module. ### Security release Security releases involve additional processes to ensure that recent releases of GitLab are properly patched while avoiding the leaking of the security details to the public until appropriate. Before beginning work on a security fix, open a new Gitaly issue with the template `Security Release` and follow the instructions at the top of the page for following the template. ## Experimental builds Push the release tag to `dev.gitlab.org/gitlab/gitaly`. After passing the test suite, the tag will automatically be built and published in . ## Patching Git The Gitaly project is the single source of truth for the Git distribution across all of GitLab: all downstream distributions use the `make git` target to build and install the Git version used at runtime. Given that there is only one central location where we define the Git version and its features, this grants us the possibility to easily apply custom patches to Git. In order for a custom patch to be accepted into the Gitaly project, it must meet the high bar of being at least in the upstream's `next` branch. The mechanism is thus intended as a process to ensure that we can test upstreamed patches faster than having to wait for the next release, not to add patches which would never be accepted upstream. Patches which were not upstreamed yet will not be accepted: at no point in time do we want to start maintaining a friendly fork of Git. In order to add a patch, you can simply add it to the `GIT_PATCHES` array in our `Makefile`. Note: while there is only a single Git distribution which is distributed across all of GitLab's official distributions, there may be unofficial ones which use a different version of Git (most importantly source-based installations). So even if you add patches to Gitaly's Makefile, you cannot assume that installations will always have these patches. As a result, all code which makes use of patched-in features must have fallback code to support the [minimum required Git version](../README.md#installation) ## RPC deprecation process First create a deprecation issue at with the title `Deprecate RPC FooBar`. Use label `Deprecation`. Below is a template for the issue description. ```markdown We are deprecating RPC FooBar because **REASONS**. - [ ] put a deprecation comment `// DEPRECATED: ` in ./proto **Merge Request LINK** - [ ] find all client-side uses of RPC and list below - [ ] update all client-side uses to no longer use RPC **ADD Merge Request LINKS** - [ ] wait for a GitLab release in which the RPC is no longer occurring in client side code **LINK TO GITLAB-CE RELEASE TAG** - [ ] delete the server side implementation of the old RPC in https://gitlab.com/gitlab-org/gitaly **Merge Request LINK** ``` ## Bumping the major version When GitLab's major version is updated, we also need to bump Gitaly's module version number. Use `make upgrade-module` for this. Gitaly version numbers match GitLab version numbers for reasons discussed below. ### Why Gitaly and GitLab versions match In [`12e0bf3ac80b7`](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3525), we added a major version to the Gitaly module. The GitLab release process tags Gitaly versions according to the GitLab versioning scheme. Because Go tries to enforce semantic versioning, when one tries to pull a tag, it expects the corresponding major version. For example, if v13.0.4 is pulled, Go expects the major version of the module to be v13. Since then, the team has discussed removing the major version because it does not really represent semantic versioning for Gitaly. With semantic versioning, major versions only change when there is a breaking non-backwards compatible change. Gitaly major versions are, instead, simply tied to GitLab versions rather than taking into account breaking changes in the Gitaly API itself. Because we'd like to make Gitaly its own standalone product one day, it would be ideal to use true semantic versioning in Gitaly. However, this would also mean we would need to change our tagging scheme. Changing this would have a large impact on the GitLab release process because much of the tooling relies on the assumption that Gitaly versions match GitLab versions. There are also other challenges to removing the major version, detailed in [this issuey](https://gitlab.com/gitlab-org/gitaly/-/issues/4714).