diff options
author | Marcia Ramos <virtua.creative@gmail.com> | 2018-12-12 16:30:01 +0300 |
---|---|---|
committer | Marcia Ramos <virtua.creative@gmail.com> | 2018-12-12 16:30:01 +0300 |
commit | 5ee900ee9d2288cbf57007d905acd2a6e9239ff4 (patch) | |
tree | fbf19e70db74f191010cae63478cc6565c1a8610 | |
parent | be449047838eb34ac420875e9d3a2eeb92eaff31 (diff) | |
parent | 8d99636eff5889ddec080ea00fd7a2ee6c911179 (diff) |
Merge branch 'docs/update-automatic-merge-documentation' into 'master'
Update automatic merging documentation
See merge request gitlab-org/gitlab-ce!23689
-rw-r--r-- | doc/development/automatic_ce_ee_merge.md | 260 |
1 files changed, 177 insertions, 83 deletions
diff --git a/doc/development/automatic_ce_ee_merge.md b/doc/development/automatic_ce_ee_merge.md index e4eb26b3aca..8208e9007ea 100644 --- a/doc/development/automatic_ce_ee_merge.md +++ b/doc/development/automatic_ce_ee_merge.md @@ -1,33 +1,27 @@ # Automatic CE->EE merge -Whenever a commit is pushed to the CE `master` branch, it is automatically -merged into the EE `master` branch. If the commit produces any conflicts, it is -instead reverted from CE `master`. When this happens, a merge request will be -set up automatically that can be used to reinstate the changes. This merge -request will be assigned to the author of the conflicting commit, or the merge -request author if the commit author could not be associated with a GitLab user. -If no author could be found, the merge request is assigned to a random member of -the Delivery team. It is then up to this team member to figure out who to assign -the merge request to. - -Because some commits can not be reverted if new commits depend on them, we also -run a job periodically that processes a range of commits and tries to merge or -revert them. This should ensure that all commits are either merged into EE -`master`, or reverted, instead of just being left behind in CE. +Commits pushed to CE `master` are automatically merged into EE `master` roughly +every 5 minutes. Changes are merged using the `ours` merge strategy in the +context of EE. This means that any merge conflicts are resolved by taking the EE +changes and discarding the CE changes. This removes the need for resolving +conflicts or reverting changes, at the cost of **absolutely requiring** EE merge +requests to be created whenever a CE merge request causes merge conflicts. +Failing to do so can result in changes not making their way into EE. + +## Always create an EE merge request if there are conflicts + +In CI there is a job called `ee_compat_check`, which checks if a CE MR causes +merge conflicts with EE. If this job reports conflicts, you **must** create an +EE merge request. If you are an external contributor you can ask the reviewer to +do this for you. ## Always merge EE merge requests before their CE counterparts **In order to avoid conflicts in the CE->EE merge, you should always merge the EE version of your CE merge request first, if present.** -The rationale for this is that as CE->EE merges are done automatically, it can -happen that: - -1. A CE merge request that needs EE-specific changes is merged. -1. The automatic CE->EE merge happens. -1. Conflicts due to the CE merge request occur since its EE merge request isn't - merged yet. -1. The CE changes are reverted. +Failing to do so will lead to CE changes being discarded when merging into EE, +if they cause merge conflicts. ## Avoiding CE->EE merge conflicts beforehand @@ -45,76 +39,184 @@ detect if the current branch's changes will conflict during the CE->EE merge. The job reports what files are conflicting and how to set up a merge request against EE. -## How to reinstate changes - -When a commit is reverted, the corresponding merge request to reinstate the -changes will include all the details necessary to ensure the changes make it -back into CE and EE. However, you still need to manually set up an EE merge -request that resolves the conflicts. +#### How the job works + +1. Generates the diff between your branch and current CE `master` +1. Tries to apply it to current EE `master` +1. If it applies cleanly, the job succeeds, otherwise... +1. Detects a branch with the `ee-` prefix or `-ee` suffix in EE +1. If it exists, generate the diff between this branch and current EE `master` +1. Tries to apply it to current EE `master` +1. If it applies cleanly, the job succeeds + +In the case where the job fails, it means you should create an `ee-<ce_branch>` +or `<ce_branch>-ee` branch, push it to EE and open a merge request against EE +`master`. +At this point if you retry the failing job in your CE merge request, it should +now pass. + +Notes: + +- This task is not a silver-bullet, its current goal is to bring awareness to + developers that their work needs to be ported to EE. +- Community contributors shouldn't be required to submit merge requests against + EE, but reviewers should take actions by either creating such EE merge request + or asking a GitLab developer to do it **before the merge request is merged**. +- If you branch is too far behind `master`, the job will fail. In that case you + should rebase your branch upon latest `master`. +- Code reviews for merge requests often consist of multiple iterations of + feedback and fixes. There is no need to update your EE MR after each + iteration. Instead, create an EE MR as soon as you see the + `ee_compat_check` job failing. After you receive the final approval + from a Maintainer (but **before the CE MR is merged**) update the EE MR. + This helps to identify significant conflicts sooner, but also reduces the + number of times you have to resolve conflicts. +- Please remember to + [always have your EE merge request merged before the CE version](#always-merge-ee-merge-requests-before-their-ce-counterparts). +- You can use [`git rerere`](https://git-scm.com/docs/git-rerere) + to avoid resolving the same conflicts multiple times. + +### Cherry-picking from CE to EE + +For avoiding merge conflicts, we use a method of creating equivalent branches +for CE and EE. If the `ee-compat-check` job fails, this process is required. + +This method only requires that you have cloned both CE and EE into your computer. +If you don't have them yet, please go ahead and clone them: + +- Clone CE repo: `git clone git@gitlab.com:gitlab-org/gitlab-ce.git` +- Clone EE repo: `git clone git@gitlab.com:gitlab-org/gitlab-ee.git` + +And the only additional setup we need is to add CE as remote of EE and vice-versa: + +- Open two terminal windows, one in CE, and another one in EE: + - In EE: `git remote add ce git@gitlab.com:gitlab-org/gitlab-ce.git` + - In CE: `git remote add ee git@gitlab.com:gitlab-org/gitlab-ee.git` + +That's all setup we need, so that we can cherry-pick a commit from CE to EE, and +from EE to CE. + +Now, every time you create an MR for CE and EE: + +1. Open two terminal windows, one in CE, and another one in EE +1. In the CE terminal: + 1. Create the CE branch, e.g., `branch-example` + 1. Make your changes and push a commit (commit A) + 1. Create the CE merge request in GitLab +1. In the EE terminal: + 1. Create the EE-equivalent branch ending with `-ee`, e.g., + `git checkout -b branch-example-ee` + 1. Fetch the CE branch: `git fetch ce branch-example` + 1. Cherry-pick the commit A: `git cherry-pick commit-A-SHA` + 1. If Git prompts you to fix the conflicts, do a `git status` + to check which files contain conflicts, fix them, save the files + 1. Add the changes with `git add .` but **DO NOT commit** them + 1. Continue cherry-picking: `git cherry-pick --continue` + 1. Push to EE: `git push origin branch-example-ee` +1. Create the EE-equivalent MR and link to the CE MR from the +description "Ports [CE-MR-LINK] to EE" +1. Once all the jobs are passing in both CE and EE, you've addressed the +feedback from your own team, and got them approved, the merge requests can be merged. +1. When both MRs are ready, the EE merge request will be merged first, and the +CE-equivalent will be merged next. + +**Important notes:** + +- The commit SHA can be easily found from the GitLab UI. From a merge request, +open the tab **Commits** and click the copy icon to copy the commit SHA. +- To cherry-pick a **commit range**, such as [A > B > C > D] use: + + ```shell + git cherry-pick "oldest-commit-SHA^..newest-commit-SHA" + ``` + + For example, suppose the commit A is the oldest, and its SHA is `4f5e4018c09ed797fdf446b3752f82e46f5af502`, + and the commit D is the newest, and its SHA is `80e1c9e56783bd57bd7129828ec20b252ebc0538`. + The cherry-pick command will be: + + ```shell + git cherry-pick "4f5e4018c09ed797fdf446b3752f82e46f5af502^..80e1c9e56783bd57bd7129828ec20b252ebc0538" + ``` + +- To cherry-pick a **merge commit**, use the flag `-m 1`. For example, suppose that the +merge commit SHA is `138f5e2f20289bb376caffa0303adb0cac859ce1`: + + ```shell + git cherry-pick -m 1 138f5e2f20289bb376caffa0303adb0cac859ce1 + ``` +- To cherry-pick multiple commits, such as B and D in a range [A > B > C > D], use: + + ```shell + git cherry-pick commmit-B-SHA commit-D-SHA + ``` + + For example, suppose commit B SHA = `4f5e4018c09ed797fdf446b3752f82e46f5af502`, + and the commit D SHA = `80e1c9e56783bd57bd7129828ec20b252ebc0538`. + The cherry-pick command will be: + + ```shell + git cherry-pick 4f5e4018c09ed797fdf446b3752f82e46f5af502 80e1c9e56783bd57bd7129828ec20b252ebc0538 + ``` + + This case is particularly useful when you have a merge commit in a sequence of + commits and you want to cherry-pick all but the merge commit. + +- If you push more commits to the CE branch, you can safely repeat the procedure +to cherry-pick them to the EE-equivalent branch. You can do that as many times as +necessary, using the same CE and EE branches. +- If you submitted the merge request to the CE repo and the `ee-compat-check` job passed, +you are not required to submit the EE-equivalent MR, but it's still recommended. If the +job failed, you are required to submit the EE MR so that you can fix the conflicts in EE +before merging your changes into CE. + +--- + +[Return to Development documentation](README.md) -Each merge request used to reinstate changes will have the "reverted" label -applied. Please do not remove this label, as it will be used to determine how -many times commits are reverted and how long it takes to reinstate the changes. - -An example merge request can be found in [CE merge request -23280](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23280). +## FAQ -## How it works +### How does automatic merging work? The automatic merging is performed using a project called [Merge -Train](https://gitlab.com/gitlab-org/merge-train/). For every commit to merge or -revert, we generate patches using `git format-patch` which we then try to apply -using `git am --3way`. If this succeeds we push the changes to EE, if this fails -we decide what to do based on the failure reason: - -1. If the patch could not be applied because it was already applied, we just - skip it. -1. If the patch caused conflicts, we revert the source commits. - -Commits are reverted in reverse order, ensuring that if commit B depends on A, -and both conflict, we first revert B followed by reverting A. +Train](https://gitlab.com/gitlab-org/merge-train/). This project will clone CE +and EE master, and merge CE master into EE master using `git merge +--strategy=ours`. This process runs roughly every 5 minutes. -## FAQ - -### Why? +For more information on the exact implementation you can refer to the source +code. -We want to work towards being able to deploy continuously, but this requires -that `master` is always stable and has all the changes we need. If CE `master` -can not be merged into EE `master` due to merge conflicts, this prevents _any_ -change from CE making its way into EE. Since GitLab.com runs on EE, this -effectively prevents us from deploying changes. +### Why merge automatically? -Past experiences and data have shown that periodic CE to EE merge requests do -not scale, and often take a very long time to complete. For example, [in this +As we work towards continuous deployments and a single repository for both CE +and EE, we need to first make sure that all CE changes make their way into CE as +fast as possible. Past experiences and data have shown that periodic CE to EE +merge requests do not scale, and often take a very long time to complete. For +example, [in this comment](https://gitlab.com/gitlab-org/release/framework/issues/49#note_114614619) we determined that the average time to close an upstream merge request is around 5 hours, with peaks up to several days. Periodic merge requests are also frustrating to work with, because they often include many changes unrelated to your own changes. -Automatically merging or reverting commits allows us to keep merging changes -from CE into EE, as we never have to wait hours for somebody to resolve a set of -merge conflicts. - -### Does the CE to EE merge take into account merge commits? - -No. When merging CE changes into EE, merge commits are ignored. +To resolve these problems, we now merge changes using the `ours` strategy to +automatically resolve merge conflicts. This removes the need for resolving +conflicts in a periodic merge request, and allows us to merge changes from CE +into EE much faster. -### My changes are reverted, but I set up an EE MR to resolve conflicts +### My CE merge request caused conflicts after it was merged. What do I do? -Most likely the automatic merge job ran before the EE merge request was merged. -If this keeps happening, consider reporting a bug in the [Merge Train issue -tracker](https://gitlab.com/gitlab-org/merge-train/issues). +If you notice this, you should set up an EE merge request that resolves these +conflicts as **soon as possible**. Failing to do so can lead to your changes not +being available in EE, which may break tests. This in turn would prevent us from +being able to deploy. -### My changes keep getting reverted, and this is really annoying! +### Won't this setup be risky? -This is understandable, but the solution to this is fairly straightforward: -simply set up an EE merge request for every CE merge request, and resolve your -conflicts before the changes are reverted. - -### Will we allow certain people to still merge changes, even if they conflict? - -No. +No, not if there is an EE merge request for every CE merge request that causes +conflicts _and_ that EE merge request is merged first. In the past we may have +been a bit more relaxed when it comes to enforcing EE merge requests, but to +enable automatic merging have to start requiring such merge requests even for +the smallest conflicts. ### Some files I work with often conflict, how can I best deal with this? @@ -123,11 +225,3 @@ so that the EE specific changes are not intertwined with CE code. For Ruby code you can do this by moving the EE code to a separate module, which can then be injected into the appropriate classes or modules. See [Guidelines for implementing Enterprise Edition features](ee_features.md) for more information. - -### Will changelog entries be reverted automatically? - -Only if the changelog was added in the commit that was reverted. If a changelog -entry was added in a separate commit, it is possible for it to be left behind. -Since changelog entries are related to the changes in question, there is no real -reason to commit the changelog separately, and as such this should not be a big -problem. |