Merge branch 'ee-merge-train' into 'master'

EE: Automatically merge CE master into EE master

See merge request gitlab-org/gitlab-ee!8657

.gitlab-ci.yml

@@ -53,6 +53,7 @@ after_script:
 stages:
   - build
   - prepare
+  - merge
   - test
   - post-test
   - pages
@@ -1203,3 +1204,26 @@ schedule:review-cleanup:
     - gem install gitlab --no-document
   script:
     - ruby -rrubygems scripts/review_apps/automated_cleanup.rb
+
+merge:master:
+  image: registry.gitlab.com/gitlab-org/merge-train
+  stage: merge
+  # The global before_script/after_script blocks break this job, or aren't
+  # necessary. These two lines result in them being ignored.
+  before_script: []
+  after_script: []
+  only:
+    refs:
+      - master
+      - schedules
+    variables:
+      - $CI_PROJECT_PATH == "gitlab-org/gitlab-ce"
+      - $MERGE_TRAIN_SSH_PUBLIC_KEY
+      - $MERGE_TRAIN_SSH_PRIVATE_KEY
+      - $MERGE_TRAIN_API_TOKEN
+  script:
+    - scripts/merge-train
+  cache:
+    paths:
+      - gitlab-ee
+    key: "merge:master"

doc/development/automatic_ce_ee_merge.md

 # Automatic CE->EE merge
 
-GitLab Community Edition is merged automatically every 3 hours into the
-Enterprise Edition (look for the [`CE Upstream` merge requests]).
-
-This merge is done automatically in a
-[scheduled pipeline](https://gitlab.com/gitlab-org/release-tools/-/jobs/43201679).
-
-## What to do if you are pinged in a `CE Upstream` merge request to resolve a conflict?
-
-1. Please resolve the conflict as soon as possible or ask someone else to do it
-   - It's ok to resolve more conflicts than the one that you are asked to resolve.
-     In that case, it's a good habit to ask for a double-check on your resolution
-     by someone who is familiar with the code you touched.
-1. Once you have resolved your conflicts, push to the branch (no force-push)
-1. Assign the merge request to the next person that has to resolve a conflict
-1. If all conflicts are resolved after your resolution is pushed, keep the merge
-   request assigned to you: **you are now responsible for the merge request to be
-   green**
-1. If you are the last person to resolve the conflicts, the pipeline is green,
-   and you have merge rights, merge the MR, but **do not** choose to squash.
-   Otherwise, assign the MR to someone that can merge.
-1. If you need any help, you can ping the current [release managers], or ask in
-   the `#ce-to-ee` Slack channel
-
-A few notes about the automatic CE->EE merge job:
-
-- If a merge is already in progress, the job
-  [doesn't create a new one](https://gitlab.com/gitlab-org/release-tools/-/jobs/43157687).
-- If there is nothing to merge (i.e. EE is up-to-date with CE), the job doesn't
-  create a new one
-- The job posts messages to the `#ce-to-ee` Slack channel to inform what's the
-  current CE->EE merge status (e.g. "A new MR has been created", "A MR is still pending")
-
-[`CE Upstream` merge requests]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests?label_name%5B%5D=CE+upstream
-[release managers]: https://about.gitlab.com/release-managers/
+Whenever a commit is pushed to the CE `master` branch, it is automatically
+merged into the EE `master` branch. If the commit produces any conflicts, it is
+instead reverted from CE `master`. When this happens, a merge request will be
+set up automatically that can be used to reinstate the changes. This merge
+request will be assigned to the author of the conflicting commit, or the merge
+request author if the commit author could not be associated with a GitLab user.
+If no author could be found, the merge request is assigned to a random member of
+the Delivery team. It is then up to this team member to figure out who to assign
+the merge request to.
+
+Because some commits can not be reverted if new commits depend on them, we also
+run a job periodically that processes a range of commits and tries to merge or
+revert them. This should ensure that all commits are either merged into EE
+`master`, or reverted, instead of just being left behind in CE.
 
 ## Always merge EE merge requests before their CE counterparts
 
 **In order to avoid conflicts in the CE->EE merge, you should always merge the
 EE version of your CE merge request first, if present.**
 
-The rationale for this is that as CE->EE merges are done automatically every few
-hours, it can happen that:
+The rationale for this is that as CE->EE merges are done automatically, it can
+happen that:
 
-1. A CE merge request that needs EE-specific changes is merged
-1. The automatic CE->EE merge happens
+1. A CE merge request that needs EE-specific changes is merged.
+1. The automatic CE->EE merge happens.
 1. Conflicts due to the CE merge request occur since its EE merge request isn't
-  merged yet
-1. The automatic merge bot will ping someone to resolve the conflict **that are
-  already resolved in the EE merge request that isn't merged yet**
-
-That's a waste of time, and that's why you should merge EE merge request before
-their CE counterpart.
+   merged yet.
+1. The CE changes are reverted.
 
 ## Avoiding CE->EE merge conflicts beforehand
 
@@ -69,136 +45,89 @@ detect if the current branch's changes will conflict during the CE->EE merge.
 The job reports what files are conflicting and how to set up a merge request
 against EE.
 
-#### How the job works
-
-1. Generates the diff between your branch and current CE `master`
-1. Tries to apply it to current EE `master`
-1. If it applies cleanly, the job succeeds, otherwise...
-1. Detects a branch with the `ee-` prefix or `-ee` suffix in EE
-1. If it exists, generate the diff between this branch and current EE `master`
-1. Tries to apply it to current EE `master`
-1. If it applies cleanly, the job succeeds
-
-In the case where the job fails, it means you should create an `ee-<ce_branch>`
-or `<ce_branch>-ee` branch, push it to EE and open a merge request against EE
-`master`.
-At this point if you retry the failing job in your CE merge request, it should
-now pass.
-
-Notes:
-
-- This task is not a silver-bullet, its current goal is to bring awareness to
-  developers that their work needs to be ported to EE.
-- Community contributors shouldn't be required to submit merge requests against
-  EE, but reviewers should take actions by either creating such EE merge request
-  or asking a GitLab developer to do it **before the merge request is merged**.
-- If you branch is too far behind `master`, the job will fail. In that case you
-  should rebase your branch upon latest `master`.
-- Code reviews for merge requests often consist of multiple iterations of
-  feedback and fixes. There is no need to update your EE MR after each
-  iteration. Instead, create an EE MR as soon as you see the
-  `ee_compat_check` job failing. After you receive the final approval
-  from a Maintainer (but **before the CE MR is merged**) update the EE MR.
-  This helps to identify significant conflicts sooner, but also reduces the
-  number of times you have to resolve conflicts.
-- Please remember to
-  [always have your EE merge request merged before the CE version](#always-merge-ee-merge-requests-before-their-ce-counterparts).
-- You can use [`git rerere`](https://git-scm.com/docs/git-rerere)
-  to avoid resolving the same conflicts multiple times.
-
-### Cherry-picking from CE to EE
-
-For avoiding merge conflicts, we use a method of creating equivalent branches
-for CE and EE. If the `ee-compat-check` job fails, this process is required.
-
-This method only requires that you have cloned both CE and EE into your computer.
-If you don't have them yet, please go ahead and clone them:
-
-- Clone CE repo: `git clone git@gitlab.com:gitlab-org/gitlab-ce.git`
-- Clone EE repo: `git clone git@gitlab.com:gitlab-org/gitlab-ee.git`
-
-And the only additional setup we need is to add CE as remote of EE and vice-versa:
-
-- Open two terminal windows, one in CE, and another one in EE:
-  - In EE: `git remote add ce git@gitlab.com:gitlab-org/gitlab-ce.git`
-  - In CE: `git remote add ee git@gitlab.com:gitlab-org/gitlab-ee.git`
-
-That's all setup we need, so that we can cherry-pick a commit from CE to EE, and
-from EE to CE.
-
-Now, every time you create an MR for CE and EE:
-
-1. Open two terminal windows, one in CE, and another one in EE
-1. In the CE terminal:
-   1. Create the CE branch, e.g., `branch-example`
-   1. Make your changes and push a commit (commit A)
-   1. Create the CE merge request in GitLab
-1. In the EE terminal:
-   1. Create the EE-equivalent branch ending with `-ee`, e.g.,
-      `git checkout -b branch-example-ee`
-   1. Fetch the CE branch: `git fetch ce branch-example`
-   1. Cherry-pick the commit A: `git cherry-pick commit-A-SHA`
-   1. If Git prompts you to fix the conflicts, do a `git status`
-      to check which files contain conflicts, fix them, save the files
-   1. Add the changes with `git add .` but **DO NOT commit** them
-   1. Continue cherry-picking: `git cherry-pick --continue`
-   1. Push to EE: `git push origin branch-example-ee`
-1. Create the EE-equivalent MR and link to the CE MR from the
-description "Ports [CE-MR-LINK] to EE"
-1. Once all the jobs are passing in both CE and EE, you've addressed the
-feedback from your own team, and got them approved, the merge requests can be merged.
-1. When both MRs are ready, the EE merge request will be merged first, and the
-CE-equivalent will be merged next.
-
-**Important notes:**
-
-- The commit SHA can be easily found from the GitLab UI. From a merge request,
-open the tab **Commits** and click the copy icon to copy the commit SHA.
-- To cherry-pick a **commit range**, such as [A > B > C > D] use:
-
-    ```shell
-    git cherry-pick "oldest-commit-SHA^..newest-commit-SHA"
-    ```
-
-    For example, suppose the commit A is the oldest, and its SHA is `4f5e4018c09ed797fdf446b3752f82e46f5af502`,
-    and the commit D is the newest, and its SHA is `80e1c9e56783bd57bd7129828ec20b252ebc0538`.
-    The cherry-pick command will be:
-
-    ```shell
-    git cherry-pick "4f5e4018c09ed797fdf446b3752f82e46f5af502^..80e1c9e56783bd57bd7129828ec20b252ebc0538"
-    ```
-
-- To cherry-pick a **merge commit**, use the flag `-m 1`. For example, suppose that the
-merge commit SHA is `138f5e2f20289bb376caffa0303adb0cac859ce1`:
-
-    ```shell
-    git cherry-pick -m 1 138f5e2f20289bb376caffa0303adb0cac859ce1
-    ```
-- To cherry-pick multiple commits, such as B and D in a range [A > B > C > D], use:
-
-    ```shell
-    git cherry-pick commmit-B-SHA commit-D-SHA
-    ```
-
-    For example, suppose commit B SHA = `4f5e4018c09ed797fdf446b3752f82e46f5af502`,
-    and the commit D SHA = `80e1c9e56783bd57bd7129828ec20b252ebc0538`.
-    The cherry-pick command will be:
-
-    ```shell
-    git cherry-pick 4f5e4018c09ed797fdf446b3752f82e46f5af502 80e1c9e56783bd57bd7129828ec20b252ebc0538
-    ```
-
-    This case is particularly useful when you have a merge commit in a sequence of
-    commits and you want to cherry-pick all but the merge commit.
-
-- If you push more commits to the CE branch, you can safely repeat the procedure
-to cherry-pick them to the EE-equivalent branch. You can do that as many times as
-necessary, using the same CE and EE branches.
-- If you submitted the merge request to the CE repo and the `ee-compat-check` job passed,
-you are not required to submit the EE-equivalent MR, but it's still recommended. If the
-job failed, you are required to submit the EE MR so that you can fix the conflicts in EE
-before merging your changes into CE.
-
----
-
-[Return to Development documentation](README.md)
+## How to reinstate changes
+
+When a commit is reverted, the corresponding merge request to reinstate the
+changes will include all the details necessary to ensure the changes make it
+back into CE and EE. However, you still need to manually set up an EE merge
+request that resolves the conflicts.
+
+Each merge request used to reinstate changes will have the "reverted" label
+applied. Please do not remove this label, as it will be used to determine how
+many times commits are reverted and how long it takes to reinstate the changes.
+
+An example merge request can be found in [CE merge request
+23280](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23280).
+
+## How it works
+
+The automatic merging is performed using a project called [Merge
+Train](https://gitlab.com/gitlab-org/merge-train/). For every commit to merge or
+revert, we generate patches using `git format-patch` which we then try to apply
+using `git am --3way`. If this succeeds we push the changes to EE, if this fails
+we decide what to do based on the failure reason:
+
+1. If the patch could not be applied because it was already applied, we just
+   skip it.
+1. If the patch caused conflicts, we revert the source commits.
+
+Commits are reverted in reverse order, ensuring that if commit B depends on A,
+and both conflict, we first revert B followed by reverting A.
+
+## FAQ
+
+### Why?
+
+We want to work towards being able to deploy continuously, but this requires
+that `master` is always stable and has all the changes we need. If CE `master`
+can not be merged into EE `master` due to merge conflicts, this prevents _any_
+change from CE making its way into EE. Since GitLab.com runs on EE, this
+effectively prevents us from deploying changes.
+
+Past experiences and data have shown that periodic CE to EE merge requests do
+not scale, and often take a very long time to complete. For example, [in this
+comment](https://gitlab.com/gitlab-org/release/framework/issues/49#note_114614619)
+we determined that the average time to close an upstream merge request is around
+5 hours, with peaks up to several days. Periodic merge requests are also
+frustrating to work with, because they often include many changes unrelated to
+your own changes.
+
+Automatically merging or reverting commits allows us to keep merging changes
+from CE into EE, as we never have to wait hours for somebody to resolve a set of
+merge conflicts.
+
+### Does the CE to EE merge take into account merge commits?
+
+No. When merging CE changes into EE, merge commits are ignored.
+
+### My changes are reverted, but I set up an EE MR to resolve conflicts
+
+Most likely the automatic merge job ran before the EE merge request was merged.
+If this keeps happening, consider reporting a bug in the [Merge Train issue
+tracker](https://gitlab.com/gitlab-org/merge-train/issues).
+
+### My changes keep getting reverted, and this is really annoying!
+
+This is understandable, but the solution to this is fairly straightforward:
+simply set up an EE merge request for every CE merge request, and resolve your
+conflicts before the changes are reverted.
+
+### Will we allow certain people to still merge changes, even if they conflict?
+
+No.
+
+### Some files I work with often conflict, how can I best deal with this?
+
+If you find you keep running into merge conflicts, consider refactoring the file
+so that the EE specific changes are not intertwined with CE code. For Ruby code
+you can do this by moving the EE code to a separate module, which can then be
+injected into the appropriate classes or modules. See [Guidelines for
+implementing Enterprise Edition features](ee_features.md) for more information.
+
+### Will changelog entries be reverted automatically?
+
+Only if the changelog was added in the commit that was reverted. If a changelog
+entry was added in a separate commit, it is possible for it to be left behind.
+Since changelog entries are related to the changes in question, there is no real
+reason to commit the changelog separately, and as such this should not be a big
+problem.

scripts/merge-train

+#!/bin/sh
+
+set -e
+
+# The name (including namespace) of the EE repository to merge commits into.
+EE_PROJECT='gitlab-org/gitlab-ee'
+
+# The directory to clone GitLab EE into.
+EE_DIRECTORY="$CI_PROJECT_DIR/gitlab-ee"
+
+# The EE branch to merge the changes into.
+EE_BRANCH='master'
+
+# Runs an incremental or periodic merge of CE to EE. This script should be run
+# from a container built using https://gitlab.com/gitlab-org/merge-train.
+
+# Merges (or reverts) commits in a batch (based on CI_COMMIT_BEFORE_SHA and
+# CI_COMMIT_SHA), or since a specific commit.
+#
+# The optional first argument of this function should be a SHA of a commit. When
+# specified, all commits since this commit will be processed.
+merge() {
+    # We need to source the configure-ssh script instead of running it with
+    # `sh`, since it uses `eval` for SSH agent and we want the result of that to
+    # persist.
+    #
+    # shellcheck disable=SC1091
+    . /app/bin/configure-ssh
+
+    # We can not perform a shallow clone, as this results in Git sometimes
+    # refusing to fetch from CE. To work around this, we perform a full clone
+    # but cache the repository in CI. If a cached repository exists, we simply
+    # just pull from `master`.
+    if [ -d "$EE_DIRECTORY" ]
+    then
+        echo "Updating existing clone of $EE_PROJECT"
+
+        git -C "$EE_DIRECTORY" pull --quiet origin "$EE_BRANCH"
+    else
+        echo "Cloning $EE_PROJECT"
+
+        git clone --quiet --single-branch --branch "$EE_BRANCH" \
+            "git@gitlab.com:$EE_PROJECT.git" "$EE_DIRECTORY"
+    fi
+
+    cd /app
+
+    env GITLAB_TOKEN="$MERGE_TRAIN_API_TOKEN" \
+        bundle exec /app/bin/merge-train "$CI_PROJECT_DIR" "$EE_DIRECTORY" \
+        --source-name "$CI_PROJECT_PATH" \
+        --target-branch "$EE_BRANCH" \
+        ${1:+--before "$1"}
+}
+
+# Merges (or reverts) all commits since a point in time as supported by `git log
+# --since`, such as "12 hours ago".
+merge_since() {
+    commit="$(git log --since="$MERGE_SINCE" --reverse --format=%H | head -n1)"
+
+    if [ "$commit" = '' ]
+    then
+        echo "There are no commits to merge since $MERGE_SINCE"
+    else
+        merge "$commit"
+    fi
+}
+
+if [ "$MERGE_SINCE" ]
+then
+    merge_since
+else
+    merge
+fi