Merge branch...

Merge branch 'refine-docs-trigger-script-and-use-a-really-long-branch-name-to-test-it' into 'master'

Refine docs trigger script and use a really long branch name to test it

For branch names more than 63 characters we were hitting the
filesystem limit and rsync created a truncated directory.
This change truncates the branch name beforehand so the Review
App URL is properly shown in the merge request widget.

See merge request gitlab-org/gitlab-ce!14354

.gitlab-ci.yml

@@ -174,7 +174,8 @@ build-package:
 # Review docs base
 .review-docs: &review-docs
   image: ruby:2.4-alpine
-  before_script: []
+  before_script:
+    - gem install gitlab --no-doc
   services: []
   variables:
     SETUP_DB: "false"
@@ -193,10 +194,9 @@ review-docs-deploy:
     name: review-docs/$CI_COMMIT_REF_NAME
     # DOCS_REVIEW_APPS_DOMAIN and DOCS_GITLAB_REPO_SUFFIX are secret variables
     # Discussion: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14236/diffs#note_40140693
-    url: http://$CI_COMMIT_REF_SLUG-built-from-ce-ee.$DOCS_REVIEW_APPS_DOMAIN/$DOCS_GITLAB_REPO_SUFFIX
+    url: http://preview-$CI_COMMIT_REF_SLUG.$DOCS_REVIEW_APPS_DOMAIN/$DOCS_GITLAB_REPO_SUFFIX
     on_stop: review-docs-cleanup
   script:
-    - gem install gitlab --no-doc
     - scripts/trigger-build-docs deploy
 
 # Cleanup remote environment of gitlab-docs
@@ -207,7 +207,6 @@ review-docs-cleanup:
     name: review-docs/$CI_COMMIT_REF_NAME
     action: stop
   script:
-    - gem install gitlab --no-doc
     - scripts/trigger-build-docs cleanup
 
 # Retrieve knapsack and rspec_flaky reports

doc/development/writing_documentation.md

@@ -106,21 +106,84 @@ CE and EE.
 
 ## Previewing the changes live
 
-If you want to preview your changes live, you can use the manual `build-docs`
-job in your merge request.
+If you want to preview the doc changes of your merge request live, you can use
+the manual `review-docs-deploy` job in your merge request.
+
+TIP: **Tip:**
+If your branch contains only documentation changes, you can use
+[special branch names](#testing) to avoid long running pipelines.
 
 ![Manual trigger a docs build](img/manual_build_docs.png)
 
 This job will:
 
 1. Create a new branch in the [gitlab-docs](https://gitlab.com/gitlab-com/gitlab-docs)
-   project named after the scheme: `<CE/EE-branch-slug>-built-from-ce-ee`
-1. Trigger a pipeline and build the docs site with your changes
-
-Look for the docs URL at the output of the `build-docs` job.
-
->**Note:**
+   project named after the scheme: `preview-<branch-slug>`
+1. Trigger a cross project pipeline and build the docs site with your changes
+
+After a few minutes, the Review App will be deployed and you will be able to
+preview the changes. The docs URL can be found in two places:
+
+- In the merge request widget
+- In the output of the `review-docs-deploy` job, which also includes the
+  triggered pipeline so that you can investigate whether something went wrong
+
+In case the Review App URL returns 404, follow these steps to debug:
+
+1. **Did you follow the URL from the merge request widget?** If yes, then check if
+   the link is the same as the one in the job output. It can happen that if the
+   branch name slug is longer than 35 characters, it is automatically
+   truncated. That means that the merge request widget will not show the proper
+   URL due to a limitation of how `environment: url` works, but you can find the
+   real URL from the output of the `review-docs-deploy` job.
+1. **Did you follow the URL from the job output?** If yes, then it means that
+   either the site is not yet deployed or something went wrong with the remote
+   pipeline. Give it a few minutes and it should appear online, otherwise you
+   can check the status of the remote pipeline from the link in the job output.
+   If the pipeline failed or got stuck, drop a line in the `#docs` chat channel.
+
+TIP: **Tip:**
+Someone that has no merge rights to the CE/EE projects (think of forks from
+contributors) will not be able to run the manual job. In that case, you can
+ask someone from the GitLab team who has the permissions to do that for you.
+
+NOTE: **Note:**
 Make sure that you always delete the branch of the merge request you were
 working on. If you don't, the remote docs branch won't be removed either,
 and the server where the Review Apps are hosted will eventually be out of
 disk space.
+
+### Behind the scenes
+
+If you want to know the hot details, here's what's really happening:
+
+1. You manually run the `review-docs-deploy` job in a CE/EE merge request.
+1. The job runs the [`scirpts/trigger-build-docs`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/scripts/trigger-build-docs)
+   script with the `deploy` flag, which in turn:
+   1. Takes your branch name and applies the following:
+      - The slug of the branch name is used to avoid special characters since
+        ultimately this will be used by NGINX.
+      - The `preview-` prefix is added to avoid conflicts if there's a remote branch
+        with the same name that you created in the merge request.
+      - The final branch name is truncated to 42 characters to avoid filesystem
+        limitations with long branch names (> 63 chars).
+   1. The remote branch is then created if it doesn't exist (meaning you can
+      re-run the manual job as many times as you want and this step will be skipped).
+   1. A new cross-project pipeline is triggered in the docs project.
+   1. The preview URL is shown both at the job output and in the merge request
+      widget. You also get the link to the remote pipeline.
+1. In the docs project, the pipeline is created and it
+   [skips the test jobs](https://gitlab.com/gitlab-com/gitlab-docs/blob/8d5d5c750c602a835614b02f9db42ead1c4b2f5e/.gitlab-ci.yml#L50-55)
+   to lower the build time.
+1. Once the docs site is built, the HTML files are uploaded as artifacts.
+1. A specific Runner tied only to the docs project, runs the Review App job
+   that downloads the artifacts and uses `rsync` to transfer the files over
+   to a location where NGINX serves them.
+
+The following GitLab features are used among others:
+
+- [Manual actions](../ci/yaml/README.md#manual-actions)
+- [Multi project pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipeline_graphs.html)
+- [Review Apps](../ci/review_apps/index.md)
+- [Artifacts](../ci/yaml/README.md#artifacts)
+- [Specific Runner](../ci/runners/README.md#locking-a-specific-runner-from-being-enabled-for-other-projects)

scripts/trigger-build-docs

@@ -2,13 +2,6 @@
 
 require 'gitlab'
 
-#
-# Give the remote branch a different name than the current one
-# in order to avoid conflicts
-#
-@docs_branch = "#{ENV["CI_COMMIT_REF_SLUG"]}-built-from-ce-ee"
-GITLAB_DOCS_REPO = 'gitlab-com/gitlab-docs'.freeze
-
 #
 # Configure credentials to be used with gitlab gem
 #
@@ -17,6 +10,26 @@ Gitlab.configure do |config|
   config.private_token  = ENV["DOCS_API_TOKEN"]  # GitLab Docs bot access token which has only Developer access to gitlab-docs
 end
 
+#
+# The remote docs project
+#
+GITLAB_DOCS_REPO = 'gitlab-com/gitlab-docs'.freeze
+
+#
+# Truncate the remote docs branch name if it's more than 63 characters
+# otherwise we hit the filesystem limit and the directory name where
+# NGINX serves the site won't match the branch name.
+#
+def docs_branch
+  # The maximum string length a file can have on a filesystem (ext4)
+  # is 63 characters. Let's use something smaller to be 100% sure.
+  max = 42
+  # Prefix the remote branch with 'preview-' in order to avoid
+  # name conflicts in the rare case the branch name already
+  # exists in the docs repo and truncate to max length.
+  "preview-#{ENV["CI_COMMIT_REF_SLUG"]}"[0...max]
+end
+
 #
 # Dummy way to find out in which repo we are, CE or EE
 #
@@ -28,18 +41,18 @@ end
 # Create a remote branch in gitlab-docs
 #
 def create_remote_branch
-  Gitlab.create_branch(GITLAB_DOCS_REPO, @docs_branch, 'master')
-  puts "Remote branch '#{@docs_branch}' created"
+  Gitlab.create_branch(GITLAB_DOCS_REPO, docs_branch, 'master')
+  puts "Remote branch '#{docs_branch}' created"
 rescue Gitlab::Error::BadRequest
-  puts "Remote branch '#{@docs_branch}' already exists"
+  puts "Remote branch '#{docs_branch}' already exists"
 end
 
 #
 # Remove a remote branch in gitlab-docs
 #
 def remove_remote_branch
-  Gitlab.delete_branch(GITLAB_DOCS_REPO, @docs_branch)
-  puts "Remote branch '#{@docs_branch}' deleted"
+  Gitlab.delete_branch(GITLAB_DOCS_REPO, docs_branch)
+  puts "Remote branch '#{docs_branch}' deleted"
 end
 
 #
@@ -50,11 +63,11 @@ def trigger_pipeline
   param_name = ee? ? 'BRANCH_EE' : 'BRANCH_CE'
 
   # The review app URL
-  app_url = "http://#{@docs_branch}.#{ENV["DOCS_REVIEW_APPS_DOMAIN"]}/#{ee? ? 'ee' : 'ce'}"
+  app_url = "http://#{docs_branch}.#{ENV["DOCS_REVIEW_APPS_DOMAIN"]}/#{ee? ? 'ee' : 'ce'}"
 
   # Create the pipeline
   puts "=> Triggering a pipeline..."
-  pipeline = Gitlab.run_trigger(GITLAB_DOCS_REPO, ENV["DOCS_TRIGGER_TOKEN"], @docs_branch, { param_name => ENV["CI_COMMIT_REF_NAME"] })
+  pipeline = Gitlab.run_trigger(GITLAB_DOCS_REPO, ENV["CI_JOB_TOKEN"], docs_branch, { param_name => ENV["CI_COMMIT_REF_NAME"] })
 
   puts "=> Pipeline created:"
   puts ""
@@ -77,4 +90,8 @@ when 'deploy'
   trigger_pipeline
 when 'cleanup'
   remove_remote_branch
+else
+  puts "Please provide a valid option:
+  deploy  - Creates the remote branch and triggers a pipeline
+  cleanup - Deletes the remote branch and stops the Review App"
 end