gitlab: Upgrade to 8.8.9 + start automation of gitlab-backup

# Upgrade

    * GitLab Software + patches ported to GitLab 8.8.9;
    * Configs synced with upstream and SlapOS-ified;
    * Fix promises due to curl upgrade

@jerome @kazuhiko @kirr: if you have any comments, I would be glad to read them. There is a [demo instance](https://softinst69126.host.vifib.net) you can try.

Check on the demo instance for the Gitlab configuration:

- [x] migrations are all `up`, when running: `gitlab-rake db:migrate:status`
- [x] promises don't fail (including slow ones)
- [x] web UI seems fine
- [x] git clone/fetch/push works well
- [x] email notifications are sent

and Nexedi edition patches:

- [x] raw blobs download
- [x] private ones with gitlab-ci token `curl -v https://user:token@testinstance/kirr/test/raw/master/hello.txt`
- [x] applying patches in merge-request
- [x] merge as topic in merge-request
- [x] "Nexedi edition" being present on front page and on page footers with ICP
- [x] clone protocol is only HTTPS (not SSH)

No changes are made in gitlab-workhorse and gitlab-shell upstream this time, but Nexedi patches and a cherry-picked commit are applied in a merge request on gitlab-ce.

Note: in order to build the gitlab SR, you need to change gitlab-ce repository to my own, as it is not merged yet:
```
--- a/software/gitlab/software.cfg
+++ b/software/gitlab/software.cfg
@@ -115,7 +115,7 @@ git-executable = ${git:location}/bin/git
 [gitlab-repository]
 <= git-repository
 #repository = https://gitlab.com/gitlab-org/gitlab-ce.git
-repository = https://lab.nexedi.com/nexedi/gitlab-ce.git
+repository = https://lab.nexedi.com/iv/gitlab-ce.git
```

# Gitlab backup (pull automation)

The idea is to regularly do a gitlab-backup pull for now, so that later, we can add some code to configure the other gitlab instance (not in production) to pull the created backup repo.

/reviewed-on !124

slapos/recipe/wrapper.py

@@ -37,6 +37,7 @@ class Recipe(GenericBaseRecipe):
         wait_files = self.options.get('wait-for-files')
         environment = self.options.get('environment')
         parameters_extra = self.options.get('parameters-extra')
+        pidfile = self.options.get('pidfile')
 
         if not wait_files and not environment:
           # Create a simple wrapper as shell script
@@ -45,6 +46,7 @@ class Recipe(GenericBaseRecipe):
              command=command_line[0],
              parameters=command_line[1:],
              parameters_extra=parameters_extra,
+             pidfile=pidfile,
           )]
 
         # More complex needs: create a Python script as wrapper
@@ -72,5 +74,7 @@ class Recipe(GenericBaseRecipe):
              name=wrapper_path,
              command=python_script,
              parameters=[],
-             parameters_extra=parameters_extra) ]
+             parameters_extra=parameters_extra,
+             pidfile=pidfile,
+        )]
 

software/gitlab/gitlab-export.in

+#!{{ bash.location }}/bin/bash -e
+# export gitlab site via `gitlab-backup pull` to a "for-export" git-backup repository
+# gitlab-export <exportto-repo>
+#
+# NOTE gitlab-backup, gitlab-rails, ... all have to be on $PATH.
+#      which site to export is determined by which gitlab-rails is on $PATH.
+
+if [ "$#" -ne 1 ]; then
+    echo "Usage: gitlab-export <exportto-repo>" 1>&2
+    exit 1
+fi
+exportto_repo="$1"
+
+# create / setup export repository if it does not exist yet
+mkdir -p "$exportto_repo"
+
+cd "$exportto_repo"
+
+# verify we are in a git repository
+if ! git rev-parse --is-inside-git-dir ; then
+  git init --bare
+fi
+
+exec gitlab-backup pull

software/gitlab/gitlab-parameters.cfg

@@ -6,7 +6,7 @@
 #
 # TODO better autogenerate from ^^^ (?)
 #
-# (last updated for omnibus-gitlab 8.7.9+ce.1-0-gf589ad7)
+# (last updated for omnibus-gitlab 8.8.9+ce.0-g25376053)
 
 [gitlab-parameters]
 configuration.external_url              = http://lab.example.com

software/gitlab/instance-gitlab-export.cfg.in

+# GitLab instance + site export support
+[buildout]
+extends = {{ instance_gitlab_cfg }}
+# TODO + ${pbsready-export:output}
+
+parts +=
+    cron-entry-gitlab-backup
+
+# -export specific instance parameters
+[instance-parameter]
+# cron frequency for gitlab backup (default: every 4h)
+configuration.backup_frequency  = 0 */4 * * *
+
+[gitlab-backup-directory]
+recipe            = slapos.cookbook:mkdirectory
+srv               = ${buildout:directory}/srv
+backup            = ${:srv}/backup
+backup-gitlab.git = ${:backup}/backup-gitlab.git
+var               = ${buildout:directory}/var
+pid               = ${:var}/pid
+
+# instance exporter script
+[exporter]
+recipe        = slapos.cookbook:wrapper
+wrapper-path  = ${buildout:directory}/bin/gitlab-exporter
+command-line  = {{ xnice_repository_location }}/bin/xnice {{ gitlab_export }} ${gitlab-backup-directory:backup-gitlab.git}
+pidfile       = ${gitlab-backup-directory:pid}/gitlab-exporter.pid
+environment   =
+# XXX: `/usr/bin` has to be in the PATH environment variable to be able to use
+# `which` command in gitlab-backup, `chrt` in xnice, ...
+  PATH=/usr/bin:${buildout:directory}/bin:{{ coreutils_location }}/bin:{{ grep_location }}/bin:{{ tar_location }}/bin:{{ gzip_location }}/bin:{{ gopath_bin }}:{{ git_location }}/bin
+
+[cron-entry-gitlab-backup]
+<= cron-entry
+# run backup script on a regular basis (given as instance parameter)
+frequency = ${instance-parameter:configuration.backup_frequency}
+command	= ${exporter:wrapper-path}

software/gitlab/instance-gitlab.cfg.in

@@ -537,7 +537,7 @@ depend  =
 
 [promise-gitlab-workhorse]
 <= promise-byurl
-url     = --unix-socket ${gitlab-workhorse:socket}  http:/static.css
+url     = --unix-socket ${gitlab-workhorse:socket}  http://localhost/static.css
 
 
 # gitlab-workhorse logs to stdout/stderr - logs are handled by slapos not us
@@ -579,7 +579,7 @@ depend  =
 
 [promise-unicorn]
 <= promise-byurl
-url     = --unix-socket ${unicorn:socket}  http:/
+url     = --unix-socket ${unicorn:socket}  http://localhost/
 
 [promise-rakebase]
 recipe  = slapos.cookbook:wrapper
@@ -627,7 +627,7 @@ log     = ${sidekiq-dir:log}
 # NOTE see queue list here:
 # https://gitlab.com/gitlab-org/gitlab-ce/blob/master/Procfile
 # https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb
-# (last updated for omnibus-gitlab 8.7.9+ce.1-0-gf589ad7)
+# (last updated for omnibus-gitlab 8.8.9+ce.0-g25376053)
 [service-sidekiq]
 recipe  = slapos.cookbook:wrapper
 wrapper-path    = ${directory:service}/sidekiq

software/gitlab/instance.cfg.in

@@ -10,15 +10,17 @@ offline = true
 
 [switch-softwaretype]
 recipe = slapos.cookbook:softwaretype
-default = $${instance-gitlab.cfg:rendered}
-# TODO -export, -import, -pull-backup
+gitlab  = $${instance-gitlab.cfg:rendered}
+gitlab-export   = $${instance-gitlab-export.cfg:rendered}
+default = $${:gitlab}
+# TODO -import, -pull-backup
 
 
-[instance-gitlab.cfg]
+# macro: render instance-*.cfg from instance-*.cfg.in
+[instance-cfg]
 recipe  = slapos.recipe.template:jinja2
 mode    = 0644
-template= ${instance-gitlab.cfg.in:target}
-rendered= $${buildout:directory}/instance-gitlab.cfg
+rendered= $${buildout:directory}/$${:_buildout_section_name_}
 context =
     import os os
     import pwd pwd
@@ -33,12 +35,16 @@ context =
     raw bash_bin                    ${bash:location}/bin/bash
     raw bzip2_location              ${bzip2:location}
     raw bundler_4gitlab             ${bundler-4gitlab:bundle}
+    raw coreutils_location          ${coreutils:location}
     raw curl_bin                    ${curl:location}/bin/curl
     raw dcron_bin                   ${dcron-output:crond}
     raw git                         ${git:location}/bin/git
     raw git_location                ${git:location}
+    raw gitlab_export               ${gitlab-export:rendered}
     raw gitlab_workhorse            ${gitlab-workhorse:location}/gitlab-workhorse
+    raw gopath_bin                  ${gopath:bin}
     raw gunzip_bin                  ${gzip:location}/bin/gunzip
+    raw grep_location               ${grep:location}
     raw gzip_bin                    ${gzip:location}/bin/gzip
     raw gzip_location               ${gzip:location}
     raw logrotate_bin               ${logrotate:location}/usr/sbin/logrotate
@@ -48,7 +54,9 @@ context =
     raw postgresql_location         ${postgresql92:location}
     raw redis_binprefix             ${redis28:location}/bin
     raw ruby_location               ${bundler-4gitlab:ruby-location}
+    raw tar_location                ${tar:location}
     raw watcher_sigkill             ${watcher-sigkill:rendered}
+    raw xnice_repository_location   ${xnice-repository:location}
 
 # config files
     raw database_yml_in             ${database.yml.in:target}
@@ -64,3 +72,16 @@ context =
     raw resque_yml_in               ${resque.yml.in:target}
     raw smtp_settings_rb_in         ${smtp_settings.rb.in:target}
     raw unicorn_rb_in               ${unicorn.rb.in:target}
+
+    $${:context-extra}
+context-extra =
+
+[instance-gitlab.cfg]
+<= instance-cfg
+template= ${instance-gitlab.cfg.in:target}
+
+[instance-gitlab-export.cfg]
+<= instance-cfg
+template= ${instance-gitlab-export.cfg.in:target}
+context-extra =
+    raw instance_gitlab_cfg         $${instance-gitlab.cfg:rendered}

software/gitlab/software.cfg

@@ -14,9 +14,12 @@ extends =
     ../../component/nginx/buildout.cfg
 
 #   for instance
+    ../../component/coreutils/buildout.cfg
     ../../component/bash/buildout.cfg
+    ../../component/grep/buildout.cfg
     ../../component/bzip2/buildout.cfg
     ../../component/curl/buildout.cfg
+    ../../component/tar/buildout.cfg
     ../../component/gzip/buildout.cfg
     ../../component/dcron/buildout.cfg
     ../../component/logrotate/buildout.cfg
@@ -36,20 +39,29 @@ parts =
     gitlab-shell/vendor
     gitlab/vendor/bundle
     gitlab-workhorse
+    git-backup
 
 #   for instance
     instance.cfg
 
+# TODO stop using slapos.cookbook-develop once 571d6514 gets into released
+# slapos.cookbook egg
+    slapos-cookbook-develop
+
     slapos-cookbook
     eggs
 
     bash
     curl
     watcher-sigkill
+    gitlab-export
     gzip
     dcron-output
     logrotate
 
+[slapos.cookbook-repository]
+revision = 571d6514f7290e8faa9439c4b86aa2f6c87df261
+
 
 ############################
 #   Software compilation   #
@@ -113,15 +125,15 @@ git-executable = ${git:location}/bin/git
 <= git-repository
 #repository = https://gitlab.com/gitlab-org/gitlab-ce.git
 repository = https://lab.nexedi.com/nexedi/gitlab-ce.git
-# 8.7.X + NXD patches:
-revision = v8.7.9-10-g7728df66b90483dac467df95948a532783e782c8
+# 8.8.X + NXD patches:
+revision = v8.8.9-10-g967afbdc3a2d2bcc58ed31ca50d5293fa2a2e9ed
 location = ${buildout:parts-directory}/gitlab
 
 [gitlab-shell-repository]
 <= git-repository
 #repository = https://gitlab.com/gitlab-org/gitlab-shell.git
 repository = https://lab.nexedi.com/nexedi/gitlab-shell.git
-# gitlab 8.7 wants gitlab-shell 2.7.2
+# gitlab 8.8 wants gitlab-shell 2.7.2
 # 2.7.2 + NXD patches
 revision = v2.6.10-50-gfbca95be784816349abc5930324659151eca50d1
 location = ${buildout:parts-directory}/gitlab-shell
@@ -130,7 +142,7 @@ location = ${buildout:parts-directory}/gitlab-shell
 <= git-repository
 #repository = https://gitlab.com/gitlab-org/gitlab-workhorse.git
 repository = https://lab.nexedi.com/nexedi/gitlab-workhorse.git
-# gitlab 8.7 wants gitlab-workhorse 0.7.1
+# gitlab 8.8 wants gitlab-workhorse 0.7.1
 # 0.7.1 + NXD patches
 revision = v0.7.1-5-gd23a3247829fc3200e3dc784dcd57b5a0febac48
 location = ${buildout:parts-directory}/gitlab-workhorse
@@ -151,6 +163,52 @@ make-binary =
 make-targets= cd ${:path} &&
     ${:bundle} install --deployment  --without development test  mysql kerberos
 
+# directories and repositories required by gitlab-backup
+[gopath]
+directory = ${buildout:directory}/go.work
+src	= ${:directory}/src
+bin	= ${:directory}/bin
+
+[go-git-repository]
+<= git-repository
+repository = https://${:go.importpath}.git
+location = ${gopath:src}/${:go.importpath}
+
+[git2go-repository]
+<= go-git-repository
+go.importpath = github.com/libgit2/git2go
+# branch 'next' is required by git-backup
+revision = next-g53594d7581617dbae7bb5960b4ac5f0ff513c184
+
+[git-backup-repository]
+<= go-git-repository
+go.importpath = lab.nexedi.com/kirr/git-backup
+revision = 3ba6cf73ba224c40f67f1fb87c855b915eb91f58
+
+
+[git-backup]
+# install git2go, git-backup, gitlab-backup in <gopath>/bin
+recipe  = slapos.recipe.cmmi
+path    = ${git-backup-repository:location}
+configure-command = :
+make-binary =
+make-targets= cd ${git2go-repository:location}
+              && git submodule update --init
+              && make install
+              && install -d ${gopath:bin}
+              && go install ${git-backup-repository:go.importpath}
+              && cp -a ${git-backup-repository:location}/contrib/gitlab-backup ${gopath:bin}
+environment =
+  PATH=${cmake:location}/bin:${pkgconfig:location}/bin:${git:location}/bin:${golang16:location}/bin:${buildout:bin-directory}:%(PATH)s
+  GOPATH=${gopath:directory}
+
+[xnice-repository]
+# to get kirr's misc repo containing xnice script for executing processes
+# with lower priority (used for backup script inside the cron)
+<= git-repository
+repository = https://lab.nexedi.com/kirr/misc.git
+revision =  4073572ea700bf1b115f3a135aebebe5b3b824e4
+location = ${buildout:parts-directory}/misc
 
 # build needed-by-gitlab-shell gems via bundler
 # ( there is not vendor/ dir in gitlab-shell, so to avoid having buildout error
@@ -194,17 +252,26 @@ eggs    =
 recipe  = slapos.recipe.template
 url     = ${:_profile_base_location_}/instance.cfg.in
 output  = ${buildout:directory}/instance.cfg
-md5sum  = ef85f02c4f6070c586d773b859a2f4e2
+md5sum  = b99a99b161c0b292845002fc3fee50cd
 
-[watcher-sigkill]
+# macro: download a shell script and put it rendered into <software>/bin/
+[binsh]
 recipe  = slapos.recipe.template:jinja2
 template= ${:_profile_base_location_}/${:_buildout_section_name_}.in
 rendered= ${buildout:bin-directory}/${:_buildout_section_name_}
 mode    = 0755
-md5sum  = 2986dcb006dc9e8508ff81f646656131
 context =
     section bash    bash
 
+[watcher-sigkill]
+<= binsh
+md5sum  = 2986dcb006dc9e8508ff81f646656131
+
+[gitlab-export]
+<= binsh
+md5sum  = a7b32680e80f34276f0a32a5e22dad50
+
+
 
 # macro: download a file named as section name
 #
@@ -232,7 +299,7 @@ md5sum  = eb1230fee50067924ba89f4dc6e82fa9
 
 [gitlab-parameters.cfg]
 <= download-file
-md5sum  = 3edd435a984b51b94539ea1d4f1b3994
+md5sum  = a74670934ec0190cc212d1f3468c11ed
 
 [gitlab-shell-config.yml.in]
 <= download-template
@@ -244,11 +311,15 @@ md5sum  = a9cb347f60aad3465932fd36cd4fe25d
 
 [gitlab.yml.in]
 <= download-template
-md5sum  = 735a78d0733fd6617d3b5f3d91bfae8c
+md5sum  = 176939a6428a7aca4767a36421b0af2b
 
 [instance-gitlab.cfg.in]
 <= download-file
-md5sum  = ba83f01fd7a313b984766c50d6e48e54
+md5sum  = 89914e4a225f6cdebfa196d46359f6f2
+
+[instance-gitlab-export.cfg.in]
+<= download-file
+md5sum  = a295e524046a9e2af569ad4b55f210cd
 
 [macrolib.cfg.in]
 <= download-file
@@ -260,11 +331,11 @@ md5sum  = a11b50d2ff2b1fa842ba4aa20041e2fe
 
 [nginx.conf.in]
 <= download-template
-md5sum  = 7da68dba86fff79eb93c27aa1aaf1055
+md5sum  = 1374f38ab6f295b850d45ea0019ec05d
 
 [rack_attack.rb.in]
 <= download-template
-md5sum  = bc1a7c1e83b7329d97bff6724f2bec3e
+md5sum  = 7d0e6dc6b826f6df6b20d8574a29e2f8
 
 [resque.yml.in]
 <= download-template
@@ -272,7 +343,7 @@ md5sum  = 7c89a730889e3224548d9abe51a2d719
 
 [smtp_settings.rb.in]
 <= download-template
-md5sum  = d66a424516ffacea34303e2f512a7d94
+md5sum  = c67ea492e17f774d0e18f1217338a55f
 
 [unicorn.rb.in]
 <= download-template

software/gitlab/template/gitlab.yml.in

@@ -2,7 +2,7 @@
 # see:
 # https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example
 # https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/gitlab.yml.erb
-# (last updated for omnibus-gitlab 8.7.9+ce.1-0-gf589ad7)
+# (last updated for omnibus-gitlab 8.8.9+ce.0-g25376053)
 
 {% from 'macrolib.cfg.in' import cfg, cfg_https, external_url  with context %}
 
@@ -86,6 +86,7 @@ production: &base
       wiki:             {{ cfg('default_projects_features.wiki') }}
       snippets:         {{ cfg('default_projects_features.snippets') }}
       builds: false {# builds not supported yet <%= @gitlab_default_projects_features_builds %> #}
+      {# container_registry: <%= @gitlab_default_projects_features_container_registry %> #}
 
     ## Webhook settings
     # Number of seconds to wait for HTTP response after sending webhook HTTP POST request (default: 10)
@@ -145,6 +146,18 @@ production: &base
     storage_path: <%= @lfs_storage_path %>
   #}
 
+  {# we do not support container registry
+  ## Container Registry
+  registry:
+    enabled: <%= @registry_enabled %>
+    host: <%= @registry_host %>
+    port: <%= @registry_port %>
+    api_url: <%= @registry_api_url %> # internal address to the registry, will be used by GitLab to directly communicate with API
+    path: <%= @registry_path %>
+    key: <%= @registry_key_path %>
+    issuer: <%= @registry_issuer %>
+  #}
+
   {# we do not support Pages
   ## GitLab Pages (EE only)
   pages:

software/gitlab/template/nginx.conf.in

@@ -2,7 +2,7 @@
 # see:
 # https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/support/nginx/gitlab-ssl
 # https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/nginx.conf.erb
-# (last updated for omnibus-gitlab 8.7.9+ce.1-0-gf589ad7)
+# (last updated for omnibus-gitlab 8.8.9+ce.0-g25376053)
 
 {% from 'macrolib.cfg.in' import cfg  with context %}
 
@@ -50,9 +50,11 @@ http {
 
   include {{ nginx_gitlab_http_conf }};
 
-  {# we don't need: ci, pages, mattermost
+  {# we don't need: ci, pages, mattermost, registry
   include <%= @gitlab_ci_http_config %>
   include <%= @gitlab_pages_http_config %>;
   include <%= @gitlab_mattermost_http_config %>
+  include <%= @gitlab_registry_http_config %>;
   #}
+
 }

software/gitlab/template/rack_attack.rb.in

@@ -2,7 +2,7 @@
 # see:
 # https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/initializers/rack_attack.rb.example
 # https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/rack_attack.rb.erb
-# (last updated for omnibus-gitlab 8.7.9+ce.1-0-gf589ad7)
+# (last updated for omnibus-gitlab 8.8.9+ce.0-g25376053)
 
 {% from 'macrolib.cfg.in' import cfg  with context %}
 
@@ -24,8 +24,9 @@ paths_to_be_protected = [
 # Create one big regular expression that matches strings starting with any of
 # the paths_to_be_protected.
 paths_regex = Regexp.union(paths_to_be_protected.map { |path| /\A#{Regexp.escape(path)}/ })
+rack_attack_enabled = Gitlab.config.rack_attack.git_basic_auth['enabled']
 
-unless Rails.env.test?
+unless Rails.env.test? || !rack_attack_enabled
   Rack::Attack.throttle('protected paths', limit: {{ cfg('rate_limit_requests_per_period') }}, period: {{ cfg('rate_limit_period') }}.seconds) do |req|
     if req.post? && req.path =~ paths_regex
       req.ip

software/gitlab/template/smtp_settings.rb.in

@@ -2,7 +2,7 @@
 # see:
 # https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/initializers/smtp_settings.rb.sample
 # https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/smtp_settings.rb.erb
-# (last updated for omnibus-gitlab 8.7.9+ce.1-0-gf589ad7)
+# (last updated for omnibus-gitlab 8.8.9+ce.0-g25376053)
 
 {% from 'macrolib.cfg.in' import cfg, cfg_bool  with context %}