slaprunner: fix resiliency test, add development-mode (bool) parameter for easy build webrunner in webrunner

This is backported to ZODB-3.10 https://github.com/zopefoundation/persistent/pull/44

The patch is needed for wendelin.core to being able to relase not needed
memory.

/reviewed-on nexedi/slapos!93

This reverts commit 605e564b.

Rationale: Stability matters:

Quoting 605e564b:
> Besides changing only recv window size at runtime breaks compatibility with
> openssh: if we only do `-W 1M` on server and try to upload data with openssh as
> client, dropbear complains
>
>     [3302] Apr 17 23:10:06 Exit (slapuser2): Bad packet size 32777
>
> and connection terminates. Thus RECV_MAX_PAYLOAD_LEN increase is also
> required, which cannot be done via option at runtime:
>
>     https://github.com/mkj/dropbear/blob/DROPBEAR_0.53.1/options.h#L268
>
>     ---- 8< ----
>     /* Maximum size of a received SSH data packet - this _MUST_ be >= 32768
>        in order to interoperate with other implementations */
>     #ifndef RECV_MAX_PAYLOAD_LEN
>     #define RECV_MAX_PAYLOAD_LEN 32768
>     #endif
>     ---- 8< ----
>
> So let's increase DEFAULT_RECV_WINDOW to 1M and RECV_MAX_PAYLOAD_LEN
> appropriately (experimentally found that at 512K the complain goes
> away).

It turned out that "Bad packet size" did not really went away. For example I've
recently hit the following:

    [14586] Aug 04 19:12:43 Pubkey auth succeeded for 'slapuser16' with key md5 b1:35:06:d3:a5:b1:0b:c6:7f:e6:59:31:ab:3a:e1:56 from 2001:67c:1254:c0::1:49886
    [14586] Aug 04 19:12:55 Exit (slapuser16): Integrity error (bad packet size 524500)

in .slappartX_runner_sshd.log of my upgraded webrunner with connection being broken.
( nexedi/slapos!68 (comment 17748) )

We could maybe try to play games with increasing RECV_MAX_PAYLOAD_LEN to
be more than DEFAULT_RECV_WINDOW but this already turned out to be error-prone.

Since when really needed we should be able to replace dropbear with openssh

    nexedi/slapos!68 (comment 7082)

which is both performant and good-compatible, to me the way is:

- make current dropbear run stable again,
- when we really need to sync large amounts of data (and we should be
  needing to do soon or already) -> work on replacing dropbear with
  openssh.

- GitLab Software + patches ported to GitLab 8.7.X;
- Configs synced with upstream;
- No base software upgrades this time because it was all recently
  upgraded during a590b03e;

TODO: allow configuration of trusted proxies

/reviewed-by TrustMe

Like for 2a835e63

    $ git diff 8.6.5+ce.0-0-g342f8be..8.7.9+ce.1-0-gf589ad7 --
            files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

is empty.

I've manually reviewed

    git diff 8.6.5+ce.0-0-g342f8be..8.7.9+ce.1-0-gf589ad7 --    \
        files/gitlab-config-template/gitlab.rb.template \
        files/gitlab-cookbooks/gitlab/attributes/default.rb

and modulo trusted proxies there are no interesting changes for us.

- config.ru template is gone - pristine gitlab-ce/config.ru can do the
  job because it obtains unicorn OOM killer setting via environment
  variables.

  https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/672

- we put TODO there for configuring trusted proxies (gitlab.yml & nginx)

- we restore our slaposified configuration from config.ru to unicorn.rb

This does almost(*) only pure merge. We will slaposify / adjust config
and corresponding md5sum in the following patches.

(*) smtp ssl option is only added as comment.

Update GitLab software to

    - gitlab-ce 8.7.9 + NXD patches

    - gitlab-shell to 2.7.2 + 1 patch to remove unneeded hooks.old in *.git

    - gitlab-workhorse stays at 0.7.1 + NXD patches because gitlab-ce 8.7.x
      sticks to this version (i.e. no workhorse upgrade for gitlab 8.6 -> 8.7)

This only updates software and begins SR update to 8.7 - for now gitlab
instance becomes non-working -- we'll pull in configuration files
updates and fixups in the following patches.

Like f6f97d72 - pristine copy from omnibus-gitlab 8.7.9+ce.1-0-gf589ad7

Changes are:

    - database.yml.erb

      * db_sslca option to specify CA for cases when DB is accessed via
        SSL (we do not need it as we access DB over unix:// only)

    - gitconfig.erb

      * turns gc.auto=0

        This is questionable to me. What they needed is to adjust
        warning reporting in git, not completely disable gc.auto and control it
        with their hands from rails.

        context: https://gitlab.com/gitlab-org/gitlab-ce/issues/14357

    - gitlab-rails-config.ru.erb removed

      with unicorn OOM killer settings moved to unicorn.rb. See:

      https://gitlab.com/gitlab-org/omnibus-gitlab/commit/cfbe6c55
      https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/672

    - gitlab.yml.erb

      * +geo_bulk_notify_worker (EE only, we do not use gitlab geo)
      * +repository_archive_cache_worker.cron   (gitlab-ce defaults to "0 * * * *")
        https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3663
      * +update_all_remote_mirrors_worker.cron  (EE only ?)
      * +omniauth.external_providers (we do not use omniauth)
      * +trusted_proxies

        this adds ability to let gitlab know trusted proxies addresses
        from which it can get and trust things like X-Forwarded-For and the
        like.

    - nginx-gitlab-http.conf.erb

      * add support for using nginx's realip module
        (http://nginx.org/en/docs/http/ngx_http_realip_module.html) for
        configuring trusted proxies and letting requests from them to
        pass through nginx with e.g. X-Forwarded-For header.

    - smtp_settings.rb.erb

      * +ssl option

        https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/730

    - unicorn.rb: see above about "gitlab-rails-config.ru.erb removed"

The following files stay the same:

    - gitlab-shell-config.yml.erb
    - nginx.conf.erb
    - rack_attack.rb.erb
    - resque.yml.erb

  CFFI is added on this directory, and it is required to bootstrap slapos.toolbox.

- GitLab Software + patches ported to GitLab 8.6.X;
- Configs synced with upstream;
- Base software upgraded where appropriate;
- misc adjustments.

Demo instance: https://softinst64196.host.vifib.net/

@jerome @kazuhiko @iv Please have a look. I've verified it works but there is always a chance one can miss some detail. If all ok I'd like to deploy this tomorrow (3 Aug) evening to lab.nexedi.com

Thanks beforehand for feedback,  
Kirill


/reviewed-on nexedi/slapos!92

Allows easier parameter input.

Starting from GitLab 8.6 pg_trgm extension becomes hard dependency of
gitlab.

    https://gitlab.com/gitlab-org/gitlab-ce/commit/d24ee2a2

The extension can be activated only by db superuser, so gitlab db
migration scripts does not activate it - it has to be done by DB
administrator or is handled by integrating code in omnibus case.

As we already handle DB setup and migrations in unicorn startup script,
as pre-action there, let's activate pg_trgm.

We'll need to invoke psql connected to gitlab db in another place, so
before doing it let's factor out the code to call psql as connected to a
separate function.

Like for 0a72505e

    $ git diff 8.5.1+ce.0-1-ge732b39..8.6.5+ce.0-0-g342f8be --
        files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

is empty.

I manually reviewed

    $ git diff 8.5.1+ce.0-1-ge732b39..8.6.5+ce.0-0-g342f8be --  \
        files/gitlab-config-template/gitlab.rb.template \
        files/gitlab-cookbooks/gitlab/attributes/default.rb

in omnibus-gitlab, and module proxy_cache and http2 changes, which we
already handled in 2 previous patches, there is nothing more interesting
for us.

Almost no changes this time: we only comment-out Nginx cache. See
details for why we do not need it in comments and in f6f97d72.

This does almost(*) only pure merge. We will slaposify / adjust config
and corresponding md5sum in the following patches.

(*) option to enable/disable HTTP/2 was in the same line as other nginx
    already jinja2'ified listen options.

    As already noted in f6f97d72 we are going to always support HTTP/2,
    that's why we do not merge-in upstream change only to through it away in
    the following patch.

Update GitLab software to

    - gitlab-ce 8.6.9 + NXD patches

      nexedi/gitlab-ce!1

    - gitlab-shell to 2.6.12 + 1 patch to remove unneeded hooks.old in *.git

      nexedi/gitlab-shell!1

    - gitlab-workhorse 0.7.1 + NXD patches.

      nexedi/gitlab-workhorse!1

      ( download speedup patches were reworked because of upstream changes.
	Please see details in the above MR and in fixup commits )

This only updates software and begins SR update to 8.6 - for now gitlab
instance becomes non-working -- we'll pull in configuration files
updates and fixups in the following patches.

The reason is: starting from GitLab 8.6 this extension becomes hard
dependency of GitLab.

References:

    https://about.gitlab.com/2016/03/22/gitlab-8-6-released/
    -> "Changes for Source installations with PostgreSQL"

    http://www.postgresql.org/docs/current/static/pgtrgm.html

NOTE

There is no way to activate only some extension building at configure
time - it is "all" or "all with all extensions" in postgresql speak (=
"world" make target).

PostgreSQL INSTALL explicitly suggests for selected-extensions install
to jump to appropriate dirs and do `make install` from there.

    http://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=contrib/README;h=5eaeb2451f29877e986f4683c57dd70edde942d5;hb=HEAD#l15

that's why we abuse slapos.recipe.cmmi a bit and do a double

    make install && make -C contrib/pg_trgm/ install