Also, stop poping it: nothing else seems to be accessing the containing
dict, so mutating it does not seem necessary.

When the outside world path does not match the Zope path (typically: Web
Site).

slapgrid is the old name of slapos node command.

Remove unused "plugin" entry.
Add locally-referenced-but-defined-by-another-file "bin".

Should have been part of:
  commit c787d04c
  Author: Hardik Juneja <hardik.juneja@nexedi.com>
  Date:   Thu Jun 8 18:39:28 2017 +0200

      remove unnecessary monitor_template section form erp5 stack

Since bd3c2b18 (Remove httpd from ERP5 and use haproxy instead, 2020-12-14)
timing in balancer access log files are in milliseconds, no longer in
microseconds as they use to be with apache.

We update apachedex to version 1.8, where it supports generating reports
from access log files with milliseconds, as long as the expected format is
passed with --logformat argument.

We changed the default value of apachedex configuration to use the expected
--logformat argument, but users with a custom apachedex config passed as
instance parameters (monitor.apachedex-configuration) will have to update
their request parameters, to also pass --logformat.
Since this is breaking change, we take this opportunity to introduce another
breaking change, this parameter is not longer a string with all arguments
on the same line, which was problematic because these arguments contained
quotes and backslashes. The arguments are now passed as a list of strings
which will become the list of arguments passed to apachedex.

This is implemented by generating an apachedex config file, which simplifies
the generation of the wrapper. For this we also needed to upate
slapos.toolbox.

The same change is reflected to the overriden profiles in slapos-master,
but for slapos-master we don't pass the --logformat argument, because
slapos-master is using httpd, so the logs timings are still in microseconds.

When root instance is updated to a new software release URL, it will re-request
all the instances with the new software release URL.
To make sure the new root instance does not appear has ready when it is
re-requested with new software release URL, introduce a promise that will check
that the instances requested by the root instance have the same software
release URL. For now we do this only for Zope instances, because they are
stateless and restart automatically on configuration changes, unlike stateful
instances like mariadb or ZEO that we don't restart automatically (yet ?).

We are using this pattern for most of our services since several
months without any issue, so let's also use it for zopes. This
makes automatic upgrade possible.

Also remove "zope running current products" promise, since we
restart we no longer need to check this.

Fixes compatibility with glibc >= 2.30 .
Update users to new usage pattern.

haproxy can be controlled with this socket, so it might be useful
to "expose" it - it's not really expose because we only use a
UNIX socket.

The apachedex reports when produced on backend will be wrong, because haproxy
logs timings in milliseconds and apachedex parses as microsecond, but as far as
I know we produce reports from frontend logs, so it should not really affect
our operations.

Two main differences of haproxy are file format for certificates and logs.

HAProxy also uses certificates in PEM format, but it expect its own server
certificate and the key to be in the same file (although recent version seems
to accept separate files, we don't use this now) and the CRL and CA certificates
also all together in the same file.
We change to use the same file for certificate and key and for CA and CRL, in
the updater script we we build PEM files by containing all CA certificates and
all CRL together.
Also, since haproxy needs to be reloaded when certificate change, we run it in
master-worker mode, with a pid file so that we can signal it to reload.

For the logs, since haproxy does not log to file, we introduce a rsyslogd to
log to a file. The log format is same as with httpd, except that timing are not
in microseconds but in milliseconds - this did not seem to be configurable.
This is a problem for apachedex reports on log, for that we plan to use an
updated version of apachedex with support for `%{ms}T` for durations.

HAProxy is configured with same timeouts, except:
 - "connect" timeout has been increased a bit (from 5 to 10s), because the
   comment "The connection should be immediate on LAN" was no longer true, now
   that haproxy is accessed from frontend.
 - the server entries for testrunner are a very long timeout (8h) because some
   ERP5 functional tests exceeed the 305s timeout.

The SSL configuration is with current "modern" config from https://ssl-config.mozilla.org/

Tests have been modified a bit, because haproxy uses HTTP/2.0 and not 1.1
like httpd was doing several haproxy features (keep alive and gzip
compression) are only available when backend uses HTTP/1.1, so we adjusted
tests to use a 1.1 backend.

There was also differences with logs, because of the time being in milliseconds.

TestPublishedURLIsReachableMixin._checkERP5IsReachable was also updated, it
was working by chance because when accessed behind httpd->haproxy->zope, zope
was producing a redirect URL that was the URL of haproxy, which could be
resolved by chance. This test was updated to access zope with a path that
contains VirtualHostMonster magic, as the shared frontend ( with "zope" software
type) is supposed to set.

This should hopefuly solve the "502 Proxy Error" that we are observing with httpd.

Revert "software/erp5: use a caucase managed certificate for balancer"

This reverts commit 74d18b9d and also follow
up fixup ( 555b26a2 ).

We are not ready to use caucase here, there are still too many problems with
caucase (keys are lost at each SR update etc) and design might still evolve,
so let's go back to self signed certificate for now.

Also remove the promise and the updater, since they are also not working and
causing problems on instances that have been updated (and where the key no
longer match the certificate)

fix balancer CSR generation:

Caucase rerequest uses a CSR *template* and use it to generate
a new CSR with a new key, so we should not use the actual key to
generate this CSR, because it is caucase rerequest job to generate
the key.
Also, we should be careful not to generate a new CSR every time this
command run, otherwise a new key will be generated and a new CSR will
be sent to caucase, but caucase will not sign it automatically (since
we configure it to sign only one certificate).

This means that the case of IP address changes is currently not
supported automatically. To support it we would need to:
  - force generation of a new CSR template
  - force caucase rerequester to request a new certificate (by removing
  existing certificate)
  - force caucased to sign the new certificate

This commit also fix indentation and remove simplefile macro that is no longer used

Make the value and its changes easier to read.

Since 0.9.6 caucase stopped using the 128bits OID arc that caddy/golang does
not support, so nothing prevents us from using a caucase certiciate now.

We don't plan using varnish like this.

Starting from slapos.toolbox 0.110, this promise will report error if the the
amount of slow queries is over the threshold - before this version due to bug
in slapos.toolbox it was not - to keep this behavior and prevent too many
errors with this promise that is still in early stages of development, we
disable this promise unless it is explicitly requested in instance parameters.

These reports are large, no need to keep them as text

done with:

     npm install
     git ls-files | grep -v '\/unstable\/' | xargs ./node_modules/.bin/eclint fix
     git ls-files | grep buildout.hash.cfg | xargs ./update-hash

stack/erp5: put a magic parameter to use the first entry of X-Forwarded-For as the source IP address.

because we anyway has never used log, but just use stderr ourput.

in HAProxy 2.0, httpclose has the same meaning as forceclose, that is deprecated.

This is a loop invariant.

Mariadb was not shared, because installing mroonga writes a plugin in
mariadb's plugin dir and it's not allowed for one part to write in
another part's folder.

The approach is to install mroonga plugin in it's own plugin directory,
then copy all mariadb default plugins in this plugin directory and
configure instance to use mroonga's plugin directory.

Groonga also has plugins and we are using groonga-normalizer-mysql
plugin. Fortunately, groonga reads plugins located in paths listed in
GRN_PLUGINS_PATH environment variable, so we can use a simpler approach
of installing plugins in their own installation folder and set
GRN_PLUGINS_PATH in the environment of processes using groonga, ie. the
mariadb server process.

We want to wait for mysql_upgrade pseudo-service to have created users
and installed groonga, not just that something listen on the port.

Instead of writing the instructions ourselves, source mroonga provided
install script.

This reverts commit 4f4354c3.
It duplicated commit d13b42b7.

As possible cover promises in tests.