This needs a recent enough erp5, with the following commit:
zopewsgi: configure threads count and large_file_threshold with argparse

haufe.requestmonitoring already supports WSGI and Python3 and seems
used and maintainted.

For ZEO4, this was implemented as a monkey patch in ERP5, but this is
now supported in ZEO5, but not enabled by default.

The setting is only avaiable through ZEO's own ZConfig, but not with
ZODB's <zeoclient> so we also adjust the config to import ZEO and use
ZEO's <clientstorage>.

SIGUSR2 for reopening log files was handled by ZServer and waitress does not
handle signals at all, so we could handle this signal but Zope official
documentation recommends using logrotate `copytruncate`:
  https://zope.readthedocs.io/en/4.7/zopebook/MaintainingZope.html#log-rotation
  https://github.com/zopefoundation/Zope/issues/809

We have two public cloudooo instances, https:///cloudooo.erp5.net/ and
https://cloudooo1.erp5.net/ but the software release uses only the first
one. This changes to use both instances by default, in a random order.
This happens for the ERP5 instance and also for the test runner.

This also includes a breaking change: the parameter is renamed from
`cloudooo-url` into `cloudooo-url-list` and instead of being a coma
delimited list of URLs, this is now a proper list of URLs.

Some obsolete code and comments about the old `erp5-cloudooo` partition
that used to part of ERP5 software release have also been removed.

On testing side, ZopeSkinsMixin._setUpClass is changed be made more
robust, by waiting that all activities are processed and using a new
xmlrpc client in every iteration ( to prevent issues like the one from
https://erp5js.nexedi.net/#/test_result_module/20230107-548523A7/10 )

The NEO/go commit kirr/neo@8c974485
changed the URL with which a NEO/go client can be loaded from

    // neo://name@master1,master2,...,masterN?options

to

    neo(s)://[credentials@]master1,master2,...,masterN/name?options

We need to apply this change in the NEO URL structure to instance-wcfs.cfg.in
to use WCFS with NEO storage.

Why?
====

The URL in instance-wcfs.cfg.in is finally send to wcfs binary, which
will open a NEO/go client with this URL via the "openClientByURL" function [1].
If the URL formatting instance-wcfs.cfg.in follows the old pattern,
NEO/go is unable to create a NEO/go client and may even raise an
exception [2], because information regarding NEO server is at the wrong
position of the URL.

---

[1] See https://lab.nexedi.com/kirr/neo/blob/87199da2b163c09ed3946b7ab9bb00a5b987d377/go/neo/client.go#L416-534
for openClientByURL.

[2] For instance when using the old pattern of putting the cluster name
in the "user" part, NEO/go will raise "credentials can be specified only
with neos:// scheme", because NEO/go reserves the "user" part for
encryption information,
see https://lab.nexedi.com/kirr/neo/blob/87199da2b163c09ed3946b7ab9bb00a5b987d377/go/neo/client.go#L449

---

/reviewed-by @kirr
/reviewed-on nexedi/slapos!1307

using link-binary is enough

Direct logs to instance's var/log and call postfix's logrotate command,
instead of the slapos-hostile syslog.
Unfortunately, rotated logs will stay in that directory, unlike the
de-facto standard for slapos.

Resolve warnings emitted by postfix commands themselves.

  The content of inline is a jinja2 template

Run wait_for_activities macro in selenium test on erp5 with activity subscribed may end with:

INFO SQLBase Got a lock error, retrying...
.....
WARNING waitress.queue Task queue depth is 1
WARNING waitress.queue Task queue depth is XXX
....
503 Service Unavailable

This repairs a NameError introduced in 397726e1 (erp5/neoppod/slapos:
Support for python3 from aurel/zope4 branch., 2022-02-03) that was
causing ERP5-IOS test suite to fail in loop.

When parameters are built by instance-erp5, we don't need to use .get,
because the key is supposed to be here - and if it's not here it would
be because of a bug so it's better to let the error propagate.

Instead of having an hardcoded timeout that users will hit anyway even
if they increase publisher timeout, set this timeout value to be
slightly higher than publisher timeout. This way publisher-timeout can
be used to allow longer requests and it's generally more consistent.

We already had haproxy configured to stop serving requests longer than
this duration, but the requests was still being processed by zope.

By adding such default in publisher-timeout, zope will stop processing
the request after the timeout.

This is slightly different behavior, because before this change the
request was still being processed, even if the client did not get the
response it may make change to databases.
Users who want to keep the previous behavior can request with
publisher-timeout parameter null.

Use slapos.recipe.template instead of slapos.recipe.template if possible

Harmonize template keys :
rendered -> output
template -> url
template = inline: -> inline =

Delete "mode" key

See merge request nexedi/slapos!1151

This was useless. dumps is enough.

Without this we sometimes hit the socket path length limit when testing
ERP5 inside Theia inside erp5testnode.

With ERP5 commit nexedi/erp5!1529
timerserver is started from wsgi startup script and the config in zope.conf
only applies to the medusa/ZServer mode.

Since ERP5 commit 8f3b77517e (Force use of the new TimerServer, 2019-09-13)
the timerserver egg is not longer used, so it's not needed to install
and configure it.

In ubuntu 21.04 (libc6 2.33-0ubuntu5) or current debian testing
(libc6 2.32-5) no longer use open to open /etc/hosts, but what appears
as "openat" in strace output - but can not be replaced by defining an
openat function.

This uses https://github.com/figiel/hosts which uses another approach
of replacing getaddrinfo, gethostbyname* and inet_aton.

Users have been updated a bit, because there are some small differences:
 - the /etc/hosts replacement file is defined by HOSTS_FILE environment
   variable, not HOSTS
 - the library name is libuserhosts.so, not userhosts.so

Other notable differences, for which we did not need code change are:
 - the new library also try to load a file when HOSTS_FILE is not set
 - the new library still use original /etc/hosts file
 - the new library supports aliases to hostnames, not only ip addresses

Since ERP5 commit d4eda7ea49 (CMFActivity: show InnoDB history list
length in console watcher, 2020-09-15), the wrapper script no longer work
because accessing history list length require PROCESS privileges.

In the case of ERP5, mariadb database is dedicated for ERP5, so we can
grant the PROCESS privilege also to erp5 default user, which is used in
the wrapper script.

We don't grant the permission to test users because they don't need it.

5s is the historical default tic periodicity, but 1s is typically used in
production instances. 1s periodicity should improve validation node
reactivity in all cases, and general activity reactivity in instances with
few processing nodes (like development instances).

By setting this in the test suite configuration, we can force ERP5 tests
to use random priorities for activity processing.

The intended usage is to set this to an empty string, in which case
ERP5's runUnitTest chooses a random value for the seed and prints the
chosen value. It may also be possible to use a value, to reproduce a
previous run.

Adapt all promise sections to changes in plugin promise recipe.

This change has no effect on stand-alone setups (which is what is
automatically deployed).
On setups configured to replicate from another database, it allows them to
takeover the role of the primary server, which allows rebuilding a
redundant setup. On such setup, the cost of this option is the extra I/O
load of binlogs being written to (in addition to the relay log and
database), and the corresponding disk space use.

This was introduced recently, in 0aecb41a (software: Use local tmp for
caucase, 2021-05-18). This is harmless, but uses a weird-looking directory

Change all `check_port_listening` promises to `check_socket_listening`
and rename all associated `config-hostname` arguments to `config-host`