See merge request !1603

This is mostly to fix some flaky tests, because with the previous
approach of just sending a signal for haproxy to reload the config
asynchronously we don't know when haproxy has reloaded.

- update zope to 5.11.1
- update zodbtools to current master ( `0.0.0.dev9-21-gb0fdb5f` )

See merge request !1670

Prevents error like this in the log for ZODB with a tmep_folder

2024-10-04 14:14:53,288 ERROR Zope.ZODBMountPoint Failed to mount database. <type 'exceptions.ImportError'> (No module named TemporaryFolder)
Traceback (most recent call last):
  File "./eggs/Products.ZODBMountPoint-1.3-py2.7.egg/Products/ZODBMountPoint/MountedObject.py", line 255, in _getOrOpenObject
    obj = self._traverseToMountedRoot(root, parent)
  File "./eggs/Products.ZODBMountPoint-1.3-py2.7.egg/Products/ZODBMountPoint/MountedObject.py", line 213, in _traverseToMountedRoot
    obj = blazer.traverseOrConstruct(real_path)
  File "./eggs/Products.ZODBMountPoint-1.3-py2.7.egg/Products/ZODBMountPoint/MountedObject.py", line 81, in traverseOrConstruct
    container = self._construct(container, part)
  File "./eggs/Products.ZODBMountPoint-1.3-py2.7.egg/Products/ZODBMountPoint/MountedObject.py", line 108, in _construct
    klass = jar.db().classFactory(jar, self.module_name, self.class_name)
  File "./eggs/Zope-4.8.7-py2.7.egg/Zope2/Startup/datatypes.py", line 288, in simpleClassFactory
    m = __import__(module, _globals, _globals, _silly)
ImportError: No module named TemporaryFolder

This ease the development and version updates

This is not needed because we are using a closing context manager

pygolang: Accompany python-interpreter with gpython-interpreter to run gpython instead of std python

With python-interpreter staying the default for pyprog, but possible to adjust.

We will need gpython-interpreter for ERP5 to use bstr+ustr as builting string types.

Remove [gpython] script in favour of gpython-interpreter since both provide the
same thing but via slightly different ways internally ([gpython] was generating
bin/gpython via setuptools and [gpython-interpreter] does it by hand with
user-visible result staying the same).

/cc @kazuhiko, @jerome

pyprog macro, added in 0ee52376 (Generalize how nxdtest python script is
generated into pyprog recipe macro) works by generating buildout code at
runtime in text form. This way it should be careful when substituting strings,
because if those strings contain \n, then intended control flow might become broken.

For example when using pyprog with eggs = ${eggs:eggs} from stack/erp5/ ,
buildout breaks because erp5 defines eggs as multiline list:

    INFO     self.buildout.parse("""
    INFO   File "/srv/slapgrid/slappart47/srv/runner/software/7f1663e8148f227ce3c6a38fc52796e2/eggs/zc.buildout-2.7.1+slapos020-py3.9.egg/zc/buildout/buildout.py", line 1352, in parse
    INFO     sections = zc.buildout.configparser.parse(
    INFO   File "/srv/slapgrid/slappart47/srv/runner/software/7f1663e8148f227ce3c6a38fc52796e2/eggs/zc.buildout-2.7.1+slapos020-py3.9.egg/zc/buildout/configparser.py", line 241, in parse
    INFO     raise e
    INFO zc.buildout.configparser.ParsingError: File contains parsing errors:
    INFO         [line 18]: 'python-barcode\n'
    INFO         [line 19]: 'SOAPpy-py3\n'
    INFO         [line 20]: 'suds-py3\n'
    INFO         [line 21]: 'neoppod[admin, ctl, master]\n'
    INFO         [line 22]: 'cython-zstd\n'
    INFO         [line 23]: 'msgpack\n'
    INFO         [line 24]: 'mysqlclient\n'
    INFO         [line 25]: 'PyMySQL\n'
    INFO         [line 26]: 'ZODB\n'
    INFO         [line 27]: 'zodbtools\n'
    INFO         [line 28]: 'psutil\n'
         ...

-> Fix it via indenting eggs list like we already do with pyinit code.

/cc @kazuhiko, @jerome

This have been a regression when upgrading to v0.40.0

We can not use the new theia parameter from 4bb65573 (software/theia: new
testing-short-embedded-instance-path parameter for tests, 2024-11-04),
because ERP5 Software release still uses some slapos.toolbox utility
hardcoding srv/runner/instance/slappart\d

https://lab.nexedi.com/nexedi/slapos.toolbox/-/blob/8fc29341d366f17544a56d7f6c8bd3716055a128/slapos/resilient/identity_script_excluding_path.py#L25

Instead, just keep using normal long paths, because they were not too
long for ERP5 in theia in testnode

These queries are too slow.

See merge request nexedi/slapos!1677

Create embedded instance with shorter paths, for resilience project
tests. This is especially needed for gitlab, which uses a lot of
sockets and a directory layout that's too deep for the default layout
of theia's embedded instances.

and small import cleanups

since a1bfe616 (software/gitlab: upgrade to version 12.10.14,
2024-01-09) we no longer manually install ajv

Instead of using a list of frontends IP addresses to determine if the backend can trust the frontend's `X-Forwarded-For` header, use the same [`authenticate-to-backend`](https://lab.nexedi.com/nexedi/slapos/-/blob/d48d682dfc67d7845f0346f01772573c9e4edc8e/software/rapid-cdn/instance-slave-input-schema.json#L215-223) approach as with ERP5: the frontend connects to the backend with a client certificate and if the backend can verify this certificate, it trusts `X-Forwarded-For` from the frontend and uses this as client IP.
Otherwise, without a verified certificate, the frontend's own IP address is uses as client IP.

This means that:
 - frontend shared instances must use `authenticate-to-backend` in parameters
 - gitlab instance must use `frontend-caucase-url-list` in parameters
 - gitlab instance no longer use `nginx_real_ip_trusted_addresses` in parameters

This branch also contains some mitigation for 503 errors we observed when too many clients were downloading archives (we had several hundreds of ongoing requests preparing archives), the approach is simply to rate-limit the download archives, implemented in nginx because gitlab does not expose rack-attack configuration for this.

See merge request nexedi/slapos!1676

nginx is not really flexible for this, but since gitlab does not make
download of archive configurable, this adds a rate limit of 1 request
per minute per source IP for archive downloads.

Introduces a new instance parameter, frontend-caucase-url-list which is
a space separated list of IP addresses (this software still uses xml
serialisation and does not have a parameter schema yet).

In 38f5053c the image has been added without
MD5SUM, and it was not stopped during merging
nexedi/slapos!1655

Fix by adding the missing MD5SUM.

Because re6stnet now uses hatchling, it needs some packages installed
for develop step.

ref: nexedi/re6stnet@88a883db

See merge request nexedi/slapos!1668

See merge request nexedi/slapos!1668

See merge request !1668

See merge request nexedi/slapos!1668