pyprog macro, added in 0ee52376 (Generalize how nxdtest python script is
generated into pyprog recipe macro) works by generating buildout code at
runtime in text form. This way it should be careful when substituting strings,
because if those strings contain \n, then intended control flow might become broken.

For example when using pyprog with eggs = ${eggs:eggs} from stack/erp5/ ,
buildout breaks because erp5 defines eggs as multiline list:

    INFO     self.buildout.parse("""
    INFO   File "/srv/slapgrid/slappart47/srv/runner/software/7f1663e8148f227ce3c6a38fc52796e2/eggs/zc.buildout-2.7.1+slapos020-py3.9.egg/zc/buildout/buildout.py", line 1352, in parse
    INFO     sections = zc.buildout.configparser.parse(
    INFO   File "/srv/slapgrid/slappart47/srv/runner/software/7f1663e8148f227ce3c6a38fc52796e2/eggs/zc.buildout-2.7.1+slapos020-py3.9.egg/zc/buildout/configparser.py", line 241, in parse
    INFO     raise e
    INFO zc.buildout.configparser.ParsingError: File contains parsing errors:
    INFO         [line 18]: 'python-barcode\n'
    INFO         [line 19]: 'SOAPpy-py3\n'
    INFO         [line 20]: 'suds-py3\n'
    INFO         [line 21]: 'neoppod[admin, ctl, master]\n'
    INFO         [line 22]: 'cython-zstd\n'
    INFO         [line 23]: 'msgpack\n'
    INFO         [line 24]: 'mysqlclient\n'
    INFO         [line 25]: 'PyMySQL\n'
    INFO         [line 26]: 'ZODB\n'
    INFO         [line 27]: 'zodbtools\n'
    INFO         [line 28]: 'psutil\n'
         ...

-> Fix it via indenting eggs list like we already do with pyinit code.

/cc @kazuhiko, @jerome

We can not use the new theia parameter from 4bb65573 (software/theia: new
testing-short-embedded-instance-path parameter for tests, 2024-11-04),
because ERP5 Software release still uses some slapos.toolbox utility
hardcoding srv/runner/instance/slappart\d

https://lab.nexedi.com/nexedi/slapos.toolbox/-/blob/8fc29341d366f17544a56d7f6c8bd3716055a128/slapos/resilient/identity_script_excluding_path.py#L25

Instead, just keep using normal long paths, because they were not too
long for ERP5 in theia in testnode

These queries are too slow.

See merge request nexedi/slapos!1677

Create embedded instance with shorter paths, for resilience project
tests. This is especially needed for gitlab, which uses a lot of
sockets and a directory layout that's too deep for the default layout
of theia's embedded instances.

and small import cleanups

since a1bfe616 (software/gitlab: upgrade to version 12.10.14,
2024-01-09) we no longer manually install ajv

Instead of using a list of frontends IP addresses to determine if the backend can trust the frontend's `X-Forwarded-For` header, use the same [`authenticate-to-backend`](https://lab.nexedi.com/nexedi/slapos/-/blob/d48d682dfc67d7845f0346f01772573c9e4edc8e/software/rapid-cdn/instance-slave-input-schema.json#L215-223) approach as with ERP5: the frontend connects to the backend with a client certificate and if the backend can verify this certificate, it trusts `X-Forwarded-For` from the frontend and uses this as client IP.
Otherwise, without a verified certificate, the frontend's own IP address is uses as client IP.

This means that:
 - frontend shared instances must use `authenticate-to-backend` in parameters
 - gitlab instance must use `frontend-caucase-url-list` in parameters
 - gitlab instance no longer use `nginx_real_ip_trusted_addresses` in parameters

This branch also contains some mitigation for 503 errors we observed when too many clients were downloading archives (we had several hundreds of ongoing requests preparing archives), the approach is simply to rate-limit the download archives, implemented in nginx because gitlab does not expose rack-attack configuration for this.

See merge request nexedi/slapos!1676

nginx is not really flexible for this, but since gitlab does not make
download of archive configurable, this adds a rate limit of 1 request
per minute per source IP for archive downloads.

Introduces a new instance parameter, frontend-caucase-url-list which is
a space separated list of IP addresses (this software still uses xml
serialisation and does not have a parameter schema yet).

In 38f5053c the image has been added without
MD5SUM, and it was not stopped during merging
nexedi/slapos!1655

Fix by adding the missing MD5SUM.

Because re6stnet now uses hatchling, it needs some packages installed
for develop step.

ref: nexedi/re6stnet@88a883db

See merge request nexedi/slapos!1668

See merge request nexedi/slapos!1668

See merge request nexedi/slapos!1668

See merge request nexedi/slapos!1668

GDAL is a library, and thus it should not mess with global settings.
This was causing the logs to be flooded with deprecation messages.

See merge request nexedi/slapos!1674

The syntax to compare strings is

      STRING1 = STRING2
                     True if the strings are equal.
      STRING1 != STRING2
                     True if the strings are not equal.

STRING1=STRING2 is not a valid syntax for strings comparisons.

This is same fix as 6cf1769d (ERP5: fix handling of repozo restoration
failure, 2024-10-24) and also a fix to a wrong error message, because
this is not restoration script, it's backup script.

Fluentbit Tail input documentation[1] says that by default maximum buffer
size is 32K which turned out to be too small in practice because we hit
a situation where enb.xlog started to have lines with ~ 34K and so
fluentbit ingestion stopped to work with the following error in
fluentbit log:

    [2024/10/23 20:30:23] [error] [input:tail:tail.0] file=/srv/slapgrid/slappart19/srv/monitor/public/enb.xlog requires a larger buffer size, lines are too long. Skipping file.

-> Fix that by increasing max buffer size to 1M which seems to be high
   enough at least for now.

Maybe it will make sense to configure this as unlimited, but I'm not
sure if going as unlimited is universally a good idea.

[1] https://docs.fluentbit.io/manual/pipeline/inputs/tail

/cc @lu.xu, @jhuge, @tomo
/reviewed-by @paul.graydon
/reviewed-on nexedi/slapos!1672

As discussed on nexedi/slapos@bb841a7b (comment 219278)
when using storage-path and passwd option, the storage file could not
be updated to the new format because of AttributeError _needs_migration.

This changes to no longer try to detect if the storage needs migration,
but just compare the expected content of the storage file during install
and overwrite the file if it is different.

This new approach also fix a behavior that re-running buildout with
storage-path option and a different passwd option did not update the
storage file. Now it is also updated.

( this also fixes a potential encoding problem on py2 )