See merge request nexedi/slapos!1677

Create embedded instance with shorter paths, for resilience project
tests. This is especially needed for gitlab, which uses a lot of
sockets and a directory layout that's too deep for the default layout
of theia's embedded instances.

and small import cleanups

since a1bfe616 (software/gitlab: upgrade to version 12.10.14,
2024-01-09) we no longer manually install ajv

Instead of using a list of frontends IP addresses to determine if the backend can trust the frontend's `X-Forwarded-For` header, use the same [`authenticate-to-backend`](https://lab.nexedi.com/nexedi/slapos/-/blob/d48d682dfc67d7845f0346f01772573c9e4edc8e/software/rapid-cdn/instance-slave-input-schema.json#L215-223) approach as with ERP5: the frontend connects to the backend with a client certificate and if the backend can verify this certificate, it trusts `X-Forwarded-For` from the frontend and uses this as client IP.
Otherwise, without a verified certificate, the frontend's own IP address is uses as client IP.

This means that:
 - frontend shared instances must use `authenticate-to-backend` in parameters
 - gitlab instance must use `frontend-caucase-url-list` in parameters
 - gitlab instance no longer use `nginx_real_ip_trusted_addresses` in parameters

This branch also contains some mitigation for 503 errors we observed when too many clients were downloading archives (we had several hundreds of ongoing requests preparing archives), the approach is simply to rate-limit the download archives, implemented in nginx because gitlab does not expose rack-attack configuration for this.

See merge request nexedi/slapos!1676

nginx is not really flexible for this, but since gitlab does not make
download of archive configurable, this adds a rate limit of 1 request
per minute per source IP for archive downloads.

Introduces a new instance parameter, frontend-caucase-url-list which is
a space separated list of IP addresses (this software still uses xml
serialisation and does not have a parameter schema yet).

In 38f5053c the image has been added without
MD5SUM, and it was not stopped during merging
nexedi/slapos!1655

Fix by adding the missing MD5SUM.

Because re6stnet now uses hatchling, it needs some packages installed
for develop step.

ref: nexedi/re6stnet@88a883db

See merge request nexedi/slapos!1668

See merge request nexedi/slapos!1668

See merge request nexedi/slapos!1668

See merge request nexedi/slapos!1668

GDAL is a library, and thus it should not mess with global settings.
This was causing the logs to be flooded with deprecation messages.

See merge request !1674

The syntax to compare strings is

      STRING1 = STRING2
                     True if the strings are equal.
      STRING1 != STRING2
                     True if the strings are not equal.

STRING1=STRING2 is not a valid syntax for strings comparisons.

This is same fix as 6cf1769d (ERP5: fix handling of repozo restoration
failure, 2024-10-24) and also a fix to a wrong error message, because
this is not restoration script, it's backup script.

Fluentbit Tail input documentation[1] says that by default maximum buffer
size is 32K which turned out to be too small in practice because we hit
a situation where enb.xlog started to have lines with ~ 34K and so
fluentbit ingestion stopped to work with the following error in
fluentbit log:

    [2024/10/23 20:30:23] [error] [input:tail:tail.0] file=/srv/slapgrid/slappart19/srv/monitor/public/enb.xlog requires a larger buffer size, lines are too long. Skipping file.

-> Fix that by increasing max buffer size to 1M which seems to be high
   enough at least for now.

Maybe it will make sense to configure this as unlimited, but I'm not
sure if going as unlimited is universally a good idea.

[1] https://docs.fluentbit.io/manual/pipeline/inputs/tail

/cc @lu.xu, @jhuge, @tomo
/reviewed-by @paul.graydon
/reviewed-on !1672

As discussed on bb841a7b (comment 219278)
when using storage-path and passwd option, the storage file could not
be updated to the new format because of AttributeError _needs_migration.

This changes to no longer try to detect if the storage needs migration,
but just compare the expected content of the storage file during install
and overwrite the file if it is different.

This new approach also fix a behavior that re-running buildout with
storage-path option and a different passwd option did not update the
storage file. Now it is also updated.

( this also fixes a potential encoding problem on py2 )

fix a SyntaxWarning on py3.9

See merge request !1664

This test is using two connection one with a client to subscribe to a
topic and wait for message and another one with publish.single to
publish to the topic.
The test was failing from time to time because the publish might have
happened after the client was subscribed.

Refactor the test to use `loop` on the client to have more control
and be able to wait for the client to be subscribed using the
`on_subscribe` callback.

The test is also factorized, instead of having the same test twice for
IPv4 and IPv6, we pass the host as parameter.

See merge request !1665

from repozo doc:

> If a full backup is created, remove any prior full or incremental
> backup files (and associated metadata files) from the repository
> directory.

This solves a problem that after a pack some old repozo files were left
around, with this option they are automatically removed.