Prevent creating 2 wrapper for the same service if hash changed. Here, one service is exited because port is used by the firt to service to start:

    slappart6:runner-sshd-4248650e36a9a26a6481df1baffd9f58-on-watch                RUNNING   pid 27835, uptime 0:03:45
    slappart6:runner-sshd-b3b68f4278ceb84691ec27521ea229eb-on-watch                EXITED    Mar 06 04:52 PM

To achieve that, update slapos.cookbook and use hash-existing-files option of wrapper recipe

hash-existing-files list all the files used for hash that are not
handled by buildout. For those files, the hash is calculated as soon as
the __init__ function so that if there is a change in those files,
buildout will remove the existing wrapper (it will uninstall the
section) and replace it with the new wrapper.

/reviewed-on nexedi/slapos!525

Caddy and ATS shall have same timeouts for the requests.

It defaults to 600s, which is good reasonable chosen before.

/reviewed-on nexedi/slapos!597

Instead of expensive and long checking of the configuration during promise run
read last stored state.

This decouples configuration validation calculation from promise check. The
validation information is updated often (on each configuration change, on each
reload, etc) and every 2 hours.

Validation happens on each configuration change, but for sure it is checked
each 2 hours.

State of configuration is calculated in separate script.

Move files from var/log/trafficserver/*old to
srv/backup/logrotate/trafficserver, xz them and clean files older than a
year.

Some arguments needs Caddy process restart, so implement it with
hash-files and also inform the master partition requester about parameters
which will result with process restart.

caddy-frontend master partition does not implement any promise in etc/promise,
all is migrated to etc/plugin

caddy-frontend-is-running-actual-software-release promise is not needed
anymore, as hash-files are used.

There is no need anymore to have two processes for normal and nginx slaves,
as nginx ones are served by caddy anyway.

Also inform the requester that type:eventsource is not implemented.

Since Caddy v0.11.4 it is possible to disable log rotation, thus disable it
and rely purely on SlapOS defined log rotation.

See https://github.com/mholt/caddy/releases/tag/v0.11.4

This also means that caddy source is fetched directly from upstream, as all
required fixes has been incorporated into the upstream.

Since https://github.com/mholt/caddy/releases/tag/v0.11.4 TLS-SNI challenge
is replaced by ACME TLS-ALPN challenge, so switch has changed.

Drop direct usage of gowork for now, in order to have caddy built using go
module, support for gowork with go modules might come later.

/reviewed-on nexedi/slapos!544

Instead of complex architecture in the profiles, reuse kedifa-updater
capability to do backward compatibility certificate management thanks to its
fall-back mechanism.

kedifa-updater uses state file to know, if it ever succeed to download
certificate from KeDiFa, and so it really makes it that pushing at least once
certificate to KeDiFa, even if it is sometimes unresponsive, will switch to
it.

Fallback certificate is used, thus each slave listens immediately on HTTP and
HTTPS. Thanks to this, asynchronous updates do not need to communicate with
slapos node instance, and slapos node instance does not care about the
certificates anymore.

Instead of fetching certificates on each slapos node instance use new
kedifa-updater, which is a tool to asynchronously fetch certificates and
has a hook to reload the server in case if new certificate is available.

custom_ssl_directory is NOT BBB

This is consistent across usage in caddy-frontend and allow better reusage.

Adapted configuration and instantiation to ATS 7.

Deployment:
 * traffic_line has been replaced with traffic_ctl
 * access log, of squid style, is ascii instead of binary, to do so
   logging.config is generated
 * ip_allow.config is configured to allow access from any host
 * RFC 5861 (stale content on error or revalidate) is implemented with core
   instead with deprecated plugin
 * trafficserver-autoconf-port renamed to trafficserver-synthetic-port
 * proxy.config.system.mmap_max removed, as it is not used by the system anymore

Tests:
 * As Via header is not returned to the client, it is dropped from the
   tests, instead its existence in the backend is checked.
 * Promise plugin trafficserver-cache-availability.py is re enabled, as
   it is expected to work immediately.

csr_id is exposed over HTTPS with short living self signed certificate,
which is transmitted via SlapOS Master. Thanks to this, it is possible to
match csr_id with certificate of given partition and take decision if it shall
be signed or not.

This is "quite secure" apporach, a bit better than blidny trusting what CSR
to sign in KeDiFa. The bootstrap information, which is short living
(certificates are valid for 5 days), resides in SlapOS Master. The csr_id
is not directly known to SlapOS Master, and shall be consumed as fast as
possible by frontend cluster operator in order to sign CSR appearing in
KeDiFa caucase. The known possible attack vector requires that attacker knows
caucased HTTP listening port and can hijack HTTPS traffic to the csr_id-url
to get the human approve his own csr_id. The second is hoped to be overcomed
by publishing certificate of this endpoint via SlapOS Master.

Unfortunately caucase-updater prefix is directly used to find real CSR, as the
one generated is just a template for rerequest, thus csr_id would be different
from really used by caucase-updater.

Use KeDiFa to store keys, and transmit the url to the requester for master
and slave partitions.

Download keys on the slave partitions level.

Use caucase to fetch main caucase CA.

kedifa-caucase-url is published in order to have access to it.

Note: caucase is prepended with kedifa, as this is that one.

Use kedifa-csr tool to generate CSR and use caucase-updater macro.

Switch to KeDiFa with SSL Auth and updated goodies.

KeDiFa endpoint URLs are randomised.

Only one (first) user certificate is going to be automatically accepted. This
one shall be operated by the cluster owner, the requester of frontend master
partition.

Then he will be able to sign certificates for other users and also for
services - so each node in the cluster.

Special trick from https://security.stackexchange.com/questions/74345/provide-subjectaltname-to-openssl-directly-on-command-line
is used for one command generation of extensions in the certificate.
Note: We could upgrade to openssl 1.1.1 in order to have it really
simplified (see https://security.stackexchange.com/a/183973 )

Improve CSR readability by creating cluster-identification, which is master
partition title, and use it as Organization of the CSR.

Reserve slots for data exchange in KeDiFa.

try_duration and try_interval are Caddy proxy's switches which allow to deal
with non working backend (https://caddyserver.com/docs/proxy)

The non working backend is the one, to which connection is lost or was not
possible to make, without sending any data.

The default try_duration=5s and try_interval=250ms are chosen, so that in
normal network conditions (with all possible problems in the network, like
lost packets) the browser will have to wait up to 5 seconds to be informed
that backend is inaccessible or for the request to start being processed,
but only a bit more than 250ms if Caddy would have to reestablish connection
to faulty backend.

In order to check it out it is advisable to setup a system, with real backend,
like apache one, and configure iptables to randomly reject packets to it:

  iptables -A INPUT -m statistic --mode random -p tcp --dport <backend_port> \
  --probability 0.05 -j REJECT --reject-with tcp-reset

Using ab or any other tool will results with lot of 502 EOF in the Caddy error
log and also reported by ab. With this configuration there are no more
errors visible to the client, which come from the problems on the network
between Caddy and the backend.

One of solutions for random 502 errors from caddy is to fully disable
HTTP2 protocol ( https://github.com/mholt/caddy/issues/1080 )

We run Caddy with HTTP2 enabled by default, as we can enable/disable it per
each slave, but in some environments it might be just better to fully avoid
HTTP2 codepaths in Caddy.

/reviewed-on !495

Caddy reloads configuration on USR1 not on HUP.

Caddy since 0.11.1 requires that certificate match the exposed site, so in
order to being able to serve ip access sites each frontend node needs
to generate certificate with its IP in the subjectAltName.

Validate only if interesting files are changed.

Simplify validation and graceful scripts, have one template for all cases.

Aviod needless repetition.

Note: There is limit of arguments to be passed to commands, see
https://www.in-ulm.de/~mascheck/various/argmax/, but we are safe for now:

  $ getconf ARG_MAX
  2097152

So we are keeping shell expansion instead of playing with find or other tools.

Caddy's -log can log to file, but there is no control over logrotation, so
it resuls with leftover logrotated files in the instance. As we have
already system to catch process logs (supervisor) move the logging there.

error-log stays for errors of catch-all and various non-slave instances.

Instead of needlessly storing information in configuration section, pass it
via jinja2 parameter. This is safe, in case if extra_slave_instance_list
would contain value like ${section:option}.