We were using caddy-log-access-header to make sure we have at least one
file to include, but this was implemented in a way that the config file
was overwritten.
Reimplement this by using caddy-log-access-empty to create an empty file
when there are no slaves, caddy-log-access otherwise.

The same caddy-log-access section was defined more than once, keep only
one implementation.
Remove some trailing spaces.

Prevent creating 2 wrapper for the same service if hash changed. Here, one service is exited because port is used by the firt to service to start:

    slappart6:runner-sshd-4248650e36a9a26a6481df1baffd9f58-on-watch                RUNNING   pid 27835, uptime 0:03:45
    slappart6:runner-sshd-b3b68f4278ceb84691ec27521ea229eb-on-watch                EXITED    Mar 06 04:52 PM

To achieve that, update slapos.cookbook and use hash-existing-files option of wrapper recipe

hash-existing-files list all the files used for hash that are not
handled by buildout. For those files, the hash is calculated as soon as
the __init__ function so that if there is a change in those files,
buildout will remove the existing wrapper (it will uninstall the
section) and replace it with the new wrapper.

/reviewed-on nexedi/slapos!525

Sorting will make dumped data "canonical", so it will limit amount of order
based changes.

It defaults to 600s, which is good reasonable chosen before.

Added part for logrotation into part_list, resulting with installing it.

/reviewed-on nexedi/slapos!606

/reviewed-on nexedi/slapos!597

In some cases domain can come from "outside" of the profile, and be filled
with "garbage", so if custom_domain is set, do not overwrite it.

/reviewed-on nexedi/slapos!575

Kedifa partition was missing monitoring at all, so add it and monitor kedifa
and exposer ip and port.

Partition running caddy was missing monitoring for exposer, so add it.

Use unreal address to avoid any tries for network connectivity.

Kedifa requires some time to process new slave, and during that time the key
download URL is not available, but as it is required for proper mapping file,
use some url to mimic it.

During buildout run no network communication is required in order to prepare
fallback certificates, so call kedifa-updater with --prepare-only

Each time new slave appears the kedifa-updater has to be run immediately, in
order for certificates to be properly setup.

Otherwise caddy can be left in non-runnable state until next kedifa-updater
would run again.

By default whole slave makes websocket connection to the backend.
With websocket-path, only the path has websocket style connections,
the rest is standard HTTP.

There is no need anymore to have two processes for normal and nginx slaves,
as nginx ones are served by caddy anyway.

Also inform the requester that type:eventsource is not implemented.

When there are no shared instances, the file was empty, but caddy
refuses to start when using an import statement on an empty file, with
this error:

```
Error during parsing: Could not read tokens while importing .../etc/log-access.conf: EOF
```

/reviewed-on nexedi/slapos!545

This also means that caddy source is fetched directly from upstream, as all
required fixes has been incorporated into the upstream.

Since https://github.com/mholt/caddy/releases/tag/v0.11.4 TLS-SNI challenge
is replaced by ACME TLS-ALPN challenge, so switch has changed.

Drop direct usage of gowork for now, in order to have caddy built using go
module, support for gowork with go modules might come later.

/reviewed-on nexedi/slapos!544

Instead of complex architecture in the profiles, reuse kedifa-updater
capability to do backward compatibility certificate management thanks to its
fall-back mechanism.

kedifa-updater uses state file to know, if it ever succeed to download
certificate from KeDiFa, and so it really makes it that pushing at least once
certificate to KeDiFa, even if it is sometimes unresponsive, will switch to
it.

Fallback certificate is used, thus each slave listens immediately on HTTP and
HTTPS. Thanks to this, asynchronous updates do not need to communicate with
slapos node instance, and slapos node instance does not care about the
certificates anymore.

Instead of fetching certificates on each slapos node instance use new
kedifa-updater, which is a tool to asynchronously fetch certificates and
has a hook to reload the server in case if new certificate is available.

custom_ssl_directory is NOT BBB

This is consistent across usage in caddy-frontend and allow better reusage.

csr_id is exposed over HTTPS with short living self signed certificate,
which is transmitted via SlapOS Master. Thanks to this, it is possible to
match csr_id with certificate of given partition and take decision if it shall
be signed or not.

This is "quite secure" apporach, a bit better than blidny trusting what CSR
to sign in KeDiFa. The bootstrap information, which is short living
(certificates are valid for 5 days), resides in SlapOS Master. The csr_id
is not directly known to SlapOS Master, and shall be consumed as fast as
possible by frontend cluster operator in order to sign CSR appearing in
KeDiFa caucase. The known possible attack vector requires that attacker knows
caucased HTTP listening port and can hijack HTTPS traffic to the csr_id-url
to get the human approve his own csr_id. The second is hoped to be overcomed
by publishing certificate of this endpoint via SlapOS Master.

Unfortunately caucase-updater prefix is directly used to find real CSR, as the
one generated is just a template for rerequest, thus csr_id would be different
from really used by caucase-updater.

Use KeDiFa to store keys, and transmit the url to the requester for master
and slave partitions.

Download keys on the slave partitions level.

Use caucase to fetch main caucase CA.

kedifa-caucase-url is published in order to have access to it.

Note: caucase is prepended with kedifa, as this is that one.

Use kedifa-csr tool to generate CSR and use caucase-updater macro.

Switch to KeDiFa with SSL Auth and updated goodies.

KeDiFa endpoint URLs are randomised.

Only one (first) user certificate is going to be automatically accepted. This
one shall be operated by the cluster owner, the requester of frontend master
partition.

Then he will be able to sign certificates for other users and also for
services - so each node in the cluster.

Special trick from https://security.stackexchange.com/questions/74345/provide-subjectaltname-to-openssl-directly-on-command-line
is used for one command generation of extensions in the certificate.
Note: We could upgrade to openssl 1.1.1 in order to have it really
simplified (see https://security.stackexchange.com/a/183973 )

Improve CSR readability by creating cluster-identification, which is master
partition title, and use it as Organization of the CSR.

Reserve slots for data exchange in KeDiFa.

Use safe JSON serialisation/deserialisation, as otherwise unusual
slave_references can lead to issues and also character case is not kept.

Also care about case of log access user, which was undetected since
slave_reference in tests were always lowercase.

try_duration and try_interval are Caddy proxy's switches which allow to deal
with non working backend (https://caddyserver.com/docs/proxy)

The non working backend is the one, to which connection is lost or was not
possible to make, without sending any data.

The default try_duration=5s and try_interval=250ms are chosen, so that in
normal network conditions (with all possible problems in the network, like
lost packets) the browser will have to wait up to 5 seconds to be informed
that backend is inaccessible or for the request to start being processed,
but only a bit more than 250ms if Caddy would have to reestablish connection
to faulty backend.

In order to check it out it is advisable to setup a system, with real backend,
like apache one, and configure iptables to randomly reject packets to it:

  iptables -A INPUT -m statistic --mode random -p tcp --dport <backend_port> \
  --probability 0.05 -j REJECT --reject-with tcp-reset

Using ab or any other tool will results with lot of 502 EOF in the Caddy error
log and also reported by ab. With this configuration there are no more
errors visible to the client, which come from the problems on the network
between Caddy and the backend.

One of solutions for random 502 errors from caddy is to fully disable
HTTP2 protocol ( https://github.com/mholt/caddy/issues/1080 )

We run Caddy with HTTP2 enabled by default, as we can enable/disable it per
each slave, but in some environments it might be just better to fully avoid
HTTP2 codepaths in Caddy.

/reviewed-on nexedi/slapos!495

/reviewed-on nexedi/slapos!466

"software/caddy-frontend: auto-restart services on SR upgrade." begun
implementation of automatic restart of services on SR upgrade, but not all
services has been caught - 6tunnel was missing.

/reviewed-on nexedi/slapos!473

This reverts commit 9243ace4

There are some bizarre issues with notebook, which happen randomly.

/reviewed-on nexedi/slapos!463

/reviewed-on nexedi/slapos!445

Caddy since 0.11.1 requires that certificate match the exposed site, so in
order to being able to serve ip access sites each frontend node needs
to generate certificate with its IP in the subjectAltName.