Instead of expensive and long checking of the configuration during promise run
read last stored state.

This decouples configuration validation calculation from promise check. The
validation information is updated often (on each configuration change, on each
reload, etc) and every 2 hours.

Validation happens on each configuration change, but for sure it is checked
each 2 hours.

State of configuration is calculated in separate script.

Move files from var/log/trafficserver/*old to
srv/backup/logrotate/trafficserver, xz them and clean files older than a
year.

As some tools are not configurable with logrotate provide a script to
"rotate" files generated by those tools.
This script more or less follows logrotate behaviour:

 * moves rotated files to logrotate directory
 * compresses them with xz
 * have to be configured to keep only files for some days

Roll logs each 24h, as it would be expected.

Stderr to null redirection happened in the wrong place.

Promises will fail with stderr filled, and it can be such if sha256sum emits
error messages, so just discard them.

Some arguments needs Caddy process restart, so implement it with
hash-files and also inform the master partition requester about parameters
which will result with process restart.

Kedifa partition was missing monitoring at all, so add it and monitor kedifa
and exposer ip and port.

Partition running caddy was missing monitoring for exposer, so add it.

Use unreal address to avoid any tries for network connectivity.

Kedifa requires some time to process new slave, and during that time the key
download URL is not available, but as it is required for proper mapping file,
use some url to mimic it.

During buildout run no network communication is required in order to prepare
fallback certificates, so call kedifa-updater with --prepare-only

It is needed by users to check certificate of KeDiFa while uploading
certificates.

Each time new slave appears the kedifa-updater has to be run immediately, in
order for certificates to be properly setup.

Otherwise caddy can be left in non-runnable state until next kedifa-updater
would run again.

caddy-frontend master partition does not implement any promise in etc/promise,
all is migrated to etc/plugin

caddy-frontend-is-running-actual-software-release promise is not needed
anymore, as hash-files are used.

By default whole slave makes websocket connection to the backend.
With websocket-path, only the path has websocket style connections,
the rest is standard HTTP.

There is no need anymore to have two processes for normal and nginx slaves,
as nginx ones are served by caddy anyway.

Also inform the requester that type:eventsource is not implemented.

Differences between tls and non-tls are minimal, so simplify the generation
as much as possible with simple tls switch.

It seems more readable than creating Jinja2 macros, which would be used only
twice.

Since Caddy v0.11.4 it is possible to disable log rotation, thus disable it
and rely purely on SlapOS defined log rotation.

See https://github.com/mholt/caddy/releases/tag/v0.11.4

When there are no shared instances, the file was empty, but caddy
refuses to start when using an import statement on an empty file, with
this error:

```
Error during parsing: Could not read tokens while importing .../etc/log-access.conf: EOF
```

/reviewed-on nexedi/slapos!545

This also means that caddy source is fetched directly from upstream, as all
required fixes has been incorporated into the upstream.

Since https://github.com/mholt/caddy/releases/tag/v0.11.4 TLS-SNI challenge
is replaced by ACME TLS-ALPN challenge, so switch has changed.

Drop direct usage of gowork for now, in order to have caddy built using go
module, support for gowork with go modules might come later.

/reviewed-on nexedi/slapos!544

This reverts commit 7993ff81.

Custom configuration checks are hard to be trusted, as they can impact too
many aspects of running frontend.

Frontend administrator knows the risks of custom configuration, and shall take
proper care.

/reviewed-on nexedi/slapos!543

Instead of complex architecture in the profiles, reuse kedifa-updater
capability to do backward compatibility certificate management thanks to its
fall-back mechanism.

kedifa-updater uses state file to know, if it ever succeed to download
certificate from KeDiFa, and so it really makes it that pushing at least once
certificate to KeDiFa, even if it is sometimes unresponsive, will switch to
it.

Fallback certificate is used, thus each slave listens immediately on HTTP and
HTTPS. Thanks to this, asynchronous updates do not need to communicate with
slapos node instance, and slapos node instance does not care about the
certificates anymore.

Instead of fetching certificates on each slapos node instance use new
kedifa-updater, which is a tool to asynchronously fetch certificates and
has a hook to reload the server in case if new certificate is available.

custom_ssl_directory is NOT BBB

This mostly useful during tests to have stable results, especially when
some slaves are rejected.

This change is expected to be no-op during normal run.

Note: The slave rejection system does not guarantee any ordering, as the sort
      order can change, because of parameters can reorder slaves. Thus, even
      if slave A was requested before slave B, and they conflict each other,
      slave A can be rejected instead of "expected" slave B.

This is consistent across usage in caddy-frontend and allow better reusage.

Section was not renamed in buildout.hash.cfg

in apache frontend, we have been using:

```
LogFormat "%h %l %{REMOTE_USER}i %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D" combined
```

The %l is (from mod_log_config docs): Remote logname (from identd, if
supplied). This will return a dash unless mod_ident is present and
IdentityCheck is set On.

In the case of apache frontend, it was always a - . This is missing in
caddy frontend and our existing log processing tools (apachedex) cannot
be used on frontend logs since we switched to Caddy.

/reviewed-on nexedi/slapos!530

Adapted configuration and instantiation to ATS 7.

Deployment:
 * traffic_line has been replaced with traffic_ctl
 * access log, of squid style, is ascii instead of binary, to do so
   logging.config is generated
 * ip_allow.config is configured to allow access from any host
 * RFC 5861 (stale content on error or revalidate) is implemented with core
   instead with deprecated plugin
 * trafficserver-autoconf-port renamed to trafficserver-synthetic-port
 * proxy.config.system.mmap_max removed, as it is not used by the system anymore

Tests:
 * As Via header is not returned to the client, it is dropped from the
   tests, instead its existence in the backend is checked.
 * Promise plugin trafficserver-cache-availability.py is re enabled, as
   it is expected to work immediately.

It is better to have automation similar to previous implementation by
default.

AIKC - Automatic Internal Kedifa's Caucase CSR signing, which can be triggered
by option automatic-internal-kedifa-caucase-csr.

It signs all CSR which match csr_id and certificate from the nodes which needs them.

csr_id is exposed over HTTPS with short living self signed certificate,
which is transmitted via SlapOS Master. Thanks to this, it is possible to
match csr_id with certificate of given partition and take decision if it shall
be signed or not.

This is "quite secure" apporach, a bit better than blidny trusting what CSR
to sign in KeDiFa. The bootstrap information, which is short living
(certificates are valid for 5 days), resides in SlapOS Master. The csr_id
is not directly known to SlapOS Master, and shall be consumed as fast as
possible by frontend cluster operator in order to sign CSR appearing in
KeDiFa caucase. The known possible attack vector requires that attacker knows
caucased HTTP listening port and can hijack HTTPS traffic to the csr_id-url
to get the human approve his own csr_id. The second is hoped to be overcomed
by publishing certificate of this endpoint via SlapOS Master.

Unfortunately caucase-updater prefix is directly used to find real CSR, as the
one generated is just a template for rerequest, thus csr_id would be different
from really used by caucase-updater.