Promise plugins are important part of the instantiated configuration, thus
have to be checked for changes.

Only TestSlave class and its subclasses are tested, as those are enough
complex scenarios: master partition, kedifa partition and frontend partition.

Instead of forcing to set monitor port in some cases, just generate them,
so it's possible to correctly instantiate caddy-frontend on one partition
scenario like in webrunner or tests.

Because we were simply using "setupClass", all classes where storing
snapshots in the same folder, so if more than one class failed we also
had an error in storing snapshot.

/reviewed-on nexedi/slapos!683

Adjust `self._cleanup` to new signature from nexedi/slapos.core!156

/reviewed-on nexedi/slapos!677

monitor-boostrap.pid existence is unpredictible.

This test just update slave dict of the class, so it is needed to request.

As timeout is the same on frontend caddy, ATS and backend caddy allow
0 or 1 accesses to the additional endpoints. The request might not reach
the backend caddy, and this is expected.

It is not part of the tested system.

Due to missing test and rare condition, if https-only and
prefer-gzip-encoding-to-backend were true accessing http:// of the
slave resulted with redirecting to the https:// url with prefer-gzip
added to the path, which is fixed and tested here.

Most important changes:

 * promise tests are dropped, as this is implemented by test case
 * slave is always instantiated, to pass promise checks
 * move out some methods from the class
 * change the way how supervisor is called
 * adapt requestDefaultInstance to request slaves

New test system requires a lot of work for testing some edge cases of
caddy-frontend instantiation, so skip some tests for now.

caddy-frontend cluster stabilises only in case if slaves are present, as
otherwise slapos node instance won't run w/o errors, so request slaves
early.

Prevent creating 2 wrapper for the same service if hash changed. Here, one service is exited because port is used by the firt to service to start:

    slappart6:runner-sshd-4248650e36a9a26a6481df1baffd9f58-on-watch                RUNNING   pid 27835, uptime 0:03:45
    slappart6:runner-sshd-b3b68f4278ceb84691ec27521ea229eb-on-watch                EXITED    Mar 06 04:52 PM

To achieve that, update slapos.cookbook and use hash-existing-files option of wrapper recipe

hash-existing-files list all the files used for hash that are not
handled by buildout. For those files, the hash is calculated as soon as
the __init__ function so that if there is a change in those files,
buildout will remove the existing wrapper (it will uninstall the
section) and replace it with the new wrapper.

/reviewed-on nexedi/slapos!525

Fixed slapos.core stabilises connection_dict during publish, so it is able
to calculate connection-parameter-hash the same way on client and on server, this
having same results when it is expected to have them.

Tasks:

 - [x] release `slapos.core` with nexedi/slapos.core!131 and use the version here

/reviewed-on nexedi/slapos!625

Destroyed nodes shall be just destroyed, and there is no need to send whole
configuration to them.

Server returns slave list with request and publish keys, but only request keys
are important.

In order to avoid needless updates and nonsense data remove those polluted
keys before publishing information about each slave.

Remove method `generateHashFromFiles` in all file `test.py`, import it from `slapos.recipe.librecipe`.

/reviewed-on nexedi/slapos!617

  The Remote User is managed on the backend apache which will unset it anyway.

As shown by https://github.com/zopefoundation/Zope/pull/655 and
https://github.com/Pylons/waitress/commit/6d4dab6bed88917b973066a6d5222917661802b7
backends usually don't accept headers with underscores.

SSL_CLIENT_SERIAL is removed because it's unused.

If backend does not send Content-Type the frontend is responsible of adding
correct one.

/reviewed-on nexedi/slapos!597

We need to sleep to give chance for Caddy to pick up the signal.

This also means that caddy source is fetched directly from upstream, as all
required fixes has been incorporated into the upstream.

Drop direct usage of gowork for now, in order to have caddy built using go
module, support for gowork with go modules might come later.

Follow new way of certificate managament in Caddy 1 as noted
https://github.com/mholt/caddy/issues/2588#issuecomment-505367152

Frontend operator shall have easy access to information about rejected
slaves, possibly the best in the JSON file.

Also the keys for the human readable information are slave's titles, not
references.

The information is published via hand crafted HTTPS endpoint.

Note: The SSL certificate is generated manually. Existing caucase is special
      for KeDiFa, this is another step to move all generated certificates (or
      otherwise self-signed) to internal, full automatic caucase.

It is ready when all on-watch processes are not EXITED.

Validation happens on each configuration change, but for sure it is checked
each 2 hours.

State of configuration is calculated in separate script.

Order of files does not matter for the assertion.