See merge request nexedi/slapos!1022

This introduces some breaking changes to the check_url_available
promise:

- http_code has been renamed to http-code
- check-secure = 1 has been deprecated; http-code = 401 should be used
  instead

See merge request nexedi/slapos!1011

also apply the change in software/theia/test/dummy/software.cfg for consistency

  This is a partial port, just to have more tests that
  weren't afected by the patches for keep apache as entry-point
  of the backend.

See merge request nexedi/slapos!1009

This reverts commit cf99b892

See merge request nexedi/slapos!1020

In 6e696faa "v↑ tempstorage 2.12.2 -> 5.2(ZODB5) 3-nxd(ZODB4-wc2)" we upgraded
tempstorage for ZODB5 and ZODB4-wc2, but were careful and left tempstorage,
that was being used with ZODB4, intact. However @tomo suggests that we should
also upgrade tempstorage for ZODB4 because of the loadBefore fixes and not to
further grow divergence in between ZODB4 and ZODB4-wc2:

    I understand that keeping the same version of tempstorage for ZODB 4 is to
    be on the safe side but I'd rather change it to tempstorage4-wc2 anyway. I mean
    we have a bugfix but we are afraid to apply it. Also, The less difference
    between ZODB 4 and ZODB 4-wc2 the better (everything will be more tested this way).
    ( nexedi/slapos!1009 (comment 137985) )

-> So let's do that.

This was introduced recently, in 0aecb41a (software: Use local tmp for
caucase, 2021-05-18). This is harmless, but uses a weird-looking directory

With the default umask of running slapos node, the default umask create files
that are only readable by user, not by group:

    slapuser10@COMP-1234:~$ ls -al /opt/slapgrid/bf0ccebdbd76d491bcbe16b7998529b2/parts/tesseract/share/tessdata/
    total 125296
    drwxr-x--- 4 slapsoft slapsoft 4096 Jul 13 05:57 .
    drwxr-x--- 3 slapsoft slapsoft 4096 Jul 13 05:57 ..
    -rw------- 1 slapsoft slapsoft 44366093 Jul 13 05:57 chi_sim.traineddata
    drwxr-x--- 2 slapsoft slapsoft 4096 Jul 13 05:57 configs
    -rw------- 1 slapsoft slapsoft 23466654 Jul 13 05:57 eng.traineddata
    -rw------- 1 slapsoft slapsoft 14213351 Jul 13 05:57 fra.traineddata
    -rw------- 1 slapsoft slapsoft 35659159 Jul 13 05:57 jpn.traineddata
    -rw------- 1 slapsoft slapsoft 10562874 Jul 13 05:57 osd.traineddata
    -rw-r--r-- 1 slapsoft slapsoft 572 Jul 13 05:57 pdf.ttf
    drwxr-x--- 2 slapsoft slapsoft 4096 Jul 13 05:57 tessconfigs

but when using multiple users, partitions users are members of slapsoft group,
so these files needs to be group readable.

See merge request !1015

See merge request nexedi/slapos!1012

listening-ip parameter is now mandatory

tempstorage 2.12.2 is from 2012 and misses loadBefore fixes which are
important when ERP5 is run with ZODB5 or ZODB4-wc2:

https://github.com/zopefoundation/tempstorage/commit/d187a86d
https://github.com/zopefoundation/tempstorage/pull/16

When the first loadBefore fix is not there, the problem manifests itself as
Zope not starting correctly with the following error in its log:

    2021-07-02 08:37:25,579 ERROR Zope.ZODBMountPoint Failed to mount database. <class 'ZODB.POSException.ReadConflictError'> (^@^@^@^@^@^@^@^@)
    Traceback (most recent call last):
     File ".../eggs/Zope2-2.13.30-py2.7.egg/Products/ZODBMountPoint/MountedObject.py", line 248, in _getOrOpenObject
       root = conn.root()
     File ".../parts/ZODB4-wc2/src/ZODB/Connection.py", line 405, in root
       return RootConvenience(self.get(z64))
     File ".../parts/ZODB4-wc2/src/ZODB/Connection.py", line 276, in get
       p, serial = self._load(oid)
     File ".../parts/ZODB4-wc2/src/ZODB/Connection.py", line 246, in _load
       return self._loadFrom(oid, self._storage)
     File ".../parts/ZODB4-wc2/src/ZODB/Connection.py", line 255, in _loadFrom
       raise ReadConflictError(oid)
    ReadConflictError: ^@^@^@^@^@^@^@^@

When the second loadBefore fix is not there, the problem manifests
itself as data corruption and sporadic POSKeyErrors
(https://github.com/zopefoundation/tempstorage/issues/8).

-> Fix both problem by using recent tempstorage with ZODB5 and ZODB4:

- ZODB5: just use tempstorage 5.2 egg
- ZODB4-wc2: use tempstorage 3 + loadBefore backports:
  https://lab.nexedi.com/nexedi/tempstorage/commits/3-nxd
- ZODB4: stays at 2.12.2 for now for this change to be as risk-free as
  possible. However, due to the fixes and e.g. https://github.com/zopefoundation/tempstorage/issues/8#issuecomment-873894941
  it makes sense to switch ZODB4 to tempstorage 3-nxd as well.

/cc @tomo, @romain, @rafael

https://github.com/restic/rest-server

Now that slapos node instance no longer use --all ( nexedi/slapos.core!278 ),
some software release test revealed some problems with softwares,
either missing promises or maybe something that needs to be handled in slapos.core.

Using --all was wrong and we want to stop doing this now. Instead of fixing these software now, we keep the old behavior
of using --all only for the softwares with missing promises.


See merge request nexedi/slapos!969

This avoids having multiple monitors trying to listen on the same port
when using replicate instances.

Thanks to minimising amount of changing parameters of the wrapper, by setting
RAM and CPU to "init" values, it's possible to not restart the wrapper in case
of device hotplugging in for changes of those parameters.

Note that this shows regressiong to enable-device-hotplug which happened
after upgrade to qemu 5.2.0 in 8dc22418.

slapos.toolbox is added in order to use qemuqmpclient.

Tests depend on pycurl, which has to be provided with component, as
otherwise is uninstallable.

Fixes 75af5cd3

This softwares have some missing promises, so we keep using slapos node instance
--all in the test, so that test keep passing after we change standalone slapos
to not use --all anymore.

monitor-globalstate (which is supposed to create monitor configuration file
accessed by HTTP as .../public//monitor.global.json) runs by cron every 2
minutes. This crontab also uses randomsleep to add a delay - but there's a
promise waiting for this monitor configuration file to be available, so the
instanciation had to wait for this and if this was sometimes causing problems
in software release tests because promise did not succeed fast enough.

To prevent this pattern where slapos node instance has to wait for a crontab
to be executed, we run that script a first time from buildout.

The two following instance parameters now exist:
- `only-sr`: a list of software releases names to test
- `unittest-args`: a list of arguments for unittest

By default:
- all software releases are tested
- the unittest arguments are ["discover", "-v"]

Single values are accepted even if they are not in a list.

Also improve the recipe `switch-softwaretype' by catching the possible errors.

- directory html5as: final version 
- directory html5as-base: initial version

See merge request nexedi/slapos!1006

See merge request nexedi/slapos!1003