[Test result](https://nexedijs.erp5.net/#/test_result_module?extended_search=%20%22Boxiang-SLAPOS-CORE-FIXUP%22&page=form&view=view)

/reviewed-on nexedi/slapos.core!149

Used to remove shared parts that are no longer referenced.

With shared part installed for example in `/opt/slapgrd/shared/name/HASH`
the heuristic is to assume that if `HASH` is present in a software's
`.installed.cfg`, then this software is using this shared part.
Similar, if `HASH` is present in
`/opt/slapgrid/shared/another/ANOTHER_HASH/.slapos*signature` then share
part another for hash `ANOTHER_HASH` is using the shared part.

This also have some support to recursively look in instances and if they
contain a `slapos.cfg` file, look for software and shared parts to see if
they use the shared part.

In other cases, this shared part can be deleted.

/reviewed-on !139

python3-wheels and python3-setuptools are installed when we installed
python3-pip only if recommended packages are installed. It's better to
force the installation of what we need.

  This normalizes the result on python2 and python3, and as well, makes the
  result more accurate.

When reinstalling a computer, 'node software' may run before
'node format' has created users, then it fails because it can't set
ownership of the directory it created. The directory already exists
on next runs and before this commit, it didn't fix the ownership.

/reviewed-on nexedi/slapos.core!132

The most important are:
 * sometimes instance was not properly destroyed, apparently we need to retry `slapos node report`
 * if instance is not properly destroy, remove the files ourself and mark test failed
 * we need to see log files and config files from the instance

there's also a few small commits not directly related.

/reviewed-on nexedi/slapos.core!145

/reviewed-on nexedi/slapos.core!143

  Introduce a new command for run promises only, from all partitions. This helps to perform quick
  checks.

  On the tests for ConsumptionReportBase, it contains a port from a test
   on slapos.toolbox done by Alain Takoudjou.

  This may case more memory usage and more 'live' data to handle,
  for this reason, I'm making it configurable

  This new report class is responsable for generate a file (serealising it per pertition),
  instead rely on concurrent calls to the sqlite.

  This is just to clean the methods a bit

  Extend monitor to grab this information, this allow us use slapos node collect more efficiently, and
  prevent others softwares to access directly the database.

- We should only try to remove leaked partitions if we detected some,
  otherwise it may produce a strange message "unable to delete leaked
  partitions".
- If some supervisor configuration is still present after partition
  deletion, they must be removed not to block next test and the current
  test must be marked failed.

destroying instance is supposed to remove these, but when there are
issues removing instances we sometimes have supervisor config left from
previous instances.

Reformating now delete theses after logging the problem.

Quite often when test fail we don't have any information about what was
wrong with the tested instance. Dumping the generated etc/* files and
the instance log files should help diagnosing such errors.

/reviewed-on nexedi/slapos.core!147

This command sometimes fail, for example with:

  2019-10-23 06:10:56 slapos[27556] ERROR
  Traceback (most recent call last):
    File "slapos/grid/SlapObject.py", line 800, in destroy
      self.cleanupFolder(self.instance_path)
    File "slapos/grid/SlapObject.py", line 836, in cleanupFolder
      os.remove(os.path.join(folder_path, file))
  FileNotFoundError: [Errno 2] No such file or directory: 'tmp/inst/defp5/.installed.cfg.tmp'

otherwise logged error and critical messages are just ignored.

maybe this is something slapos node instance should do instead ...

When having more than one read_only_shared_part_list, the template was
first rendered with indentation and then dedented, but since dedent only
remove the common indentation, we ended up with slapos.cfg like this:

```
$ cat /data/slappart11_testnode/cqg/inst/test0-9/tmp/etc/slapos.cfg

            [slapos]
            software_root = /data/slappart11_testnode/cqg/inst/test0-9/tmp/soft
            instance_root = /data/slappart11_testnode/cqg/inst/test0-9/tmp/inst
            shared_part_list =
              /srv/slapgrid/slappart11/srv/shared
/srv/slapgrid/slappart11/srv/testnode/cqg/shared
              /data/slappart11_testnode/cqg/inst/test0-9/tmp/shared
            master_url = http://10.0.12.198:46312
            computer_id = local
            root_check = False
```

First dedent, then render the template to prevent this issue. Apply this to
other uses of textwrap.dedent for consistency.

/reviewed-on nexedi/slapos.core!140

When the testnode installing `slapos_subscription_request`, it reports:
```
2019-10-08 19:09:59,997 INFO     runTestSuite: Adding erp5_web_renderjs_ui business template ... (imported in 2.513s) done (21.342s)
2019-10-08 19:09:59,997 INFO     runTestSuite: Adding slapos_subscription_request business template ... (imported in 2.692s) Traceback (most recent call last):
.....
2019-10-08 19:10:00,008 INFO     runTestSuite: ValueError: "commerce_validation_workflow" is not a workflow ID
```

I found the `commerce_validation_workflow` is inside `erp5_commerce` bt. So I add `erp5_commerce` as the dependency of `slapos_subscription_request`.

/reviewed-on nexedi/slapos.core!141

This new option for slapos.cfg allow to define a list of paths to use
for shared parts feature of `slapos.recipe.cmmi`.
This will be injected in the buildout profile used to install software,
as `${buildout:shared-part-list}` which is the option `slapos.recipe.cmmi`
use since 0.11.

Note that the option changed in `slapos.recipe.cmmi` because this was
incompatible. In 0.10, the option was named `${buildout:shared-parts}` and
it was supposed to be a single path. `slapos.recipe.cmmi` 0.10 was trying
to create this path. If we continued using this `${buildout:shared-parts}`
option and changed to passe a newline-separated list of paths, then
`slapos.recipe.cmmi` would just create one directory with \n in the name
and will probably fail, which would make all software from 1.0.73 to 1.0.121
not installable on next `slapos.core`, so we used another buildout option
to prevent this problem.

Because generally, profiles using slapos.recipe.cmmi 0.10 were not
"safely" shared, we don't even provide compatibility with old option.

/reviewed-on nexedi/slapos.core!46

This is about replacing the `utils.py` we have duplicated in each software release tests by a more "official API".

This introduce a  "sandalone slapos API" which are helpers around slapos API to create a recursive SlapOS. This can eventually replace implementation of erp5testnode and slaprunner (but there are no plans for that)

/reviewed-on nexedi/slapos.core!64