Symlinks can point anywhere outside the parts. Also symlinks can be
broken.

By using lstat instead of stat, the symlink is not considered as a
regular file.

See merge request nexedi/slapos.recipe.cmmi!14

Don't force strip_top_level_dir option to be false when calling for slapos.recipe.build:downloadunpacked - let the user decide.

Effectively adds strip_top_level_dir option to the script which allows for shorter and more consistent paths.
Also it can help mitigating the variability of path length between archives downloaded for branch, commit hash and release.

Add some doctests to describe the use cases of shared parts with slapos,
including the need for our `${:_profile_base_location_}` buildout patch
and the need to save signature files.

It is redundant with the fact that slapos.buildout already rebuilds
if any dependency is modified. slapos.buildout does not yet understand
the new => syntax from upstream buildout but meanwhile a dummy option
can be set to any options of required parts.

The value was always '${:_profile_base_location_}'

In conjunction with the new Options.barrier() in buildout,
parts will be able to modify the environment during install/update.

If not set, the commands in the middle of custom configure-command,
pre-build, post-build etc can fail without resulting in SystemError
for user and without stopping cmmi process, which thus can formally
succeed despite being misconfigured or in some cases with the code
not even built.
This is because when executing a multiline command, only the exit
status of the last one actually gets caught by check_call().

Setting -e makes debugging easier by allowing shell to terminate
mid-way if errors (non-zero exit status) occur.

-e only covers 'simple commands', and does not affect 'if' tests
and more complex cases where non-zero exit status is expected to
occur normally.
Detailed description of -e effects can be seen in the POSIX
standard:

https://pubs.opengroup.org/onlinepubs/009695399/utilities/set.html

When build process is interrupted (like when the buildout process
terminated by SIGKILL), building with shared leave the system in a state
that looks like installation succeeded, so next execution will assume
the part was properly installed.


The behavior was something like this:


    if isSharedPartSignatureOK():
      useThisAlreadyInstalledPart()
    else:
      writeSignatureInSharedPart()
      try:
        build()
      except:
        removeSharedPart()
        raise


and when program is terminated during build step, the signature is
there.

The fix is to write signature later, algorithm becomes:


    if isSharedPartSignatureOK():
      useThisAlreadyInstalledPart()
    else:
      try:
        build()
      except:
        removeSharedPart()
        raise
      writeSignatureInSharedPart()

/reviewed-on !10

Use a new option `buildout:shared-path-list` that is now a new line separated list of path, only the last part is used to install new parts, other are read-only.

Also some cleanups in the test suite.

/reviewed-on !9

on python3.6, subprocess error message is different, use a
zope.testing.renormalizing rule to normalize the message.

This will allow nested slapos where the inner slapos can re-use shared
parts from outer slapos, but not write into the outer shared parts.

For compatibility reasons, the syntax of the buildout option changed,
instead of the old ${buildout:shared-parts} that use to be one path, we
now use ${buildout:shared-part-list} as a \n separated list of paths.
All paths can be used to lookup existing parts, only the last one can be
used to install new parts.
There is no compatibility with the old option as it was not used much,
only in profiles which were not really compatible with shared options.

https://pypi.org/project/zope.testing/ features "renormalizing" that
this test was already using to make this test independent from actual
paths that can be different but use some paths that looks always the
same (like /sample_buildout/).
Also use this in our patches for shared support so that the test is a
bit more readable and also so that we can assert the actual hash being
used.

This is restructured text.

This is not the default with Python 2.

This reverts daa833ea ( nexedi/slapos.recipe.cmmi!3 ) and extend it to not set `$TMP` either.

De facto, setting only `$TMP` was never effective ( see also [#20120530-132221B](https://nexedi.erp5.net/bug_module/20120530-132221B) ) and as we saw in nexedi/slapos.recipe.cmmi!3 (comment 67059) it causes more harm than good.

The conclusion so far is that environment variables related to temporary files should be handled by the parent process. The parent process is responsible for providing a directory short enough so that we can create unix sockets in `$TMPDIR`  and to clean up this directory afterwards.


In a pure buildout it could be:

```bash
tmpdir=$(mktemp -d)
trap 'rm -rf $tmpdir' SIGINT SIGTERM
TEMP=$tmpdir TMPDIR=$tmpdir TMP=$tmpdir buildout
rm -rf $tmpdir
```

/reviewed-on nexedi/slapos.recipe.cmmi!8

It was inconsistent to only set TMP and not TEMP / TEMPDIR. In the case
of slapos, it should be set by slapos node software

Most installation seem to cleanup tmp properly, so the use case cleaning
up seem not critical.

This TMP was also used to know what new files have been created, by
considering files newer that the creation date of this directory.
Compare the file dates with the creation of the source directory instead
( xxx__compile__ )

This reverts commit daa833ea.

To make it easyier to diagnose build errors.

/reviewed-on !7