_install_and_load is slow, using cache saves time when there are
many sections.

Introduce a workaround for the shebang line length limitation
in zc.buildout.easy_install.

Useful when recipes generate non-string values to be reused by other
recipes.

We want two identical sections at different URL to be able to produce
same signature. This is feature is useful for slapos.recipe.cmmi's
shared parts.

Rebase instructions:
- squash with "Support ${:_profile_base_location_}."
- XXX: the logic changed since then, so hard to do

Add referred parts' hash strings in __buildout_signature__, that
invokes rebuild of a part when one of its (recursive) dependencies
are modified.

Also remove duplicates and sort entries in __buildout_signature__.

Also, updating a part does not put it anymore at the end of the list of
installed parts, that was making .installed.cfg too big.

In SlapOS, bootstrap is called each time software release is invoked and
we do not want to delete develop-eggs directory.

This reverts commit 55d76b34.

https://bugs.launchpad.net/zc.buildout/+bug/144228

Even though such configuration is wrong...

This test asserts buildout's behavior with regards to download options,
and this was changed by the new algorithm for extends. Tests for the
new behavior have not been written yet.

The new algorithm avoids fetching the same extended file more than once
and correctly handles overriding values and += and -=:

The new algorithm starts as if there was a buildout file containing

```
[buildout]
extends =
  user/defaults.cfg # if it exists
  buildout.cfg # if it exists
  command_line_extends.cfg # if passed on the command line
```

The files are then fetched in depth-first-search postorder and fetching
child nodes in the order given by the extends directive, ignoring files
that have already been fetched.

The buildout dicts are then collected in order, and this linearisation
is then merged at the end, overriding the first configs collected with
the later ones. The first dict in the linearisation is not from a file,
but the dict of buildout's (hardcoded) defaults. This is equivalent to
acting as though every file that does not extend anything extends these
defaults.

The first time a file must be downloaded from a url, the linearisation
is merged with the configs already collected, and the resulting options
are then used to determine the download options for this download, and
every subsequent download.

This is a break with buildout's current logic for download options.

By analogy with classes in Python, we are computing a linearisation of
the class hierarchy to determine the method resolution order (MRO).
This algorithm is not the same as Python's MRO since Python 2.3 (C3).

It could be good to switch to a C3 linearisation like Python.

This avoids unecessary copies. This is a preparatory step to
reimplementing the extends algorithm. It may be that this breaks
the extends algorithm as it is currently implemented.

This avoids unecessary deepcopies. This is a preparatory step to
reimplementing the extends algorithm. It may be that this breaks
the extends algorithm as it is currently implemented.

If slapos.libnetworkcache is importable, install it in bootstrap
as though it were a dependency of zc.buildout.

This is a hack to propagate libnetworkcache as a soft dependency.

If zc.buildout or its dependencies have pinned versions that do not
match the currently running versions, they are now installed in the
local eggs directory from scratch according to the pinned versions.

In offline mode this merely ensures that versions that satisfy the
requirements are already available. This is the case when the eggs
are already installed, or when the running versions are a match to
the pinned versions or the absence of a pinned version.

If after this matching versions of zc.buildout and its dependencies
are not located in the local eggs or develop-eggs directories, they
are copied there as was already the case before.

When generating an environment dict for subprocess calls to pip,
os.environ was accidentally modified despite efforts to copy it
and modify only the copy, as copy.copy(os.environ) is not enough.

[ci skip]

[skip ci]

[ci skip]

[ci skip]

[skip ci]

For example: `[versions:python_version <= "3.9"]`.
Fixes https://github.com/buildout/buildout/issues/621

[skip ci]

[skip ci]