On Python 3, these made the build output very hard to read (e.g., https://travis-ci.org/buildout/buildout/jobs/394026829)

On Python 3.7, these could actually break the doctests.

Closing the files should get us closer to being able to pass the tests with PyPy.

Add a RENormalizer so the easy_install.txt will be happy with scripts
generated on Python 2 (using open(..., 'U')) and on Python 3 (using just
open(...)).

This was suggested by @sallner.

(cherry picked from commit 2cf42066)

Python 3.7 began warning about open's 'U' mode. Universal newline mode (newline=None) is default in Python 3 so avoid specifying it on Python 3.

(cherry picked from commit 978a6af9)

fixup! Apply += & -= in correct order with respect to extends

If a.cfg extends b.cfg and both contain [a],
it was already not possible for a.cfg:[a] to combine <= & +=.
But it worked when a.cfg:[a] does not extend anything, and
commit 161fb191 broke that.

The only proper way to solve all this seems to process <= before
+= & -=. But since processing <= while extending files would cause
other issues, we'd rather process +=&-= when initializing parts.

This is useful when using OS Python & eggs.

Useless for SlapOS.

The version of Python should not affect the behaviour of a recipe.
Anyway, it was already ignored for DEVELOP_DIST eggs.

This makes the slapos mechanism to share parts more efficient. And with
the upcoming changes in buildout & slapos.recipe.cmmi, there would be
no way for the slapos.reboostrap extension to prevent everything from
being rebuilt when reboostrapping to a different version of Python,
if bootstrap parts are shared.

The monkey-patch by slapos.reboostrap now becomes useless.

When buildout encounter an error, program exists with error code 1, this
should also be the case when exiting from a debugger session when
running with -D option.

/reviewed-on nexedi/slapos.buildout!17

When a recipe is developed in a git working copy, .git administrative
database folder should be ignored, like we did with CVS and svn.

/reviewed-on nexedi/slapos.buildout!14

_install_and_load is slow, using cache saves time when there are many sections.

Contrary to bash, dash filters out environment variables containing
'-' characters. And for example, /bin/sh is dash on Debian.

<rebased>

Useful when recipes generate non-string values to be reused by other
recipes.

We want two identical sections at different URL to be able to produce
same signature. This is feature is useful for slapos.recipe.cmmi's
shared parts.

Rebase instructions:
- squash with "apply "Support ${:_profile_base_location_}."

Add referred parts' hash strings in __buildout_signature__, that invokes rebuild of a part when one of its (recursive) dependencies are modified.

Also remove duplicates and sort entries in __buildout_signature__.

Also, updating a part does not put it anymore at the end of the list of
installed parts, that was making .installed.cfg too big.

In SlapOS, bootstrap is called each time software release is invoked and
we do not want to delete develop-eggs directory.

This reverts commit 55d76b34.

https://bugs.launchpad.net/zc.buildout/+bug/144228

Even though such configuration is wrong...

extends can be interpreted as inheritance in OOP, but the original
behaviour was against what is commonly (always?) seen everywhere.

It is however good practice a file extends all files it needs directly
(and only them). Then if two files A & B (possibly unrelated) extends
the same third C, A was unable to overrides C values. It was even
error-prone because someone who don't use B yet could override C values
in A and later extending B would break A.

For some of our common use cases, this new algorithm is also 9x faster
(time to annotate: ~2.3s with -> ~.29s).

Other changes:
- ~/ is now expanded for non-url extends.
- An absolute (non-url) path is not longer treated like a local path
  if the base is a url.
- Better path/url normalization.

Rebase instructions:
- squash with "Chomp ../ from beginging of filenames"
- split and apply "Support ${:_profile_base_location_}." after

Sqaushed with

Chomp ../ from beginging of filenames.

In order to have as canonical as possible paths, chomp ../ from filenames and
recalculate base.

--setuptools-version and --buildout-version options in bootstrap script still have the priority.

One big issue in commit e129e187 is that the
initial value of buildout_and_setuptools_path can cause buildout to use a lot
of CPU at startup. The following change would help:

    def buildout_and_setuptools_path():
        r = [get_python_lib(standard_lib=1)]
        for d in pkg_resources.working_set:
            d = d.location
            if d not in r:
                r.append(d)
        return r[1:]
    buildout_and_setuptools_path = buildout_and_setuptools_path()

but initializing buildout_and_setuptools_dists also takes time to compute,
and it's only used for bootstrap.

Because the commit is not required anymore since setuptools 36, it's easier
for us to revert partially, so we can keep applying our patches.

(cherry picked from commit 18964211)