This currently cause distributions to be downloaded twice, even with a
download cache

TODO: see XXX comments in the doctest

This changes to use hashlib directly, the module was introduced in python2.5
we don't need to support older versions.

This redoes 0abf873d (Install egg from wheel only if explictly required),
because that switch to install wheels only when explicitly requested
broke things in several places[1,2]. We can avoid such breakage if
instead of rejecting all wheels by default, we accept wheels, but allow
only pure-python wheels to be installed.

As slapos!1026 (comment 138639)
shows detecting whether a dist is pure-python wheel is just

	(dist.precedence == WHL_DIST and dist.platform is None)

So let's use this and rework egg installer accordingly.

This patch is completely untested. However the way to check whether a
wheel is source-only was tested at slapos!1026 (comment 138639).

If this patch is accepted and new slapos.buildout release is made, then
all :whl usage in slapos.git can be dropped.

/cc @tomo, @jerome, @jm

[1] slapos!1026
[2] slapos!1026 (comment 138590)

Egg is install from wheel the requested version ends with :whl

This replaces:
  6d9b7483 Avoid installing eggs from wheels when possible
  ff2ee218 Do never install from wheels

This relaxes commit ff2ee218
("Do never install from wheels") because for some eggs like plantuml,
there's no tarball.

About the implementation, no need to sort by version (as it was done)
because the list is already filtered to keep the best one.

See merge request nexedi/slapos.buildout!22

other tests have been disabled by mistake.
test was randomly failing because of glob.

When recipes mutate the options, we should not allow invalid syntax otherwise
it gets written in .installed.cfg and generate invalid configparser syntax.

When recipe mutate options, options values are written to .installed.cfg
without escaping buildout substitution syntax, so if a recipe sets an option
value that could be interpreted as a buildout substitution, it is written as is
in .installed.cfg.
This can be a problem if options read from _read_installed_part_options are
accessed, like it's the case with slapos patched buildout which saves
installed options after an error with part installation or after each part
installation when running in verbose mode.

Contrary to bash, dash filters out environment variables containing
'-' characters. And for example, /bin/sh is dash on Debian.

Rebase instructions:
- squash with 93369b04
  and keep the above commit message

fixup! Fix testing index when /bin/sh is dash (like on Debian)

python ZipFile module does not support updating an entry in place, instead make
a new zip file and copy all entries.

On Python 3, these made the build output very hard to read (e.g., https://travis-ci.org/buildout/buildout/jobs/394026829)

On Python 3.7, these could actually break the doctests.

Closing the files should get us closer to being able to pass the tests with PyPy.

Because updating a section no longer add the section t the end of parts in
.installed, uninstallation happens in this order.

fixup! Add referred parts' hash strings in __buildout_signature__, that invokes rebuild of a part when one of its (recursive) dependencies are modified.

egg signatures are always {egg_name}-{egg_verson}, there should not be a "egg" prefix for setuptools

fixup! Add referred parts' hash strings in __buildout_signature__, that invokes rebuild of a part when one of its (recursive) dependencies are modified.

Comment by Julien Muchembled:

    Maybe we again have the original behaviour because of
    commit dffdeffc.

    Anyway, the behaviour of buildout at this point is meaningless.
    Buildout was previously invoked with install d3 d4 command-line
    arguments and I consider that d2 should have been reinstalled,
    because a section shall not access the option of another section
    if the latter is not properly installed.

Only build egg at this point if we have some patches to apply otherwise
buildout would install dependent egg without logging this operation, which
would cause many tests to fail.

Also cover the whole scenario of egg patching in a new doctest.

fixup! Add referred parts' hash strings in __buildout_signature__, that invokes rebuild of a part when one of its (recursive) dependencies are modified.

Add a RENormalizer so the easy_install.txt will be happy with scripts
generated on Python 2 (using open(..., 'U')) and on Python 3 (using just
open(...)).

This was suggested by @sallner.

(cherry picked from commit 2cf42066)

Python 3.7 began warning about open's 'U' mode. Universal newline mode (newline=None) is default in Python 3 so avoid specifying it on Python 3.

(cherry picked from commit 978a6af9)

If a.cfg extends b.cfg and both contain [a],
it was already not possible for a.cfg:[a] to combine <= & +=.
But it worked when a.cfg:[a] does not extend anything, and
commit 161fb191 broke that.

The only proper way to solve all this seems to process <= before
+= & -=. But since processing <= while extending files would cause
other issues, we'd rather process +=&-= when initializing parts.

extends can be interpreted as inheritance in OOP, but the original
behaviour was against what is commonly (always?) seen everywhere.

It is however good practice a file extends all files it needs directly
(and only them). Then if two files A & B (possibly unrelated) extends
the same third C, A was unable to overrides C values. It was even
error-prone because someone who don't use B yet could override C values
in A and later extending B would break A.

For some of our common use cases, this new algorithm is also 9x faster
(time to annotate: ~2.3s with -> ~.29s).

Other changes:
- ~/ is now expanded for non-url extends.
- An absolute (non-url) path is not longer treated like a local path
  if the base is a url.
- Better path/url normalization.

Rebase instructions:
- squash with "Chomp ../ from beginging of filenames"
- split and apply "Support ${:_profile_base_location_}." after

Revert "Cache downloaded data in zc/buildout/buildout.py:_open() in memory to accelerate remote extends."

This reverts commit 260bc0c4 partially.

The remaining should be squashed with "Write .installed.cfg only once, in safe way and only if there's any change."