See merge request nexedi/slapos.buildout!29

Remove hardcoded dependencies of zc.buildout in bootstrap: easy_install
will anyway install all the actual dependencies of the pinned version
of zc.buildout, and return a working set containing them all.

Before this, when the pinned zc.buildout had other dependencies than
the ones hardcoded, they were installed correctly but bootstrap took
only the hardcoded ones into account and wrongly failed due to the
other dependencies being missing.

This is needed for buildout 3 because the dependencies now include
pip and wheel in addition to setuptools.

This fixes rmtree with broken symlinks on python >= 3.8:
after https://github.com/python/cpython/issues/86180 func is os.open

They are deprecated since Python 3.8

Rebase instructions:
- split and fixup commits
    171c2459
    c9e9b267

Fixes #484.

(cherry picked from commit 5aa3e5bd)

This fixes filtering of distributions, like py_version for EGG_DIST.

See commit fb785cb3.

This is a rewrite of commit 6e4b8efe
("Support network cache in Download.download()").

This reverts commit e6e348fa.

Commit c3903c7e was error-prone
because (lambda f1, f2: False) may be wrong in other cases than where
it's currently used.

To be squashed with commit 4ccb79e9.

Like for URL that contain credentials, we still skip auth challenge
because it's faster and:
- we only support one auth scheme (basic)
- netrc provides no way to specify realms, which seem anyway to be
  less and less used (https://stackoverflow.com/q/69303610 reports
  that recent browsers don't display them anymore)

See merge request nexedi/slapos.buildout!25

- Py3: stop using legacy API of urllib.request and
       fix download of http(s) URLs containing user:passwd@
- Py2: avoid OOM when downloading huge files

This is implemented as a method in case we want to make it configurable
via [buildout].

To be dropped once all buildout in the wild are able to upgrade to a
version that supports new names in expression of conditional sections
(see previous commit).

Adding new names for expression is currently not possible because
buildout aborts before it tries to upgrade (in-place or bootstrap).

(cherry picked from commit f1589378)

For example, existing values were not enough to distinguish
'arm-linux-gnueabi' from 'arm-linux-gnueabihf'.

The 'multiarch' value is the output of

  $CC -dumpmachine

where CC defaults to 'gcc'.

See also https://wiki.debian.org/Multiarch/Tuples

Sometimes, most parts are reinstalled for a reason that the user
didn't think about and it can take time to understand why.
Explaining for all parts would be too verbose and useless because
many are reinstalled just because their dependencies changed.

This alternate URL is used in case of HTTPError with the main one.
In any case, the main URL is used for both downloading & uploading
from/to networkcache.

This will be exposed by slapos.recipe.build:download* and
slapos.recipe.cmmi recipes.

An optimization is to avoid temporary file when possible: a rename
(or hard link) is not always possible (different mount points).

Another one is to not check md5sum twice when using cache file.

Fall-back mode is ignored if an MD5 checksum is given.

In case of checksum mismatch for a cached path, remove it and
download again, mainly to cover the following cases:
- the url content changes and the user updates the checksum
- buildout killed while downloading directly to cache
  (see above optimization)
- shutil.copyfile is interrupted

Egg is install from wheel the requested version ends with :whl

This replaces:
  6d9b7483 Avoid installing eggs from wheels when possible
  ff2ee218 Do never install from wheels

This relaxes commit ff2ee218
("Do never install from wheels") because for some eggs like plantuml,
there's no tarball.

About the implementation, no need to sort by version (as it was done)
because the list is already filtered to keep the best one.

See merge request nexedi/slapos.buildout!22

other tests have been disabled by mistake.
test was randomly failing because of glob.

When recipes mutate the options, we should not allow invalid syntax otherwise
it gets written in .installed.cfg and generate invalid configparser syntax.