Add referred parts' hash strings in __buildout_signature__, that invokes rebuild of a part when one of its (recursive) dependencies are modified.

Also remove duplicates and sort entries in __buildout_signature__.

Also, updating a part does not put it anymore at the end of the list of
installed parts, that was making .installed.cfg too big.

In SlapOS, bootstrap is called each time software release is invoked and
we do not want to delete develop-eggs directory.

This reverts commit 55d76b34.

https://bugs.launchpad.net/zc.buildout/+bug/144228

Even though such configuration is wrong...

Currently only zc.recipe.egg:custom supports setting environment
variables, and zc.recipe.egg:develop does not.

My motivation for allowing setting environment in :develop is
wendelin.core

    https://lab.nexedi.cn/nexedi/slapos/blob/b5faab3b/component/wendelin.core/buildout.cfg

There we have [wendelin.core] part which installs released egg from
pypi, and [wendelin.core-dev] part which installs wendelin.core from
its latest git version via zc.recipe.egg:develop .

The problem is, wendelin.core for setup.py to work, needs git available,
and with slapos we usually don't have git available on base system, so
we build it by our own and do something like

    [wendelin.core-dev]
    recipe = zc.recipe.egg:develop
    environment = wendelin.core-dev-env

    [wendelin.core-dev-env]
    # wendelin.core-dev needs git to build
    PATH = ${git:location}/bin:%(PATH)s

and the problem is environment does not currently work for
zc.recipe.egg:develop, and thus git is not found -> build fails.

~~~~

In order to support environment in :develop, we just move environment
setting/restoring bits from Custom to Base, and provide Base.install() which
uses this bits. Custom & Develop .install() becomes ._install() which gets
hooked into Base.install() .

I've tested the patch only manually, because currently automated tests are
broken in a lot of places for slapos.buildout and zc.recipe.egg .

/cc @kazuhiko, @Tyagov

- Support on the fly patches in zc.recipe.egg by ``EGGNAME-patches``,
  ``EGGNAME-patch-options``, ``EGGNAME-patch-binary`` (or
  ``patch-binary``) and ``EGGNAME-patch-revision`` options.

- Support on the fly patches in zc.recipe.egg:custom by ``patches``,
  ``patch-options``, ``patch-binary`` and ``patch-revision`` options.
  (options ``EGGNAME-*`` are also supported as well).

In order to have as canonical as possible paths, chomp ../ from filenames and
recalculate base.

--setuptools-version and --buildout-version options in bootstrap script still have the priority.

One big issue in commit e129e187 is that the
initial value of buildout_and_setuptools_path can cause buildout to use a lot
of CPU at startup. The following change would help:

    def buildout_and_setuptools_path():
        r = [get_python_lib(standard_lib=1)]
        for d in pkg_resources.working_set:
            d = d.location
            if d not in r:
                r.append(d)
        return r[1:]
    buildout_and_setuptools_path = buildout_and_setuptools_path()

but initializing buildout_and_setuptools_dists also takes time to compute,
and it's only used for bootstrap.

Because the commit is not required anymore since setuptools 36, it's easier
for us to revert partially, so we can keep applying our patches.

(cherry picked from commit 18964211252e158f52c52152799cd4fcd34521ba)

(cherry picked from commit 73b1377c27d9b4c490792cbdcfc2a2e88d83d3d1)

PyPI redirects HTTP to HTTPS by default now so using HTTPS directly
avoids the potential for that redirect being modified in flight,
helping prevent MITM attacks.

Fixes #114.

(cherry picked from commit fac9979f)

[ci skip]

Fixes #326

Mainly to disable it when just trying to get distributions for
buildout and setuptools.

That I can no longer reproduce.

NFC

Fixed a bug introduced in 2.6.0:
zc.buildout and its dependeoncies were reported as picked even when
their versions were fixed in a ``versions`` section.  Worse, when the
``update-versions-file`` option was used, the ``versions`` section was
updated needlessly on every run.

[ci skip]

[ci skip]

Added a buildout option, ``abi-tag-eggs``

As suggested in: https://github.com/buildout/buildout/pull/325

Also simplified _setup_directories to not bother with the eggs
directory, since that's handled elsewhere.

that, when true, causes the `ABI tag
<https://www.python.org/dev/peps/pep-0425/#abi-tag>`_ for the buildout
environment to be added to the eggs directory name.

This is useful when switching Python implementations (e.g. CPython
vs PyPI or debug builds vs regular builds), especially when
environment differences aren't reflected in egg names.  It also has
the side benefit of making eggs directories smaller, because eggs
for different Python versions are in different directories.

Python 3.6