Installer: Encapsulate all uses of `._env` and `._path` (#352)

* buildout.cfg: Remove redundancy

Share the eggs definition between the the online and offline tests.

* Remove redundant code

Trying to insert `self._dest` in `self._path`  in `Installer.install()`
is unnecessary since:

 * it's already done in `__init__()`

 * nothing changes either `self._path` or `self._dest` during the
   lifetime of `Installer`

Remove `Installer._load_dist(self, dist)` since it has never been
used in the whole history of Buildout.

Finally, simplify the interface to `._get_dist()` and  `_call_easy_install()`:

All callers to `._get_dist()` add the resulting dist to the WorkingSet. No
reason to do it in `._get_dist()` or to pass the `ws` into
`._call_easy_install()`.

* Be stricter when adding setuptools as requirements

Only do it for namespace package dists that actually need it. I.e. those
that use `pkg_resources.declare_namespace()` instead of
`pkg_util.extend_path()` or PEP 420.

* easy_install: `eggs-directory` may contain more than eggs

Modify the `easy_install.Installer` class to locate not only eggs, but
anything that has a `.dist-info` inside `._dest` (i.e. the Buildout
`eggs-directory`).

This makes it easier for extensions like `buildout.wheel` to add non eggs
inside it: by installing a distribution inside a subdirectory of `._dest` as if
that subdirectory was `site-packages`, you can later locate that
distribution with the Installer.

Conversely, make sure any distribution found in a direct child of `.dest` is
treated as if it was an `egg` distribution (as opposed to a develop
distribution or a site-packages distribution).

All access to `Installer._path` and all access that modifies `._env` is now
done through `Installer` methods, so that Buildout extensions can install
subclasses of `Installer` that alter their behaviour.

* Refactor `_get_dist()` and `_call_easy_install()`

Move the actual invocation of `easy_install` to a module global function
`call_easy_install()`. Simplify its signature so it's simpler to
override in Buildout extensions.

Refactor `._get_dist()` and remove all unpack/install logic into
`_move_to_eggs_dir_and_compile()`, bypassing
`Installer._call_easy_install()`.

`._get_dist()` now calls directly `call_easy_install()` instead. But
only as a fallback to a dictionary lookup of filename extensions to
distribution unpacking methods like:

 - `.egg`: `unpack_egg()`
 - `.whl`: `unpack_wheel()`

This will make it easier for extensions to add support to new dist
formats.

buildout.cfg

@@ -25,9 +25,7 @@ eggs =
 # Tests that can be run wo a network
 [oltest]
 recipe = zc.recipe.testrunner
-eggs =
-  zc.buildout[test]
-  zc.recipe.egg
+eggs = ${test:eggs}
 defaults =
   [
   '-t',

src/zc/buildout/tests.py

@@ -836,12 +836,15 @@ and we will get setuptools included in the working set.
     ...        'zc.buildout.easy_install', level=logging.WARNING)
     >>> logging.getLogger('zc.buildout.easy_install').propagate = False
 
-    >>> [dist.project_name
-    ...  for dist in zc.buildout.easy_install.working_set(
-    ...    ['foox'], sys.executable,
-    ...    [join(sample_buildout, 'eggs'),
-    ...     join(sample_buildout, 'develop-eggs'),
-    ...     ])]
+    >>> def get_working_set(*project_names):
+    ...     paths = [join(sample_buildout, 'eggs'),
+    ...              join(sample_buildout, 'develop-eggs')]
+    ...     return [
+    ...        dist.project_name
+    ...        for dist in zc.buildout.easy_install.working_set(
+    ...            project_names, sys.executable, paths)
+    ...     ]
+    >>> get_working_set('foox')
     ['foox', 'setuptools']
 
     >>> print_(handler)
@@ -851,13 +854,15 @@ and we will get setuptools included in the working set.
 
     >>> handler.clear()
 
-On the other hand, if we have a regular egg, rather than a develop egg:
+On the other hand, if we have a zipped egg, rather than a develop egg:
 
     >>> os.remove(join('develop-eggs', 'foox.egg-link'))
 
-    >>> _ = system(join('bin', 'buildout') + ' setup foo bdist_egg -d'
-    ...            + join(sample_buildout, 'eggs'))
-
+    >>> _ = system(join('bin', 'buildout') + ' setup foo bdist_egg')
+    >>> foox_dist = join('foo', 'dist')
+    >>> import glob
+    >>> [foox_egg] = glob.glob(join(foox_dist, 'foox-*.egg'))
+    >>> _ = shutil.copy(foox_egg, join(sample_buildout, 'eggs'))
     >>> ls('develop-eggs')
     -  zc.recipe.egg.egg-link
 
@@ -870,17 +875,27 @@ On the other hand, if we have a regular egg, rather than a develop egg:
 
 We do not get a warning, but we do get setuptools included in the working set:
 
-    >>> [dist.project_name
-    ...  for dist in zc.buildout.easy_install.working_set(
-    ...    ['foox'], sys.executable,
-    ...    [join(sample_buildout, 'eggs'),
-    ...     join(sample_buildout, 'develop-eggs'),
-    ...     ])]
+    >>> get_working_set('foox')
     ['foox', 'setuptools']
 
     >>> print_(handler, end='')
 
-We get the same behavior if the it is a dependency that uses a
+Likewise for an unzipped egg:
+
+    >>> foox_egg_basename = os.path.basename(foox_egg)
+    >>> os.remove(join(sample_buildout, 'eggs', foox_egg_basename))
+    >>> _ = zc.buildout.easy_install.install(
+    ...     ['foox'], join(sample_buildout, 'eggs'), links=[foox_dist],
+    ...     index='file://' + foox_dist)
+    >>> ls('develop-eggs')
+    -  zc.recipe.egg.egg-link
+
+    >>> get_working_set('foox')
+    ['foox', 'setuptools']
+
+    >>> print_(handler, end='')
+
+We get the same behavior if it is a dependency that uses a
 namespace package.
 
 
@@ -903,12 +918,7 @@ namespace package.
     Develop: '/sample-buildout/foo'
     Develop: '/sample-buildout/bar'
 
-    >>> [dist.project_name
-    ...  for dist in zc.buildout.easy_install.working_set(
-    ...    ['bar'], sys.executable,
-    ...    [join(sample_buildout, 'eggs'),
-    ...     join(sample_buildout, 'develop-eggs'),
-    ...     ])]
+    >>> get_working_set('bar')
     ['bar', 'foox', 'setuptools']
 
     >>> print_(handler, end='')
@@ -916,6 +926,32 @@ namespace package.
       Develop distribution: foox 0.0.0
     uses namespace packages but the distribution does not require setuptools.
 
+On the other hand, if the distribution uses ``pkgutil.extend_path()`` to
+implement its namespaces, even if just as fallback from the absence of
+``pkg_resources``, then ``setuptools`` shoudn't be added as requirement to
+its unzipped egg:
+
+    >>> foox_installed_egg = join(sample_buildout, 'eggs', foox_egg_basename)
+    >>> namespace_init = join(foox_installed_egg, 'stuff', '__init__.py')
+    >>> write(namespace_init,
+    ... """try:
+    ...     __import__('pkg_resources').declare_namespace(__name__)
+    ... except ImportError:
+    ...     __path__ = __import__('pkgutil').extend_path(__path__, __name__)
+    ... """)
+
+    >>> os.remove(join('develop-eggs', 'foox.egg-link'))
+    >>> os.remove(join('develop-eggs', 'bar.egg-link'))
+    >>> get_working_set('foox')
+    ['foox']
+
+The same goes for packages using PEP420 namespaces
+
+    >>> os.remove(namespace_init)
+    >>> get_working_set('foox')
+    ['foox']
+
+Cleanup:
 
     >>> logging.getLogger('zc.buildout.easy_install').propagate = True
     >>> handler.uninstall()
@@ -3155,8 +3191,7 @@ class UnitTests(unittest.TestCase):
         def wheel_to_egg(dist, dest):
             newloc = os.path.join(dest, egg_name)
             shutil.copy(dist.location, newloc)
-            return pkg_resources.Distribution.from_location(newloc,
-                                                            'demo-0.3.whl')
+            return pkg_resources.Distribution.from_filename(newloc)
         zc.buildout.easy_install.wheel_to_egg = wheel_to_egg
         egg_dir = os.path.join(self.sample_buildout, 'eggs')
         self.assertFalse(egg_name in os.listdir(egg_dir))