Factor out setuptools support from the buildout

git-svn-id: http://svn.zope.org/repos/main/zc.buildout/trunk@69874 62d5b8a3-27da-0310-9561-8e5933582275

zc.buildoutsupport/README.txt

+=============
+Zope Buildout
+=============
+
+.. contents::
+
+The Zope Buildout project provides support for creating applications,
+especially Python applications.  It provides tools for assembling
+applications from multiple parts, Python or otherwise.  An application
+may actually contain multiple programs, processes, and configuration
+settings.
+
+Here's an example of such an application that we built with an earlier
+prototype of the buildout system. We have a Zope application consisting of:
+
+- Multiple Zope instances
+
+- 4 ZEO servers
+
+- An LDAP server
+
+- Cache-invalidation and Mail delivery servers
+
+- Dozens of add-on packages
+
+- Multiple test runners
+
+- Multiple deployment modes, including dev, stage, and prod, 
+  with prod deployment over multiple servers
+
+Parts installed include:
+
+- Application software installs, including Zope, ZEO and LDAP
+  software
+
+- Add-on packages
+
+- Bundles of configuration that define Zope, ZEO and LDAP instances
+
+- Utility scripts such as test runners, server-control
+  scripts, cron jobs.
+
+This is all defined using configuration files and recipes, which are
+software that build and installs parts based on configuration data.
+The prototype system has minimal documentation and no tests and
+has no egg support.  (It build on earlier make-based systems that had
+no documentation or tests.)  
+
+This project provides a non-prototype implementation of the ideas and
+knowledge gained from earlier efforts and leverages setuptools to make
+recipe management cleaner and to provide better Python package and
+script management.
+
+The word "buildout" refers to a description of a set of parts and the
+software to create and assemble them.  It is often used informally to
+refer to an installed system based on a buildout definition.  For
+example, if we are creating an application named "Foo", then "the Foo
+buildout" is the collection of configuration and application-specific
+software that allows an instance of the application to be created.  We
+may refer to such an instance of the application informally as "a Foo
+buildout".  
+
+I expect that, for many Zope packages, we'll arrange the package
+projects in subversion as buildouts.  To work on the package, someone
+will check the project out of Subversion and build it.  Building it
+will assemble all of packages and programs needed to work on it.  For
+example, a buildout for a project to provide a new security policy
+will include the source of the policy and specifications to build the
+application for working on it, including:
+
+- a test runner
+
+- a web server for running the user interface
+
+- supporting packages
+
+A buildout will typically contain a copy of bootstrap.py.  When
+someone checks out the project, they'll run bootstrap.py, which will
+
+- create support directories, like bin, eggs, and work, as needed,
+
+- download and install the zc.buildout and setuptools eggs,
+
+- run bin/build (created by installing zc.buildout) to build the
+  application.
+
+Buildouts are defined using configuration files.  These files are
+based on the Python ConfigParser module with some variable-definition
+and substitution extensions.  
+
+Installation
+============
+
+There are two ways to install zc,buildout
+
+1. Install it as an egg using `easy_install
+   <http://peak.telecommunity.com/DevCenter/EasyInstall>`_ into a
+   Python instaallation. Then just use the buildout script from your
+   Python bin or Scripts directory.
+
+2. Use the `bootstrap script
+   <http://dev.zope.org/Buildout/bootstrap.py>`_ to install setuptools
+   and the buildout software into your buildout.  Typically, you'll
+   check the bootstrap script into your project so that, whenever you
+   checkout your project, you can turn it into a buildout by just
+   running the bootstrap script.
+
+More information
+================
+
+The detailed documentation for the various parts of buildout can be
+found in the following files:
+
+`buildout.txt <http://dev.zope.org/Buildout/buildout.html>`_
+   Describes how to define and run buildouts.  It also describes how
+   to write recipes.
+
+`easy_install.txt <http://dev.zope.org/Buildout/easy_install.html>`_
+   Describes an Python APIs for invoking easy_install for generation
+   of scripts with paths baked into them.
+
+
+Download
+========
+
+You can download zc.buildout and many buildout recipes from the
+`Python Package Index <http://www.python.org/pypi>`_.
+
+Recipes
+=======
+
+Existing recipes include:
+
+`zc.recipe.egg <http://dev.zope.org/Buildout/egg.html>`_
+   The egg recipe installes one or more eggs, with their
+   dependencies.  It installs their console-script entry points with
+   the needed eggs included in their paths.
+
+`zc.recipe.testrunner <http://dev.zope.org/Buildout/testrunner.html>`_
+   The testrunner egg installs creates a test runner script for one or
+   more eggs.
+
+`zc.recipe.zope3checkout <http://dev.zope.org/Buildout/zope3checkout.html>`_
+   The zope3checkout recipe installs a Zope 3 checkout into a
+   buildout.
+
+`zc.recipe.zope3instance <http://dev.zope.org/Buildout/zope3instance.html>`_
+   The zope3instance recipe sets up a Zope 3 instance.
+
+`zc.recipe.filestorage <http://dev.zope.org/Buildout/filestorage.html>`_
+   The filestorage recipe sets up a ZODB file storage for use in a
+   Zope 3 instance creayed by the zope3instance recipe.
+
+Buildout examples
+=================
+
+Some simple buildout examples:
+
+`The zc.buildout project <http://svn.zope.org/zc.buildout/trunk>`_
+   This is the project for the buildout software itself, which is
+   developed as a buildout. 
+
+`The zc sharing project <http://svn.zope.org/zc.sharing/trunk>`_
+   This project illistrates using the buildout software with Zope 3.
+   Note that the bootstrap.py file is checked in so that a buildout
+   can be made when the project is checked out.  The buildout.cfg
+   specified everything needed to create a Zope 3 installation with
+   the zc.sharing package installed in development mode.
+
+Status
+======
+
+The buildout system is under active development. Some near term
+priorities include:
+
+- Better error reporing
+
+- Windows support
+
+- Handling of egg extras
+
+- More recipes
+
+Questions
+=========
+
+You can send questions to jim@zope.com.
+
+Change History
+==============
+
+1.0.0b3
+-------
+
+- Added Windows support.
+
+- Fixed some bugs in variable substitutions.  
+
+  The characters "-", "." and " ", weren't allowed in section or
+  option names.
+
+  Substitutions with invalid names were ignored, which caused
+  missleading failures downstream.
+
+- Improved error handling.  No longer show tracebacks for user errors.
+
+- Now require a recipe option (and therefore a section) for every part.
+
+1.0.0b2
+-------
+
+Added support for specifying some build_ext options when installing eggs
+from source distributions.
+
+1.0.0b1
+-------
+
+- Changed the bootstrapping code to only install setuptools and
+  zc.buildout. The bootstrap code no-longer runs the buildout itself.
+  This was to fix a bug that caused parts to be recreated
+  unnecessarily because the recipe signature in the initial buildout
+  reflected temporary locations for setuptools and zc.buildout.
+
+- Now create a minimal setup.py if it doesn't exist and issue a
+  warning that it is being created.
+
+- Fixed bug in saving installed configuration data.  %'s and extra
+  spaces weren't quoted.
+
+1.0.0a1
+-------
+
+Initial public version

zc.buildoutsupport/bootstrap/bootstrap.py

+##############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Bootstrap a buildout-based project
+
+Simply run this script in a directory containing a buildout.cfg.
+The script accepts buildout command-line options, so you can
+use the -c option to specify an alternate configuration file.
+
+$Id$
+"""
+
+import os, shutil, sys, tempfile, urllib2
+
+tmpeggs = tempfile.mkdtemp()
+
+ez = {}
+exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
+                     ).read() in ez
+ez['use_setuptools'](to_dir=tmpeggs, download_delay=0)
+
+import pkg_resources
+
+cmd = 'from setuptools.command.easy_install import main; main()'
+if sys.platform == 'win32':
+    cmd = '"%s"' % cmd # work around spawn lamosity on windows
+
+ws = pkg_resources.working_set
+assert os.spawnle(
+    os.P_WAIT, sys.executable, sys.executable,
+    '-c', cmd, '-mqNxd', tmpeggs, 'zc.buildout',
+    dict(os.environ,
+         'PYTHONPATH'=
+         ws.find(pkg_resources.Requirement.parse('setuptools')).location
+         ),
+    ) == 0
+
+ws.add_entry(tmpeggs)
+ws.require('zc.buildout')
+import zc.buildout.buildout
+zc.buildout.buildout.main(sys.argv[1:] + ['bootstrap'])
+shutil.rmtree(tmpeggs)

zc.buildoutsupport/buildout.cfg

+[buildout]
+develop = zc.recipe.egg_ zc.recipe.testrunner zc.buildoutsupport
+parts = test
+
+# prevent slow access to cheeseshop:
+index = http://download.zope.org
+
+[test]
+recipe = zc.recipe.testrunner
+eggs = 
+  zc.buildout 
+  zc.recipe.egg 
+  zc.recipe.testrunner

zc.buildoutsupport/dev.py

+##############################################################################
+#
+# Copyright (c) 2005 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Bootstrap the buildout project itself.
+
+This is different from a normal boostrapping process because the
+buildout egg itself is installed as a develop egg.
+
+$Id$
+"""
+
+import os, sys, urllib2
+
+for d in 'eggs', 'develop-eggs', 'bin':
+    if not os.path.exists(d):
+        os.mkdir(d)
+
+ez = {}
+exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
+                     ).read() in ez
+ez['use_setuptools'](to_dir='eggs', download_delay=0)
+
+import pkg_resources
+
+os.spawnle(os.P_WAIT, sys.executable, sys.executable, 'setup.py',
+           '-q', 'develop', '-m', '-x', '-d', 'develop-eggs',
+           {'PYTHONPATH': os.path.dirname(pkg_resources.__file__)},
+           )
+pkg_resources.working_set.add_entry('src')
+
+import zc.buildout.easy_install
+zc.buildout.easy_install.scripts(
+    ['zc.buildout'], pkg_resources.working_set , sys.executable, 'bin')
+sys.exit(os.spawnl(os.P_WAIT, 'bin/buildout', 'bin/buildout'))

zc.buildoutsupport/setup.py

+from setuptools import setup, find_packages
+
+name = "zc.buildout"
+setup(
+    name = name,
+    version = "1.0.0b2",
+    author = "Jim Fulton",
+    author_email = "jim@zope.com",
+    description = "System for managing development buildouts",
+    long_description=open('README.txt').read(),
+    license = "ZPL 2.1",
+    keywords = "development build",
+    url='http://svn.zope.org/zc.buildout',
+
+    data_files = [('.', ['README.txt'])],
+    packages = ['zc', 'zc.buildout'],
+    package_dir = {'': 'src'},
+    namespace_packages = ['zc'],
+    install_requires = 'setuptools',
+    include_package_data = True,
+    tests_require = ['zope.testing'],
+    test_suite = name+'.tests.test_suite',
+    entry_points = {'console_scripts':
+                    ['buildout = %s.buildout:main' % name]}, 
+    dependency_links = ['http://download.zope.org/distribution/'],
+    )

zc.buildoutsupport/specifications/README.txt

+This directory is (an experimental) place to manage Launchpad
+specification bodies.  Files here are referenced (via svn.zope.org
+URLs) from the buildout project specifications in Launchpad, 
+https://features.launchpad.net/products/zc.buildout.

zc.buildoutsupport/specifications/repeatable.txt

+Repeatable (taggable) buildouts
+===============================
+
+It's important to be able to tag a buildout in a software repository
+in such a way that, months, or even years later, the buildout tag can
+be checked out and used to construct the same collection of parts,
+with the same versions. (Note that parts could still behave
+differently due to changes in parts of the environment, such as system
+libraries, not controlled by the buildout.)
+
+A feature of the buildout is it's use of eggs and the automatic
+resolution of dependencies.  The latest versions of dependencies are
+automatically downloaded and installed.  This is great during
+development or when using the buildout for casual software
+development, but it doesn't work very well for reproducing an old
+buildout.
+
+What's needed is some way to, when needed, record information about
+the versions of eggs (and any other bits) who's versions are
+determined dynamically.
+
+Proposal
+--------
+
+We'll add a buildout option, create-repeatable. The option will
+specify a file into which option information should be saved to create
+a repeatable buildout.  The data will be saved in a form that can be
+used by the buildout or recipes in a later run.  To make a tagged
+buildout, a user would run the buildout with the create-repeatable
+option set to a file name and then modify the buildout to be
+extended-by this file.
+
+Consider the following example buildout.cfg::
+
+  [buildout]
+  parts = foo
+
+  [foo]
+  recipe = zc.recipe.eggs
+  eggs = foo 
+         eek
+
+Now assume that:
+
+- The current version of foo is 1.1
+
+- Foo depends on bar =, which depends on baz.  The current versions of
+  bar and bas are 1.1 and 2.1.
+
+- The current version of eek is 1.5
+
+- eek depends on ook, which is as version 1.3.
+
+- zc.recipe.egg is at version 1.0b5
+
+If we run the buildout with the command-line option::
+
+  buildout:create-repeatable=reapeatable.cfg
+
+we'll get a repeatable.cfg file that looks something like::
+
+  [foo]
+  recipe = zc.recipe.eggs ==1.0b5
+  static = true
+  eggs = foo ==1.1
+         bar ==1.1
+         baz ==2.1
+         eek ==1.5
+         ook ==1.3
+
+The file contains options for the foo part.  The buildout software
+itself added an entry for the recipe that fixes the recipe version
+at the version used by the buildout.
+
+The zc.recipe.eggs recipe added the eggs option that lists the
+specifoc releases that were assembled.
+
+Finally the buildout.cfg file can be modified to use the
+repeatable.cfg file::
+
+  [buildout]
+  parts = foo
+  extended-by: repeatable.cfg
+
+  [foo]
+  recipe = zc.recipe.eggs
+  eggs = foo 
+         eek
+
+When the buildout is run, the options in repeatable.cfg will override
+the onces in buildout.cfg, providing a repeatable buildout
+
+Python API
+----------
+
+The recipe API will grow a repeatable method that is called after the
+install method and is passed a dictionary that a recipe can store
+option data in.  A recipe instance will only be able to provide repeatable data
+for it's part.
+

zc.buildoutsupport/src/zc/__init__.py

+try:
+    __import__('pkg_resources').declare_namespace(__name__)
+except:
+    # bootstrapping
+    pass

zc.buildoutsupport/src/zc/buildout/__init__.py

+#

zc.buildoutsupport/todo.txt

+
+- Use setuptools PackageIndex objects to improve performance by
+  deciding whether we need to download anything without using
+  easy_install.
+
+- tests
+
+  - distribution dependency links
+
+  - offline mode (there is an indirect test in the testrunner tests)
+
+
+- Load from urls
+
+- control python for develop (probbaly a new recipe)
+
+- proper handling of extras
+
+- Common recipes
+
+  - configure-make-make-install
+
+  - download, checkout
+
+  - Should it be possible to provide multiple recipies?
+    Or should recipies be combined through inheritence (or
+    composition)?
+
+  - Python
+
+- Some way to freeze versions so we can have reproducable buildouts.
+
+  Maybe simple approach:
+
+  - Egg recipe outputs dependency info with debug logging
+
+  - Egg recipe has option to specify dependencies.  When used, 
+    don't automatically fetch newer data.
+
+- Option to search python path for distros
+
+- Part dependencies
+
+- custom uninstall
+
+- Fix develop so thet ordinary eggs fetched as dependencies end up
+  in eggs directory.
+
+  "fixed" except that fix is ineffective due to setuptools bug. :(
+  
+
+- spelling :)
+
+- document recipe initialization order
+
+
+
+
+Issues
+
+- Should we include setuptools and buildout eggs for buildout process
+  in environment when searching for requirements?
+
+- We don't want to look for new versions of setuptools all the time.
+  For now, we always use a local dist if there is one.  Needs more
+  thought.

zc.buildoutsupport/zc.recipe.egg_/README.txt

+Buildout Egg-Installation Recipe
+================================
+
+The egg-installation recipe installes eggs into a buildout eggs
+directory.  It also generates scripts in a buildout bin directory with 
+egg paths baked into them.
+
+The recipe provides the following options:
+
+eggs
+    A list of eggs to install given as one ore more setuptools
+    requirement strings.  Each string must be given on a separate
+    line.
+
+find-links
+    One or more addresses of link servers to be searched for
+    distributions.  This is optional.  If not specified, links
+    specified in the buildout section will be used, if any.
+
+index
+    The optional address of a distribution index server.  If not
+    specified, then the option from the buildout section will be
+    used.  If not specified in the part data or in the buildout
+    section, then http://www.python.org/pypi is used.
+
+python
+    The name of a section defining the Python executable to use.
+    This defaults to buildout.
+
+scripts
+   Control which scripts are generated.  The value should be a list of
+   zero or more tokens.  Each token is either a name, or a name
+   followed by an '=' and a new name.  Only the named scripts are
+   generated.  If no tokens are given, then script generation is
+   disabled.  If the option isn't given at all, then all scripts
+   defined by the named eggs will be generated.
+
+Custom eggs
+-----------
+
+The zc.recipe.egg:custom recipe supports building custom eggs,
+currently with specialized options for building extensions.
+
+extra-paths
+   Extra paths to include in a generates script.
+
+To do
+-----
+
+- Some way to freeze the egg-versions used.  This includes some way to
+  record which versions were selected dynamially and then a way to
+  require that the recorded versions be used in a later run.
+
+- More control over script generation.  In particular, some way to 
+  specify data t be recored in the script.
+
+Change History
+==============
+
+1.0.0a2
+-------
+
+Added a new recipe for building custom eggs from source distributions,
+specifying custom distutils build_ext options.
+
+1.0.0a1
+-------
+
+Initial public version

zc.buildoutsupport/zc.recipe.egg_/setup.py

+from setuptools import setup, find_packages
+
+name = "zc.recipe.egg"
+setup(
+    name = name,
+    version = "1.0.0a2",
+    author = "Jim Fulton",
+    author_email = "jim@zope.com",
+    description = "Recipe for installing Python package distributions as eggs",
+    long_description = open('README.txt').read(),
+    license = "ZPL 2.1",
+    keywords = "development build",
+    url='http://svn.zope.org/zc.buildout',
+
+    packages = find_packages('src'),
+    include_package_data = True,
+    package_dir = {'':'src'},
+    namespace_packages = ['zc', 'zc.recipe'],
+    install_requires = ['zc.buildout', 'setuptools'],
+    tests_require = ['zope.testing'],
+    test_suite = name+'.tests.test_suite',
+    entry_points = {'zc.buildout': ['default = %s:Egg' % name,
+                                    'custom = %s:Custom' % name,
+                                    ]
+                    },    
+    dependency_links = ['http://download.zope.org/distribution/'],
+    )

zc.buildoutsupport/zc.recipe.egg_/src/zc/__init__.py

+__import__('pkg_resources').declare_namespace(__name__)

zc.buildoutsupport/zc.recipe.egg_/src/zc/recipe/__init__.py

+__import__('pkg_resources').declare_namespace(__name__)

zc.buildoutsupport/zc.recipe.egg_/src/zc/recipe/egg/README.txt

+Installation of distributions as eggs
+=====================================
+
+The zc.recipe.egg recipe can be used to install various types if
+distutils distributions as eggs.  It takes a number of options:
+
+eggs
+    A list of eggs to install given as one ore more setuptools
+    requirement strings.  Each string must be given on a separate
+    line.
+
+find-links
+   A list of URLs, files, or directories to search for distributions.
+
+index
+   The URL of an index server, or almost any other valid URL. :)
+
+   If not specified, the Python Package Index,
+   http://cheeseshop.python.org/pypi, is used.  You can specify an
+   alternate index with this option.  If you use the links option and
+   if the links point to the needed distributions, then the index can
+   be anything and will be largely ignored.  In the examples, here,
+   we'll just point to an empty directory on our link server.  This 
+   will make our examples run a little bit faster.
+
+python
+   The name of a section to get the Python executable from.
+   If not specified, then the buildout python option is used.  The
+   Python executable is found in the executable option of the named
+   section. 
+
+scripts
+   Control which scripts are generated.  The value should be a list of
+   zero or more tokens.  Each token is either a name, or a name
+   followed by an '=' and a new name.  Only the named scripts are
+   generated.  If no tokens are given, then script generation is
+   disabled.  If the option isn't given at all, then all scripts
+   defined by the named eggs will be generated.
+
+extra-paths
+   Extra paths to include in a generates script.
+
+We have a link server that has a number of eggs:
+
+    >>> print get(link_server),
+    <html><body>
+    <a href="demo-0.1-py2.3.egg">demo-0.1-py2.3.egg</a><br>
+    <a href="demo-0.2-py2.3.egg">demo-0.2-py2.3.egg</a><br>
+    <a href="demo-0.3-py2.3.egg">demo-0.3-py2.3.egg</a><br>
+    <a href="demoneeded-1.0-py2.3.egg">demoneeded-1.0-py2.3.egg</a><br>
+    <a href="demoneeded-1.1-py2.3.egg">demoneeded-1.1-py2.3.egg</a><br>
+    <a href="index/">index/</a><br>
+    <a href="other-1.0-py2.3.egg">other-1.0-py2.3.egg</a><br>
+    </body></html>
+
+
+We have a sample buildout.  Let's update it's configuration file to
+install the demo package. 
+
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... """
+    ... [buildout]
+    ... parts = demo
+    ...
+    ... [demo]
+    ... recipe = zc.recipe.egg
+    ... eggs = demo<0.3
+    ... find-links = %(server)s
+    ... index = %(server)s/index
+    ... """ % dict(server=link_server))
+
+In this example, we limited ourself to revisions before 0.3. We also
+specified where to find distributions using the find-links option.
+
+Let's run the buildout:
+
+    >>> import os
+    >>> os.chdir(sample_buildout)
+    >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout')
+    >>> print system(buildout),
+    
+Now, if we look at the buildout eggs directory:
+
+    >>> ls(sample_buildout, 'eggs')
+    -  demo-0.2-py2.3.egg
+    -  demoneeded-1.1-py2.3.egg
+    -  setuptools-0.6-py2.3.egg
+    -  zc.buildout-1.0-py2.3.egg
+
+We see that we got an egg for demo that met the requirement, as well
+as the egg for demoneeded, wich demo requires.  (We also see an egg
+link for the recipe.  This egg link was actually created as part of
+the sample buildout setup. Normally, when using the recipe, you'll get
+a regular egg installation.)
+
+The demo egg also defined a script and we see that the script was
+installed as well:
+
+    >>> ls(sample_buildout, 'bin')
+    -  buildout
+    -  demo
+    -  py-demo
+    -  py-zc.buildout
+
+Here, in addition to the buildout script, we see the demo script,
+demo, and we see a script, py-demo, for giving us a Python prompt with
+the path for demo and any eggs it depends on included in sys.path.
+This is useful for debugging and testing.
+
+If we run the demo script, it prints out some minimal data:
+
+    >>> print system(os.path.join(sample_buildout, 'bin', 'demo')),
+    2 1
+
+The value it prints out happens to be some values defined in the
+modules installed.
+
+We can also run the py-demo script.  Here we'll just print out
+the bits if the path added to reflect the eggs:
+
+    >>> print system(os.path.join(sample_buildout, 'bin', 'py-demo'),
+    ... """import os, sys
+    ... for p in sys.path:
+    ...     if 'demo' in p:
+    ...         print os.path.basename(p)
+    ...
+    ... """).replace('>>> ', '').replace('... ', ''),
+    ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
+    demo-0.2-py2.4.egg
+    demoneeded-1.1-py2.4.egg
+
+The recipe gets the most recent distribution that satisfies the
+specification. For example, We remove the restriction on demo:
+
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... """
+    ... [buildout]
+    ... parts = demo
+    ...
+    ... [demo]
+    ... recipe = zc.recipe.egg
+    ... find-links = %(server)s
+    ... index = %(server)s/index
+    ... """ % dict(server=link_server))
+
+    >>> print system(buildout),
+
+Then we'll get a new demo egg:
+
+    >>> ls(sample_buildout, 'eggs')
+    -  demo-0.2-py2.3.egg
+    -  demo-0.3-py2.3.egg
+    -  demoneeded-1.0-py2.3.egg
+    -  setuptools-0.6-py2.4.egg
+    -  zc.buildout-1.0-py2.4.egg
+
+Note that we removed the eggs option, and the eggs
+defaulted to the part name.
+
+The script is updated too:
+
+    >>> print system(os.path.join(sample_buildout, 'bin', 'demo')),
+    3 1
+
+You can control which scripts get generated using the scripts option.
+For example, to suppress scripts, use the scripts option without any
+arguments:
+
+
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... """
+    ... [buildout]
+    ... parts = demo
+    ...
+    ... [demo]
+    ... recipe = zc.recipe.egg
+    ... find-links = %(server)s
+    ... index = %(server)s/index
+    ... scripts =
+    ... """ % dict(server=link_server))
+
+
+    >>> print system(buildout),
+
+    >>> ls(sample_buildout, 'bin')
+    -  buildout
+    -  py-zc.buildout
+
+You can also control the name used for scripts:
+
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... """
+    ... [buildout]
+    ... parts = demo
+    ...
+    ... [demo]
+    ... recipe = zc.recipe.egg
+    ... find-links = %(server)s
+    ... index = %(server)s/index
+    ... scripts = demo=foo
+    ... """ % dict(server=link_server))
+
+    >>> print system(buildout),
+
+    >>> ls(sample_buildout, 'bin')
+    -  buildout
+    -  foo
+    -  py-zc.buildout
+
+If we need to include extra paths in a script, we can use the
+extra-paths option:
+
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... """
+    ... [buildout]
+    ... parts = demo
+    ...
+    ... [demo]
+    ... recipe = zc.recipe.egg
+    ... find-links = %(server)s
+    ... index = %(server)s/index
+    ... scripts = demo=foo
+    ... extra-paths = 
+    ...    /foo/bar
+    ...    /spam/eggs
+    ... """ % dict(server=link_server))
+
+    >>> print system(buildout),
+
+Let's look at the script that was generated:
+
+    >>> cat(sample_buildout, 'bin', 'foo') # doctest: +NORMALIZE_WHITESPACE
+    #!/usr/local/bin/python2.3
+    <BLANKLINE>
+    import sys
+    sys.path[0:0] = [
+      '/tmp/xyzsample-install/demo-0.3-py2.3.egg',
+      '/tmp/xyzsample-install/demoneeded-1.1-py2.3.egg',
+      '/foo/bar',
+      '/spam/eggs'
+      ]
+    <BLANKLINE>
+    import eggrecipedemo
+    <BLANKLINE>
+    if __name__ == '__main__':
+        eggrecipedemo.main()
+
+
+
+Offline mode
+------------
+
+If the buildout offline option is set to "true", then no attempt will
+be made to contact an index server:
+
+
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... """
+    ... [buildout]
+    ... parts = demo
+    ... offline = true
+    ...
+    ... [demo]
+    ... recipe = zc.recipe.egg
+    ... index = eek!
+    ... scripts = demo=foo
+    ... """ % dict(server=link_server))
+
+    >>> print system(buildout),

zc.buildoutsupport/zc.recipe.egg_/src/zc/recipe/egg/__init__.py

+from zc.recipe.egg.egg import Egg
+from zc.recipe.egg.custom import Custom

zc.buildoutsupport/zc.recipe.egg_/src/zc/recipe/egg/api.txt

+Egg Recipe API for other Recipes
+================================
+
+It is common for recipes to accept a collection of egg specifications
+and generate scripts based on the resulting working sets.  The egg
+recipe provides an API that other recipes can use.
+
+A recipe can reuse the egg recipe, supporting the eggs, find-links,
+index, and python options.  This is done by creating an egg recipe
+instance in a recipes's contructor.  In the recipe's install script,
+the egg-recipe instance's working_set method is used to collect the
+requested eggs and working set.
+
+To illustrate, we create a sample recipe that is a very thin layer
+around the egg recipe:
+
+    >>> mkdir(sample_buildout, 'sample')
+    >>> write(sample_buildout, 'sample', 'sample.py', 
+    ... """
+    ... import logging, os
+    ... import zc.recipe.egg
+    ...
+    ... class Sample:
+    ...
+    ...     def __init__(self, buildout, name, options):
+    ...         self.egg = zc.recipe.egg.Egg(buildout, name, options)
+    ...         self.name = name
+    ...         self.options = options
+    ...
+    ...     def install(self):
+    ...         extras = self.options['extras'].split()
+    ...         requirements, ws = self.egg.working_set(extras)
+    ...         print 'Part:', self.name
+    ...         print 'Egg requirements:'
+    ...         for r in requirements:
+    ...             print r
+    ...         print 'Working set:'
+    ...         for d in ws:
+    ...             print d
+    ... """)
+
+Here we instantiated the egg recipe in the constructor, saving it in
+an attribute.  This also initialized the options dictionary.
+
+In our install method, we called the working_set method on the
+instance we saved.  The working_set method takes an optional sequence
+of extra requirements to be included in the working set.
+
+    >>> write(sample_buildout, 'sample', 'setup.py',
+    ... """
+    ... from setuptools import setup
+    ... 
+    ... setup(
+    ...     name = "sample",
+    ...     entry_points = {'zc.buildout': ['default = sample:Sample']},
+    ...     install_requires = 'zc.recipe.egg',
+    ...     )
+    ... """)
+
+
+    >>> write(sample_buildout, 'sample', 'README.txt', " ")
+
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... """
+    ... [buildout]
+    ... develop = sample
+    ... parts = sample-part
+    ...
+    ... [sample-part]
+    ... recipe = sample
+    ... eggs = demo<0.3
+    ... find-links = %(server)s
+    ... index = %(server)sindex
+    ... extras = other
+    ... """ % dict(server=link_server))
+
+    >>> import os
+    >>> os.chdir(sample_buildout)
+    >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout')
+    >>> print system(buildout),
+    Part: sample-part
+    Egg requirements:
+    demo<0.3
+    Working set:
+    demo 0.2
+    other 1.0
+    demoneeded 1.1
+
+We can see that the options were augmented with additionl data
+computed by the egg recipe by looking at .installed.cfg:
+
+    >>> cat(sample_buildout, '.installed.cfg')
+    [buildout]
+    parts = sample-part
+    <BLANKLINE>
+    [sample-part]
+    __buildout_installed__ = 
+    __buildout_signature__ = sample-6aWMvV2EJ9Ijq+bR8ugArQ==
+            zc.recipe.egg-cAsnudgkduAa/Fd+WJIM6Q==
+            setuptools-0.6-py2.4.egg
+            zc.buildout-+rYeCcmFuD1K/aB77XTj5A==
+    _b = /tmp/tmpb7kP9bsample-buildout/bin
+    _d = /tmp/tmpb7kP9bsample-buildout/develop-eggs
+    _e = /tmp/tmpb7kP9bsample-buildout/eggs
+    eggs = demo<0.3
+    executable = /usr/local/bin/python2.3
+    extras = other
+    find-links = http://localhost:27071/
+    index = http://localhost:27071/index
+    recipe = sample

zc.buildoutsupport/zc.recipe.egg_/src/zc/recipe/egg/custom.py

+##############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Install packages as eggs
+
+$Id$
+"""
+
+import os, re, zipfile
+import zc.buildout.easy_install
+
+class Custom:
+
+    def __init__(self, buildout, name, options):
+        self.buildout = buildout
+        self.name = name
+        self.options = options
+        links = options.get('find-links',
+                            buildout['buildout'].get('find-links'))
+        if links:
+            links = links.split()
+            options['find-links'] = '\n'.join(links)
+        else:
+            links = ()
+        self.links = links
+
+        index = options.get('index', buildout['buildout'].get('index'))
+        if index is not None:
+            options['index'] = index
+        self.index = index
+
+        options['_b'] = buildout['buildout']['bin-directory']
+        options['_e'] = buildout['buildout']['eggs-directory']
+        options['_d'] = buildout['buildout']['develop-eggs-directory']
+
+        assert options.get('unzip') in ('true', 'false', None)
+
+        python = options.get('python', buildout['buildout']['python'])
+        options['executable'] = buildout[python]['executable']
+
+        build_ext = {}
+        for be_option in ('include-dirs', 'library-dirs', 'rpath'):
+            value = options.get(be_option)
+            if value is None:
+                continue
+            value = [
+                os.path.join(
+                    buildout['buildout']['directory'],
+                    v.strip()
+                    )
+                for v in value.strip().split('\n')
+                if v.strip()
+            ]
+            build_ext[be_option] = ':'.join(value)
+            options[be_option] = ':'.join(value)
+        self.build_ext = build_ext
+
+    def install(self):
+        if self.buildout['buildout'].get('offline') == 'true':
+            return
+        options = self.options
+        distribution = options.get('eggs', self.name).strip()
+        build_ext = dict([
+            (k, options[k])
+            for k in ('include-dirs', 'library-dirs', 'rpath')
+            if k in options
+            ])
+        zc.buildout.easy_install.build(
+            distribution, options['_d'], self.build_ext,
+            self.links, self.index, options['executable'], [options['_e']],
+            )
+        

zc.buildoutsupport/zc.recipe.egg_/src/zc/recipe/egg/custom.txt

+Custon eggs
+===========
+
+Sometimes, It's necessary to provide extra control over how an egg is
+created.  This is commonly true for eggs with extension modules that
+need to access libraries or include files.
+
+The zc.recipe.egg:custom recipe can be used to define an egg with
+custom build parameters.  The currently defined parameters are:
+
+include-dirs
+   A new-line separated list of directories to search for include
+   files.
+
+library-dirs
+   A new-line separated list of directories to search for libraries
+   to link with.
+
+rpath
+   A new-line separated list of directories to search for dynamic libraries
+   at run time.
+
+In addition, the following options can be used to specify the egg:
+
+egg
+    An eggs to install given as a setuptools requirement string.
+    This defaults to the part name.
+
+find-links
+   A list of URLs, files, or directories to search for distributions.
+
+index
+   The URL of an index server, or almost any other valid URL. :)
+
+   If not specified, the Python Package Index,
+   http://cheeseshop.python.org/pypi, is used.  You can specify an
+   alternate index with this option.  If you use the links option and
+   if the links point to the needed distributions, then the index can
+   be anything and will be largely ignored.  In the examples, here,
+   we'll just point to an empty directory on our link server.  This 
+   will make our examples run a little bit faster.
+
+python
+   The name of a section to get the Python executable from.
+   If not specified, then the buildout python option is used.  The
+   Python executable is found in the executable option of the named
+   section. 
+
+To illustrate this, we'll define a buildout that builds an egg for a
+package that has a simple extension module::
+
+  #include <Python.h>
+  #include <extdemo.h>
+
+  static PyMethodDef methods[] = {};
+
+  PyMODINIT_FUNC
+  initextdemo(void)
+  {
+      PyObject *d;
+      d = Py_InitModule3("extdemo", methods, "");
+      PyDict_SetItemString(d, "val", PyInt_FromLong(EXTDEMO));    
+  }
+
+The extension depends on a system-dependnt include file, extdemo.h,
+that defines a constant, EXTDEMO, that is exposed by the extension.
+
+The extension module is available as a source distribution,
+extdemo-1.4.tar.gz, on a distribution server.
+
+We have a sample buildout that we'll add an include directory to with
+the necessary include file:
+
+    >>> mkdir(sample_buildout, 'include')
+    >>> import os
+    >>> open(os.path.join(sample_buildout, 'include', 'extdemo.h'), 'w').write(
+    ...    "#define EXTDEMO 42\n")
+
+We'll also update the buildout configuration file to define a part for
+the egg:
+
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... """
+    ... [buildout]
+    ... parts = extdemo
+    ...
+    ... [extdemo]
+    ... recipe = zc.recipe.egg:custom
+    ... find-links = %(server)s
+    ... index = %(server)s/index
+    ... include-dirs = include
+    ... """ % dict(server=link_server))
+
+    >>> os.chdir(sample_buildout)
+    >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout')
+
+    >>> print system(buildout),
+    zip_safe flag not set; analyzing archive contents...
+
+We got the zip_safe warning because the source distribution we used
+wasn't setuptools based and thus didn't set the option.
+
+The egg is created in the develop-eggs directory *not* the eggs
+directory because it depends on buildout-specific parameters and the
+eggs directory can be shared across multiple buildouts.
+
+    >>> ls(sample_buildout, 'develop-eggs')
+    d  extdemo-1.4-py2.4-unix-i686.egg
+    -  zc.recipe.egg.egg-link
+
+Note that no scripts or dependencies are installed.  To install
+dependencies or scripts for a custom egg, define another part and use
+the zc.recipe.egg recipe, listing the custom egg as one of the eggs to
+be installed.  The zc.recipe.egg recipe will use the installed egg.

zc.buildoutsupport/zc.recipe.egg_/src/zc/recipe/egg/egg.py

+##############################################################################
+#
+# Copyright (c) 2006 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Install packages as eggs
+
+$Id$
+"""
+
+import os, re, zipfile
+import zc.buildout.easy_install
+
+class Egg:
+
+    def __init__(self, buildout, name, options):
+        self.buildout = buildout
+        self.name = name
+        self.options = options
+        links = options.get('find-links',
+                            buildout['buildout'].get('find-links'))
+        if links:
+            links = links.split()
+            options['find-links'] = '\n'.join(links)
+        else:
+            links = ()
+        self.links = links
+
+        index = options.get('index', buildout['buildout'].get('index'))
+        if index is not None:
+            options['index'] = index
+        self.index = index
+
+        self.extra_paths = [
+            os.path.join(buildout['buildout']['directory'], p.strip())
+            for p in options.get('extra-paths', '').split('\n')
+            if p.strip()
+            ]
+        if self.extra_paths:
+            options['extra-paths'] = '\n'.join(self.extra_paths)
+
+        options['_b'] = buildout['buildout']['bin-directory']
+        options['_e'] = buildout['buildout']['eggs-directory']
+        options['_d'] = buildout['buildout']['develop-eggs-directory']
+
+        assert options.get('unzip') in ('true', 'false', None)
+
+        python = options.get('python', buildout['buildout']['python'])
+        options['executable'] = buildout[python]['executable']
+
+    def working_set(self, extra=()):
+        """Separate method to just get the working set
+
+        This is intended for reuse by similar recipes.
+        """
+        options = self.options
+
+        distributions = [
+            r.strip()
+            for r in options.get('eggs', self.name).split('\n')
+            if r.strip()]
+        orig_distributions = distributions[:]
+        distributions.extend(extra)
+
+        if self.buildout['buildout'].get('offline') == 'true':
+            ws = zc.buildout.easy_install.working_set(
+                distributions, options['executable'],
+                [options['_d'], options['_e']]
+                )
+        else:
+            ws = zc.buildout.easy_install.install(
+                distributions, options['_e'],
+                links = self.links,
+                index = self.index, 
+                executable = options['executable'],
+                always_unzip=options.get('unzip') == 'true',
+                path=[options['_d']]
+                )
+
+        return orig_distributions, ws
+
+    def install(self):
+        distributions, ws = self.working_set()
+        options = self.options
+
+        scripts = options.get('scripts')
+        if scripts or scripts is None:
+            if scripts is not None:
+                scripts = scripts.split()
+                scripts = dict([
+                    ('=' in s) and s.split('=', 1) or (s, s)
+                    for s in scripts
+                    ])
+            return zc.buildout.easy_install.scripts(
+                distributions, ws, options['executable'],
+                options['_b'],
+                scripts=scripts,
+                extra_paths=self.extra_paths)
+

zc.buildoutsupport/zc.recipe.testrunner/README.txt

+Test-Runner Recipe
+==================
+
+This recipe generates zope.testing test-runenr scripts for testing a
+collection of eggs.  The eggs must already be installed (using the
+zc.recipe.egg recipe)
+
+The test-runner recipe has 2 options:
+
+- The eggs option takes the names of the eggs to be
+  tested.  These are not installed by the recipe. They must be
+  installed by some other recipe (or using the buildout develop
+  option).  The distributions are in the form os setuptools
+  requirements.  Multiple distributions must be listed on separate
+  lines. This option is required.
+
+- The script option gives the name of the script to generate, in the
+  buildout bin directory.  Of the option isn't used, the part name
+  will be used.
+
+To do
+-----
+
+- Support specifying testrunner defaults (e.g. verbosity, test file 
+  patterns, etc.)
+

zc.buildoutsupport/zc.recipe.testrunner/setup.py

+from setuptools import setup, find_packages
+
+name = "zc.recipe.testrunner"
+setup(
+    name = name,
+    version = "1.0.0a1",
+    author = "Jim Fulton",
+    author_email = "jim@zope.com",
+    description = "ZC Buildout recipe for creating test runners",
+    long_description=open('README.txt').read(),
+    license = "ZPL 2.1",
+    keywords = "development build testing",
+    url='http://svn.zope.org/zc.buildout',
+
+    packages = find_packages('src'),
+    include_package_data = True,
+    package_dir = {'':'src'},
+    namespace_packages = ['zc', 'zc.recipe'],
+    install_requires = ['zc.buildout', 'zope.testing', 'setuptools',
+                        'zc.recipe.egg',
+                        ],
+    test_suite = name+'.tests.test_suite',
+    entry_points = {'zc.buildout': ['default = %s:TestRunner' % name]},
+    dependency_links = ['http://download.zope.org/distribution/'],
+    )

zc.buildoutsupport/zc.recipe.testrunner/src/zc/__init__.py

+__import__('pkg_resources').declare_namespace(__name__)

zc.buildoutsupport/zc.recipe.testrunner/src/zc/recipe/__init__.py

+__import__('pkg_resources').declare_namespace(__name__)