Commit 9affb95b authored by Jim Fulton's avatar Jim Fulton

Checking in initial work. Still need more tests and features.

parent deefd960
Zope Buildout
=============
The Zope Buildout project provides support for creating applications,
especially Pyton applications. It provides tools for assembling
applications from multiple parts, Python or otherwise. An application
may actually contain multiple programs, processes, and configuration
settings.
The word "buildout" refers to a description of a set of parts and the
software to create ans assemble them. It is often used informally to
refer to an installed system based on a buildout definition. For
example, if we are creating an application named "Foo", then "the Foo
buildout" is the collection of configuration and application-specific
software that allows an instance of the application to be created. We
may refer to such an instance of the application informally as "a Foo
buildout".
I expect that, for many Zope packages, we'll arrange the package
projects in subversion as buildouts. To work on the package, someone
will check the project out of Subversion and build it. Building it
will assemble all of packages and progras needed to work on it. For
example, a buildout for a project to provide a new security policy
will include the source of the policy and specifications to build the
application for working on it, including:
- a test runner
- a web server for running the user interface
- supporting packages
A buildout will typically contain a copy of bootstrap.py. When
someone checks out the project, they'll run bootstrap.py, which will
- create support directories, like bin, eggs, and work, as needed,
- download and install the zc.buildout and setuptools eggs,
- run bin/build (created by installing zc.buildout) to build the
application.
Buildouts are defined using configuration files. These files are
based on the Python ConfigParser module with some variable-definition
and substitution extensions.
The detailed documentation for the various parts of bukdout can be
found in the following files:
bootstrap.txt
Describes how to use the bootstrapping script
buildout.txt
Describes how to define and run buildouts. It also describes how
to write recipes.
recipes.txt
Documents the few built-in recipes.
##############################################################################
#
# Copyright (c) 2005 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Bootstrap a buildout
$Id$
"""
import os, sys, urllib2
for d in 'eggs', 'bin':
if not os.path.exists(d):
os.mkdir(d)
ez = {}
exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
).read() in ez
ez['use_setuptools'](to_dir='eggs', download_delay=0)
import setuptools.command.easy_install
import pkg_resources
import setuptools.package_index
import distutils.dist
os.spawnle(os.P_WAIT, sys.executable, sys.executable, 'setup.py',
'-q', 'develop', '-m', '-x', '-d', 'eggs',
{'PYTHONPATH': os.path.dirname(pkg_resources.__file__)},
)
## easy = setuptools.command.easy_install.easy_install(
## distutils.dist.Distribution(),
## multi_version=True,
## exclude_scripts=True,
## sitepy_installed=True,
## install_dir='eggs',
## outputs=[],
## quiet=True,
## zip_ok=True,
## args=['zc.buildout'],
## )
## easy.finalize_options()
## easy.easy_install('zc.buildout')
env = pkg_resources.Environment(['eggs'])
ws = pkg_resources.WorkingSet()
sys.path[0:0] = [
d.location
for d in ws.resolve([pkg_resources.Requirement.parse('zc.buildout')], env)
]
import zc.buildout.egglinker
zc.buildout.egglinker.scripts(['zc.buildout'], 'bin', ['eggs'])
sys.exit(os.spawnl(os.P_WAIT, 'bin/buildout', 'bin/buildout'))
[buildout]
develop = eggrecipe testrunnerrecipe
parts = test
[test]
recipe = zc.recipe.testrunner
distributions = zc.buildout zc.recipe.egg
Buildout recipe for installing Python distutils distributions as eggs
from setuptools import setup, find_packages
setup(
name = "zc.recipe.egg",
version = "0.1",
packages = find_packages('src'),
include_package_data = True,
package_dir = {'':'src'},
namespace_packages = ['zc', 'zc.recipe'],
install_requires = ['zc.buildout'],
tests_require = ['zope.testing'],
test_suite = 'zc.recipe.eggs.tests.test_suite',
author = "Jim Fulton",
author_email = "jim@zope.com",
description = "Recipe for installing Python package distributions as eggs",
license = "ZPL 2.1",
keywords = "development build",
entry_points = {'zc.buildout': ['default = zc.recipe.egg:Egg']},
)
__import__('pkg_resources').declare_namespace(__name__)
__import__('pkg_resources').declare_namespace(__name__)
Installation of distributions as eggs
=====================================
The zc.recipe.egg ewcipe can be used to install various types if
distutils distributions as eggs. It takes a number of options:
distribution
The distribution specifies the distribution requirement.
This is a requirement as defined by setuptools.
find_links
A list of URLs, files, or directories to search for distributions.
To illustrate this, we've created a directory with some sample eggs:
>>> ls(sample_eggs)
- demo-0.1-py2.3.egg
- demo-0.2-py2.3.egg
- demo-0.3-py2.3.egg
- demoneeded-1.0-py2.3.egg
We have a sample buildout. Let's update it's configuration file to
install the demo package.
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = zc.recipe.egg
... distribution = demo <0.3
... find_links = %s
... """ % sample_eggs)
In this example, we limited ourself to revisions before 0.3. We also
specified where to find distributions using the find_links option.
Let's run the buildout:
>>> import os
>>> os.chdir(sample_buildout)
>>> runscript = os.path.join(sample_buildout, 'bin', 'buildout')
>>> print system(runscript),
Now, if we look at the buildout eggs directory:
>>> ls(sample_buildout, 'eggs')
- demo-0.2-py2.3.egg
- demoneeded-1.0-py2.3.egg
- zc.recipe.egg.egg-link
We see that we got an egg for demo that met the requirement, as well
as the egg for demoneeded, wich demo requires. (We also see an egg
link for the recipe. This egg link was actually created as part of
the sample buildout setup. Normally, when using the recipe, you'll get
a regular egg installation.)
The demo egg also defined a script and we see that the script was
installed as well:
>>> ls(sample_buildout, 'bin')
- buildout
- demo
- py_demo
Here, in addition to the buildout script, we see the demo script,
demo, and we see a script, py_demo, for giving us a Python prompt with
the path for demo and any eggs it depends on included in sys.path.
This is useful for testing.
If we run the demo script, it prints out some minimal data:
>>> print system(os.path.join(sample_buildout, 'bin', 'demo')),
2 1
The value it prints out happens to be some values defined in the
modules installed.
We can also run the py_demo script. Here we'll just print out
the bits if the path added to reflect the eggs:
>>> print system(os.path.join(sample_buildout, 'bin', 'py_demo'),
... """for p in sys.path[:3]:
... print p
... """).replace('>>> ', '').replace('... ', ''),
... # doctest: +ELLIPSIS
<BLANKLINE>
/usr/local/python/2.3.5/lib/python/setuptools-0.6b2-py2.3.egg
/tmp/tmpcy8MvGbuildout-tests/eggs/demo-0.2-py2.3.egg
/tmp/tmpcy8MvGbuildout-tests/eggs/demoneeded-1.0-py2.3.egg
<BLANKLINE>
from zc.recipe.egg.egg import Egg
##############################################################################
#
# Copyright (c) 2006 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Install packages as eggs
$Id$
"""
import zc.buildout.egglinker
import zc.buildout.easy_install
class Egg:
def __init__(self, buildout, name, options):
self.buildout = buildout
self.name = name
self.options = options
def install(self):
distribution = self.options.get('distribution', self.name)
links = self.options.get(
'find_links',
self.buildout['buildout'].get('find_links'),
)
if links:
links = links.split()
else:
links = ()
buildout = self.buildout
zc.buildout.easy_install.install(
distribution,
buildout.eggs,
[buildout.buildout_path(link) for link in links],
always_copy = True,
)
zc.buildout.egglinker.scripts(
[distribution], buildout.bin, [buildout.eggs],
)
##############################################################################
#
# Copyright (c) 2006 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
import os, re, shutil, sys, tempfile
import pkg_resources
import zc.buildout.testing
import unittest
from zope.testing import doctest, renormalizing
def runsetup(d):
here = os.getcwd()
try:
os.chdir(d)
os.spawnle(
os.P_WAIT, sys.executable, sys.executable,
'setup.py', '-q', 'bdist_egg',
{'PYTHONPATH': os.path.dirname(pkg_resources.__file__)},
)
shutil.rmtree('build')
finally:
os.chdir(here)
def dirname(d, level=1):
if level == 0:
return d
return dirname(os.path.dirname(d), level-1)
def setUp(test):
zc.buildout.testing.buildoutSetUp(test)
open(os.path.join(test.globs['sample_buildout'],
'eggs', 'zc.recipe.egg.egg-link'),
'w').write(dirname(__file__, 4))
sample = tempfile.mkdtemp('eggtest')
test.globs['_sample_eggs_container'] = sample
test.globs['sample_eggs'] = os.path.join(sample, 'dist')
zc.buildout.testing.write(sample, 'README.txt', '')
zc.buildout.testing.write(sample, 'eggrecipedemobeeded.py', 'y=1\n')
zc.buildout.testing.write(
sample, 'setup.py',
"from setuptools import setup\n"
"setup(name='demoneeded', py_modules=['eggrecipedemobeeded'],"
" zip_safe=True, version='1.0')\n"
)
runsetup(sample)
os.remove(os.path.join(sample, 'eggrecipedemobeeded.py'))
for i in (1, 2, 3):
zc.buildout.testing.write(
sample, 'eggrecipedemo.py',
'import eggrecipedemobeeded\n'
'x=%s\n'
'def main(): print x, eggrecipedemobeeded.y\n'
% i)
zc.buildout.testing.write(
sample, 'setup.py',
"from setuptools import setup\n"
"setup(name='demo', py_modules=['eggrecipedemo'],"
" install_requires = 'demoneeded',"
" entry_points={'console_scripts': ['demo = eggrecipedemo:main']},"
" zip_safe=True, version='0.%s')\n" % i
)
runsetup(sample)
def tearDown(test):
shutil.rmtree(test.globs['_sample_eggs_container'])
zc.buildout.testing.buildoutTearDown(test)
def test_suite():
return unittest.TestSuite((
#doctest.DocTestSuite(),
doctest.DocFileSuite(
'README.txt',
setUp=setUp, tearDown=tearDown,
checker=renormalizing.RENormalizing([
(re.compile('\S+[/%(sep)s]'
'(\\w+-)[^ \t\n%(sep)s/]+.egg'
% dict(sep=os.path.sep)
),
'\\1-VVV-egg')
])
),
))
if __name__ == '__main__':
unittest.main(defaultTest='test_suite')
from setuptools import setup, find_packages
setup(
name = "zc.buildout",
version = "0.1",
packages = ['zc.buildout'],
package_dir = {'':'src'},
namespace_packages = ['zc'],
include_package_data = True,
tests_require = ['zope.testing'],
test_suite = 'zc.buildout.tests.test_suite',
author = "Jim Fulton",
author_email = "jim@zope.com",
description = "System for managing development buildouts",
license = "ZPL 2.1",
keywords = "development build",
install_requires = 'setuptools',
entry_points = {'console_scripts': ['buildout = zc.buildout.build:main']},
)
try:
__import__('pkg_resources').declare_namespace(__name__)
except:
# bootstrapping
pass
##############################################################################
#
# Copyright (c) 2005 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Buildout main script
$Id$
"""
import md5
import os
import pprint
import re
import shutil
import sys
import ConfigParser
import zc.buildout.easy_install
import pkg_resources
import zc.buildout.easy_install
import zc.buildout.egglinker
class MissingOption(KeyError):
"""A required option was missing
"""
class Options(dict):
def __init__(self, buildout, section, data):
self.buildout = buildout
self.section = section
super(Options, self).__init__(data)
def __getitem__(self, option):
try:
return super(Options, self).__getitem__(option)
except KeyError:
# XXX need test
raise MissingOption("Missing option", self.section, option)
def copy(self):
return Options(self.buildout, self.section, self)
class Buildout(dict):
def __init__(self):
self._buildout_dir = os.path.abspath(os.getcwd())
self._config_file = self.buildout_path('buildout.cfg')
super(Buildout, self).__init__(self._open(
directory = self._buildout_dir,
eggs_directory = 'eggs',
bin_directory = 'bin',
parts_directory = 'parts',
installed = '.installed.cfg',
))
options = self['buildout']
links = options.get('find_links', '')
self._links = links and links.split() or ()
# XXX need tests for alternate directory locations
for name in ('bin', 'parts', 'eggs'):
d = self.buildout_path(options[name+'_directory'])
setattr(self, name, d)
if not os.path.exists(d):
os.mkdir(d)
_template_split = re.compile('([$]{\w+:\w+})').split
def _open(self, **predefined):
# Open configuration files
parser = ConfigParser.SafeConfigParser()
parser.add_section('buildout')
for k, v in predefined.iteritems():
parser.set('buildout', k, v)
parser.read(self._config_file)
data = dict([
(section,
Options(self, section,
[(k, v.strip()) for (k, v) in parser.items(section)])
)
for section in parser.sections()
])
converted = {}
for section, options in data.iteritems():
for option, value in options.iteritems():
if '$' in value:
value = self._dosubs(section, option, value,
data, converted, [])
options[option] = value
converted[(section, option)] = value
# XXX need various error tests
return data
def _dosubs(self, section, option, value, data, converted, seen):
key = section, option
r = converted.get(key)
if r is not None:
return r
if key in seen:
raise ValueError('Circular references', seen, key)
seen.append(key)
value = '$$'.join([self._dosubs_esc(s, data, converted, seen)
for s in value.split('$$')
])
seen.pop()
return value
def _dosubs_esc(self, value, data, converted, seen):
value = self._template_split(value)
subs = []
for s in value[1::2]:
s = tuple(s[2:-1].split(':'))
v = converted.get(s)
if v is None:
options = data.get(s[0])
if options is None:
raise KeyError("Referenced section does not exist", s[0])
v = options.get(s[1])
if v is None:
raise KeyError("Referenced option does not exist", *s)
if '$' in v:
v = _dosubs(s[0], s[1], v, data, converted, seen)
options[s[1]] = v
converted[s] = v
subs.append(v)
subs.append('')
return ''.join([''.join(v) for v in zip(value[::2], subs)])
# XXX test
def buildout_path(self, *names):
return os.path.join(self._buildout_dir, *names)
# XXX test
def distributions_path(self, specs):
return zc.buildout.egglinker.path(specs, [self.eggs])
def install(self):
self._develop()
new_part_options = self._gather_part_info()
installed_part_options = self._read_installed_part_options()
old_parts = installed_part_options['buildout']['parts'].split()
old_parts.reverse()
new_old_parts = []
for part in old_parts:
installed_options = installed_part_options[part].copy()
installed = installed_options.pop('__buildout_installed__')
if installed_options != new_part_options.get(part):
self._uninstall(installed)
del installed_part_options[part]
else:
new_old_parts.append(part)
new_old_parts.reverse()
new_parts = []
try:
for part in new_part_options['buildout']['parts'].split():
installed = self._install(part)
new_part_options[part]['__buildout_installed__'] = installed
new_parts.append(part)
installed_part_options[part] = new_part_options[part]
new_old_parts = [p for p in new_old_parts if p != part]
finally:
new_parts.extend(new_old_parts)
installed_part_options['buildout']['parts'] = ' '.join(new_parts)
self._save_installed_options(installed_part_options)
def _develop(self):
"""Install sources by running setup.py develop on them
"""
develop = self['buildout'].get('develop')
if develop:
here = os.getcwd()
try:
for setup in develop.split():
setup = self.buildout_path(setup)
if os.path.isdir(setup):
setup = os.path.join(setup, 'setup.py')
os.chdir(os.path.dirname(setup))
os.spawnle(
os.P_WAIT, sys.executable, sys.executable,
setup, '-q', 'develop', '-m', '-x',
'-f', ' '.join(self._links),
'-d', self.eggs,
{'PYTHONPATH':
os.path.dirname(pkg_resources.__file__)},
)
finally:
os.chdir(os.path.dirname(here))
def _gather_part_info(self):
"""Get current part info, including part options and recipe info
"""
parts = self['buildout']['parts']
part_info = {'buildout': {'parts': parts}}
recipes_requirements = []
pkg_resources.working_set.add_entry(self.eggs)
parts = parts and parts.split() or []
for part in parts:
options = self.get(part)
if options is None:
options = self[part] = {}
options = options.copy()
recipe, entry = self._recipe(part, options)
zc.buildout.easy_install.install(
recipe, self.eggs, self._links)
recipes_requirements.append(recipe)
part_info[part] = options
# Load up the recipe distros
pkg_resources.require(recipes_requirements)
base = self.eggs + os.path.sep
for part in parts:
options = part_info[part]
recipe, entry = self._recipe(part, options)
req = pkg_resources.Requirement.parse(recipe)
sig = _dists_sig(pkg_resources.working_set.resolve([req]), base)
options['__buildout_signature__'] = ' '.join(sig)
return part_info
def _recipe(self, part, options):
recipe = options.get('recipe', part)
if ':' in recipe:
recipe, entry = recipe.split(':')
else:
entry = 'default'
return recipe, entry
def _read_installed_part_options(self):
old = self._installed_path()
if os.path.isfile(old):
parser = ConfigParser.SafeConfigParser()
parser.read(old)
return dict([(section, dict(parser.items(section)))
for section in parser.sections()])
else:
return {'buildout': {'parts': ''}}
def _installed_path(self):
return self.buildout_path(self['buildout']['installed'])
def _uninstall(self, installed):
for f in installed.split():
f = self.buildout_path(f)
if os.path.isdir(f):
shutil.rmtree(f)
elif os.path.isfile(f):
os.remove(f)
def _install(self, part):
options = self[part]
recipe, entry = self._recipe(part, options)
recipe_class = pkg_resources.load_entry_point(
recipe, 'zc.buildout', entry)
installed = recipe_class(self, part, options).install()
if installed is None:
installed = []
elif isinstance(installed, basestring):
installed = [installed]
base = self.buildout_path('')
installed = [d.startswith(base) and d[len(base):] or d
for d in installed]
return ' '.join(installed)
def _save_installed_options(self, installed_options):
parser = ConfigParser.SafeConfigParser()
for section in installed_options:
parser.add_section(section)
for option, value in installed_options[section].iteritems():
parser.set(section, option, value)
parser.write(open(self._installed_path(), 'w'))
def _dir_hash(dir):
hash = md5.new()
for (dirpath, dirnames, filenames) in os.walk(dir):
filenames[:] = [f for f in filenames
if not (f.endswith('pyc') or f.endswith('pyo'))
]
hash.update(' '.join(dirnames))
hash.update(' '.join(filenames))
for name in filenames:
hash.update(open(os.path.join(dirpath, name)).read())
return hash.digest().encode('base64').strip()
def _dists_sig(dists, base):
result = []
for dist in dists:
location = dist.location
if dist.precedence == pkg_resources.DEVELOP_DIST:
result.append(dist.project_name + '-' + _dir_hash(location))
else:
if location.startswith(base):
location = location[len(base):]
result.append(location)
return result
def main():
Buildout().install()
Defining Buildouts
==================
This document describes how to define buildouts using buildout
configuation files and recipes. It doesn't describe how to bootstrap
a buildout. To find out how to do that, see bootstrap.txt. For the
examples we show here, we've created a sample buildout that already
contains the mimimal software needed for a buildout.
Buildouts are defined using configuration files. These are in the
format defined by the Python ConfigParser module, with an extension
that we'll describe later. When a buildout is run, it looks for
the file buildout.cfg in the directory where the buidout is run. It
will optionally look for buildout-instance.cfg. Typically, buidout.cfg
contains information common to all instances of a buildout and is
checked in, and buildout-instance.cfg has instance-specific information.
We have a sample buildout that has already been created for us. It
has the absolute minimum information. We have bin, eggs and parts
directories, and a configuration file:
>>> ls(sample_buildout)
- .installed.cfg
d bin
- buildout.cfg
d eggs
d parts
The bin directory contains scripts. A minimal buildout has a build
script and a py_zc.buildout script:
>>> ls(sample_buildout, 'bin')
- buildout
The build script is what we run to build things out. The
py_zc.buildout script gives us a Python prompt with the Python path
set to that needed by the zc.buildout package.
The eggs directory is initially empty. This is typically the case
when the zc.buildout and setuptools are installed externally to the
buildout:
>>> ls(sample_buildout, 'eggs')
They can also be installed locally in a buildout, in which case they's
show up as eggs in the eggs directory.
The parts directory is initially empty:
>>> ls(sample_buildout, 'parts')
The parts directory provides an area where recipies can install
part data. For example, if we built a custom Python, we would
install it in the part directory. Part data is stored in a
subdirectory of the parts directory with the same name as the part.
The file .installed.cfg contains information about previously installed
parts. Because this is a new buildout, this file isn't very
interesting:
>>> cat(sample_buildout, '.installed.cfg')
[buildout]
parts =
The minimal configuration file has a buildout section that defines no
parts:
>>> cat(sample_buildout, 'buildout.cfg')
[buildout]
parts =
A part is simply something to be created by a buildout. It can be
almost anything, such as a Python package, a program, a directory, or
a confguration file.
A part is created by a recipe. Recipes are always installed as Python
eggs. They can be downloaded from an package server, such as the
Python package index, or they can be developed as part of a project.
Let's create a recipe as part of the sample project. We'll create a
recipe for creating directories.
First, we'll create a recipes directory for
our local recipes:
>>> mkdir(sample_buildout, 'recipes')
and then we'll create a source file for our mkdir recipe:
>>> write(sample_buildout, 'recipes', 'mkdir.py',
... """
... import os
...
... class Mkdir:
...
... def __init__(self, buildout, name, options):
... self.buildout = buildout
... self.name = name
... self.options = options
...
... def install(self):
... path = self.buildout.buildout_path(self.options['path'])
... if not os.path.isdir(path):
... print 'Creating directory', os.path.basename(path)
... os.mkdir(path)
... return path
... """)
The recipe defines a constructor that takes a buildout object, a part
name, and an options dictionary. It saves them in instance attributes.
The install method is responsible for creating the part. In this
case, we need the path of the directory to create. We'll use a
buildout option from our options dictionary. If the path is relative,
we'll interpret it relative to the buildout directory. The buildout
buildout_path method gives us a path relative to the buildout. It
uses os.path.join, so if we pass it an absolute path, we'll get the
absolute path back. (If no arguments are passed to base_path, then the
buildout directory is returned.)
We made the method chatty so that we can observe what it's doing.
We return the path that we installed. If the part is unistalled or
reinstalled, then the path returned will be removed by the buildout
machinery. A recipe install method is expected to return None, a
string, or an iterable of strings containing paths to be removed if a
part is uninstalled.
We need to provide packaging information so that our recipe can be
installed as an egg. We need to define a setup script for this:
>>> write(sample_buildout, 'recipes', 'setup.py',
... """
... from setuptools import setup
...
... setup(
... name = "recipes",
... entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']},
... )
... """)
Here we've defined a package containing just our module. We've also
defined an entry point. Entry points provide a way for an egg to
define the services it provides. Here we've said that we define a
zc.buildout entry point named default. Recipe classes must be exposed
as entry points in the zc.buildout group. we give entry points names
within the group. The name "default" is somewhat special because it
allows a recipe to be referenced using a package name without naming
an entry point.
We also need a README.txt for our recipes to avoid a warning:
>>> write(sample_buildout, 'recipes', 'README.txt', " ")
Now let's update our buildout.cfg:
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... develop = recipes
... parts = data_dir
...
... [data_dir]
... recipe = recipes:mkdir
... path = mystuff
... """)
Let's go through the changes one by one::
develop = recipes
This tells the buildout to install a development egg for our recipes.
Any number of paths can be listed. The paths can be relative or
absolute. If relative, they are treated as relative to the buidlout
directory. They can be directory or file paths. If a file path is
given, it should point to a Python setup script. If a directory path
is given, it should point to a directory containing a setup.py.
Development eggs are installed before building any parts, as they may
provide locally-defined recipes needed by the parts.
::
parts = data_dir
Here we've named a part to be "built". We can use any name we want
except that different part names must be unique and recipes will often
use the part name to decide what to do.
::
[data_dir]
recipe = recipes:mkdir
path = mystuff
Generally, when we name a part, we also create a section of the same
name that contains part data. In this section, we'll usually define
the recipe to be used to install the part. In this case, we also
specify the path to be created.
Let's run the buildout. We do so by running the build script in the
buildout:
>>> import os
>>> os.chdir(sample_buildout)
>>> runscript = os.path.join(sample_buildout, 'bin', 'buildout')
>>> print system(runscript),
Creating directory mystuff
We see that the recipe created the directory, as expected:
>>> ls(sample_buildout)
- .installed.cfg
d bin
- buildout.cfg
d eggs
d mystuff
d parts
d recipes
In addition, .installed.cfg has been updated to reflect the part we
installed:
>>> cat(sample_buildout, '.installed.cfg')
[buildout]
parts = data_dir
<BLANKLINE>
[data_dir]
path = mystuff
recipe = recipes:mkdir
__buildout_signature__ = recipes-O3ypTgKOkHMqMwKvMfvHnA==
__buildout_installed__ = mystuff
<BLANKLINE>
Note that the directory we installed is included in .installed.cfg.
If we change the name of the directory in the configuration file,
we'll see that the directory gets removed and recreated:
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... develop = recipes
... parts = data_dir
...
... [data_dir]
... recipe = recipes:mkdir
... path = mydata
... """)
>>> print system(runscript),
Creating directory mydata
>>> ls(sample_buildout)
- .installed.cfg
d bin
- buildout.cfg
d eggs
d mydata
d parts
d recipes
Variable substitutions
----------------------
Buildout configuration files support two kinds of substitutions,
standard ConfigParser substitutions, and string-template
substitutions. To illustrate this, we'll create an debug recipe to
allow us to see interactions with the buildout:
>>> write(sample_buildout, 'recipes', 'debug.py',
... """
... class Debug:
...
... def __init__(self, buildout, name, options):
... self.buildout = buildout
... self.name = name
... self.options = options
...
... def install(self):
... items = self.options.items()
... items.sort()
... for option, value in items:
... print option, value
... """)
In this example, we've used a simple base class that provides a
boilerplate constructor. This recipe doesn't actually create
anything. The install method doesn't return anything, because it
didn't create any files or directories.
We also have to update our setup script:
>>> write(sample_buildout, 'recipes', 'setup.py',
... """
... from setuptools import setup
... entry_points = (
... '''
... [zc.buildout]
... mkdir = mkdir:Mkdir
... debug = debug:Debug
... ''')
... setup(name="recipes", entry_points=entry_points)
... """)
We've rearranged the script a bit to make the entry points easier to
edit.
Let's update our configuration to provide variable substitution
examples:
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... develop = recipes
... parts = data_dir debug
...
... [debug]
... recipe = recipes:debug
... file1 = ${data_dir:path}/file
... file2 = %(file1)s.out
... file3 = %(base)s/file3
...
... [data_dir]
... recipe = recipes:mkdir
... path = mydata
...
... [DEFAULT]
... base = var
... """)
In this example, we've used ConfigParser substitutions for file2 and
file3. This type of substitution uses Python string format syntax.
Valid names are option in the same section and options defined in the
DEFAULT section. We used a string-template substitution for file1.
This type of substituion uses the string.Template syntax. Names
substited are qualified option names, consisting of a section name and
option name joined by a colon.
Now, if we run the buildout, we'll see the options with the values
substituted.
>>> print system(runscript),
Creating directory mydata
base var
file1 mydata/file
file2 mydata/file.out
file3 var/file3
recipe recipes:debug
It might seem surprising that mydata was created again. This is
because we changed our recipes package by adding the debug module.
The buildout system didn't know if this module could effect the mkdir
recipe, so it assumed it could and reinstalled mydata. If we rerun
the buildout:
>>> print system(runscript),
base var
file1 mydata/file
file2 mydata/file.out
file3 var/file3
recipe recipes:debug
We can see that mydata was not recreated.
##############################################################################
#
# Copyright (c) 2005 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Python easy_install API
This module provides a high-level Python API for installing packages.
It doesn't install scripts. It uses setuptools and requires it to be
installed.
$Id$
"""
# XXX needs doctest
import setuptools.command.easy_install
import pkg_resources
import setuptools.package_index
import distutils.dist
import distutils.log
def install(spec, dest, links=(), **kw):
index = setuptools.package_index.PackageIndex()
index.add_find_links(links)
easy = setuptools.command.easy_install.easy_install(
distutils.dist.Distribution(),
multi_version=True,
exclude_scripts=True,
sitepy_installed=True,
install_dir=dest,
outputs=[],
verbose = 0,
args = [spec],
find_links = links,
**kw
)
easy.finalize_options()
old_warn = distutils.log.warn
distutils.log.warn = lambda *a, **k: None
easy.easy_install(spec, deps=True)
distutils.log.warn = old_warn
##############################################################################
#
# Copyright (c) 2005 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Egg linker -- Link eggs together to build applications
Egg linker is a script that generates startup scripts for eggs that
include an egg's working script in the generated script.
The egg linker module also exports helper functions of varous kinds to
assist in custom script generation.
$Id$
"""
# XXX needs doctest
import os
import sys
import pkg_resources
def distributions(reqs, eggss):
env = pkg_resources.Environment(eggss)
ws = pkg_resources.WorkingSet()
reqs = [pkg_resources.Requirement.parse(r) for r in reqs]
reqs.append(pkg_resources.Requirement.parse('setuptools'))
return ws.resolve(reqs, env=env)
def path(reqs, eggss):
dists = distributions(reqs, eggss)
return [dist.location for dist in dists]
def location(spec, eggss):
env = pkg_resources.Environment(eggss)
req = pkg_resources.Requirement.parse(spec)
dist = env.best_match(req, pkg_resources.WorkingSet())
return dist.location
def scripts(reqs, dest, eggss):
dists = distributions(reqs, eggss)
reqs = [pkg_resources.Requirement.parse(r) for r in reqs]
projects = [r.project_name for r in reqs]
path = "',\n '".join([dist.location for dist in dists])
for dist in dists:
if dist.project_name in projects:
for name in pkg_resources.get_entry_map(dist, 'console_scripts'):
_script(dist, name, path, os.path.join(dest, name))
_pyscript(path, os.path.join(dest, 'py_'+dist.project_name))
def _script(dist, name, path, dest):
open(dest, 'w').write(script_template % dict(
python = sys.executable,
path = path,
project = dist.project_name,
name = name,
))
try:
os.chmod(dest, 0755)
except (AttributeError, os.error):
pass
script_template = '''\
#!%(python)s
import sys
sys.path[0:0] = [
'%(path)s'
]
from pkg_resources import load_entry_point
sys.exit(
load_entry_point('%(project)s', 'console_scripts', '%(name)s')()
)
'''
def _pyscript(path, dest):
open(dest, 'w').write(py_script_template % dict(
python = sys.executable,
path = path,
))
try:
os.chmod(dest,0755)
except (AttributeError, os.error):
pass
py_script_template = '''\
#!%(python)s -i
import sys
sys.path[0:0] = [
'%(path)s'
]
'''
def main():
import pdb; pdb.set_trace()
##############################################################################
#
# Copyright (c) 2004 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""XXX short summary goes here.
$Id$
"""
import os, re, shutil, sys, tempfile, unittest
from zope.testing import doctest, renormalizing
def cat(dir, *names):
path = os.path.join(dir, *names)
print open(path).read(),
def ls(dir, *subs):
if subs:
dir = os.path.join(dir, *subs)
names = os.listdir(dir)
names.sort()
for name in names:
if os.path.isdir(os.path.join(dir, name)):
print 'd ',
else:
print '- ',
print name
def mkdir(dir, *subs):
if subs:
dir = os.path.join(dir, *subs)
os.mkdir(dir)
def write(dir, *args):
open(os.path.join(dir, *(args[:-1])), 'w').write(args[-1])
def system(command, input=''):
i, o = os.popen4(command)
if input:
i.write(input)
i.close()
return o.read()
def dirname(path, n=1):
if n <= 0:
return path
return dirname(os.path.dirname(path), n-1)
def buildoutSetUp(test):
sample = tempfile.mkdtemp('buildout-tests')
for name in ('bin', 'eggs', 'parts'):
os.mkdir(os.path.join(sample, name))
# make sure we can import zc.buildout and setuptools
import zc.buildout, setuptools
# Generate buildout script
dest = os.path.join(sample, 'bin', 'buildout')
open(dest, 'w').write(
script_template % dict(python=sys.executable, path=sys.path)
)
try:
os.chmod(dest, 0755)
except (AttributeError, os.error):
pass
open(os.path.join(sample, 'buildout.cfg'), 'w').write(
"[buildout]\nparts =\n"
)
open(os.path.join(sample, '.installed.cfg'), 'w').write(
"[buildout]\nparts =\n"
)
test.globs.update(dict(
__here = os.getcwd(),
sample_buildout = sample,
ls = ls,
cat = cat,
mkdir = mkdir,
write = write,
system = system,
__original_wd__ = os.getcwd(),
))
def buildoutTearDown(test):
shutil.rmtree(test.globs['sample_buildout'])
os.chdir(test.globs['__original_wd__'])
script_template = '''\
#!%(python)s
import sys
sys.path[0:0] = %(path)r
from pkg_resources import load_entry_point
sys.exit(load_entry_point('zc.buildout', 'console_scripts', 'buildout')())
'''
##############################################################################
#
# Copyright (c) 2004 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""XXX short summary goes here.
$Id$
"""
import unittest
from zope.testing import doctest
from zc.buildout.testing import buildoutSetUp, buildoutTearDown
def test_suite():
return unittest.TestSuite((
#doctest.DocTestSuite(),
doctest.DocFileSuite(
'buildout.txt',
setUp=buildoutSetUp, tearDown=buildoutTearDown,
),
))
if __name__ == '__main__':
unittest.main(defaultTest='test_suite')
Recipe for generating a custom test runner.
from setuptools import setup, find_packages
setup(
name = "zc.recipe.testrunner",
version = "0.1",
packages = find_packages('src'),
include_package_data = True,
package_dir = {'':'src'},
namespace_packages = ['zc', 'zc.recipe'],
install_requires = ['zc.buildout', 'zope.testing'],
dependency_links = ['http://download.zope.org/distribution/'],
test_suite = 'zc.recipe.testrunner.tests.test_suite',
author = "Jim Fulton",
author_email = "jim@zope.com",
description = "ZC Buildout recipe for creating test runners",
license = "ZPL 2.1",
keywords = "development build",
entry_points = {'zc.buildout':
['default = zc.recipe.testrunner:TestRunner']},
)
__import__('pkg_resources').declare_namespace(__name__)
__import__('pkg_resources').declare_namespace(__name__)
##############################################################################
#
# Copyright (c) 2006 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""A few built-in recipes
$Id$
"""
# XXX need tests
import os, sys
class TestRunner:
def __init__(self, buildout, name, options):
self.buildout = buildout
self.name = name
self.options = options
def install(self):
distributions = self.options['distributions'].split()
path = self.buildout.distributions_path(distributions+['zope.testing'])
locations = [self.buildout.distribution_location(distribution)
for distribution in distributions]
script = self.options.get('script', self.name)
script = self.buildout.buildout_path('bin', script)
open(script, 'w').write(tests_template % dict(
PYTHON=sys.executable,
PATH="',\n '".join(path),
TESTPATH="',\n '--test-path', '".join(locations),
))
try:
os.chmod(script, 0755)
except (AttributeError, os.error):
pass
tests_template = """#!%(PYTHON)s
import sys
sys.path[0:0] = [
'%(PATH)s',
]
from zope.testing import testrunner
defaults = [
'--test-path', '%(TESTPATH)s',
]
sys.exit(testrunner.run(defaults))
"""
- Missing tests. See XXXs
- Buildout command-line options:
--config -C specify a config files
--option -O specify options
- Python discovery support
(Perhaps this is best handled by DEFAULT section.
- Common recipes
- configure-make-make-install
- download, checkout
- Should ot be possible to provide multiple recipies?
Or should recipies be combined through inheritence (or
composition)?
- Python
- Need to better understand the way upgrading works in setuptools.
- Multiple setupfiles,
o extends, optionally-extends, extended-by, optionally-exteded-by
o instance.ini
o ~/buildout/default.ini
- Offline mode
- Local download cache
- Logging
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment