Skip to content

S
slapos.buildout
Commit 2530eeb2 authored by Jim Fulton's avatar Jim Fulton
Browse files
Options

Major refactoring. The original motivation was to get the newest 

distributions available. This required working around issues with
easy_install's --upgrade option:

- Upgrade is not recursive. Upgrading a distro doesn't update it's
  dependencies.

- Upgrade doesn't try very hard to avoid searching.  If we require a
  specific version of a distribution, and we already have that
  distribution, there's no point in looking for a newer one.

- easy_install has kind of odd rules for deciding when to look at an
  index.   Now that we use upgrade all the time, easy_install always
  wants to look at an index.

- We get warnings when connecting to index servers, like PyPI that 
  return text/plain not found messages.

We now have much greater control over how dependencies are
managed. We've essentially taken this over from easy_install.

Because we now always talk to an index server and because we want to
control anything we do in a test, many of the tests actually run their
own web servers.

Anyway:

- Now handle upgrades correctly, I think.

- The egg recipe can now install multiple distributions.

- We have the beginnings of offline mode.

- The internal architeture is much cleaner.

- We've merged the easy_install and egglinker modules, tossing
  some superfluois apis in the egglinker module.
parent e74cdf1a
Hide whitespace changes
Inline Side-by-side
Showing with 877 additions and 531 deletions
bootstrap.py
View file @ 2530eeb2
...@@ -25,42 +25,17 @@ for d in 'eggs', 'develop-eggs', 'bin': ...@@ -25,42 +25,17 @@ for d in 'eggs', 'develop-eggs', 'bin':
ez = {} ez = {}
exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py' exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
).read() in ez ).read() in ez
ez['use_setuptools'](to_dir='eggs', download_delay=0) ez['use_setuptools'](to_dir='eggs', download_delay=0)
import setuptools.command.easy_install
import pkg_resources import pkg_resources
import setuptools.package_index
import distutils.dist
os.spawnle(os.P_WAIT, sys.executable, sys.executable, 'setup.py', os.spawnle(os.P_WAIT, sys.executable, sys.executable, 'setup.py',
'-q', 'develop', '-m', '-x', '-d', 'develop-eggs', '-q', 'develop', '-m', '-x', '-d', 'develop-eggs',
{'PYTHONPATH': os.path.dirname(pkg_resources.__file__)}, {'PYTHONPATH': os.path.dirname(pkg_resources.__file__)},
) )
pkg_resources.working_set.add_entry('src')
## easy = setuptools.command.easy_install.easy_install( import zc.buildout.easy_install
## distutils.dist.Distribution(), zc.buildout.easy_install.scripts(
## multi_version=True, ['zc.buildout'], pkg_resources.working_set , sys.executable, 'bin')
## exclude_scripts=True,
## sitepy_installed=True,
## install_dir='eggs',
## outputs=[],
## quiet=True,
## zip_ok=True,
## args=['zc.buildout'],
## )
## easy.finalize_options()
## easy.easy_install('zc.buildout')
env = pkg_resources.Environment(['develop-eggs', 'eggs'])
ws = pkg_resources.WorkingSet()
sys.path[0:0] = [
d.location
for d in ws.resolve([pkg_resources.Requirement.parse('zc.buildout')], env)
]
import zc.buildout.egglinker
zc.buildout.egglinker.scripts(['zc.buildout'], 'bin', ['eggs'])
sys.exit(os.spawnl(os.P_WAIT, 'bin/buildout', 'bin/buildout')) sys.exit(os.spawnl(os.P_WAIT, 'bin/buildout', 'bin/buildout'))
buildout.cfg
View file @ 2530eeb2
...@@ -2,10 +2,12 @@ ...@@ -2,10 +2,12 @@
develop = eggrecipe testrunnerrecipe develop = eggrecipe testrunnerrecipe
parts = test parts = test
# prevent slow access to cheeseshop:
index = http://download.zope.org
[test] [test]
recipe = zc.recipe.testrunner recipe = zc.recipe.testrunner
distributions = distributions =
zc.buildout zc.buildout
zc.recipe.egg zc.recipe.egg
zc.recipe.testrunner zc.recipe.testrunner
eggrecipe/setup.py
View file @ 2530eeb2
...@@ -7,7 +7,7 @@ setup( ...@@ -7,7 +7,7 @@ setup(
include_package_data = True, include_package_data = True,
package_dir = {'':'src'}, package_dir = {'':'src'},
namespace_packages = ['zc', 'zc.recipe'], namespace_packages = ['zc', 'zc.recipe'],
install_requires = ['zc.buildout'], install_requires = ['zc.buildout', 'setuptools'],
tests_require = ['zope.testing'], tests_require = ['zope.testing'],
test_suite = 'zc.recipe.eggs.tests.test_suite', test_suite = 'zc.recipe.eggs.tests.test_suite',
author = "Jim Fulton", author = "Jim Fulton",
......
eggrecipe/src/zc/recipe/egg/README.txt
View file @ 2530eeb2
...@@ -11,9 +11,23 @@ distribution ...@@ -11,9 +11,23 @@ distribution
If not specified, the distribution defaults to the part name. If not specified, the distribution defaults to the part name.
Multiple requirements can be given, separated by newlines. Each
requirement has to be on a separate line.
find-links find-links
A list of URLs, files, or directories to search for distributions. A list of URLs, files, or directories to search for distributions.
index
The URL of an index server, or almost any other valid URL. :)
If not specified, the Python Package Index,
http://cheeseshop.python.org/pypi, is used. You can specify an
alternate index with this option. If you use the links option and
if the links point to the needed distributions, then the index can
be anything and will be largely ignored. In the examples, here,
we'll just point to an empty directory on our link server. This
will make our examples run a little bit faster.
python python
The name of a section to get the Python executable from. The name of a section to get the Python executable from.
If not specified, then the buildout python option is used. The If not specified, then the buildout python option is used. The
...@@ -26,13 +40,20 @@ unzip ...@@ -26,13 +40,20 @@ unzip
only effective when an egg is installed. If a zipped egg already only effective when an egg is installed. If a zipped egg already
exists in the eggs directory, it will not be unzipped. exists in the eggs directory, it will not be unzipped.
To illustrate this, we've created a directory with some sample eggs:
>>> ls(sample_eggs) We have a link server that has a number of eggs:
- demo-0.1-py2.3.egg
- demo-0.2-py2.3.egg >>> print get(link_server),
- demo-0.3-py2.3.egg <html><body>
- demoneeded-1.0-py2.3.egg <a href="demo-0.1-py2.3.egg">demo-0.1-py2.3.egg</a><br>
<a href="demo-0.2-py2.3.egg">demo-0.2-py2.3.egg</a><br>
<a href="demo-0.3-py2.3.egg">demo-0.3-py2.3.egg</a><br>
<a href="demoneeded-1.0-py2.3.egg">demoneeded-1.0-py2.3.egg</a><br>
<a href="demoneeded-1.1-py2.3.egg">demoneeded-1.1-py2.3.egg</a><br>
<a href="index/">index/</a><br>
<a href="other-1.0-py2.3.egg">other-1.0-py2.3.egg</a><br>
</body></html>
We have a sample buildout. Let's update it's configuration file to We have a sample buildout. Let's update it's configuration file to
install the demo package. install the demo package.
...@@ -44,9 +65,10 @@ install the demo package. ...@@ -44,9 +65,10 @@ install the demo package.
... ...
... [demo] ... [demo]
... recipe = zc.recipe.egg ... recipe = zc.recipe.egg
... distribution = demo <0.3 ... distribution = demo<0.3
... find-links = %s ... find-links = %(server)s
... """ % sample_eggs) ... index = %(server)s/index
... """ % dict(server=link_server))
In this example, we limited ourself to revisions before 0.3. We also In this example, we limited ourself to revisions before 0.3. We also
specified where to find distributions using the find-links option. specified where to find distributions using the find-links option.
...@@ -55,14 +77,14 @@ Let's run the buildout: ...@@ -55,14 +77,14 @@ Let's run the buildout:
>>> import os >>> import os
>>> os.chdir(sample_buildout) >>> os.chdir(sample_buildout)
>>> runscript = os.path.join(sample_buildout, 'bin', 'buildout') >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout')
>>> print system(runscript), >>> print system(buildout),
Now, if we look at the buildout eggs directory: Now, if we look at the buildout eggs directory:
>>> ls(sample_buildout, 'eggs') >>> ls(sample_buildout, 'eggs')
- demo-0.2-py2.3.egg - demo-0.2-py2.3.egg
- demoneeded-1.0-py2.3.egg - demoneeded-1.1-py2.3.egg
We see that we got an egg for demo that met the requirement, as well We see that we got an egg for demo that met the requirement, as well
as the egg for demoneeded, wich demo requires. (We also see an egg as the egg for demoneeded, wich demo requires. (We also see an egg
...@@ -114,21 +136,22 @@ specification. For example, We remove the restriction on demo: ...@@ -114,21 +136,22 @@ specification. For example, We remove the restriction on demo:
... ...
... [demo] ... [demo]
... recipe = zc.recipe.egg ... recipe = zc.recipe.egg
... find-links = %s ... find-links = %(server)s
... index = %(server)s/index
... unzip = true ... unzip = true
... """ % sample_eggs) ... """ % dict(server=link_server))
We also used the unzip uption to request a directory, rather than We also used the unzip uption to request a directory, rather than
a zip file. a zip file.
>>> print system(runscript), >>> print system(buildout),
Then we'll get a new demo egg: Then we'll get a new demo egg:
>>> ls(sample_buildout, 'eggs') >>> ls(sample_buildout, 'eggs')
- demo-0.2-py2.3.egg - demo-0.2-py2.3.egg
d demo-0.3-py2.3.egg d demo-0.3-py2.3.egg
- demoneeded-1.0-py2.3.egg d demoneeded-1.0-py2.3.egg
Note that we removed the distribution option, and the distribution Note that we removed the distribution option, and the distribution
defaulted to the part name. defaulted to the part name.
...@@ -150,12 +173,13 @@ arguments: ...@@ -150,12 +173,13 @@ arguments:
... ...
... [demo] ... [demo]
... recipe = zc.recipe.egg ... recipe = zc.recipe.egg
... find-links = %s ... find-links = %(server)s
... index = %(server)s/index
... scripts = ... scripts =
... """ % sample_eggs) ... """ % dict(server=link_server))
>>> print system(runscript), >>> print system(buildout),
>>> ls(sample_buildout, 'bin') >>> ls(sample_buildout, 'bin')
- buildout - buildout
...@@ -169,11 +193,12 @@ You can also control the name used for scripts: ...@@ -169,11 +193,12 @@ You can also control the name used for scripts:
... ...
... [demo] ... [demo]
... recipe = zc.recipe.egg ... recipe = zc.recipe.egg
... find-links = %s ... find-links = %(server)s
... index = %(server)s/index
... scripts = demo=foo ... scripts = demo=foo
... """ % sample_eggs) ... """ % dict(server=link_server))
>>> print system(runscript), >>> print system(buildout),
>>> ls(sample_buildout, 'bin') >>> ls(sample_buildout, 'bin')
- buildout - buildout
......
eggrecipe/src/zc/recipe/egg/egg.py
View file @ 2530eeb2
...@@ -16,8 +16,7 @@ ...@@ -16,8 +16,7 @@
$Id$ $Id$
""" """
import os, zipfile import os, re, zipfile
import zc.buildout.egglinker
import zc.buildout.easy_install import zc.buildout.easy_install
class Egg: class Egg:
...@@ -29,14 +28,17 @@ class Egg: ...@@ -29,14 +28,17 @@ class Egg:
links = options.get('find-links', links = options.get('find-links',
buildout['buildout'].get('find-links')) buildout['buildout'].get('find-links'))
if links: if links:
buildout_directory = buildout['buildout']['directory'] links = links.split()
links = [os.path.join(buildout_directory, link)
for link in links.split()]
options['find-links'] = '\n'.join(links) options['find-links'] = '\n'.join(links)
else: else:
links = () links = ()
self.links = links self.links = links
index = options.get('index', buildout['buildout'].get('index'))
if index is not None:
options['index'] = index
self.index = index
options['_b'] = buildout['buildout']['bin-directory'] options['_b'] = buildout['buildout']['bin-directory']
options['_e'] = buildout['buildout']['eggs-directory'] options['_e'] = buildout['buildout']['eggs-directory']
options['_d'] = buildout['buildout']['develop-eggs-directory'] options['_d'] = buildout['buildout']['develop-eggs-directory']
...@@ -48,13 +50,20 @@ class Egg: ...@@ -48,13 +50,20 @@ class Egg:
def install(self): def install(self):
options = self.options options = self.options
distribution = options.get('distribution', self.name) distributions = [
r.strip()
for r in options.get('distribution', self.name).split('\n')
if r.strip()]
zc.buildout.easy_install.install( ws = zc.buildout.easy_install.install(
distribution, options['_e'], self.links, options['executable'], distributions, options['_e'],
always_unzip=options.get('unzip') == 'true') links = self.links,
index = self.index,
executable = options['executable'],
always_unzip=options.get('unzip') == 'true',
path=[options['_d']]
)
eggss = [options['_d'], options['_e']]
scripts = options.get('scripts') scripts = options.get('scripts')
if scripts or scripts is None: if scripts or scripts is None:
if scripts is not None: if scripts is not None:
...@@ -63,7 +72,7 @@ class Egg: ...@@ -63,7 +72,7 @@ class Egg:
('=' in s) and s.split('=', 1) or (s, s) ('=' in s) and s.split('=', 1) or (s, s)
for s in scripts for s in scripts
]) ])
return zc.buildout.egglinker.scripts( return zc.buildout.easy_install.scripts(
[distribution], options['_b'], eggss, distributions, ws, options['executable'],
scripts=scripts, executable=options['executable']) options['_b'], scripts=scripts)
eggrecipe/src/zc/recipe/egg/selecting-python.txt
View file @ 2530eeb2
...@@ -9,17 +9,24 @@ We can specify the python to use by specifying the name of a section ...@@ -9,17 +9,24 @@ We can specify the python to use by specifying the name of a section
to read the Python executable from. The default is the section to read the Python executable from. The default is the section
defined by the python buildout option. defined by the python buildout option.
We have a directory with some sample eggs: We have a link server:
>>> ls(sample_eggs) >>> print get(link_server),
- demo-0.1-py2.3.egg <html><body>
- demo-0.1-py2.4.egg <a href="demo-0.1-py2.3.egg">demo-0.1-py2.3.egg</a><br>
- demo-0.2-py2.3.egg <a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br>
- demo-0.2-py2.4.egg <a href="demo-0.2-py2.3.egg">demo-0.2-py2.3.egg</a><br>
- demo-0.3-py2.3.egg <a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br>
- demo-0.3-py2.4.egg <a href="demo-0.3-py2.3.egg">demo-0.3-py2.3.egg</a><br>
- demoneeded-1.0-py2.3.egg <a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br>
- demoneeded-1.0-py2.4.egg <a href="demoneeded-1.0-py2.3.egg">demoneeded-1.0-py2.3.egg</a><br>
<a href="demoneeded-1.0-py2.4.egg">demoneeded-1.0-py2.4.egg</a><br>
<a href="demoneeded-1.1-py2.3.egg">demoneeded-1.1-py2.3.egg</a><br>
<a href="demoneeded-1.1-py2.4.egg">demoneeded-1.1-py2.4.egg</a><br>
<a href="index/">index/</a><br>
<a href="other-1.0-py2.3.egg">other-1.0-py2.3.egg</a><br>
<a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br>
</body></html>
We have a sample buildout. Let's update it's configuration file to We have a sample buildout. Let's update it's configuration file to
install the demo package using Python 2.3. install the demo package using Python 2.3.
...@@ -33,9 +40,10 @@ install the demo package using Python 2.3. ...@@ -33,9 +40,10 @@ install the demo package using Python 2.3.
... [demo] ... [demo]
... recipe = zc.recipe.egg ... recipe = zc.recipe.egg
... distribution = demo <0.3 ... distribution = demo <0.3
... find-links = %s ... find-links = %(server)s
... index = %(server)s/index
... python = python2.3 ... python = python2.3
... """ % sample_eggs) ... """ % dict(server=link_server))
In our default.cfg file in the .buildout subdirectiry of our In our default.cfg file in the .buildout subdirectiry of our
directory, we have something like:: directory, we have something like::
...@@ -59,7 +67,7 @@ we'll get the Python 2.3 eggs for demo and demoneeded: ...@@ -59,7 +67,7 @@ we'll get the Python 2.3 eggs for demo and demoneeded:
>>> ls(sample_buildout, 'eggs') >>> ls(sample_buildout, 'eggs')
- demo-0.2-py2.3.egg - demo-0.2-py2.3.egg
- demoneeded-1.0-py2.3.egg - demoneeded-1.1-py2.3.egg
And the generated scripts invoke Python 2.3: And the generated scripts invoke Python 2.3:
...@@ -71,7 +79,7 @@ And the generated scripts invoke Python 2.3: ...@@ -71,7 +79,7 @@ And the generated scripts invoke Python 2.3:
import sys import sys
sys.path[0:0] = [ sys.path[0:0] = [
'/private/tmp/tmpOEtRO8sample-buildout/eggs/demo-0.2-py2.3.egg', '/private/tmp/tmpOEtRO8sample-buildout/eggs/demo-0.2-py2.3.egg',
'/private/tmp/tmpOEtRO8sample-buildout/eggs/demoneeded-1.0-py2.3.egg' '/private/tmp/tmpOEtRO8sample-buildout/eggs/demoneeded-1.1-py2.3.egg'
] ]
<BLANKLINE> <BLANKLINE>
import eggrecipedemo import eggrecipedemo
...@@ -87,7 +95,7 @@ And the generated scripts invoke Python 2.3: ...@@ -87,7 +95,7 @@ And the generated scripts invoke Python 2.3:
import sys import sys
sys.path[0:0] = [ sys.path[0:0] = [
'/tmp/tmpOBTxDMsample-buildout/eggs/demo-0.2-py2.3.egg', '/tmp/tmpOBTxDMsample-buildout/eggs/demo-0.2-py2.3.egg',
'/tmp/tmpOBTxDMsample-buildout/eggs/demoneeded-1.0-py2.3.egg' '/tmp/tmpOBTxDMsample-buildout/eggs/demoneeded-1.1-py2.3.egg'
] ]
If we change the Python version to 2.4, we'll use Python 2.4 eggs: If we change the Python version to 2.4, we'll use Python 2.4 eggs:
...@@ -101,17 +109,18 @@ If we change the Python version to 2.4, we'll use Python 2.4 eggs: ...@@ -101,17 +109,18 @@ If we change the Python version to 2.4, we'll use Python 2.4 eggs:
... [demo] ... [demo]
... recipe = zc.recipe.egg ... recipe = zc.recipe.egg
... distribution = demo <0.3 ... distribution = demo <0.3
... find-links = %s ... find-links = %(server)s
... index = %(server)s/index
... python = python2.4 ... python = python2.4
... """ % sample_eggs) ... """ % dict(server=link_server))
>>> print system(buildout), >>> print system(buildout),
>>> ls(sample_buildout, 'eggs') >>> ls(sample_buildout, 'eggs')
- demo-0.2-py2.3.egg - demo-0.2-py2.3.egg
- demo-0.2-py2.4.egg - demo-0.2-py2.4.egg
- demoneeded-1.0-py2.3.egg - demoneeded-1.1-py2.3.egg
- demoneeded-1.0-py2.4.egg - demoneeded-1.1-py2.4.egg
>>> f = open(os.path.join(sample_buildout, 'bin', 'demo')) >>> f = open(os.path.join(sample_buildout, 'bin', 'demo'))
>>> f.readline().strip() == '#!' + python2_4_executable >>> f.readline().strip() == '#!' + python2_4_executable
...@@ -121,7 +130,7 @@ If we change the Python version to 2.4, we'll use Python 2.4 eggs: ...@@ -121,7 +130,7 @@ If we change the Python version to 2.4, we'll use Python 2.4 eggs:
import sys import sys
sys.path[0:0] = [ sys.path[0:0] = [
'/private/tmp/tmpOEtRO8sample-buildout/eggs/demo-0.2-py2.4.egg', '/private/tmp/tmpOEtRO8sample-buildout/eggs/demo-0.2-py2.4.egg',
'/private/tmp/tmpOEtRO8sample-buildout/eggs/demoneeded-1.0-py2.4.egg' '/private/tmp/tmpOEtRO8sample-buildout/eggs/demoneeded-1.1-py2.4.egg'
] ]
<BLANKLINE> <BLANKLINE>
import eggrecipedemo import eggrecipedemo
...@@ -137,7 +146,7 @@ If we change the Python version to 2.4, we'll use Python 2.4 eggs: ...@@ -137,7 +146,7 @@ If we change the Python version to 2.4, we'll use Python 2.4 eggs:
import sys import sys
sys.path[0:0] = [ sys.path[0:0] = [
'/tmp/tmpOBTxDMsample-buildout/eggs/demo-0.2-py2.4.egg', '/tmp/tmpOBTxDMsample-buildout/eggs/demo-0.2-py2.4.egg',
'/tmp/tmpOBTxDMsample-buildout/eggs/demoneeded-1.0-py2.4.egg' '/tmp/tmpOBTxDMsample-buildout/eggs/demoneeded-1.1-py2.4.egg'
] ]
eggrecipe/src/zc/recipe/egg/tests.py
View file @ 2530eeb2
...@@ -29,10 +29,16 @@ def setUp(test): ...@@ -29,10 +29,16 @@ def setUp(test):
'develop-eggs', 'zc.recipe.egg.egg-link'), 'develop-eggs', 'zc.recipe.egg.egg-link'),
'w').write(dirname(__file__, 4)) 'w').write(dirname(__file__, 4))
zc.buildout.testing.create_sample_eggs(test) zc.buildout.testing.create_sample_eggs(test)
test.globs['link_server'] = (
'http://localhost:%s/'
% zc.buildout.testing.start_server(zc.buildout.testing.make_tree(test))
)
def tearDown(test): def tearDown(test):
shutil.rmtree(test.globs['_sample_eggs_container']) shutil.rmtree(test.globs['_sample_eggs_container'])
zc.buildout.testing.buildoutTearDown(test) zc.buildout.testing.buildoutTearDown(test)
zc.buildout.testing.stop_server(test.globs['link_server'])
def setUpPython(test): def setUpPython(test):
zc.buildout.testing.buildoutSetUp(test, clear_home=False) zc.buildout.testing.buildoutSetUp(test, clear_home=False)
...@@ -42,6 +48,10 @@ def setUpPython(test): ...@@ -42,6 +48,10 @@ def setUpPython(test):
'w').write(dirname(__file__, 4)) 'w').write(dirname(__file__, 4))
zc.buildout.testing.multi_python(test) zc.buildout.testing.multi_python(test)
test.globs['link_server'] = (
'http://localhost:%s/'
% zc.buildout.testing.start_server(zc.buildout.testing.make_tree(test))
)
def test_suite(): def test_suite():
return unittest.TestSuite(( return unittest.TestSuite((
...@@ -54,7 +64,8 @@ def test_suite(): ...@@ -54,7 +64,8 @@ def test_suite():
'(\\w+-)[^ \t\n%(sep)s/]+.egg' '(\\w+-)[^ \t\n%(sep)s/]+.egg'
% dict(sep=os.path.sep) % dict(sep=os.path.sep)
), ),
'\\2-VVV-egg') '\\2-VVV-egg'),
(re.compile('-py\d[.]\d.egg'), '-py2.4.egg'),
]) ])
), ),
doctest.DocFileSuite( doctest.DocFileSuite(
......
src/zc/buildout/buildout.py
View file @ 2530eeb2
...@@ -28,7 +28,6 @@ import ConfigParser ...@@ -28,7 +28,6 @@ import ConfigParser
import zc.buildout.easy_install import zc.buildout.easy_install
import pkg_resources import pkg_resources
import zc.buildout.easy_install import zc.buildout.easy_install
import zc.buildout.egglinker
class MissingOption(KeyError): class MissingOption(KeyError):
"""A required option was missing """A required option was missing
...@@ -262,32 +261,50 @@ class Buildout(dict): ...@@ -262,32 +261,50 @@ class Buildout(dict):
os.chdir(os.path.dirname(setup)) os.chdir(os.path.dirname(setup))
os.spawnle( os.spawnle(
os.P_WAIT, sys.executable, sys.executable, os.P_WAIT, sys.executable, sys.executable,
setup, '-q', 'develop', '-m', '-x', setup, '-q', 'develop', '-m', '-x', '-N',
'-f', ' '.join(self._links), '-f', ' '.join(self._links),
'-d', self['buildout']['develop-eggs-directory'], '-d', self['buildout']['develop-eggs-directory'],
{'PYTHONPATH': {'PYTHONPATH':
os.path.dirname(pkg_resources.__file__)}, os.path.dirname(pkg_resources.__file__)},
) )
finally: finally:
os.chdir(os.path.dirname(here)) os.chdir(here)
def _load_recipes(self, parts): def _load_recipes(self, parts):
recipes = {} recipes = {}
if not parts:
return recipes
recipes_requirements = [] recipes_requirements = []
pkg_resources.working_set.add_entry( pkg_resources.working_set.add_entry(
self['buildout']['develop-eggs-directory']) self['buildout']['develop-eggs-directory'])
pkg_resources.working_set.add_entry(self['buildout']['eggs-directory']) pkg_resources.working_set.add_entry(self['buildout']['eggs-directory'])
# Install the recipe distros # Gather requirements
for part in parts: for part in parts:
options = self.get(part) options = self.get(part)
if options is None: if options is None:
options = self[part] = {} options = self[part] = {}
recipe, entry = self._recipe(part, options) recipe, entry = self._recipe(part, options)
zc.buildout.easy_install.install(
recipe, self['buildout']['eggs-directory'], self._links)
recipes_requirements.append(recipe) recipes_requirements.append(recipe)
# Install the recipe distros
offline = self['buildout'].get('offline', 'false')
if offline not in ('true', 'false'):
self._error('Invalif value for offline option: %s', offline)
if offline == 'true':
ws = zc.buildout.easy_install.working_set(
recipes_requirements, sys.executable,
[self['buildout']['eggs-directory'],
self['buildout']['develop-eggs-directory'],
],
)
else:
ws = zc.buildout.easy_install.install(
recipes_requirements, self['buildout']['eggs-directory'],
links=self._links, index=self['buildout'].get('index'),
path=[self['buildout']['develop-eggs-directory']])
# Add the distros to the working set # Add the distros to the working set
pkg_resources.require(recipes_requirements) pkg_resources.require(recipes_requirements)
...@@ -503,6 +520,10 @@ def main(args=None): ...@@ -503,6 +520,10 @@ def main(args=None):
else: else:
verbosity -= 10 verbosity -= 10
op = op[1:] op = op[1:]
if op == 'd':
op = op[1:]
import pdb; pdb.set_trace()
if op[:1] == 'c': if op[:1] == 'c':
op = op[1:] op = op[1:]
if op: if op:
......
src/zc/buildout/easy_install.py
View file @ 2530eeb2
...@@ -20,18 +20,299 @@ installed. ...@@ -20,18 +20,299 @@ installed.
$Id$ $Id$
""" """
import os, sys import logging, os, re, sys
import pkg_resources
import zc.buildout
def install(spec, dest, links, executable=sys.executable, always_unzip=False): logger = logging.getLogger('zc.buildout.easy_install')
# Include buildout and setuptools eggs in paths
buildout_and_setuptools_path = [
(('.egg' in m.__file__)
and m.__file__[:m.__file__.rfind('.egg')+4]
or os.path.dirname(m.__file__)
)
for m in (pkg_resources,)
]
buildout_and_setuptools_path += [
(('.egg' in m.__file__)
and m.__file__[:m.__file__.rfind('.egg')+4]
or os.path.dirname(os.path.dirname(os.path.dirname(m.__file__)))
)
for m in (zc.buildout,)
]
_versions = {sys.executable: '%d.%d' % sys.version_info[:2]}
def _get_version(executable):
try:
return _versions[executable]
except KeyError:
i, o = os.popen4(executable + ' -V')
i.close()
version = o.read().strip()
o.close()
pystring, version = version.split()
assert pystring == 'Python'
version = re.match('(\d[.]\d)[.]\d$', version).group(1)
_versions[executable] = version
return version
def _satisfied(req, env):
dists = env[req.project_name]
best = None
for dist in dists:
if (dist.precedence == pkg_resources.DEVELOP_DIST) and (dist in req):
if best is not None and best.location != dist.location:
raise ValueError('Multiple devel eggs for', req)
best = dist
if best is not None:
return best
specs = [(pkg_resources.parse_version(v), op) for (op, v) in req.specs]
specs.sort()
maxv = None
greater = False
lastv = None
for v, op in specs:
if op == '==' and not greater:
maxv = v
elif op in ('>', '>=', '!='):
maxv = None
greater == True
elif op == '<':
maxv = None
greater == False
elif op == '<=':
maxv = v
greater == False
if v == lastv:
# Repeated versions values are undefined, so
# all bets are off
maxv = None
greater = True
else:
lastv = v
if maxv is not None:
for dist in dists:
if dist.parsed_version == maxv:
return dist
return None
def _call_easy_install(spec, dest, links=(),
index = None,
executable=sys.executable,
always_unzip=False,
):
prefix = sys.exec_prefix + os.path.sep prefix = sys.exec_prefix + os.path.sep
path = os.pathsep.join([p for p in sys.path if not p.startswith(prefix)]) path = os.pathsep.join([p for p in sys.path if not p.startswith(prefix)])
args = ( args = (
'-c', 'from setuptools.command.easy_install import main; main()', '-c', 'from setuptools.command.easy_install import main; main()',
'-mqxd', dest) '-mUNxd', dest)
if links: if links:
args += ('-f', ' '.join(links)) args += ('-f', ' '.join(links))
if index:
args += ('-i', index)
if always_unzip: if always_unzip:
args += ('-Z', ) args += ('-Z', )
args += (spec, dict(PYTHONPATH=path)) level = logger.getEffectiveLevel()
if level > logging.DEBUG:
args += ('-q', )
elif level < logging.DEBUG:
args += ('-v', )
args += (spec, )
if level <= logging.DEBUG:
logger.debug('Running easy_install:\n%s "%s"\npath=%s\n',
executable, '" "'.join(args), path)
os.spawnle(os.P_WAIT, executable, executable, *args) args += (dict(PYTHONPATH=path), )
sys.stdout.flush() # We want any pending output first
exit_code = os.spawnle(os.P_WAIT, executable, executable, *args)
# We may overwrite distributions, so clear importer
# cache.
sys.path_importer_cache.clear()
assert exit_code == 0
def _get_dist(requirement, env, ws,
dest, links, index, executable, always_unzip):
# Maybe an existing dist is already the best dist that satisfies the
# requirement
dist = _satisfied(requirement, env)
# XXX Special case setuptools because:
# 1. Almost everything depends on it and
# 2. It is expensive to checl for.
# Need to think of a cleaner way to handle this.
# If we already have a satisfactory version, use it.
if dist is None and requirement.project_name == 'setuptools':
dist = env.best_match(requirement, ws)
if dist is None:
if dest is not None:
# May need a new one. Call easy_install
_call_easy_install(str(requirement), dest, links, index,
executable, always_unzip)
# Because we may have added new eggs, we need to rescan
# the destination directory. A possible optimization
# is to get easy_install to recod the files installed
# and either firgure out the distribution added, or
# only rescan if any files have been added.
env.scan([dest])
dist = env.best_match(requirement, ws)
# XXX Need test for this
if dist.has_metadata('dependency_links.txt'):
for link in dist.get_metadata_lines('dependency_links.txt'):
link = link.strip()
if link not in links:
links.append(link)
return dist
def install(specs, dest,
links=(), index=None,
executable=sys.executable, always_unzip=False,
path=None):
logger.debug('Installing %r', specs)
path = path and path[:] or []
if dest is not None:
path.insert(0, dest)
path += buildout_and_setuptools_path
links = list(links) # make copy, because we may need to mutate
# For each spec, see if it is already installed. We create a working
# set to keep track of what we've collected and to make sue than the
# distributions assembled are consistent.
env = pkg_resources.Environment(path, python=_get_version(executable))
requirements = [pkg_resources.Requirement.parse(spec) for spec in specs]
ws = pkg_resources.WorkingSet([])
for requirement in requirements:
ws.add(_get_dist(requirement, env, ws,
dest, links, index, executable, always_unzip)
)
# OK, we have the requested distributions and they're in the working
# set, but they may have unmet requirements. We'll simply keep
# trying to resolve requirements, adding missing requirements as they
# are reported.
#
# Note that we don't pass in the environment, because we
# want to look for new eggs unless what we have is the best that matches
# the requirement.
while 1:
try:
ws.resolve(requirements)
except pkg_resources.DistributionNotFound, err:
[requirement] = err
if dest:
logger.debug('Getting required %s', requirement)
ws.add(_get_dist(requirement, env, ws,
dest, links, index, executable, always_unzip)
)
else:
break
return ws
def working_set(specs, executable, path):
return install(specs, None, executable=executable, path=path)
def scripts(reqs, working_set, executable, dest, scripts=None):
reqs = [pkg_resources.Requirement.parse(r) for r in reqs]
projects = [r.project_name for r in reqs]
path = "',\n '".join([dist.location for dist in working_set])
generated = []
for dist in working_set:
if dist.project_name in projects:
for name in pkg_resources.get_entry_map(dist, 'console_scripts'):
if scripts is not None:
sname = scripts.get(name)
if sname is None:
continue
else:
sname = name
sname = os.path.join(dest, sname)
generated.append(sname)
_script(dist, 'console_scripts', name, path, sname, executable)
name = 'py_'+dist.project_name
if scripts is not None:
sname = scripts.get(name)
else:
sname = name
if sname is not None:
sname = os.path.join(dest, sname)
generated.append(sname)
_pyscript(path, sname, executable)
return generated
def _script(dist, group, name, path, dest, executable):
entry_point = dist.get_entry_info(group, name)
open(dest, 'w').write(script_template % dict(
python = executable,
path = path,
project = dist.project_name,
name = name,
module_name = entry_point.module_name,
attrs = '.'.join(entry_point.attrs),
))
try:
os.chmod(dest, 0755)
except (AttributeError, os.error):
pass
script_template = '''\
#!%(python)s
import sys
sys.path[0:0] = [
'%(path)s'
]
import %(module_name)s
if __name__ == '__main__':
%(module_name)s.%(attrs)s()
'''
def _pyscript(path, dest, executable):
open(dest, 'w').write(py_script_template % dict(
python = executable,
path = path,
))
try:
os.chmod(dest,0755)
except (AttributeError, os.error):
pass
py_script_template = '''\
#!%(python)s -i
import sys
sys.path[0:0] = [
'%(path)s'
]
'''
src/zc/buildout/easy_install.txt
View file @ 2530eeb2
...@@ -2,64 +2,263 @@ Minimal Python interface to easy_install ...@@ -2,64 +2,263 @@ Minimal Python interface to easy_install
======================================== ========================================
The easy_install module provides a minimal interface to the setuptools The easy_install module provides a minimal interface to the setuptools
easy_install command. This API is likely to grow, although I hope easy_install command that provides some additional semantics:
that it will ultimately be replaced by a setuptools-provided API.
- By default, we look for new packages *and* the packages that
they depend on. This is somewhat like (and uses) the --upgrade
option of easy_install, except that we also upgrade required
packages.
- If the highest-revision package satisfying a specification is
already present, then we don't try to get another one. This saves a
lot of search time in the common case that packages are pegged to
specific versions.
- If there is a develop egg that satisfies a requirement, we don't
look for additional distributions. We always give preference to
develop eggs.
The easy_install module provides a single method, install. The The easy_install module provides a single method, install. The
install function takes 3 arguments: install function takes 2 positional arguments:
- A setuptools requirement specification for a distribution to be - An iterable of setuptools requirement strings for the distributions
installed, to be installed, and
- A destination egg directory to install to and to satisfy - A destination directory to install to and to satisfy
requirements from, and requirements from.
- a sequence of lications to look for distributions. It supports a number of optional keyword arguments:
For example, given the sample eggs: links
a sequence of URLs, file names, or directories to look for
links to distributions,
>>> ls(sample_eggs) index
- demo-0.1-py2.3.egg The URL of an index server, or almost any other valid URL. :)
- demo-0.1-py2.4.egg
- demo-0.2-py2.3.egg If not specified, the Python Package Index,
- demo-0.2-py2.4.egg http://cheeseshop.python.org/pypi, is used. You can specify an
- demo-0.3-py2.3.egg alternate index with this option. If you use the links option and
- demo-0.3-py2.4.egg if the links point to the needed distributions, then the index can
- demoneeded-1.0-py2.3.egg be anything and will be largely ignored. In the examples, here,
- demoneeded-1.0-py2.4.egg we'll just point to an empty directory on our link server. This
will make our examples run a little bit faster.
let's make directory and install the demo egg to it: executable
A path to a Python executable. Distributions will ne installed
using this executable and will be for the matching Python version.
path
A list of additional directories to search for locally-installed
distributions.
always_unzip
A flag indicating that newly-downloaded distributions should be
directories even if they could be installed as zip files.
The install method returns a working set containing the distributions
needed to meet the given requirements.
We have a link server that has a number of eggs:
>>> print get(link_server),
<html><body>
<a href="demo-0.1-py2.3.egg">demo-0.1-py2.3.egg</a><br>
<a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br>
<a href="demo-0.2-py2.3.egg">demo-0.2-py2.3.egg</a><br>
<a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br>
<a href="demo-0.3-py2.3.egg">demo-0.3-py2.3.egg</a><br>
<a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br>
<a href="demoneeded-1.0-py2.3.egg">demoneeded-1.0-py2.3.egg</a><br>
<a href="demoneeded-1.0-py2.4.egg">demoneeded-1.0-py2.4.egg</a><br>
<a href="demoneeded-1.1-py2.3.egg">demoneeded-1.1-py2.3.egg</a><br>
<a href="demoneeded-1.1-py2.4.egg">demoneeded-1.1-py2.4.egg</a><br>
<a href="index/">index/</a><br>
<a href="other-1.0-py2.3.egg">other-1.0-py2.3.egg</a><br>
<a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br>
</body></html>
let's make directory and install the demo egg to it, using the demo:
>>> import tempfile >>> import tempfile
>>> dest = tempfile.mkdtemp() >>> dest = tempfile.mkdtemp('sample-install')
>>> import zc.buildout.easy_install >>> import zc.buildout.easy_install
>>> zc.buildout.easy_install.install('demo', dest, [sample_eggs]) >>> ws = zc.buildout.easy_install.install(
... ['demo==0.2'], dest,
... links=[link_server], index=link_server+'index/')
We requested version 0.2 of the demo distribution to be installed into
the destination server. We specified that we should search for links
on the link server and that we should use the (empty) link server
index directory as a package index.
The working set contains the distributions we retrieved.
>>> for dist in ws:
... print dist
demo 0.2
demoneeded 1.1
And the actual eggs were added to the eggs directory.
>>> ls(dest)
- demo-0.2-py2.3.egg
- demoneeded-1.1-py2.3.egg
If we ask for the demo distribution without a version restriction,
we'll get the newer version:
>>> ws = zc.buildout.easy_install.install(
... ['demo'], dest, links=[link_server], index=link_server+'index/')
>>> ls(dest) >>> ls(dest)
- demo-0.2-py2.3.egg
- demo-0.3-py2.3.egg
- demoneeded-1.1-py2.3.egg
We can supply additional distributions. We can also supply
specifications for distributions that would normally be found via
dependencies. We might do this to specify a sprcific version.
>>> ws = zc.buildout.easy_install.install(
... ['demo', 'other', 'demoneeded==1.0'], dest,
... links=[link_server], index=link_server+'index/')
>>> for dist in ws:
... print dist
demo 0.3
other 1.0
demoneeded 1.0
>>> ls(dest)
- demo-0.2-py2.3.egg
- demo-0.3-py2.3.egg - demo-0.3-py2.3.egg
- demoneeded-1.0-py2.3.egg - demoneeded-1.0-py2.3.egg
- demoneeded-1.1-py2.3.egg
- other-1.0-py2.3.egg
We can specify an alternate Python executable, and we can specify We can specify an alternate Python executable, and we can specify
that, when we retrieve (or create) an egg, it should be unzipped. that, when we retrieve (or create) an egg, it should be unzipped.
>>> import shutil >>> import shutil
>>> shutil.rmtree(dest) >>> shutil.rmtree(dest)
>>> dest = tempfile.mkdtemp() >>> dest = tempfile.mkdtemp('sample-install')
>>> zc.buildout.easy_install.install( >>> ws = zc.buildout.easy_install.install(
... 'demo', dest, [sample_eggs], ... ['demo'], dest, links=[link_server], index=link_server+'index/',
... always_unzip=True, executable= python2_3_executable) ... always_unzip=True, executable= python2_3_executable)
>>> ls(dest) >>> ls(dest)
d demo-0.3-py2.3.egg d demo-0.3-py2.3.egg
d demoneeded-1.0-py2.3.egg d demoneeded-1.1-py2.3.egg
>>> shutil.rmtree(dest) >>> shutil.rmtree(dest)
>>> dest = tempfile.mkdtemp() >>> dest = tempfile.mkdtemp('sample-install')
>>> zc.buildout.easy_install.install( >>> ws = zc.buildout.easy_install.install(
... 'demo', dest, [sample_eggs], ... ['demo'], dest, links=[link_server], index=link_server+'index/',
... always_unzip=True, executable= python2_4_executable) ... always_unzip=True, executable=python2_4_executable)
>>> ls(dest) >>> ls(dest)
d demo-0.3-py2.4.egg d demo-0.3-py2.4.egg
d demoneeded-1.0-py2.4.egg d demoneeded-1.1-py2.4.egg
Script generation
-----------------
The easy_install module provides support for creating scripts from
eggs. It provides a function similar to setuptools except that it
provides facilities for baking a script's path into the script. This
has two advantages:
- The eggs to be used by a script are not chosen at run time, making
startup faster and, more importantly, deterministic.
- The script doesn't have to import pkg_resources because the logic
that pkg_resources would execute at run time is executed at
script-creation time.
The scripts method can be used to generate scripts. Let's create a
destination directory for it to place them in:
>>> import tempfile
>>> bin = tempfile.mkdtemp()
Now, we'll use the scripts method to generate scripts in this directory
from the demo egg:
>>> scripts = zc.buildout.easy_install.scripts(
... ['demo==0.1'], ws, python2_4_executable, bin)
the four arguments we passed were:
1. A sequence of distribution requirements. These are of the same
form as setuptools requirements. Here we passed a single
requirement, for the version 0.1 demo distribution.
2. A working set,
3. The Python executable to use, and
3. The destination directory.
The bin directory now contains 2 generated scripts:
>>> ls(bin)
- demo
- py_demo
The return value is a list of the scripts generated:
>>> import os
>>> scripts == [os.path.join(bin, 'demo'), os.path.join(bin, 'py_demo')]
True
The demo script run the entry point defined in the demo egg:
>>> cat(bin, 'demo')
#!/usr/local/bin/python2.3
<BLANKLINE>
import sys
sys.path[0:0] = [
'/tmp/xyzsample-install/demo-0.3-py2.3.egg',
'/tmp/xyzsample-install/demoneeded-1.1-py2.3.egg'
]
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
eggrecipedemo.main()
Some things to note:
- The demo and demoneeded eggs are added to the beginning of sys.path.
- The module for the script entry point is imported and the entry
point, in this case, 'main', is run.
The py_demo script simply run the Python interactive interpreter with
the path set:
>>> cat(bin, 'py_demo')
#!/usr/local/bin/python2.3 -i
<BLANKLINE>
import sys
sys.path[0:0] = [
'/tmp/xyzsample-install/demo-0.3-py2.3.egg',
'/tmp/xyzsample-install/demoneeded-1.1-py2.3.egg'
]
An additional argumnet can be passed to define which scripts to install
and to provie script names. The argument is a dictionary mapping
original script names to new script names.
>>> import shutil
>>> shutil.rmtree(bin)
>>> bin = tempfile.mkdtemp()
>>> scripts = zc.buildout.easy_install.scripts(
... ['demo==0.1'], ws, python2_4_executable, bin, dict(demo='run'))
>>> scripts == [os.path.join(bin, 'run')]
True
>>> ls(bin)
- run
>>> print system(os.path.join(bin, 'run')),
3 1
src/zc/buildout/egglinker.py deleted 100644 → 0
View file @ e74cdf1a
##############################################################################
#
# Copyright (c) 2005 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Egg linker -- Link eggs together to build applications
Egg linker is a script that generates startup scripts for eggs that
include an egg's working script in the generated script.
The egg linker module also exports helper functions of varous kinds to
assist in custom script generation.
$Id$
"""
# XXX need to deal with extras
import os
import re
import sys
import pkg_resources
_versions = {sys.executable: '%d.%d' % sys.version_info[:2]}
def _get_version(executable):
try:
return _versions[executable]
except KeyError:
i, o = os.popen4(executable + ' -V')
i.close()
version = o.read().strip()
o.close()
pystring, version = version.split()
assert pystring == 'Python'
version = re.match('(\d[.]\d)[.]\d$', version).group(1)
_versions[executable] = version
return version
def distributions(reqs, eggss, executable=sys.executable):
env = pkg_resources.Environment(eggss, python=_get_version(executable))
ws = pkg_resources.WorkingSet()
reqs = [pkg_resources.Requirement.parse(r) for r in reqs]
return ws.resolve(reqs, env=env)
def path(reqs, eggss, executable=sys.executable):
dists = distributions(reqs, eggss, executable)
return [dist.location for dist in dists]
def location(spec, eggss, executable=sys.executable):
env = pkg_resources.Environment(eggss, python=_get_version(executable))
req = pkg_resources.Requirement.parse(spec)
dist = env.best_match(req, pkg_resources.WorkingSet())
return dist.location
def scripts(reqs, dest, eggss, scripts=None, executable=sys.executable):
dists = distributions(reqs, eggss, executable)
reqs = [pkg_resources.Requirement.parse(r) for r in reqs]
projects = [r.project_name for r in reqs]
path = "',\n '".join([dist.location for dist in dists])
generated = []
for dist in dists:
if dist.project_name in projects:
for name in pkg_resources.get_entry_map(dist, 'console_scripts'):
if scripts is not None:
sname = scripts.get(name)
if sname is None:
continue
else:
sname = name
sname = os.path.join(dest, sname)
generated.append(sname)
_script(dist, 'console_scripts', name, path, sname, executable)
name = 'py_'+dist.project_name
if scripts is not None:
sname = scripts.get(name)
else:
sname = name
if sname is not None:
sname = os.path.join(dest, sname)
generated.append(sname)
_pyscript(path, sname, executable)
return generated
def _script(dist, group, name, path, dest, executable):
entry_point = dist.get_entry_info(group, name)
open(dest, 'w').write(script_template % dict(
python = executable,
path = path,
project = dist.project_name,
name = name,
module_name = entry_point.module_name,
attrs = '.'.join(entry_point.attrs),
))
try:
os.chmod(dest, 0755)
except (AttributeError, os.error):
pass
script_template = '''\
#!%(python)s
import sys
sys.path[0:0] = [
'%(path)s'
]
import %(module_name)s
if __name__ == '__main__':
%(module_name)s.%(attrs)s()
'''
def _pyscript(path, dest, executable):
open(dest, 'w').write(py_script_template % dict(
python = executable,
path = path,
))
try:
os.chmod(dest,0755)
except (AttributeError, os.error):
pass
py_script_template = '''\
#!%(python)s -i
import sys
sys.path[0:0] = [
'%(path)s'
]
'''
def main():
import pdb; pdb.set_trace()
src/zc/buildout/egglinker.txt deleted 100644 → 0
View file @ e74cdf1a
Custom script support
=====================
The egg-linker module provides support for creating scripts from
eggs. It provides a function similar to setup tools except that it
provides facilities for baking a script's path into the script. This
has two advantages:
- The eggs to be used by a script are not chosen at run time, making
startup faster and, more importantly, deterministic.
- The script doesn't have to import pkg_resources because the logic
that pkg_resources would execute at run time is executed at
script-creation time.
We have a directory with some sample eggs in it:
>>> ls(sample_eggs)
- demo-0.1-py2.3.egg
- demo-0.1-py2.4.egg
- demo-0.2-py2.3.egg
- demo-0.2-py2.4.egg
- demo-0.3-py2.3.egg
- demo-0.3-py2.4.egg
- demoneeded-1.0-py2.3.egg
- demoneeded-1.0-py2.4.egg
The demo package depends on the demoneeded package.
The egglinker module can be used to generate scripts. Let's create a
desitnation directory for it to place them in:
>>> import tempfile
>>> bin = tempfile.mkdtemp()
Now, we'll use the egg linker to generate scripts in this directory
from the demo egg:
>>> import zc.buildout.egglinker
>>> scripts = zc.buildout.egglinker.scripts(['demo==0.1'], bin,
... [sample_eggs])
the three arguments we passed were:
1. A sequence of distribution requirements. These are of the same
form as setuptools requirements. Here we passed a single
requirement, for the version 0.1 demo distribution.
2. The destination directory.
3, A sequence of egg directories, which are searched for suitable
distributions.
The bin directory now contains 2 generated scripts:
>>> ls(bin)
- demo
- py_demo
The return value is a list of the scripts generated:
>>> import os
>>> scripts == [os.path.join(bin, 'demo'), os.path.join(bin, 'py_demo')]
True
The demo script run the entry point defined in the demo egg:
>>> cat(bin, 'demo')
#!/usr/local/bin/python2.3
<BLANKLINE>
import sys
sys.path[0:0] = [
'/tmp/xyzsample-eggs/demo-0.1-py2.3.egg',
'/tmp/xyzsample-eggs/demoneeded-1.0-py2.3.egg'
]
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
eggrecipedemo.main()
Some things to note:
- The demo and demoneeded eggs are added to the beginning of sys.path.
- The module for the script entry point is imported and the entry
point, in this case, 'main', is run.
The py_demo script simply run the Python interactive interpreter with
the path set:
>>> cat(bin, 'py_demo')
#!/usr/local/bin/python2.3 -i
<BLANKLINE>
import sys
sys.path[0:0] = [
'/tmp/xyzsample-eggs/demo-0.1-py2.3.egg',
'/tmp/xyzsample-eggs/demoneeded-1.0-py2.3.egg'
]
An additional argumnet can be passed to define which scripts to install
and to provie script names. The argument is a dictionary mapping
original script names to new script names.
>>> import shutil
>>> shutil.rmtree(bin)
>>> bin = tempfile.mkdtemp()
>>> scripts = zc.buildout.egglinker.scripts(
... ['demo==0.1'], bin, [sample_eggs],
... dict(demo='run'))
>>> scripts == [os.path.join(bin, 'run')]
True
>>> ls(bin)
- run
>>> print system(os.path.join(bin, 'run')),
1 1
Sometimes we need more control over script generation. Some
lower-level APIs are available to help us generate scripts ourselves.
These apis are a little bit higher level than those provided by
the pkg_resources from the setuptools distribution.
The path method returns a path to a set of eggs satisftying a sequence
of requirements from a sequence of egg directories:
>>> zc.buildout.egglinker.path(['demo==0.1'], [sample_eggs])
... # doctest: +NORMALIZE_WHITESPACE
['/tmp/xyzsample-eggs/demo-0.1-py2.3.egg',
'/tmp/xyzsample-eggs/demoneeded-1.0-py2.3.egg']
The location method returns the distribution location for an egg that
satisfies a requirement:
>>> zc.buildout.egglinker.location('demo==0.1', [sample_eggs])
'/tmp/xyzsample-eggs/demo-0.1-py2.3.egg'
The distributions function can retrieve a list of distributions found
ineg directories that match a sequence of requirements:
>>> [(d.project_name, d.version) for d in
... zc.buildout.egglinker.distributions(['demo==0.1'], [sample_eggs])]
[('demo', '0.1'), ('demoneeded', '1.0')]
Using a custom Python interpreter
---------------------------------
You can pass an executable argument to egglinker methods:
>>> scripts = zc.buildout.egglinker.scripts(
... ['demo==0.1'], bin, [sample_eggs],
... executable=python2_3_executable)
>>> f = open(os.path.join(bin, 'demo'))
>>> f.readline().strip() == '#!' + python2_3_executable
True
>>> print f.read(),
<BLANKLINE>
import sys
sys.path[0:0] = [
'/tmp/sample-eggs/dist/demo-0.1-py2.3.egg',
'/tmp/sample-eggs/dist/demoneeded-1.0-py2.3.egg'
]
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
eggrecipedemo.main()
>>> zc.buildout.egglinker.path(['demo==0.1'], [sample_eggs],
... python2_3_executable)
... # doctest: +NORMALIZE_WHITESPACE
['/tmp/sample-eggs/dist/demo-0.1-py2.3.egg',
'/tmp/sample-eggs/dist/demoneeded-1.0-py2.3.egg']
>>> zc.buildout.egglinker.location('demo==0.1', [sample_eggs],
... python2_3_executable)
'/tmp/sample-eggs/demo-0.1-py2.3.egg'
>>> [(d.project_name, d.version) for d in
... zc.buildout.egglinker.distributions(
... ['demo==0.1'], [sample_eggs], python2_3_executable)]
[('demo', '0.1'), ('demoneeded', '1.0')]
>>> scripts = zc.buildout.egglinker.scripts(
... ['demo==0.1'], bin, [sample_eggs],
... executable=python2_4_executable)
>>> f = open(os.path.join(bin, 'demo'))
>>> f.readline().strip() == '#!' + python2_4_executable
True
>>> print f.read(),
<BLANKLINE>
import sys
sys.path[0:0] = [
'/tmp/sample-eggs/dist/demo-0.1-py2.4.egg',
'/tmp/sample-eggs/dist/demoneeded-1.0-py2.4.egg'
]
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
eggrecipedemo.main()
>>> zc.buildout.egglinker.path(['demo==0.1'], [sample_eggs],
... python2_4_executable)
... # doctest: +NORMALIZE_WHITESPACE
['/tmp/sample-eggs/dist/demo-0.1-py2.4.egg',
'/tmp/sample-eggs/dist/demoneeded-1.0-py2.4.egg']
>>> shutil.rmtree(bin)
src/zc/buildout/testing.py
View file @ 2530eeb2
...@@ -16,11 +16,13 @@ ...@@ -16,11 +16,13 @@
$Id$ $Id$
""" """
import ConfigParser, os, re, shutil, sys, tempfile, unittest
import BaseHTTPServer, ConfigParser, os, random, re, shutil, socket, sys
import tempfile, threading, time, urllib2, unittest
from zope.testing import doctest, renormalizing from zope.testing import doctest, renormalizing
import pkg_resources import pkg_resources
def cat(dir, *names): def cat(dir, *names):
path = os.path.join(dir, *names) path = os.path.join(dir, *names)
print open(path).read(), print open(path).read(),
...@@ -52,6 +54,9 @@ def system(command, input=''): ...@@ -52,6 +54,9 @@ def system(command, input=''):
i.close() i.close()
return o.read() return o.read()
def get(url):
return urllib2.urlopen(url).read()
def buildoutSetUp(test, clear_home=True): def buildoutSetUp(test, clear_home=True):
if clear_home: if clear_home:
# we both need to make sure that HOME isn't set and be prepared # we both need to make sure that HOME isn't set and be prepared
...@@ -91,6 +96,7 @@ def buildoutSetUp(test, clear_home=True): ...@@ -91,6 +96,7 @@ def buildoutSetUp(test, clear_home=True):
mkdir = mkdir, mkdir = mkdir,
write = write, write = write,
system = system, system = system,
get = get,
__original_wd__ = os.getcwd(), __original_wd__ = os.getcwd(),
)) ))
...@@ -133,14 +139,25 @@ def create_sample_eggs(test, executable=sys.executable): ...@@ -133,14 +139,25 @@ def create_sample_eggs(test, executable=sys.executable):
test.globs['sample_eggs'] = os.path.join(sample, 'dist') test.globs['sample_eggs'] = os.path.join(sample, 'dist')
write(sample, 'README.txt', '') write(sample, 'README.txt', '')
write(sample, 'eggrecipedemobeeded.py', 'y=1\n') for i in (0, 1):
write(sample, 'eggrecipedemobeeded.py', 'y=%s\n' % i)
write(
sample, 'setup.py',
"from setuptools import setup\n"
"setup(name='demoneeded', py_modules=['eggrecipedemobeeded'],"
" zip_safe=True, version='1.%s')\n"
% i
)
runsetup(sample, executable)
write( write(
sample, 'setup.py', sample, 'setup.py',
"from setuptools import setup\n" "from setuptools import setup\n"
"setup(name='demoneeded', py_modules=['eggrecipedemobeeded']," "setup(name='other', zip_safe=True, version='1.0', "
" zip_safe=True, version='1.0')\n" "py_modules=['eggrecipedemobeeded'])\n"
) )
runsetup(sample, executable) runsetup(sample, executable)
os.remove(os.path.join(sample, 'eggrecipedemobeeded.py')) os.remove(os.path.join(sample, 'eggrecipedemobeeded.py'))
for i in (1, 2, 3): for i in (1, 2, 3):
...@@ -170,5 +187,131 @@ def multi_python(test): ...@@ -170,5 +187,131 @@ def multi_python(test):
create_sample_eggs(test, executable=p24) create_sample_eggs(test, executable=p24)
test.globs['python2_3_executable'] = p23 test.globs['python2_3_executable'] = p23
test.globs['python2_4_executable'] = p24 test.globs['python2_4_executable'] = p24
def make_tree(test):
sample_eggs = test.globs['sample_eggs']
tree = dict(
[(n, open(os.path.join(sample_eggs, n), 'rb').read())
for n in os.listdir(sample_eggs)
])
tree['index'] = {}
return tree
class Server(BaseHTTPServer.HTTPServer):
def __init__(self, tree, *args):
BaseHTTPServer.HTTPServer.__init__(self, *args)
self.tree = tree
__run = True
def serve_forever(self):
while self.__run:
self.handle_request()
def handle_error(self, *_):
self.__run = False
class Handler(BaseHTTPServer.BaseHTTPRequestHandler):
def __init__(self, request, address, server):
self.tree = server.tree
BaseHTTPServer.BaseHTTPRequestHandler.__init__(
self, request, address, server)
def do_GET(self):
if '__stop__' in self.path:
raise SystemExit
tree = self.tree
for name in self.path.split('/'):
if not name:
continue
tree = tree.get(name)
if tree is None:
self.send_response(404, 'Not Found')
out = '<html><body>Not Found</body></html>'
self.send_header('Content-Length', str(len(out)))
self.send_header('Content-Type', 'text/html')
self.end_headers()
self.wfile.write(out)
return
self.send_response(200)
if isinstance(tree, dict):
out = ['<html><body>\n']
items = tree.items()
items.sort()
for name, v in items:
if isinstance(v, dict):
name += '/'
out.append('<a href="%s">%s</a><br>\n' % (name, name))
out.append('</body></html>\n')
out = ''.join(out)
self.send_header('Content-Length', str(len(out)))
self.send_header('Content-Type', 'text/html')
else:
out = tree
self.send_header('Content-Length', len(out))
if name.endswith('.egg'):
self.send_header('Content-Type', 'application/zip')
else:
self.send_header('Content-Type', 'text/html')
self.end_headers()
self.wfile.write(out)
def log_request(*s):
pass
def _run(tree, port):
server_address = ('localhost', port)
httpd = Server(tree, server_address, Handler)
httpd.serve_forever()
def get_port():
for i in range(10):
port = random.randrange(20000, 30000)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
try:
s.connect(('localhost', port))
except socket.error:
return port
finally:
s.close()
raise RuntimeError, "Can't find port"
def start_server(tree):
port = get_port()
threading.Thread(target=_run, args=(tree, port)).start()
wait(port, up=True)
return port
def stop_server(url):
try:
urllib2.urlopen(url+'__stop__')
except Exception:
pass
def wait(port, up):
addr = 'localhost', port
for i in range(120):
time.sleep(0.25)
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(addr)
s.close()
if up:
break
except socket.error, e:
if e[0] not in (errno.ECONNREFUSED, errno.ECONNRESET):
raise
s.close()
if not up:
break
else:
if up:
raise
else:
raise SystemError("Couln't stop server")
src/zc/buildout/tests.py
View file @ 2530eeb2
...@@ -66,21 +66,26 @@ It is an error to create a variable-reference cycle: ...@@ -66,21 +66,26 @@ It is an error to create a variable-reference cycle:
def linkerSetUp(test): def linkerSetUp(test):
zc.buildout.testing.buildoutSetUp(test, clear_home=False) zc.buildout.testing.buildoutSetUp(test, clear_home=False)
zc.buildout.testing.multi_python(test) zc.buildout.testing.multi_python(test)
test.globs['link_server'] = (
'http://localhost:%s/'
% zc.buildout.testing.start_server(zc.buildout.testing.make_tree(test))
)
def linkerTearDown(test): def linkerTearDown(test):
shutil.rmtree(test.globs['_sample_eggs_container']) shutil.rmtree(test.globs['_sample_eggs_container'])
zc.buildout.testing.buildoutTearDown(test) zc.buildout.testing.buildoutTearDown(test)
zc.buildout.testing.stop_server(test.globs['link_server'])
def buildoutTearDown(test): def buildoutTearDown(test):
shutil.rmtree(test.globs['extensions']) shutil.rmtree(test.globs['extensions'])
shutil.rmtree(test.globs['home']) shutil.rmtree(test.globs['home'])
zc.buildout.testing.buildoutTearDown(test) zc.buildout.testing.buildoutTearDown(test)
class PythonNormalizing(renormalizing.RENormalizing): class PythonNormalizing(renormalizing.RENormalizing):
def _transform(self, want, got): def _transform(self, want, got):
if '/xyzsample-eggs/' in want: if '/xyzsample-install/' in want:
got = got.replace('-py2.4.egg', '-py2.3.egg') got = got.replace('-py2.4.egg', '-py2.3.egg')
firstg = got.split('\n')[0] firstg = got.split('\n')[0]
firstw = want.split('\n')[0] firstw = want.split('\n')[0]
...@@ -149,14 +154,15 @@ def test_suite(): ...@@ -149,14 +154,15 @@ def test_suite():
), ),
doctest.DocFileSuite( doctest.DocFileSuite(
'egglinker.txt', 'easy_install.txt', 'easy_install.txt',
setUp=linkerSetUp, tearDown=linkerTearDown, setUp=linkerSetUp, tearDown=linkerTearDown,
checker=PythonNormalizing([ checker=PythonNormalizing([
(re.compile("'%(sep)s\S+sample-eggs%(sep)s(dist%(sep)s)?" (re.compile("'%(sep)s\S+sample-install%(sep)s(dist%(sep)s)?"
% dict(sep=os.path.sep)), % dict(sep=os.path.sep)),
'/sample-eggs/'), '/sample-eggs/'),
(re.compile("(- demo(needed)?-\d[.]\d-py)\d[.]\d[.]egg"), (re.compile("(- (demo(needed)?|other)"
"-\d[.]\d-py)\d[.]\d[.]egg"),
'\\1V.V.egg'), '\\1V.V.egg'),
]), ]),
), ),
......
testrunnerrecipe/setup.py
View file @ 2530eeb2
...@@ -7,7 +7,7 @@ setup( ...@@ -7,7 +7,7 @@ setup(
include_package_data = True, include_package_data = True,
package_dir = {'':'src'}, package_dir = {'':'src'},
namespace_packages = ['zc', 'zc.recipe'], namespace_packages = ['zc', 'zc.recipe'],
install_requires = ['zc.buildout', 'zope.testing'], install_requires = ['zc.buildout', 'zope.testing', 'setuptools'],
dependency_links = ['http://download.zope.org/distribution/'], dependency_links = ['http://download.zope.org/distribution/'],
test_suite = 'zc.recipe.testrunner.tests.test_suite', test_suite = 'zc.recipe.testrunner.tests.test_suite',
author = "Jim Fulton", author = "Jim Fulton",
......
testrunnerrecipe/src/zc/recipe/testrunner/README.txt
View file @ 2530eeb2
...@@ -75,6 +75,7 @@ develop egg and to create the test script: ...@@ -75,6 +75,7 @@ develop egg and to create the test script:
... [buildout] ... [buildout]
... develop = demo demo2 ... develop = demo demo2
... parts = testdemo ... parts = testdemo
... offline = true
... ...
... [testdemo] ... [testdemo]
... recipe = zc.recipe.testrunner ... recipe = zc.recipe.testrunner
...@@ -87,6 +88,8 @@ develop egg and to create the test script: ...@@ -87,6 +88,8 @@ develop egg and to create the test script:
Note that we specified both demo and demo2 in the distributions Note that we specified both demo and demo2 in the distributions
section and that we put them on separate lines. section and that we put them on separate lines.
We also specified the offline option to run the buildout in offline mode.
Now when we run the buildout: Now when we run the buildout:
>>> import os >>> import os
...@@ -113,6 +116,7 @@ script will get it's name from the part: ...@@ -113,6 +116,7 @@ script will get it's name from the part:
... [buildout] ... [buildout]
... develop = demo ... develop = demo
... parts = testdemo ... parts = testdemo
... offline = true
... ...
... [testdemo] ... [testdemo]
... recipe = zc.recipe.testrunner ... recipe = zc.recipe.testrunner
......
testrunnerrecipe/src/zc/recipe/testrunner/__init__.py
View file @ 2530eeb2
...@@ -17,7 +17,8 @@ $Id$ ...@@ -17,7 +17,8 @@ $Id$
""" """
import os, sys import os, sys
import zc.buildout.egglinker import pkg_resources
import zc.buildout.easy_install
class TestRunner: class TestRunner:
...@@ -30,25 +31,28 @@ class TestRunner: ...@@ -30,25 +31,28 @@ class TestRunner:
) )
options['_e'] = buildout['buildout']['eggs-directory'] options['_e'] = buildout['buildout']['eggs-directory']
options['_d'] = buildout['buildout']['develop-eggs-directory'] options['_d'] = buildout['buildout']['develop-eggs-directory']
python = options.get('python', buildout['buildout']['python'])
options['executable'] = buildout[python]['executable']
def install(self): def install(self):
distributions = [ options = self.options
req.strip() requirements = [r.strip()
for req in self.options['distributions'].split('\n') for r in options['distributions'].split('\n')
if req.split() if r.strip()]
]
path = zc.buildout.egglinker.path( ws = zc.buildout.easy_install.working_set(
distributions+['zope.testing'], requirements+['zope.testing'],
[self.options['_d'], self.options['_e']], executable = options['executable'],
path=[options['_d'], options['_e']]
) )
locations = [zc.buildout.egglinker.location( path = [dist.location for dist in ws]
distribution, locations = [dist.location for dist in ws
[self.options['_d'], self.options['_e']]) if dist.project_name != 'zope.testing']
for distribution in distributions]
script = self.options['script'] script = options['script']
open(script, 'w').write(tests_template % dict( open(script, 'w').write(tests_template % dict(
PYTHON=sys.executable, PYTHON=options['executable'],
PATH="',\n '".join(path), PATH="',\n '".join(path),
TESTPATH="',\n '--test-path', '".join(locations), TESTPATH="',\n '--test-path', '".join(locations),
)) ))
......
testrunnerrecipe/src/zc/recipe/testrunner/tests.py
View file @ 2530eeb2
...@@ -17,7 +17,8 @@ import pkg_resources ...@@ -17,7 +17,8 @@ import pkg_resources
import zc.buildout.testing import zc.buildout.testing
import unittest import unittest
from zope.testing import doctest, renormalizing import zope.testing
from zope.testing import doctest
def dirname(d, level=1): def dirname(d, level=1):
if level == 0: if level == 0:
...@@ -29,6 +30,11 @@ def setUp(test): ...@@ -29,6 +30,11 @@ def setUp(test):
open(os.path.join(test.globs['sample_buildout'], open(os.path.join(test.globs['sample_buildout'],
'eggs', 'zc.recipe.testrunner.egg-link'), 'eggs', 'zc.recipe.testrunner.egg-link'),
'w').write(dirname(__file__, 4)) 'w').write(dirname(__file__, 4))
# XXX assumes that zope.testing egg is a directory
open(os.path.join(test.globs['sample_buildout'],
'eggs', 'zope.testing.egg-link'),
'w').write(dirname(zope.testing.__file__, 3))
def tearDown(test): def tearDown(test):
zc.buildout.testing.buildoutTearDown(test) zc.buildout.testing.buildoutTearDown(test)
......
todo.txt
View file @ 2530eeb2
- tests
- distribution dependency links
- offline mode (there is an indirect test in the testrunner tests)
- Windows support - Windows support
- Load from urls - Load from urls
- control python for develop (probbaly a new recipe)
- proper handling of extras
- Common recipes - Common recipes
- configure-make-make-install - configure-make-make-install
...@@ -14,10 +24,6 @@ ...@@ -14,10 +24,6 @@
- Python - Python
- Need to better understand the way upgrading works in setuptools.
- Offline mode
- Some way to freeze versions so we can have reproducable buildouts. - Some way to freeze versions so we can have reproducable buildouts.
Maybe simple approach: Maybe simple approach:
...@@ -27,6 +33,8 @@ ...@@ -27,6 +33,8 @@
- Egg recipe has option to specify dependencies. When used, - Egg recipe has option to specify dependencies. When used,
don't automatically fetch newer data. don't automatically fetch newer data.
- Option to search python path for distros
- Part dependencies - Part dependencies
- custom uninstall - custom uninstall
...@@ -34,14 +42,21 @@ ...@@ -34,14 +42,21 @@
- Fix develop so thet ordinary eggs fetched as dependencies end up - Fix develop so thet ordinary eggs fetched as dependencies end up
in eggs directory. in eggs directory.
"fixed" except that fix is ineffective due to setuptools bug. :(
- spelling :) - spelling :)
- document recipe initialization order - document recipe initialization order
Issues Issues
- Want to be able to control whether eggs get unzipped when they are - Should we include setuptools and buildout eggs for buildout process
installed. This requires looking at a distribution after it's in environment when searching for requirements?
installed and unzipping it if it's zipped.
- We don't want to look for new versions of setuptools all the time.
For now, we always use a local dist if there is one. Needs more
thought.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7