Skip to content

S
slapos.rebootstrap
Commit 225767ad authored by Kazuhiko Shiozaki's avatar Kazuhiko Shiozaki
Browse files
Options

Rewrite for zc.buildout >= 2.0.0. 

 * Same parts directory as the normal buildout is used so that we can
   build faster and also make the code simpler.
 * Python part is now specified by `python` parameter in `buildout`
   section.
 * Restart automatically with the original python when running python
   part is removed.
parent 26f6d42a
Show whitespace changes
Inline Side-by-side
Showing with 89 additions and 181 deletions
CHANGES.txt
View file @ 225767ad
......@@ -4,7 +4,13 @@ Changes
3.2 (unreleased)
----------------
* No changes yet.
* Support zc.buildout >= 2.0.0.
* Same parts directory as the normal buildout is used so that we can
build faster and also make the code simpler.
* Python part is now specified by `python` parameter in `buildout`
section.
* Restart automatically with the original python when running python
part is removed.
3.1 (2011-06-24)
----------------
......
README.txt
View file @ 225767ad
Chicken and egg python zc.buildout extension
============================================
Rerun buildout with a python provided by buildout
===============================================
This extensions for zc.buildout is created to solve chicken and egg problem
while working with buildout and when some exact version of python, which is
provided by buildout shall be used to *execute* this buildout itself.
Buildout 2.0 no-longer supports using multiple versions of Python in a
single buildout. This extension makes it possible to use 'yet another
single' version of python, by building a specified python part and its
dependencies first then reinvoking buildout with the executable.
Usage
-----
Part to build python is required. By convention slapos.rebootstrap will try to
find python executable in:
A part to build python is required. You need to specify it by `python`
parameter in `buildout` section, just same as buildout 1.x. The python
section must provide executable option that gives the path to a Python
executable.
special.parts.directory/partname/bin/partname
And add slapos.rebootstrap to `extensions` parameter in `buildout` section.
But when needed python-path parameter can be used to point rebootstrap to find
python in:
special.parts.directory/partname/python-path
Add slapos.rebootstrap to extensions and set rebooter-section to above section.
Use whatever python to bootstrap and run buildout. If reboot will detect that
sys.executable used to run buildout is different then executable provided in
python section it will try to find this executable. If it does not exists it
will install this section and then reinstall buildout using new python
executable. Later buildout run will continue using new python.
Because external buildout is used to provide buildout version parameter is
introduced to being able to upgrade not in place python part. This parameter
is required and becomes part of suffix.
Whenever developer-mode is set to true no cleanup will be done in case of
failure. Then user is responsible to cleanup directories.
Use whatever python to bootstrap and run buildout. If this extension
detects that sys.executable used to run buildout is different than
executable provided in python section, it will try to find this
executable. If it does not exists, it will install this section and
then reinstall buildout using new python executable. Later buildout
run will continue using new python.
Example profile and invocation
------------------------------
......@@ -39,14 +29,11 @@ Example profile and invocation
[buildout]
extensions = slapos.rebootstrap
python = slapospython
parts =
realrun
[rebootstrap]
section = slapospython
version = 1
[slapospython]
recipe = plone.recipe.command
stop-on-error = true
......@@ -61,7 +48,7 @@ Example profile and invocation
After bootstrapping and running this buildout it will print:
Running with python /path/to/buildout/parts.rebootstrap.1/slapospython/bin/slapospython
Running with python /path/to/buildout/parts/slapospython/bin/slapospython
Running tests
-------------
......
setup.py
View file @ 225767ad
from setuptools import setup, find_packages
version = '3.2-dev'
version = '3.2'
name = "slapos.rebootstrap"
long_description = open("README.txt").read() + '\n\n'
......@@ -38,7 +38,7 @@ setup(
zip_safe=True,
install_requires=[
'setuptools',
'zc.buildout'
'zc.buildout >=2.0.0'
],
tests_require=[
'zope.testing',
......
slapos/rebootstrap/__init__.py
View file @ 225767ad
......@@ -12,7 +12,6 @@
#
##############################################################################
import os
import shutil
import zc.buildout
import sys
import logging
......@@ -25,53 +24,22 @@ class Rebootstrap:
def __init__(self, buildout):
self.logger = logging.getLogger(__name__)
self.buildout = buildout
# fetch section of rebootstrap, obligatory
rebootstrap_section = buildout['rebootstrap']
# leave garbage? only in case of developer mode
self.developer_mode = rebootstrap_section.get('developer-mode',
'false') == 'true'
# fetch section to build python, obligatory
self.python_section = rebootstrap_section['section'].strip()
# fetch version of python build, obligatory
version = rebootstrap_section['version'].strip()
self.python_section = (
buildout['buildout'].get('python') or \
buildout['rebootstrap']['section']
).strip()
self.wanted_python = buildout[self.python_section]['executable']
# usualy python will be try to found by convention in:
# prefixed-parts/part-name/bin/part-name
# but in case if part is non controlable it will be possible to
# find it as
# prefixed-parts/part-name/python-path
# where python-path is relative path to python interpreter
relative_python = rebootstrap_section.get('python-path', ''
).strip() or os.path.join('bin', self.python_section)
# query for currently running python
self.running_python = sys.executable
# generate rebootstrap.realname.version
prefix = 'rebootstrap.' + version + '.'
self.installed = '.' + prefix + 'installed.cfg'
self.parts_directory = os.path.join(self.buildout['buildout'][
'directory'], prefix + 'parts')
# query for wanted python, which is combination of suffixed parts
# and working recipe
self.wanted_python = os.path.join(self.parts_directory,
self.python_section, relative_python)
# support additional eggs
self.eggs = []
eggs = rebootstrap_section.get('eggs')
if eggs:
eggs = [q.strip() for q in eggs.split() if q.strip()]
self.eggs = eggs
def __call__(self):
if self.running_python != self.wanted_python:
self.install_section()
self.reboot()
def reboot(self):
if self.running_python == self.wanted_python:
return
message = """
************ REBOOTSTRAP: IMPORTANT NOTICE ************
bin/buildout is being reinstalled right now, as new python:
......@@ -83,72 +51,16 @@ Buildout will be restarted automatically to have this change applied.
""" % dict(wanted_python=self.wanted_python,
running_python=self.running_python)
self.logger.info(message)
options = self.buildout['buildout']
self.eggs.append('zc.buildout')
# XXX: A lot of below code is took from zc.buildout.buildout.py:_maybe_upgrade
# which is code duplication issue, but even if newer buildout with needed
# hooks will be released, this extension shall work on older ones
if zc.buildout.easy_install.is_distribute:
self.eggs.append('distribute')
else:
self.eggs.append('setuptools')
ws = zc.buildout.easy_install.install(
[
(spec + ' ' + options.get(spec+'-version', '')).strip()
for spec in self.eggs
],
options['eggs-directory'],
links = options.get('find-links', '').split(),
index = options.get('index'),
path = [options['develop-eggs-directory']],
prefer_final=False,
executable=self.wanted_python
)
# the new dist is different, so we've upgraded.
# Update the scripts and return True
# Ideally the new version of buildout would get a chance to write the
# script. Not sure how to do that.
partsdir = os.path.join(options['parts-directory'], 'buildout')
if os.path.exists(partsdir):
# This is primarily for unit tests, in which .py files change too
# fast for Python to know to regenerate the .pyc/.pyo files.
shutil.rmtree(partsdir)
os.mkdir(partsdir)
script_initialization = ''
# (Honor the relative-paths option.)
relative_paths = options.get('relative-paths', 'false')
if relative_paths == 'true':
relative_paths = options['directory']
else:
assert relative_paths == 'false'
relative_paths = ''
zc.buildout.easy_install.sitepackage_safe_scripts(
options['bin-directory'], ws, self.wanted_python, partsdir,
reqs=['zc.buildout'], relative_paths=relative_paths,
include_site_packages=False,
script_initialization=script_initialization,
)
# Restart
args = map(zc.buildout.easy_install._safe_arg, sys.argv)
# We want to make sure that our new site.py is used for rerunning
# buildout, so we put the partsdir in PYTHONPATH for our restart.
# This overrides any set PYTHONPATH, but since we generally are
# trying to run with a completely "clean" python (only the standard
# library) then that should be fine.
env = os.environ.copy()
env['PYTHONPATH'] = partsdir
sys.stdout.flush()
env = os.environ
env['ORIG_PYTHON'] = sys.executable
os.execve(self.wanted_python, [self.wanted_python] + args, env)
def install_section(self):
try_install = False
try:
if not os.path.exists(self.parts_directory) \
or not os.path.exists(self.wanted_python):
try_install = True
self.logger.info('Installing section %r to provide %r in %r' % (
self.python_section, self.wanted_python, self.parts_directory))
env = os.environ
if not os.path.exists(self.wanted_python):
self.logger.info('Installing section %r to provide %r' % (
self.python_section, self.wanted_python))
args = map(zc.buildout.easy_install._safe_arg, sys.argv)
if 'install' in args:
args = args[:args.index('install')]
......@@ -162,11 +74,8 @@ Buildout will be restarted automatically to have this change applied.
args.extend([
# install only required section with dependencies
'buildout:parts=%s' % self.python_section,
# do not load extensions (even other ones)
# do not load this extension
'buildout:extensions=%s' % ' '.join(extension_list),
# create virutal python environment
'buildout:parts-directory=%s' % self.parts_directory,
'buildout:installed=%s' % self.installed,
])
self.logger.info('Rerunning buildout to install section %s with '
'arguments %r.'% (self.python_section, args))
......@@ -180,17 +89,23 @@ Buildout will be restarted automatically to have this change applied.
'%r\nUnfortunately even after installing this section executable was'
' not found.\nThis is section responsibility to provide python (eg. '
'by compiling it).' % (self.python_section, self.wanted_python))
except zc.buildout.UserError:
if try_install:
if not self.developer_mode:
if os.path.exists(self.parts_directory):
shutil.rmtree(self.parts_directory)
if os.path.exists(self.installed):
os.unlink(self.installed)
else:
self.logger.warning('Developer mode active.')
self.logger.warning('Directory %r and file %r are left for further '
'analysis.' % (self.parts_directory, self.installed))
self.logger.warning('Developer is responsible for removing those '
'directories.')
raise
_uninstall_part_orig = zc.buildout.buildout.Buildout._uninstall_part
def _uninstall_part(self, part, installed_part_options):
_uninstall_part_orig(self, part, installed_part_options)
location = self[part].get('location')
if location and sys.executable.startswith(location):
message = """
************ REBOOTSTRAP: IMPORTANT NOTICE ************
%r part that provides the running Python is uninstalled.
Buildout will be restarted automatically with the original Python.
************ REBOOTSTRAP: IMPORTANT NOTICE ************
""" % part
self._logger.info(message)
if getattr(self, 'dry_run', False):
sys.exit()
args = map(zc.buildout.easy_install._safe_arg, sys.argv)
env = os.environ
orig_python = env['ORIG_PYTHON']
os.execve(orig_python, [orig_python] + args, env)
zc.buildout.buildout.Buildout._uninstall_part = _uninstall_part
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7