Skip to content

S
slapos.recipe.cmmi
Commit 5239a27d authored by Xiaowu Zhang's avatar Xiaowu Zhang
1 Browse files
Options

Reimplement shared features 

1.use shared-part in [buildout] section to define shared path
2.shared option is True or Flase
3.a magic substitution @@LOCATION@@ is used to for dynamic path
parent aa5a651b
Show whitespace changes
Inline Side-by-side
Showing with 159 additions and 48 deletions
slapos/recipe/cmmi/README.txt
View file @ 5239a27d
......@@ -26,25 +26,13 @@ Supported options
``--prefix`` injection takes place. You can also set the ``--prefix``
parameter explicitly in ``configure-options``.
``share``
``shared``
Specify the path in which this package is shared by many other
packages. When it's unset or blank, it means the package isn't
shared. Otherwise, it shared by many packages. Recipe will return
an empty list so that buildout will not uninstall the package when
uninstalling part.
In share mode, ``promises`` should be set so that recipe can tell
whether the package is installed. Otherwise nohting to do.
If ``share`` is not empty, ``location`` will be set to value of
``share``, and remove trailing '/'. So in case ``share`` is '/',
``location`` will be set to blank. Thus any reference like
"${part:location}/bin" in other parts will get the correct value.
This option is experiment now. Recipe doesn't try to set prefix
with the valule of this opton in the configure or make command,
you should specify them accordingly by yourself.
packages.
Shared-part should be defined in [buildout] section
Shared option is True or False
The package will be installed on path/name/hash of options.
``md5sum``
......@@ -1019,10 +1007,17 @@ Look, "package" is reinstalled either:
building package
installing package
Install share package
Install shared package
=====================
Use option ``share`` to install a share pacakge.
Use option ``shared`` to install a shared pacakge.
>>> import os
>>> _ = system('rm -rf %s' % join(os.path.dirname(__file__), 'shared'))
>>> shared_dir = join(os.path.dirname(__file__), 'shared')
>>> os.mkdir(shared_dir)
If no shared-part is set, and shared is True, an error is raised:
>>> write('buildout.cfg',
... """
... [buildout]
......@@ -1032,41 +1027,111 @@ Use option ``share`` to install a share pacakge.
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... share = /usr/local
... promises = ${:share}/bin/mypackage.exe
... """ % (src,))
>>> print system(buildout)
... shared = True
... """% src)
>>> print system(buildout) #doctest:+ELLIPSIS
While:
Installing.
Getting section package.
Initializing section package.
...
ValueError: Set ${buildout:shared-part} for shared feature
If shared-part is set and shared is True, build package failed, the build directory is removed, a build directory__complile__ is left for debugging
>>> _ = system('mv %s/package-0.0.0.tar.gz %s/package-0.0.0.tar.gz.bak' % (src, src))
>>> import tarfile
>>> from io import BytesIO
>>> import sys
>>> tarpath = os.path.join(src, 'package-0.0.0.tar.gz')
>>> with tarfile.open(tarpath, 'w:gz') as tar:
... configure = 'invalid'
... info = tarfile.TarInfo('configure.off')
... info.size = len(configure)
... info.mode = 0o755
... tar.addfile(info, BytesIO(configure))
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part = %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... shared = True
... """ % (shared_dir, src))
>>> print system(buildout) #doctest:+ELLIPSIS
package: shared directory .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/... set for package
Uninstalling package.
Uninstalling package-2.
Installing package.
package: Checking whether package is installed at share path: /usr/local
package: could not find promise "/usr/local/bin/mypackage.exe"
package: [ENV] TMP = /sample_buildout/parts/package/tmp
configure --prefix=/usr/local
package: Checking whether package is installed at shared path: .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...
package: [ENV] TMP = .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/.../tmp
package: Command failed with exit code 127: ./configure --prefix=".../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/..."
package: Compilation error. The package is left as is at .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...__compile__ where you can inspect what went wrong
/bin/sh: 1: ./configure: not found
While:
Installing package.
Error: System error
If shared-part is set and shared is True, package will be installed in shared_part/package/a hash of the recipe's configuration options
>>> _ = system('mv %s/package-0.0.0.tar.gz.bak %s/package-0.0.0.tar.gz' % (src, src))
>>> print system(buildout) #doctest:+ELLIPSIS
package: shared directory .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/... set for package
Installing package.
package: Checking whether package is installed at shared path: .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...
package: [ENV] TMP = .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/.../tmp
package: Removing already existing directory .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...__compile__
configure --prefix=.../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...
building package
installing package
package: could not find promise "/usr/local/bin/mypackage.exe"
Do nothing if one package has been installed.
>>> remove('.installed.cfg')
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part = %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... share = /usr/local/bin
... promises =
... """ % (src,))
... shared = True
... """ % (shared_dir, src))
>>> print system(buildout) #doctest:+ELLIPSIS
package: shared directory .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/... set for package
Installing package.
package: Checking whether package is installed at shared path: .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...
package: This shared package has been installed by other package
>>> print system(buildout)
If options change, reinstall in different location:
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part = %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... shared =True
... change = True
... """ % (shared_dir, src))
>>> print system(buildout) #doctest:+ELLIPSIS
package: shared directory .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/... set for package
Uninstalling package.
Installing package.
package: Checking whether package is installed at share path: /usr/local/bin
package: This shared package has been installed by other package
package: Checking whether package is installed at shared path: .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...
package: [ENV] TMP = .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/.../tmp
configure --prefix=.../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...
building package
installing package
For even more specific needs you can write your own recipe that uses
``slapos.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``true``.
......
slapos/recipe/cmmi/__init__.py
View file @ 5239a27d
......@@ -29,7 +29,7 @@ class Recipe(object):
self.options = options
self.buildout = buildout
self.name = name
log = logging.getLogger(self.name)
# Merge options if there is a matched platform section
platform_options = self.buildout.get(
"%s:%s:%s" % (name, sys.platform, self.get_machine()),
......@@ -40,12 +40,41 @@ class Recipe(object):
self.original_options = options.copy()
options.update(platform_options)
options['share'] = options.get('share', '').strip()
options['default-location'] = os.path.join(
shared = options.get('shared', '')
if shared:
shared_part = buildout['buildout'].get('shared-part', None)
if not shared_part:
raise ValueError(
" Set ${buildout:shared-part} for shared feature")
shared = os.path.join(shared_part.strip().rstrip('/'), self.name)
if not os.path.exists(shared):
os.makedirs(shared)
self._debug_signature_text = []
# compute hash from options
m = md5()
buildout_directory = buildout['buildout']['directory']
profile_base_location = options.get('_profile_base_location_', '')
for k, v in sorted(options.items()):
# Key not vary on profile base location
if profile_base_location:
v = v.replace(profile_base_location, '${:_profile_base_location_}')
option_signature = ('%r: %r' % (k, v)).encode()
log.debug("Updating hash with %s", option_signature)
self._debug_signature_text.append(option_signature)
# Or maybe warn and disable cache
assert buildout_directory not in option_signature
m.update(option_signature)
shared = os.path.join(shared, m.hexdigest())
log.info('shared directory %s set for %s' % (shared, self.name))
options['shared'] = shared
default_location = options['default-location'] = os.path.join(
buildout['buildout']['parts-directory'],
self.name)
share = options['share']
options['location'] = options['default-location'] if share == '' else share.rstrip('/')
self.name) if shared == '' else shared
options['location'] = options['default-location']
prefix = options.get('prefix', '').strip()
if prefix == '':
......@@ -88,6 +117,11 @@ class Recipe(object):
else:
options['compile-directory'] = options['path']
from copy import copy
for k, v in copy(options.items()):
  • Julien Muchembled @jm ·
    Owner

    options.items() already returns a copy so the copy module must not be used.

    /cc @yusei

Please register or sign in to reply
if '@@LOCATION@@' in v:
options[k] = v.replace('@@LOCATION@@', default_location)
self.environ = {}
self.original_environment = os.environ.copy()
......@@ -250,10 +284,10 @@ class Recipe(object):
log = logging.getLogger(self.name)
parts = []
# In share mode, do nothing if package has been installed.
if (not self.options['share'] == ''):
log.info('Checking whether package is installed at share path: %s' % self.options['share'])
if self.check_promises(log):
# In shared mode, do nothing if package has been installed.
if (not self.options['shared'] == ''):
log.info('Checking whether package is installed at shared path: %s' % self.options['shared'])
if os.path.exists(self.options['shared']):
log.info('This shared package has been installed by other package')
return parts
......@@ -314,6 +348,13 @@ class Recipe(object):
shutil.rmtree(tmp_path, True)
os.mkdir(tmp_path)
# Create a debug file with the signatures.
if self.options['shared']:
with open(os.path.join(
self.options["shared"],
".slapos.recipe.cmmi.signature"), 'w') as f:
f.write('\n'.join(self._debug_signature_text))
try:
try:
# We support packages that either extract contents to the $PWD
......@@ -369,16 +410,21 @@ class Recipe(object):
log.info('Executing post-install')
self.run(post_install_cmd)
if (self.buildout_prefix != ''
and self.options['share'] == ''
and self.options['shared'] == ''
and os.path.exists(self.buildout_prefix)):
log.info('Getting installed file lists')
parts.extend(self.get_installed_files(tmp_path))
except:
log.error('Compilation error. The package is left as is at %s where '
'you can inspect what went wrong' % os.getcwd())
# Delete shared directory if not correctly installed
if self.options.get('shared'):
shutil.rmtree(self.options['shared'])
raise
finally:
os.chdir(current_dir)
# If in shared mode and have error during installed, tmp_path was already deleted
if os.path.isdir(tmp_path):
shutil.rmtree(tmp_path)
# Check promises
......@@ -395,7 +441,7 @@ class Recipe(object):
shutil.rmtree(compile_dir)
del self.options['compile-directory']
if self.options['share'] == '':
if self.options['shared'] == '':
parts.append(self.options['default-location'])
self.fix_shebang(self.options['default-location'])
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7