Skip to content

S
slapos.recipe.cmmi
Commit deda4108 authored by Julien Muchembled's avatar Julien Muchembled
Browse files
Options

Switch to slapos.recipe.build implementation of environment & shared options 

'environment-section' option is dropped.
parent 3b128565
Pipeline #18545 passed with stage
    in 0 seconds
    Expand all Show whitespace changes
    Inline Side-by-side
    Showing with 102 additions and 363 deletions
    setup.py
    View file @ deda4108
    ...@@ -43,7 +43,7 @@ setup( ...@@ -43,7 +43,7 @@ setup(
    install_requires=[ install_requires=[
    'zc.buildout>=2', 'zc.buildout>=2',
    'setuptools', 'setuptools',
    'slapos.recipe.build>=0.40', 'slapos.recipe.build>=0.49',
    ], ],
    extras_require={ extras_require={
    'test': ['zope.testing', 'manuel'], 'test': ['zope.testing', 'manuel'],
    ......
    slapos/recipe/cmmi/README-SlapOS.rst
    View file @ deda4108
    This document describe the usages of ``slapos.recipe.cmmi`` in SlapOS. This document describes some usages of ``slapos.recipe.cmmi`` in SlapOS.
    Nothing here is specific to ``slapos.recipe.cmmi``:
    - ``${:_profile_base_location_}`` support comes from SlapOS buildout
    - the mechanism to share parts is inherited from ``slapos.recipe.build``
    SlapOS ``${:_profile_base_location_}`` support SlapOS ``${:_profile_base_location_}`` support
    ============================================== ==============================================
    ...@@ -52,7 +56,7 @@ We have profiles for this, they are located at URL ``URL1``. ...@@ -52,7 +56,7 @@ We have profiles for this, they are located at URL ``URL1``.
    ... recipe = slapos.recipe.cmmi ... recipe = slapos.recipe.cmmi
    ... url = file://%s ... url = file://%s
    ... patches = ${:_profile_base_location_}/../dummy.patch ... patches = ${:_profile_base_location_}/../dummy.patch
    ... shared = True ... shared = true
    ... """ % package_path) ... """ % package_path)
    >>> write('URL1/haproxy.cfg', >>> write('URL1/haproxy.cfg',
    ...@@ -65,7 +69,7 @@ We have profiles for this, they are located at URL ``URL1``. ...@@ -65,7 +69,7 @@ We have profiles for this, they are located at URL ``URL1``.
    ... recipe = slapos.recipe.cmmi ... recipe = slapos.recipe.cmmi
    ... url = file://%s ... url = file://%s
    ... patches = ${:_profile_base_location_}/../dummy.patch ... patches = ${:_profile_base_location_}/../dummy.patch
    ... shared = True ... shared = true
    ... environment= ... environment=
    ... CFLAGS=-I${package:location}/include/ ... CFLAGS=-I${package:location}/include/
    ... """ % package_path) ... """ % package_path)
    ...@@ -87,16 +91,14 @@ We have a buildout using these profiles from ``URL1``: ...@@ -87,16 +91,14 @@ We have a buildout using these profiles from ``URL1``:
    package: shared at /shared/package/<MD5SUM:0> package: shared at /shared/package/<MD5SUM:0>
    haproxy: shared at /shared/haproxy/<MD5SUM:1> haproxy: shared at /shared/haproxy/<MD5SUM:1>
    Installing package. Installing package.
    package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
    package: Applying patches package: Applying patches
    patching file dummy.txt patching file dummy.txt
    configure --prefix=/shared/package/<MD5SUM:0> configure --prefix=/shared/package/<MD5SUM:0>
    building package building package
    installing package installing package
    Installing haproxy. Installing haproxy.
    haproxy: Checking whether package is installed at shared path: /shared/haproxy/<MD5SUM:1>
    haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:0>/include/
    haproxy: Applying patches haproxy: Applying patches
    haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:0>/include/
    patching file dummy.txt patching file dummy.txt
    configure --prefix=/shared/haproxy/<MD5SUM:1> configure --prefix=/shared/haproxy/<MD5SUM:1>
    building package building package
    ...@@ -139,11 +141,9 @@ and not installed again. ...@@ -139,11 +141,9 @@ and not installed again.
    Uninstalling haproxy. Uninstalling haproxy.
    Uninstalling package. Uninstalling package.
    Installing package. Installing package.
    package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0> package: shared part is already installed
    package: This shared package has been installed by other package
    Installing haproxy. Installing haproxy.
    haproxy: Checking whether package is installed at shared path: /shared/haproxy/<MD5SUM:1> haproxy: shared part is already installed
    haproxy: This shared package has been installed by other package
    On the other hand, if the ``package`` becomes different, then it will be re-installed at another On the other hand, if the ``package`` becomes different, then it will be re-installed at another
    ...@@ -155,7 +155,7 @@ shared location and ``haproxy``, which depend on ``package`` will also be re-ins ...@@ -155,7 +155,7 @@ shared location and ``haproxy``, which depend on ``package`` will also be re-ins
    ... recipe = slapos.recipe.cmmi ... recipe = slapos.recipe.cmmi
    ... url = file://%s ... url = file://%s
    ... # no patch this time ... # no patch this time
    ... shared = True ... shared = true
    ... """ % package_path) ... """ % package_path)
    >>> print(system(buildout)) >>> print(system(buildout))
    ...@@ -164,14 +164,12 @@ shared location and ``haproxy``, which depend on ``package`` will also be re-ins ...@@ -164,14 +164,12 @@ shared location and ``haproxy``, which depend on ``package`` will also be re-ins
    Uninstalling haproxy. Uninstalling haproxy.
    Uninstalling package. Uninstalling package.
    Installing package. Installing package.
    package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:2>
    configure --prefix=/shared/package/<MD5SUM:2> configure --prefix=/shared/package/<MD5SUM:2>
    building package building package
    installing package installing package
    Installing haproxy. Installing haproxy.
    haproxy: Checking whether package is installed at shared path: /shared/haproxy/<MD5SUM:3>
    haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:2>/include/
    haproxy: Applying patches haproxy: Applying patches
    haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:2>/include/
    patching file dummy.txt patching file dummy.txt
    configure --prefix=/shared/haproxy/<MD5SUM:3> configure --prefix=/shared/haproxy/<MD5SUM:3>
    building package building package
    ...@@ -213,15 +211,17 @@ tool will be able to delete this previous version of ``haproxy``. ...@@ -213,15 +211,17 @@ tool will be able to delete this previous version of ``haproxy``.
    False False
    We also needed something to know the dependences between shared parts. For this, we are using We also needed something to know the dependences between shared parts. For this, we are using
    ``.slapos.recipe.cmmi.signature`` files in folders where shared parts are installed. Because ``.buildout-shared.json`` files in folders where shared parts are installed. Because
    ``haproxy`` depends on ``package`` version ``<MD5SUM:2>``, we can see ``<MD5SUM:2>`` in its ``haproxy`` depends on ``package`` version ``<MD5SUM:2>``, we can see ``<MD5SUM:2>`` in its
    signature file. signature file.
    >>> haproxy_location = read_installed()['haproxy']['location'] >>> haproxy_location = read_installed()['haproxy']['location']
    >>> haproxy_location >>> haproxy_location
    '/shared/haproxy/<MD5SUM:3>' '/shared/haproxy/<MD5SUM:3>'
    >>> cat(join(haproxy_location, '.slapos.recipe.cmmi.signature')) # doctest: +ELLIPSIS >>> cat(join(haproxy_location, '.buildout-shared.json')) # doctest: +ELLIPSIS
    '__buildout_signature__': ... {
    "__buildout_signature__": ...
    ...<MD5SUM:2>... ...<MD5SUM:2>...
    }
    This is also useful during testing, we check that signatures do not have references to non-shared parts. This is also useful during testing, we check that signatures do not have references to non-shared parts.
    slapos/recipe/cmmi/README.rst
    View file @ deda4108
    ...@@ -33,11 +33,7 @@ Supported options ...@@ -33,11 +33,7 @@ Supported options
    ``shared`` ``shared``
    Specify the path in which this package is shared by many other See documentation of slapos.recipe.build's default recipe.
    packages.
    ``shared-part-list`` should be defined in ``[buildout]`` section
    Shared option is True or False
    The package will be installed on path/name/hash of options.
    ``md5sum`` ``md5sum``
    ...@@ -179,40 +175,9 @@ Supported options ...@@ -179,40 +175,9 @@ Supported options
    If any item doesn't exist, the recipe shows a warning message. The If any item doesn't exist, the recipe shows a warning message. The
    default value is empty. default value is empty.
    ``environment-section``
    Name of a section that provides environment variables that will be used to
    augment the variables read from ``os.environ`` before executing the
    recipe.
    This recipe does not modify ``os.environ`` directly. External commands
    run as part of the recipe (e.g. make, configure, etc.) get an augmented
    environment when they are forked. Python hook scripts are passed the
    augmented as a parameter.
    The values of the environment variables may contain references to other
    existing environment variables (including themselves) in the form of
    Python string interpolation variables using the dictionary notation. These
    references will be expanded using values from ``os.environ``. This can be
    used, for example, to append to the ``PATH`` variable, e.g.::
    [component]
    recipe = slapos.recipe.cmmi
    environment-section =
    environment
    [environment]
    PATH = %(PATH)s:${buildout:directory}/bin
    ``environment`` ``environment``
    A sequence of ``KEY=VALUE`` pairs separated by newlines that define See documentation of slapos.recipe.build's default recipe.
    additional environment variables used to update ``os.environ`` before
    executing the recipe.
    The semantics of this option are the same as ``environment-section``. If
    both ``environment-section`` and ``environment`` are provided the values from
    the former will be overridden by the latter allowing per-part customization.
    Additionally, the recipe honors the ``download-cache`` option set Additionally, the recipe honors the ``download-cache`` option set
    in the ``[buildout]`` section and stores the downloaded files under in the ``[buildout]`` section and stores the downloaded files under
    ...@@ -444,17 +409,13 @@ with a new buildout and provide more options. ...@@ -444,17 +409,13 @@ with a new buildout and provide more options.
    ... newest = false ... newest = false
    ... parts = package ... parts = package
    ... ...
    ... [build-environment]
    ... CFLAGS = -I/sw/include
    ... LDFLAGS = -I/sw/lib
    ...
    ... [package] ... [package]
    ... recipe = slapos.recipe.cmmi ... recipe = slapos.recipe.cmmi
    ... url = file://%s ... url = file://%s
    ... md5sum = 6b94295c042a91ea3203857326bc9209 ... md5sum = 6b94295c042a91ea3203857326bc9209
    ... prefix = /somewhere/else ... prefix = /somewhere/else
    ... environment-section = build-environment
    ... environment = ... environment =
    ... CFLAGS=-I/sw/include
    ... LDFLAGS=-L/sw/lib -L/some/extra/lib ... LDFLAGS=-L/sw/lib -L/some/extra/lib
    ... configure-options = ... configure-options =
    ... --with-threads ... --with-threads
    ...@@ -467,16 +428,16 @@ with a new buildout and provide more options. ...@@ -467,16 +428,16 @@ with a new buildout and provide more options.
    ... patches/Makefile.dist.patch ... patches/Makefile.dist.patch
    ... """ % package_path) ... """ % package_path)
    This configuration uses custom configure options, an environment section, This configuration uses custom configure options, environment variables,
    per-part customization to the environment, custom prefix, multiple make custom prefix, multiple make targets and also patches the source code
    targets and also patches the source code before the scripts are run. before the scripts are run.
    >>> print(system(buildout)) >>> print(system(buildout))
    Uninstalling package. Uninstalling package.
    Installing package. Installing package.
    package: Applying patches
    package: [ENV] CFLAGS = -I/sw/include package: [ENV] CFLAGS = -I/sw/include
    package: [ENV] LDFLAGS = -L/sw/lib -L/some/extra/lib package: [ENV] LDFLAGS = -L/sw/lib -L/some/extra/lib
    package: Applying patches
    patching file configure patching file configure
    patching file Makefile.dist patching file Makefile.dist
    patched-configure --prefix=/somewhere/else --with-threads --without-foobar patched-configure --prefix=/somewhere/else --with-threads --without-foobar
    ...@@ -847,166 +808,6 @@ replaced with the recipe final prefix. ...@@ -847,166 +808,6 @@ replaced with the recipe final prefix.
    package: Executing pre-install package: Executing pre-install
    installing package at /sample_buildout/parts/package -lib installing package at /sample_buildout/parts/package -lib
    Install shared package
    ======================
    Use option ``shared`` to install a shared package.
    >>> import subprocess
    >>> shared_dir = tmpdir('shared')
    >>> another_shared_dir = tmpdir('another_shared_dir')
    >>> __tear_downs.insert(0, lambda: subprocess.call(
    ... ('chmod', '-R', 'u+w', shared_dir, another_shared_dir)))
    If no ``shared-part-list`` is set, and ``shared`` is True, ``shared`` feature
    is not used.
    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... newest = false
    ... parts = package
    ...
    ... [package]
    ... recipe = slapos.recipe.cmmi
    ... url = file://%s
    ... shared = True
    ... """ % package_path)
    >>> print(system(buildout)) #doctest:+ELLIPSIS
    Uninstalling package.
    Installing package.
    configure --prefix=/sample_buildout/parts/package
    building package
    installing package
    If ``shared-part-list`` is set, shared is True, and build package fails, the
    part location is left for debugging.
    Also a shell script with the environment variable is created, so that
    developer can try same build process as the recipe tried.
    >>> os.rename(package_path, package_path + '.bak')
    >>> import tarfile
    >>> from io import BytesIO
    >>> import sys
    >>> with tarfile.open(package_path, 'w:gz') as tar:
    ... configure = b'invalid'
    ... info = tarfile.TarInfo('configure.off')
    ... info.size = len(configure)
    ... info.mode = 0o755
    ... tar.addfile(info, BytesIO(configure))
    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... newest = false
    ... parts = package
    ... shared-part-list = %s
    ...
    ... [package]
    ... recipe = slapos.recipe.cmmi
    ... url = file://%s
    ... shared = True
    ... environment =
    ... FOO=bar
    ... """ % (shared_dir, package_path))
    >>> print(system(buildout)) #doctest:+ELLIPSIS
    package: shared at /shared/package/<MD5SUM:0>
    Uninstalling package.
    Installing package.
    package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
    package: [ENV] FOO = bar
    package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
    package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>/.build where you can inspect what went wrong.
    A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
    /bin/sh: 1: ./configure: not found
    While:
    Installing package.
    Error: System error
    >>> import glob
    >>> cat(glob.glob(join(shared_dir, 'package/*/.build/slapos.recipe.build.env.sh'))[0])
    export FOO=bar
    ...
    Next time buildout runs, it detects that the build failed, remove the compile dir and retry.
    >>> print(system(buildout)) #doctest:+ELLIPSIS
    package: shared at /shared/package/<MD5SUM:0>
    Installing package.
    package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
    package: [ENV] FOO = bar
    package: Removing already existing directory /shared/package/<MD5SUM:0>
    package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
    package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>/.build where you can inspect what went wrong.
    A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
    /bin/sh: 1: ./configure: not found
    While:
    Installing package.
    Error: System error
    If ``shared-part-list`` is set as an option in buildout section and
    ``shared`` is True, package will be installed in shared_part/package
    and a hash of the recipe's configuration options.
    There can be multiple path listed in ``shared-part-list``, the recipe
    will look in each of these paths if package was already installed and
    if not, it will install the package in the last entry.
    >>> os.rename(package_path + '.bak', package_path)
    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... newest = false
    ... parts = package
    ... shared-part-list =
    ... %s
    ... not/exists
    ... %s
    ...
    ... [package]
    ... recipe = slapos.recipe.cmmi
    ... url = file://%s
    ... shared = True
    ... environment =
    ... FOO=bar
    ... """ % (shared_dir, another_shared_dir, package_path))
    >>> print(system(buildout)) #doctest:+ELLIPSIS
    package: shared at /shared/package/<MD5SUM:0>
    Installing package.
    package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
    package: [ENV] FOO = bar
    package: Removing already existing directory /shared/package/<MD5SUM:0>
    configure --prefix=/shared/package/<MD5SUM:0>
    building package
    installing package
    If options change, reinstall in different location:
    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... newest = false
    ... parts = package
    ... shared-part-list =
    ... %s
    ... not/exists
    ... %s
    ...
    ... [package]
    ... recipe = slapos.recipe.cmmi
    ... url = file://%s
    ... shared =True
    ... change = True
    ... """ % (shared_dir, another_shared_dir, package_path))
    >>> print(system(buildout)) #doctest:+ELLIPSIS
    package: shared at /another_shared_dir/package/<MD5SUM:1>
    Uninstalling package.
    Installing package.
    package: Checking whether package is installed at shared path: /another_shared_dir/package/<MD5SUM:1>
    configure /another_shared_dir/package/<MD5SUM:1>
    building package
    installing package
    For even more specific needs you can write your own recipe that uses For even more specific needs you can write your own recipe that uses
    ``slapos.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``true``. ``slapos.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``true``.
    You can then continue from where this recipe finished by reading the location You can then continue from where this recipe finished by reading the location
    ......
    slapos/recipe/cmmi/__init__.py
    View file @ deda4108
    This diff is collapsed.
    Markdown is supported
    0% or
    You are about to add 0 people to the discussion. Proceed with caution.
    Finish editing this message first!
    Please register or to comment
    GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7