Commit 0bb65331 authored by Jérome Perrin's avatar Jérome Perrin

shared: Fix recovery after an interrupted build

When build process is interrupted (like when the buildout process
terminated by SIGKILL), building with shared leave the system in a state
that looks like installation succeeded, so next execution will assume
the part was properly installed.


The behavior was something like this:


    if isSharedPartSignatureOK():
      useThisAlreadyInstalledPart()
    else:
      writeSignatureInSharedPart()
      try:
        build()
      except:
        removeSharedPart()
        raise


and when program is terminated during build step, the signature is
there.

The fix is to write signature later, algorithm becomes:


    if isSharedPartSignatureOK():
      useThisAlreadyInstalledPart()
    else:
      try:
        build()
      except:
        removeSharedPart()
        raise
      writeSignatureInSharedPart()

/reviewed-on nexedi/slapos.recipe.cmmi!10
parent dc5b5bac
......@@ -1070,14 +1070,64 @@ developer can try same build process as the recipe tried.
export FOO="bar"
...
If ``shared-part-list`` is set as an option in buildout section and
``shared`` is True, package will be installed in shared_part/package
and a hash of the recipe's configuration options.
Next time buildout runs, it detects that the build failed, remove the compile dir and retry.
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared directory /shared/package/FIRST_SHARED_PACKAGE_HASH set for package
Installing package.
package: Checking whether package is installed at shared path: /shared/package/FIRST_SHARED_PACKAGE_HASH
package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/FIRST_SHARED_PACKAGE_HASH__compile__
package: Command './configure --prefix="/shared/package/FIRST_SHARED_PACKAGE_HASH"' returned non-zero exit status 127.
package: Compilation error. The package is left as is at /shared/package/FIRST_SHARED_PACKAGE_HASH__compile__ where you can inspect what went wrong.
A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
/bin/sh: 1: ./configure: not found
While:
Installing package.
Error: System error
But we had a bug with version v0.11 is that if build was interrupted in the middle of the build of a
shared part. The shared part was left in an inconsistent state that looked like installation succeeded.
Let's simulate a scenario where buildout is terminated in the middle of a build.
For this, we need a package that takes a lot of time to install.
>>> tarpath = os.path.join(src, 'package-0.0.0.tar.gz')
>>> with tarfile.open(tarpath, 'w:gz') as tar:
... configure = b'#!/bin/sh\necho configure started\nsleep 300;'
... info = tarfile.TarInfo('configure')
... info.size = len(configure)
... info.mode = 0o755
... tar.addfile(info, BytesIO(configure))
We also need a bit more complex method to run buildout so that we send a termination signal in the
middle of build process.
>>> import subprocess
>>> import signal
>>> import time
>>> buildout_process = subprocess.Popen(buildout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
>>> output_line = ''
>>> for _ in range(10):
... time.sleep(1)
... output_line = buildout_process.stdout.readline()
... if output_line and b'configure started' in output_line:
... print ('configure started')
... buildout_process.send_signal(signal.SIGKILL)
... print ('buildout terminated')
... break
configure started
buildout terminated
>>> _ = buildout_process.wait()
If we run buildout again, the compile dir is removed again and installation is retried. This time
installation can succeed.
This was not the case in version 0.11 of this recipe. If installation of a shared part failed, this was
not detected and the part was considered as installed.
There can be multiple path listed in ``shared-part-list``, the recipe
will look in each of these paths if package was already installed and
if not, it will install the package in the last entry the last entry
from the list of ``shared-part-list``.
>>> _ = system('mv %s/package-0.0.0.tar.gz.bak %s/package-0.0.0.tar.gz' % (src, src))
>>> print(system(buildout)) #doctest:+ELLIPSIS
......@@ -1090,6 +1140,17 @@ from the list of ``shared-part-list``.
building package
installing package
If ``shared-part-list`` is set as an option in buildout section and
``shared`` is True, package will be installed in shared_part/package
and a hash of the recipe's configuration options.
There can be multiple path listed in ``shared-part-list``, the recipe
will look in each of these paths if package was already installed and
if not, it will install the package in the last entry the last entry
from the list of ``shared-part-list``.
If package was already installed in any of the ``shared-part-list`` used, it will be
used instead of installing if one package has been installed.
......
......@@ -332,9 +332,6 @@ class Recipe(object):
os.mkdir(location)
os.chdir(compile_dir)
if self.options['shared']:
self._signature.save(self.options["shared"])
try:
try:
# We support packages that either extract contents to the $PWD
......@@ -420,6 +417,9 @@ class Recipe(object):
finally:
os.chdir(current_dir)
if self.options['shared']:
self._signature.save(self.options["shared"])
# Check promises
self.check_promises(log)
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment