boost-lib: enable parallel build

component/boost-lib/buildout.cfg

@@ -13,9 +13,9 @@ url = http://downloads.sourceforge.net/sourceforge/boost/boost_1_67_0.tar.bz2
 md5sum = ced776cb19428ab8488774e1415535ab
 location = @@LOCATION@@
 configure-command = ./bootstrap.sh --prefix=${:location} --with-python=${python2.7:location}/bin/python2.7
-make-binary =
-make-options =
-make-targets = ./b2 link=shared dll-path=${:location}/lib:${bzip2:location}/lib:${gcc:location}/lib:${gcc:location}/lib64:${zlib:location}/lib install
+make-binary = ./b2
+# XXX: we assume that $MAKEFLAGS is only used for -j
+make-options = $MAKEFLAGS link=shared dll-path=${:location}/lib:${bzip2:location}/lib:${gcc:location}/lib:${gcc:location}/lib64:${zlib:location}/lib
 environment =
   PATH=${gcc:location}/bin:%(PATH)s
   BZIP2_INCLUDE=${bzip2:location}/include

[Jérome Perrin]

We should not assume this. Testnodes uses something like MAKEFLAGS="-j10 -l10" (here).

That's also what I use and it seem to cause a strange error like this:

...
gcc.compile.c++ bin.v2/libs/wave/build/gcc-5.5.0/release/threading-multi/instantiate_re2c_lexer_str.o
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
10 second time limit exceeded
...

If I change to MAKEFLAGS="-j10 -l4", it fails with:

...
4 second time limit exceeded
...

so it seems -l apply a time limit to commands and that must be the cause of the failures we have sometimes with boost on test nodes.

I don't see -l option being documented in https://boostorg.github.io/build/manual/master/index.html, but I see:

-j N

Run up to N commands in parallel. Default number of jobs is the number of detected available CPU threads. Note: There are circumstances when that default can be larger than the allocated cpu resources, for instance in some virtualized container installs.

I have not verified if this is true. I'm not even sure this doc applies to this ./b2 we are using here, but I guess we better not pass $MAKEFLAGS like this.

[Ivan Tyagov]

@jerome, might be wrong but I observed a case when cc1 was OOM killed with same message (while compiling groonga in the standalone tests inside a 2GB RAM VM). Might that be the reason (you can check kernel's logs) ?

[Julien Muchembled]

The b2 documentation is wrong (at least for 1.67.0): the default is 1. I also can't find any documentation for -l. Another strange thing is that 'b2' eats 100% CPU even without parallel build.

While debugging, I found a slapos.recipe.cmmi bug: __buildout_installed__ is empty for shared parts.

[Jérome Perrin]

Thanks for feedback and for the fix in 4cd0606c

@Tyagov yes, I should have mentioned it, I first suspected something like OOM, but this was not in dmesg.

@jm maybe this doc does not apply for the ./b2 script. Anyway this should be OK on master now.

While debugging, I found a slapos.recipe.cmmi bug: __buildout_installed__ is empty for shared parts.

I'm not sure we're talking about the same thing, but if a part is installed in shared mode, the software which installed the part does not "own" the part and should not uninstall that part, because other softwares might also be using it. The only way to uninstall shared part is slapos node prune command that was added recently.

[Julien Muchembled]

if a part is installed in shared mode, the software which installed the part does not "own" the part and should not uninstall that part

Oops, yes, of course. The problem I encountered is that I manually deleted a shared part and expected it to be rebuilt. A missing __buildout_installed__ path is enough to trigger a reinstallation, but since __buildout_installed__ has to be empty, we'd need some other mechanism.

[Julien Muchembled]

Oh, I've just remembered about zc.buildout.uninstall entry point. We could specify a function that does nothing if __buildout_installed__ points to a shared location.

[Jérome Perrin]

The problem I encountered is that I manually deleted a shared part and expected it to be rebuilt.

Maybe it works only if the shared part is directly referenced. If for example we have curl component which depends on zlib, and an erp5 software which uses curl:

• if we delete curl and reinstall erp5 software, curl part should be reinstalled, because the directory is missing, slapos.recipe.cmmi's init should detect this
• if we delete zlib and reinstall erp5 software, zlib part will not be reinstalled, because erp5 only depends on curl, and curl looks installed, the directory is present.

This __buildout_installed__ trick could work, but not in all cases I think. If installing erp5 installed curl and zlib, they will be in erp5 software's .installed.cfg, but not in other softwares. Later if we delete zlib and reinstall another software using curl (for example cloudooo), then zlib would not be seen as missing. Or maybe we can find a way to write dependent parts in __buildout_installed__.

I prefer if we can say that this use case is not supported, but maybe we could implement this use case with a script that would delete a shared part and all shared parts depending on it (or maybe slapos node prune could detect this and "repair" by deleting "orphean" parts).

[Julien Muchembled]

Buildout does not work like this. It always establishes the full graph of dependencies, regardless parts are installed or not. In your example of ERP5 SR where zlib is deleted but not curl, both are uninstalled (the first one because of missing path, the second because of uninstalled dependency) and installed again if __buildout_installed__ was set.

The only strange thing with shared parts is that the installation of curl would be a no-op, even though zlib was rebuilt. In theory, this should work because builds are supposed to be reproducible (well, modulo non-important things like profile-guided optimization).

[Jérome Perrin]

Yes, you're right, if we make slapos.recipe.cmmi always return shared part location, not only on install but also when it detects the part was already installed (this was what I missed), it should be enough. About the zc.buildout.uninstall entry point, we are using it in slapos.recipe.build:gitclone (here) to prevent buildout from deleting a git repository with local changes.