OBS: Build slapos-node using either python2 or python3

This commit also simplifies the preparation of the cache for OBS compilation.

The slapos-node package can be built in any Linux distribution (old or
new).

WARNING: Do not use PYTHON make variable since it messes up with compilation. Instead
use SLAPOS_BOOTSTRAP_SYSTEM_PYTHON which explains the idea of the variable
and does not overrides make's PYTHON variable in other places.

obs/slapos/README

@@ -19,7 +19,7 @@ Requirements
 ------------
 - A computer
 - Internet connection
-- gcc, g++, make, patch, wget, python, chrpath, osc, devscripts
+- gcc, g++, make, patch, wget, python3, chrpath, osc, devscripts
 - An account on OBS
 
 Documentation
@@ -31,20 +31,20 @@ Documentation
 Directory Organisation
 ----------------------
 
-- prepare_slapos_build.sh : This is the script you have to run to prepare the packet.
-  You can choose the version you want to build in it.
+- Makefile : makefile containing all the commands to prepare the packet.
+- release_configuration.sh : script defining the version of slapos-node
+  package. Version is automaticallmy detected but you can decide to override
+  it.
 - ./home:VIFIBnexedi:branches:home:VIFIBnexedi/SlapOS-Node : This is the directory for OBS.
   It is the one used to generate unreleased package and testing it.
-  If you have/want to use another directory, update its path in prepare_slapos_build.sh
+  If you have/want to use another directory, update its path in release_configuration.sh
 - templates/ : Contain templates to build the package.
-  	     - debian/ : contain configuration files for .deb packaging
 	     - Makefile.in : Makefile to build slapos
-	     - offline.sh.in : Prepare network-cache build building once slapos
 	     - slapos.dsc.in : File needed by obs to build .deb package
 	     - slapos.spec.in : configuration file for rpm packaging
+- debian/ : contain configuration files for .deb packaging
 - slapos-node/ : Template to build package
   	       - Makefile : Makefile to build slapos and add files for package
-	       - template/ : template files for package (cron.d, service)
 	       - slapos/ : to build slapos. Contain buildout.cfg.in
 
 
@@ -59,15 +59,15 @@ You need in particular an OBS directory (for example, the vifib test directory):
   $ cd home:VIFIBnexedi:branches:home:VIFIBnexedi/SlapOS-Node
   $ osc up
 
-SlapOS is build using buildout and need an internet connection to fetch all elements.
-On obs package are build on virtual machines without internet access.
-prepare_slapos_build.sh will compile slapos on your computer and build a network-cache thanks to offline.sh script.
+SlapOS is built using buildout and need an internet connection to fetch all elements.
+On OBS, packages are built on virtual machines without internet access.
+Makefile will compile slapos on your computer and build a network-cache thanks to prepare_download_cache.sh script.
 At the end it remove slapos installation and only keep cache.
-It makes a tarball out of it to be used by obs.
-It then fetch packaging configuration from templates directory. Everything is then commit to obs thanks to osc.
+It makes a tarball out of it to be used by OBS.
+It then fetch packaging configuration from templates directory. Everything is then commit to OBS thanks to osc.
 
 So to build package just run:
-# bash prepare_slapos_build.sh
+# make all
 
 About cron file
 ---------------

obs/slapos/debian/control

@@ -4,7 +4,7 @@ Section: net
 Priority: optional
 Build-Depends: debhelper,
                chrpath,
-               python
+               python3 (>=3.7) | python
 Homepage: https://slapos.nexedi.com
 
 Package: slapos-node

obs/slapos/prepare_download_cache.sh

 #!/bin/bash
 
+set -e
+
 source release_configuration.sh
 
 ##########
@@ -9,7 +11,7 @@ source release_configuration.sh
 TARGET_DIRECTORY=/opt/slapos
 BUILD_ROOT_DIRECTORY="$CURRENT_DIRECTORY/$SLAPOS_DIRECTORY/slapos/build"
 BUILD_DIRECTORY=$BUILD_ROOT_DIRECTORY$TARGET_DIRECTORY
-BUILDOUT_VERSION="2.5.2+slapos013"
+BUILDOUT_VERSION="2.7.1+slapos019"
 
 rm -rf $BUILD_ROOT_DIRECTORY
 
@@ -17,63 +19,77 @@ echo "Preparing source tarball (recipe version: $RECIPE_VERSION)"
 echo " Build Directory: $BUILD_DIRECTORY "
 echo " Buildroot Directory: $BUILD_ROOT_DIRECTORY "
 
-mkdir -p $BUILD_DIRECTORY/{eggs,extends-cache,download-cache,download-cache/dist,tmp-networkcached/eggs}
+########################################################
+# build the package once keeping every source in cache #
+########################################################
 
-set -e
+mkdir -p $BUILD_DIRECTORY/{eggs,extends-cache,download-cache/dist}
+cd $BUILD_DIRECTORY
 
+# 1) boostrap with old version
+echo "bootsrapping buildout"
+sed  "s/\%RECIPE_VERSION\%/$RECIPE_VERSION/g;s|\%PATCHES_DIRECTORY\%|$PATCHES_DIRECTORY|g;s|\%TARGET_DIRECTORY\%|$TARGET_DIRECTORY|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_ROOT_DIRECTORY|g;s|\%BUILD_DIRECTORY\%|$BUILD_DIRECTORY|g" $BUILD_ROOT_DIRECTORY/../slapos.buildout.cfg.in > buildout.cfg
+
+wget https://lab.nexedi.com/nexedi/slapos.buildout/raw/master/bootstrap/bootstrap.py
+python3 -S bootstrap.py \
+  --setuptools-version 40.8.0 \
+  --setuptools-to-dir eggs
+
+# 2) get newest version of zc.buildout and setuptools
+#    note that we can't directly do setuptools + zc.buildout +
+#    slapos.libnetworkcache because buildout would be relaunched in the middle
+#    without the "-S" option to python
+echo "downloading good version of setuptools and zc.buildout"
+sed -i '1s/$/ -S/' bin/buildout
+sed -i "/def _satisfied(/s/\(\bsource=\)None/\11/" eggs/zc.buildout-*/zc/buildout/easy_install.py # no wheel
+bin/buildout buildout:newest=true -v
+ls download-cache/dist/*.whl && { echo "There shouldn't be any wheel in download-cache" ; exit 1 ; }
+
+# 3) compile very simple buildout with networkcache
 echo "Preparing networkcached zc.buildout"
-NETWORKCACHED_DIRECTORY=$BUILD_DIRECTORY/tmp-networkcached
-sed  "s/\%RECIPE_VERSION\%/$RECIPE_VERSION/g;s|\%PATCHES_DIRECTORY\%|$PATCHES_DIRECTORY|g;s|\%TARGET_DIRECTORY\%|$TARGET_DIRECTORY|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_ROOT_DIRECTORY|g;s|\%BUILD_DIRECTORY\%|$BUILD_DIRECTORY|g" $BUILD_ROOT_DIRECTORY/../networkcached.cfg.in > $NETWORKCACHED_DIRECTORY/buildout.cfg
-cd $NETWORKCACHED_DIRECTORY
-# Download  bootstrap file
-wget https://bootstrap.pypa.io/bootstrap-buildout.py -O bootstrap.py
-(python2.7 -S bootstrap.py \
-  --buildout-version $BUILDOUT_VERSION \
-  --setuptools-version 44.1.1 \
-  --setuptools-to-dir eggs \
-  -f http://www.nexedi.org/static/packages/source/slapos.buildout/ && \
-./bin/buildout -v)
-
-# build locally everything with gcc
-sed  "s/\%RECIPE_VERSION\%/$RECIPE_VERSION/g;s|\%PATCHES_DIRECTORY\%|$PATCHES_DIRECTORY|g;s|\%TARGET_DIRECTORY\%|$TARGET_DIRECTORY|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_ROOT_DIRECTORY|g;s|\%BUILD_DIRECTORY\%|$BUILD_DIRECTORY|g" $BUILD_ROOT_DIRECTORY/../buildout_with_gcc.cfg.in > $BUILD_DIRECTORY/buildout.cfg
-
-# Build first time to get download-cache and extends-cache ready
-cd $BUILD_DIRECTORY
+sed  "s/\%RECIPE_VERSION\%/$RECIPE_VERSION/g;s|\%PATCHES_DIRECTORY\%|$PATCHES_DIRECTORY|g;s|\%TARGET_DIRECTORY\%|$TARGET_DIRECTORY|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_ROOT_DIRECTORY|g;s|\%BUILD_DIRECTORY\%|$BUILD_DIRECTORY|g" $BUILD_ROOT_DIRECTORY/../networkcached.cfg.in > buildout.cfg
+sed -i '1s/$/ -S/' bin/buildout
+bin/buildout buildout:newest=true -v
 
-echo "$BUILD_ROOT_DIRECTORY" > $CURRENT_DIRECTORY/$SLAPOS_DIRECTORY/slapos/original_directory
+# 4) build locally everything with gcc to get download-cache and extends-cache ready
+echo "Launch the big buildout to compile everything"
+sed  "s/\%RECIPE_VERSION\%/$RECIPE_VERSION/g;s|\%PATCHES_DIRECTORY\%|$PATCHES_DIRECTORY|g;s|\%TARGET_DIRECTORY\%|$TARGET_DIRECTORY|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_ROOT_DIRECTORY|g;s|\%BUILD_DIRECTORY\%|$BUILD_DIRECTORY|g" $BUILD_ROOT_DIRECTORY/../buildout_with_gcc.cfg.in > buildout.cfg
 
-($NETWORKCACHED_DIRECTORY/bin/buildout bootstrap  --buildout-version $BUILDOUT_VERSION \
-                        --setuptools-to-dir eggs \
-                        -f http://www.nexedi.org/static/packages/source/slapos.buildout/ && \
-    ./bin/buildout -v) || (./bin/buildout -v || (echo "Failed to run buildout, exiting." && exit 1))
+bin/buildout buildout:newest=true -v
+
+###################################################
+# remove all files from build keeping only caches #
+###################################################
 
-# remove all files from build keeping only caches
 echo "Deleting unecessary files to reduce source tarball size"
 
 # TODO: Figure out why there is no write permission even for
 #       the owner
 chmod -R u+w .
 
-cp -R eggs/slapos.rebootstrap* $BUILD_ROOT_DIRECTORY/..
-
 rm -fv .installed.cfg environment.*
 rm -rfv ./{downloads,parts,eggs,develop-eggs,bin,rebootstrap}
 
 # Removing empty directories
 find . -type d -empty -prune -exec rmdir '{}' ';'
 
-mkdir -p $BUILD_DIRECTORY/eggs
-$NETWORKCACHED_DIRECTORY/bin/buildout bootstrap --buildout-version $BUILDOUT_VERSION \
-                       --setuptools-to-dir eggs \
-                       -f http://www.nexedi.org/static/packages/source/ \
-                       -f http://www.nexedi.org/static/packages/source/slapos.buildout/ 
-
-cp -R $BUILD_ROOT_DIRECTORY/../slapos.rebootstrap* eggs
-
 # Removing Python byte-compiled files (as it will be done upon
 # package installation) and static libraries
 find . -regextype posix-extended -type f \
 	-iregex '.*/*\.(py[co]|[l]?a|exe|bat)$$' -exec rm -fv '{}' ';'
 
+#TODO remove git files
+
+##################################
+# prepare compilation inside OBS #
+##################################
+# we need the very first bootstrap script
+cp $CURRENT_DIRECTORY/../re6st/bootstrap $BUILD_DIRECTORY
+
+# we need the original directory to do a sed inside OBS
+# TODO remove this and properly use extends-cache instead
+echo "$BUILD_ROOT_DIRECTORY" > $CURRENT_DIRECTORY/$SLAPOS_DIRECTORY/slapos/original_directory
+
 # in OBS build, don't force gcc build
 sed  "s/\%RECIPE_VERSION\%/$RECIPE_VERSION/g;s|\%PATCHES_DIRECTORY\%|$PATCHES_DIRECTORY|g;s|\%TARGET_DIRECTORY\%|$TARGET_DIRECTORY|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_ROOT_DIRECTORY|g;s|\%BUILD_DIRECTORY\%|$BUILD_DIRECTORY|g" $BUILD_ROOT_DIRECTORY/../buildout_without_gcc.cfg.in > $BUILD_DIRECTORY/buildout.cfg
+

obs/slapos/slapos-node/Makefile

@@ -2,15 +2,13 @@
 # -*- makefile -*-
 
 build: 
-	cd slapos; make build
+	make -C slapos build SLAPOS_BOOTSTRAP_SYSTEM_PYTHON=$(shell which python2.7 2> /dev/null || echo python3)
 
 all: build
 
 install: all
-	cd slapos; make install
+	make -C slapos install
 	mkdir -p $(DESTDIR)/usr/bin/
-	mkdir -p $(DESTDIR)/usr/sbin/
-	sed -i "s/\/opt\/slapos\/rebootstrap\/parts/\/opt\/slapos\/parts/g" $(DESTDIR)/opt/slapos/bin/*
 	cp $(DESTDIR)/opt/slapos/bin/slapos $(DESTDIR)/usr/bin/
 	mkdir -p $(DESTDIR)/etc/opt/slapos/
 	mkdir -p $(DESTDIR)/etc/firewalld/

obs/slapos/slapos-node/slapos/buildout_with_gcc.cfg.in

@@ -4,6 +4,8 @@ destdir = %BUILD_ROOT_DIRECTORY%
 builddir = %BUILD_DIRECTORY%
 extends =
   %BUILD_ROOT_DIRECTORY%/../slapos_repository/component/slapos/obs.cfg
+extends-cache = extends-cache
+download-cache = download-cache
 
 [gcc]
 # force usage of gcc from slapos to have it in the cache

obs/slapos/slapos-node/slapos/buildout_without_gcc.cfg.in

@@ -4,4 +4,5 @@ destdir = %BUILD_ROOT_DIRECTORY%
 builddir = %BUILD_DIRECTORY%
 extends =
   %BUILD_ROOT_DIRECTORY%/../slapos_repository/component/slapos/obs.cfg
+download-cache = download-cache
 

obs/slapos/slapos-node/slapos/networkcached.cfg.in

@@ -2,14 +2,25 @@
 rootdir = %TARGET_DIRECTORY%
 destdir = %BUILD_ROOT_DIRECTORY%
 builddir = %BUILD_DIRECTORY%
-extends =
-  %BUILD_ROOT_DIRECTORY%/../slapos_repository/component/slapos/obs.cfg
 extensions =
+extends-cache = extends-cache
+download-cache = download-cache
 parts =
   networkcached
 
+# Add location for modified non-official slapos.buildout
+find-links +=
+  http://www.nexedi.org/static/packages/source/
+  http://www.nexedi.org/static/packages/source/slapos.buildout/
+
 [networkcached]
 recipe = zc.recipe.egg
 eggs =
   slapos.libnetworkcache
   zc.buildout
+
+[versions]
+setuptools = 44.1.1
+zc.buildout = 2.7.1+slapos019
+slapos.libnetworkcache = 0.25
+zc.recipe.egg = 2.0.3+slapos003

obs/slapos/slapos-node/slapos/slapos.buildout.cfg.in

+[buildout]
+rootdir = %TARGET_DIRECTORY%
+destdir = %BUILD_ROOT_DIRECTORY%
+builddir = %BUILD_DIRECTORY%
+extensions =
+extends-cache = extends-cache
+download-cache = download-cache
+parts =
+  zc.buildout
+
+# Add location for modified non-official slapos.buildout
+find-links +=
+  http://www.nexedi.org/static/packages/source/
+  http://www.nexedi.org/static/packages/source/slapos.buildout/
+
+[zc.buildout]
+recipe = zc.recipe.egg
+eggs =
+  zc.buildout
+
+[versions]
+setuptools = 44.1.1
+zc.buildout = 2.7.1+slapos019
+zc.recipe.egg = 2.0.3+slapos003

obs/slapos/slapos-node/slapos/slapos.cfg

-[slapos]
-software_root = /opt/slapgrid
-instance_root = /srv/slapgrid
-master_url = http://127.0.0.1:5000
-computer_id = abeautifulbutuniquename
-pidfile = /var/opt/slapos/run/slapgrid.pid
-logfile = /var/opt/slapos/log/slapgrid.log
-supervisord-socket = /var/opt/slapos/run/slapgrid-supervisord.sock
-#key_file = /etc/opt/slapos/ssl/computer.key
-#cert_file = /etc/opt/slapos/ssl/computer.crt
-#certificate_repository_path = /var/opt/slapos/lib/certificates
-
-[slapformat]
-computer_xml = /var/opt/slapos/lib/computer.xml
-log_file = /var/opt/slapos/log/slapformat.log
-partition_amount = 10
-ipv4_local_network = 10.0.0.0/16
-bridge_name = slapbr0
-tap_base_name = slaptap
-partition_base_name = slappart
-user_base_name = slapuser
-
-[slapproxy]
-host = 127.0.0.1
-port = 5000
-database_uri = /var/opt/slapos/lib/proxy.db
-

obs/slapos/templates/Makefile.in

@@ -4,7 +4,6 @@
 VERSION = %VERSION%
 RECIPE_VERSION = %RECIPE_VERSION%
 
-PY = $(PYTHON)
 INSTALL_DIRECTORY = $(DESTDIR)/opt/slapos
 TARGET_DIRECTORY = /opt/slapos
 BUILD_ROOT_DIRECTORY := $(shell pwd)/build
@@ -22,9 +21,13 @@ build-stamp:
 		xargs sed -i 's#$(ORIGINAL_DIRECTORY)#$(BUILD_ROOT_DIRECTORY)#g' || \
 		echo "No path to fix."
 
+	@echo "Bootstrapping buildout"
+	cd $(BUILD_DIRECTORY) && \
+		$(SLAPOS_BOOTSTRAP_SYSTEM_PYTHON) ./bootstrap
+
 	@echo 'Preparing source tarball (recipe version: $(RECIPE_VERSION))'
 	cd $(BUILD_DIRECTORY) && \
-		$(PY) ./bin/buildout -v
+		$(SLAPOS_BOOTSTRAP_SYSTEM_PYTHON) ./bin/buildout -v
 
 	@touch build-stamp
 

obs/slapos/templates/PKGBUILD.in

-# Maintainer: Rafael Monnerat <rafael@nexedi.com>
+# Maintainer: Thomas Gambier <thomas.gambier@nexedi.com>
 pkgname=slapos-node
 pkgver=%VERSION%+%RECIPE_VERSION%+%RELEASE%
 pkgdesc="Slapos node"
@@ -6,15 +6,15 @@ pkgrel=1
 arch=('x86_64' 'i686')
 url="http://www.slapos.org"
 license=('GPL')
-depends=(python2 python2-distribute iptables iproute2)
-makedepends=(patch wget chrpath inetutils)
+depends=(iptables iproute2)
+makedepends=(patch wget chrpath inetutils python3)
 install='slapos-node.install'
 source=(${pkgname}_${pkgver}.tar.gz)
 md5sums=(%SOURCEMD5%)
 
 build() {
 	cd "${srcdir}/${pkgname}_${pkgver}"
-	make PYTHON=python2
+	make
 }
 
 package() {

obs/slapos/templates/slapos.dsc.in

@@ -3,10 +3,10 @@ Source: slapos-node
 Binary: slapos-node
 Architecture: any
 Version: %VERSION%+%RECIPE_VERSION%+%RELEASE%-1
-Maintainer: Rafael Monnerat <rafael@nexedi.com>
-Homepage: http://community.slapos.org
+Maintainer: Thomas Gambier <thomas.gambier@nexedi.com>
+Homepage: http://slapos.nexedi.com
 Standards-Version: 3.9.2
-Build-Depends: debhelper (>= 4.1.16), chrpath, po-debconf, wget, python, python-dev, libssl-dev
+Build-Depends: debhelper (>= 4.1.16), chrpath, po-debconf, wget, python3 (>= 3.7) | python, python3-dev (>= 3.7) | python-dev, libssl-dev
 Files: 
  1c2803e4693c74042985d753c9c3ba53 73736102 slapos-node_%VERSION%+%RECIPE_VERSION%+%RELEASE%.orig.tar.gz
  1c2803e4693c74042985d753c9c3ba53 6102 slapos-node_%VERSION%+%RECIPE_VERSION%+%RELEASE%-1.diff.tar.gz

obs/slapos/templates/slapos.spec.in

@@ -44,11 +44,22 @@ Group: Application/Network
 Source0: slapos-node_%{unmangled_version}+%{slapversion}+%{release_number}.tar.gz
 URL: http://community.slapos.org/ 
 Vendor: Vifib
-Packager: Rafael Monnerat <rafael@nexedi.com>
-
-BuildRequires: gcc-c++, make, patch, wget, python2, python-devel, chrpath, openssl-devel
-
-Requires: bridge-utils, python2, gcc-c++, make, patch, wget, lm_sensors
+Packager: Thomas Gambier <thomas.gambier@nexedi.com>
+
+BuildRequires: gcc-c++
+BuildRequires: make
+BuildRequires: patch
+BuildRequires: wget
+BuildRequires: chrpath
+BuildRequires: openssl-devel
+%if 0%{?centos_version} <= 700 || 0%{?rhel_version} <= 700 || 0%{?suse_version} < 1300 || 0%{?sle_version} < 130000
+BuildRequires: python
+BuildRequires: python-devel
+%else
+BuildRequires: python3
+BuildRequires: python3-devel
+%endif
+Requires: gcc-c++, make, patch, wget, lm_sensors
 
 AutoReqProv: no
 
@@ -77,7 +88,7 @@ zcat $RPM_SOURCE_DIR/slapos-node_%{unmangled_version}+%{slapversion}+%{release_n
 
 %build
 cd $RPM_BUILD_DIR/slapos-node_%{unmangled_version}+%{slapversion}+%{release_number}
-make PYTHON=python2
+make
 
 %install
 cd $RPM_BUILD_DIR/slapos-node_%{unmangled_version}+%{slapversion}+%{release_number}/
@@ -89,14 +100,6 @@ cp $RPM_BUILD_DIR/slapos-node_%{unmangled_version}+%{slapversion}+%{release_numb
 mkdir -p $RPM_BUILD_ROOT/etc/systemd/system/
 mkdir -p $RPM_BUILD_ROOT/opt/slapos/log
 rm -rf $RPM_BUILD_ROOT/opt/slapos/eggs/setuptools-*.egg/setuptools/tests/test_easy_install.py
-rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/test_httpservers.py
-rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/badsyntax_nocaret.py
-rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/badsyntax_future*.py
-rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/bad_*.py
-rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/lib2to3/tests/data/py3_test_grammar.py
-rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/test_grammar.py
-rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/test_compile.py
-rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/_mock_backport.py
 
 %files
 /opt/slapos