Document recent development

README.rst

@@ -437,6 +437,14 @@ The default git command is `git`, if for a any reason you don't
 have git in your path, you can specify git binary path with `git-command`
 option.
 
+Sparse checkout
+~~~~~~~~~~~~~~~
+
+The value of the `sparse-checkout` option is written to the
+``$GITDIR/info/sparse-checkout`` file, which is used to populate the working
+directory sparsely. See the `SPARSE CHECKOUT` section of ``git-read-tree``
+command. This feature is disabled if the value is empty or unset.
+
 Ignore SSL certificate
 ~~~~~~~~~~~~~~~~~~~~~~
 
@@ -560,3 +568,270 @@ Specific version
   modules =
     express@1.0.2
   node = node-0.6
+
+==========================
+ slapos.recipe.build:vm.*
+==========================
+
+This is a set of recipes to build Virtual Machine images and execute commands
+inside them. They rely on QEMU and OpenSSH: executables are found via the PATH
+environment variable. They do nothing on update.
+
+Common options
+--------------
+
+location
+    Folder where the recipe stores any produced file.
+    Default: ${buildout:parts-directory}/<section_name>
+
+environment
+    Extra environment for the spawn executables. It can either be the name of a
+    section or a list of variables (1 per line, in the form ``key=value``).
+    Values are expanded with current environment using Python %-dict formatting.
+
+mem
+    Python expression evaluating to an integer that specifies the
+    RAM size in MB for the VM.
+
+smp
+    Number of CPUs for the VM. Default: 1
+
+Example
+~~~~~~~
+
+::
+
+  [vm-run-environment]
+  PATH = ${openssh:location}/bin:${qemu:location}/bin:%(PATH)s
+
+  [vm-run-base]
+  recipe = slapos.recipe.build:vm.run
+  environment = vm-run-environment
+  mem = 256 * (${:smp} + 1)
+  smp = 4
+
+slapos.recipe.build:vm.install-debian
+-------------------------------------
+
+Install Debian from an ISO image. Additional required binaries:
+
+- ``7z`` (from 7zip), to extract kernel/initrd from the ISO;
+- ``file``, which is used to test that the VM image is bootable.
+
+Currently, it only produces `raw` images, in `discard` mode (see ``-drive``
+QEMU option): combined the use of ``discard`` mount option, this minimizes
+the used space on disk.
+
+If the ``ssh`` package is installed, which is required for
+`slapos.recipe.build:vm.run`_, an SSH key is automatically created with
+``ssh-keygen``, and it will be used to connect as `root` user.
+
+Options
+~~~~~~~
+
+location
+    Produced files: ``<dist>.img`` (1 for each token of `dists`), ``passwd``
+    and optionally ``ssh.key``
+
+arch
+    QEMU architecture (the recipe runs the ``qemu-system-<arch>`` executable).
+    It is also used to select the ISO in the sections refered by `dists`.
+    Default to host architecture.
+
+dists
+    List of VMs to build: each token refers to a buildout section name that
+    describes the ISOs to use. See `ISO sections`_ below.
+
+size
+    Size of the VM image. This must be an integer, optionally followed by a
+    IEC or SI suffix.
+
+mem
+    Default: 256
+
+preseed.<preseed>
+    Set the <preseed> value for the installation. The recipe has many default
+    preseed values: you can see the list in the ``InstallDebianRecipe.preseed``
+    class attribute (file ``slapos/recipe/vm.py``). Note however that:
+
+    - aliases are not recognized so you must check this list if you want to
+      override any default value;
+    - all values are passed via the kernel command line, so beware complex
+      values (double-quotes are automatically added when there are spaces).
+
+debconf.<owner>
+    List of debconf value for <owner> (usually a package name),
+    each line with 2 whitespace-separated parts: <key> <value>.
+
+late-command
+    Shell commands to execute at the end of the installation. They are run
+    inside the target system. This is a reliable alternative to the
+    ``preseed.preseed/late_command`` option.
+
+packages
+    Extra packages to install. Defaults to ``ssh``.
+    Like for `late-command`, do not use ``preseed.pkgsel/include``.
+
+ISO sections
+~~~~~~~~~~~~
+
+<arch>.iso
+    Name of the section that provides the ISO image, for example by downloading
+    it. This section must define 2 options: `location` is the folder
+    containing the ISO, and `filename` is the file name of the ISO.
+
+<arch>.kernel
+    Path to kernel image inside the ISO.
+
+<arch>.initrd
+    Path to initrd image inside the ISO.
+
+Example
+~~~~~~~
+
+::
+
+  [vm-install-environment]
+  # vm-run-environment refers to the section in common options
+  PATH = ${file:location}/bin:${p7zip:location}/bin:${vm-run-environment:PATH}
+
+  [vm-debian]
+  recipe = slapos.recipe.build:vm.install-debian
+  environment = vm-install-environment
+  dists = debian-jessie debian-stretch
+  size = 2Gi
+  late-command =
+  # rdnssd causes too much trouble with QEMU 2.7, because the latter acts as
+  # a DNS proxy on both IPv4 and IPv6 without translating queries to what the
+  # host supports.
+    dpkg -P rdnssd
+  debconf.debconf =
+    debconf/frontend noninteractive
+    debconf/priority critical
+  # minimal size
+  preseed.recommends = false
+  preseed.tasks =
+  packages = localepurge ssh
+
+  [debian-jessie]
+  x86_64.iso = debian-amd64-netinst.iso
+  x86_64.kernel = install.amd/vmlinuz
+  x86_64.initrd = install.amd/initrd.gz
+
+  [debian-stretch]
+  <= debian-jessie
+  x86_64.iso = debian-amd64-testing-netinst.iso
+
+  [debian-amd64-netinst.iso]
+  ...
+
+slapos.recipe.build:vm.run
+--------------------------
+
+Execute shell commands inside a VM, in snapshot mode (the VM image is not
+modified).
+
+``${buildout:directory}`` is always mounted as `/mnt/buildout` inside the VM.
+
+Options
+~~~~~~~
+
+location
+    Folder where to store any produce file. Inside the guest, it is pointed to
+    by the PARTDIR environment variable. It is also used as temporary storage
+    for changes to the VM image.
+
+vm
+    Folder containing the VM images and the `ssh.key`` file. See the `location`
+    option of the `vm.install-*` recipes.
+
+dist
+    VM image to use inside the `vm` folder.
+
+commands
+    List of <command> options, each one being a shell script to execute via
+    SSH. They are processed in sequence. This is usually only required if you
+    want to reboot the VM. Default: command
+
+mount.<name>
+    Extra mount point. The value is a host folder that is mounted as
+    ``/mnt/<name>``.
+
+stop-ssh
+    Tell `reboot` function how to stop SSH (see Helpers_).
+    Default: systemctl stop ssh
+
+wait-ssh
+    Time to wait for (re)boot. The recipe fails if it can't connect to the SSH
+    server after this number of seconds. Default: 60
+
+Helpers
+~~~~~~~
+
+Before commands are executed, all `mount.<name>` are mounted
+and a few helpers are set to make scripting easier.
+
+set -e
+    This is done before anything else, to make buildout abort if any untested
+    command fails.
+
+reboot
+    Function to safely reboot the guest. The next command in `commands` will be
+    executed once the SSH server is back.
+
+map <host_path>
+    Function to map a folder inside ``${buildout:directory}``.
+
+PARTDIR
+    Folder where to store any produced file. Inside the guest, it actually
+    maps to `location` on the host. This is useful because you can't write
+    ``PARTDIR=`map ${:location}``` if you don't explicitly set `location`.
+
+Example
+~~~~~~~
+
+::
+
+  [vm-run-base]
+  # extends above example in common options
+  vm = ${vm-debian:location}
+  dist = debian-jessie
+
+  [vm-debian]
+  # extends above example in vm.install-debian
+  packages += build-essential git equivs
+
+  [userhosts-repository]
+  recipe = slapos.recipe.build:gitclone
+  repository = https://lab.nexedi.com/nexedi/userhosts.git
+  # we don't need a working directory on the host
+  sparse-checkout = /.gitignore
+
+  [build-userhosts-map]
+  <= vm-run-base
+  repository = `map ${userhosts-repository:location}`
+  command =
+    git clone -s ${:repository} /userhosts
+    cd /userhosts
+    mk-build-deps -irt 'apt-get -y'
+    dpkg-buildpackage -uc -b -jauto
+    cd ..
+    mv *.changes *.deb $PARTDIR
+
+  # Alternate way, which is required if [userhosts-repository] is extended
+  # in such way that the repository is outside ${buildout:directory}.
+  [build-userhosts-mount]
+  <= build-userhosts-map
+  mount.userhosts = ${userhosts-repository:location}
+  repository = /mnt/userhosts
+
+  [test-reboot]
+  <= vm-run-base
+  commands = hello world
+  hello =
+    uptime -s
+    echo Hello ...
+    reboot
+  world =
+    uptime -s
+    echo ... world!