We provide neotest SR which builds current NEO/go together with the
tools which are needed to run the benchmarks.

The SR does not yet provide automated service to run the tests
automatically periodically and to upload the results to ERP5 (all marked
as TODO).

However with present state it provides all the infrastructure for people
to try to run NEO tests in their webrunner: a `neotest` program is
installed in buildout top-level bin/ . One should use it to run the
tests.

An example run output could be seen here:

https://lab.nexedi.com/snippets/258

With benchstat (https://godoc.org/golang.org/x/perf/cmd/benchstat) it can be summarized or compared to another run. Here is e.g. summarization:

```
name                                                                 pystone/s
vifibcloud-onlinenet-hosting-004/pystone                              156k ± 6%

name                                                                 µs/op
vifibcloud-onlinenet-hosting-004/sha1/py/1024B                        1.71 ± 6%
vifibcloud-onlinenet-hosting-004/sha1/go/1024B                        2.26 ±73%
vifibcloud-onlinenet-hosting-004/sha1/py/4096B                        6.39 ± 4%
vifibcloud-onlinenet-hosting-004/sha1/go/4096B                        5.66 ± 2%

name                                                                 us/op
vifibcloud-onlinenet-hosting-004/disk/randread/direct/4K-min          75.7 ± 4%
vifibcloud-onlinenet-hosting-004/disk/randread/direct/4K-avg          95.2 ± 1%
vifibcloud-onlinenet-hosting-004/disk/randread/pagecache/4K-avg       1.84 ± 0%

name                                                                 time/op
vifibcloud-onlinenet-hosting-004/disk/randread/pagecache/4K-min      288ns ± 5%
vifibcloud-onlinenet-hosting-004/disk/randread/pagecache/4K-avg      876ns ± 9%

name                                                                 µs/object
dataset:wczblk1-8
vifibcloud-onlinenet-hosting-004/fs1/zhash.py                         22.2 ±11%
vifibcloud-onlinenet-hosting-004/fs1/zhash.py-P16                     40.6 ±19%
vifibcloud-onlinenet-hosting-004/fs1/zhash.go                         4.00 ±40%
vifibcloud-onlinenet-hosting-004/fs1/zhash.go+prefetch128             5.58 ±24%
vifibcloud-onlinenet-hosting-004/fs1/zhash.go-P16                     3.68 ±88%
vifibcloud-onlinenet-hosting-004/zeo/zhash.py                          626 ±40%
vifibcloud-onlinenet-hosting-004/zeo/zhash.py-P16                    2.17k ± 3%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py                823 ±17%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py-P16          2.04k ± 3%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go                666 ± 9%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go+prefetch128    138 ±10%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go-P16          1.96k ± 0%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py                 1.02k ±71%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py-P16             3.09k ± 1%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go                   602 ±29%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go+prefetch128       213 ±14%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go-P16             4.66k ± 1%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.py                       326 ±32%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.py-P16                   554 ± 7%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.go                      39.6 ± 7%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.go+prefetch128          24.3 ±18%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.go-P16                   213 ± 6%
vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go               42.8 ±13%
vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go+prefetch128   24.5 ±19%
vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go-P16           70.2 ±62%
dataset:prod1-1024
vifibcloud-onlinenet-hosting-004/fs1/zhash.py                         18.8 ±22%
vifibcloud-onlinenet-hosting-004/fs1/zhash.py-P16                     33.6 ±21%
vifibcloud-onlinenet-hosting-004/fs1/zhash.go                         3.46 ±25%
vifibcloud-onlinenet-hosting-004/fs1/zhash.go+prefetch128             5.36 ±25%
vifibcloud-onlinenet-hosting-004/fs1/zhash.go-P16                     2.56 ±45%
vifibcloud-onlinenet-hosting-004/zeo/zhash.py                          617 ±23%
vifibcloud-onlinenet-hosting-004/zeo/zhash.py-P16                    2.19k ± 1%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py                717 ±46%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py-P16          2.02k ± 1%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go                297 ±14%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go+prefetch128    130 ±10%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go-P16          1.86k ± 7%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py                   680 ±61%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py-P16             4.76k ± 1%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go                   269 ±13%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go+prefetch128       178 ± 8%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go-P16             3.03k ± 1%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.py                       227 ±10%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.py-P16                   654 ± 5%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.go                      32.4 ± 8%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.go+prefetch128          15.5 ±12%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.go-P16                   261 ±12%
vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go               24.2 ± 1%
vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go+prefetch128   13.9 ±25%
vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go-P16            270 ±10%
```

The infrastructure to build Go projects under SlapOS is also introduced along the way
(was applied separetely as nexedi/slapos@1b540151).

Please see details in the individual commit messages. 

/cc @nexedi 

P.S. the results are noisy becuase under regular webrunner I do not have root and cannot run e.g. `cpupower frequency-set -g performance`.

/reviewed-on nexedi/slapos!242

We provide neotest SR which builds current NEO/go together with the
tools which are needed to run the benchmarks. Go part uses
just-introduced in previous commit gowork infrastructure.

The SR does not yet provide automated service to run the tests
automatically periodically and to upload the results to ERP5 (all marked
as TODO).

However with present state it provides all the infrastructure for people
to try to run NEO tests in their webrunner: a `neotest` program is
installed in buildout top-level bin/ . One should use it to run the
tests.

For example:

	slapuser14@vifibcloud-onlinenet-hosting-004:~/srv/runner/t$ ../software/4db21ec948dce895b38ba93c1def3ab1/bin/neotest
	Neotest is a tool to functionally test and benchmark NEO.

	Usage:

	        neotest command [arguments]

	The commands are:

	        bench-local     run benchmarks when client and server are both on the same localhost
	        bench-cluster   run benchmarks when server is local and client is on another node

	        run-client      run client benchmarks against separate server
	        bench-disk      benchmark local disk (already part of bench-{local,cluster})
	        bench-cpu       benchmark local cpu  (already part of bench-{local,cluster})

	        deploy          deploy NEO & needed software for tests to remote host
	        deploy-local    deploy NEO & needed software for tests locally

	        info            print information about a node
	        info-local      print information about local deployment

	Additional utility commands:

	        cpustat         run a command and print CPU-related statistics

	slapuser14@vifibcloud-onlinenet-hosting-004:~/srv/runner/t$ ../software/4db21ec948dce895b38ba93c1def3ab1/bin/neotest info-local
	# Thu, 19 Oct 2017 18:13:35 +0300
	# slapuser14@sd-112617.dedibox.fr (2001:67c:1254:e:8c::1 (+ 20·ipv6) 163.172.70.8 (+ 20·ipv4))
	# Linux vifibcloud-onlinenet-hosting-004 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
	# cpu: Intel(R) Xeon(R) CPU D-1531 @ 2.20GHz
	# cpu[0-11]: freq: intel_pstate/powersave [.80GHz - 2.70GHz]
	# cpu[0-11]: idle: acpi_idle/menu: POLL(0μs) C1(1μs) C2(41μs)
	# cpu: WARNING: frequency not fixed - benchmark timings won't be stable
	# cpu: WARNING: C-state exit-latency is max 41μs - up to that can add to networked and IPC request-reply latency
	# md1 (raid0) -> sda3  sdb3
	# sda: SAMSUNG MZ7LN256  rev 100Q 238.5G
	# sdb: SAMSUNG MZ7LN256  rev 100Q 238.5G
	# eth0: Intel Corporation I350 Gigabit Network Connection rev 01
	# eth0: features: rx tx sg tso !ufo gso gro !lro rxvlan txvlan !ntuple rxhash ...
	# eth0: coalesce: rxc: 3μs/0f/0μs-irq/0f-irq,  txc: 0μs/0f/0μs-irq/0f-irq
	# eth0: up, speed=1000, mtu=1500, txqlen=1000, !gro_flush_timeout
	# eth1: Intel Corporation I350 Gigabit Network Connection rev 01
	# eth1: features: rx tx sg tso !ufo gso gro !lro rxvlan txvlan !ntuple rxhash ...
	# eth1: coalesce: rxc: 3μs/0f/0μs-irq/0f-irq,  txc: 0μs/0f/0μs-irq/0f-irq
	# eth1: down, speed=?, mtu=1500, txqlen=1000, !gro_flush_timeout
	# Python 2.7.14
	# go version go1.9.1 linux/amd64
	# sqlite 3.19.0 (py mod 2.6.0)
	# mysqld  Ver 10.1.28-MariaDB for Linux on x86_64 (MariaDB Server)
	# neo             : v1.8-1326-g4d0cd894
	# zodb            : 4.4.5
	# zeo             : 4.3.1
	# mysqlclient     : 1.3.12
	# wendelin.core   : v0.11-4-g38fbc83

slapuser14@vifibcloud-onlinenet-hosting-004:~/srv/runner/t$ time ../software/4db21ec948dce895b38ba93c1def3ab1/bin/neotest bench-local 2>&1 |tee 1.txt
...
*** FileStorage
Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 24.4 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.052s  # POLL·0 C1·49 C2·853
Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 24.6 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.052s  # POLL·0 C1·63 C2·894
Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 20.7 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.044s  # POLL·0 C1·39 C2·803
Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 20.8 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.044s  # POLL·1 C1·18 C2·629
Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 20.7 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.044s  # POLL·0 C1·12 C2·669
...

lmbench is a bit outdated tool to do OS performance analysis. The
upcoming neotest SR will use lat_tcp from this packages to measure TCP
round-trip-time latency.

We use a bit patched version with fixes to lat_tcp for errors not to go
unnoticed and also lat_tcp.go added to see how using Go compares to
plain lat_tcp.c for latencies.

In the upcoming neotest SR both lat_tcp C & Go versions will be used to
automatically benchmark TCP round-trip-times over network link in between
2 machines.

This tool can be used to measure disk IO latency for various workloads
(random read, sequential read, direct or cached, write, ...).

We use a bit patched version shows not only avg latency but also its
distribution. (the patches are currently dirty/not very robust and not
yet sent to upstream).

In the upcoming neotest SR ioping will be used to automatically
benchmark underlying disk characteristics.

ethtool can be used to adjust NIC settings, such as e.g. interrupt
coalescing and various others. It usually requires root to do so.

However regular users can query NIC settings via ethtool just ok.

This way upcoming neotest SR will use it to query/display NIC setting
which are essential to get good networked latency and automatically
produce warnings if it is not.

Starting from this version zodbtools requires zodburi to open ZODB
storages by URL (see nexedi/zodbtools@db3b84ec
and nexedi/zodbtools@82b06413)

zodburi in turn requires ZEO (maybe we might patch it to only optionally
depend on it), so we move ZEO requirement from stack/ERP5 to the place
where it is actually first needed.

In Go world development workflow is organized around so-called workspace
where multiple packages can be installed / worked on etc. The following
page describes Go workspaces:

https://golang.org/doc/code.html

A new [gowork] section and infrastructure around it is introduced.
Quoting code:

    # gowork is a top-level section representing workspace
    #
    # users should add `install` field to [gowork] to describe packages they want to
    # be installed (+ automatically their dependencies are installed too). e.g.
    #
    #   [gowork]
    #   install =
    #       lab.nexedi.com/kirr/neo/go/...  \
    #       github.com/pkg/profile          \
    #       golang.org/x/perf/cmd/benchstat

The way it all works inside is:

- gowork organizes to create go.work/ directory in buildout root
- inside it creates env.sh which when sources in shell adjusts all paths
  so that appropriate go compiler is in path, GOPATH is also set
  appropriately, etc - in other words everything needs for using/working
  on this workspace is setup in the environment.
- in actual user software profile the list of Go projects which needs to
  be installed by SlapOS has to be listed. The [gowork] machinery takes
  care about git-cloning these projects and `go install`s them after.
- by SlapOS design builds have to be reproducible and so every component
  state/version has to be fixed.

  A convenient helper (gowork-snapshot) to get snapshot of git go
  packages installed with their exact revisions is added. This should be
  used to automatically (re-)generate list of go git repositories &
  their pinning for a workspace.

For a new Go project that needs to be slaposified the workflow should be:

- first get a project into working state via any convenient way (`go get`, etc.)
- generate list of go packages & their pinning with gowork-snapshot
- extend golang/buildout.cfg and in project software profile add

	[gowork]
  	install = your-top-level-packages

  as it was outlined above.

P.S.

[golang] is removed because

1. it is not used anywhere in slapos.git tree, and
2. the canonical way to work on go projects from now on will be to use
   [gowork]. There latest go version is preconfigured, but users can
   override it to some particular other version in their projects as they
   need.

/reviewed-on	 nexedi/slapos!242
/also-needed-for nexedi/slapos!243

This reverts merge request !243

The first proof of concept  of Caddy in Slapos. 

For now, everything is inside software/caddy but in the next merge request we will move to stack. 

@rafael @romain 


/reviewed-on nexedi/slapos!243

WIP

  Environment requires new lines instead spaces

   This is required got the usage of git

  MD5SUM update was missing.

Experimental. Not automatically executed.

Restores given backup, configures service as a slave to given master,
starts replication.

Like for "normal" backup restoration, any running mariadb is killed,
started with backup-restoration-friendly parameters, and killed again once
restored.

As there is AFAIK no way to ask supervisor to start even one of our own
services, this script unfortunately leaves the partition in a
not-immediately useful state.

Nothing references this section otherwise.

This value is supposed to come from configuration file, which is already
implicitly provided inside used wrapper.