* master:
  go/zodb: Loader: Document that loaded buffer must not be modified
  go/zodb: Split Loader & co. interfaces from IStorage
  go/zodb: Document that Tid(0) is invalid while Oid(0) is valid.
  fixup! go/zodb: Tweak documentation a bit so it renders more well in godoc

The reason for this is that the buffer can be shared with other loaders
via cache.

There are situations when we want to work with, or emulate, only subset
of whole IStorage functionality, with one already established example
being Cache. For this reason split IStorage interface into finer-grained
ones. For now:

	- Loader
	- Prefetcher
	- Iterator
	- Committer
	- Notifier

Oid(0) represents root database object. We cannot change this.

Missed few places in 8685b742.

* master: (29 commits)
  go/zodb: Tweak documentation a bit so it renders more well in godoc
  go/neo/t/nxd/runTestSuite: Also run go & py unit tests in addition to bench-local
  go/neo/t/nxd/runTestSuite: Tee tested process stdout,stderr to testnode logs incrementally
  go/neo/t/nxd/runTestSuite: New wrapper to run neotest as part of Nexedi testing infrastructure
  go/neo/t/benchplot: New program to visualise neotest benchmarks (draft)
  go/neo/t/neotest: Add bench-local and bench-cluster commands
  go/internal/xzlib: Switch to github.com/DataDog/czlib to zlib Decompression
  go/internal/xzlib: Try to reuse zlib decoders
  go/neo/t/neotest: bench-cpu += unzlib for wczblk1 and prod1 objects
  go/neo/t/neotest: Switch to zwrk to simulate parallel load from multiple clients
  go/zodb: Allow to open a storage in "direct" mode - without local cache
  go/neo/t/neotest: ZODB benchmarks
  go/neo/t/neotest: Network information & benchmarks
  go/neo/t/neotest: Disk information & benchmarks
  go/neo/t/neotest: CPU information & benchmarks
  go/neo/t: Neotest start
  go/zodb/zeo: Add support for protocols preceding ZEO5
  go/zodb: Teach ZODB/go to access ZEO (draft)
  go/zodb: Start putting pickle-related utilities into internal/pickletools package
  go/neo/neonet: Lightweight mode
  ...

While at it add draft overview of data model & friends to package
documentation.

We send output from tested process to master. We also print it to
stdout,stderr so it appears in testnode logs.

However till now it was like, whole output first read, and only then
emitted to log as a whole, thus not allowing to oversee current test
progress by watching testnode log tail.

Fix it by implementing the teeing process manually.

Some draft history related to this patch:

	lab.nexedi.com/kirr/neo/commit/aa370ca3        fixup! X neotest/runTestSuite: Tee tested process stdout,stderr to testnode logs incrementally
	lab.nexedi.com/kirr/neo/commit/096550b1        fixup! X neotest/runTestSuite: Tee tested process stdout,stderr to testnode logs incrementally
	lab.nexedi.com/kirr/neo/commit/63956f43        fixup! X neotest/runTestSuite: Tee tested process stdout,stderr to testnode logs incrementally
	lab.nexedi.com/kirr/neo/commit/b9819d0e        X neotest/runTestSuite: Tee tested process stdout,stderr to testnode logs incrementally

The wrapper runs `neotest bench-local` in neotest SlapOS instance:

	https://lab.nexedi.com/nexedi/slapos/blob/ab8705f4/software/neotest/instance.cfg.in

Extracted from slapos!282

Add the program that reads results from either bench-local or bench-cluster
neotest output and visualizes it. It uses benchlib.py module to read data
in Go benchmark format(*), processes them and plots scalability and other
graphs via matplotlib.

There are lots of hacks and rough edges, and in particular callout coordinate
calculation is completely wrong. However even in this state benchplot was used
to prepare the graphs in http://navytux.spb.ru/~kirr/neo.html and
http://navytux.spb.ru/~kirr/misc/neo·P4.html .

Some draft history related to this patch:

	lab.nexedi.com/kirr/neo/commit/078c9ac3        X move benchlib to -> https://lab.nexedi.com/kirr/pygolang
	lab.nexedi.com/kirr/neo/commit/0edd5129        X benchplot: Teach it to understand benchmark names for partitioned NEO clusters
	lab.nexedi.com/kirr/neo/commit/a1dde3c9        X deco-rio timings
	lab.nexedi.com/kirr/neo/commit/916782b6        X normalize/convert units, so that disk and ping/tcp latencies could be plotted too
	lab.nexedi.com/kirr/neo/commit/f5fec740        X switch node info to labels; start adding that to plot
	lab.nexedi.com/kirr/neo/commit/906462a3        X neotest: Move cluster / node out fro benchmark name to label in environment
	lab.nexedi.com/kirr/neo/commit/cceca65f        X benchplot: Start of automated plotting for neotest benchmark data
	lab.nexedi.com/kirr/neo/commit/a9b10a45        X benchlib/benchstat: Emit label:value info for several labels on one line, similary to go version
	lab.nexedi.com/kirr/neo/commit/502d9477        X benchlib: Python module to read & work with data in Go benchmark format

(*) benchlib.py is now part of pygolang: https://pypi.org/project/pygolang .

These commands do full benchmarking for localhost and networked cases:

- show system info
- do server & client cpu benchmarks
- do server disk benchmarks
- for networked case: do network benchmarks
- tail to either zbench-local or zbench-cluster

It was full `neotest bench-local` that was used to prepare benchmarks
for http://navytux.spb.ru/~kirr/neo.html and http://navytux.spb.ru/~kirr/misc/neo·P4.html

name                 old time/op    new time/op    delta
unzlib/py/wczdata      20.8µs ± 2%    20.7µs ± 1%     ~     (p=0.421 n=5+5)
unzlib/go/wczdata      64.4µs ± 1%    21.3µs ± 0%  -66.89%  (p=0.008 n=5+5)
unzlib/py/prod1-avg    4.00µs ± 1%    4.02µs ± 1%     ~     (p=0.421 n=5+5)
unzlib/go/prod1-avg    10.4µs ± 1%     4.3µs ± 1%  -58.72%  (p=0.008 n=5+5)

There is also unsafe interface with czlib.UnsafeDecompress & friends which I
had not tried because even using safe interface brings ~ 3x speedup.

name                 old time/op    new time/op    delta
unzlib/py/wczdata      20.7µs ± 2%    20.8µs ± 2%     ~     (p=0.548 n=5+5)
unzlib/go/wczdata      70.6µs ± 0%    64.4µs ± 1%   -8.85%  (p=0.008 n=5+5)
unzlib/py/prod1-avg    4.02µs ± 1%    4.00µs ± 1%     ~     (p=0.167 n=5+5)
unzlib/go/prod1-avg    15.2µs ± 0%    10.4µs ± 1%  -31.59%  (p=0.008 n=5+5)

still on wczdata and prod1 much slower compared to py/c zlib.

NEO uses zlib compression for data, and this way client has to spend
time decompressing it. Benchmark how much time zlib decompression takes.
With stdlib zlib decompressor out of the box it looks like:

	name                 time/op
	unzlib/py/wczdata    20.7µs ± 2%
	unzlib/go/wczdata    70.6µs ± 0%
	unzlib/py/prod1-avg  4.02µs ± 1%
	unzlib/go/prod1-avg  15.2µs ± 0%

i.e. much not in favour of Go.

We'll be fixing that in the following patches.

With zwrk for ZODB being similar to what wrk is for HTTP.

Rationale: simulating multiple clients is:

1. noisy - the timings from run to run are changing sometimes up to 50%
2. with significant additional overhead - there are constant OS-level
   process switches in between client processes and this prevents to
   actually create the load.
3. the above load from "2" actually takes resources from the server in
   localhost case.

So let's switch to simulating many requests in lightweight way similarly
to how it is done in wrk - in one process and not so many threads (it
can be just 1) with many connections opened to server and epolly way to
load it with Go providing epoll-goroutine matching.

Example summarized zbench-local output:

	x/src/lab.nexedi.com/kirr/neo/go/neo/t$ benchstat -split node,cluster,dataset x.txt
	name                             time/object
	cluster:rio dataset:wczblk1-8
	fs1-zhash.py                             23.7µs ± 5%
	fs1-zhash.go                             5.68µs ± 8%
	fs1-zhash.go+prefetch128                 6.44µs ±16%
	zeo/py/fs1-zhash.py                       376µs ± 4%
	zeo/py/fs1-zhash.go                       130µs ± 3%
	zeo/py/fs1-zhash.go+prefetch128          72.3µs ± 4%
	neo/py(!log)/sqlite·P1-zhash.py           565µs ± 4%
	neo/py(!log)/sql·P1-zhash.py              491µs ± 8%
	cluster:rio dataset:prod1-1024
	fs1-zhash.py                             19.5µs ± 2%
	fs1-zhash.go                             3.92µs ±12%
	fs1-zhash.go+prefetch128                 4.42µs ± 6%
	zeo/py/fs1-zhash.py                       365µs ± 9%
	zeo/py/fs1-zhash.go                       120µs ± 1%
	zeo/py/fs1-zhash.go+prefetch128          68.4µs ± 3%
	neo/py(!log)/sqlite·P1-zhash.py           560µs ± 5%
	neo/py(!log)/sql·P1-zhash.py              482µs ± 8%

	name                             req/s
	cluster:rio dataset:wczblk1-8
	fs1-zwrk.go·1                              380k ± 2%
	fs1-zwrk.go·2                              666k ± 3%
	fs1-zwrk.go·3                              948k ± 1%
	fs1-zwrk.go·4                             1.24M ± 1%
	fs1-zwrk.go·8                             1.62M ± 0%
	fs1-zwrk.go·12                            1.70M ± 0%
	fs1-zwrk.go·16                            1.71M ± 0%
	zeo/py/fs1-zwrk.go·1                      8.29k ± 1%
	zeo/py/fs1-zwrk.go·2                      10.4k ± 2%
	zeo/py/fs1-zwrk.go·3                      11.2k ± 1%
	zeo/py/fs1-zwrk.go·4                      11.7k ± 1%
	zeo/py/fs1-zwrk.go·8                      12.1k ± 2%
	zeo/py/fs1-zwrk.go·12                     12.3k ± 1%
	zeo/py/fs1-zwrk.go·16                     12.3k ± 2%
	cluster:rio dataset:prod1-1024
	fs1-zwrk.go·1                              594k ± 7%
	fs1-zwrk.go·2                             1.14M ± 4%
	fs1-zwrk.go·3                             1.60M ± 2%
	fs1-zwrk.go·4                             2.09M ± 1%
	fs1-zwrk.go·8                             2.74M ± 1%
	fs1-zwrk.go·12                            2.76M ± 0%
	fs1-zwrk.go·16                            2.76M ± 1%
	zeo/py/fs1-zwrk.go·1                      9.42k ± 9%
	zeo/py/fs1-zwrk.go·2                      10.4k ± 1%
	zeo/py/fs1-zwrk.go·3                      11.4k ± 1%
	zeo/py/fs1-zwrk.go·4                      11.7k ± 2%
	zeo/py/fs1-zwrk.go·8                      12.4k ± 1%
	zeo/py/fs1-zwrk.go·12                     12.5k ± 1%
	zeo/py/fs1-zwrk.go·16                     13.4k ±11%

	name                             latency-time/object
	cluster:rio dataset:wczblk1-8
	fs1-zwrk.go·1                            2.63µs ± 2%
	fs1-zwrk.go·2                            3.00µs ± 3%
	fs1-zwrk.go·3                            3.16µs ± 1%
	fs1-zwrk.go·4                            3.23µs ± 1%
	fs1-zwrk.go·8                            4.94µs ± 0%
	fs1-zwrk.go·12                           7.06µs ± 0%
	fs1-zwrk.go·16                           9.36µs ± 0%
	zeo/py/fs1-zwrk.go·1                      121µs ± 1%
	zeo/py/fs1-zwrk.go·2                      192µs ± 2%
	zeo/py/fs1-zwrk.go·3                      267µs ± 1%
	zeo/py/fs1-zwrk.go·4                      343µs ± 1%
	zeo/py/fs1-zwrk.go·8                      660µs ± 2%
	zeo/py/fs1-zwrk.go·12                     977µs ± 1%
	zeo/py/fs1-zwrk.go·16                    1.30ms ± 2%
	cluster:rio dataset:prod1-1024
	fs1-zwrk.go·1                            1.69µs ± 7%
	fs1-zwrk.go·2                            1.76µs ± 4%
	fs1-zwrk.go·3                            1.88µs ± 2%
	fs1-zwrk.go·4                            1.91µs ± 1%
	fs1-zwrk.go·8                            2.92µs ± 1%
	fs1-zwrk.go·12                           4.34µs ± 0%
	fs1-zwrk.go·16                           5.80µs ± 1%
	zeo/py/fs1-zwrk.go·1                      107µs ± 9%
	zeo/py/fs1-zwrk.go·2                      192µs ± 1%
	zeo/py/fs1-zwrk.go·3                      263µs ± 1%
	zeo/py/fs1-zwrk.go·4                      342µs ± 2%
	zeo/py/fs1-zwrk.go·8                      648µs ± 1%
	zeo/py/fs1-zwrk.go·12                     957µs ± 1%
	zeo/py/fs1-zwrk.go·16                    1.20ms ±10%

The scalability graphs in http://navytux.spb.ru/~kirr/neo.html were
made with simulating client load by zwrk, not many client OS processes.
http://navytux.spb.ru/~kirr/neo.html#performance-tests has some
additional notes on zwrk.

Some draft history related to this patch:

	lab.nexedi.com/kirr/neo/commit/ca0d828b	X neotest: Tzwrk1 - place to control running time of 1 zwrk iteration
	lab.nexedi.com/kirr/neo/commit/bbfb5006	X zwrk: Make sure we warm up connections to all NEO storages when cluster is partitioned
	lab.nexedi.com/kirr/neo/commit/7f22bba6	X zwrk: New tool to simulate paralell load from multiple clients

zodb/go provides generic cache (see 7233b4c0 "zodb/go: In-RAM client
cache") primarily in order for prefetch to work. However if we need to
benchmark a storage with loading some objects several times, this cache
can hide the actual time it takes for an object to load.

For such use cases add NoCache open option so that opening does not
create a cache and always conveys load operations directly to storage
driver. The option will be used by zwrk tool (see next patch).

Add to neotest zbench-local and zbench-cluster commands that perform
ZODB benchmarks on FileStorage, ZEO and NEO with Python and Go clients
either locally, or with a server and client running on 2 different nodes.

There are 2 client programs: tzodb.py and tzodb.go which for now compute
hash of whole latest objects stream in a ZODB database. On server side
neotest is taught to launch ZEO and various NEO clusters and to execute
client load on them.

Two test datasets are used: wczblk1-8 - the dataset with wendelin.core ZBlk1
objects covering 8M array, and prod1-1024 - synthethic dataset that tries to
represent regular ERP5 instance. Both datasets are very small and so we can
assume they reside completely in server disk cache while running benchmarks.
Benchmark timings will thus give pure storage software processing latency, as
pagecache hit time is on par, or less, to 1µs.

Example output:

	x/src/lab.nexedi.com/kirr/neo/go/neo/t$ ./neotest zbench-local
	dataset:	wczblk1-8
	node:
	cluster:	deco

	*** generating fs1 data...
	I: RAM:  7.47GB
	I: WORK: 0.01GB
	gen signal t=0...1.05e+06  float64  (= 0.01GB)
	gen signal blk [0:1048576]  (100.0%)
	VIRT: 297 MB	RSS: 48MB

	*** generating sqlite data...
	I: RAM:  7.47GB
	I: WORK: 0.01GB
	gen signal t=0...1.05e+06  float64  (= 0.01GB)
	gen signal blk [0:1048576]  (100.0%)
	VIRT: 386 MB	RSS: 58MB
	2018-07-10 19:57:35.7065 ERROR     NEO        [           app: 91] primary master is down
	Cluster state changed

	*** generating sql data...
	2018-07-10 19:57:35 140115116649600 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6+b1) starting as process 27574 ...
	2018-07-10 19:57:39 140205509999744 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6+b1) starting as process 27603 ...
	2018-07-10 19:57:42 139692109810816 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6+b1) starting as process 27633 ...
	2018-07-10 19:57:45 139759221546112 [Note] mysqld (mysqld 10.1.29-MariaDB-6+b1) starting as process 27662 ...
	I: RAM:  7.47GB
	I: WORK: 0.01GB
	gen signal t=0...1.05e+06  float64  (= 0.01GB)
	gen signal blk [0:1048576]  (100.0%)
	VIRT: 387 MB	RSS: 59MB
	2018-07-10 19:57:48.2565 ERROR     NEO        [           app: 91] primary master is down
	Cluster state changed

	*** FileStorage
	Benchmarkfs1-zhash.py 2127 16.3 µs/object	# crc32:14640593  nread=8540363  t=0.035s	# POLL·2 C1·73 C1E·38 C3·12 C6·36 C7s·0 C8·112 C9·0 C10·62
	Benchmarkfs1-zhash.py 2127 16.6 µs/object	# crc32:14640593  nread=8540363  t=0.035s	# POLL·0 C1·113 C1E·21 C3·16 C6·56 C7s·0 C8·136 C9·0 C10·41
	Benchmarkfs1-zhash.py 2127 15.9 µs/object	# crc32:14640593  nread=8540363  t=0.034s	# POLL·0 C1·71 C1E·36 C3·22 C6·50 C7s·0 C8·167 C9·0 C10·47
	Benchmarkfs1-zhash.py 2127 15.9 µs/object	# crc32:14640593  nread=8540363  t=0.034s	# POLL·0 C1·77 C1E·32 C3·11 C6·55 C7s·0 C8·184 C9·0 C10·31
	Benchmarkfs1-zhash.py 2127 16.0 µs/object	# crc32:14640593  nread=8540363  t=0.034s	# POLL·0 C1·78 C1E·15 C3·12 C6·51 C7s·0 C8·140 C9·0 C10·44

	# 16 clients in parallel
	Benchmarkfs1-zhash.py·P16 2127 129.0 µs/object	# crc32:14640593  nread=8540363  t=0.274s
	Benchmarkfs1-zhash.py·P16 2127 132.6 µs/object	# crc32:14640593  nread=8540363  t=0.282s
	Benchmarkfs1-zhash.py·P16 2127 135.0 µs/object	# crc32:14640593  nread=8540363  t=0.287s
	Benchmarkfs1-zhash.py·P16 2127 135.3 µs/object	# crc32:14640593  nread=8540363  t=0.288s
	Benchmarkfs1-zhash.py·P16 2127 136.6 µs/object	# crc32:14640593  nread=8540363  t=0.291s
	Benchmarkfs1-zhash.py·P16 2127 122.8 µs/object	# crc32:14640593  nread=8540363  t=0.261s
	Benchmarkfs1-zhash.py·P16 2127 130.9 µs/object	# crc32:14640593  nread=8540363  t=0.279s
	Benchmarkfs1-zhash.py·P16 2127 126.4 µs/object	# crc32:14640593  nread=8540363  t=0.269s
	Benchmarkfs1-zhash.py·P16 2127 125.8 µs/object	# crc32:14640593  nread=8540363  t=0.268s
	Benchmarkfs1-zhash.py·P16 2127 108.3 µs/object	# crc32:14640593  nread=8540363  t=0.230s
	Benchmarkfs1-zhash.py·P16 2127 131.0 µs/object	# crc32:14640593  nread=8540363  t=0.279s
	Benchmarkfs1-zhash.py·P16 2127 124.1 µs/object	# crc32:14640593  nread=8540363  t=0.264s
	Benchmarkfs1-zhash.py·P16 2127 129.3 µs/object	# crc32:14640593  nread=8540363  t=0.275s
	Benchmarkfs1-zhash.py·P16 2127 125.0 µs/object	# crc32:14640593  nread=8540363  t=0.266s
	Benchmarkfs1-zhash.py·P16 2127 131.5 µs/object	# crc32:14640593  nread=8540363  t=0.280s
	Benchmarkfs1-zhash.py·P16 2127 131.4 µs/object	# crc32:14640593  nread=8540363  t=0.280s
	# POLL·0 C1·4 C1E·13 C3·11 C6·79 C7s·0 C8·14 C9·0 C10·0

	...

And its summary via benchstat:

	x/src/lab.nexedi.com/kirr/neo/go/neo/t$ benchstat -split node,cluster,dataset x.log
	name                                 time/object
	cluster:deco dataset:wczblk1-8
	fs1-zhash.py                         16.1µs ± 3%
	fs1-zhash.py·P16                      130µs ± 5%
	fs1-zhash.go                         3.00µs ±10%
	fs1-zhash.go+prefetch128             3.40µs ±18%
	fs1-zhash.go·P16                     10.2µs ±71%
	zeo/py/fs1-zhash.py                   336µs ± 3%
	zeo/py/fs1-zhash.py·P16              3.22ms ± 6%
	zeo/py/fs1-zhash.go                   112µs ± 2%
	zeo/py/fs1-zhash.go+prefetch128      60.9µs ± 1%
	zeo/py/fs1-zhash.go·P16              1.07ms ± 5%
	neo/py(!log)/sqlite·P1-zhash.py       291µs ± 2%
	neo/py(!log)/sqlite·P1-zhash.py·P16  2.86ms ± 1%
	neo/py(!log)/sql·P1-zhash.py          318µs ± 4%
	neo/py(!log)/sql·P1-zhash.py·P16     3.99ms ± 0%
	cluster:deco dataset:prod1-1024
	fs1-zhash.py                         12.3µs ± 1%
	fs1-zhash.py·P16                      106µs ±10%
	fs1-zhash.go                         2.56µs ±10%
	fs1-zhash.go+prefetch128             2.68µs ± 8%
	fs1-zhash.go·P16                     9.48µs ±43%
	zeo/py/fs1-zhash.py                   319µs ± 3%
	zeo/py/fs1-zhash.py·P16              3.13ms ± 3%
	zeo/py/fs1-zhash.go                   101µs ± 5%
	zeo/py/fs1-zhash.go+prefetch128      56.9µs ± 1%
	zeo/py/fs1-zhash.go·P16              1.19ms ± 4%
	neo/py(!log)/sqlite·P1-zhash.py       281µs ± 3%
	neo/py(!log)/sqlite·P1-zhash.py·P16  2.80ms ± 1%
	neo/py(!log)/sql·P1-zhash.py          316µs ± 1%
	neo/py(!log)/sql·P1-zhash.py·P16     3.91ms ± 1%

Since there is no NEO/go support yet, corresponding neotest parts are merged,
but commented-out with appropriate remark.

Parallel access is simulated with spawning many OS processes for now.
This will change in the nearby followup patch to zwrk.

Results of ZODB benchmarking were discussed in

	http://navytux.spb.ru/~kirr/neo.html#performance-tests		, and
	http://navytux.spb.ru/~kirr/neo.html#results-and-discussion

Some draft history related to this patch:

	lab.nexedi.com/kirr/neo/commit/e0d875bc	X neotest: Teach it to benchmark NEO with storage partitioned to several nodes
	lab.nexedi.com/kirr/neo/commit/590f0a46	X neo/py uses n(replica) as n(real-replica) - 1
	lab.nexedi.com/kirr/neo/commit/b655da26	X save time not benchmarking things we do not show
	lab.nexedi.com/kirr/neo/commit/f834f40d	X zhash: Show N(obj) read, not 1, in place of N(iter)
	lab.nexedi.com/kirr/neo/commit/a16e8d52	X teach golang to access ZEO
	lab.nexedi.com/kirr/neo/commit/b9827725	X switch to using no compression, because this way it is more fair for comparing storage latencies
	lab.nexedi.com/kirr/neo/commit/c0067335	X neotest: Don't depend on killall
	lab.nexedi.com/kirr/neo/commit/2bcd6ebb	X neotest: add zbench-local & zbench-cluster subcomands
	lab.nexedi.com/kirr/neo/commit/fb165ad9	X neotest: Also benchmark NEO/py with logging disabled
	lab.nexedi.com/kirr/neo/commit/2118ba38	X neotest: Help mysqlk_install_db find its basedir under SlapOS
	lab.nexedi.com/kirr/neo/commit/80eaa05e	X zgenprod1 tool
	lab.nexedi.com/kirr/neo/commit/eb0e516f	X check hash result and error if mismatch (zhash.* part); neotest part pending
	lab.nexedi.com/kirr/neo/commit/046370db	X benchify rest of bench-cluster
	lab.nexedi.com/kirr/neo/commit/2d13818e	X bench-local + zhash: Add output in std bench format
	lab.nexedi.com/kirr/neo/commit/1d692a3b	X add NEO/go with SHA1 disabled (both Sgo and Cgo to regular benchmarks)