Wendelin.core 2 needs this.
See nexedi/ZODB!1 for details.

- factor bits that constitute ZODB stack into component/ZODB;
- add support for testing ZODB on Nexedi testing infrastructure;

This are preparatory steps to integrate
nexedi/ZODB!1.

Please see details in the individual commits.

/cc @nexedi
/proposed-for-review-on: nexedi/slapos!867

Organize testing of ZODB 3/4/5 on Nexedi testing infrastructure.

ZODB is core component that we rely on everywhere, and given that this days
upstream is not very active, and we are going to add our own patches to ZODB,
we should add automatic testing to make sure that our ZODB is in good shape.

We'll need persistent as git checkout in the next patch when organizing testing
of ZODB stack, because persistent tests want to discover in-tree files that are
not present in persistent egg when it is installed in non-development mode:

https://erp5.nexedi.net/test_result_module/20201123-3F859E35/7
(look for "AssertionError: could not find my setup.py")

https://github.com/zopefoundation/persistent/blob/4.6.4-0-g7ed95cf/persistent/tests/test_docs.py#L37-L43

NEO/ZODB3 was using transaction 1.1.1, but ZODB3 was fixed to follow transaction 1.6.1 in 2016:

https://github.com/zopefoundation/ZODB/commit/d8409616

Actually ZODB3 tests fail with transaction 1.1.1

https://erp5.nexedi.net/test_result_module/20201123-60855473/6

Factor-out things that provide components for ZODB stack from software/neoppod/
into component/ZODB/. We switch ZODB 3/4/5 to git checkouts, but used revisions
correspond to ZODB versions we were using before, so this should be a
non-functional change.

We need ZODB to be git checkouts for better control, because we'll want to add
our own patches to ZODB4, for example nexedi/ZODB!1.

If that change will be done, it will be done as a separate step.

We will also need git checkouts to add testing support for ZODB, so that we
could setup automatic testing for this essential-to-Nexedi component on our own
testing infrastructure.

https://github.com/zopefoundation/ZEO/commit/503dccb1

We upgraded ZEO5 to 5.2.2 in commit 34ebf8b5 (ZEO5: v↑ (5.2.0 -> 5.2.2)), and
so now it works with msgpack 0.6.2 provided by stack/slapos.cfg.

The component remains on wendelin.core 1, but the build environment is
adjusted so that both (any of) wendelin.core 1 or wendelin.core 2 could
be built:

- Add zodbtools that wendelin.core 2 requires,
- Add Go and zlib to gowork that are neede to build WCFS.

GOPATH is going to go away in Go 1.17 and most of in-tree SlapOS things are
already built using modules - including upcoming wendelin.core 2 that also
builds/uses NEO/go. This way maintaining GOPATH-based approach becomes just
unneccesary burden (modulo development, where it is still needed sometime
unfortunately https://github.com/golang/go/issues/37755#issuecomment-771927771)

What was "latest" in 2018 is outdated by now.
-> Just use the versions that is provided by component/wendelin.core that neotest extends from.

Add support for using Go modules to golang/gowork infrastructure:

- Users can now request to install a module via gowork:install as. e.g.
  in the following example:

        [gowork]
        install =
            lab.nexedi.com/kirr/neo/go/...@v0.0.0-20210103165133-f3effa6c535f
            golang.org/x/tools/gopls@v0.4.3
            ${helloweb:location}/go:./...

  The first two request to install programs from an external module at particular
  revision/version. The latter requests to install programs from locally
  cloned/checked-out module source.

  The documentation now talks only about programs, because "package
  installation" became unnecessary long time ago as Go toolchain uses
  right packages and recompiles things as needed automatically since
  introduction of the Go build cache in go 1.10.

- The change comes accompanied by corresponding helloweb change that
  reworks it to a) become a module itself, and b) to use other modules -
  that are not explicitly cloned by buildout - so that we can be sure
  that module way of fetching/building things actually works.

  kirr/helloweb@a7c788ae

- Non-module way - e.g. build via GOPATH - is still supported (because
  e.g. software/gitlab still uses it), but not explicitly documented and 
  scheduled to be deprecated and removed. The reason for this is that
  upstream Go is going to remove support for GOPATH and leave only
  module-based approach in Go1.17

  https://github.com/golang/go/issues/37755#issuecomment-771879911

/cc @jerome, @luke, @tomo, @alain.takoudjou
/reviewed-on !924

Add support for using Go modules to golang/gowork infrastructure:

- Users can now request to install a module via gowork:install as. e.g.
  in the following example:

	[gowork]
	install =
	    lab.nexedi.com/kirr/neo/go/...@v0.0.0-20210103165133-f3effa6c535f
	    golang.org/x/tools/gopls@v0.4.3
	    ${helloweb:location}/go:./...

  The first two request to install programs from an external module at particular
  revision/version. The latter requests to install programs from locally
  cloned/checked-out module source.

  The documentation now talks only about programs, because "package
  installation" became unnecessary long time ago as Go toolchain uses
  right packages and recompiles things as needed automatically since
  introduction of the Go build cache in go 1.10.

- The change comes accompanied by corresponding helloweb change that
  reworks it to a) become a module itself, and b) to use other modules -
  that are not explicitly cloned by buildout - so that we can be sure
  that module way of fetching/building things actually works.

  kirr/helloweb@a7c788ae

- Non-module way - e.g. build via GOPATH - is still supported (because
  e.g. software/gitlab still uses it), but not explicitly documented and
  scheduled to be deprecated and removed. The reason for this is that
  upstream Go is going to remove support for GOPATH and leave only
  module-based approach in Go1.17

  https://github.com/golang/go/issues/37755#issuecomment-771879911

Some software releases - e.g. wendelin.core - only use ${go:exe} and
does not put anything into ${gowork:install}.

Put emphasis on that gowork defines Go workspace and explain first
settings that are related to that definition. Only after that say how to
use gowork.install. The reason for this is that gowork.install will
become optional in the next patch.

gowork.cfg idea is to specify a GOPATH snapshot with a list of
repositories and their revisions. GOPATH way of building things is going
to go away and we'll soon switch default mode to build go things to do
so via Go modules.

Refactor helloweb a bit before doing that as a preparatory step.

This allows factorising its version pins.

Update PEM to the latest available version.

Instead of having softwares install yarn, unify this in nodejs stack.

Yarn usage is similar to nodejs usage, if a specific version is needed,
software should use macro to expose which yarn version to use, example:

    [yarn]
    <= yarn-1.17.3

Then sections can use yarn by having ${yarn:location}/bin/ in their path.

yarn will use the default [nodejs], so to another nodejs version, the same
pattern can be used:

    [nodejs]
    <= nodejs-10.6.0

Since gmp is used by nft and nft is used by firewalld which is included
in slapos-node package, we need to have support for all CPU (disable-assembly).
Otherwise, you can see this kind of errors in dmesg:

firewalld[16932] trap invalid opcode ip:7f5f26ad61fb sp:7fff95cf7480 error:0 in libgmp.so.10.3.1[7f5f26ac7000+51000]

Note that --host=none-pc-linux-gnu is needed to enable the shared libraries.

We need the newest version anyway and it does not trigger a problem in OBS

Changes:
- Add comments and reorganise instance.cfg.in for clarity
- Select free ports instead of hardcoded ports
- Upgrade to slapos.core 1.6.5 to prefix forwarded requests

This makes it possible to recursively nest theias into theias.

See merge request nexedi/slapos!919

Consider a Go package that is defined via go-git-package, for example

    [helloweb]
    <= go-git-package
    go.importpath = lab.nexedi.com/nexedi/helloweb
    repository    = https://lab.nexedi.com/nexedi/helloweb.git

Currently, since go-git-package references ${gowork:src}, it creates
helloweb -> gowork dependency. gowork, in turn, depends on
gowork.goinstall, which gets list of things to install from ${gowork:install}.

Currently we put only plain strings into ${gowork.install}, e.g.

    [gowork]
    install =
       lab.nexedi.com/nexedi/helloweb/go/...

but for Go modules support and for properly expressing what depends on what,
we'll want in the next patch to be able to specify something like

    [gowork]
    install =
       ${helloweb:location}/go:./...

which will create helloweb ⇄ gowork cycle.

Unfortunately buildout does not detect nor report an error for such cycles, and
simply processes parts in an order, which leads to situation where e.g.
helloweb was not yet cloned, but gowork.goinstall tries to `go install` it and
complains "no such helloweb directory".

-> Fix it by leaving gowork to use by component/golang/ users, and putting
settings about where gowork directories is into underlying gowork.dir section.

Not sure which commit broke it, but this diff is now polluting my commits.

GOPATH is going to be removed in Go1.17 (see e.g. https://github.com/golang/go/issues/37755#issuecomment-771879911).

-> Prevent programs suddenly become installed into $HOME/go/bin instead of
gowork/bin, and mod cache to become something like $HOME/go/... instead of
being kept under gowork/

No change in behaviour for Go ≤ 1.16

This reverts commit 685f611d partially
because it breaks Mroonga (https://github.com/mroonga/mroonga/pull/394).
It's not so urgent to upgrade MariaDB so let's wait for a new release
of Mroonga.

Go1.16 is incremental improvement over Go1.15 with better and faster compiler,
runtime and module-mode improvements:

https://blog.golang.org/go1.16
https://golang.org/doc/go1.16

Since this Go release switches GO111MODULE default from "auto" to "on", but we
still have in-tree GOPATH-mode uses, add GO111MODULE=auto to our goenv.sh to
preserve previous behaviour for now.

Don't drop support for Go1.14, Go1.13, Go1.12 yet, as that no longer supported
Go releases are still being used by in-tree components(*).

Remain default at Go1.15 yet.

Switch helloworld to Go1.16 and test this patch on that
software-release.

(*) theia, caddy, galene, replication-manager, restic, gitlab and grafana

Go1.14 is end-of-life by today as it will no longer be updated after Go1.16 was
released in Feb 2021:

        https://golang.org/doc/devel/release.html#go1.14
        https://golang.org/doc/devel/release.html#go1.16

Go1.15 is current oldstable - i.e. it is still supported and mature.

Going Go1.14.12 -> Go1.14.15 brings in fixes to compiler, runtime, standard
library and build security.

https://golang.org/doc/devel/release.html#go1.14.minor

Tested on helloworld SR (by adjusting it to use go1.14 instead of go1.15)

Go1.14.15 is the last Go1.14.x release, because, after Go1.16 was released, Go1.14
becomes end-of-life and will not receive any updates nor fixes for security bugs.

/cc @jerome (for theia which is on go1.14)
/cc @luke (for caddy which is on go1.14)

/cc @tomo, @alain.takoudjou (for galene which is still on EOL go1.13)
/cc @alain.takoudjou (for replication-manager which is still on EOL go1.13)
/cc @alain.takoudjou (for restic which is still on EOL go1.13)

/cc @jerome, @alain.takoudjou (for gitlab which is still on EOL go1.12)
/cc @jerome (for grafana which is still on EOL go1.12)

Going Go1.15.5 -> 1.15.8 brings in fixes to compiler, runtime, standard library
and build security.

https://golang.org/doc/devel/release.html#go1.15.minor

Tested on helloworld SR.

* Update md5sum
* Add extra egg plone command

This brings in the following changes:

- nexedi/nxdtest@1e6a1cc6
- nexedi/nxdtest@b5a74214

The first one makes nxdtest more robust to detect an error in its worker
thread, while the second one makes nxdtest more human-friendly when used
while developing.

/reviewed-by @jerome
/reviewed-on nexedi/slapos!918

buildout has no knowledge of setup_requires, when installing a package with
setup_requires, setuptools will fetch the package from pypi and install its latest
version, regardless of buildout's versions or allow-picked-versions options, that
are critical for us to ensure repeatability.

These patches were produced by running buildout with a modified setuptools version which does not install setup_requires ( setuptools@54f42c38 ) and by modifying the profiles to use setup-eggs option in all the sections installing packages using setup_requires.

See merge request nexedi/slapos!912

It does -X PUT + --data-binary @file.

It's available only on the slave, so it is not part of master partition
configuration.

Due to the cleanup change the parameter parsing logic has to be improved.

Improvements:

 * link from README to schemas
 * drop incorrect entries in README
 * improve entries description in README
 * make parameter description more understandable in SCHEMA