See merge request nexedi/slapos!1499

/reviewed-by @kirr and @jerome
/reviewed-on nexedi/slapos!1494

wendelin.core 2.0.alpha3.post9 supports golang 1.21: with this new
version we can set golang 1.21 as the new default.

/reviewed-by @kirr and @jerome
/reviewed-on nexedi/slapos!1494

Go1.21 already has the fifth minor revision (released 2023-12-05) and
should therefore already be sufficiently stable. Furthermore we need it
to fix a bug in a NEO/go dependency [1].

Please find all details in the official release note:

    https://go.dev/doc/go1.21

It was released on 2023-08-08 [2]. Due to the go promise of compatibility
most software should still compile without any problems.

In golang < 1.21 we needed to patch golang to fix
https://github.com/golang/go/issues/42525. In golang 1.21 we still need
to apply a fix, but can't apply the old patch because the code changed.
In golang > 1.21 this problem is already fixed with https://go-review.googlesource.com/c/go/+/520055.

Because of https://github.com/golang/go/commit/092671423cd95eaa6df93eb29442fef41504d097
'TestUnshareMountNameSpace' fails on golang 1.21 [3]. In golang 1.20 this test
was skipped [4]. To fix this failure the additional patch
'skip-unshare-mount-test.patch' has been added.

---

[1] nexedi/wendelin.core!22 (comment 195769)]
[2] https://go.dev/doc/devel/release
[3] --- FAIL: TestUnshareMountNameSpace (0.18s)
    exec_linux_test.go:243: unshare failed: exit status 2
        unshare: mount /tmp/TestUnshareMountNameSpace2210137852/001 failed: 0x1
[4] === RUN TestUnshareMountNameSpace exec_linux_test.go:333: kernel prohibits unshare in unprivileged process, unless using user namespace — SKIP: TestUnshareMountNameSpace (0.00s)

/reviewed-by @kirr and @jerome
/reviewed-on nexedi/slapos!1494

This version is compatible with gcc 12

using https://github.com/ilevkivskyi/com2ann

drop versions not running on debian 12 and add new versions

This was disabled to "keep compatibility with current font selection
problems", but I don't think we need to care about compatibility for
this. This was anyway causing a problem that system fontconfig will be
used if it's available, so it's better to define things explictly to
prevent falling back to system configuration.

This caused cloudooo to select different fonts, because LibreOffice-bin
comes with some Noto fonts and fontconfig now prefers Noto font

Testing conversion of HTML with font-family:"FontFamily FontVariant"
CSS rule does not really make sense. Remove a few cases for which the
behavior is not stable.

Using RegExp to validate hostnames is a bad practice, and has a lot of reasons to be wrong.
On top of that, the JSON Schema specification allows, since draft 7, to validate hostnames
against an IDN hostname, by using the `idn-hostname` format.

With these changes, IDN are now supported (.рф and .中國 for instance), and long TLD
should not be a problem anymore.

With backported fixes so that old software supports debian 12

In 9636d79c (Nextcloud Upgrade fixes, 2023-12-13) parameter changed:
 - instance.trusted-domain-* parameters are replaced by instance.trusted-domain-list
 - the frontend is part of trusted domain, so with "bypassed" frontends
from slapos proxy the backend appears twice (because the backend URL is
returned as frontend URL)

This actually updates firefox from version 68 to version 115

So that we can see when something changes.

See merge request !1486

Use runTestSuite interface to launch tests that request instances
on actual SlapOS cloud.

As described in https://github.com/msgpack/msgpack-python#major-breaking-changes-in-msgpack-10
raw is False by default for unpacker so we receive strings, not bytes

See merge request !1490

ZODB4 does not support python3

These must be installed through dependencies, but we explicitly need
scripts from supervisor because this is used by slapos-core tests

on python3 we use lzma directly

This software only use nodejs for a simple build step, it should be OK
to use any version.

Hello up there.

I've noticed that wendelin.core tests became failing to build and
automatically disabled some time ago. Not good. I've tried to fix build
failures. Please see individual patches for details.

/cc @levin.zimmermann
/reviewed-by @jerome
/reviewed-on nexedi/slapos!1492

After probably 02fad6a8 (component/{numpy,scipy}: Version up for Python3.)
pygolang test build started to fail in numpy compilation as shown in the
appendix. The failure is correct as pygolang never explicitly depended on
${numpy:egg}. We did not noticed before probably because in earlier versions
numpy did not required Cython to be present at install time.

-> Fix it by explicitly adding ${numpy:egg} to pygolang[all_test] dependencies.

Apendix. Log of test build failure

    networkcache: Trying to download pypi:numpy=1.22.0 from network cache...
    Getting distribution for 'numpy==1.22.0'.
    WARNING: The easy_install command is deprecated and will be removed in a future version.
    Running from numpy source directory.
    Processing numpy/random/_bounded_integers.pxd.in
    Processing numpy/random/_pcg64.pyx
    Traceback (most recent call last):
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/tools/cythonize.py", line 53, in process_pyx
        import Cython
    ModuleNotFoundError: No module named 'Cython'

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/tools/cythonize.py", line 234, in <module>
        main()
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/tools/cythonize.py", line 230, in main
        find_process_files(root_dir)
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/tools/cythonize.py", line 221, in find_process_files
        process(root_dir, fromfile, tofile, function, hash_db)
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/tools/cythonize.py", line 187, in process
        processor_function(fromfile, tofile)
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/tools/cythonize.py", line 60, in process_pyx
        raise OSError(msg) from e
    OSError: Cython needs to be installed in Python as a module
    Traceback (most recent call last):
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 154, in save_modules
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 195, in setup_context
      File "<string>", line 7, in setup_context
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 250, in run_setup
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 45, in _execfile
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/setup.py", line 450, in <module>
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/setup.py", line 432, in setup_package
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/setup.py", line 237, in generate_cython
    RuntimeError: Running cythonize failed!

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 10, in <module>
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 2317, in main
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/__init__.py", line 162, in setup
      File "/srv/slapgrid/slappart49/srv/runner/shared/python3/a066243faa0ae8c256b7daa6207cc9a6/lib/python3.9/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/srv/slapgrid/slappart49/srv/runner/shared/python3/a066243faa0ae8c256b7daa6207cc9a6/lib/python3.9/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/srv/slapgrid/slappart49/srv/runner/shared/python3/a066243faa0ae8c256b7daa6207cc9a6/lib/python3.9/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 424, in run
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 666, in easy_install
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 711, in install_item
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 896, in install_eggs
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 1164, in build_and_install
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/command/easy_install.py", line 1150, in run_setup
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 253, in run_setup
      File "/srv/slapgrid/slappart49/srv/runner/shared/python3/a066243faa0ae8c256b7daa6207cc9a6/lib/python3.9/contextlib.py", line 137, in __exit__
        self.gen.throw(typ, value, traceback)
      File "<string>", line 7, in setup_context
      File "/srv/slapgrid/slappart49/srv/runner/shared/python3/a066243faa0ae8c256b7daa6207cc9a6/lib/python3.9/contextlib.py", line 137, in __exit__
        self.gen.throw(typ, value, traceback)
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 195, in setup_context
      File "/srv/slapgrid/slappart49/srv/runner/shared/python3/a066243faa0ae8c256b7daa6207cc9a6/lib/python3.9/contextlib.py", line 137, in __exit__
        self.gen.throw(typ, value, traceback)
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 166, in save_modules
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 141, in resume
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/_vendor/six.py", line 685, in reraise
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 154, in save_modules
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 195, in setup_context
      File "<string>", line 7, in setup_context
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 250, in run_setup
      File "/srv/slapgrid/slappart49/srv/runner/software/0919a77f026d6fc6870b3d6ec703b69d/eggs/setuptools-44.1.1-py3.9.egg/setuptools/sandbox.py", line 45, in _execfile
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/setup.py", line 450, in <module>
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/setup.py", line 432, in setup_package
      File "/tmp/easy_install-z2of107_/numpy-1.22.0/setup.py", line 237, in generate_cython
    RuntimeError: Running cythonize failed!
    Cythonizing sources
    An error occurred when trying to install numpy 1.22.0. Look above this message for any errors that were output by easy_install.
    While:
      Installing gpython.
      Base installation request: 'pygolang[all_test]'
        Requirement of pygolang[all_test]: wheel
        Requirement of pygolang[all_test]: setuptools_dso>=2.8
        Requirement of pygolang[all_test]: setuptools
        Requirement of pygolang[all_test]: pytest
        Requirement of pygolang[all_test]: numpy
        Requirement of pygolang[all_test]: ipython
        Requirement of pygolang[all_test]: cython<3
        Requirement of pygolang[all_test]: geventmp
        Requirement of pygolang[all_test]: setuptools_dso>=2.8
        Requirement of pygolang[all_test]: decorator
        Requirement of pygolang[all_test]: six
        Requirement of pygolang[all_test]: gevent
        Requirement of setuptools_dso>=2.8: setuptools
        Requirement of pytest: wcwidth
        Requirement of pytest: six>=1.10.0
        Requirement of pytest: py>=1.5.0
        Requirement of pytest: pluggy<1.0,>=0.12
        Requirement of pytest: packaging
        Requirement of pytest: more-itertools>=4.0.0
        Requirement of pytest: attrs>=17.4.0
        Requirement of pytest: atomicwrites>=1.0
      Getting distribution for 'numpy==1.22.0'.
    Error: Couldn't install: numpy 1.22.0

I already fixed ZEO4-wc2 installation once in 97832d95 (ZEO: Fix
ZEO4-wc2 installation). That worked but, as it turned out, not fully:
Even though correct ZEO4-wc2-repository is cloned, and correct ZEO4-wc2
develop-egg is installed, we don't tell buildout which egg-version of
ZEO to use. This leads to the situation when buildout picks up _latest_
ZEO egg, when another egg, which depends on ZEO, is installed.

For example wendelin.core/test-zodb4-wc2 was failing as

    Updating ZEO4-wc2-repository.
    Installing ZEO4-wc2.
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    Installing ZEO.
    Installing zodbtools.
    While:
      Installing zodbtools.
      Base installation request: 'zodbtools'
        Requirement of zodbtools==0.0.0.dev8: dateparser
        Requirement of zodbtools==0.0.0.dev8: six
        Requirement of zodbtools==0.0.0.dev8: pygolang>=0.0.0.dev6
        Requirement of zodbtools==0.0.0.dev8: zope.interface
        Requirement of zodbtools==0.0.0.dev8: zodburi
        Requirement of zodbtools==0.0.0.dev8: ZODB
        Requirement of dateparser: tzlocal
        Requirement of dateparser: regex!=2019.02.19
        Requirement of dateparser: pytz
        Requirement of dateparser: python-dateutil
        Requirement of pygolang>=0.0.0.dev6: Importing
        Requirement of pygolang>=0.0.0.dev6: decorator
        Requirement of pygolang>=0.0.0.dev6: six
        Requirement of pygolang>=0.0.0.dev6: gevent
        Requirement of zope.interface: setuptools
        Requirement of zodburi: ZEO
        Requirement of zodburi: ZConfig
        Requirement of zodburi: ZODB
        Requirement of ZODB: zodbpickle>=0.6.0
        Requirement of ZODB: zope.interface
        Requirement of ZODB: zc.lockfile
        Requirement of ZODB: six
        Requirement of ZODB: transaction>=1.5.0
        Requirement of ZODB: ZConfig
        Requirement of ZODB: BTrees>=4.2.0
        Requirement of ZODB: persistent>=4.2.0
        Requirement of tzlocal: pytz
        Requirement of python-dateutil: six>=1.5
        Requirement of gevent: greenlet>=0.4.17
        Requirement of gevent: setuptools
        Requirement of gevent: zope.interface
        Requirement of gevent: zope.event
        Requirement of ZEO: trollius
        Requirement of ZEO: futures
        Requirement of ZEO: zope.interface
        Requirement of ZEO: zdaemon
        Requirement of ZEO: ZConfig
        Requirement of ZEO: zc.lockfile
        Requirement of ZEO: persistent>=4.1.0
        Requirement of ZEO: transaction>=2.0.3
        Requirement of ZEO: six
        Requirement of ZEO: ZODB>=5.1.1
        Requirement of zodbpickle>=0.6.0: setuptools
        Requirement of zc.lockfile: setuptools
        Requirement of transaction>=1.5.0: zope.interface
        Requirement of BTrees>=4.2.0: zope.interface>=5.0.0
        Requirement of BTrees>=4.2.0: persistent>=4.1.0
        Requirement of persistent>=4.2.0: cffi
        Requirement of persistent>=4.2.0: zope.interface
        Requirement of zope.event: setuptools
      Getting distribution for 'trollius'.
    Error: Picked: trollius = 2.2.1

-> Fix that by specifying which egg-version ZEO4/ZEO4-wc2 should be using.

For the reference for ZODB we already do the same - specify egg-version for all
ZODB5, ZODB4-wc2 and ZODB4:

https://lab.nexedi.com/nexedi/slapos/blob/6765c349/component/ZODB/buildout.cfg#L52-72

ZEO[test] requires ZopeUndo, but we now never pin that anywhere in e.g. stack/slapos.cfg .
Previously it was working without explicit pinning because Zope2, contrary to Zope4, pins ZopeUndo to 2.12.0.

Build of test-zodb5.cfg was failing as

    Installing wendelin.core-python.
    While:
      Installing wendelin.core-python.
      Base installation request: 'wendelin.core[test]', 'pygolang[pyx.build]', 'neoppod[tests]', 'ZEO[test]'
        Requirement of ZEO[test]==5.4.0: zope.testrunner
        Requirement of ZEO[test]==5.4.0: zdaemon
        Requirement of ZEO[test]==5.4.0: msgpack<1
        Requirement of ZEO[test]==5.4.0: mock
        Requirement of ZEO[test]==5.4.0: transaction
        Requirement of ZEO[test]==5.4.0: manuel
        Requirement of ZEO[test]==5.4.0: zope.testing
        Requirement of ZEO[test]==5.4.0: ZopeUndo
        Requirement of ZEO[test]==5.4.0: ZODB>=5.5.1
        Requirement of ZEO[test]==5.4.0: ZConfig
        Requirement of ZEO[test]==5.4.0: trollius
        Requirement of ZEO[test]==5.4.0: futures
        Requirement of ZEO[test]==5.4.0: zope.interface
        Requirement of ZEO[test]==5.4.0: zdaemon
        Requirement of ZEO[test]==5.4.0: ZConfig
        Requirement of ZEO[test]==5.4.0: zc.lockfile
        Requirement of ZEO[test]==5.4.0: persistent>=4.1.0
        Requirement of ZEO[test]==5.4.0: transaction>=2.0.3
        Requirement of ZEO[test]==5.4.0: six
        Requirement of ZEO[test]==5.4.0: ZODB>=5.1.1
        Requirement of neoppod[tests]==1.12.0: neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-pymysql,storage-sqlite]
        Requirement of neoppod[tests]==1.12.0: psutil>=2
        Requirement of neoppod[tests]==1.12.0: zope.testing
        Requirement of neoppod[tests]==1.12.0: coverage
        Requirement of neoppod[tests]==1.12.0: python-dateutil
        Requirement of neoppod[tests]==1.12.0: msgpack>=0.5.6
        Requirement of pygolang[pyx.build]==0.1: wheel
        Requirement of pygolang[pyx.build]==0.1: setuptools_dso>=1.7
        Requirement of pygolang[pyx.build]==0.1: setuptools
        Requirement of pygolang[pyx.build]==0.1: cython
        Requirement of pygolang[pyx.build]==0.1: Importing
        Requirement of pygolang[pyx.build]==0.1: decorator
        Requirement of pygolang[pyx.build]==0.1: six
        Requirement of pygolang[pyx.build]==0.1: gevent
        Requirement of wendelin.core[test]: ZEO
        Requirement of wendelin.core[test]: neoppod
        Requirement of wendelin.core[test]: scipy
        Requirement of wendelin.core[test]: pytest
        Requirement of wendelin.core[test]: psutil
        Requirement of wendelin.core[test]: six
        Requirement of wendelin.core[test]: pygolang>=0.1
        Requirement of wendelin.core[test]: zodbtools>=0.0.0.dev8
        Requirement of wendelin.core[test]: ZODB>=4
        Requirement of wendelin.core[test]: numpy
        Requirement of zope.testrunner: zope.interface
        Requirement of zope.testrunner: zope.exceptions
        Requirement of zope.testrunner: six
        Requirement of zope.testrunner: setuptools
        Requirement of zdaemon: setuptools
        Requirement of zdaemon: ZConfig
        Requirement of mock: funcsigs>=1
        Requirement of mock: six
        Requirement of transaction: zope.interface
        Requirement of manuel: six
        Requirement of manuel: setuptools
        Requirement of zope.testing: setuptools
      Getting distribution for 'ZopeUndo'.
    Error: Picked: ZopeUndo = 6.0

-> pin ZopeUndo to 5.0 - the last version that supports both py2 and py3.