1. 20 Oct, 2020 40 commits
    • Kirill Smelkov's avatar
      Move dependent egg-versions for added components to stack/slapos.cfg · 97444968
      Kirill Smelkov authored
      Move versions for eggs that component/{ZEO,pygolang,zodbtools,pytest}
      depend on out of component/ and into stack/slapos.cfg
      
      Leave version of component <X> inside component/<X>.
      
      I was asked to do so:
      
      !839 (comment 119170)
      97444968
    • Kirill Smelkov's avatar
    • Jérome Perrin's avatar
      fixup! gdb: New component · ffc50ed4
      Jérome Perrin authored
      I tried to use this gdb in a software release installed by slapos-sr-testing (which perform some extra checks that we only do in slapos-sr-testing at the moment) and it complains that:
      
      ```
      ======================================================================
      ERROR: setUpModule (test)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/srv/slapgrid/slappart9/srv/slapos/soft/24930952d96110d7e0142b49eba918a8/parts/slapos.core-repository/slapos/testing/testcase.py", line 168, in setUpModule
          installSoftwareUrlList(cls, [software_url], debug=debug)
        File "/srv/slapgrid/slappart9/srv/slapos/soft/24930952d96110d7e0142b49eba918a8/parts/slapos.core-repository/slapos/testing/testcase.py", line 378, in installSoftwareUrlList
          checkSoftware(cls.slap, software_url)
        File "/srv/slapgrid/slappart9/srv/slapos/soft/24930952d96110d7e0142b49eba918a8/parts/slapos.core-repository/slapos/testing/testcase.py", line 336, in checkSoftware
          raise RuntimeError('\n'.join(error_list))
      RuntimeError: /srv/slapgrid/slappart9/srv/slapos/inst/slappart7/tmp/shared/gdb/d14016b094c3637c4ebf6a3df4a8c64d/bin/gdb uses system library /usr/lib/x86_64-linux-gnu/libexpat.so.1.6.8 for libexpat.so.1
      
      ----------------------------------------------------------------------
      Ran 0 tests in 462.686s
      ```
      
      I saw this on a debian KVM on which I don't think I installed any system package except `slapos-node`, but `libexpat1-dev` is installed for some reason, so configure detected it and the built binary is linked against it. I checked some test nodes and `/usr/include/expat.h` seems present on some testnodes, so probably some test nodes will also have this problem. For that, I suggest to build a gdb using slapos `libexpat`, so that we build something more reproductible and not depending on the host system (that's important if we want to use binary cache, but can also be a problem if system packages get remove/upgraded). As you might be thinking now, that's a kind of infinite problem, because we can not add anything that configure might detect, but that's how we usually do for this for now.
      
      While looking at this, I also realised that we were using a .tar.xz URL here, for which rely our recipe will use `xz` command. Because some machines might not have that command, we usually put `xz-utils` in PATH.
      
      How about including something like this ?
      ffc50ed4
    • Kirill Smelkov's avatar
      wendelin.core: Add way to run tests · 805374b4
      Kirill Smelkov authored
      Just like with pygolang and zodbtools add way to run wendelin.core tests
      via nxdtest with test instance organized with help of stack/nxdtest.cfg
      
      Test agains ZODB4 and ZODB5 for the reasons explained in previous patch.
      
      Wendelin.core already had [wendelin.core-dev], so we only have to add
      test*.cfg in this patch.
      805374b4
    • Kirill Smelkov's avatar
      zodbtools: Add way to run tests · 9a5cfb73
      Kirill Smelkov authored
      Following approach used for pygolang in the previous patch lets add
      testing support for zodbtools:
      
      - Add zodbtools/buildout-dev.cfg that overrides [zodbtools] to use the
        software from git checkout.
      
      - Add zodbtools/test<X>.cfg that is software-release to create a test
        instance to be run under testnode. To help itself on this task this
        software release uses just-added stack/nxdtest.cfg and
        [python-interpreter] from pygolang, so the code in zodbtools is
        minimal.
      
      Zodbtools can be tested against both ZODB4 and ZODB5 because we still
      use ZODB4 as our primary ZODB version, but it is no longer supported by
      upstream which considers only ZODB5 as current.
      9a5cfb73
    • Kirill Smelkov's avatar
      pygolang: Add way to run tests · 662b1a0b
      Kirill Smelkov authored
      - Add pygolang/buildout-dev.cfg that overrides [pygolang] to use the
        software from git checkout.
      
      - Add pygolang/test.cfg that is software-release to create a test
        instance to be run under testnode. This software release uses
        just-added stack/nxdtest.cfg to help itself on this task, so the code
        in pygolang is minimal.
      
      A new section [python-interpreter] is added, because python interpreters
      that zc.recipe.egg generates don't process `-m args` correctly and
      handle subprocess well. I had to workaround that with code from
      gpython.pymain to be able to run `python -m pytest --<pytestarg>` and to
      spawn children processes with preserving sys.path. Comments around and
      inside [python-interpreter] has more details on this topic.
      662b1a0b
    • Kirill Smelkov's avatar
      pytest: New component · 97527e35
      Kirill Smelkov authored
      Pytest is used to test wendelin.core, zodbtools and pygolang
      It is also used in component/unstable/pyodide, but currently via pip.
      97527e35
    • Kirill Smelkov's avatar
      stack/nxdtest · 0dd3e119
      Kirill Smelkov authored
      Currently Nexedi takes mostly a copy-paste approach when it comes to
      organizing bin/runTestSuite and an instance to run this inside testnode:
      
          kirr@deco:~/src/wendelin/slapos/slapos-master$ git ls-files |grep runTestSuite
          software/build-rina/runTestSuite.in
          software/buildout-testing/runTestSuite.in
          software/cython-test/runTestSuite.in
          software/erp5testnode/testsuite/deploy-test/runTestSuite.py
          software/jstestnode/runTestSuite.in
          software/neoppod/runTestSuite.in
          software/neoppod/stress-testing/runTestSuite.in
          software/unstable/pyodide/runTestSuite.in
      
      and adding new tests to be run this way will only increase the entropy.
      
      Let's stop this, at least for new tests, and use a uniform approach without copying:
      
      1) I've created nxdtest tool[1] which allows a project to specify in
         declarative way how to test itself: given such declaration nxdtest
         takes on itself the responsibility to communicate with testnode
         master, to drive test commands spawning etc, and leaves to the
         project only semantic part that is specific to the particular
         project.
      
         Nxdtest is based on my go/neo/t/nxd/runTestSuite[2,3] and on the code
         I've seen elsewhere scattered in other runTestSuite.in files.
      
      2) Let's add stack/nxdtest.cfg to SlapOS that provides easy way for a
         software or component to organize testing of itself.
      
      This patch adds stack/nxdtest and migrates software/neotest to use it
      (because stack/nxdtest is based on it and this way it helps to track the
      changes).
      
      In the following patches we will add testing support for pygolang,
      zodbtools and wendelin.core .
      
      [1] https://lab.nexedi.com/kirr/nxdtest/blob/master/nxdtest
      [2] kirr/neo@51b18490
      [3] kirr/neo@f67c147d
      0dd3e119
    • Kirill Smelkov's avatar
      golang: Provide ${go:exe} · fee8b968
      Kirill Smelkov authored
      ${go:exe} is standalone executable that runs go in activated gowork environment.
      It is handy to use in recipes where one cannot source ${gowork:env} -
      in particular parts with recipe=zc.recipe.egg
      
      Wendelin.core 2 will use `GO=${go:exe}` in its build environment.
      fee8b968
    • Kirill Smelkov's avatar
      53e050f3
    • Kirill Smelkov's avatar
      wendelin.core: Add pygolang · 8ffe80c1
      Kirill Smelkov authored
      Starting from 0.14 (aka 2) wendelin.core requires pygolang to be present
      both at runtime and at build time.
      8ffe80c1
    • Kirill Smelkov's avatar
      neotest: v↑ everything *.go · ff633c08
      Kirill Smelkov authored
      Update lab.nexedi.com/kirr/neo.git and lab.nexedi.com/kirr/go123 to latest.
      Update all other go packages.
      
      The update puls in the following new dependencies:
      
          github.com/shamaton/msgpack
          github.com/tinylib/msgp
            github.com/philhofer/fwd
            github.com/ttacon/chalk
            golang.org/x/mod
            golang.org/x/xerrors
      
          github.com/davecgh/go-spew
          github.com/pmezard/go-difflib
          gopkg.in/yaml.v3
      ff633c08
    • Kirill Smelkov's avatar
    • Kirill Smelkov's avatar
      zodbtools: v↑ (0.0.0.dev4 -> 0.0.0.dev8) · 4e45e0c5
      Kirill Smelkov authored
      Going from 0.0.0.dev4 to -> 0.0.0.dev8 zodbtools:
      
      - Stabilized `zodb dump` format and layed ground for `zodb restore`;
      - Taught `zodb analyze` to work with any ZODB storage (instead of being
        FileStorage-only tool) and to analyze a particular range of history
        (instead of crunching data for days on a large database);
      - Added `zodb commit` tool that is handy in testing;
      - Added ability to specify tid ranges in human-readable format, as in
        e.g. `zodb analyze data.fs 2018-01-01T10:30:00Z..yesterday`.
      - Progressed on Python3 support.
      
      See https://pypi.org/project/zodbtools/#zodbtools-change-history
      for details.
      4e45e0c5
    • Kirill Smelkov's avatar
      zodbtools: v↑ zodburi (2.3.0 -> 2.4.0) · fcfa3195
      Kirill Smelkov authored
      Close to non-functional change, but removes PendingDeprecationWarning
      about cgi.parse_qsl
      fcfa3195
    • Kirill Smelkov's avatar
      pygolang: v↑ (0.0.0.dev4 -> 0.0.7.post1) · e1d269b4
      Kirill Smelkov authored
      0.0.0.dev4 to 0.0.7.post1 goes a long way. See
      https://pypi.org/project/pygolang/#pygolang-change-history for details.
      
      Recent pygolang is needed for wendelin.core 2.
      Zodbtools also uses it starting from v0.0.0.dev5.
      e1d269b4
    • Kirill Smelkov's avatar
      6b9c863b
    • Kirill Smelkov's avatar
      e83edfc9
    • Kirill Smelkov's avatar
      ZEO: Factor it to separate component · 83618cd6
      Kirill Smelkov authored
      We already patch ZEO4 with TCP_NODELAY patch (see 5cf4cf1f "ERP5:
      enable TCP_NODELAY for ZEO") and we will need to backport more patches
      to ZEO4 branch for wendelin.core 2 to work correctly.
      
      It's not only software/neoppod which uses ZEO, and it is not convenient for
      all other software-releases to inherit from neoppod to use correct
      version and build of ZEO egg. For this reason factor out details of ZEO
      egg building into component/ZEO and let users use ${ZEO:egg} where ZEO
      is needed. This way ZEO will be correctly installed for all users.
      
      This patch should be a non-functional change. We switch to
      ZEO@5114f909 revision which corresponds to ZEO 4.3.1 +
      TCP_NODELAY.patch
      
      Adding other patches to ZEO4 needed by wendelin.core 2 will be done as a
      separate step.
      83618cd6
    • Thomas Gambier's avatar
      slapos: version up slapos.toolbox 0.112 · a8c0eb15
      Thomas Gambier authored
      a8c0eb15
    • Łukasz Nowak's avatar
      Feature/caddy frontend improvement 202010 · d1334b4b
      Łukasz Nowak authored
      See merge request nexedi/slapos!836
      d1334b4b
    • Łukasz Nowak's avatar
      ff9a04e2
    • Łukasz Nowak's avatar
      caddy-frontend: Avoid stop-on-error during instantiation · bc594e1c
      Łukasz Nowak authored
      stop-on-error during instantiation can lead to endless instantiation in some
      cases, which disallows to create services required for given part to pass, and
      in the same time in many cases the called scripts are smart enough to continue
      and restart on error.
      bc594e1c
    • Łukasz Nowak's avatar
      f7021142
    • Łukasz Nowak's avatar
      caddy-frontend: Name backend explicitly in haproxy backend · 49cce32f
      Łukasz Nowak authored
      Thanks to this, it's much easier to find relevant configuration, and also logs
      are becoming much more informative.
      49cce32f
    • Łukasz Nowak's avatar
      caddy-frontend: Improve generated haproxy backend configuration · 1cc2ae69
      Łukasz Nowak authored
      Avoid creating needless entries and minimise needless newlines, and reflect
      this in test data, as in such case no backend_log is created, as nothing is
      configured on backend Haproxy level.
      1cc2ae69
    • Łukasz Nowak's avatar
    • Łukasz Nowak's avatar
      cd202751
    • Łukasz Nowak's avatar
      caddy-frontend: Drop not used parameters · 63828339
      Łukasz Nowak authored
      SSL cache is unusable in context of caddy, same for the so called HTTPD home.
      Those are old Apache leftovers.
      63828339
    • Łukasz Nowak's avatar
      0470062a
    • Łukasz Nowak's avatar
      9071e3fd
    • Łukasz Nowak's avatar
      37fe226a
    • Łukasz Nowak's avatar
      2412f59b
    • Łukasz Nowak's avatar
      446b8e84
    • Łukasz Nowak's avatar
      caddy-frontend: Simplify instance.cfg.in · 4146eef0
      Łukasz Nowak authored
      4146eef0
    • Łukasz Nowak's avatar
      caddy-frontend: Use instance_parameter_dict name · 61ebfefe
      Łukasz Nowak authored
      It's a dict, and in SlapOS usage of Jinja2 it's good to see the type of
      a variable immediately.
      61ebfefe
    • Łukasz Nowak's avatar
      caddy-frontend: Name passed parameter correctly · 041a49c3
      Łukasz Nowak authored
      "parameter_dict" says nothing, whereas "software_parameter_dict" explains
      source and purpose of the information.
      041a49c3
    • Łukasz Nowak's avatar
    • Łukasz Nowak's avatar
      caddy-frontend: Reuse passed parameter_dict from instance.cfg.in · edcf83eb
      Łukasz Nowak authored
      There is needless duplication of information.
      edcf83eb
    • Łukasz Nowak's avatar
      caddy-frontend: Share parameter_dict · 5e720d5b
      Łukasz Nowak authored
      5e720d5b