1. 21 Oct, 2020 2 commits
    • Kirill Smelkov's avatar
      . · 73b9c447
      Kirill Smelkov authored
      73b9c447
    • Kirill Smelkov's avatar
      X go/neo: tests: Terminate spawned processses with SIGTERM instead of SIGKILL · e9af596e
      Kirill Smelkov authored
      For example NEOCluster needs to shutdown gracefully, else there are
      processes left for e.g. storage nodes and they dump somthing as below on
      the terminal after tests completion:
      
          === RUN   TestLoad
          2020/10/21 14:33:00 zodb: FIXME: open ../zodb/storage/fs1/testdata/1.fs: raw cache is not ready for invalidations -> NoCache forced
          === RUN   TestLoad/py
          I: runneo.py: /tmp/neo445013868/1: Started master(s): 127.0.0.1:24661
          WARNING: This is not the recommended way to import data to NEO: you should use the Importer backend instead.
          NEO also does not implement IStorageRestoreable interface, which means that undo information is not preserved when using this tool: conflict resolution could happen when undoing an old transaction.
          Migrating from ../zodb/storage/fs1/testdata/1.fs to 127.0.0.1:24661
          Migration done in 0.19877
          --- PASS: TestLoad (0.75s)
              --- PASS: TestLoad/py (0.74s)
          PASS
          ok      lab.nexedi.com/kirr/neo/go/neo  0.749s
          (neo) (z-dev) (g.env) kirr@deco:~/src/neo/src/lab.nexedi.com/kirr/neo/go/neo$ Traceback (most recent call last):
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/tests/functional/__init__.py", line 182, in start
              getattr(neo.scripts,  command).main()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/scripts/neostorage.py", line 66, in main
              app.run()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 147, in run
              self._run()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 178, in _run
              self.doOperation()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 266, in doOperation
              poll()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 87, in _poll
              self.em.poll(1)
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 155, in poll
              self._poll(blocking)
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 253, in _poll
              timeout_object.onTimeout()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 259, in onTimeout
              on_timeout()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/manager.py", line 207, in _deferredCommit
              self.commit()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/manager.py", line 193, in commit
              self._commit()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/sqlite.py", line 90, in _commit
              retry_if_locked(self.conn.commit)
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/sqlite.py", line 45, in retry_if_locked
              return f(*args)
          OperationalError: disk I/O error
      e9af596e
  2. 20 Oct, 2020 2 commits
  3. 19 Oct, 2020 2 commits
    • Kirill Smelkov's avatar
      X go/neo: NEO/py represents data deletion by returning data=None · cd5bbb87
      Kirill Smelkov authored
      This fixes
      
      	=== RUN   TestLoad/py
      	    xtesting.go:297: load 0285cbacc06d3a4c:0000000000000007: returned err unexpected:
      	        have: <nil>
      	        want: neo://1@127.0.0.1:27145: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
      	    xtesting.go:297: load 0285cbacc06d3a4c:0000000000000007: returned tid unexpected: 0285cbacc06d3a4c  ; want: 0285cbacc06d3a4c
      	    xtesting.go:297: load 0285cbacc06d3a4c:0000000000000007: returned buf != nil
      	    xtesting.go:297: load 0285cbad858bf2e6:0000000000000006: returned err unexpected:
      	        have: <nil>
      	        want: neo://1@127.0.0.1:27145: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
      	    xtesting.go:297: load 0285cbad858bf2e6:0000000000000006: returned tid unexpected: 0285cbad858bf2e6  ; want: 0285cbad858bf2e6
      	    xtesting.go:297: load 0285cbad858bf2e6:0000000000000006: returned buf != nil
      	    xtesting.go:315: load 7fffffffffffffff:0000000000000006: returned err unexpected:
      	        have: <nil>
      	        want: neo://1@127.0.0.1:27145: load 7fffffffffffffff:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
      	    xtesting.go:315: load 7fffffffffffffff:0000000000000006: returned tid unexpected: 0285cbad858bf2e6  ; want: 0285cbad858bf2e6
      	    xtesting.go:315: load 7fffffffffffffff:0000000000000006: returned buf != nil
      	    xtesting.go:315: load 7fffffffffffffff:0000000000000007: returned err unexpected:
      	        have: <nil>
      	        want: neo://1@127.0.0.1:27145: load 7fffffffffffffff:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
      	    xtesting.go:315: load 7fffffffffffffff:0000000000000007: returned tid unexpected: 0285cbacc06d3a4c  ; want: 0285cbacc06d3a4c
      	    xtesting.go:315: load 7fffffffffffffff:0000000000000007: returned buf != nil
      cd5bbb87
    • Kirill Smelkov's avatar
      X go/neo: Switch to NEO-kind SHA1 · bc2cddfc
      Kirill Smelkov authored
      NEO/py sends all zeros for empty data. Without this fix tests fail:
      
          --- FAIL: TestLoad/py (0.74s)
              xtesting.go:297: load 0285cbacc06d3a4c:0000000000000007: returned err unexpected:
                  have: neo://1@127.0.0.1:25910: load 0285cbacc06d3a4c:0000000000000007: data corrupt: checksum mismatch
                  want: neo://1@127.0.0.1:25910: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
              xtesting.go:297: load 0285cbad858bf2e6:0000000000000006: returned err unexpected:
                  have: neo://1@127.0.0.1:25910: load 0285cbad858bf2e6:0000000000000006: data corrupt: checksum mismatch
                  want: neo://1@127.0.0.1:25910: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
              xtesting.go:315: load 7fffffffffffffff:0000000000000006: returned err unexpected:
                  have: neo://1@127.0.0.1:25910: load 7fffffffffffffff:0000000000000006: data corrupt: checksum mismatch
                  want: neo://1@127.0.0.1:25910: load 7fffffffffffffff:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
              xtesting.go:315: load 7fffffffffffffff:0000000000000007: returned err unexpected:
                  have: neo://1@127.0.0.1:25910: load 7fffffffffffffff:0000000000000007: data corrupt: checksum mismatch
                  want: neo://1@127.0.0.1:25910: load 7fffffffffffffff:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
      bc2cddfc
  4. 18 Oct, 2020 17 commits
  5. 16 Oct, 2020 5 commits
    • Kirill Smelkov's avatar
      . · 6410356a
      Kirill Smelkov authored
      6410356a
    • Kirill Smelkov's avatar
      . · 8b37a895
      Kirill Smelkov authored
      8b37a895
    • Kirill Smelkov's avatar
      Merge branch 'master' into t · bd1e3d27
      Kirill Smelkov authored
      * master:
        go/zodb/zeo: Skip handshake/py/msgpack=true on ZEO4
      bd1e3d27
    • Kirill Smelkov's avatar
      go/zodb/zeo: Skip handshake/py/msgpack=true on ZEO4 · 6d9a8046
      Kirill Smelkov authored
      ZEO4 does not have msgpack support and does not take $ZEO_MSGPACK into
      account. With ZEO4 this test was failing before:
      
      	--- FAIL: TestHandshake (0.46s)
      	    --- FAIL: TestHandshake/py/msgpack=true (0.24s)
      	        zeo_test.go:241: handshake: encoding=Z  ; want M
      
      We don't have infrastructure to check python packages versions, so
      check it by verifying ZEO.asyncio presence.
      6d9a8046
    • Julien Muchembled's avatar
      Fix a few issues with ZODB5 · d7f3bc57
      Julien Muchembled authored
      In the Importer storage backend, the repickler code never really worked with
      ZODB 5 (use of protocol > 1), and now the test does not pass anymore.
      
      The other issues caused by ZODB commit 12ee41c47310156027a674932df34b60de86ba36
      are fixed:
      
        TypeError: list indices must be integers, not binary
      
        ValueError: unsupported pickle protocol: 3
      
      Although not necessary as long as we don't support Python 3,
      this commit also replaces `str` by `bytes` in a few places.
      d7f3bc57
  6. 13 Oct, 2020 2 commits
  7. 12 Oct, 2020 3 commits
    • Kirill Smelkov's avatar
      Merge branch 'master' into t · 28494187
      Kirill Smelkov authored
      * master:
        go/zodb/zeo: Fix initialization when server database is empty
        neotest: info-local: Fix NIC display for virtio devices
        neotest: info-local: Fix it when CPU frequence/idle information is not available
      28494187
    • Kirill Smelkov's avatar
      go/zodb/zeo: Fix initialization when server database is empty · 1df5acab
      Kirill Smelkov authored
      In that case at0 was initialized as 0 and still considered uninitialized
      by flushEventq0:
      
          (neo) (z-dev) (g.env) kirr@deco:~/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo$ go test -run Empty
          ------
          2020-10-12T07:39:25 INFO ZEO.runzeo (146240) opening storage '1' using FileStorage
          ------
          2020-10-12T07:39:25 INFO ZEO.StorageServer StorageServer created RW with storages: 1:RW:/tmp/zeo905263273/1.fs
          ------
          2020-10-12T07:39:25 INFO ZEO.asyncio.server listening on /tmp/zeo905263273/1.fs.zeosock
          ------
          2020-10-12T07:39:25 INFO ZEO.asyncio.base Connected server protocol
          ------
          2020-10-12T07:39:25 INFO ZEO.asyncio.server received handshake 'Z5'
          2020/10/12 07:39:25 /tmp/zeo905263273/1.fs.zeosock: EOF
          --- FAIL: TestEmptyDB (0.22s)
              --- FAIL: TestEmptyDB/py/msgpack=false (0.22s)
          panic: flush, but .at0 not yet initialized [recovered]
                  panic: flush, but .at0 not yet initialized
      
          goroutine 7 [running]:
          testing.tRunner.func1.1(0x644a60, 0x6e1a50)
                  /home/kirr/src/tools/go/go/src/testing/testing.go:1072 +0x30d
          testing.tRunner.func1(0xc000001e00)
                  /home/kirr/src/tools/go/go/src/testing/testing.go:1075 +0x41a
          panic(0x644a60, 0x6e1a50)
                  /home/kirr/src/tools/go/go/src/runtime/panic.go:969 +0x175
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.(*zeo).flushEventq0(0xc00018a000)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo.go:180 +0xf3
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.openByURL(0x6e9ca0, 0xc000016108, 0xc000138120, 0xc000153d98, 0x0, 0x0, 0x0, 0x0, 0x0)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo.go:488 +0x5ba
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.zeoOpen(0xc000018740, 0x1e, 0xc000049d98, 0x0, 0x0, 0x0, 0x0)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:285 +0x17b
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.withZEO.func1(0xc000001e00, 0x6e9ea0, 0xc00005e6c0)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:219 +0xd0
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.withZEOSrv.func2.1(0xc0000185c0, 0x16)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:205 +0xfb
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.withZEOSrv.func1(0xc000001e00, 0xc00000e5a0)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:185 +0x129
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.withZEOSrv.func2(0xc000001e00)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:197 +0x105
          testing.tRunner(0xc000001e00, 0xc00000e440)
                  /home/kirr/src/tools/go/go/src/testing/testing.go:1123 +0xef
          created by testing.(*T).Run
                  /home/kirr/src/tools/go/go/src/testing/testing.go:1168 +0x2b3
          exit status 2
          FAIL    lab.nexedi.com/kirr/neo/go/zodb/storage/zeo     0.227s
      
      -> Fix it by using dedicated field marking whether .at0 was initialized or not yet.
      1df5acab
    • Kirill Smelkov's avatar
      go/zodb/zeo: Fix initialization when server database is empty · dcdf98c0
      Kirill Smelkov authored
      In that case at0 was initialized as 0 and still considered uninitialized
      by flushEventq0:
      
          (neo) (z-dev) (g.env) kirr@deco:~/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo$ go test -run Empty
          ------
          2020-10-12T07:39:25 INFO ZEO.runzeo (146240) opening storage '1' using FileStorage
          ------
          2020-10-12T07:39:25 INFO ZEO.StorageServer StorageServer created RW with storages: 1:RW:/tmp/zeo905263273/1.fs
          ------
          2020-10-12T07:39:25 INFO ZEO.asyncio.server listening on /tmp/zeo905263273/1.fs.zeosock
          ------
          2020-10-12T07:39:25 INFO ZEO.asyncio.base Connected server protocol
          ------
          2020-10-12T07:39:25 INFO ZEO.asyncio.server received handshake 'Z5'
          2020/10/12 07:39:25 /tmp/zeo905263273/1.fs.zeosock: EOF
          --- FAIL: TestEmptyDB (0.22s)
              --- FAIL: TestEmptyDB/py/msgpack=false (0.22s)
          panic: flush, but .at0 not yet initialized [recovered]
                  panic: flush, but .at0 not yet initialized
      
          goroutine 7 [running]:
          testing.tRunner.func1.1(0x644a60, 0x6e1a50)
                  /home/kirr/src/tools/go/go/src/testing/testing.go:1072 +0x30d
          testing.tRunner.func1(0xc000001e00)
                  /home/kirr/src/tools/go/go/src/testing/testing.go:1075 +0x41a
          panic(0x644a60, 0x6e1a50)
                  /home/kirr/src/tools/go/go/src/runtime/panic.go:969 +0x175
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.(*zeo).flushEventq0(0xc00018a000)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo.go:180 +0xf3
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.openByURL(0x6e9ca0, 0xc000016108, 0xc000138120, 0xc000153d98, 0x0, 0x0, 0x0, 0x0, 0x0)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo.go:488 +0x5ba
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.zeoOpen(0xc000018740, 0x1e, 0xc000049d98, 0x0, 0x0, 0x0, 0x0)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:285 +0x17b
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.withZEO.func1(0xc000001e00, 0x6e9ea0, 0xc00005e6c0)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:219 +0xd0
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.withZEOSrv.func2.1(0xc0000185c0, 0x16)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:205 +0xfb
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.withZEOSrv.func1(0xc000001e00, 0xc00000e5a0)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:185 +0x129
          lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.withZEOSrv.func2(0xc000001e00)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:197 +0x105
          testing.tRunner(0xc000001e00, 0xc00000e440)
                  /home/kirr/src/tools/go/go/src/testing/testing.go:1123 +0xef
          created by testing.(*T).Run
                  /home/kirr/src/tools/go/go/src/testing/testing.go:1168 +0x2b3
          exit status 2
          FAIL    lab.nexedi.com/kirr/neo/go/zodb/storage/zeo     0.227s
      
      -> Fix it by using dedicated field marking whether .at0 was initialized or not yet.
      dcdf98c0
  8. 30 Sep, 2020 1 commit
  9. 29 Sep, 2020 1 commit
  10. 27 Sep, 2020 1 commit
  11. 25 Sep, 2020 4 commits