- 19 Jan, 2022 2 commits
-
-
Kirill Smelkov authored
* master: wcfs: Setup basic logging for warnings/errors to go to stderr when invoked as e.g. `wcfs serve` wcfs: Fix crash if on invalidation handledδZ needs to access ZODB t/qemu-runlinux: Use multidevs=remaps for 9P setup fixup! *: Use defer for dbclose & friends wcfs: client: tests: Turn SIGSEGV in tMapping.assertBlk into exception wcfs: Server.stop: Don't log "after SIGTERM" when first wait for wcfs.go exit failed wcfs: Server.stop: Don't report first unmount failure to outside
-
Kirill Smelkov authored
If we do not setup logging explicitly, it will print only No handlers could be found for logger "wcfs" instead of emitting useful details in before e.g. RuntimeError: fuse_unmount /dev/shm/wcfs/<X>: failed: fusermount: failed to unmount /dev/shm/wcfs/<X>: Device or resource busy (more details logged) -> Fix it.
-
- 18 Jan, 2022 1 commit
-
-
Kirill Smelkov authored
The invalidation logic is generally right, but invalidateBlk -> ΔFtail.BlkRevAt was being called with ctx without transaction. As the result it was panicking as panic: transaction: no current transaction goroutine 41 [running]: lab.nexedi.com/kirr/neo/go/transaction.currentTxn({0x9696d8, 0xc0000d8080}) /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/transaction/transaction.go:59 +0x77 lab.nexedi.com/kirr/neo/go/transaction.Current(...) /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/transaction/api.go:206 lab.nexedi.com/kirr/neo/go/zodb.(*Connection).checkTxnCtx(...) /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/connection.go:374 lab.nexedi.com/kirr/neo/go/zodb.(*Connection).Get(0xc00010c640, {0x9696d8, 0xc0000d8080}, 0x4) /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/connection.go:331 +0x73 lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/zdata.(*ΔFtail).BlkRevAt(0xc000077d40, {0x9696d8, 0xc0000d8080}, 0xc000064f60, 0x0, 0x3e5983329bbd100) /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/zdata/δftail.go:1140 +0x39d main.(*BigFile).invalidateBlk.func1(0xc000164400, {0x9696d8, 0xc0000d8080}, 0xc0005a0000, 0x200000, 0x200000, {0xc0005a0000, 0x200000, 0x200000}) /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/wcfs.go:1089 +0xb8 main.(*BigFile).invalidateBlk(0xc000164400, {0x9696d8, 0xc0000d8080}, 0x0) /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/wcfs.go:1105 +0x3bb main.(*Root).handleδZ.func3({0x9696d8, 0xc0000d8080}) /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/wcfs.go:898 +0x34 lab.nexedi.com/kirr/go123/xsync.(*WorkGroup).Go.func1() /home/kirr/src/neo/src/lab.nexedi.com/kirr/go123/xsync/xsync.go:86 +0x68 created by lab.nexedi.com/kirr/go123/xsync.(*WorkGroup).Go /home/kirr/src/neo/src/lab.nexedi.com/kirr/go123/xsync/xsync.go:83 +0x92 on any new change to tracked file block whose previous history is not covered by ΔFtail/ΔBtail. Problem reported by @Francois.
-
- 26 Nov, 2021 1 commit
-
-
Kirill Smelkov authored
Fixes the following warning that started to appear: kirr@deca:~/src/wendelin/wendelin.core/t$ ./qemu-runlinux -g /home/kirr/src/linux/obj-qemu_debug/arch/x86/boot/bzImage /bin/bash qemu-system-x86_64: warning: 9p: Multiple devices detected in same VirtFS export, which might lead to file ID collisions and severe misbehaviours on guest! You should either use a separate export for each device shared from host or use virtfs option 'multidevs=remap'! See https://wiki.qemu.org/Documentation/9psetup for documentation of multidevs option.
-
- 23 Nov, 2021 4 commits
-
-
Kirill Smelkov authored
In 5c8340d2 we said: dbclose now uses defer almost everywhere - there are still few places in tests, where one test function is opening/closing test database multiple times - those were not (yet ?) converted. Let's convert those remaining places now, because when wendelin.core tests are run wrt plain ZODB4 (contrary to ZODB4-wc2), many tests fail at fileh_open time, e.g. @func def test_bigfile_filezodb_fileh_gc(): root1= dbopen() conn1= root1._p_jar db = conn1.db() defer(db.close) root1['zfile4'] = f1 = ZBigFile(blksize) transaction.commit() > fh1 = f1.fileh_open() bigfile/tests/test_filezodb.py:588: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ bigfile/file_zodb.py:603: in fileh_open fileh = _ZBigFileH(self, _use_wcfs) bigfile/file_zodb.py:664: in __init__ self.zfileh = zfile._v_file.fileh_open(use_wcfs) bigfile/_file_zodb.pyx:112: in wendelin.bigfile._file_zodb._ZBigFile.fileh_open pywconn = wczsync.pywconnOf(zconn) wcfs/client/_wczsync.pyx:56: in wendelin.wcfs.client._wczsync.pywconnOf wconn = wc.connect(zconn_at(zconn)) lib/zodb.py:163: in zconn_at "ZODB!1") _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ patch = 'conn:MVCC-via-loadBefore-only', details_link = 'ZODB!1' def _zassertHasNXDPatch(patch, details_link): if not _zhasNXDPatch(patch): raise AssertionError( "ZODB%s is not patched with required Nexedi patch %r\n\tSee %s for details" % > (zmajor, patch, details_link)) E AssertionError: ZODB4 is not patched with required Nexedi patch 'conn:MVCC-via-loadBefore-only' E See ZODB!1 for details and DB is left unclosed. This change should reduce, if not completely fix, the number of leaked /tmp/testdb_* directories for Wendelin.core.UnitTest-ZODB4(xfail) testsuite.
-
Kirill Smelkov authored
When WCFS-mmapped memory is accessed, it can get SIGBUS on IO error (and automatically on WCFS crash), and SIGSEGV when accessed client mapping is closed. tFile.assertBlk in wcfs_test.py already converts SIGSEGV into python exception when accessing on-wcfs file's block. However tMapping.assertBlk was not doing so, which, instead of providing proper details, leads to test crashes if something goes wrong. For example when wendelin.core tests are run wrt plain ZODB4 (contrary to ZODB4-wc2, see ZODB!1 and slapos@e256ed97), it first fails in pinner and then gets SIGSEGV on data access, because, to mimic SIGBUS on EIO, pinner shutdowns all mappings on its failure: https://lab.nexedi.com/nexedi/wendelin.core/blob/49f826b1/wcfs/client/wcfs.cpp#L477-501 https://nexedijs.erp5.net/#/test_result_module/20211118-7C45220A/25 -> Fix it by wrapping test block access with appropriate read_exfault variant. Before this patch: .../wendelin.core$ WENDELIN_CORE_TEST_DB='<zeo>' WENDELIN_CORE_VIRTMEM='r:wcfs+w:uvmm' python -m pytest -vsx wcfs/ -k test_wcfs_client ... wcfs/client/client_test.py::test_wcfs_client -------------------- live log call --------------------- INFO wcfs:__init__.py:293 starting for zeo://localhost:28866 ... I1122 19:17:14.376182 110032 wcfs.go:2384] start "/dev/shm/wcfs/ef87339c054c3e0e48d494fa584bb209518844b2" "zeo://localhost:28866" I1122 19:17:14.376291 110032 wcfs.go:2390] (built with go1.17.3) W1122 19:17:14.380882 110032 storage.go:152] zodb: FIXME: open zeo://localhost:28866: raw cache is not ready for invalidations -> NoCache forced INFO wcfs:__init__.py:334 started pid110032 @ /dev/shm/wcfs/ef87339c054c3e0e48d494fa584bb209518844b2 M: commit -> @at0 (03e452313dddbc00) M: commit -> @at1 (03e452313e0f3b99) M: f<0000000000000002> [2, 3] M: commit -> @at2 (03e452313e1adb55) M: f<0000000000000002> [2] M: commit -> @at3 (03e452313e3be500) M: f<0000000000000002> [3, 4] W1122 19:17:14.597654 110032 wcfs.go:2050] /@03e452313d343c88/bigfile: lookup "0000000000000002": bigfopen 0000000000000002 @03e452313d343c88: invalid argument: Get 0000000000000002: Get 03e452313d343c88:0000000000000002: zeo://localhost:28866: load 03e452313d343c88:0000000000000002: 0000000000000002: no such object E1122 19:17:14.597759 110032 wcfs.go:1220] /head/bigfile/0000000000000002: readblk #4: pin watchers: wlink1: f<0000000000000002>: pin #4 @03e452313d343c88: expect "ack"; got "nak: _remmapblk #4 @03e452313d343c88: open /dev/shm/wcfs/ef87339c054c3e0e48d494fa584bb209518844b2/@03e452313d343c88/bigfile/0000000000000002: Invalid argument" F1122 19:17:14.597803 110050 wcfs/client/wcfs.cpp:487] CRITICAL: pinner: pin f<0000000000000002> #4 @03e452313d343c88: _remmapblk #4 @03e452313d343c88: open /dev/shm/wcfs/ef87339c054c3e0e48d494fa584bb209518844b2/@03e452313d343c88/bigfile/0000000000000002: Invalid argument F1122 19:17:14.597835 110050 wcfs/client/wcfs.cpp:488] CRITICAL: wcfs server will likely kill us soon. CRITICAL: pinner: pin f<0000000000000002> #4 @03e452313d343c88: _remmapblk #4 @03e452313d343c88: open /dev/shm/wcfs/ef87339c054c3e0e48d494fa584bb209518844b2/@03e452313d343c88/bigfile/0000000000000002: Invalid argument CRITICAL: wcfs server will likely kill us soon. Segmentation fault: read @00007ff7b9534000 /home/kirr/src/wendelin/wendelin.core/wcfs/client/./../../bigfile/liblibvirtmem.so(dump_traceback+0x34)[0x7ff7d6b5c279] /home/kirr/src/wendelin/wendelin.core/wcfs/client/./../../bigfile/liblibvirtmem.so(+0x27b0)[0x7ff7d6b577b0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140)[0x7ff7da078140] python(PyString_FromStringAndSize+0x228)[0x5627feb96b58] python(PyEval_EvalFrameEx+0x603e)[0x5627febb7a4e] python(PyEval_EvalCodeEx+0x57c)[0x5627febb03cc] ... python(PyObject_Call+0x43)[0x5627feb9d903] python(+0x18a7e1)[0x5627fec5d7e1] python(Py_Main+0x3ad)[0x5627fec4b8ed] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7ff7d9d59d0a] python(_start+0x2a)[0x5627fec4b46a] Ошибка сегментирования (стек памяти сброшен на диск) After this patch: .../wendelin.core$ WENDELIN_CORE_TEST_DB='<zeo>' WENDELIN_CORE_VIRTMEM='r:wcfs+w:uvmm' python -m pytest -vsx wcfs/ -k test_wcfs_client ... wcfs/client/client_test.py::test_wcfs_client -------------------- live log call --------------------- INFO wcfs:__init__.py:293 starting for zeo://localhost:22854 ... I1122 18:17:22.486445 102541 wcfs.go:2384] start "/dev/shm/wcfs/c818c147676f8d6f3b408b02f727aca5e3229e98" "zeo://localhost:22854" I1122 18:17:22.486525 102541 wcfs.go:2390] (built with go1.17.3) W1122 18:17:22.489908 102541 storage.go:152] zodb: FIXME: open zeo://localhost:22854: raw cache is not ready for invalidations -> NoCache forced INFO wcfs:__init__.py:334 started pid102541 @ /dev/shm/wcfs/c818c147676f8d6f3b408b02f727aca5e3229e98 M: commit -> @at0 (03e451f560834477) M: commit -> @at1 (03e451f560a2aa77) M: f<0000000000000002> [2, 3] M: commit -> @at2 (03e451f560adafcc) M: f<0000000000000002> [2] M: commit -> @at3 (03e451f560d02111) M: f<0000000000000002> [3, 4] W1122 18:17:22.703710 102541 wcfs.go:2050] /@03e451f55fcc4c77/bigfile: lookup "0000000000000002": bigfopen 0000000000000002 @03e451f55fcc4c77: invalid argument: Get 0000000000000002: Get 03e451f55fcc4c77:0000000000000002: zeo://localhost:22854: load 03e451f55fcc4c77:0000000000000002: 0000000000000002: no such object E1122 18:17:22.703840 102541 wcfs.go:1220] /head/bigfile/0000000000000002: readblk #4: pin watchers: wlink1: f<0000000000000002>: pin #4 @03e451f55fcc4c77: expect "ack"; got "nak: _remmapblk #4 @03e451f55fcc4c77: open /dev/shm/wcfs/c818c147676f8d6f3b408b02f727aca5e3229e98/@03e451f55fcc4c77/bigfile/0000000000000002: Invalid argument" F1122 18:17:22.704380 102558 wcfs/client/wcfs.cpp:487] CRITICAL: pinner: pin f<0000000000000002> #4 @03e451f55fcc4c77: _remmapblk #4 @03e451f55fcc4c77: open /dev/shm/wcfs/c818c147676f8d6f3b408b02f727aca5e3229e98/@03e451f55fcc4c77/bigfile/0000000000000002: Invalid argument F1122 18:17:22.704639 102558 wcfs/client/wcfs.cpp:488] CRITICAL: wcfs server will likely kill us soon. CRITICAL: pinner: pin f<0000000000000002> #4 @03e451f55fcc4c77: _remmapblk #4 @03e451f55fcc4c77: open /dev/shm/wcfs/c818c147676f8d6f3b408b02f727aca5e3229e98/@03e451f55fcc4c77/bigfile/0000000000000002: Invalid argument CRITICAL: wcfs server will likely kill us soon. >>> Change history by file: f<0000000000000002>: 0 1 2 3 4 5 6 7 a b c d e f g h @at0 (03e451f560834477) @at1 (03e451f560a2aa77) 2 3 @at2 (03e451f560adafcc) 2 @at3 (03e451f560d02111) 3 4 INFO wcfs:__init__.py:400 unmount/stop wcfs pid102541 @ /dev/shm/wcfs/c818c147676f8d6f3b408b02f727aca5e3229e98 I1122 18:17:22.728452 102541 wcfs.go:2560] stop "/dev/shm/wcfs/c818c147676f8d6f3b408b02f727aca5e3229e98" "zeo://localhost:22854" FAILED ======================= FAILURES ======================= ___________________ test_wcfs_client ___________________ @func def test_wcfs_client(): t = tDB(); zf = t.zfile; at0=t.at0 defer(t.close) pinned = lambda fh: fhpinned(t, fh) at1 = t.commit(zf, {2:'c1', 3:'d1'}) at2 = t.commit(zf, {2:'c2'}) wconn = t.wc.connect(at1) defer(wconn.close) fh = wconn.open(zf._p_oid) defer(fh.close) # create mmap with 1 block beyond file size m1 = fh.mmap(2, 3) defer(m1.unmap) assert m1.blk_start == 2 assert m1.blk_stop == 5 assert len(m1.mem) == 3*zf.blksize tm1 = tMapping(t, m1) assert pinned(fh) == {} # verify initial data reads tm1.assertBlk(2, 'c1', {2:at1}) tm1.assertBlk(3, 'd1', {2:at1}) tm1.assertBlk(4, '', {2:at1}) # commit with growing file size -> verify data read as the same, #3 pinned. # (#4 is not yet pinned because it was not accessed) at3 = t.commit(zf, {3:'d3', 4:'e3'}) assert pinned(fh) == {2:at1} tm1.assertBlk(2, 'c1', {2:at1}) tm1.assertBlk(3, 'd1', {2:at1, 3:at1}) tm1.assertBlk(4, '', {2:at1, 3:at1}) # resync at1 -> at2: #2 must unpin to @head; #4 must stay as zero wconn.resync(at2) assert pinned(fh) == {3:at1} tm1.assertBlk(2, 'c2', { 3:at1}) tm1.assertBlk(3, 'd1', { 3:at1}) > tm1.assertBlk(4, '', { 3:at1, 4:at0}) # XXX at0->ø ? wcfs/client/client_test.py:158: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ wcfs/client/client_test.py:86: in assertBlk _ = read_exfault_withgil(blkview[0:1]) wcfs/internal/wcfs_test.pyx:90: in wendelin.wcfs.internal.wcfs_test.read_exfault_withgil return _read_exfault(mem, withgil=True) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > raise SegmentationFault() E SegmentationFault wcfs/internal/wcfs_test.pyx:120: SegmentationFault ------------------ Captured log call ------------------- INFO wcfs:__init__.py:293 starting for zeo://localhost:22854 ... INFO wcfs:__init__.py:334 started pid102541 @ /dev/shm/wcfs/c818c147676f8d6f3b408b02f727aca5e3229e98 INFO wcfs:__init__.py:400 unmount/stop wcfs pid102541 @ /dev/shm/wcfs/c818c147676f8d6f3b408b02f727aca5e3229e98
-
Kirill Smelkov authored
Here wcfs.go should have exited due to either unmount request, _or_ SIGTERM.
-
Kirill Smelkov authored
If first unmount fails, e.g. due to "device or resource is busy", we are trying to unmount the filesystem the second time after force kill/FUSE-abort (see 5f684a49 "wcfs: Server.stop: Make sure to remove mount entry even if we had to use FUSE abort"). This way the caller of Server.stop should get an error only if that second unmount fails, not on unmount-1 error, which should be considered as internal to Server.stop implementation. If we don't hide that unmount-1 error and raise it to the caller, from outside it can confusingly look like "the server is successfully stopped, but nevertheless we are raised with an error".
-
- 17 Nov, 2021 3 commits
-
-
Kirill Smelkov authored
* t2: . wendelin.core v2.0.alpha1
-
Kirill Smelkov authored
* master: wendelin.core v2.0.alpha1
-
Kirill Smelkov authored
-
- 16 Nov, 2021 13 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
* t2: *: Cosmetics . . lib/zodb: Mark test_zconn_at as xfail on plain ZODB4 . . wcfs: Server.stop: Make sure to remove mount entry even if we had to use FUSE abort tests: Don't leak WCFS log files tests: Remove test NEO database after test run is over nxdtest: Don't run test.go for multiple GOMAXPROCS wcfs: Make sure to remove mountpoint directory on Server.stop nxdtest: Run WCFS-related tests in verbose mode on testnodes setup: Fix egg_info after addition of δbtail.go
-
Kirill Smelkov authored
* master: *: Cosmetics
-
Kirill Smelkov authored
-
Kirill Smelkov authored
* master: lib/zodb: Mark test_zconn_at as xfail on plain ZODB4
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
This way on plain ZODB4 the following non-wcfs tests will continue to pass test.py/fs-!wcfs test.py/zeo-!wcfs test.py/neo-!wcfs instead of failing as e.g. in here: https://nexedijs.erp5.net/#/test_result_module/20211116-123A66706 On plain ZODB4 WCFS-related functionality - which uses zconn_at - will continue to raise corresponding assertion in WCFS-related tests, as e.g. in https://nexedijs.erp5.net/#/test_result_module/20211116-123A66706/6
-
Kirill Smelkov authored
-
Kirill Smelkov authored
* master: wcfs: Server.stop: Make sure to remove mount entry even if we had to use FUSE abort tests: Don't leak WCFS log files tests: Remove test NEO database after test run is over nxdtest: Don't run test.go for multiple GOMAXPROCS wcfs: Make sure to remove mountpoint directory on Server.stop nxdtest: Run WCFS-related tests in verbose mode on testnodes setup: Fix egg_info after addition of δbtail.go
-
Kirill Smelkov authored
-
Kirill Smelkov authored
* t2: . fixup! wcfs: Handle ZODB invalidations . . wcfs/internal/mm: Complete the package fixup! wcfs: client: Provide client package to care about isolation protocol details lib/zodb: zconn_at: Fix how ZODB4 is asserted to be patched . . . . .
-
Kirill Smelkov authored
-
- 15 Nov, 2021 1 commit
-
-
Kirill Smelkov authored
Server.stop currently tries to unmount, and if that fails invokes FUSE abort and kills wcfs.go . However it does not call unmount the second time after such abort, and this way the filesystem remains mounted (in ENOTCONN state) and rmdir(mountpoint) fails. -> Fix it by calling unmount the second time if we had to abort FUSE connection. In that second try use lazy unmounting, because regular unmount can still fail with "Device or resource busy" since there could be still client file descriptors left pointing to the mounted filesystem. With lazy mode unmounting + followup rmdir, hopefully, always succeeds. Here is example test run where one test timed out, FUSE connection was aborted, but neither the filesystem was unmounted, nor mountpoint directory was deleted, which led to all followup tests failing in setup assert that testmountpoint does not exist: https://nexedijs.erp5.net/#/test_result_module/20211112-1ACEA62D/22 This patch should fix those followup failures + fix another leakage of WCFS mounts in real services.
-
- 12 Nov, 2021 4 commits
-
-
Kirill Smelkov authored
By default every WCFS run creates several files in /tmp/wcfs.*.log.* and without explicit cleanup those files are left hanging on testnodes. Over last ~6 months we accumulated ~ 300K such files. Don't allow those files to be leaked by instructing WCFS to log to stderr during test run. This should be also useful to see details in the test output.
-
Kirill Smelkov authored
With NEO we were creating test database on /tmp but we were not deleting it in the end. As the result many /tmp/neo_XXXXXX non-empty directories were being leaked. -> Fix it by creating testdb directory outselves and removing it at the end, similarly to FileStorage and ZEO. Fixes: 7fc4ec66 (tests: Allow to test with ZEO & NEO ZODB storages)
-
Kirill Smelkov authored
We run tests with different GOMAXPROCS because some WCFS bugs are only likely to trigger when there is only 1 or 2 main OS thread(s) in WCFS. However test.go does not exercise filesystem functionality - it runs unit tests for ZBlk decoding, ΔBtail and similar. At the same time test.go:* currently occupies ~ 50% of whole time to run full testsuite with the main consumer being ΔBtail random testing. -> Run test.go only once. This should save ~ 1000s for each run and lower whole time to run wendelin.core testsuite on testnode from ~60m -> to ~40 minutes.
-
Kirill Smelkov authored
Else every time test.py/wcfs is run several empty directories are left in /dev/shm/wcfs - each corresponding to WCFS server that was automatically spawned and stopped at the end of the test. Over time this can accumulate to some big number as e.g. ~20000 of such directories were left on the testnode during last 6 months.
-
- 09 Nov, 2021 2 commits
-
-
Kirill Smelkov authored
This are the early days of WCFS - we want full details which in default configuration might not be available to see if WCFS gets stuck for one reason or another. See added comments for details.
-
Kirill Smelkov authored
`python setup.py egg_info` stopped working after we added non-ASCII files, e.g. δbtail.go in 2ab4be93 (wcfs: xbtree: ΔBtail) and δftail.go in f980471f (wcfs: zdata: ΔFtail): (neo) (z-dev) (g.env) kirr@deca:~/src/neo/src/lab.nexedi.com/nexedi/wendelin.core$ python setup.py egg_info running egg_info writing requirements to wendelin.core.egg-info/requires.txt writing wendelin.core.egg-info/PKG-INFO writing top-level names to wendelin.core.egg-info/top_level.txt writing dependency_links to wendelin.core.egg-info/dependency_links.txt writing entry points to wendelin.core.egg-info/entry_points.txt package init file '__init__.py' not found (or not a regular file) /usr/lib/python2.7/distutils/filelist.py:64: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal sortable_files.sort() Traceback (most recent call last): File "setup.py", line 416, in <module> """.splitlines()] File "/home/kirr/src/tools/go/pygolang/golang/pyx/build.py", line 118, in setup setuptools_dso.setup(**kw) File "/home/kirr/src/wendelin/venv/z-dev/lib/python2.7/site-packages/setuptools_dso/__init__.py", line 37, in setup _setup(**kws) File "/home/kirr/src/wendelin/venv/z-dev/lib/python2.7/site-packages/setuptools/__init__.py", line 162, in setup return distutils.core.setup(**attrs) File "/usr/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/kirr/src/wendelin/venv/z-dev/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 296, in run self.find_sources() File "/home/kirr/src/wendelin/venv/z-dev/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 303, in find_sources mm.run() File "/home/kirr/src/wendelin/venv/z-dev/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 538, in run self.filelist.sort() File "/usr/lib/python2.7/distutils/filelist.py", line 64, in sort sortable_files.sort() UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 0: ordinal not in range(128) This happens becuase by default setuptools collects filenames as str, not unicode, and our git_lsfiles - also registered into setuptools.file_finders entrypoint - collects filenames as unicode. Previously everything was working because there was no on-ASCII filenames, and so unicode vs str coercion worked automatically. But now, after there is filename like 'δbtail.go', it stopped to work and raises UnicodeDecodeError. -> Fix it by adjusting git_lsfiles to collect filenames as UTF-8 encoded strings instead of unicode.
-
- 08 Nov, 2021 8 commits
-
-
Kirill Smelkov authored
* master: (40 commits) fixup! wcfs: Handle ZODB invalidations wcfs/internal/mm: Complete the package fixup! wcfs: client: Provide client package to care about isolation protocol details lib/zodb: zconn_at: Fix how ZODB4 is asserted to be patched lib/zodb: zstor_2zurl: Explicitly reject MappingStorage bigfile/zodb: Teach ZBigFile backend to use WCFS wcfs: client: Provide virtmem integration wcfs: client: Add wczsync package to maintain WCFS connection in sync to ZODB connection lib/zodb: Teach zconn_at to work on ZODB4 lib/zodb: Add ZODB.Connection.onShutdownCallback lib/zodb: Teach Connection.onResyncCallback to work on ZODB4 bigfile/py: Allow PyBigFile backend to expose "mmap overlay" functionality bigfile/virtmem: Introduce "mmap overlay" mode wcfs: client: Provide client package to care about isolation protocol details wcfs: Provide isolation to clients wcfs: Handle ZODB invalidations wcfs: Add FileSock FUSE utility wcfs: zdata: ΔFtail wcfs: xbtree: ΔBtail wcfs: xbtree: BTree-diff algorithm ...
-
Kirill Smelkov authored
-
Kirill Smelkov authored
Fix last-minute error that crept in during 4af54da9 : (neo) (z-dev) (g.env) kirr@deca:~/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs$ go test # lab.nexedi.com/nexedi/wendelin.core/wcfs ./wcfs.go:957:4: Errorf format %s has arg sk of wrong type *lab.nexedi.com/nexedi/wendelin.core/wcfs.FileSock Amends 4430de41.
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
Add two functions, that were developed during wendelin.core 2 α, to the package for completeness: - map_zero_into_ro complements map_zero_ro, but mmaps into user-provided buffer. - sync calls msync on the provided memory.
-
Kirill Smelkov authored
Remove outdated TODO because test_wcfs_watch_before_create passes this days. It was fixed after ΔFtail was taught about epochs and the fix was reflected in 63ae8326. Amends 10f7153a.
-
Kirill Smelkov authored
Fix how unpatched ZODB4 is reported to lack required patch: Before: Traceback (most recent call last): File "/home/kirr/src/wendelin/wendelin.core/lib/tests/test_zodb.py", line 251, in test_zconn_at assert zconn_at(conn1) == at0 File "/home/kirr/src/wendelin/wendelin.core/lib/zodb.py", line 162, in zconn_at assert 'conn:MVCC-via-loadBefore-only' in ZODB.nxd_patches, \ AttributeError: 'module' object has no attribute 'nxd_patches' After: Traceback (most recent call last): File "/home/kirr/src/wendelin/wendelin.core/lib/tests/test_zodb.py", line 251, in test_zconn_at assert zconn_at(conn1) == at0 File "/home/kirr/src/wendelin/wendelin.core/lib/zodb.py", line 163, in zconn_at "nexedi/ZODB!1") File "/home/kirr/src/wendelin/wendelin.core/lib/zodb.py", line 191, in _zassertHasNXDPatch (zmajor, patch, details_link)) AssertionError: ZODB4 is not patched with required Nexedi patch 'conn:MVCC-via-loadBefore-only' See nexedi/ZODB!1 for details Fixes 1f866c00 (lib/zodb: Teach zconn_at to work on ZODB4).
-
- 28 Oct, 2021 1 commit
-
-
Kirill Smelkov authored
* master: (36 commits) lib/zodb: zstor_2zurl: Explicitly reject MappingStorage bigfile/zodb: Teach ZBigFile backend to use WCFS wcfs: client: Provide virtmem integration wcfs: client: Add wczsync package to maintain WCFS connection in sync to ZODB connection lib/zodb: Teach zconn_at to work on ZODB4 lib/zodb: Add ZODB.Connection.onShutdownCallback lib/zodb: Teach Connection.onResyncCallback to work on ZODB4 bigfile/py: Allow PyBigFile backend to expose "mmap overlay" functionality bigfile/virtmem: Introduce "mmap overlay" mode wcfs: client: Provide client package to care about isolation protocol details wcfs: Provide isolation to clients wcfs: Handle ZODB invalidations wcfs: Add FileSock FUSE utility wcfs: zdata: ΔFtail wcfs: xbtree: ΔBtail wcfs: xbtree: BTree-diff algorithm wcfs: xbtree: blib += PPTreeSubSet, ΔPPTreeSubSet wcfs: xbtree: blib += RangedMap, RangedKeySet wcfs: tests: Tree-based testing environment wcfs: Set package ...
-