Commit 91e9eba8 authored by Kirill Smelkov's avatar Kirill Smelkov

X wcfs: test: Register tFile to tDB early

tFile registers itself to tDB so that defer(tdb.close) that each test does,
automatically close all files before trying to unmount wcfs. However if an
error happens inside tFile constructor - before the place where
`tdb._files.add(t)` was previously located, tDB.close won't close tFile.f,
which will leave opened file on the wcfs -> wcfs unmount will fail with EBUSY,
and wcfs will be killed with huge backtrace.

If we register tFile early, the problem - even if it happens inside tFile
constor - will be handled gracefully: opened file closed and wcfs unmounted
clearnly - for example with the following short and easy to understand traceback

    (1.venv) kirr@deb9:~/src/wendelin.core$ python -m pytest -x wcfs/
    ============================================== test session starts ==============================================
    platform linux2 -- Python 2.7.13, pytest-4.6.11, py-1.9.0, pluggy-0.13.1
    rootdir: /home/kirr/src/wendelin.core
    collected 28 items

    wcfs/wcfs_test.py ..F

    =================================================== FAILURES ====================================================
    ________________________________________________ test_wcfs_basic ________________________________________________

        @func
        def test_wcfs_basic():
            t = tDB(); zf = t.zfile
            defer(t.close)

            # >>> lookup non-BigFile -> must be rejected
            with raises(OSError) as exc:
                t.wc._stat("head/bigfile/%s" % h(t.nonzfile._p_oid))
            assert exc.value.errno == EINVAL

            # >>> file initially empty
    >       f = t.open(zf)

    wcfs/wcfs_test.py:1080:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    wcfs/wcfs_test.py:322: in open
        return tFile(t, zf, at=at)
    wcfs/wcfs_test.py:477: in __init__
        mm.lock(t.fmmap, mm.MLOCK_ONFAULT)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    >   PyErr_SetFromErrno(OSError)
    E   OSError: [Errno 1] Operation not permitted

    wcfs/internal/mm.pyx:112: OSError
    --------------------------------------------- Captured stdout call ----------------------------------------------

    M: commit -> @at0 (03db288d33c34955)
    >>> Change history by file:

    --------------------------------------------- Captured stderr call ----------------------------------------------
    wcfs: 2020/10/08 19:13:12 zodb: FIXME: open file:///tmp/testdb_fs.1HuVZF/1.fs: raw cache is not ready for invalidations -> NoCache forced
    E1008 19:13:12.030532    3251 wcfs.go:2450] kernel FUSE (API 7.26) does not support explicit data cache invalidation
    E1008 19:13:12.030780    3251 wcfs.go:2450] -> performance will be AWFUL.
    E1008 19:13:12.030796    3251 wcfs.go:2450] -> you need kernel which includes git.kernel.org/linus/ad2ba64dd489.
    E1008 19:13:12.030809    3251 wcfs.go:2450] -> (Linux 5.2+, or nxd-fuse-dkms package installed from navytux.spb.ru/pkg)
    W: wcfs: kernel FUSE (API 7.26) does not support explicit data cache invalidation
    W: wcfs: -> performance will be AWFUL.
    W: wcfs: -> you need kernel which includes git.kernel.org/linus/ad2ba64dd489.
    W: wcfs: -> (Linux 5.2+, or nxd-fuse-dkms package installed from navytux.spb.ru/pkg)
    ----------------------------------------------- Captured log call -----------------------------------------------
    WARNING  ZODB.FileStorage:FileStorage.py:398 Ignoring index for /tmp/testdb_fs.1HuVZF/1.fs
    ====================================== 1 failed, 2 passed in 2.10 seconds =======================================
parent d995819d
...@@ -438,6 +438,8 @@ class tFile: ...@@ -438,6 +438,8 @@ class tFile:
t.at = at t.at = at
t.f = tdb.wc._open(zf, at=at) t.f = tdb.wc._open(zf, at=at)
t.blksize = zf.blksize t.blksize = zf.blksize
t.fmmap = None
tdb._files.add(t)
# make sure that wcfs reports zf.blksize as preffered block size for IO. # make sure that wcfs reports zf.blksize as preffered block size for IO.
# wcfs.py also uses .st_blksize in blk -> byte offset computation. # wcfs.py also uses .st_blksize in blk -> byte offset computation.
...@@ -495,11 +497,10 @@ class tFile: ...@@ -495,11 +497,10 @@ class tFile:
mm.advise(blkmmap[:_], mm.MADV_SEQUENTIAL) mm.advise(blkmmap[:_], mm.MADV_SEQUENTIAL)
mm.advise(blkmmap[_:], mm.MADV_RANDOM) mm.advise(blkmmap[_:], mm.MADV_RANDOM)
tdb._files.add(t)
def close(t): def close(t):
t.tdb._files.remove(t) t.tdb._files.remove(t)
mm.unmap(t.fmmap) if t.fmmap is not None:
mm.unmap(t.fmmap)
t.f.close() t.f.close()
# _blk returns memoryview of file[blk]. # _blk returns memoryview of file[blk].
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment