• Kirill Smelkov's avatar
    tests: Add test for open vs invalidation race · 5b4dd5f7
    Kirill Smelkov authored
    Add test that exercises open vs invalidation race condition that, if
    happen, leads to data corruption. We are seeing such race happening on
    storage level in ZEO (https://github.com/zopefoundation/ZEO/issues/166),
    and previously we've seen it also to happen on Connection level
    (https://github.com/zopefoundation/ZODB/issues/290). By adding this test
    to be exercised wrt all storages we make sure that all storages stay
    free from this race.
    
    And it payed out. Besides catching original problems from
    https://github.com/zopefoundation/ZODB/issues/290 and
    https://github.com/zopefoundation/ZEO/issues/166 , this test also
    discovered a concurrency bug in MVCCMappingStorage:
    
        Failure in test check_race_open_vs_invalidate (ZODB.tests.testMVCCMappingStorage.MVCCMappingStorageTests)
        Traceback (most recent call last):
          File "/usr/lib/python2.7/unittest/case.py", line 329, in run
            testMethod()
          File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/BasicStorage.py", line 492, in check_race_open_vs_invalidate
            self.fail(failure[0])
          File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
            raise self.failureException(msg)
        AssertionError: T1: obj1.value (24)  !=  obj2.value (23)
    
    The problem with MVCCMappingStorage was that instance.poll_invalidations
    was correctly taking main_lock with intention to make sure main data is
    not mutated during analysis, but instance.tpc_finish and
    instance.tpc_abort did _not_ taken main lock, which was leading to
    committed data to be propagating into main storage in non-atomic way.
    
    This bug was also observable if both obj1 and obj2 in the added test
    were always loaded from the storage (added obj2._p_invalidate after
    obj1._p_invalidate).
    
    -> Fix MVCCMappingStorage by correctly locking main MVCCMappingStorage
    instance when processing transaction completion.
    
    /cc @d-maurer, @jamadden, @jmuchemb
    /reviewed-on https://github.com/zopefoundation/ZODB/pull/345
    5b4dd5f7
MVCCMappingStorage.py 4.63 KB