- 30 Nov, 2020 1 commit
-
-
Kirill Smelkov authored
-
- 23 Nov, 2020 2 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
- 19 Nov, 2020 1 commit
-
-
Kirill Smelkov authored
X Stub ZODB/persistent/BTrees/ZEO have to be installed manually for forward-compatibility with ZODB4 and ZODB5 Else installing any egg that depends on 'ZODB' (not 'ZODB3') will pull in ZODB5. Example of those eggs are zodbtools, zodburi, ...
-
- 16 Nov, 2020 1 commit
-
-
Kirill Smelkov authored
-
- 10 Nov, 2020 1 commit
-
-
Kirill Smelkov authored
-
- 31 Jul, 2020 3 commits
-
-
Kirill Smelkov authored
[ This is ZODB3 backport of commit bb9bf539 (https://github.com/zopefoundation/ZODB/pull/298) ] ZODB tries to avoid saving empty transactions to storage on `transaction.commit()`. The way it works is: if no objects were changed during ongoing transaction, ZODB.Connection does not join current TransactionManager, and transaction.commit() performs two-phase commit protocol only on joined DataManagers. In other words if no objects were changed, no tpc_*() methods are called at all on ZODB.Connection at transaction.commit() time. This way application servers like Zope/ZServer/ERP5/... can have something as try: # process incoming request transaction.commit() # processed ok except: transaction.abort() # problem: log + reraise in top-level code to process requests without creating many on-disk transactions with empty data changes just because read-only requests were served. Everything is working as intended. However at storage level, FileStorage currently also checks whether transaction that is being committed also comes with empty data changes, and _skips_ saving transaction into disk *at all* for such cases, even if it has been explicitly told to commit the transaction via two-phase commit protocol calls done at storage level. This creates the situation, where contrary to promise in ZODB/interfaces.py(*), after successful tpc_begin/tpc_vote/tpc_finish() calls made at storage level, transaction is _not_ made permanent, despite tid of "committed" transaction being returned to caller. In other words FileStorage, when asked to commit a transaction, even if one with empty data changes, reports "ok" and gives transaction ID to the caller, without creating corresponding transaction record on disk. This behaviour is a) redundant to application-level avoidance to create empty transaction on storage described in the beginning, and b) creates problems: The first problem is that application that works at storage-level might be interested in persisting transaction, even with empty changes to data, just because it wants to save the metadata similarly to e.g. `git commit --allow-empty`. The other problem is that an application view and data in database become inconsistent: an application is told that a transaction was created with corresponding transaction ID, but if the storage is actually inspected, e.g. by iteration, the transaction is not there. This, in particular, can create problems if TID of committed transaction is reported elsewhere and that second database client does not find the transaction it was told should exist. I hit this particular problem with wendelin.core. In wendelin.core, there is custom virtual memory layer that keeps memory in sync with data in ZODB. At commit time, the memory is inspected for being dirtied, and if a page was changed, virtual memory layer joins current transaction _and_ forces corresponding ZODB.Connection - via which it will be saving data into ZODB objects - to join the transaction too, because it would be too late to join ZODB.Connection after 2PC process has begun(+). One of the format in which data are saved tries to optimize disk space usage, and it actually might happen, that even if data in RAM were dirtied, the data itself stayed the same and so nothing should be saved into ZODB. However ZODB.Connection is already joined into transaction and it is hard not to join it because joining a DataManager when the 2PC is already ongoing does not work. This used to work ok with wendelin.core 1, but with wendelin.core 2 - where separate virtual filesystem is also connected to the database to provide base layer for arrays mappings - this creates problem, because when wcfs (the filesystem) is told to synchronize to view the database @tid of committed transaction, it can wait forever waiting for that, or later, transaction to appear on disk in the database, creating application-level deadlock. I agree that some more effort might be made at wendelin.core side to avoid committing transactions with empty data at storage level. However the most clean way to fix this problem in my view is to fix FileStorage itself, because if at storage level it was asked to commit something, it should not silently skip doing so and dropping even non-empty metadata + returning ok and committed transaction ID to the caller. As described in the beginning this should not create problems for application-level ZODB users, while at storage-level the implementation is now consistently matching interface and common sense. ---- (*) tpc_finish: Finish the transaction, making any transaction changes permanent. Changes must be made permanent at this point. ... https://github.com/zopefoundation/ZODB/blob/5.5.1-35-gb5895a5c2/src/ZODB/interfaces.py#L828-L831 (+) https://lab.nexedi.com/kirr/wendelin.core/blob/9ff5ed32/bigfile/file_zodb.py#L788-822
-
Julien Muchembled authored
---- kirr: This is likely to be ZODB3-backport of https://github.com/zopefoundation/ZODB/pull/69 Taken from https://lab.nexedi.com/nexedi/neoppod/blob/v1.12-52-gfb746e6b/ZODB3.patch
-
Kirill Smelkov authored
People say that ZODB3 is kind of closed: https://github.com/zopefoundation/ZODB/pull/306#issuecomment-606735225 Givent that even for ZEO4 we were suggested that 4 is considered "dead" by upstream: https://github.com/zopefoundation/ZEO/pull/161#pullrequestreview-447245642 Let's keep on patching ZODB3 in our local fork until we keep on using it.
-
- 19 Apr, 2017 4 commits
-
-
Godefroid Chapelle authored
-
Jim Fulton authored
-
Jim Fulton authored
-
Jim Fulton authored
-
- 14 Sep, 2016 2 commits
-
-
Julien Muchembled authored
-
Vincent Pelletier authored
-
- 13 Sep, 2016 6 commits
-
-
Julien Muchembled authored
-
Julien Muchembled authored
Changes are backported from commits 227953b9 and 03a326be.
-
Julien Muchembled authored
-
Kirill Smelkov authored
( This is backport of https://github.com/zopefoundation/persistent/pull/44 to ZODB-3.10 ) On ._p_deactivate() and ._p_invalidate(), when an object goes to ghost state, objects referenced by all its attributes, except related to persistence machinery, are released, this way freeing memory (if they were referenced only from going-to-ghost object). That's the idea - an object in ghost state is simply a stub, which loads its content on first access (via hooking into get/set attr) while occupying minimal memory in not-yet-loaded state. However the above is not completely true right now, as currently on ghostification only object's .__dict__ is released, while in-slots objects are retained attached to ghost object staying in RAM: ---- 8< ---- from ZODB import DB from persistent import Persistent import gc db = DB(None) jar = db.open() class C: def __init__(self, v): self.v = v def __del__(self): print 'released (%s)' % self.v class P1(Persistent): pass class P2(Persistent): __slots__ = ('aaa') p1 = P1() jar.add(p1) p1.aaa = C(1) p2 = P2() jar.add(p2) p2.aaa = C(2) p1._p_invalidate() # "released (1)" is printed p2._p_invalidate() gc.collect() # "released (2)" is NOT printed <-- ---- 8< ---- So teach ghostify() & friends to release objects in slots to free-up memory when an object goes to ghost state. NOTE PyErr_Occurred() added after ghostify() calls because pickle_slotnames() can raise an error, but we do not want to change ghostify() prototype for backward compatibility reason - as it is used in cPersistenceCAPIstruct. ( I hit this bug with wendelin.core which uses proxies to load data from DB to virtual memory manager and then deactivate proxy right after load has been completed: https://lab.nexedi.com/nexedi/wendelin.core/blob/f7803634/bigfile/file_zodb.py#L239 https://lab.nexedi.com/nexedi/wendelin.core/blob/f7803634/bigfile/file_zodb.py#L295 )
-
Julien Muchembled authored
-
Julien Muchembled authored
This reverts to the behaviour of 3.10.3 and older. (cherry picked from commit b74eef76) Conflicts: src/ZODB/tests/testconflictresolution.py
-
- 28 Apr, 2016 2 commits
-
-
Julien Muchembled authored
-
Julien Muchembled authored
Multi-threaded IO support, which is new to ZODB 3.10, allows clients to read data (load & loadBefore) even after tpc_vote has started to write a new transaction to disk. This is done by using different 'file' objects. Issues start when a transaction is rolled back after data has been appended (using the writing file object). Truncating is not enough because the FilePool may have been used concurrently to read the end of the last transaction: file objects have their own read buffers which, in this case, may also contain the beginning of the aborted transaction. So a solution is to invalidate read buffers whenever they may contain wrong data. This patch does it on truncation, which happens rarely enough to not affect performance. We discovered this bug in the following conditions: - ZODB splitted in several FileStorage - many conflicts in the first committed DB, but always resolved - unresolved conflict in another DB If the transaction is replayed with success (no more conflict in the other DB), a subsequent load of the object that could be resolved in the first DB may, for example, return a wrong serial (tid of the aborted transaction) if the layout of the committed transaction matches that of the aborted one. The bug usually manifests with POSKeyError & CorruptedDataError exceptions in ZEO logs, for example while trying to resolve a conflict (and restarting the transaction does not help, causing Site Errors in Zope). But theorically, this could also cause silent corruption or unpickling errors at client side. (cherry picked from commit 028b1922) Conflicts: src/ZODB/FileStorage/FileStorage.py
-
- 17 Mar, 2016 4 commits
-
-
Tres Seaver authored
Fixing uncaught exception problem on shutdown
-
Patrick Gerken authored
-
Patrick Gerken authored
-
Patrick Gerken authored
-
- 16 Mar, 2016 1 commit
-
-
Patrick Gerken authored
Nested connections try to clean up themselves multiple times. This does not work
-
- 10 Mar, 2014 3 commits
-
-
Godefroid Chapelle authored
-
Godefroid Chapelle authored
-
Gediminas Paulauskas authored
-
- 03 Mar, 2014 2 commits
-
-
Godefroid Chapelle authored
-
Godefroid Chapelle authored
-
- 03 Feb, 2014 6 commits
-
-
Godefroid Chapelle authored
-
Godefroid Chapelle authored
-
Godefroid Chapelle authored
-
-
Godefroid Chapelle authored
-
-
- 23 Dec, 2013 1 commit
-
-
Godefroid Chapelle authored
-