Skip to content

Z
Zope
Commit 9933d242 authored by Barry Warsaw's avatar Barry Warsaw
Browse files
Options

The sematics of object serial numbers across abortVersions has 

(finally) been defined: the serial number of objects in the
non-version after an abortVersion should be the last serial number of
the object before it was modified in the version.

This actually breaks the symmetry between serial numbers and
transaction ids, so it must be handled specially.

Specific changes include:

_setupDBs(): Updated documentation.

_finish(): Recognize the 'a' opcode, which is subtly different than
the 'o' opcode.  When the 'a' opcode is seen, we'll store both the
"live revision id" and the current transaction id in the _serials
table.  The former id is the serial number of the last change to the
non-version data.  If we're seeing the 'o' or 'x' opcodes, we'll save
None as the serial number, which tells the loop at the end of the
function that the object serial number and the transaction id are the
same (overwhelmingly, the most common case).

When serial# <> tid, we'll concatenate the two values together into
serial+tid as the value of the _serials's oid key.  So the value can
now be an 8-byte string (the common case), or a 16-byte string when
serial# <> tid.

_getSerialAndTid(): A new function which returns a 2-tuple of the
oid's current serial number and the tid of the last transaction that
modified the object.  Usually these are the same, but in the face of
abortVersion, they're not.

_getSerialAndTidMissingOk(): Like _getSerialAndTid() but if the oid
isn't in the _serials table, return (None, None) instead of raising a
KeyError.  Sometimes we need different table access semantics.

abortVersion(), commitVersion(), modifiedInVersion(), load(),
getSerial(), transactionalUndo(), history(), _dopack(): Use the new
_getSerialAndTid() access method to get the serial# and tid for the
last object modification, instead of accessing the _serials table
directly.

store(), _rootreachable(): Use _getSerialAndTidMissingOk()
parent 77c8d867
Hide whitespace changes
Inline Side-by-side
Showing with 172 additions and 70 deletions
lib/python/BDBStorage/BDBFullStorage.py
View file @ 9933d242
...@@ -18,7 +18,7 @@ See Minimal.py for an implementation of Berkeley storage that does not support ...@@ -18,7 +18,7 @@ See Minimal.py for an implementation of Berkeley storage that does not support
undo or versioning. undo or versioning.
""" """
__version__ = '$Revision: 1.41 $'.split()[-2:][0] __version__ = '$Revision: 1.42 $'.split()[-2:][0]
import sys import sys
import struct import struct
...@@ -87,26 +87,30 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -87,26 +87,30 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
def _setupDBs(self): def _setupDBs(self):
# Data Type Assumptions: # Data Type Assumptions:
# #
# object ids (oid) are 8-bytes # - object ids (oid) are 8-bytes
# object serial numbers are 8-bytes # - object revision ids (revid) are 8-bytes
# transaction ids (tid) are 8-bytes # - transaction ids (tid) are 8-bytes
# revision ids (revid) are the same as transaction ids, just used in a # - version ids (vid) are 8-bytes
# different context. # - data pickles are of arbitrary length
# version ids (vid) are 8-bytes #
# data pickles are of arbitrary length # Note that revids are usually the same as tids /except/ in the face
# of abortVersion().
# #
# Create the tables used to maintain the relevant information. The # Create the tables used to maintain the relevant information. The
# full storage needs a bunch of tables. These two are defined by the # full storage needs a bunch of tables. These two are defined by the
# base class infrastructure and are shared by the Minimal # base class infrastructure and are shared by the Minimal
# implementation. # implementation.
# #
# serials -- {oid -> serial} # serials -- {oid -> serial} | {oid -> serial + tid}
# Maps oids to object serial numbers. The serial number is # Maps oids to revids. Usually the revision id of the object is
# essentially a timestamp used to determine if conflicts have # the transaction id of the last transaction that object is
# arisen, and serial numbers double as transaction ids and object # modified in. This doesn't work for transactions which contained
# revision ids. If an attempt is made to store an object with a # an abortVersion() because in that case, the serial number of the
# serial number that is different than the current serial number # object is the revid of the last non-version revision before the
# for the object, a ConflictError is raised. # version was created (IOW, by definition, a serial number doesn't
# change when an abortVersion occurs). However, the tid /does/
# change so in those cases, the value of the oid key is a 16 byte
# value instead of an 8 byte value. This is considered rare.
# #
# pickles -- {oid+revid -> pickle} # pickles -- {oid+revid -> pickle}
# Maps the concrete object referenced by oid+revid to that # Maps the concrete object referenced by oid+revid to that
...@@ -301,7 +305,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -301,7 +305,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
if rec is None: if rec is None:
break break
op, data = rec op, data = rec
if op in 'ox': if op in 'aox':
# This is a `versioned' object record. Information about # This is a `versioned' object record. Information about
# this object must be stored in the pickle table, the # this object must be stored in the pickle table, the
# object metadata table, the currentVersions tables , and # object metadata table, the currentVersions tables , and
...@@ -330,7 +334,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -330,7 +334,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
for roid in refdoids: for roid in refdoids:
refcounts[roid] = refcounts.get(roid, 0) + 1 refcounts[roid] = refcounts.get(roid, 0) + 1
# Update the metadata table # Update the metadata table
if op == 'o': if op <> 'x':
# `x' opcode does an immediate write to metadata # `x' opcode does an immediate write to metadata
metadata.append( metadata.append(
(key, ''.join((vid,nvrevid,lrevid,prevrevid)))) (key, ''.join((vid,nvrevid,lrevid,prevrevid))))
...@@ -340,7 +344,13 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -340,7 +344,13 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# than to weed them out now. # than to weed them out now.
if vid <> ZERO: if vid <> ZERO:
self._currentVersions.put(vid, oid, txn=txn) self._currentVersions.put(vid, oid, txn=txn)
serials.append((oid, tid)) # If this was an abortVersion, we need to store the
# revid+tid as the value of the key, since in this one
# instance, the two are not equivalent
if op == 'a':
serials.append((oid, lrevid, tid))
else:
serials.append((oid, None, tid))
# Update the pickle's reference count. Remember, the # Update the pickle's reference count. Remember, the
# refcount is stored as a string, so we have to do the # refcount is stored as a string, so we have to do the
# string->long->string dance. # string->long->string dance.
...@@ -363,11 +373,14 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -363,11 +373,14 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
c.close() c.close()
# It's actually faster to boogie through this list twice # It's actually faster to boogie through this list twice
#print >> sys.stderr, 'start:', self._lockstats() #print >> sys.stderr, 'start:', self._lockstats()
for oid, tid in serials: for oid, serial, tid in serials:
self._txnoids.put(tid, oid, txn=txn) self._txnoids.put(tid, oid, txn=txn)
#print >> sys.stderr, 'post-txnoids:', self._lockstats() #print >> sys.stderr, 'post-txnoids:', self._lockstats()
for oid, tid in serials: for oid, serial, tid in serials:
self._serials.put(oid, tid, txn=txn) if serial is None:
self._serials.put(oid, tid, txn=txn)
else:
self._serials.put(oid, serial+tid, txn=txn)
#print >> sys.stderr, 'post-serials:', self._lockstats() #print >> sys.stderr, 'post-serials:', self._lockstats()
for key, data in metadata: for key, data in metadata:
self._metadata.put(key, data, txn=txn) self._metadata.put(key, data, txn=txn)
...@@ -449,7 +462,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -449,7 +462,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
if oids.has_key(oid): if oids.has_key(oid):
# We've already dealt with this oid... # We've already dealt with this oid...
continue continue
revid = self._serials[oid] serial, tid = self._getSerialAndTid(oid)
revid = tid
meta = self._metadata[oid+revid] meta = self._metadata[oid+revid]
curvid, nvrevid = struct.unpack('>8s8s', meta[:16]) curvid, nvrevid = struct.unpack('>8s8s', meta[:16])
# Make sure that the vid in the metadata record is the same as # Make sure that the vid in the metadata record is the same as
...@@ -519,7 +533,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -519,7 +533,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
rec = c.next_dup() rec = c.next_dup()
if oids.has_key(oid): if oids.has_key(oid):
continue continue
revid = self._serials[oid] serial, tid = self._getSerialAndTid(oid)
revid = tid
meta = self._metadata[oid+revid] meta = self._metadata[oid+revid]
curvid, nvrevid, lrevid = struct.unpack('>8s8s8s', meta[:24]) curvid, nvrevid, lrevid = struct.unpack('>8s8s8s', meta[:24])
# Our database better be consistent. # Our database better be consistent.
...@@ -551,7 +566,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -551,7 +566,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
self._lock_acquire() self._lock_acquire()
try: try:
# Let KeyErrors percolate up # Let KeyErrors percolate up
revid = self._serials[oid] serial, tid = self._getSerialAndTid(oid)
revid = tid
vid = self._metadata[oid+revid][:8] vid = self._metadata[oid+revid][:8]
if vid == ZERO: if vid == ZERO:
# Not in a version # Not in a version
...@@ -579,7 +595,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -579,7 +595,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
try: try:
# Get the current revid for the object. As per the protocol, let # Get the current revid for the object. As per the protocol, let
# any KeyErrors percolate up. # any KeyErrors percolate up.
revid = self._serials[oid] serial, tid = self._getSerialAndTid(oid)
revid = tid
# Get the metadata associated with this revision of the object. # Get the metadata associated with this revision of the object.
# All we really need is the vid, the non-version revid and the # All we really need is the vid, the non-version revid and the
# pickle pointer revid. # pickle pointer revid.
...@@ -591,7 +608,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -591,7 +608,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# object is living in is the one that was requested, we simply # object is living in is the one that was requested, we simply
# return the current revision's pickle. # return the current revision's pickle.
if vid == ZERO or self._versions.get(vid) == version: if vid == ZERO or self._versions.get(vid) == version:
return self._pickles[oid+lrevid], revid return self._pickles[oid+lrevid], serial
# The object was living in a version, but not the one requested. # The object was living in a version, but not the one requested.
# Semantics here are to return the non-version revision. # Semantics here are to return the non-version revision.
lrevid = self._metadata[oid+nvrevid][16:24] lrevid = self._metadata[oid+nvrevid][16:24]
...@@ -599,6 +616,34 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -599,6 +616,34 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
finally: finally:
self._lock_release() self._lock_release()
def _getSerialAndTid(self, oid):
# For the object, return the curent serial number and transaction id
# of the last transaction that modified the object. Usually these
# will be the same, unless the last transaction was an abortVersion
self._lock_acquire()
try:
data = self._serials[oid]
finally:
self._lock_release()
if len(data) == 8:
return data, data
return data[:8], data[8:]
def _getSerialAndTidMissingOk(self, oid):
# For the object, return the curent serial number and transaction id
# of the last transaction that modified the object. Usually these
# will be the same, unless the last transaction was an abortVersion
self._lock_acquire()
try:
data = self._serials.get(oid)
finally:
self._lock_release()
if data is None:
return None, None
if len(data) == 8:
return data, data
return data[:8], data[8:]
def _loadSerialEx(self, oid, serial): def _loadSerialEx(self, oid, serial):
# Just like loadSerial, except that it returns both the pickle and the # Just like loadSerial, except that it returns both the pickle and the
# version this object revision is living in. # version this object revision is living in.
...@@ -629,7 +674,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -629,7 +674,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# irrespective of any versions. # irrespective of any versions.
self._lock_acquire() self._lock_acquire()
try: try:
return self._serials[oid] serial, tid = self._getSerialAndTid(oid)
return serial
finally: finally:
self._lock_release() self._lock_release()
...@@ -691,7 +737,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -691,7 +737,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# a single transaction. It's not clear though under what # a single transaction. It's not clear though under what
# situations that can occur or what the semantics ought to be. # situations that can occur or what the semantics ought to be.
# For now, we'll assume this doesn't happen. # For now, we'll assume this doesn't happen.
oserial = orevid = self._serials.get(oid) oserial, orevid = self._getSerialAndTidMissingOk(oid)
if oserial is None: if oserial is None:
# There's never been a previous revision of this object, so # There's never been a previous revision of this object, so
# set its non-version revid to zero. # set its non-version revid to zero.
...@@ -778,7 +824,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -778,7 +824,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# that would be restored if we were undoing the current # that would be restored if we were undoing the current
# revision. Otherwise, we attempt application level conflict # revision. Otherwise, we attempt application level conflict
# resolution. If that fails, we raise an exception. # resolution. If that fails, we raise an exception.
revid = self._serials[oid] cserial, ctid = self._getSerialAndTid(oid)
revid = ctid
if revid == tid: if revid == tid:
vid, nvrevid, lrevid, prevrevid = struct.unpack( vid, nvrevid, lrevid, prevrevid = struct.unpack(
'>8s8s8s8s', self._metadata[oid+tid]) '>8s8s8s8s', self._metadata[oid+tid])
...@@ -1012,7 +1059,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -1012,7 +1059,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# start with the most recent revision of the object, then search # start with the most recent revision of the object, then search
# the transaction records backwards until we find enough records. # the transaction records backwards until we find enough records.
history = [] history = []
revid = self._serials[oid] serial, tid = self._getSerialAndTid(oid)
revid = tid
# BAW: Again, let KeyErrors percolate up # BAW: Again, let KeyErrors percolate up
while len(history) < size: while len(history) < size:
# Some information comes out of the revision metadata... # Some information comes out of the revision metadata...
...@@ -1037,7 +1085,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -1037,7 +1085,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
d = {'time' : TimeStamp(revid).timeTime(), d = {'time' : TimeStamp(revid).timeTime(),
'user_name' : user, 'user_name' : user,
'description': desc, 'description': desc,
'serial' : revid, 'serial' : serial,
'version' : retvers, 'version' : retvers,
'size' : len(data), 'size' : len(data),
} }
...@@ -1047,7 +1095,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -1047,7 +1095,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# revision, stopping when we've reached the end. # revision, stopping when we've reached the end.
if previd == ZERO: if previd == ZERO:
break break
revid = previd serial = revid = previd
return history return history
finally: finally:
self._lock_release() self._lock_release()
...@@ -1070,7 +1118,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -1070,7 +1118,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
seen[oid] = 1 seen[oid] = 1
reachables[oid] = 1 reachables[oid] = 1
# Get the pickle data for the object's current version # Get the pickle data for the object's current version
revid = self._serials.get(oid) serial, tid = self._getSerialAndTidMissingOk(oid)
revid = tid
if revid is None: if revid is None:
# BAW: how can this happen?! This means that an object is # BAW: how can this happen?! This means that an object is
# holding references to an object that we know nothing about. # holding references to an object that we know nothing about.
...@@ -1280,8 +1329,10 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -1280,8 +1329,10 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
packablerevs.setdefault(oid, []).append(key) packablerevs.setdefault(oid, []).append(key)
# Otherwise, if this isn't the current revision for this # Otherwise, if this isn't the current revision for this
# object, then it's packable. # object, then it's packable.
elif self._serials[oid] <> key[8:]: else:
packablerevs.setdefault(oid, []).append(key) serial, tid = self._getSerialAndTid(oid)
if tid <> key[8:]:
packablerevs.setdefault(oid, []).append(key)
finally: finally:
c.close() c.close()
# We now have all the packable revisions we're going to handle. For # We now have all the packable revisions we're going to handle. For
......
lib/python/BDBStorage/Full.py
View file @ 9933d242
...@@ -18,7 +18,7 @@ See Minimal.py for an implementation of Berkeley storage that does not support ...@@ -18,7 +18,7 @@ See Minimal.py for an implementation of Berkeley storage that does not support
undo or versioning. undo or versioning.
""" """
__version__ = '$Revision: 1.41 $'.split()[-2:][0] __version__ = '$Revision: 1.42 $'.split()[-2:][0]
import sys import sys
import struct import struct
...@@ -87,26 +87,30 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -87,26 +87,30 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
def _setupDBs(self): def _setupDBs(self):
# Data Type Assumptions: # Data Type Assumptions:
# #
# object ids (oid) are 8-bytes # - object ids (oid) are 8-bytes
# object serial numbers are 8-bytes # - object revision ids (revid) are 8-bytes
# transaction ids (tid) are 8-bytes # - transaction ids (tid) are 8-bytes
# revision ids (revid) are the same as transaction ids, just used in a # - version ids (vid) are 8-bytes
# different context. # - data pickles are of arbitrary length
# version ids (vid) are 8-bytes #
# data pickles are of arbitrary length # Note that revids are usually the same as tids /except/ in the face
# of abortVersion().
# #
# Create the tables used to maintain the relevant information. The # Create the tables used to maintain the relevant information. The
# full storage needs a bunch of tables. These two are defined by the # full storage needs a bunch of tables. These two are defined by the
# base class infrastructure and are shared by the Minimal # base class infrastructure and are shared by the Minimal
# implementation. # implementation.
# #
# serials -- {oid -> serial} # serials -- {oid -> serial} | {oid -> serial + tid}
# Maps oids to object serial numbers. The serial number is # Maps oids to revids. Usually the revision id of the object is
# essentially a timestamp used to determine if conflicts have # the transaction id of the last transaction that object is
# arisen, and serial numbers double as transaction ids and object # modified in. This doesn't work for transactions which contained
# revision ids. If an attempt is made to store an object with a # an abortVersion() because in that case, the serial number of the
# serial number that is different than the current serial number # object is the revid of the last non-version revision before the
# for the object, a ConflictError is raised. # version was created (IOW, by definition, a serial number doesn't
# change when an abortVersion occurs). However, the tid /does/
# change so in those cases, the value of the oid key is a 16 byte
# value instead of an 8 byte value. This is considered rare.
# #
# pickles -- {oid+revid -> pickle} # pickles -- {oid+revid -> pickle}
# Maps the concrete object referenced by oid+revid to that # Maps the concrete object referenced by oid+revid to that
...@@ -301,7 +305,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -301,7 +305,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
if rec is None: if rec is None:
break break
op, data = rec op, data = rec
if op in 'ox': if op in 'aox':
# This is a `versioned' object record. Information about # This is a `versioned' object record. Information about
# this object must be stored in the pickle table, the # this object must be stored in the pickle table, the
# object metadata table, the currentVersions tables , and # object metadata table, the currentVersions tables , and
...@@ -330,7 +334,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -330,7 +334,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
for roid in refdoids: for roid in refdoids:
refcounts[roid] = refcounts.get(roid, 0) + 1 refcounts[roid] = refcounts.get(roid, 0) + 1
# Update the metadata table # Update the metadata table
if op == 'o': if op <> 'x':
# `x' opcode does an immediate write to metadata # `x' opcode does an immediate write to metadata
metadata.append( metadata.append(
(key, ''.join((vid,nvrevid,lrevid,prevrevid)))) (key, ''.join((vid,nvrevid,lrevid,prevrevid))))
...@@ -340,7 +344,13 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -340,7 +344,13 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# than to weed them out now. # than to weed them out now.
if vid <> ZERO: if vid <> ZERO:
self._currentVersions.put(vid, oid, txn=txn) self._currentVersions.put(vid, oid, txn=txn)
serials.append((oid, tid)) # If this was an abortVersion, we need to store the
# revid+tid as the value of the key, since in this one
# instance, the two are not equivalent
if op == 'a':
serials.append((oid, lrevid, tid))
else:
serials.append((oid, None, tid))
# Update the pickle's reference count. Remember, the # Update the pickle's reference count. Remember, the
# refcount is stored as a string, so we have to do the # refcount is stored as a string, so we have to do the
# string->long->string dance. # string->long->string dance.
...@@ -363,11 +373,14 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -363,11 +373,14 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
c.close() c.close()
# It's actually faster to boogie through this list twice # It's actually faster to boogie through this list twice
#print >> sys.stderr, 'start:', self._lockstats() #print >> sys.stderr, 'start:', self._lockstats()
for oid, tid in serials: for oid, serial, tid in serials:
self._txnoids.put(tid, oid, txn=txn) self._txnoids.put(tid, oid, txn=txn)
#print >> sys.stderr, 'post-txnoids:', self._lockstats() #print >> sys.stderr, 'post-txnoids:', self._lockstats()
for oid, tid in serials: for oid, serial, tid in serials:
self._serials.put(oid, tid, txn=txn) if serial is None:
self._serials.put(oid, tid, txn=txn)
else:
self._serials.put(oid, serial+tid, txn=txn)
#print >> sys.stderr, 'post-serials:', self._lockstats() #print >> sys.stderr, 'post-serials:', self._lockstats()
for key, data in metadata: for key, data in metadata:
self._metadata.put(key, data, txn=txn) self._metadata.put(key, data, txn=txn)
...@@ -449,7 +462,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -449,7 +462,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
if oids.has_key(oid): if oids.has_key(oid):
# We've already dealt with this oid... # We've already dealt with this oid...
continue continue
revid = self._serials[oid] serial, tid = self._getSerialAndTid(oid)
revid = tid
meta = self._metadata[oid+revid] meta = self._metadata[oid+revid]
curvid, nvrevid = struct.unpack('>8s8s', meta[:16]) curvid, nvrevid = struct.unpack('>8s8s', meta[:16])
# Make sure that the vid in the metadata record is the same as # Make sure that the vid in the metadata record is the same as
...@@ -519,7 +533,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -519,7 +533,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
rec = c.next_dup() rec = c.next_dup()
if oids.has_key(oid): if oids.has_key(oid):
continue continue
revid = self._serials[oid] serial, tid = self._getSerialAndTid(oid)
revid = tid
meta = self._metadata[oid+revid] meta = self._metadata[oid+revid]
curvid, nvrevid, lrevid = struct.unpack('>8s8s8s', meta[:24]) curvid, nvrevid, lrevid = struct.unpack('>8s8s8s', meta[:24])
# Our database better be consistent. # Our database better be consistent.
...@@ -551,7 +566,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -551,7 +566,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
self._lock_acquire() self._lock_acquire()
try: try:
# Let KeyErrors percolate up # Let KeyErrors percolate up
revid = self._serials[oid] serial, tid = self._getSerialAndTid(oid)
revid = tid
vid = self._metadata[oid+revid][:8] vid = self._metadata[oid+revid][:8]
if vid == ZERO: if vid == ZERO:
# Not in a version # Not in a version
...@@ -579,7 +595,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -579,7 +595,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
try: try:
# Get the current revid for the object. As per the protocol, let # Get the current revid for the object. As per the protocol, let
# any KeyErrors percolate up. # any KeyErrors percolate up.
revid = self._serials[oid] serial, tid = self._getSerialAndTid(oid)
revid = tid
# Get the metadata associated with this revision of the object. # Get the metadata associated with this revision of the object.
# All we really need is the vid, the non-version revid and the # All we really need is the vid, the non-version revid and the
# pickle pointer revid. # pickle pointer revid.
...@@ -591,7 +608,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -591,7 +608,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# object is living in is the one that was requested, we simply # object is living in is the one that was requested, we simply
# return the current revision's pickle. # return the current revision's pickle.
if vid == ZERO or self._versions.get(vid) == version: if vid == ZERO or self._versions.get(vid) == version:
return self._pickles[oid+lrevid], revid return self._pickles[oid+lrevid], serial
# The object was living in a version, but not the one requested. # The object was living in a version, but not the one requested.
# Semantics here are to return the non-version revision. # Semantics here are to return the non-version revision.
lrevid = self._metadata[oid+nvrevid][16:24] lrevid = self._metadata[oid+nvrevid][16:24]
...@@ -599,6 +616,34 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -599,6 +616,34 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
finally: finally:
self._lock_release() self._lock_release()
def _getSerialAndTid(self, oid):
# For the object, return the curent serial number and transaction id
# of the last transaction that modified the object. Usually these
# will be the same, unless the last transaction was an abortVersion
self._lock_acquire()
try:
data = self._serials[oid]
finally:
self._lock_release()
if len(data) == 8:
return data, data
return data[:8], data[8:]
def _getSerialAndTidMissingOk(self, oid):
# For the object, return the curent serial number and transaction id
# of the last transaction that modified the object. Usually these
# will be the same, unless the last transaction was an abortVersion
self._lock_acquire()
try:
data = self._serials.get(oid)
finally:
self._lock_release()
if data is None:
return None, None
if len(data) == 8:
return data, data
return data[:8], data[8:]
def _loadSerialEx(self, oid, serial): def _loadSerialEx(self, oid, serial):
# Just like loadSerial, except that it returns both the pickle and the # Just like loadSerial, except that it returns both the pickle and the
# version this object revision is living in. # version this object revision is living in.
...@@ -629,7 +674,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -629,7 +674,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# irrespective of any versions. # irrespective of any versions.
self._lock_acquire() self._lock_acquire()
try: try:
return self._serials[oid] serial, tid = self._getSerialAndTid(oid)
return serial
finally: finally:
self._lock_release() self._lock_release()
...@@ -691,7 +737,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -691,7 +737,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# a single transaction. It's not clear though under what # a single transaction. It's not clear though under what
# situations that can occur or what the semantics ought to be. # situations that can occur or what the semantics ought to be.
# For now, we'll assume this doesn't happen. # For now, we'll assume this doesn't happen.
oserial = orevid = self._serials.get(oid) oserial, orevid = self._getSerialAndTidMissingOk(oid)
if oserial is None: if oserial is None:
# There's never been a previous revision of this object, so # There's never been a previous revision of this object, so
# set its non-version revid to zero. # set its non-version revid to zero.
...@@ -778,7 +824,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -778,7 +824,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# that would be restored if we were undoing the current # that would be restored if we were undoing the current
# revision. Otherwise, we attempt application level conflict # revision. Otherwise, we attempt application level conflict
# resolution. If that fails, we raise an exception. # resolution. If that fails, we raise an exception.
revid = self._serials[oid] cserial, ctid = self._getSerialAndTid(oid)
revid = ctid
if revid == tid: if revid == tid:
vid, nvrevid, lrevid, prevrevid = struct.unpack( vid, nvrevid, lrevid, prevrevid = struct.unpack(
'>8s8s8s8s', self._metadata[oid+tid]) '>8s8s8s8s', self._metadata[oid+tid])
...@@ -1012,7 +1059,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -1012,7 +1059,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# start with the most recent revision of the object, then search # start with the most recent revision of the object, then search
# the transaction records backwards until we find enough records. # the transaction records backwards until we find enough records.
history = [] history = []
revid = self._serials[oid] serial, tid = self._getSerialAndTid(oid)
revid = tid
# BAW: Again, let KeyErrors percolate up # BAW: Again, let KeyErrors percolate up
while len(history) < size: while len(history) < size:
# Some information comes out of the revision metadata... # Some information comes out of the revision metadata...
...@@ -1037,7 +1085,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -1037,7 +1085,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
d = {'time' : TimeStamp(revid).timeTime(), d = {'time' : TimeStamp(revid).timeTime(),
'user_name' : user, 'user_name' : user,
'description': desc, 'description': desc,
'serial' : revid, 'serial' : serial,
'version' : retvers, 'version' : retvers,
'size' : len(data), 'size' : len(data),
} }
...@@ -1047,7 +1095,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -1047,7 +1095,7 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
# revision, stopping when we've reached the end. # revision, stopping when we've reached the end.
if previd == ZERO: if previd == ZERO:
break break
revid = previd serial = revid = previd
return history return history
finally: finally:
self._lock_release() self._lock_release()
...@@ -1070,7 +1118,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -1070,7 +1118,8 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
seen[oid] = 1 seen[oid] = 1
reachables[oid] = 1 reachables[oid] = 1
# Get the pickle data for the object's current version # Get the pickle data for the object's current version
revid = self._serials.get(oid) serial, tid = self._getSerialAndTidMissingOk(oid)
revid = tid
if revid is None: if revid is None:
# BAW: how can this happen?! This means that an object is # BAW: how can this happen?! This means that an object is
# holding references to an object that we know nothing about. # holding references to an object that we know nothing about.
...@@ -1280,8 +1329,10 @@ class Full(BerkeleyBase, ConflictResolvingStorage): ...@@ -1280,8 +1329,10 @@ class Full(BerkeleyBase, ConflictResolvingStorage):
packablerevs.setdefault(oid, []).append(key) packablerevs.setdefault(oid, []).append(key)
# Otherwise, if this isn't the current revision for this # Otherwise, if this isn't the current revision for this
# object, then it's packable. # object, then it's packable.
elif self._serials[oid] <> key[8:]: else:
packablerevs.setdefault(oid, []).append(key) serial, tid = self._getSerialAndTid(oid)
if tid <> key[8:]:
packablerevs.setdefault(oid, []).append(key)
finally: finally:
c.close() c.close()
# We now have all the packable revisions we're going to handle. For # We now have all the packable revisions we're going to handle. For
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7