- 09 Oct, 2018 1 commit
-
-
Kirill Smelkov authored
-
- 08 Oct, 2018 4 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
- 02 Oct, 2018 3 commits
-
-
Kirill Smelkov authored
* master: go/zodb: Connection += .At()
-
Kirill Smelkov authored
To know database state corresponding to the connection.
-
Kirill Smelkov authored
-
- 01 Oct, 2018 3 commits
-
-
Kirill Smelkov authored
* master: go/zodb: Don't truncate Tid time precision to 1µs
-
Kirill Smelkov authored
The format of tid assumes ~ ns precision, and it is only formatted to µs precision by default. So don't truncate TimeStamp value when computing it from Tid, and perform the µs-rounding only on formatting. The float numbers are not always exactly as in python. For example the following program tidv = [ 0x0000000000000000, 0x0285cbac258bf266, 0x0285cbad27ae14e6, 0x037969f722a53488, 0x03b84285d71c57dd, 0x03caa84275fc1166, ] for tid in tidv: t = TimeStamp.TimeStamp(p64(tid)) print '0x%016x %s %.9f\t%.9f' % (tid, t, t.timeTime(), t.second()) prints: 0x0000000000000000 1900-01-01 00:00:00.000000 -2208988800.000000000 0.000000000 0x0285cbac258bf266 1979-01-03 21:00:08.800000 284245208.800000191 8.800000185 0x0285cbad27ae14e6 1979-01-03 21:01:09.300001 284245269.300001621 9.300001496 <-- ex here 0x037969f722a53488 2008-10-24 05:11:08.120000 1224825068.119999886 8.119999878 0x03b84285d71c57dd 2016-07-01 09:41:50.416574 1467366110.416574001 50.416573989 0x03caa84275fc1166 2018-10-01 16:34:27.652650 1538411667.652649879 27.652650112 the difference is due to floating point operation ordering, because TimeStamp.timeTime() looses precision - e.g. for marked case: In [8]: '%.10f' % (281566860.000000000 + 9.300001496) Out[8]: '281566869.3000015020' We don't try to mimic float64 behaviour to Python exactly - because it is even different for PURE_PYTHON=y or C TimeStamp implementations. However we don't limit due to that our timestamp precision to only 1µs. In other words we keep on maintaining exact compatibility with Python on printing, but timestamp values itself are now ~ ns precision.
-
Kirill Smelkov authored
-
- 30 Sep, 2018 3 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
- 28 Sep, 2018 3 commits
-
-
Kirill Smelkov authored
* master: go/zodb/fs1/index: Don't rely on []byte being pickled as string
-
Kirill Smelkov authored
As https://github.com/kisielk/og-rek/pull/57 maybe shows []byte was pickling as string only unintentionally and that might change. We are already explicitly checking for string in corresponding index load place: https://lab.nexedi.com/kirr/neo/blob/2dba8607/go/zodb/storage/fs1/index.go#L282 so it is better we also explicitly save the bits as string. If we don't and https://github.com/kisielk/og-rek/pull/57 gets accepted, tests will fail: --- FAIL: TestIndexSaveLoad (0.00s) index_test.go:176: index load: /tmp/t-index893650059/458967662/1.fs.index: pickle @6: invalid oidPrefix: type []uint8 Traceback (most recent call last): File "./py/indexcmp", line 41, in <module> main() File "./py/indexcmp", line 29, in main d2 = fsIndex.load(path2) File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/fsIndex.py", line 138, in load data[ensure_bytes(k)] = fsBucket().fromString(ensure_bytes(v)) File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/fsIndex.py", line 71, in ensure_bytes return s.encode('ascii') if not isinstance(s, bytes) else s AttributeError: 'bytearray' object has no attribute 'encode' --- FAIL: TestIndexSaveToPy (0.04s) index_test.go:218: zodb/py read/compare index: exit status 1
-
Kirill Smelkov authored
-
- 04 Sep, 2018 2 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
- 03 Sep, 2018 3 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
- 31 Aug, 2018 1 commit
-
-
Kirill Smelkov authored
-
- 10 Aug, 2018 5 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
- 09 Aug, 2018 12 commits
-
-
Kirill Smelkov authored
* master: go/zodb/btree: New package to work with ZODB BTrees (draft) go/zodb: DB - application-level handle to database (very draft) go/zodb: Connection: Allow applications to tune live-cache eviction policy go/zodb: Implement Connection go/zodb/internal/weak: New package to handle weak references go/zodb: PyStateful persistency support go/zodb: pydata: Factor out class extraction logic into xpyclass go/zodb: Persistent - the base type to implement IPersistent objects go/zodb: IPersistent + Connection stub go/zodb: pkgdoc: Add section overviewing storage drivers go/zodb: pkgdoc: Put zodbtools reference into "Miscellaneous" section go/zodb: pkgdoc: "Operations" -> "Storage layer" go/zodb: pkgdoc: Stress that objects can reference each other in the database go/zodb: Pkgdoc cosmetics go/zodb: Provide invalid Oid value go/zodb: OpenOptions: Document that NoCache disables prefetch go/transaction: New package to deal with transactions (very draft)
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
Provide minimal support for BTrees.LOBTree Get for now.
-
Kirill Smelkov authored
DB represents a handle to database at application level and contains pool of connections. DB.Open opens database connection. The connection will be automatically put back into DB pool for future reuse after corresponding transaction is complete. DB thus provides service to maintain live objects cache and reuse live objects from transaction to transaction. Note that it is possible to have several DB handles to the same database. This might be useful if application accesses distinctly different sets of objects in different transactions and knows beforehand which set it will be next time. Then, to avoid huge cache misses, it makes sense to keep DB handles opened for every possible case of application access. TODO handle invalidations.
-
Kirill Smelkov authored
For example Wendelin.core wcfs will need to keep some types of objects (e.g. BigFile index) always in RAM for efficiency. Provide corresponding interface that application could use to install such live-cache eviction decision-making tuning.
-
Kirill Smelkov authored
Connection represents an application-level view of a ZODB database. It has groups of in-RAM application-level objects associated with it. The objects are isolated from both changes in further database transactions and from changes to in-RAM objects in other connections. Connection, as objects group manager, is responsible for handling object -> object database references. For this to work it keeps {} oid -> obj dict and uses it to find already loaded object when another object persistently references particular oid. Since it related pydata handling of persistent references is correspondingly implemented in this patch. The dict must keep weak references on objects. The following text explains the rationale: if Connection keeps strong link to obj, just obj.PDeactivate will not fully release obj if there are no references to it from other objects: - deactivate will release obj state (ok) - but there will be still reference from connection `oid -> obj` map to this object, which means the object won't be garbage-collected. -> we can solve it by using "weak" pointers in the map. NOTE we cannot use regular map and arbitrarily manually "gc" entries there periodically: since for an obj we don't know whether other objects are referencing it, we can't just remove obj's oid from the map - if we do so and there are other live objects that reference obj, user code can still reach obj via those references. On the other hand, if another, not yet loaded, object also references obj and gets loaded, traversing reference from that loaded object will load second copy of obj, thus breaking 1 object in db <-> 1 live object invariant: A → B → C ↓ | D <--------- - - -> D2 (wrong) - A activate - D activate - B activate - D gc, A still keeps link on D - C activate -> it needs to get to D, but D was removed from objtab -> new D2 is wrongly created that's why we have to depend on Go's GC to know whether there are still live references left or not. And that in turn means finalizers and thus weak references. some link on the subject: https://groups.google.com/forum/#!topic/golang-nuts/PYWxjT2v6ps
-
Kirill Smelkov authored
We will need weak references to handle {} oid -> obj inside zodb.Connection . In Go world they often say that weak references are not needed at all. Please see however the next patch for detailed rationale for why weak references (finalizers and cooperation from Go's GC in other words) are _required_ in that case.
-
Kirill Smelkov authored
As promised in 354e0e51 (go/zodb: Persistent - the base type to implement IPersistent objects) add support to persistency machinery to set object state from python pickles serialized by ZODB/py. Persistent references are not yet handled. As promised add some very minimal persistent tests.
-
Kirill Smelkov authored
Currently we handle many ways ZODB could serialize a Python class in PyData.ClassName. Since we'll be using this functionality in other places soon - extract it into dedicated function. Since will be also frequently using class.__module__ + "." + class.__name__ don't inline it in ClassName and instead put it into pyclassPath() right away.
-
Kirill Smelkov authored
Add the base type, that types which want to implement persistency could embed, and this way inherit persistent functionality. For example type MyObject struct { Persistent ... } type myObjectState MyObject func (o *myObjectState) DropState() { ... } func (o *myObjectState) SetState(state *mem.Buf) error { ... } Here state management methods (DropState and SetState) will be automatically used by persistency machinery on activation and deactivation. For this to work MyObject class has to be registered to ZODB func init() { t := reflect.TypeOf zodb.RegisterClass("mymodule.MyObject", t(MyObject{}), t(myObjectState)) } and new instances of MyObject has to be created via zodb.NewPersistent: obj := zodb.NewPersistent(reflect.TypeOf(MyObject{}), jar).(*MyObject) SetState corresponds to __setstate__ in Python. However in Go version it is explicitly separated from class's public API - as it is the contract between a class and persistency machinery, not between the class and its user. Notice that SetState takes raw buffer as its argument. In the following patch we'll add SetState cousing (PySetState) that will be taking unpickled objects as the state - exactly how __setstate__ operates in Python. Classes will be able to choose whether to accept state as raw bytes or as a python object. The activation/deactivation is implemented via reference counting. Tests are pending (for PySetState).
-
Kirill Smelkov authored
Add to ZODB/go IPersistent - the interface that is used to represent in-RAM application-level objects that are mirroring objects in database. The interface is modelled after Python's IPersistent https://github.com/zopefoundation/ZODB/blob/3.10.7-4-gb8d7a8567/src/persistent/interfaces.py#L22 but is not exactly equal to it. In particular we support concurrent access to an object from multiple goroutines simultaneously. Due to concurrency support there is no STICKY state, because STICKY is used in CPython version to temporarily pin object in RAM briefly and is not safe to use from multiple threads there. Correspondingly the semantic of PActivate is a bit different from _p_activate - in Go, after an object has been activated, it is guaranteed that it will remain present in RAM until it is explicitly deactivated by user. Please see details of the activation protocol in IPersistent documentation. ZODB/py uses interface (IDataManager) for a persistent-object's jar, but in Go I decided, at least for now, to go without explicit interface at that level. For this reason a concrete type - Connection - will be used, and so its stub is also introduced in the patch, since IPersistent wants to return the connection via PJar.
-