Merge branch 'y/restore' into x/catobj

* y/restore: (56 commits)
  zodbrestore - Tool to restore content of a ZODB database from zodbdump output
  zodbcommit: Prepare to compute current serial of an oid lazily
  zodbcommit: Don't forget to call tpc_abort on an error
  Drop support for ZODB3
  tox: Don't run tests agains ZODB+PR183 anymore
  Add way to run tests via nxdtest
  tidrange: test: Fix for py3
  *: dict.keys() returns sequence, not [] on py3
  *: Pass bytes literal into BytesIO
  zodbdump: Use bytes to emit its output
  *: Zodbdump format is semi text-binary: Mark it as such + handle zdump output as binary
  *: Don't use %r to print/report lines/bytes to outside
  zodbinfo: Provide "head" as command to query DB head; Turn "last_tid" into deprecated alias for head
  test/gen_testdata: Fix for ZODB5 > 5.5.1 + preserve database compatibility with ZODB3/py2
  tox: Don't duplicate setup.py on which for-tests dependencies we need
  zodbdump: Default out to stdout in binary mode
  *: s.decode('hex') -> fromhex(s)
  utils: Initialize hashers with bytes
  *: Pass bytes - not unicode - literals to sha1()
  util: Fix ashex for Python3
  ...

.gitignore

 /dist
 /zodbtools.egg-info
+
+/.tox/

.nxdtest

+# setup to run tests on Nexedi testing infrastructure.
+# https://stack.nexedi.com/test_status
+
+TestCase('pytest', ['python',  '-m', 'pytest'], summaryf=PyTest.summary)

CHANGELOG.rst

+Zodbtools change history
+========================
+
+0.0.0.dev8 (2019-03-07)
+-----------------------
+
+- Support using absolute and relative time in tidrange.  One example usage is:
+  ``zodb analyze data.fs 2018-01-01T10:30:00Z..yesterday`` (commit__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/4037002c
+
+- Python3 support progressed (`commit 1`__, 2__, 3__), but zodbtools does not
+  support python3 yet. The test suite was extended to run on python3 (commit__)
+  and also was extended to also run on ZODB with raw extensions from ongoing
+  pull request `#183`__  (commit__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/d6bde57c
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/f16ccfd4
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/b338d004
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/eaa3aec7
+  __ https://github.com/zopefoundation/ZODB/pull/183
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/c50bfb00
+
+
+0.0.0.dev7 (2019-01-11)
+-----------------------
+
+- Fix zodbtools to work with all ZODB3, ZODB4 and ZODB5 (`commit 1`__, 2__,
+  3__, 4__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/425e6656
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/0e5d2f81
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/7a94e312
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/8ff7020c
+
+- Fix `zodb analyze` for the case when history range is empty (`commit 1`__,
+  2__, 3__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/b4824ad5
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/d37746c6
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/474a0559
+
+- Zodbtools is not yet Python3-ready (commit__), but we started to fix it
+  step-by-step (`commit 1`__, 2__, 3__, 4__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/7c5bb0b5
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/7d24147b
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/55853615
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/79aa0c45
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/5e2ed5e7
+
+
+0.0.0.dev6 (2018-12-30)
+-----------------------
+
+- `zodb analyze` can now work with any ZODB storage and supports analyzing a
+  particular range of history (`commit 1`__, 2__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/3ce22f28
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/7ad9e1df
+
+- Add help for specifying TID ranges (commit__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/f7eff5fe
+
+- Always close opened storages (commit__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/9dbe70f3
+
+0.0.0.dev5 (2018-12-13)
+-----------------------
+
+- Start to stabilize `zodb dump` format. The format is close to be stable now
+  and will likely be changed, if at all, only in minor ways (`commit 1`__, 2__,
+  3__, 4__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/75c03368
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/33230940
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/7f0bbf7e
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/624aeb09
+
+- Add `DumpReader` - class to read/parse input in `zodbdump` format (commit__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/dd959b28
+
+- Add `zodb commit` subcommand to commit new transaction into ZODB (commit__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/960c5e17
+
+
+0.0.0.dev4 (2017-04-05)
+-----------------------
+
+- Clarify licensing (`commit 1`__, 2__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/9e4305b8
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/79cf177a
+
+- Add `zodb` tool to drive all subcommands (commit__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/984cfe22
+
+- Add `zodb info` subcommand to print general information about a ZODB database
+  (commit__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/37b9fbde
+
+- Switch to open ZODB storages by URL, not only via ZConfig files. URL support
+  comes from `zodburi` (`commit 1`__, 2__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/82b06413
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/bfeb1690
+
+
+0.0.0.dev3 (2016-11-17)
+-----------------------
+
+- Move Nexedi version of `zodbanalyze` from ERP5 into zodbtools.
+
+  Compared to original `zodbanalyze` Nexedi version is faster, prints not only
+  total, but also current sizes, and supports running on bigger databases where
+  keeping all working set to analyze in RAM is not feasible. It also supports
+  analyzing a Repozo deltafs file directly.
+  (`commit 1`__, 2__, 3__, 4__, 5__, 6__, 7__, 8__, 9__)
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/ab17cf2d
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/1e506a81
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/d86d04dc
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/5fd2c0eb
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/a9346784
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/1a489502
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/8dc37247
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/e4d4762a
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/2e834aaf
+
+
+0.0.0.dev2 (2016-11-17)
+-----------------------
+
+- Add initial draft of `zodbdump` - tool to dump content of a ZODB database
+  (`commit 1`__, 2__).
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/c0a6299f
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/d955f79a
+
+0.0.0.dev1 (2016-11-16)
+-----------------------
+
+- Initial release of zodbtools with `zodbcmp` (`commit 1`__, 2__, 3__).
+
+  We originally tried to put `zodbcmp` into ZODB itself, but Jim Fulton asked__
+  not to load ZODB with scripts anymore. This way zodbtools was created.
+
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/fd6ad1b9
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/66a03ae5
+  __ https://lab.nexedi.com/nexedi/zodbtools/commit/66946b8d
+  __ https://github.com/zopefoundation/ZODB/pull/128#issuecomment-260970932

MANIFEST.in

+include COPYING LICENSE-ZPL.txt README.rst CHANGELOG.rst tox.ini
+recursive-include zodbtools/test/testdata *.fs *.index *.ok *.txt

README.rst

@@ -8,7 +8,9 @@ scripts anymore. So we are here:
 
 __ https://github.com/zopefoundation/ZODB/pull/128#issuecomment-260970932
 
-- `zodb analyze` - analyze FileStorage or repozo deltafs usage.
+- `zodb analyze` - analyze ZODB database or repozo deltafs usage.
 - `zodb cmp` - compare content of two ZODB databases bit-to-bit.
+- `zodb commit` - commit new transaction into a ZODB database.
 - `zodb dump` - dump content of a ZODB database.
+- `zodb restore` - restore content of a ZODB database.
 - `zodb info` - print general information about a ZODB database.

setup.py

@@ -8,9 +8,10 @@ def readfile(path):
 
 setup(
     name        = 'zodbtools',
-    version     = '0.0.0.dev4',
+    version     = '0.0.0.dev8',
     description = 'ZODB-related utilities',
-    long_description = readfile('README.rst'),
+    long_description = '%s\n----\n\n%s' % (
+                            readfile('README.rst'), readfile('CHANGELOG.rst')),
     url         = 'https://lab.nexedi.com/nexedi/zodbtools',
     license     = 'GPLv3+ with wide exception for Open-Source; ZPL 2.1',
     author      = 'Nexedi + Zope Foundation + Community',
@@ -19,23 +20,21 @@ setup(
     keywords    = 'zodb utility tool',
 
     packages    = find_packages(),
-    install_requires = ['ZODB', 'zodburi', 'pygolang >= 0.0.0.dev3', 'six'],
+    install_requires = ['ZODB', 'zodburi', 'zope.interface', 'pygolang >= 0.0.0.dev6', 'six', 'dateparser'],
 
     extras_require = {
-                  'test': ['pytest'],
+                  'test': ['pytest', 'freezegun', 'pytz', 'mock;python_version<="2.7"'],
     },
 
     entry_points= {'console_scripts': ['zodb = zodbtools.zodb:main']},
 
+    # FIXME restore py3 support
     classifiers = [_.strip() for _ in """\
         Development Status :: 3 - Alpha
         Intended Audience :: Developers
         Operating System :: POSIX :: Linux
         Programming Language :: Python :: 2
         Programming Language :: Python :: 2.7
-        Programming Language :: Python :: 3
-        Programming Language :: Python :: 3.4
-        Programming Language :: Python :: 3.5
         Topic :: Database
         Topic :: Utilities
         Framework :: ZODB\

tox.ini

+# zodbtools | tox setup
+[tox]
+envlist = py{27,36,37}-ZODB{4,5}
+
+[testenv]
+deps	=
+    .[test]
+
+    # latest current ZODB 4
+    ZODB4:  ZODB >=4.0, <5.0dev
+    ZODB4:  ZEO  >=4.0, <5.0dev
+    # ZEO4 depends on transaction <2
+    ZODB4:  transaction <2.0dev
+
+    # latest current ZODB 5
+    ZODB5:  ZODB >=5.6,	<6.0dev
+    ZODB5:  ZEO	 >=5.0, <6.0dev
+
+commands= {envpython} -m pytest

zodbtools/help.py

+# -*- coding: utf-8 -*-
 # zodbtools - help topics
-# Copyright (C) 2017  Nexedi SA and Contributors.
-#                     Kirill Smelkov <kirr@nexedi.com>
+# Copyright (C) 2017-2018  Nexedi SA and Contributors.
+#                          Kirill Smelkov <kirr@nexedi.com>
 #
 # This program is free software: you can Use, Study, Modify and Redistribute
 # it under the terms of the GNU General Public License version 3, or (at your
@@ -51,4 +52,39 @@ Please see zodburi documentation for full details:
 http://docs.pylonsproject.org/projects/zodburi/
 """
 
-topic_dict['zurl'] = "specifying database URL", help_zurl
+help_tidrange = """\
+Many zodb commands can be invoked on specific range of database history and
+accept <tidrange> parameter for that. The syntax for <tidrange> is
+
+    tidmin..tidmax
+
+where tidmin and tidmax specify [tidmin, tidmax] range of transactions, ends
+inclusive. Both tidmin and tidmax are optional and default to
+
+    tidmin: 0   (start of database history)
+    tidmax: +∞  (end of database history)
+
+If a tid (tidmin or tidmax) is given, it has to be specified as follows:
+
+    - a 16-digit hex number specifying transaction ID, e.g. 0285cbac258bf266
+    - absolute timestamp, in RFC3339 or RFC822 formats
+    - relative timestamp, e.g. yesterday, 1 week ago
+
+Example tid ranges:
+
+    ..                                  whole database history
+    000000000000aaaa..                  transactions starting from 000000000000aaaa till latest
+    ..000000000000bbbb                  transactions starting from database beginning till 000000000000bbbb
+    000000000000aaaa..000000000000bbbb  transactions starting from 000000000000aaaa till 000000000000bbbb
+    1985-04-12T23:20:50.52Z..2018-01-01T10:30:00Z
+                                        transactions starting from 1985-04-12 at 23 hours
+                                        20 minutes 50 seconds and 520000000 nano seconds
+                                        in UTC till 2018-01-01 at 10 hours 30 minutes in UTC
+    1_week_ago..yesterday               transactions from one week ago until yesterday.
+
+In commands <tidrange> is optional - if it is not given at all, it defaults to
+0..+∞, i.e. to whole database history.
+"""
+
+topic_dict['zurl']      = "specifying database URL",    help_zurl
+topic_dict['tidrange']  = "specifying history range",   help_tidrange

zodbtools/test/conftest.py

+# Copyright (C) 2019  Nexedi SA and Contributors.
+#                     Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+
+import pytest
+from zodbtools.test.testutil import zext_supported
+
+# zext is a test fixture function object that allows to exercise 2 cases:
+#
+# - when ZODB does not have txn.extension_bytes support
+# - when ZODB might have    txn.extension_bytes support
+#
+# in a test, zext should be used as as follows:
+#
+#   def test_something(zext):
+#       # bytes for an extension dict
+#       raw_ext = dumps({...})
+#
+#       # will be either same as raw_ext, or b'' if ZODB lacks txn.extension_bytes support
+#       raw_ext = zext(raw_ext)
+#
+#       # zext.disabled indicates whether testing for non-empty extension was disabled.
+#       if zext.disabled:
+#           ...
+@pytest.fixture(params=['!zext', 'zext'])
+def zext(request):
+    if request.param == '!zext':
+        # txn.extension_bytes is not working - always test with empty extension
+        def _(ext):
+            return b''
+        _.disabled = True
+        return _
+    else:
+        # txn.extension_bytes might be working - test with given extension and
+        # xfail if ZODB does not have necessary support.
+        def _(ext):
+            return ext
+        _.disabled = False
+        if not zext_supported():
+            request.applymarker(pytest.mark.xfail(reason='ZODB does not have txn.extension_bytes support'))
+        return _

zodbtools/test/gen_testdata.py

 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-# Copyright (C) 2017  Nexedi SA and Contributors.
-#                     Kirill Smelkov <kirr@nexedi.com>
+# Copyright (C) 2017-2021  Nexedi SA and Contributors.
+#                          Kirill Smelkov <kirr@nexedi.com>
 #
 # This program is free software: you can Use, Study, Modify and Redistribute
 # it under the terms of the GNU General Public License version 3, or (at your
@@ -39,6 +39,7 @@
 
 from ZODB.FileStorage import FileStorage
 from ZODB import DB
+from ZODB.Connection import TransactionMetaData
 from ZODB.POSException import UndoError
 from persistent import Persistent
 import transaction
@@ -60,7 +61,12 @@ def hex64(packed):
     return '0x%016x' % unpack64(packed)
 
 # make time.time() predictable
-_xtime = time.mktime(time.strptime("04 Jan 1979", "%d %b %Y"))
+_xtime0 = time.mktime(time.strptime("04 Jan 1979", "%d %b %Y"))
+def xtime_reset():
+    global _xtime
+    _xtime = _xtime0
+xtime_reset()
+
 def xtime():
     global _xtime
     _xtime += 1.1
@@ -94,7 +100,7 @@ class Object(Persistent):
 
 # prepare extension dictionary for subject
 alnum = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
-def ext(subj):
+def ext4subj(subj):
     d = {"x-generator": "zodb/py%s (%s)" % (sys.version_info.major, subj)}
 
     # also add some random 'x-cookie'
@@ -107,7 +113,7 @@ def ext(subj):
     d[xcookie] = cookie
 
     # shufle extension dict randomly - to likely trigger different ordering on save
-    keyv = d.keys()
+    keyv = list(d.keys())
     random.shuffle(keyv)
     ext = {}
     for key in keyv:
@@ -115,8 +121,66 @@ def ext(subj):
 
     return ext
 
-# gen_testdb generates test FileStorage database @ outfs_path
-def gen_testdb(outfs_path):
+# run_with_zodb4py2_compat(f) runs f preserving database compatibility with
+# ZODB4/py2, which generates pickles encoded with protocol < 3.
+#
+# ZODB5 started to use protocol 3 and binary for oids starting from ZODB 5.4.0:
+# https://github.com/zopefoundation/ZODB/commit/12ee41c4
+# Undo it, while we generate test database.
+def run_with_zodb4py2_compat(f):
+    import ZODB.ConflictResolution
+    import ZODB.Connection
+    import ZODB.ExportImport
+    import ZODB.FileStorage.FileStorage
+    import ZODB._compat
+    import ZODB.broken
+    import ZODB.fsIndex
+    import ZODB.serialize
+    binary    = getattr(ZODB.serialize, 'binary', None)
+    _protocol = getattr(ZODB.serialize, '_protocol', None)
+    Pz4 = 2
+    try:
+        ZODB.serialize.binary    = bytes
+        # XXX cannot change just ZODB._compat._protocol, because many modules
+        # do `from ZODB._compat import _protocol` and just `import ZODB`
+        # imports many ZODB.X modules. In other words we cannot change
+        # _protocol just in one place.
+        ZODB.ConflictResolution._protocol       = Pz4
+        ZODB.Connection._protocol               = Pz4
+        ZODB.ExportImport._protocol             = Pz4
+        ZODB.FileStorage.FileStorage._protocol  = Pz4
+        ZODB._compat._protocol                  = Pz4
+        ZODB.broken._protocol                   = Pz4
+        ZODB.fsIndex._protocol                  = Pz4
+        ZODB.serialize._protocol                = Pz4
+
+        f()
+    finally:
+        ZODB.serialize.binary    = binary
+        ZODB.ConflictResolution._protocol       = _protocol
+        ZODB.Connection._protocol               = _protocol
+        ZODB.ExportImport._protocol             = _protocol
+        ZODB.FileStorage.FileStorage._protocol  = _protocol
+        ZODB._compat._protocol                  = _protocol
+        ZODB.broken._protocol                   = _protocol
+        ZODB.fsIndex._protocol                  = _protocol
+        ZODB.serialize._protocol                = _protocol
+
+# gen_testdb generates test FileStorage database @ outfs_path.
+#
+# zext indicates whether or not to include non-empty extension into transactions.
+def gen_testdb(outfs_path, zext=True):
+    def _():
+        _gen_testdb(outfs_path, zext)
+    run_with_zodb4py2_compat(_)
+
+def _gen_testdb(outfs_path, zext):
+    xtime_reset()
+
+    ext = ext4subj
+    if not zext:
+        def ext(subj): return {}
+
     logging.basicConfig()
 
     # generate random changes to objects hooked to top-level root by a/b/c/... key
@@ -163,7 +227,7 @@ def gen_testdb(outfs_path):
                 break
 
         # delete an object
-        name = random.choice(root.keys())
+        name = random.choice(list(root.keys()))
         obj = root[name]
         root[name] = Object("%s%i*" % (name, i))
         # NOTE user/ext are kept empty on purpose - to also test this case
@@ -178,14 +242,16 @@ def gen_testdb(outfs_path):
                             ''.join(chr(_) for _ in range(32)),     # <- NOTE all control characters
                         u"delete %i\nalpha beta gamma'delta\"lambda\n\nqqq ..." % i,
                         ext("delete %s" % unpack64(obj._p_oid)))
-        stor.tpc_begin(txn)
-        stor.deleteObject(obj._p_oid, obj_tid_lastchange, txn)
-        stor.tpc_vote(txn)
+        # at low level stor requires ZODB.IStorageTransactionMetaData not txn (ITransaction)
+        txn_stormeta = TransactionMetaData(txn.user, txn.description, txn.extension)
+        stor.tpc_begin(txn_stormeta)
+        stor.deleteObject(obj._p_oid, obj_tid_lastchange, txn_stormeta)
+        stor.tpc_vote(txn_stormeta)
         # TODO different txn status vvv
         # XXX vvv it does the thing, but py fs iterator treats this txn as EOF
         #if i != Niter-1:
-        #    stor.tpc_finish(txn)
-        stor.tpc_finish(txn)
+        #    stor.tpc_finish(txn_stormeta)
+        stor.tpc_finish(txn_stormeta)
 
         # close db & rest not to get conflict errors after we touched stor
         # directly a bit. everything will be reopened on next iteration.
@@ -196,13 +262,22 @@ def gen_testdb(outfs_path):
 # ----------------------------------------
 
 from zodbtools.zodbdump import zodbdump
+from zodbtools.test.testutil import zext_supported
 
 def main():
+    # check that ZODB supports txn.extension_bytes; refuse to work if not.
+    if not zext_supported():
+        raise RuntimeError("gen_testdata must be used with ZODB that supports txn.extension_bytes")
+
     out = "testdata/1"
-    gen_testdb("%s.fs" % out)
-    stor = FileStorage("%s.fs" % out, read_only=True)
-    with open("%s.zdump.ok" % out, "w") as f:
-        zodbdump(stor, None, None, out=f)
+    for zext in [True, False]:
+        dbname = out
+        if not zext:
+            dbname += "_!zext"
+        gen_testdb("%s.fs" % dbname, zext=zext)
+        stor = FileStorage("%s.fs" % dbname, read_only=True)
+        with open("%s.zdump.ok" % dbname, "wb") as f:
+            zodbdump(stor, None, None, out=f)
 
 if __name__ == '__main__':
     main()

zodbtools/test/test_analyze.py

+# -*- coding: utf-8 -*-
+# Copyright (C) 2019 Nexedi SA and Contributors.
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+
+from zodbtools.zodbanalyze import analyze, report
+import os.path
+
+
+def test_zodbanalyze(capsys):
+    for use_dbm in (False, True):
+        report(
+            analyze(
+                os.path.join(os.path.dirname(__file__), "testdata", "1.fs"),
+                use_dbm=use_dbm,
+                delta_fs=False,
+                tidmin=None,
+                tidmax=None,
+            ),
+            csv=False,
+        )
+        captured = capsys.readouterr()
+        assert "Processed 68 records in 59 transactions" in captured.out
+        assert captured.err == ""
+
+    # csv output
+    report(
+        analyze(
+            os.path.join(os.path.dirname(__file__), "testdata", "1.fs"),
+            use_dbm=False,
+            delta_fs=False,
+            tidmin=None,
+            tidmax=None,
+        ),
+        csv=True,
+    )
+    captured = capsys.readouterr()
+    assert (
+        """Class Name,T.Count,T.Bytes,Pct,AvgSize,C.Count,C.Bytes,O.Count,O.Bytes
+persistent.mapping.PersistentMapping,10,1578,45.633314%,157.800000,1,213,9,1365
+__main__.Object,56,1880,54.366686%,33.571429,9,303,47,1577
+"""
+        == captured.out
+    )
+    assert captured.err == ""
+
+    # empty range
+    report(
+        analyze(
+            os.path.join(os.path.dirname(__file__), "testdata", "1.fs"),
+            use_dbm=False,
+            delta_fs=False,
+            tidmin="ffffffffffffffff",
+            tidmax=None,
+        ),
+        csv=False,
+    )
+    captured = capsys.readouterr()
+    assert "# ø\nNo transactions processed\n" == captured.out.encode('utf-8')
+    assert captured.err == ""

zodbtools/test/test_commit.py

+# -*- coding: utf-8 -*-
+# Copyright (C) 2018-2020  Nexedi SA and Contributors.
+#                          Kirill Smelkov <kirr@nexedi.com>
+#                          Jérome Perrin <jerome@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+
+from zodbtools.zodbcommit import zodbcommit
+from zodbtools.zodbdump import zodbdump, Transaction, ObjectData, ObjectDelete, ObjectCopy
+from zodbtools.util import storageFromURL, sha1
+from ZODB.utils import p64, u64, z64
+from ZODB._compat import BytesIO, dumps, _protocol   # XXX can't yet commit with arbitrary ext.bytes
+
+from tempfile import mkdtemp
+from shutil import rmtree
+from golang import func, defer
+
+# verify zodbcommit.
+@func
+def test_zodbcommit(zext):
+    tmpd = mkdtemp('', 'zodbcommit.')
+    defer(lambda: rmtree(tmpd))
+
+    stor = storageFromURL('%s/2.fs' % tmpd)
+    defer(stor.close)
+
+    head = stor.lastTransaction()
+
+    # commit some transactions via zodbcommit and verify if storage dump gives
+    # what is expected.
+    t1 = Transaction(z64, ' ', b'user name', b'description ...', zext(dumps({'a': 'b'}, _protocol)), [
+        ObjectData(p64(1), b'data1', 'sha1', sha1(b'data1')),
+        ObjectData(p64(2), b'data2', 'sha1', sha1(b'data2'))])
+
+    t1.tid = zodbcommit(stor, head, t1)
+
+    t2 = Transaction(z64, ' ', b'user2', b'desc2', b'', [
+        ObjectDelete(p64(2))])
+
+    t2.tid = zodbcommit(stor, t1.tid, t2)
+
+
+    buf = BytesIO()
+    zodbdump(stor, p64(u64(head)+1), None, out=buf)
+    dumped = buf.getvalue()
+
+    assert dumped == b''.join([_.zdump() for _ in (t1, t2)])
+
+    # ObjectCopy. XXX zodbcommit handled ObjectCopy by actually copying data,
+    # not referencing previous transaction via backpointer.
+    t3 = Transaction(z64, ' ', b'user3', b'desc3', b'', [
+        ObjectCopy(p64(1), t1.tid)])
+
+    t3.tid = zodbcommit(stor, t2.tid, t3)
+
+    data1_1, _, _ = stor.loadBefore(p64(1), p64(u64(t1.tid)+1))
+    data1_3, _, _ = stor.loadBefore(p64(1), p64(u64(t3.tid)+1))
+    assert data1_1 == data1_3
+    assert data1_1 == b'data1'  # just in case

zodbtools/test/test_dump.py

-# Copyright (C) 2017  Nexedi SA and Contributors.
-#                     Kirill Smelkov <kirr@nexedi.com>
+# -*- coding: utf-8 -*-
+# Copyright (C) 2017-2020  Nexedi SA and Contributors.
+#                          Kirill Smelkov <kirr@nexedi.com>
+#                          Jérome Perrin <jerome@nexedi.com>
 #
 # This program is free software: you can Use, Study, Modify and Redistribute
 # it under the terms of the GNU General Public License version 3, or (at your
@@ -17,21 +19,129 @@
 # See COPYING file for full licensing terms.
 # See https://www.nexedi.com/licensing for rationale and options.
 
-from zodbtools.zodbdump import zodbdump
+from zodbtools.zodbdump import (
+        zodbdump, DumpReader, Transaction, ObjectDelete, ObjectCopy,
+        ObjectData, HashOnly
+    )
+from zodbtools.util import fromhex
 from ZODB.FileStorage import FileStorage
-from cStringIO import StringIO
+from ZODB.utils import p64
+from io import BytesIO
 
 from os.path import dirname
 
+from zodbtools.test.testutil import zext_supported
+from pytest import raises, xfail
+
 # verify zodbdump output against golden
-def test_zodbdump():
-    tdir = dirname(__file__)
-    stor = FileStorage('%s/testdata/1.fs' % tdir, read_only=True)
+def test_zodbdump(zext):
+    tdir  = dirname(__file__)
+    zkind = '_!zext' if zext.disabled else ''
+    stor  = FileStorage('%s/testdata/1%s.fs' % (tdir, zkind), read_only=True)
 
-    with open('%s/testdata/1.zdump.ok' % tdir) as f:
+    with open('%s/testdata/1%s.zdump.ok' % (tdir, zkind), 'rb') as f:
         dumpok = f.read()
 
-    out = StringIO()
+    out = BytesIO()
     zodbdump(stor, None, None, out=out)
 
     assert out.getvalue() == dumpok
+
+
+# verify zodbdump.DumpReader
+def test_dumpreader():
+    in_ = b"""\
+txn 0123456789abcdef " "
+user "my name"
+description "o la-la..."
+extension "zzz123 def"
+obj 0000000000000001 delete
+obj 0000000000000002 from 0123456789abcdee
+obj 0000000000000003 54 adler32:01234567 -
+obj 0000000000000004 4 sha1:9865d483bc5a94f2e30056fc256ed3066af54d04
+ZZZZ
+obj 0000000000000005 9 crc32:52fdeac5
+ABC
+
+DEF!
+
+txn 0123456789abcdf0 " "
+user "author2"
+description "zzz"
+extension "qqq"
+
+"""
+
+    r = DumpReader(BytesIO(in_))
+    t1 = r.readtxn()
+    assert isinstance(t1, Transaction)
+    assert t1.tid == fromhex('0123456789abcdef')
+    assert t1.user              == b'my name'
+    assert t1.description       == b'o la-la...'
+    assert t1.extension_bytes   == b'zzz123 def'
+    assert len(t1.objv)         == 5
+    _ = t1.objv[0]
+    assert isinstance(_, ObjectDelete)
+    assert _.oid        == p64(1)
+    _ = t1.objv[1]
+    assert isinstance(_, ObjectCopy)
+    assert _.oid        == p64(2)
+    assert _.copy_from  == fromhex('0123456789abcdee')
+    _ = t1.objv[2]
+    assert isinstance(_, ObjectData)
+    assert _.oid        == p64(3)
+    assert _.data       == HashOnly(54)
+    assert _.hashfunc   == 'adler32'
+    assert _.hash_      == fromhex('01234567')
+    _ = t1.objv[3]
+    assert isinstance(_, ObjectData)
+    assert _.oid        == p64(4)
+    assert _.data       == b'ZZZZ'
+    assert _.hashfunc   == 'sha1'
+    assert _.hash_      == fromhex('9865d483bc5a94f2e30056fc256ed3066af54d04')
+    _ = t1.objv[4]
+    assert isinstance(_, ObjectData)
+    assert _.oid        == p64(5)
+    assert _.data       == b'ABC\n\nDEF!'
+    assert _.hashfunc   == 'crc32'
+    assert _.hash_      == fromhex('52fdeac5')
+
+    t2 = r.readtxn()
+    assert isinstance(t2, Transaction)
+    assert t2.tid == fromhex('0123456789abcdf0')
+    assert t2.user              == b'author2'
+    assert t2.description       == b'zzz'
+    assert t2.extension_bytes   == b'qqq'
+    assert t2.objv              == []
+
+    assert r.readtxn() == None
+
+    z = b''.join([_.zdump() for _ in (t1, t2)])
+    assert z == in_
+
+    # unknown hash function
+    r = DumpReader(BytesIO(b"""\
+txn 0000000000000000 " "
+user ""
+description ""
+extension ""
+obj 0000000000000001 1 xyz:0123 -
+
+"""))
+    with raises(RuntimeError) as exc:
+        r.readtxn()
+    assert exc.value.args == ("""+5: invalid line: unknown hash function "xyz" ("obj 0000000000000001 1 xyz:0123 -")""",)
+
+    # data integrity error
+    r = DumpReader(BytesIO(b"""\
+txn 0000000000000000 " "
+user ""
+description ""
+extension ""
+obj 0000000000000001 5 crc32:01234567
+hello
+
+"""))
+    with raises(RuntimeError) as exc:
+        r.readtxn()
+    assert exc.value.args == ("""+6: data corrupt: crc32 = 3610a686, expected 01234567""",)

zodbtools/test/test_restore.py

+# -*- coding: utf-8 -*-
+# Copyright (C) 2021  Nexedi SA and Contributors.
+#                     Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+
+from __future__ import print_function
+
+from zodbtools.zodbrestore import zodbrestore
+from zodbtools.util import storageFromURL
+
+from os.path import dirname
+from tempfile import mkdtemp
+from shutil import rmtree
+from golang import func, defer
+
+# verify zodbrestore.
+@func
+def test_zodbrestore():
+    tmpd = mkdtemp('', 'zodbrestore.')
+    defer(lambda: rmtree(tmpd))
+
+    # restore from testdata/1.zdump.ok and verify it gives result that is
+    # bit-to-bit identical to testdata/1.fs
+    tdata = dirname(__file__) + "/testdata"
+    @func
+    def _():
+        zdump = open("%s/1.zdump.ok" % tdata, 'rb')
+        defer(zdump.close)
+
+        stor = storageFromURL('%s/2.fs' % tmpd)
+        defer(stor.close)
+
+        zodbrestore(stor, zdump)
+    _()
+
+    zfs1 = _readfile("%s/1.fs" % tdata)
+    zfs2 = _readfile("%s/2.fs" % tmpd)
+    assert zfs1 == zfs2
+
+
+# _readfile reads file at path.
+def _readfile(path): # -> data(bytes)
+    with open(path, 'rb') as _:
+        return _.read()

zodbtools/test/test_tidrange.py

+# -*- coding: utf-8 -*-
+# Copyright (C) 2019-2020 Nexedi SA and Contributors.
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+
+import datetime
+import os
+import time
+
+import pytest
+import pytz
+from freezegun import freeze_time
+import tzlocal
+
+from zodbtools.util import TidRangeInvalid, TidInvalid, ashex, parse_tid, parse_tidrange
+from golang import b
+
+
+@pytest.fixture
+def fake_time():
+    """Pytest's fixture to run this test as if now() was 2009-08-30T19:20:00Z
+    and if the machine timezone was Europe/Paris
+    """
+    initial_tz = os.environ.get("TZ")
+    os.environ["TZ"] = "Europe/Paris"
+    time.tzset()
+    tzlocal.reload_localzone()
+
+    reference_time = datetime.datetime(2009, 8, 30, 19, 20, 0, 0,
+                                       pytz.utc).astimezone(
+                                           pytz.timezone("Europe/Paris"))
+    with freeze_time(reference_time):
+        yield
+    del os.environ["TZ"]
+    if initial_tz:
+        os.environ["TZ"] = initial_tz
+    time.tzset()
+
+
+def test_tidrange_tid():
+    assert (
+        b"\x00\x00\x00\x00\x00\x00\xaa\xaa",
+        b"\x00\x00\x00\x00\x00\x00\xbb\xbb",
+    ) == parse_tidrange("000000000000aaaa..000000000000bbbb")
+
+    assert (b"\x00\x00\x00\x00\x00\x00\xaa\xaa",
+            None) == parse_tidrange("000000000000aaaa..")
+
+    assert (None, b"\x00\x00\x00\x00\x00\x00\xbb\xbb"
+            ) == parse_tidrange("..000000000000bbbb")
+
+    assert (None, None) == parse_tidrange("..")
+
+    with pytest.raises(TidRangeInvalid) as exc:
+        parse_tidrange("inv.alid")
+    assert exc.value.args == ("inv.alid", )
+
+    # range is correct, but a TID is invalid
+    with pytest.raises(TidInvalid) as exc:
+        parse_tidrange("invalid..")
+    assert exc.value.args == ("invalid", )
+
+
+def test_tidrange_date():
+    assert (
+        b"\x03\xc4\x85v\x00\x00\x00\x00",
+        b"\x03\xc4\x88\xa0\x00\x00\x00\x00",
+    ) == parse_tidrange(
+        "2018-01-01T10:30:00Z..2018-01-02T00:00:00.000000+00:00")
+
+
+def test_parse_tid():
+    assert b"\x00\x00\x00\x00\x00\x00\xbb\xbb" == parse_tid("000000000000bbbb")
+
+    with pytest.raises(TidInvalid) as exc:
+        parse_tid("invalid")
+    assert exc.value.args == ("invalid", )
+
+    with pytest.raises(TidInvalid) as exc:
+        parse_tid('')
+    assert exc.value.args == ('', )
+
+
+test_parameters = [] # of (reference_time, reference_tid, input_time)
+with open(
+        os.path.join(
+            os.path.dirname(__file__), "testdata",
+            "tid-time-format.txt")) as f:
+    for line in f:
+        line = line.strip()
+        if line and not line.startswith("#"):
+            test_parameters.append(line.split(" ", 2))
+
+
+@pytest.mark.parametrize("reference_time,reference_tid,input_time",
+                         test_parameters)
+def test_parse_tid_time_format(fake_time, reference_time, reference_tid,
+                               input_time):
+    assert b(reference_tid) == ashex(parse_tid(input_time))
+    # check that the reference_tid matches the reference time, mainly
+    # to check that input is defined correctly.
+    assert b(reference_tid) == ashex(parse_tid(reference_time))

zodbtools/test/test_zodb.py

+# -*- coding: utf-8 -*-
+# Copyright (C) 2019  Nexedi SA and Contributors.
+#                     Jérome Perrin <jerome@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+
+import sys
+try:
+    from unittest import mock
+except ImportError:
+    # BBB python2
+    import mock
+
+import pytest
+
+from zodbtools import zodb
+from zodbtools import help as help_module
+
+
+# zodbrun runs zodb.main with argv and returns exit code + captured stdout/stderr.
+def zodbrun(capsys, *argv):
+    with mock.patch.object(sys, 'argv', ('zodb',) + argv), \
+         pytest.raises(SystemExit) as excinfo:
+        zodb.main()
+    assert len(excinfo.value.args) == 1
+    ecode = excinfo.value.args[0]
+    return ecode, capsys.readouterr()
+
+
+def test_main(capsys):
+    e, _ = zodbrun(capsys)
+    assert e == 2
+    assert "" == _.out
+    assert "Zodb is a tool for managing ZODB databases." in _.err
+
+    e, _ = zodbrun(capsys, '-h')
+    assert e == 0
+    assert "Zodb is a tool for managing ZODB databases." in _.out
+    assert "" == _.err
+
+
+@pytest.mark.parametrize(
+    "help_topic",
+    tuple(zodb.command_dict) + tuple(help_module.topic_dict))
+def test_help(capsys, help_topic):
+    e, _ = zodbrun(capsys, 'help', help_topic)
+    assert e == 0
+    assert _.err == ""
+    assert _.out != ""

zodbtools/test/testdata/tid-time-format.txt

+# This is the supported time formats for zodbutils <tidrange>
+# Format of this file is:
+#   <reference time> <tid in hex format> <time input format>
+#
+# These must be run with current time: 2009-08-30T19:20:00Z
+# in Europe/Paris timezone.
+# ( as a timestamp: 1251660000 )
+
+
+# some absolute date formats:
+# RFC3339
+2018-01-01T10:30:00Z 03c4857600000000 2018-01-01T10:30:00Z
+1985-04-12T23:20:50.520000Z 02b914f8d78d4fdf 1985-04-12T23:20:50.52Z
+1996-12-20T00:39:57Z 03189927f3333333 1996-12-19T16:39:57-08:00
+2018-01-01T05:30:00Z 03c4844a00000000 2018-01-01T10:30:00+05:00
+
+# RFC822
+1976-08-26T14:29:00Z 02728aa500000000 26 Aug 76 14:29 GMT
+1976-08-26T12:29:00Z 02728a2d00000000 26 Aug 76 14:29 +02:00
+
+
+# RFC850 -> not supported (by go implementation)
+#2006-01-02T22:04:05Z 036277cc15555555 Monday, 02-Jan-06 15:04:05 MST
+
+# RFC1123 -> not supported (by go implementation)
+#2006-01-02T22:04:05Z 036277cc15555555 Mon, 02 Jan 2006 15:04:05 MST
+#2006-01-02T22:04:05Z 036277cc15555555 Mon, 02 Jan 2006 23:04:05 GMT+1
+
+# explicit UTC timezone
+2018-01-01T10:30:00Z 03c4857600000000 2018-01-01 10:30:00 UTC
+2018-01-02T00:00:00Z 03c488a000000000 2018-01-02 UTC
+
+
+# Relative formats, based on git's test for approxidate
+# (adapted for timezone Europe/Paris and extended a bit)
+2009-08-30T19:20:00Z 03805ec800000000 now
+2009-08-30T19:19:55Z 03805ec7eaaaaaaa 5 seconds ago
+2009-08-30T19:19:55Z 03805ec7eaaaaaaa 5.seconds.ago
+2009-08-30T19:10:00Z 03805ebe00000000 10.minutes.ago
+2009-08-29T19:20:00Z 0380592800000000 yesterday
+2009-08-27T19:20:00Z 03804de800000000 3.days.ago
+2009-08-09T19:20:00Z 037fe8a800000000 3.weeks.ago
+2009-05-30T19:20:00Z 037e53a800000000 3.months.ago
+2009-08-30T19:19:00Z 03805ec700000000 1 minute ago
+2009-08-29T19:20:00Z 0380592800000000 1 day ago
+2009-07-30T19:20:00Z 037fb06800000000 1 month ago
+
+# go's when does not support "chaining" like this
+#2007-05-30T19:20:00Z 036dfaa800000000 2.years.3.months.ago
+
+2009-08-29T04:00:00Z 0380559000000000 6am yesterday
+2009-08-29T16:00:00Z 0380586000000000 6pm yesterday
+
+2009-08-30T01:00:00Z 03805a7c00000000 3:00
+2009-08-30T13:00:00Z 03805d4c00000000 15:00
+2009-08-30T10:00:00Z 03805c9800000000 noon today
+2009-08-29T10:00:00Z 038056f800000000 noon yesterday
+
+# this input is a bit weird also, what does "noon pm" mean?
+# it seems to trigger a bug in python's parser
+#  TypeError: can't compare offset-naive and offset-aware datetimes
+#2009-01-05T12:00:00Z 037b0bd000000000 January 5th noon pm
+
+# this input is "ambiguous"
+#2009-08-29T12:00:00Z 0380577000000000 10am noon
+
+# not supported by date parser
+#2009-08-25T19:20:00Z 038042a800000000 last tuesday
+# non consistent behavior ( go keep current hour:minutes - python use midnight )
+#  this also TypeError on python
+#2009-07-05T00:00:00Z 037f1f4000000000 July 5th
+
+# parsed as month/day (at least for me ... it might depend on some locale settings other than $TZ ?)
+#2009-05-06T00:00:00Z 037dc82000000000 06.05.2009
+
+# go parser is wrong on this one
+#2009-06-06T05:00:00Z 037e77ac00000000 Jun 6, 5AM
+# go parser is wrong on this one
+#2009-06-06T05:00:00Z 037e77ac00000000 5AM Jun 6
+
+2009-06-07T04:00:00Z 037e7d1000000000 6AM, June 7, 2009
+
+# python and go disagree on these two, go see them as 00:00 UTC
+#2008-11-30T23:00:00Z 037a3e4400000000 2008-12-01
+#2009-11-30T23:00:00Z 03826ac400000000 2009-12-01
+#2009-06-04T22:00:00Z 037e706800000000 06/05/2009
+# ( end of tests from git )
+
+# more tests
+
+### works with python implementation, but not supported:
+#2018-01-01T09:30:00Z 03c4853a00000000 le 1er janvier 2018 à 10h30
+#2018-01-01T23:00:00Z 03c4886400000000 2018年1月2日
+
+### some invalid formats that "looks OK"
+# wrong format on timezone (should be 2009-06-01T22:00:00+09:00)
+#2009-06-01T01:00:00Z 037e5a9c00000000 2009-06-01T10:00:00:+09:00
+# day is 34
+# ERROR XXX 2009-06-34T22:00:00Z
+# one digits hour minutes
+# ERROR XXX 2009-06-01T1:2:3
+# month use a captital o instead of O
+# ERROR XXX 2009-O6-01T22:00:00Z

zodbtools/test/testutil.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# Copyright (C) 2019  Nexedi SA and Contributors.
+#                     Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+"""utilities for testing"""
+
+from ZODB.FileStorage import FileStorage
+from ZODB import DB
+import transaction
+
+from tempfile import mkdtemp
+from shutil import rmtree
+from golang import func, defer
+
+# zext_supported checks whether ZODB supports txn.extension_bytes .
+_zext_supported_memo = None
+def zext_supported():
+    global _zext_supported_memo
+    if _zext_supported_memo is not None:
+        return _zext_supported_memo
+
+    _ = _zext_supported_memo = _zext_supported()
+    return _
+
+@func
+def _zext_supported():
+    tmpd = mkdtemp('', 'zext_check.')
+    defer(lambda: rmtree(tmpd))
+    dbfs = tmpd + '/1.fs'
+
+    stor = FileStorage(dbfs, create=True)
+    db   = DB(stor)
+    conn = db.open()
+    root = conn.root()
+    root._p_changed = True
+
+    txn = transaction.get()
+    txn.setExtendedInfo('a', 'b')
+    txn.commit()
+
+    for last_txn in stor.iterator(start=stor.lastTransaction()):
+        break
+    else:
+        assert False, "cannot see details of last transaction"
+
+    assert last_txn.extension == {'a': 'b'}
+    return hasattr(last_txn, 'extension_bytes')

zodbtools/util.py

+# -*- coding: utf-8 -*-
 # zodbtools - various utility routines
-# Copyright (C) 2016-2017  Nexedi SA and Contributors.
+# Copyright (C) 2016-2019  Nexedi SA and Contributors.
 #                          Kirill Smelkov <kirr@nexedi.com>
+#                          Jérome Perrin <jerome@nexedi.com>
 #
 # This program is free software: you can Use, Study, Modify and Redistribute
 # it under the terms of the GNU General Public License version 3, or (at your
@@ -18,14 +20,23 @@
 # See COPYING file for full licensing terms.
 # See https://www.nexedi.com/licensing for rationale and options.
 
-import hashlib
+import hashlib, struct, codecs, io
 import zodburi
 from six.moves.urllib_parse import urlsplit, urlunsplit
+from zlib import crc32, adler32
+from ZODB.TimeStamp import TimeStamp
+import dateparser
 
 def ashex(s):
-    return s.encode('hex')
+    # type: (bytes) -> bytes
+    return codecs.encode(s, 'hex')
+
+def fromhex(s):
+    # type: (Union[str,bytes]) -> bytes
+    return codecs.decode(s, 'hex')
 
 def sha1(data):
+    # type: (bytes) -> bytes
     m = hashlib.sha1()
     m.update(data)
     return m.digest()
@@ -57,20 +68,73 @@ def txnobjv(txn):
     return objv
 
 # "tidmin..tidmax" -> (tidmin, tidmax)
-class TidRangeInvalid(Exception):
+class TidInvalid(ValueError):
+    pass
+
+
+class TidRangeInvalid(ValueError):
     pass
 
+
+def parse_tid(tid_string, raw_only=False):
+    """Try to parse `tid_string` as a time and returns the
+    corresponding raw TID.
+    If `tid_string` cannot be parsed as a time, assume it was
+    already a TID.
+    This function also raise TidRangeInvalid when `tid_string`
+    is invalid.
+    """
+    assert isinstance(tid_string, (str, bytes))
+
+    # If it "looks like a TID", don't try to parse it as time,
+    # because parsing is slow.
+    if len(tid_string) == 16:
+        try:
+            return fromhex(tid_string)
+        except ValueError:
+            pass
+
+    if raw_only:
+        # either it was not 16-char string or hex decoding failed
+        raise TidInvalid(tid_string)
+
+    # preprocess to support `1.day.ago` style formats like git log does.
+    if "ago" in tid_string:
+        tid_string = tid_string.replace(".", " ").replace("_", " ")
+    parsed_time = dateparser.parse(
+        tid_string,
+        settings={
+            'TO_TIMEZONE': 'UTC',
+            'RETURN_AS_TIMEZONE_AWARE': True
+        })
+
+    if not parsed_time:
+        # parsing as date failed
+        raise TidInvalid(tid_string)
+
+    # build a ZODB.TimeStamp to convert as a TID
+    return TimeStamp(
+            parsed_time.year,
+            parsed_time.month,
+            parsed_time.day,
+            parsed_time.hour,
+            parsed_time.minute,
+            parsed_time.second + parsed_time.microsecond / 1000000.).raw()
+
+
+# parse_tidrange parses a string into (tidmin, tidmax).
+#
+# see `zodb help tidrange` for accepted tidrange syntax.
 def parse_tidrange(tidrange):
     try:
         tidmin, tidmax = tidrange.split("..")
     except ValueError:  # not exactly 2 parts in between ".."
         raise TidRangeInvalid(tidrange)
 
-    try:
-        tidmin = tidmin.decode("hex")
-        tidmax = tidmax.decode("hex")
-    except TypeError:   # hex decoding error
-        raise TidRangeInvalid(tidrange)
+    if tidmin:
+        tidmin = parse_tid(tidmin)
+    if tidmax:
+        tidmax = parse_tid(tidmax)
 
     # empty tid means -inf / +inf respectively
     # ( which is None in IStorage.iterator() )
@@ -101,3 +165,73 @@ def storageFromURL(url, read_only=None):
     stor = stor_factory()
 
     return stor
+
+# ---- hashing ----
+
+# hasher that discards data
+class NullHasher:
+    name = "null"
+    digest_size = 1
+
+    def update(self, data):
+        pass
+
+    def digest(self):
+        return b'\0'
+
+    def hexdigest(self):
+        return "00"
+
+# adler32 in hashlib interface
+class Adler32Hasher:
+    name = "adler32"
+    digest_size = 4
+
+    def __init__(self):
+        self._h = adler32(b'')
+
+    def update(self, data):
+        self._h = adler32(data, self._h)
+
+    def digest(self):
+        return struct.pack('>I', self._h & 0xffffffff)
+
+    def hexdigest(self):
+        return '%08x' % (self._h & 0xffffffff)
+
+# crc32 in hashlib interface
+class CRC32Hasher:
+    name = "crc32"
+    digest_size = 4
+
+    def __init__(self):
+        self._h = crc32(b'')
+
+    def update(self, data):
+        self._h = crc32(data, self._h)
+
+    def digest(self):
+        return struct.pack('>I', self._h & 0xffffffff)
+
+    def hexdigest(self):
+        return '%08x' % (self._h & 0xffffffff)
+
+# {} name -> hasher
+hashRegistry = {
+    "null":     NullHasher,
+    "adler32":  Adler32Hasher,
+    "crc32":    CRC32Hasher,
+    "sha1":     hashlib.sha1,
+    "sha256":   hashlib.sha256,
+    "sha512":   hashlib.sha512,
+}
+
+# ---- IO ----
+
+# asbinstream return binary stream associated with stream.
+# For example on py3 sys.stdout is io.TextIO which does not allow to write binary data to it.
+def asbinstream(stream):
+    # type: (IO) -> BinaryIO
+    if isinstance(stream, io.TextIOBase):
+        return stream.buffer
+    return stream

zodbtools/zodb.py

 #!/usr/bin/env python
-# Copyright (C) 2017  Nexedi SA and Contributors.
-#                     Kirill Smelkov <kirr@nexedi.com>
+# -*- coding: utf-8 -*-
+# Copyright (C) 2017-2021  Nexedi SA and Contributors.
+#                          Kirill Smelkov <kirr@nexedi.com>
+#                          Jérome Perrin <jerome@nexedi.com>
 #
 # This program is free software: you can Use, Study, Modify and Redistribute
 # it under the terms of the GNU General Public License version 3, or (at your
@@ -35,7 +37,7 @@ def register_command(cmdname):
     command_module = importlib.import_module('zodbtools.zodb' + cmdname)
     command_dict[cmdname] = command_module
 
-for _ in ('analyze', 'catobj', 'cmp', 'dump', 'info'):
+for _ in ('analyze', 'catobj', 'cmp', 'dump', 'info', 'restore'):
     register_command(_)
 
 
@@ -51,10 +53,7 @@ Usage:
 The commands are:
 """, file=out)
 
-    cmdv = command_dict.keys()
-    cmdv.sort()
-    for cmd in cmdv:
-        cmd_module = command_dict[cmd]
+    for cmd, cmd_module in sorted(command_dict.items()):
         print("    %-11s %s" % (cmd, cmd_module.summary), file=out)
 
     print("""\

zodbtools/zodbcmp.py

 # -*- coding: utf-8 -*-
-# Copyright (C) 2016-2017  Nexedi SA and Contributors.
+# Copyright (C) 2016-2018  Nexedi SA and Contributors.
 #                          Kirill Smelkov <kirr@nexedi.com>
 #
 # This program is free software: you can Use, Study, Modify and Redistribute
@@ -20,7 +20,7 @@
 """Zodbcmp - Tool to compare two ZODB databases
 
 Zodbcmp compares two ZODB databases in between tidmin..tidmax transaction range
-with default range being -∞..+∞ - (whole database).
+with default range being 0..+∞ - (whole database).
 
 For comparison both databases are scanned at storage layer and every
 transaction content is compared bit-to-bit between the two. The program stops
@@ -34,6 +34,7 @@ from __future__ import print_function
 from zodbtools.util import ashex, inf, nextitem, txnobjv, parse_tidrange, TidRangeInvalid,  \
         storageFromURL
 from time import time
+from golang import func, defer
 
 # compare two storage transactions
 # 0 - equal, 1 - non-equal
@@ -120,6 +121,7 @@ Usage: zodb cmp [OPTIONS] <storage1> <storage2> [tidmin..tidmax]
 Compare two ZODB databases.
 
 <storageX> is an URL (see 'zodb help zurl') of a ZODB-storage.
+<tidrange> is a history range (see 'zodb help tidrange') to compare.
 
 Options:
 
@@ -127,6 +129,7 @@ Options:
     -h  --help      show this help
 """, file=out)
 
+@func
 def main2(argv):
     verbose = False
 
@@ -159,8 +162,8 @@ def main2(argv):
             print("E: invalid tidrange: %s" % e, file=sys.stderr)
             sys.exit(2)
 
-    stor1 = storageFromURL(storurl1, read_only=True)
-    stor2 = storageFromURL(storurl2, read_only=True)
+    stor1 = storageFromURL(storurl1, read_only=True);   defer(stor1.close)
+    stor2 = storageFromURL(storurl2, read_only=True);   defer(stor2.close)
 
     zcmp = storcmp(stor1, stor2, tidmin, tidmax, verbose)
     sys.exit(1 if zcmp else 0)

zodbtools/zodbcommit.py

+# Copyright (C) 2018-2021  Nexedi SA and Contributors.
+#                          Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+"""Zodbcommit - Commit new transaction into a ZODB database
+
+Zodbcommit reads transaction description from stdin and commits read data into
+ZODB. The transaction to be committed is read in zodbdump format, but without
+first 'txn' header line. For example::
+
+    user "author"
+    description "change 123"
+    extension ""
+    obj 0000000000000001 4 null:00
+    ZZZZ
+
+On success the ID of committed transaction is printed to stdout.
+On conflict or other problem - the error is printed to stderr exit code is !0.
+
+Zodbcommit requires `at` parameter to be given. This specifies caller idea
+about its current database view and is used to detect conflicting simultaneous
+commits. `at` is required because zodbcommit is plumbing-level command and
+implicitly using storage last_tid instead of it could hide bugs. In scripts one
+can query current database head (last_tid) with `zodb info <stor> last_tid`.
+"""
+
+from __future__ import print_function
+from zodbtools import zodbdump
+from zodbtools.util import ashex, fromhex, storageFromURL
+from ZODB.interfaces import IStorageRestoreable
+from ZODB.utils import p64, u64, z64
+from ZODB.POSException import POSKeyError
+from ZODB._compat import BytesIO
+from golang import func, defer, panic
+import warnings
+
+
+# zodbcommit commits new transaction into ZODB storage with data specified by
+# zodbdump transaction.
+#
+# txn.tid acts as a flag:
+# - with tid=0 the transaction is committed regularly.
+# - with tid=!0 the transaction is recreated with exactly that tid via IStorageRestoreable.
+#
+# tid of created transaction is returned.
+_norestoreWarned = set() # of storage class
+def zodbcommit(stor, at, txn):
+    assert isinstance(txn, zodbdump.Transaction)
+
+    want_restore = (txn.tid != z64)
+    have_restore = IStorageRestoreable.providedBy(stor)
+    # warn once if stor does not implement IStorageRestoreable
+    if want_restore and (not have_restore):
+        if type(stor) not in _norestoreWarned:
+            warnings.warn("restore: %s does not provide IStorageRestoreable ...\n"
+                          "\t... -> will try to emulate it on best-effort basis." %
+                          type(stor), RuntimeWarning)
+            _norestoreWarned.add(type(stor))
+
+    if want_restore:
+        # even if stor might be not providing IStorageRestoreable and not
+        # supporting .restore, it can still support .tpc_begin(tid=...). An example
+        # of this is NEO. We anyway need to be able to specify which transaction ID
+        # we need to restore transaction with.
+        stor.tpc_begin(txn, tid=txn.tid)
+    else:
+        stor.tpc_begin(txn)
+
+    def _():
+        def current_serial(oid):
+            return _serial_at(stor, oid, at)
+        for obj in txn.objv:
+            data = None # data do be committed - setup vvv
+            copy_from = None
+            if isinstance(obj, zodbdump.ObjectCopy):
+                copy_from = obj.copy_from
+                data, _, _ = stor.loadBefore(obj.oid, p64(u64(obj.copy_from)+1))
+
+            elif isinstance(obj, zodbdump.ObjectDelete):
+                data = None
+
+            elif isinstance(obj, zodbdump.ObjectData):
+
+                if isinstance(obj.data, zodbdump.HashOnly):
+                    raise ValueError('cannot commit transaction with hashonly object')
+
+                data = obj.data
+
+            else:
+                panic('invalid object record: %r' % (obj,))
+
+            # we have the data -> restore/store the object.
+            # if it will be ConflictError - we just fail and let the caller retry.
+            if data is None:
+                stor.deleteObject(obj.oid, current_serial(obj.oid), txn)
+            else:
+                if want_restore and have_restore:
+                    stor.restore(obj.oid, txn.tid, data, '', copy_from, txn)
+                else:
+                    # FIXME we don't handle copy_from on commit
+                    # NEO does not support restore, and even if stor supports restore,
+                    # going that way requires to already know tid for transaction we are
+                    # committing. -> we just imitate copy by actually copying data and
+                    # letting the storage deduplicate it.
+                    stor.store(obj.oid, current_serial(obj.oid), data, '', txn)
+
+    try:
+        _()
+        stor.tpc_vote(txn)
+    except:
+        stor.tpc_abort(txn)
+        raise
+
+    # in ZODB >= 5 tpc_finish returns tid directly, but on ZODB 4 it
+    # does not do so. Since we still need to support ZODB 4, utilize tpc_finish
+    # callback to know with which tid the transaction was committed.
+    _ = []
+    stor.tpc_finish(txn, lambda tid: _.append(tid))
+    assert len(_) == 1
+    tid = _[0]
+    if want_restore and (tid != txn.tid):
+        panic('restore: restored transaction has tid=%s, but requested was tid=%s' %
+              (ashex(tid), ashex(txn.tid)))
+    return tid
+
+# _serial_at returns oid's serial as of @at database state.
+def _serial_at(stor, oid, at):
+    before = p64(u64(at)+1)
+    try:
+        xdata = stor.loadBefore(oid, before)
+    except POSKeyError:
+        serial = z64
+    else:
+        if xdata is None:
+            serial = z64
+        else:
+            _, serial, _ = xdata
+    return serial
+
+
+# ----------------------------------------
+import sys, getopt
+
+summary = "commit new transaction into a ZODB database"
+
+def usage(out):
+    print("""\
+Usage: zodb commit [OPTIONS] <storage> <at> < input
+Commit new transaction into a ZODB database.
+
+The transaction to be committed is read from stdin in zodbdump format without
+first 'txn' header line.
+
+<storage> is an URL (see 'zodb help zurl') of a ZODB-storage.
+<at> is transaction ID of what is caller idea about its current database view.
+
+On success the ID of committed transaction is printed to stdout.
+
+Options:
+
+    -h  --help      show this help
+""", file=out)
+
+@func
+def main(argv):
+    try:
+        optv, argv = getopt.getopt(argv[1:], "h", ["help"])
+    except getopt.GetoptError as e:
+        print(e, file=sys.stderr)
+        usage(sys.stderr)
+        sys.exit(2)
+
+    for opt, _ in optv:
+        if opt in ("-h", "--help"):
+            usage(sys.stdout)
+            sys.exit(0)
+
+    if len(argv) != 2:
+        usage(sys.stderr)
+        sys.exit(2)
+
+    storurl = argv[0]
+    at = fromhex(argv[1])
+
+    stor = storageFromURL(storurl)
+    defer(stor.close)
+
+    # artificial transaction header with tid=0 to request regular commit
+    zin = b'txn 0000000000000000 " "\n'
+
+    zin += sys.stdin.read()
+    zin = BytesIO(zin)
+    zr = zodbdump.DumpReader(zin)
+    zr.lineno -= 1                      # we prepended txn header
+    txn = zr.readtxn()
+    tail = zin.read()
+    if tail:
+        print('E: +%d: garbage after transaction' % zr.lineno, file=sys.stderr)
+        sys.exit(1)
+
+    tid = zodbcommit(stor, at, txn)
+    print(ashex(tid))

zodbtools/zodbinfo.py

 # -*- coding: utf-8 -*-
-# Copyright (C) 2017  Nexedi SA and Contributors.
-#                     Kirill Smelkov <kirr@nexedi.com>
+# Copyright (C) 2017-2020  Nexedi SA and Contributors.
+#                          Kirill Smelkov <kirr@nexedi.com>
 #
 # This program is free software: you can Use, Study, Modify and Redistribute
 # it under the terms of the GNU General Public License version 3, or (at your
@@ -22,13 +22,19 @@
 from __future__ import print_function
 from zodbtools.util import ashex, storageFromURL
 from collections import OrderedDict
+from golang import func, defer
 import sys
 
+def _last_tid(stor):
+    print("W: last_tid is deprecated alias for head", file=sys.stderr)
+    return infoDict["head"](stor)
+
 # {} parameter_name -> get_parameter(stor)
 infoDict = OrderedDict([
     ("name", lambda stor: stor.getName()),
     ("size", lambda stor: stor.getSize()),
-    ("last_tid", lambda stor: ashex(stor.lastTransaction())),
+    ("head", lambda stor: ashex(stor.lastTransaction())),
+    ("last_tid", _last_tid),
 ])
 
 def zodbinfo(stor, parameterv):
@@ -71,6 +77,7 @@ Options:
     -h  --help      show this help
 """, file=out)
 
+@func
 def main(argv):
     try:
         optv, argv = getopt.getopt(argv[1:], "h", ["help"])
@@ -91,5 +98,6 @@ def main(argv):
         sys.exit(2)
 
     stor = storageFromURL(storurl, read_only=True)
+    defer(stor.close)
 
     zodbinfo(stor, argv[1:])

zodbtools/zodbrestore.py

+# Copyright (C) 2021  Nexedi SA and Contributors.
+#                     Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+"""Zodbrestore - Restore content of a ZODB database.
+
+Zodbrestore reads transactions from zodbdump output and recreates them in a
+ZODB storage. See Zodbdump documentation for details.
+"""
+
+from __future__ import print_function
+from zodbtools.zodbdump import DumpReader
+from zodbtools.zodbcommit import zodbcommit
+from zodbtools.util import asbinstream, ashex, storageFromURL
+from golang import func, defer
+
+
+# zodbrestore restores transactions read from reader r in zodbdump format.
+#
+# restoredf, if !None, is called for every restored transaction.
+def zodbrestore(stor, r, restoredf=None):
+    zr = DumpReader(r)
+    at = stor.lastTransaction()
+    while 1:
+        txn = zr.readtxn()
+        if txn is None:
+            break
+
+        zodbcommit(stor, at, txn)
+        if restoredf != None:
+            restoredf(txn)
+        at = txn.tid
+
+
+# ----------------------------------------
+import sys, getopt
+
+summary = "restore content of a ZODB database"
+
+def usage(out):
+    print("""\
+Usage: zodb restore [OPTIONS] <storage> < input
+Restore content of a ZODB database.
+
+The transactions to restore are read from stdin in zodbdump format.
+On success the ID of every restored transaction is printed to stdout.
+
+<storage> is an URL (see 'zodb help zurl') of a ZODB-storage.
+
+Options:
+
+    -h  --help      show this help
+""", file=out)
+
+@func
+def main(argv):
+    try:
+        optv, argv = getopt.getopt(argv[1:], "h", ["help"])
+    except getopt.GetoptError as e:
+        print(e, file=sys.stderr)
+        usage(sys.stderr)
+        sys.exit(2)
+
+    for opt, _ in optv:
+        if opt in ("-h", "--help"):
+            usage(sys.stdout)
+            sys.exit(0)
+
+    if len(argv) != 1:
+        usage(sys.stderr)
+        sys.exit(2)
+
+    storurl = argv[0]
+
+    stor = storageFromURL(storurl)
+    defer(stor.close)
+
+    def _(txn):
+        print(ashex(txn.tid))
+    zodbrestore(stor, asbinstream(sys.stdin), _)