• Kirill Smelkov's avatar
    go/zodbpickle: Switch to pickling with protocol=3 · 9c9d5304
    Kirill Smelkov authored
    Pickle protocol 3 allows to natively support all py bytes and string
    types and is supported by zodbpickle/py on both py2 and py3 out of the
    box. ZODB/py3 is using protocol=3 since 2013 and ZODB/py2 since 2018. In
    other words nowdays protocol=3 is the default protocol with which
    ZODB emits pickles, and it can be read practically everywhere.
    
    The difference in between protocol=2 and protocol=3 is addition of
    BINBYTES and SHORT_BINBYTES opcodes to represent bytes. Without those
    opcodes bytes are encoded as
    
        `_codecs.encode(byt.decode('latin1'), 'latin1')`
    
    which, when unpickled, results in bytes on py3 and str on py2.
    
    Compared to using `BINBYTES byt` this form is much bigger in size, but
    what is more important might turn bytes back into str when decoded and
    reencoded on py2.
    
    This form of bytes encoding is also not accepted by ZEO/py which rejects
    it with
    
        2024-07-18T20:44:39 ERROR ZEO.asyncio.server Can't deserialize message
        Traceback (most recent call last):
          File "/home/kirr/src/wendelin/z/ZEO5/src/ZEO/asyncio/server.py", line 100, in message_received
            message_id, async_, name, args = self.decode(message)
          File "/home/kirr/src/wendelin/z/ZEO5/src/ZEO/asyncio/marshal.py", line 119, in pickle_server_decode
            return unpickler.load()  # msgid, flags, name, args
          File "/home/kirr/src/wendelin/z/ZEO5/src/ZEO/asyncio/marshal.py", line 176, in server_find_global
            raise ImportError("Module not allowed: %s" % (module,))
        ImportError: Module not allowed: _codecs
    
    All in all using pickle protocol=3 is ok from both py2 and py3 point of
    view, brings size optimization and correctness, and fixes ZEO and
    probably other issues.
    
    So let the pickles ZODB/go saves and other Go places emit be encoded with protocol=3 now as well.
    
    For the reference, the
    
        // TODO 2 -> 3 since ZODB5 switched to it and uses zodbpickle.
    
    is from 2019 added in a16c9e06 (go/zodb: Teach Persistent to serialize
    itself).
    9c9d5304
zodbpickle.go 4.17 KB