• Kirill Smelkov's avatar
    encoder: Fix string wrt protocol version · e7d96969
    Kirill Smelkov authored
    - we can use BINSTRING* only if protocol >= 1;
    - at protocol 0 we thus have to use text STRING;
    - if protocol >= 3 we have to emit the string as unicode pickle object
      the same way as Python3 does. If we don't do - Python3 won't be
      generally able to load our pickle:
    
    	In [1]: s = b'U\x06\xd0\xbc\xd0\xb8\xd1\x80q\x00.'
    
      	In [2]: from pickletools import dis
    
      	In [3]: dis(s)
      	    0: U    SHORT_BINSTRING 'миÑ\x80'
      	    8: q    BINPUT     0
      	   10: .    STOP
      	highest protocol among opcodes = 1
    
      	In [4]: import pickle
    
      	In [5]: pickle.loads(s)
      	---------------------------------------------------------------------------
      	UnicodeDecodeError                        Traceback (most recent call last)
      	<ipython-input-5-764e4625bc41> in <module>()
      	----> 1 pickle.loads(s)
    
      	UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)
    
    We already decode unicode pickle objects into string, this way
    decode(encode(string)) remains always idempotent.
    e7d96969
encode.go 11.1 KB