1. 28 Feb, 2023 5 commits
    • Levin Zimmermann's avatar
      pandas: Fix unpickle np arrays with py2+pd>0.19.x · d223aede
      Levin Zimmermann authored
      Pandas 0.20.0 introduced a bug fix [1] which changed the behaviour of
      'DataFrame.to_records()', so that the resulting Record objects dtype names are
      unicodes if the data frames column names were unicode. Before this bug fix
      the dtype names were str, no matter whether the column names were str or unicode.
      
      Unfortunately np unpickle breaks if dtype names are unicode [2]. Since
      many of our data frame columns are unicode, loading arrays often
      fails. In python3 this isn't a problem anymore, so until then we fix
      this by introducing a simple monkey patch to pandas, which basically
      reverts the mentioned bug fix.
      
      [1] https://github.com/pandas-dev/pandas/issues/11879
      [2] Small example to reproduce this error:
      
      ''
      import os
      
      import numpy as np
      import pandas as pd
      
      r = pd.DataFrame({u'A':[1,2,3]}).to_records()
      a = np.ndarray(shape=r.shape, dtype=r.dtype.fields)
      p = "t"
      
      try:
        os.remove(p)
      except:
        pass
      
      with open(p, 'wb') as f:
        np.save(f, a)
      with open(p, 'rb') as f:
        np.load(f)
      ''
      
      /reviewed-on nexedi/erp5!1738
      /reviewed-by @jerome @klaus
      d223aede
    • Levin Zimmermann's avatar
      erp5_data_notebook += test un/pickle pd.DF=>np.rec · 611419d0
      Levin Zimmermann authored
      Add test which verifies that we can un/pickle 'pd.DataFrame' which were
      converted to 'np.recarray' via the '.to_records()' method.
      
      This test is necessary because it turns out that the combination of
      pandas >= 0.20.x + python2 doesn't support this functionality by
      default if the data frames column names are unicodes.
      
      Please see nexedi/erp5!1738 (comment 179298) for
      additional context.
      
      /reviewed-on nexedi/erp5!1738
      /reviewed-by @jerome @klaus
      611419d0
    • Levin Zimmermann's avatar
      Restricted: Allow newer pandas versions · f8301a19
      Levin Zimmermann authored
      There are newer pandas versions which still support python2. The latest
      one which still supports py2 (0.24.x) has initially been released in 2019
      and should therefore be quite stable and better supported. We should
      therefore prepare ERP5 so that it support those newer versions.
      
      This patch mostly fixes movements within the internal pandas structure.
      The patch still supports old object locations of previous pandas versions.
      
      /reviewed-on nexedi/erp5!1738
      /reviewed-by @jerome @klaus
      f8301a19
    • Levin Zimmermann's avatar
      restricted: Allow usage of resampler · b5f3e9d8
      Levin Zimmermann authored
      We already use '.resample.mean()' in some projects.
      
      /reviewed-on nexedi/erp5!1738
      /reviewed-by @jerome @klaus
      b5f3e9d8
    • Levin Zimmermann's avatar
      erp5_core_test: restricted: Test pandas resampler · a5696ae9
      Levin Zimmermann authored
      In some projects we use pandas resampler inside 'portal_callables'
      (so with restricted Python). But we miss tests which check
      whether pandas resamplers are allowed inside the restricted Python.
      Due to this lack of tests we won't notice whether the usage of pandas
      resamplers still work with a newer pandas version. With this patch
      we can recognize if an internal change of pandas makes the resampler unusuable
      in restricted Python.
      
      /reviewed-on nexedi/erp5!1738
      /reviewed-by @jerome @klaus
      a5696ae9
  2. 27 Feb, 2023 4 commits
  3. 24 Feb, 2023 4 commits
  4. 23 Feb, 2023 1 commit
  5. 22 Feb, 2023 6 commits
  6. 21 Feb, 2023 5 commits
  7. 18 Feb, 2023 1 commit
  8. 17 Feb, 2023 7 commits
  9. 16 Feb, 2023 7 commits