1. 01 Jul, 2015 2 commits
  2. 30 Jun, 2015 5 commits
  3. 29 Jun, 2015 4 commits
    • Kirill Smelkov's avatar
      erp5_wendelin/test/test_01_IngestionFromFluentd: Verify resulting array in full explicitly · a81baf75
      Kirill Smelkov authored
      Instead of verifying only min/max/len of the result, we can verify that the
      result is explicitly the array we expect, especially that it is easier to do
      and less lines with just appropriate arange() and array_equal().
      
      /cc @Tyagov
      a81baf75
    • Kirill Smelkov's avatar
      erp5_wendelin/test/test_01_IngestionFromFluentd: Make sure we process all numbers in 'real_data' · 40acb891
      Kirill Smelkov authored
      As explained in previous commit, real_data tail was ending without \n
      
          99988,99989\n99990,99991,99992,99993,99994,99995,99996,99997,99998,99999\n100000
      
      and because DataStream_copyCSVToDataArray() processes data in full lines only,
      the tail was lost.
      
      Fix t by making sure the last line is always terminated properly with \n.
      
      /cc @Tyagov
      40acb891
    • Kirill Smelkov's avatar
      erp5_wendelin/test/test_01_IngestionFromFluentd: Resulting zarray should be 100000 not 99999 · c3fa8672
      Kirill Smelkov authored
      Consider this:
      
          In [1]: l = range(100000)
      
          In [2]: min(l)
          Out[2]: 0
      
          In [3]: max(l)
          Out[3]: 99999
      
          In [4]: len(l)
          Out[4]: 100000
      
      so if we assert that zarray min=0 and max=99999 the length should be max+1
      which is 100000.
      
      NOTE the length is not 100001, as one would guess from test number sequence
      created at the beginning of the test:
      
        def chunks(l, n):
          """Yield successive n-sized chunks from l."""
          for i in xrange(0, len(l), n):
            yield l[i:i+n]
      
        ...
      
          number_string_list = []
          for my_list in list(chunks(range(0, 100001), 10)):
            number_string_list.append(','.join([str(x) for x in my_list]))
          real_data = '\n'.join(number_string_list)
      
      because processing code "eats" numbers till last \n and for 10001 last \n is located before 100000:
      
        99988,99989\n99990,99991,99992,99993,99994,99995,99996,99997,99998,99999\n100000
      
      I will fix input data generation in the following patch.
      
      /cc @Tyagov
      c3fa8672
    • Kirill Smelkov's avatar
      erp5_wendelin: Fix thinko in DataStream_copyCSVToDataArray() ZBigArray init · bf0adf22
      Kirill Smelkov authored
      When we conditionally create new BigArray for appending data, we should create
      it as empty, because in DataStream_copyCSVToDataArray() creation is done lazily
      only when destination array is not yet initialized and we anyway append data to
      the array in the following code block.
      
      Creating BigArray with initial shape of appending part will result in
      destination array being longer than neccessary by first-appended-chunk length
      with this-way-introduced extra header reading as all zeros.
      
      Fix it.
      
      /cc @Tyagov
      bf0adf22
  4. 26 Jun, 2015 1 commit
  5. 25 Jun, 2015 1 commit
  6. 24 Jun, 2015 3 commits
  7. 22 Jun, 2015 4 commits
  8. 10 Jun, 2015 1 commit
  9. 09 Jun, 2015 2 commits
  10. 05 Jun, 2015 4 commits
  11. 04 Jun, 2015 2 commits
  12. 02 Jun, 2015 6 commits
  13. 29 May, 2015 3 commits
  14. 28 May, 2015 2 commits