See merge request nexedi/wendelin!54

See merge request nexedi/wendelin!52

See merge request nexedi/wendelin!51

See merge request nexedi/wendelin!50

Since ERP5's 4773fac27bb33f1beec4ed44e5ba05df009a2654 BigFile component is no longer on file-system thus adjust imports accordingly (and add its bt5).

See merge request nexedi/wendelin!47

See merge request nexedi/wendelin!46

See merge request nexedi/wendelin!48

See merge request nexedi/wendelin!49

erp5_wendelin_data: add new category Big Data for business applications and Stream Ingestion category to "use"

- state after ingestion is "validated"

Be explicit and set desired portal_type as otherwise by default Data Stream Bucket will be created and tests will fail for missing API.

See merge request nexedi/wendelin!45

erp5_wendelin_data_lake_ingestion: split ingestion validation is done right after last chunk (eof) is ingested instead of waiting for the alarm
- better handling of data stream hash calculation and publication

erp5_wendelin_data_lake_ingestion:  split files are no longer appended/processed and no data streams are removed anymore
- all ingestions and data streams corresponding to split parts are kept
- client will receive the list of all data streams and it will be in charge of merging the parts during the download
- validate chunk data streams only when full file was ingested
- only process split ingestions when full file was ingested
- calculate full split file size
- calculate hash and add state control during data stream validation
- stop invalidated ingestions

- do not return invalid data streams (filter by reference and state)
- groups split file datastreams by reference
- returns full file size and large hash