Skip to content

W
wendelin
Commit ed280d8d authored by Xiaowu Zhang's avatar Xiaowu Zhang
Browse files
Options

erp5_wendelin: add data mapping 

data mapping is used to map complex data into a single value

the idea is if an object has complet data to process, like (object, value1, value2, value3,value4....), we can mapping (object, value1, value2, value3,value4....) into a single value X, then we process only X to make it faster

here is an use case:

we have two 2D data arrays with 5 columns, the first column is the name of object, the other fours are the differents value of this object

Data ArrayA:

| object | value1 | value2 | value3 | value4 |
| ------ | ------ | ------ | ------ | ------ |
| X | 1 | 2 | 3 | 4 |
| Y | 5 |  6 | 7 | 8 |
| Z | 9 |  10 | 11 | 12 |

Data ArrayB:

| object | value1 | value2 | value3 | value4 |
| ------ | ------ | ------ | ------ | ------ |
| X | 1 | 2 | 3 | 4 |
| Y | 5 |  8 | 7 | 8 |
| Z | 9 |  10 | 192 | 12 |

now we need to compare data array A to data array B to find which object inside A has different value.

without data mapping, we need to compare each object's 4 values, the complexity is O(2^n)

with data mapping:

we map those values:

(X, 1, 2, 3, 4)  ==> 1

(Y, 5, 6, 7, 8)  ==> 2

(Z, 9, 10, 11,12) ==>3

(Y, 5, 8, 7, 8) ==> 4

(Z, 9, 10, 192, 12) ==> 5

Data ArrayA:

| object |
| ------ |
| 1 |
| 2 |
| 3 |

Data ArrayB:

| object |
| ------ |
| 1 |
| 4 |
| 5 |

then compare 1D array is fast, the complexity is O(n)
parent 390704d9
Pipeline #25376 passed with stage
    in 0 seconds
    Hide whitespace changes
    Inline Side-by-side
    Showing with 836 additions and 4 deletions
    bt5/erp5_wendelin/ActionTemplateItem/portal_types/Data%20Mapping%20Module/view.xml 0 → 100644
    View file @ ed280d8d
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="ActionInformation" module="Products.CMFCore.ActionInformation"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>action</string> </key>
    <value>
    <persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
    </value>
    </item>
    <item>
    <key> <string>categories</string> </key>
    <value>
    <tuple>
    <string>action_type/object_list</string>
    </tuple>
    </value>
    </item>
    <item>
    <key> <string>category</string> </key>
    <value> <string>object_list</string> </value>
    </item>
    <item>
    <key> <string>condition</string> </key>
    <value> <string></string> </value>
    </item>
    <item>
    <key> <string>description</string> </key>
    <value> <string></string> </value>
    </item>
    <item>
    <key> <string>icon</string> </key>
    <value> <string></string> </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>view</string> </value>
    </item>
    <item>
    <key> <string>permissions</string> </key>
    <value>
    <tuple>
    <string>View</string>
    </tuple>
    </value>
    </item>
    <item>
    <key> <string>portal_type</string> </key>
    <value> <string>Action Information</string> </value>
    </item>
    <item>
    <key> <string>priority</string> </key>
    <value> <float>1.0</float> </value>
    </item>
    <item>
    <key> <string>title</string> </key>
    <value> <string>View</string> </value>
    </item>
    <item>
    <key> <string>visible</string> </key>
    <value> <int>1</int> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    <record id="2" aka="AAAAAAAAAAI=">
    <pickle>
    <global name="Expression" module="Products.CMFCore.Expression"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>text</string> </key>
    <value> <string>string:${object_url}/DataMappingModule_viewDataMappingList</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_wendelin/ActionTemplateItem/portal_types/Data%20Mapping/view.xml 0 → 100644
    View file @ ed280d8d
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="ActionInformation" module="Products.CMFCore.ActionInformation"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>action</string> </key>
    <value>
    <persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
    </value>
    </item>
    <item>
    <key> <string>categories</string> </key>
    <value>
    <tuple>
    <string>action_type/object_view</string>
    </tuple>
    </value>
    </item>
    <item>
    <key> <string>category</string> </key>
    <value> <string>object_view</string> </value>
    </item>
    <item>
    <key> <string>condition</string> </key>
    <value> <string></string> </value>
    </item>
    <item>
    <key> <string>description</string> </key>
    <value> <string></string> </value>
    </item>
    <item>
    <key> <string>icon</string> </key>
    <value> <string></string> </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>view</string> </value>
    </item>
    <item>
    <key> <string>permissions</string> </key>
    <value>
    <tuple>
    <string>View</string>
    </tuple>
    </value>
    </item>
    <item>
    <key> <string>portal_type</string> </key>
    <value> <string>Action Information</string> </value>
    </item>
    <item>
    <key> <string>priority</string> </key>
    <value> <float>1.0</float> </value>
    </item>
    <item>
    <key> <string>title</string> </key>
    <value> <string>View</string> </value>
    </item>
    <item>
    <key> <string>visible</string> </key>
    <value> <int>1</int> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    <record id="2" aka="AAAAAAAAAAI=">
    <pickle>
    <global name="Expression" module="Products.CMFCore.Expression"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>text</string> </key>
    <value> <string>string:${object_url}/DataMapping_view</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_wendelin/DocumentTemplateItem/portal_components/document.erp5.DataMapping.py 0 → 100644
    View file @ ed280d8d
    # -*- coding: utf-8 -*-
    ##############################################################################
    #
    # Copyright (c) 2022 Nexedi SA and Contributors. All Rights Reserved.
    # Xiaowu Zhang <xiaowu.zhang@nexedi.com>
    #
    # WARNING: This program as such is intended to be used by professional
    # programmers who take the whole responsability of assessing all potential
    # consequences resulting from its eventual inadequacies and bugs
    # End users who are looking for a ready-to-use solution with commercial
    # garantees and support are strongly adviced to contract a Free Software
    # Service Company
    #
    # This program is Free Software; you can redistribute it and/or
    # modify it under the terms of the GNU General Public License
    # as published by the Free Software Foundation; either version 2
    # of the License, or (at your option) any later version.
    #
    # This program is distributed in the hope that it will be useful,
    # but WITHOUT ANY WARRANTY; without even the implied warranty of
    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
    # GNU General Public License for more details.
    #
    # You should have received a copy of the GNU General Public License
    # along with this program; if not, write to the Free Software
    # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
    #
    ##############################################################################
    from BTrees.OLBTree import OLBTree
    from BTrees.LOBTree import LOBTree
    from AccessControl import ClassSecurityInfo
    from Products.ERP5Type import Permissions, PropertySheet
    from erp5.component.document.Document import Document
    class DataMapping(Document):
    """
    data mapping is used to map complex data into a single value
    the idea is if an object has complet data to process, like (object, value1, value2, value3,value4....), we can mapping (object, value1, value2, value3,value4....) into a single value X, then we process only X to make it faster
    here is an use case:
    we have two 2D data arrays with 5 columns, the first column is the name of object, the other fours are the differents value of this object
    Data ArrayA:
    | object | value1 | value2 | value3 | value4 |
    | ------ | ------ | ------ | ------ | ------ |
    | X | 1 | 2 | 3 | 4 |
    | Y | 5 | 6 | 7 | 8 |
    | Z | 9 | 10 | 11 | 12 |
    Data ArrayB:
    | object | value1 | value2 | value3 | value4 |
    | ------ | ------ | ------ | ------ | ------ |
    | X | 1 | 2 | 3 | 4 |
    | Y | 5 | 8 | 7 | 8 |
    | Z | 9 | 10 | 192 | 12 |
    now we need to compare data array A to data array B to find which object inside A has different value.
    without data mapping, we need to compare each object's 4 values, the complexity is O(2^n)
    with data mapping:
    we map those values:
    (X, 1, 2, 3, 4) ==> 1
    (Y, 5, 6, 7, 8) ==> 2
    (Z, 9, 10, 11,12) ==>3
    (Y, 5, 8, 7, 8) ==> 4
    (Z, 9, 10, 192, 12) ==> 5
    Data ArrayA:
    | object |
    | ------ |
    | 1 |
    | 2 |
    | 3 |
    Data ArrayB:
    | object |
    | ------ |
    | 1 |
    | 4 |
    | 5 |
    then compare 1D array is fast, the complexity is O(n)
    """
    meta_type = 'ERP5 Data Mapping'
    portal_type = 'Data Mapping'
    add_permission = Permissions.AddPortalContent
    # Declarative security
    security = ClassSecurityInfo()
    security.declareObjectProtected(Permissions.AccessContentsInformation)
    # Declarative properties
    property_sheets = (
    PropertySheet.CategoryCore,
    PropertySheet.SortIndex
    )
    def __init__(self, *args, **kw):
    self._last_value = 0
    self._object_to_index_tree = OLBTree()
    self._index_to_object_tree = LOBTree()
    Document.__init__(self, *args, **kw)
    security.declareProtected(Permissions.AccessContentsInformation, 'addObject')
    def addObject(self, obj):
    if obj in self._object_to_index_tree:
    return self._object_to_index_tree[obj]
    self._object_to_index_tree[obj] = self._last_value
    self._index_to_object_tree[self._last_value] = obj
    self._last_value += 1
    return self._object_to_index_tree[obj]
    security.declareProtected(Permissions.AccessContentsInformation, 'getValueFromObject')
    def getValueFromObject(self, obj):
    if obj in self._object_to_index_tree:
    return self._object_to_index_tree[obj]
    else:
    return None
    security.declareProtected(Permissions.AccessContentsInformation, 'getObjectFromValue')
    def getObjectFromValue(self, value):
    if value in self._index_to_object_tree:
    return self._index_to_object_tree[value]
    else:
    return None
    security.declareProtected(Permissions.AccessContentsInformation, 'getData')
    def getData(self):
    data_list = []
    for obj in self._object_to_index_tree.keys():
    data_list.append((obj, self._object_to_index_tree[obj]))
    return data_list
    security.declareProtected(Permissions.AccessContentsInformation, 'getSize')
    def getSize(self):
    return len(self._object_to_index_tree.keys())
    bt5/erp5_wendelin/DocumentTemplateItem/portal_components/document.erp5.DataMapping.xml 0 → 100644
    View file @ ed280d8d
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="Document Component" module="erp5.portal_type"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>default_reference</string> </key>
    <value> <string>DataMapping</string> </value>
    </item>
    <item>
    <key> <string>default_source_reference</string> </key>
    <value>
    <none/>
    </value>
    </item>
    <item>
    <key> <string>description</string> </key>
    <value>
    <none/>
    </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>document.erp5.DataMapping</string> </value>
    </item>
    <item>
    <key> <string>portal_type</string> </key>
    <value> <string>Document Component</string> </value>
    </item>
    <item>
    <key> <string>sid</string> </key>
    <value>
    <none/>
    </value>
    </item>
    <item>
    <key> <string>text_content_error_message</string> </key>
    <value>
    <tuple/>
    </value>
    </item>
    <item>
    <key> <string>text_content_warning_message</string> </key>
    <value>
    <tuple/>
    </value>
    </item>
    <item>
    <key> <string>version</string> </key>
    <value> <string>erp5</string> </value>
    </item>
    <item>
    <key> <string>workflow_history</string> </key>
    <value>
    <persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
    </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    <record id="2" aka="AAAAAAAAAAI=">
    <pickle>
    <global name="PersistentMapping" module="Persistence.mapping"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>data</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>component_validation_workflow</string> </key>
    <value>
    <persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
    </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    <record id="3" aka="AAAAAAAAAAM=">
    <pickle>
    <global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_log</string> </key>
    <value>
    <list>
    <dictionary>
    <item>
    <key> <string>action</string> </key>
    <value> <string>validate</string> </value>
    </item>
    <item>
    <key> <string>validation_state</string> </key>
    <value> <string>validated</string> </value>
    </item>
    </dictionary>
    </list>
    </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_wendelin/ModuleTemplateItem/data_mapping_module.xml 0 → 100644
    View file @ ed280d8d
    <module>
    <id>data_mapping_module</id>
    <permission_list>
    <permission type='tuple'>
    <name>Access Transient Objects</name>
    <role>Assignee</role>
    <role>Assignor</role>
    <role>Associate</role>
    <role>Auditor</role>
    <role>Author</role>
    <role>Manager</role>
    </permission>
    <permission type='tuple'>
    <name>Access contents information</name>
    <role>Assignee</role>
    <role>Assignor</role>
    <role>Associate</role>
    <role>Auditor</role>
    <role>Author</role>
    <role>Manager</role>
    </permission>
    <permission type='tuple'>
    <name>Access session data</name>
    <role>Assignee</role>
    <role>Assignor</role>
    <role>Associate</role>
    <role>Auditor</role>
    <role>Author</role>
    <role>Manager</role>
    </permission>
    <permission type='tuple'>
    <name>Add portal content</name>
    <role>Assignor</role>
    <role>Author</role>
    <role>Manager</role>
    </permission>
    <permission type='tuple'>
    <name>Add portal folders</name>
    <role>Assignor</role>
    <role>Author</role>
    <role>Manager</role>
    </permission>
    <permission type='tuple'>
    <name>Change local roles</name>
    <role>Assignor</role>
    <role>Manager</role>
    </permission>
    <permission type='tuple'>
    <name>Copy or Move</name>
    <role>Assignee</role>
    <role>Assignor</role>
    <role>Associate</role>
    <role>Auditor</role>
    <role>Author</role>
    <role>Manager</role>
    </permission>
    <permission type='tuple'>
    <name>Delete objects</name>
    <role>Assignor</role>
    <role>Manager</role>
    </permission>
    <permission type='tuple'>
    <name>List folder contents</name>
    <role>Assignee</role>
    <role>Assignor</role>
    <role>Associate</role>
    <role>Auditor</role>
    <role>Author</role>
    <role>Manager</role>
    </permission>
    <permission type='tuple'>
    <name>Modify portal content</name>
    <role>Assignor</role>
    <role>Manager</role>
    </permission>
    <permission type='tuple'>
    <name>View</name>
    <role>Assignee</role>
    <role>Assignor</role>
    <role>Associate</role>
    <role>Auditor</role>
    <role>Manager</role>
    </permission>
    <permission type='tuple'>
    <name>View History</name>
    <role>Assignee</role>
    <role>Assignor</role>
    <role>Associate</role>
    <role>Auditor</role>
    <role>Author</role>
    <role>Manager</role>
    </permission>
    </permission_list>
    <portal_type>Data Mapping Module</portal_type>
    <title>Data Mappings</title>
    </module>
    \ No newline at end of file
    bt5/erp5_wendelin/PortalTypeAllowedContentTypeTemplateItem/allowed_content_types.xml
    View file @ ed280d8d
    ......@@ -40,6 +40,9 @@
    <portal_type id="Data License Module">
    <item>Data License</item>
    </portal_type>
    <portal_type id="Data Mapping Module">
    <item>Data Mapping</item>
    </portal_type>
    <portal_type id="Data Operation Module">
    <item>Data Operation</item>
    </portal_type>
    ......
    bt5/erp5_wendelin/PortalTypeBaseCategoryTemplateItem/base_category_list.xml
    View file @ ed280d8d
    ......@@ -38,6 +38,9 @@
    <portal_type id="Data License Module">
    <item>business_application</item>
    </portal_type>
    <portal_type id="Data Mapping Module">
    <item>business_application</item>
    </portal_type>
    <portal_type id="Data Operation Module">
    <item>business_application</item>
    </portal_type>
    ......
    bt5/erp5_wendelin/PortalTypeTemplateItem/portal_types/Data%20Mapping%20Module.xml 0 → 100644
    View file @ ed280d8d
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="Base Type" module="erp5.portal_type"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_property_domain_dict</string> </key>
    <value>
    <persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
    </value>
    </item>
    <item>
    <key> <string>content_icon</string> </key>
    <value> <string>folder_icon.gif</string> </value>
    </item>
    <item>
    <key> <string>factory</string> </key>
    <value> <string>addFolder</string> </value>
    </item>
    <item>
    <key> <string>group_list</string> </key>
    <value>
    <tuple>
    <string>module</string>
    </tuple>
    </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>Data Mapping Module</string> </value>
    </item>
    <item>
    <key> <string>portal_type</string> </key>
    <value> <string>Base Type</string> </value>
    </item>
    <item>
    <key> <string>type_class</string> </key>
    <value> <string>Folder</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    <record id="2" aka="AAAAAAAAAAI=">
    <pickle>
    <global name="PersistentMapping" module="Persistence.mapping"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>data</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>short_title</string> </key>
    <value>
    <persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
    </value>
    </item>
    <item>
    <key> <string>title</string> </key>
    <value>
    <persistent> <string encoding="base64">AAAAAAAAAAQ=</string> </persistent>
    </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    <record id="3" aka="AAAAAAAAAAM=">
    <pickle>
    <global name="TranslationInformation" module="Products.ERP5Type.TranslationProviderBase"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>domain_name</string> </key>
    <value> <string>erp5_ui</string> </value>
    </item>
    <item>
    <key> <string>property_name</string> </key>
    <value> <string>short_title</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    <record id="4" aka="AAAAAAAAAAQ=">
    <pickle>
    <global name="TranslationInformation" module="Products.ERP5Type.TranslationProviderBase"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>domain_name</string> </key>
    <value> <string>erp5_ui</string> </value>
    </item>
    <item>
    <key> <string>property_name</string> </key>
    <value> <string>title</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_wendelin/PortalTypeTemplateItem/portal_types/Data%20Mapping.xml 0 → 100644
    View file @ ed280d8d
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="Base Type" module="erp5.portal_type"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>content_icon</string> </key>
    <value>
    <none/>
    </value>
    </item>
    <item>
    <key> <string>description</string> </key>
    <value>
    <none/>
    </value>
    </item>
    <item>
    <key> <string>factory</string> </key>
    <value> <string>addXMLObject</string> </value>
    </item>
    <item>
    <key> <string>group_list</string> </key>
    <value>
    <tuple>
    <string>item</string>
    </tuple>
    </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>Data Mapping</string> </value>
    </item>
    <item>
    <key> <string>init_script</string> </key>
    <value>
    <none/>
    </value>
    </item>
    <item>
    <key> <string>permission</string> </key>
    <value>
    <none/>
    </value>
    </item>
    <item>
    <key> <string>portal_type</string> </key>
    <value> <string>Base Type</string> </value>
    </item>
    <item>
    <key> <string>type_class</string> </key>
    <value> <string>DataMapping</string> </value>
    </item>
    <item>
    <key> <string>type_interface</string> </key>
    <value>
    <tuple/>
    </value>
    </item>
    <item>
    <key> <string>type_mixin</string> </key>
    <value>
    <tuple/>
    </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_wendelin/PortalTypeWorkflowChainTemplateItem/workflow_chain_type.xml
    View file @ ed280d8d
    ......@@ -51,6 +51,10 @@
    <type>Data License</type>
    <workflow>edit_workflow, validation_workflow</workflow>
    </chain>
    <chain>
    <type>Data Mapping</type>
    <workflow>edit_workflow, validation_workflow</workflow>
    </chain>
    <chain>
    <type>Data Operation</type>
    <workflow>edit_workflow, validation_workflow</workflow>
    ......
    bt5/erp5_wendelin/TestTemplateItem/portal_components/test.erp5.testWendelin.py
    View file @ ed280d8d
    ......@@ -759,4 +759,114 @@ result = [x for x in data_bucket_stream.getBucketIndexKeySequenceByIndex()]
    for data_analysis_title in data_analysis_title_list:
    data_analysis = getDataAnalysisByTitle(data_analysis_title)
    self.assertNotEqual(data_analysis, None)
    self.addCleanup(self._removeDocument, data_analysis)
    \ No newline at end of file
    self.addCleanup(self._removeDocument, data_analysis)
    def test_17_DataMapping(self):
    """
    """
    portal = self.portal
    data_mapping = portal.data_mapping_module.newContent(portal_type='Data Mapping')
    self.assertEqual(0, data_mapping.getSize())
    data_list = [
    ('/usr/bin/2to3-2.7', '3ea002bead53f6bdf7', 'fade8568285eb14146a7244', 'f631570af55ee08ecef78f3'),
    ('/usr/bin/R', 'b4c48d52345ae2eb7ca0455db', '59441ddbc00b6521da571', 'a92be1a7acc03f3846'),
    ('/usr/bin/Rscript', 'e97842e556f90be5f7e5', '806725443a01bcae802','1829d887e0c3380ec8f463527'),
    ('/usr/bin/[', '5c348873d0e0abe26d56cc752f0', 'ebb7ae1a78018b62224','9162fdf5b9598c9e9d'),
    ('/usr/bin/aa-enabled', 'c2336b14b7e9d1407d9', 'c8c7af8e6d14a4d1061', '09726542c6c4ca907417c676b4'),
    ('/usr/bin/aa-exec', '7f66cd90da4b27703de299', 'b4bfe90f53a690deb2687342', 'ad36c9002fa84d4e86'),
    ('/usr/bin/aclocal-1.16', '6ba134fb4f97d79a5', '64e518309fc9544b01c164c2e48', 'cc4595fba3251aaa9b48'),
    ('/usr/bin/activate-global-python-argcomplete3', '4007899eba237603a', '8a2584480e9094f9e4761f7', '893ebeeca071795ecd457ed2'),
    ('/usr/bin/addpart', '078a10c409b8b21c3b5137d', '481766f1aa1d393e4a3a', '668374ce14a57b2dd14bb'),
    ('/usr/bin/ansible', 'cb8a161dabae51cf2616c4a85', '31173931d09486697c05', '00ee4c55e8b46f69a54'),
    ('/usr/bin/ansible-connection', 'ffbef34c8d2ae633031f4e8', '61cf10e6b6e4ff33c7ded21244', 'c670ce6af9b645c'),
    ('/usr/bin/ansible-test', '40f528bebb16199d2b11a43b', 'ea6d8680d833fb36e9548a', 'a2c1b309b22cd4285a'),
    ('/usr/bin/appres', '7ccb78e306838a87b68d2c', '7d089e41ee491fc64b2b', 'c3e24ec3fee558e9f06bd0'),
    ('/usr/bin/apt', '3ca60d2b26761b7a50d812', '8b624ea3f31040bd838dfc', '36b5070dd02c87b47512'),
    ('/usr/bin/apt-cache', 'f0e82d30aa1d8a80e', '7da7f514d8056033c4844a6f0', 'be07d8cb7140399252c51c'),
    ('/usr/bin/apt-cdrom', '073483cad694b013a1', 'fb7b20b9450ab7abb4bcc', '2ad240a6670f486e2e1da9daa'),
    ('/usr/bin/apt-config', '32856a9a4e703346d6b8', '407560704903fb078e45dac', 'ee559cd54c2a34f3ad3d4')
    ]
    data_mapped_list = []
    # each different object return a different value
    for data in data_list:
    data_mapped_list.append(data_mapping.addObject(data))
    self.assertEqual(len(data_mapped_list), len(data_list))
    self.assertEqual(len(data_list), data_mapping.getSize())
    # ensure add again same data return always samething
    tmp_list = []
    for data in data_list:
    tmp_list.append(data_mapping.addObject(data))
    self.assertEqual(tmp_list, data_mapped_list)
    # size still same
    self.assertEqual(len(data_list), data_mapping.getSize())
    # ensure we can get original value
    for index in range(len(data_mapped_list)):
    self.assertEqual(data_mapping.getObjectFromValue(data_mapped_list[index]), data_list[index])
    # ensure we can get mapped value
    for index in range(len(data_mapped_list)):
    self.assertEqual(data_mapping.getValueFromObject(data_list[index]), data_mapped_list[index])
    # another data list, /usr/bin/2to3-2.7, /usr/bin/appres, /usr/bin/aclocal-1.16 's value are different compare to previous data
    # so 3 values are different
    another_data_list = [
    ('/usr/bin/2to3-2.7', 'ModifiedValue', 'fade8568285eb14146a7244', 'f631570af55ee08ecef78f3'),
    ('/usr/bin/R', 'b4c48d52345ae2eb7ca0455db', '59441ddbc00b6521da571', 'a92be1a7acc03f3846'),
    ('/usr/bin/Rscript', 'e97842e556f90be5f7e5', '806725443a01bcae802','1829d887e0c3380ec8f463527'),
    ('/usr/bin/[', '5c348873d0e0abe26d56cc752f0', 'ebb7ae1a78018b62224','9162fdf5b9598c9e9d'),
    ('/usr/bin/aa-enabled', 'c2336b14b7e9d1407d9', 'c8c7af8e6d14a4d1061', '09726542c6c4ca907417c676b4'),
    ('/usr/bin/aa-exec', '7f66cd90da4b27703de299', 'b4bfe90f53a690deb2687342', 'ad36c9002fa84d4e86'),
    ('/usr/bin/aclocal-1.16', '6ba134fb4f97d79a5', 'ModifiedValue', 'cc4595fba3251aaa9b48'),
    ('/usr/bin/activate-global-python-argcomplete3', '4007899eba237603a', '8a2584480e9094f9e4761f7', '893ebeeca071795ecd457ed2'),
    ('/usr/bin/addpart', '078a10c409b8b21c3b5137d', '481766f1aa1d393e4a3a', '668374ce14a57b2dd14bb'),
    ('/usr/bin/ansible', 'cb8a161dabae51cf2616c4a85', '31173931d09486697c05', '00ee4c55e8b46f69a54'),
    ('/usr/bin/ansible-connection', 'ffbef34c8d2ae633031f4e8', '61cf10e6b6e4ff33c7ded21244', 'c670ce6af9b645c'),
    ('/usr/bin/ansible-test', '40f528bebb16199d2b11a43b', 'ea6d8680d833fb36e9548a', 'a2c1b309b22cd4285a'),
    ('/usr/bin/appres', '7ccb78e306838a87b68d2c', 'ModifiedValue', 'c3e24ec3fee558e9f06bd0'),
    ('/usr/bin/apt', '3ca60d2b26761b7a50d812', '8b624ea3f31040bd838dfc', '36b5070dd02c87b47512'),
    ('/usr/bin/apt-cache', 'f0e82d30aa1d8a80e', '7da7f514d8056033c4844a6f0', 'be07d8cb7140399252c51c'),
    ('/usr/bin/apt-cdrom', '073483cad694b013a1', 'fb7b20b9450ab7abb4bcc', '2ad240a6670f486e2e1da9daa'),
    ('/usr/bin/apt-config', '32856a9a4e703346d6b8', '407560704903fb078e45dac', 'ee559cd54c2a34f3ad3d4')
    ]
    another_data_mapped_list = []
    for data in another_data_list:
    another_data_mapped_list.append(data_mapping.addObject(data))
    self.assertEqual(len(another_data_mapped_list), len(data_list))
    array = np.array(data_mapped_list)
    another_array = np.array(another_data_mapped_list)
    # simply call setdiff1d to get the different between two datas
    diff_array = np.setdiff1d(another_array, array)
    self.assertEqual(diff_array.size, 3)
    diff_object_list = []
    for value in diff_array:
    diff_object_list.append(data_mapping.getObjectFromValue(value))
    self.assertEqual(diff_object_list, [('/usr/bin/2to3-2.7', 'ModifiedValue', 'fade8568285eb14146a7244', 'f631570af55ee08ecef78f3'),
    ('/usr/bin/aclocal-1.16', '6ba134fb4f97d79a5', 'ModifiedValue', 'cc4595fba3251aaa9b48'),
    ('/usr/bin/appres', '7ccb78e306838a87b68d2c', 'ModifiedValue', 'c3e24ec3fee558e9f06bd0')])
    # same data value as "another_data_list" but in other format
    other_format_data_list = [
    ['/usr/bin/2to3-2.7', 'ModifiedValue', 'fade8568285eb14146a7244', 'f631570af55ee08ecef78f3'],
    ['/usr/bin/R', 'b4c48d52345ae2eb7ca0455db', '59441ddbc00b6521da571', 'a92be1a7acc03f3846'],
    ['/usr/bin/Rscript', 'e97842e556f90be5f7e5', '806725443a01bcae802','1829d887e0c3380ec8f463527']
    ]
    other_format_data_mapped_list = []
    original_size = data_mapping.getSize()
    for data in other_format_data_list:
    other_format_data_mapped_list.append(data_mapping.addObject(data))
    self.assertEqual(original_size + 3, data_mapping.getSize())
    other_format_array = np.array(other_format_data_mapped_list)
    # ensure "even data values are same but format is different" is considered different
    diff_array = np.setdiff1d(other_format_array, another_array)
    self.assertEqual(diff_array.size, 3)
    diff_object_list = []
    for value in diff_array:
    diff_object_list.append(data_mapping.getObjectFromValue(value))
    self.assertEqual(diff_object_list, [['/usr/bin/2to3-2.7', 'ModifiedValue', 'fade8568285eb14146a7244', 'f631570af55ee08ecef78f3'],
    ['/usr/bin/R', 'b4c48d52345ae2eb7ca0455db', '59441ddbc00b6521da571', 'a92be1a7acc03f3846'],
    ['/usr/bin/Rscript', 'e97842e556f90be5f7e5', '806725443a01bcae802','1829d887e0c3380ec8f463527']])
    bt5/erp5_wendelin/bt/template_action_path_list
    View file @ ed280d8d
    ......@@ -35,6 +35,8 @@ Data Ingestion Module | view
    Data Ingestion | view
    Data License Module | view
    Data License | view
    Data Mapping Module | view
    Data Mapping | view
    Data Operation Module | view
    Data Operation | view
    Data Order Line | view
    ......
    bt5/erp5_wendelin/bt/template_document_id_list
    View file @ ed280d8d
    ......@@ -7,4 +7,5 @@ document.erp5.DataArrayViewLine
    document.erp5.DataArrayLineExistenceConstraint
    document.erp5.DataBucketStream
    document.erp5.DataStream
    document.erp5.DataTransformation
    \ No newline at end of file
    document.erp5.DataTransformation
    document.erp5.DataMapping
    \ No newline at end of file
    bt5/erp5_wendelin/bt/template_module_id_list
    View file @ ed280d8d
    ......@@ -8,6 +8,7 @@ data_event_module
    data_ingestion_batch_module
    data_ingestion_module
    data_license_module
    data_mapping_module
    data_operation_module
    data_order_module
    data_product_module
    ......
    bt5/erp5_wendelin/bt/template_portal_type_allowed_content_type_list
    View file @ ed280d8d
    ......@@ -13,6 +13,7 @@ Data Ingestion Batch Module | Data Ingestion Batch
    Data Ingestion Module | Data Ingestion
    Data Ingestion | Data Ingestion Line
    Data License Module | Data License
    Data Mapping Module | Data Mapping
    Data Operation Module | Data Operation
    Data Order Module | Data Order
    Data Order | Data Order Line
    ......
    bt5/erp5_wendelin/bt/template_portal_type_base_category_list
    View file @ ed280d8d
    ......@@ -13,6 +13,7 @@ Data Event | source
    Data Ingestion Batch Module | business_application
    Data Ingestion Module | business_application
    Data License Module | business_application
    Data Mapping Module | business_application
    Data Operation Module | business_application
    Data Order Module | business_application
    Data Product Module | business_application
    ......@@ -24,4 +25,4 @@ Data Stream | successor
    Data Supply Module | business_application
    Data Transformation Module | business_application
    Device Configuration Module | business_application
    Progress Indicator Module | business_application
    Progress Indicator Module | business_application
    \ No newline at end of file
    bt5/erp5_wendelin/bt/template_portal_type_id_list
    View file @ ed280d8d
    ......@@ -24,6 +24,8 @@ Data Ingestion Line
    Data Ingestion Module
    Data License
    Data License Module
    Data Mapping
    Data Mapping Module
    Data Operation
    Data Operation Module
    Data Order
    ......
    bt5/erp5_wendelin/bt/template_portal_type_workflow_chain_list
    View file @ ed280d8d
    ......@@ -21,6 +21,8 @@ Data Ingestion | data_simulation_workflow
    Data Ingestion | edit_workflow
    Data License | edit_workflow
    Data License | validation_workflow
    Data Mapping | edit_workflow
    Data Mapping | validation_workflow
    Data Operation | edit_workflow
    Data Operation | validation_workflow
    Data Order Line | edit_workflow
    ......@@ -47,4 +49,4 @@ Ingestion Policy | validation_workflow
    Progress Indicator | edit_workflow
    Progress Indicator | validation_workflow
    Sensor | edit_workflow
    Sensor | validation_workflow
    Sensor | validation_workflow
    \ No newline at end of file
    Markdown is supported
    0% or
    You are about to add 0 people to the discussion. Proceed with caution.
    Finish editing this message first!
    Please register or to comment
    GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7