Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
W
wendelin
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Nikola Balog
wendelin
Commits
fd728553
Commit
fd728553
authored
Apr 30, 2020
by
Ivan Tyagov
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Add a new interaction workflow which will publish / invalidate all Data Streams per Data Set.
parent
562b39c3
Changes
16
Show whitespace changes
Inline
Side-by-side
Showing
16 changed files
with
560 additions
and
1 deletion
+560
-1
bt5/erp5_wendelin_data_lake_ingestion/PortalTypeWorkflowChainTemplateItem/workflow_chain_type.xml
...rtalTypeWorkflowChainTemplateItem/workflow_chain_type.xml
+1
-1
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataSet_getDataStreamList.py
...kins/erp5_wendelin_data_lake/DataSet_getDataStreamList.py
+13
-0
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataSet_getDataStreamList.xml
...ins/erp5_wendelin_data_lake/DataSet_getDataStreamList.xml
+62
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow.xml
...ateItem/portal_workflow/data_set_interaction_workflow.xml
+46
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions.xml
...l_workflow/data_set_interaction_workflow/interactions.xml
+28
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions/invalidate.xml
...data_set_interaction_workflow/interactions/invalidate.xml
+100
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions/publish.xml
...ow/data_set_interaction_workflow/interactions/publish.xml
+100
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts.xml
...portal_workflow/data_set_interaction_workflow/scripts.xml
+28
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_invalidateDataStreamList.py
...tion_workflow/scripts/DataSet_invalidateDataStreamList.py
+6
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_invalidateDataStreamList.xml
...ion_workflow/scripts/DataSet_invalidateDataStreamList.xml
+62
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_publishDataStreamList.py
...raction_workflow/scripts/DataSet_publishDataStreamList.py
+6
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_publishDataStreamList.xml
...action_workflow/scripts/DataSet_publishDataStreamList.xml
+62
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/variables.xml
...rtal_workflow/data_set_interaction_workflow/variables.xml
+22
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/worklists.xml
...rtal_workflow/data_set_interaction_workflow/worklists.xml
+22
-0
bt5/erp5_wendelin_data_lake_ingestion/bt/template_portal_type_workflow_chain_list
...ake_ingestion/bt/template_portal_type_workflow_chain_list
+1
-0
bt5/erp5_wendelin_data_lake_ingestion/bt/template_workflow_id_list
...wendelin_data_lake_ingestion/bt/template_workflow_id_list
+1
-0
No files found.
bt5/erp5_wendelin_data_lake_ingestion/PortalTypeWorkflowChainTemplateItem/workflow_chain_type.xml
View file @
fd728553
<workflow_chain>
<workflow_chain>
<chain>
<chain>
<type>
Data Set
</type>
<type>
Data Set
</type>
<workflow>
-validation_workflow, data_validation_workflow
</workflow>
<workflow>
-validation_workflow, data_
set_interaction_workflow, data_
validation_workflow
</workflow>
</chain>
</chain>
<chain>
<chain>
<type>
Data Stream
</type>
<type>
Data Stream
</type>
...
...
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataSet_getDataStreamList.py
0 → 100644
View file @
fd728553
"""
Return list of Data Streams belonging to a Date Set.
Data Ingestion line aggregates both Data Set and Data Stream.
Note: This code is quite computationally costly (for Data Streams having thousands of iles) as it needs to:
1. Query MariaDB to find ingestion lines
2. Read from ZODB both Data Ingestion Lines and Data Streams (whoch itself can be big too)
"""
data_ingestion_line_list
=
context
.
portal_catalog
(
portal_type
=
"Data Ingestion Line"
,
aggregate_uid
=
context
.
getUid
())
return
[
x
.
getAggregateValue
(
portal_type
=
"Data Stream"
)
\
for
x
in
data_ingestion_line_list
]
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataSet_getDataStreamList.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"PythonScript"
module=
"Products.PythonScripts.PythonScript"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
Script_magic
</string>
</key>
<value>
<int>
3
</int>
</value>
</item>
<item>
<key>
<string>
_bind_names
</string>
</key>
<value>
<object>
<klass>
<global
name=
"NameAssignments"
module=
"Shared.DC.Scripts.Bindings"
/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key>
<string>
_asgns
</string>
</key>
<value>
<dictionary>
<item>
<key>
<string>
name_container
</string>
</key>
<value>
<string>
container
</string>
</value>
</item>
<item>
<key>
<string>
name_context
</string>
</key>
<value>
<string>
context
</string>
</value>
</item>
<item>
<key>
<string>
name_m_self
</string>
</key>
<value>
<string>
script
</string>
</value>
</item>
<item>
<key>
<string>
name_subpath
</string>
</key>
<value>
<string>
traverse_subpath
</string>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key>
<string>
_params
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
DataSet_getDataStreamList
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"InteractionWorkflowDefinition"
module=
"Products.ERP5.InteractionWorkflow"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
_objects
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
creation_guard
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
description
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
groups
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
data_set_interaction_workflow
</string>
</value>
</item>
<item>
<key>
<string>
manager_bypass
</string>
</key>
<value>
<int>
0
</int>
</value>
</item>
<item>
<key>
<string>
title
</string>
</key>
<value>
<string>
Data Set Interaction Workflow
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"Interaction"
module=
"Products.ERP5.Interaction"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
_mapping
</string>
</key>
<value>
<dictionary/>
</value>
</item>
<item>
<key>
<string>
_objects
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
interactions
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions/invalidate.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"InteractionDefinition"
module=
"Products.ERP5.Interaction"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
actbox_category
</string>
</key>
<value>
<string>
workflow
</string>
</value>
</item>
<item>
<key>
<string>
actbox_name
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
actbox_url
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
activate_script_name
</string>
</key>
<value>
<list>
<string>
DataSet_invalidateDataStreamList
</string>
</list>
</value>
</item>
<item>
<key>
<string>
after_script_name
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
before_commit_script_name
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
description
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
guard
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
invalidate
</string>
</value>
</item>
<item>
<key>
<string>
method_id
</string>
</key>
<value>
<list>
<string>
invalidate
</string>
</list>
</value>
</item>
<item>
<key>
<string>
once_per_transaction
</string>
</key>
<value>
<int>
1
</int>
</value>
</item>
<item>
<key>
<string>
portal_type_filter
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
portal_type_group_filter
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
script_name
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
temporary_document_disallowed
</string>
</key>
<value>
<int>
1
</int>
</value>
</item>
<item>
<key>
<string>
title
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
trigger_type
</string>
</key>
<value>
<int>
2
</int>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions/publish.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"InteractionDefinition"
module=
"Products.ERP5.Interaction"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
actbox_category
</string>
</key>
<value>
<string>
workflow
</string>
</value>
</item>
<item>
<key>
<string>
actbox_name
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
actbox_url
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
activate_script_name
</string>
</key>
<value>
<list>
<string>
DataSet_publishDataStreamList
</string>
</list>
</value>
</item>
<item>
<key>
<string>
after_script_name
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
before_commit_script_name
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
description
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
guard
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
publish
</string>
</value>
</item>
<item>
<key>
<string>
method_id
</string>
</key>
<value>
<list>
<string>
publish
</string>
</list>
</value>
</item>
<item>
<key>
<string>
once_per_transaction
</string>
</key>
<value>
<int>
1
</int>
</value>
</item>
<item>
<key>
<string>
portal_type_filter
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
portal_type_group_filter
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
script_name
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
temporary_document_disallowed
</string>
</key>
<value>
<int>
1
</int>
</value>
</item>
<item>
<key>
<string>
title
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
trigger_type
</string>
</key>
<value>
<int>
2
</int>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"Scripts"
module=
"Products.DCWorkflow.Scripts"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
_mapping
</string>
</key>
<value>
<dictionary/>
</value>
</item>
<item>
<key>
<string>
_objects
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
scripts
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_invalidateDataStreamList.py
0 → 100644
View file @
fd728553
"""
Publish all Data Streams for a Data Set.
"""
data_set
=
state_change
[
'object'
]
for
data_stream
in
data_set
.
DataSet_getDataStreamList
():
data_stream
.
activate
().
invalidate
()
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_invalidateDataStreamList.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"PythonScript"
module=
"Products.PythonScripts.PythonScript"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
Script_magic
</string>
</key>
<value>
<int>
3
</int>
</value>
</item>
<item>
<key>
<string>
_bind_names
</string>
</key>
<value>
<object>
<klass>
<global
name=
"NameAssignments"
module=
"Shared.DC.Scripts.Bindings"
/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key>
<string>
_asgns
</string>
</key>
<value>
<dictionary>
<item>
<key>
<string>
name_container
</string>
</key>
<value>
<string>
container
</string>
</value>
</item>
<item>
<key>
<string>
name_context
</string>
</key>
<value>
<string>
context
</string>
</value>
</item>
<item>
<key>
<string>
name_m_self
</string>
</key>
<value>
<string>
script
</string>
</value>
</item>
<item>
<key>
<string>
name_subpath
</string>
</key>
<value>
<string>
traverse_subpath
</string>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key>
<string>
_params
</string>
</key>
<value>
<string>
state_change,**kw
</string>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
DataSet_invalidateDataStreamList
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_publishDataStreamList.py
0 → 100644
View file @
fd728553
"""
Publish all Data Streams for a Data Set.
"""
data_set
=
state_change
[
'object'
]
for
data_stream
in
data_set
.
DataSet_getDataStreamList
():
data_stream
.
activate
().
publish
()
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_publishDataStreamList.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"PythonScript"
module=
"Products.PythonScripts.PythonScript"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
Script_magic
</string>
</key>
<value>
<int>
3
</int>
</value>
</item>
<item>
<key>
<string>
_bind_names
</string>
</key>
<value>
<object>
<klass>
<global
name=
"NameAssignments"
module=
"Shared.DC.Scripts.Bindings"
/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key>
<string>
_asgns
</string>
</key>
<value>
<dictionary>
<item>
<key>
<string>
name_container
</string>
</key>
<value>
<string>
container
</string>
</value>
</item>
<item>
<key>
<string>
name_context
</string>
</key>
<value>
<string>
context
</string>
</value>
</item>
<item>
<key>
<string>
name_m_self
</string>
</key>
<value>
<string>
script
</string>
</value>
</item>
<item>
<key>
<string>
name_subpath
</string>
</key>
<value>
<string>
traverse_subpath
</string>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key>
<string>
_params
</string>
</key>
<value>
<string>
state_change,**kw
</string>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
DataSet_publishDataStreamList
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/variables.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"Variables"
module=
"Products.DCWorkflow.Variables"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
_mapping
</string>
</key>
<value>
<dictionary/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
variables
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/worklists.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"Worklists"
module=
"Products.DCWorkflow.Worklists"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
_mapping
</string>
</key>
<value>
<dictionary/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
worklists
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/bt/template_portal_type_workflow_chain_list
View file @
fd728553
Data Set | -validation_workflow
Data Set | -validation_workflow
Data Set | data_set_interaction_workflow
Data Set | data_validation_workflow
Data Set | data_validation_workflow
Data Stream | -validation_workflow
Data Stream | -validation_workflow
Data Stream | data_validation_workflow
Data Stream | data_validation_workflow
\ No newline at end of file
bt5/erp5_wendelin_data_lake_ingestion/bt/template_workflow_id_list
View file @
fd728553
data_set_interaction_workflow
data_validation_workflow
data_validation_workflow
\ No newline at end of file
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment