Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
W
wendelin
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Douglas
wendelin
Commits
75f5f2ba
Commit
75f5f2ba
authored
Oct 08, 2015
by
Ivan Tyagov
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Pass data stream start end offset only. Adjust test.
parent
503c9033
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
14 additions
and
12 deletions
+14
-12
bt5/erp5_wendelin/SkinTemplateItem/portal_skins/erp5_wendelin/DataStream_transformTail.xml
...m/portal_skins/erp5_wendelin/DataStream_transformTail.xml
+8
-6
bt5/erp5_wendelin/TestTemplateItem/portal_components/test.erp5.testWendelin.py
...tTemplateItem/portal_components/test.erp5.testWendelin.py
+1
-1
bt5/erp5_wendelin/WorkflowTemplateItem/portal_workflow/data_stream_interaction_workflow/scripts/DataStream_appendData.xml
...am_interaction_workflow/scripts/DataStream_appendData.xml
+5
-5
No files found.
bt5/erp5_wendelin/SkinTemplateItem/portal_skins/erp5_wendelin/DataStream_transformTail.xml
View file @
75f5f2ba
...
@@ -50,7 +50,9 @@
...
@@ -50,7 +50,9 @@
</item>
</item>
<item>
<item>
<key>
<string>
_body
</string>
</key>
<key>
<string>
_body
</string>
</key>
<value>
<string>
"""\n
<value>
<string
encoding=
"cdata"
>
<![CDATA[
"""\n
Read tail of a Data Stream and aplly needed transformations.\n
Read tail of a Data Stream and aplly needed transformations.\n
This script is called every time we appendData to a Stream\n
This script is called every time we appendData to a Stream\n
using data_stream_interaction_workflow.\n
using data_stream_interaction_workflow.\n
...
@@ -58,15 +60,15 @@
...
@@ -58,15 +60,15 @@
The idea is to provide close to real time data transformations.\n
The idea is to provide close to real time data transformations.\n
As transformation is quite specific we leave this script empty so developers\n
As transformation is quite specific we leave this script empty so developers\n
can hook in and add needed transformations.\n
can hook in and add needed transformations.\n
\n
Interesting here is to use OffsetIndex property sheet which allows to place\n
marker till where a Data Stream has been processed.\n
"""\n
"""\n
</string>
</value>
assert start_offset < end_offset\n
]]>
</string>
</value>
</item>
</item>
<item>
<item>
<key>
<string>
_params
</string>
</key>
<key>
<string>
_params
</string>
</key>
<value>
<string>
*argument_lis
t
</string>
</value>
<value>
<string>
start_offset, end_offse
t
</string>
</value>
</item>
</item>
<item>
<item>
<key>
<string>
id
</string>
</key>
<key>
<string>
id
</string>
</key>
...
...
bt5/erp5_wendelin/TestTemplateItem/portal_components/test.erp5.testWendelin.py
View file @
75f5f2ba
...
@@ -150,7 +150,7 @@ class Test(ERP5TypeTestCase):
...
@@ -150,7 +150,7 @@ class Test(ERP5TypeTestCase):
# override DataStream_transformTail to actually do transformation on appenData
# override DataStream_transformTail to actually do transformation on appenData
start
=
data_stream
.
getSize
()
start
=
data_stream
.
getSize
()
script_id
=
'DataStream_transformTail'
script_id
=
'DataStream_transformTail'
script_content_list
=
[
"
*argument_lis
t"
,
"""
script_content_list
=
[
"
start_offset, end_offse
t"
,
"""
# created by testWendelin.test_01_1_IngestionTail
# created by testWendelin.test_01_1_IngestionTail
start = %s
start = %s
end = %s
end = %s
...
...
bt5/erp5_wendelin/WorkflowTemplateItem/portal_workflow/data_stream_interaction_workflow/scripts/DataStream_appendData.xml
View file @
75f5f2ba
...
@@ -57,11 +57,11 @@ data_stream = state_change[\'object\']\n
...
@@ -57,11 +57,11 @@ data_stream = state_change[\'object\']\n
argument_list = state_change[\'kwargs\'][\'workflow_method_args\']\n
argument_list = state_change[\'kwargs\'][\'workflow_method_args\']\n
\n
\n
# call you own script to handle newly appended data which \n
# call you own script to handle newly appended data which \n
# is not processed yet\n
# is not processed yet
, pass data stream start end offset only
\n
# Note: this will serialize argument_list to activity\'s mysql table
\n
end_offset = data_stream.getSize()
\n
# in case of big packets this can be slow, in =case of small appends it can be
\n
packet_size = len(argument_list[0])
\n
# acceptable.
\n
start_offset = end_offset - packet_size
\n
data_stream.activate().DataStream_transformTail(
*argument_lis
t)\n
data_stream.activate().DataStream_transformTail(
start_offset, end_offse
t)\n
</string>
</value>
</string>
</value>
</item>
</item>
<item>
<item>
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment