Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
E
ebulk
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
ebulk
Commits
e34943b0
Commit
e34943b0
authored
Nov 15, 2018
by
roqueporchetto@gmail.com
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
constants are requested to server script
parent
eab67c55
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
14 additions
and
3 deletions
+14
-3
ebulk-data/embulk-wendelin-dataset-tool/lib/embulk/dataset_utils.rb
.../embulk-wendelin-dataset-tool/lib/embulk/dataset_utils.rb
+14
-3
No files found.
ebulk-data/embulk-wendelin-dataset-tool/lib/embulk/dataset_utils.rb
View file @
e34943b0
require_relative
'filelogger'
require
'digest/md5'
require
'fileutils'
require
'open-uri'
# class that handles dataset tasks report
class
DatasetUtils
begin
uri
=
URI
(
URI
.
escape
(
"https://softinst104003.host.vifib.net/erp5/getIngestionConstantsJson"
))
res
=
open
(
uri
).
read
json
=
JSON
.
parse
(
res
)
EOF
=
json
[
'split_end_suffix'
]
NONE_EXT
=
json
[
'none_extension'
]
REFERENCE_SEPARATOR
=
json
[
'reference_separator'
]
rescue
EOF
=
"EOF"
NONE_EXT
=
"none"
REFERENCE_SEPARATOR
=
"/"
end
DATASET_REPORT_FILE
=
".dataset-task-report"
DATASET_TEMP_REPORT_FILE
=
".temp-dataset-task-report"
DATASET_COMPLETED_FILE
=
".dataset-completed"
...
...
@@ -42,10 +56,7 @@ class DatasetUtils
OUTPUT_RENAMED
=
"renamed: "
MEGA
=
1000000
EOF
=
"EOF"
CHUNK_SIZE
=
50000000
#50mb
NONE_EXT
=
"none"
REFERENCE_SEPARATOR
=
"/"
RECORD_SEPARATOR
=
";"
DATE_FORMAT
=
"%Y-%m-%d-%H-%M-%S"
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment