Skip to content

slapos
slapos
Commit 8206dcdd authored by Ayush Tiwari's avatar Ayush Tiwari Committed by Kirill Smelkov
Browse files
Options

erp5 kernel jupyter 

ipython_notebook SR hooked with ERP5 kernel.
This kernel helps in interaction between erp5 and Jupyter frontend.
The patches have been cleaned up

Features:

- All the code execution is being done at erp5 side, Jupyter just acts as dumb client.
- Receives result as string and its mime_type and thanks to kernel, displays it accordingly.
- Interactions b/w erp5 and Jupyter frontend are based on HTTP requests.

Major changes:

- Addition of erp5 kernel
- Improvement in code according to guidelines(name, section name)
- Use jinja template as instance file and make it more dynamic
- Debugging added for ipython_notebook service.

Note: The certificate authentication changed has been reverted to the previous
one(done by creating wrapper around openssl command) for now.

/cc @Tyagov
/reviewed-by @kirr, @jerome  (on nexedi/slapos!33)
parents ae1aa162 837c05c6
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 617 additions and 126 deletions
software/ipython_notebook/instance.cfg deleted 100644 → 0
View file @ ae1aa162
[buildout]
parts =
instance
publish-connection-parameter
## Monitoring part XXX whe should not have to specify all parts like this
## Parts to add for monitoring
certificate-authority
cron
cron-entry-monitor
cron-entry-rss
deploy-index
deploy-settings-cgi
deploy-status-cgi
deploy-status-history-cgi
setup-static-files
certificate-authority
zero-parameters
public-symlink
cgi-httpd-wrapper
cgi-httpd-graceful-wrapper
monitor-promise
monitor-instance-log-access
## Monitor for ipython
monitor-current-log-access
monitor-deploy-set-password-cgi
extends = ${monitor-template:output}
eggs-directory = ${buildout:eggs-directory}
develop-eggs-directory = ${buildout:develop-eggs-directory}
offline = true
[slapconfiguration]
recipe = slapos.cookbook:slapconfiguration
computer = $${slap_connection:computer_id}
partition = $${slap_connection:partition_id}
url = $${slap_connection:server_url}
key = $${slap_connection:key_file}
cert = $${slap_connection:cert_file}
[instance-parameter]
port = 8888
host = $${slapconfiguration:ipv6-random}
cert_file = $${generate-certificate:cert_file}
key_file = $${generate-certificate:key_file}
logfile = $${directory:log}/ipython_notebook.log
notebook_dir = $${directory:notebook_dir}
[generate-certificate]
; TODO: there is a slapos recipe to generate certificates. Use it instead
recipe = plone.recipe.command
command =
if [ ! -e $${instance-parameter:key_file} ]
then
${openssl-output:openssl} req -x509 -nodes -days 3650 \
-subj "/C=AA/ST=X/L=X/O=Dis/CN=$${instance-parameter:host}" \
-newkey rsa:1024 -keyout $${instance-parameter:key_file} \
-out $${instance-parameter:cert_file}
fi
update-command = $${:command}
cert_file = $${directory:etc}/ipython_notebook_cert.crt
key_file = $${directory:etc}/ipython_notebook_cert.key
[instance]
recipe = slapos.cookbook:wrapper
command-line =
${buildout:bin-directory}/ipython notebook
--no-browser
--matplotlib=inline
--ip=$${instance-parameter:host}
--port=$${instance-parameter:port}
--port-retries=0
--certfile=$${instance-parameter:cert_file}
--keyfile=$${instance-parameter:key_file}
--notebook-dir=$${instance-parameter:notebook_dir}
--logfile=$${instance-parameter:logfile}
--config=$${ipython-notebook-config:rendered}
wrapper-path = $${directory:service}/ipython_notebook
parameters-extra = true
[ipython-notebook-config]
recipe = slapos.recipe.template:jinja2
template = ${ipython-notebook-config:location}/${ipython-notebook-config:filename}
rendered = $${directory:etc}/ipython_notebook_config.py
mode = 0744
context =
raw config_cfg $${buildout:directory}/knowledge0.cfg
[monitor-current-log-access]
< = monitor-directory-access
source = $${instance-parameter:logfile}
[monitor-deploy-set-password-cgi]
recipe = slapos.recipe.template:jinja2
template = ${ipython-notebook-set-password:location}/${ipython-notebook-set-password:filename}
rendered = $${monitor-directory:knowledge0-cgi}/$${:filename}
filename = ipython-notebook-password.cgi
mode = 0744
context =
raw config_cfg $${buildout:directory}/knowledge0.cfg
raw python_executable ${buildout:bin-directory}/ipython
key pwd monitor-directory:knowledge0-cgi
key this_file :filename
key httpd_graceful cgi-httpd-graceful-wrapper:rendered
[directory]
recipe = slapos.cookbook:mkdirectory
home = $${buildout:directory}
etc = $${:home}/etc
var = $${:home}/var
script = $${:etc}/run/
service = $${:etc}/service
promise = $${:etc}/promise/
log = $${:var}/log
notebook_dir = $${:var}/notebooks
[publish-connection-parameter]
recipe = slapos.cookbook:publish
url = https://[$${instance-parameter:host}]:$${instance-parameter:port}
monitor_url = $${monitor-parameters:url}
software/ipython_notebook/instance.cfg.in 0 → 100644
View file @ 8206dcdd
[buildout]
parts =
instance
publish-connection-parameter
## Monitoring part XXX whe should not have to specify all parts like this
## Parts to add for monitoring
certificate-authority
cron
cron-entry-monitor
cron-entry-rss
deploy-index
deploy-settings-cgi
deploy-status-cgi
deploy-status-history-cgi
setup-static-files
certificate-authority
zero-parameters
public-symlink
cgi-httpd-wrapper
cgi-httpd-graceful-wrapper
monitor-promise
monitor-instance-log-access
## Monitor for ipython
monitor-current-log-access
monitor-deploy-set-password-cgi
erp5-kernel
kernel-json
custom-js
extends = {{ monitor_template }}
eggs-directory = {{ eggs_directory }}
develop-eggs-directory = {{ develop_eggs_directory }}
offline = true
[slapconfiguration]
recipe = slapos.cookbook:slapconfiguration
computer = ${slap-connection:computer-id}
partition = ${slap-connection:partition-id}
url = ${slap-connection:server-url}
key = ${slap-connection:key-file}
cert = ${slap-connection:cert-file}
[instance-parameter]
port = 8888
host = ${slapconfiguration:ipv6-random}
cert_file = ${generate-certificate:cert_file}
key_file = ${generate-certificate:key_file}
logfile = ${directory:log}/ipython_notebook.log
notebook_dir = ${directory:notebook_dir}
[dynamic-jinja2-template-base]
recipe = slapos.recipe.template:jinja2
mode = 0644
[generate-certificate]
; TODO: there is a slapos recipe to generate certificates. Use it instead
recipe = plone.recipe.command
command =
if [ ! -e ${instance-parameter:key_file} ]
then
{{ openssl_output }} req -x509 -nodes -days 3650 \
-subj "/C=AA/ST=X/L=X/O=Dis/CN=${instance-parameter:host}" \
-newkey rsa:1024 -keyout ${instance-parameter:key_file} \
-out ${instance-parameter:cert_file}
fi
update-command = ${:command}
cert_file = ${directory:etc}/ipython_notebook_cert.crt
key_file = ${directory:etc}/ipython_notebook_cert.key
[instance]
recipe = slapos.cookbook:wrapper
command-line =
{{ bin_directory }}/ipython notebook
--no-browser
--ip=${instance-parameter:host}
--port=${instance-parameter:port}
--port-retries=0
--certfile=${instance-parameter:cert_file}
--keyfile=${instance-parameter:key_file}
--notebook-dir=${instance-parameter:notebook_dir}
--logfile=${instance-parameter:logfile}
--config=${ipython-notebook-config:rendered}
# Add 'log-level' alias to 'DEBUG' so as to log requests being done while running
# of service ipython_notebook.
--log-level="DEBUG"
wrapper-path = ${directory:service}/ipython_notebook
parameters-extra = true
# Explicitly define IPython Directory to be used while starting ipython notebook
environment = IPYTHONDIR=${directory:ipython_dir}
[ipython-notebook-config]
<= dynamic-jinja2-template-base
template = {{ ipython_notebook_config_location }}/{{ ipython_notebook_config_filename }}
rendered = ${directory:etc}/ipython_notebook_config.py
mode = 0744
context =
raw config_cfg ${buildout:directory}/knowledge0.cfg
[monitor-current-log-access]
< = monitor-directory-access
source = ${instance-parameter:logfile}
[monitor-deploy-set-password-cgi]
<= dynamic-jinja2-template-base
template = {{ ipython_notebook_set_password_location }}/{{ ipython_notebook_set_password_filename }}
rendered = ${monitor-directory:knowledge0-cgi}/${:filename}
filename = ipython-notebook-password.cgi
mode = 0744
context =
raw config_cfg ${buildout:directory}/knowledge0.cfg
raw python_executable {{ bin_directory }}/ipython
key pwd monitor-directory:knowledge0-cgi
key this_file :filename
key httpd_graceful cgi-httpd-graceful-wrapper:rendered
[directory]
recipe = slapos.cookbook:mkdirectory
home = ${buildout:directory}
etc = ${:home}/etc
var = ${:home}/var
script = ${:etc}/run/
service = ${:etc}/service
promise = ${:etc}/promise/
log = ${:var}/log
notebook_dir = ${:var}/notebooks
# Add folders to explicitly define ipython directory
ipython_dir = ${:home}/ipython
ipython_kernel_dir = ${:ipython_dir}/kernels
erp5_kernel_dir = ${:ipython_kernel_dir}/ERP5
[publish-connection-parameter]
recipe = slapos.cookbook:publish.serialised
url = https://[${instance-parameter:host}]:${instance-parameter:port}
monitor_url = ${monitor-parameters:url}
[erp5-kernel]
<= dynamic-jinja2-template-base
template = {{ erp5_kernel_location }}/{{ erp5_kernel_filename }}
rendered = ${directory:erp5_kernel_dir}/ERP5kernel.py
# Use ipython as executable python as we'll be needing requests library in kernel
context =
raw python_executable {{ bin_directory }}/ipython
[kernel-json]
<= dynamic-jinja2-template-base
template = {{ kernel_json_location }}/{{ kernel_json_filename }}
rendered = ${directory:erp5_kernel_dir}/kernel.json
# Use python2.7 executable bin file for kernel config
context =
raw python_executable {{ python_executable }}
key kernel_dir erp5-kernel:rendered
raw display_name ERP5
raw language_name python
[custom-js]
<= dynamic-jinja2-template-base
template = {{ custom_js_location }}/{{ custom_js_filename }}
rendered = ${directory:ipython_dir}/profile_default/static/custom/custom.js
mode = 0744
software/ipython_notebook/software.cfg
View file @ 8206dcdd
......@@ -12,13 +12,13 @@ parts =
monitor-eggs
slapos-cookbook
ipython-notebook
ipython-notebook-set-password
instance
instance-jupyter
[ipython-notebook]
; In the ipython notebook software, we use more eggs than in the minimal
; ipython notebook component
eggs +=
requests
${scipy:egg}
${pandas:egg}
${scikit-learn:egg}
......@@ -40,13 +40,44 @@ md5sum = a5bc4ee8539109d1de7ab33b4c2c97ea
filename = ipython_set_password.cgi.jinja
md5sum = d7d4a7e19d55bf14007819258bf42100
[instance]
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/instance.cfg
output = ${buildout:directory}/instance.cfg
[erp5-kernel]
<= download-file-base
filename = ERP5kernel.py.jinja
md5sum = a6a0ad790d100167c42c354fe5744e11
[kernel-json]
<= download-file-base
filename = kernel.json.jinja
md5sum = ab6e78ea20855e07d388b5b86d1770fe
[custom-js]
<= download-file-base
filename = custom.js.jinja
md5sum = 0e8262d04a6dafbc1b77d95aea2192bc
[instance-jupyter]
recipe = slapos.recipe.template:jinja2
template = ${:_profile_base_location_}/instance.cfg.in
rendered = ${buildout:directory}/template.cfg
mode = 0644
md5sum = 1a993b1f8fa3f001c45075fe95a48332
context =
key bin_directory buildout:bin-directory
key develop_eggs_directory buildout:develop-eggs-directory
key eggs_directory buildout:eggs-directory
key monitor_template monitor-template:output
key openssl_output openssl-output:openssl
key python_executable python2.7:executable
key ipython_notebook_config_location ipython-notebook-config:location
key ipython_notebook_config_filename ipython-notebook-config:filename
key ipython_notebook_set_password_location ipython-notebook-set-password:location
key ipython_notebook_set_password_filename ipython-notebook-set-password:filename
key erp5_kernel_location erp5-kernel:location
key erp5_kernel_filename erp5-kernel:filename
key kernel_json_location kernel-json:location
key kernel_json_filename kernel-json:filename
key custom_js_location custom-js:location
key custom_js_filename custom-js:filename
[versions]
PyRSS2Gen = 1.1
......@@ -65,6 +96,7 @@ scipy = 0.15.1
slapos.recipe.template = 2.8
terminado = 0.5
tornado = 4.2
requests = 2.7.0
# Required by:
# tornado==4.2
......
software/ipython_notebook/template/ERP5kernel.py.jinja 0 → 100644
View file @ 8206dcdd
This diff is collapsed.
software/ipython_notebook/template/custom.js.jinja 0 → 100644
View file @ 8206dcdd
// leave at least 2 line with only a star on it below, or doc generation fails
/**
*
*
* Placeholder for custom user javascript
* mainly to be overridden in profile/static/custom/custom.js
* This will always be an empty file in IPython
*
* User could add any javascript in the `profile/static/custom/custom.js` file.
* It will be executed by the ipython notebook at load time.
*
* Same thing with `profile/static/custom/custom.css` to inject custom css into the notebook.
*
*
* The object available at load time depend on the version of IPython in use.
* there is no guaranties of API stability.
*
* The example below explain the principle, and might not be valid.
*
* Instances are created after the loading of this file and might need to be accessed using events:
* define([
* 'base/js/namespace',
* 'base/js/events'
* ], function(IPython, events) {
* events.on("app_initialized.NotebookApp", function () {
* IPython.keyboard_manager....
* });
* });
*
* __Example 1:__
*
* Create a custom button in toolbar that execute `%qtconsole` in kernel
* and hence open a qtconsole attached to the same kernel as the current notebook
*
* define([
* 'base/js/namespace',
* 'base/js/events'
* ], function(IPython, events) {
* events.on('app_initialized.NotebookApp', function(){
* IPython.toolbar.add_buttons_group([
* {
* 'label' : 'run qtconsole',
* 'icon' : 'icon-terminal', // select your icon from http://fortawesome.github.io/Font-Awesome/icons
* 'callback': function () {
* IPython.notebook.kernel.execute('%qtconsole')
* }
* }
* // add more button here if needed.
* ]);
* });
* });
*
* __Example 2:__
*
* At the completion of the dashboard loading, load an unofficial javascript extension
* that is installed in profile/static/custom/
*
* define([
* 'base/js/events'
* ], function(events) {
* events.on('app_initialized.DashboardApp', function(){
* require(['custom/unofficial_extension.js'])
* });
* });
*
* __Example 3:__
*
* Use `jQuery.getScript(url [, success(script, textStatus, jqXHR)] );`
* to load custom script into the notebook.
*
* // to load the metadata ui extension example.
* $.getScript('/static/notebook/js/celltoolbarpresets/example.js');
* // or
* // to load the metadata ui extension to control slideshow mode / reveal js for nbconvert
* $.getScript('/static/notebook/js/celltoolbarpresets/slideshow.js');
*
*
* @module IPython
* @namespace IPython
* @class customjs
* @static
*/
$([IPython.events]).on('notebook_loaded.Notebook', function(){
var kernelname = IPython.notebook.kernel_selector.current_selection;
var display_text="<div class='output_subarea output_text output_result'>\
<pre>Follow these steps to customize your notebook with ERP5 kernel :-</br>\
1. Add <b>%erp5_url &lt;your_erp5_url&gt;/erp5/Base_executeJupyter</b>.</br>\
Make sure you have 'erp5_data_notebook' business template installed in your erp5</br>\
2. <b>%notebook_set_reference &lt;your_notebook_refernce&gt;</b></br>\
It would be better to set the reference to match with erp5 reference pattern.</br>\
3. <b>%erp5_user &lt;your_erp5_username&gt;</b></br>\
4. <b>%erp5_password &lt;your_erp5_password&gt;</b></br>\
5. As soon as you see 'Please Proceed' message you can now access your erp5 using notebook.</br>\
<p><u>OTHER USEFUL MAGICS</u> -</br>\
<b>%my_notebooks</b> -This is used to display all the notebooks created by the specific user.</br>\
NOTE: Do not dynamically alter imported module objects as they are not being saved in DB, </br>\
so changes to them would be disregarded and would throw an error.</br>\
</pre></div>";
if (kernelname=="erp5"){
$('div#notebook-container').prepend(display_text);
}
});
\ No newline at end of file
software/ipython_notebook/template/kernel.json.jinja 0 → 100644
View file @ 8206dcdd
{
"argv": [
"{{python_executable}}",
"{{kernel_dir}}",
"-f",
"{connection_file}"
],
"display_name": "{{display_name}}",
"language": "{{language_name}}",
"language_info": {"name": "python"}
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7