erp5_data_notebook: environment object implementation and refactoring to ERP5 kernel

@Tyagov review, please. I'm creating a test suite now and will post about the test results as soon as they are available. 

- An environment object was implemented to help us deal with the multiprocess
architecture of ERP5 and objects that cannot be easily stored in the ZODB.
It stores definition of functions, classes and variables as string. The implementation
uses a dumb Environment class to allow users to make `define` and `undefine` calls,
which are captured and processed by an AST transformer before code execution.

- Along with the environment object, an automatic "import fixer" was created. It does
not allow users to import modules as they normally would, because this may cause
collateral effects on other users' code. A good example is the plot settings in the
matplotlib module. It will fix normal imports, make them use the environment object
mentione earlier automatically and warn the user about it.

Some bugs were fixed with this new implementation: 

- https://nexedi.erp5.net/bug_module/20160318-7098DD, which reports an inconsistency
on portal catalog queries between Jupyter and Python (Script) objects. Probably an
issue with user context storage in ActiveProcess

- https://nexedi.erp5.net/bug_module/20160330-13AA193, which reports an error related
to acquisition when trying to plot images, which happened in other situations, although
this is not officially reported in Nexedi's ERP5. This probably also was happening because
of old user context storage.


/reviewed-on !131

bt5/erp5_data_notebook/PropertySheetTemplateItem/portal_property_sheets/JupyterNotebook/local_varibale_property.xml → bt5/erp5_data_notebook/PropertySheetTemplateItem/portal_property_sheets/JupyterNotebook/notebook_context_property.xml

@@ -10,7 +10,7 @@
             <key> <string>categories</string> </key>
             <value>
               <tuple>
-                <string>elementary_type/string</string>
+                <string>elementary_type/object</string>
               </tuple>
             </value>
         </item>
@@ -22,7 +22,7 @@
         </item>
         <item>
             <key> <string>id</string> </key>
-            <value> <string>local_varibale_property</string> </value>
+            <value> <string>notebook_context_property</string> </value>
         </item>
         <item>
             <key> <string>portal_type</string> </key>

bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_addLocalVariableDict.xml → bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_createNotebookContext.xml

@@ -8,7 +8,7 @@
       <dictionary>
         <item>
             <key> <string>_function</string> </key>
-            <value> <string>AddNewLocalVariableDict</string> </value>
+            <value> <string>createNotebookContext</string> </value>
         </item>
         <item>
             <key> <string>_module</string> </key>
@@ -16,7 +16,7 @@
         </item>
         <item>
             <key> <string>id</string> </key>
-            <value> <string>Base_addLocalVariableDict</string> </value>
+            <value> <string>Base_createNotebookContext</string> </value>
         </item>
         <item>
             <key> <string>title</string> </key>

bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_executeJupyter.py

-"""
-Python script to create Data Notebook or update existing Data Notebooks
-identifying notebook by reference from user.
-
-Expected behaviour from this script:-
-1. Return unauthorized message for non-developer user.
-2. Create new 'Data Notebook' for new reference.
-3. Add new 'Data Notebook Line'to the existing Data Notebook on basis of reference.
-4. Return python dictionary containing list of all notebooks for 'request_reference=True'
-"""
-
-portal = context.getPortalObject()
-# Check permissions for current user and display message to non-authorized user 
-if not portal.Base_checkPermission('portal_components', 'Manage Portal'):
-  return "You are not authorized to access the script"
-
-import json
-
-# Convert the request_reference argument string to their respeced boolean values
-request_reference = {'True': True, 'False': False}.get(request_reference, False)
-
-# Return python dictionary with title and reference of all notebooks
-# for request_reference=True
-if request_reference:
-  data_notebook_list = portal.portal_catalog(portal_type='Data Notebook')
-  notebook_detail_list = [{'reference': obj.getReference(), 'title': obj.getTitle()} for obj in data_notebook_list]
-  return notebook_detail_list
-
-if not reference:
-  message = "Please set or use reference for the notebook you want to use"
-  return message
-
-# Take python_expression as '' for empty code from jupyter frontend
-if not python_expression:
-  python_expression = ''
-
-# Get Data Notebook with the specific reference
-data_notebook = portal.portal_catalog.getResultValue(portal_type='Data Notebook',
-                      reference=reference)
-
-# Create new Data Notebook if reference doesn't match with any from existing ones
-if not data_notebook:
-  notebook_module = portal.getDefaultModule(portal_type='Data Notebook')
-  data_notebook = notebook_module.DataNotebookModule_addDataNotebook(
-    title=title,
-    reference=reference,
-    batch_mode=True
-  )
-
-# Add new Data Notebook Line to the Data Notebook
-data_notebook_line = data_notebook.DataNotebook_addDataNotebookLine(
-  notebook_code=python_expression,
-  batch_mode=True
-)
-
-# Get active_process associated with data_notebook object
-process_id = data_notebook.getProcess()
-active_process = portal.portal_activities[process_id]
-# Add a result object to Active Process object
-result_list = active_process.getResultList()
-
-# Get local variables saves in Active Result, local varibales are saved as
-# persistent mapping object
-old_local_variable_dict = result_list[0].summary
-if not old_local_variable_dict:
-  old_local_variable_dict = context.Base_addLocalVariableDict()
-
-# Pass all to code Base_runJupyter external function which would execute the code
-# and returns a dict of result
-final_result = context.Base_runJupyter(python_expression, old_local_variable_dict)
-code_result = final_result['result_string']
-new_local_variable_dict = final_result['local_variable_dict']
-ename = final_result['ename']
-evalue = final_result['evalue']
-traceback = final_result['traceback']
-status = final_result['status']
-mime_type = final_result['mime_type']
-
-# Call to function to update persistent mapping object with new local variables
-# and save the variables in the Active Result pertaining to the current Data Notebook
-new_dict = context.Base_updateLocalVariableDict(new_local_variable_dict)
-result_list[0].edit(summary=new_dict)
-
-result = {
-  u'code_result': code_result,
-  u'ename': ename,
-  u'evalue': evalue,
-  u'traceback': traceback,
-  u'status': status,
-  u'mime_type': mime_type
-}
-
-# Catch exception while seriaizing the result to be passed to jupyter frontend
-# and in case of error put code_result as None and status as 'error' which would
-# be shown by Jupyter frontend
-try:
-  serialized_result = json.dumps(result)
-except UnicodeDecodeError:
-  result = {
-    u'code_result': None,
-    u'ename': u'UnicodeDecodeError',
-    u'evalue': None,
-    u'traceback': None,
-    u'status': u'error',
-    u'mime_type': mime_type
-  }
-  serialized_result = json.dumps(result)
-
-data_notebook_line.edit(notebook_code_result=code_result, mime_type=mime_type)
-
-return serialized_result

bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_executeJupyter.xml

@@ -2,71 +2,26 @@
 <ZopeData>
   <record id="1" aka="AAAAAAAAAAE=">
     <pickle>
-      <global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
+      <global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
     </pickle>
     <pickle>
       <dictionary>
         <item>
-            <key> <string>Script_magic</string> </key>
-            <value> <int>3</int> </value>
-        </item>
-        <item>
-            <key> <string>_Access_contents_information_Permission</string> </key>
-            <value>
-              <tuple>
-                <string>Authenticated</string>
-                <string>Author</string>
-                <string>Manager</string>
-                <string>Owner</string>
-              </tuple>
-            </value>
-        </item>
-        <item>
-            <key> <string>_bind_names</string> </key>
-            <value>
-              <object>
-                <klass>
-                  <global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
-                </klass>
-                <tuple/>
-                <state>
-                  <dictionary>
-                    <item>
-                        <key> <string>_asgns</string> </key>
-                        <value>
-                          <dictionary>
-                            <item>
-                                <key> <string>name_container</string> </key>
-                                <value> <string>container</string> </value>
-                            </item>
-                            <item>
-                                <key> <string>name_context</string> </key>
-                                <value> <string>context</string> </value>
-                            </item>
-                            <item>
-                                <key> <string>name_m_self</string> </key>
-                                <value> <string>script</string> </value>
-                            </item>
-                            <item>
-                                <key> <string>name_subpath</string> </key>
-                                <value> <string>traverse_subpath</string> </value>
-                            </item>
-                          </dictionary>
-                        </value>
-                    </item>
-                  </dictionary>
-                </state>
-              </object>
-            </value>
+            <key> <string>_function</string> </key>
+            <value> <string>Base_executeJupyter</string> </value>
         </item>
         <item>
-            <key> <string>_params</string> </key>
-            <value> <string>python_expression=None, reference=None, title=None, request_reference=False, **kw</string> </value>
+            <key> <string>_module</string> </key>
+            <value> <string>JupyterCompile</string> </value>
         </item>
         <item>
             <key> <string>id</string> </key>
             <value> <string>Base_executeJupyter</string> </value>
         </item>
+        <item>
+            <key> <string>title</string> </key>
+            <value> <string></string> </value>
+        </item>
       </dictionary>
     </pickle>
   </record>

bt5/erp5_data_notebook/PropertySheetTemplateItem/portal_property_sheets/JupyterNotebook/process_property.xml → bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_getErrorMessageForException.xml

@@ -2,31 +2,25 @@
 <ZopeData>
   <record id="1" aka="AAAAAAAAAAE=">
     <pickle>
-      <global name="Standard Property" module="erp5.portal_type"/>
+      <global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
     </pickle>
     <pickle>
       <dictionary>
         <item>
-            <key> <string>categories</string> </key>
-            <value>
-              <tuple>
-                <string>elementary_type/string</string>
-              </tuple>
-            </value>
+            <key> <string>_function</string> </key>
+            <value> <string>getErrorMessageForException</string> </value>
         </item>
         <item>
-            <key> <string>description</string> </key>
-            <value>
-              <none/>
-            </value>
+            <key> <string>_module</string> </key>
+            <value> <string>JupyterCompile</string> </value>
         </item>
         <item>
             <key> <string>id</string> </key>
-            <value> <string>process_property</string> </value>
+            <value> <string>Base_getErrorMessageForException</string> </value>
         </item>
         <item>
-            <key> <string>portal_type</string> </key>
-            <value> <string>Standard Property</string> </value>
+            <key> <string>title</string> </key>
+            <value> <string></string> </value>
         </item>
       </dictionary>
     </pickle>

bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_runJupyter.xml

@@ -8,7 +8,7 @@
       <dictionary>
         <item>
             <key> <string>_function</string> </key>
-            <value> <string>Base_compileJupyterCode</string> </value>
+            <value> <string>Base_runJupyterCode</string> </value>
         </item>
         <item>
             <key> <string>_module</string> </key>

bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/DataNotebookModule_addDataNotebook.py

 """
 Python script to add a new notebook to Data Notebook module.
-This script also concerns for assigning an Active Process for each data notebook
-created.
+This script also concerns for assigning an empty notebook context for each data
+notebook created.
 """
-from Products.CMFActivity.ActiveResult import ActiveResult
-
-# Comment out person in case addition of person required to Data Notebook object
-#person = context.ERP5Site_getAuthenticatedMemberPersonValue()
-
-# Create new ActiveProcess object and getting its id
-active_process = context.portal_activities.newActiveProcess()
-active_process_id = active_process.getId()
-
-# Creating new dictionary via external method to save results in ZODB
-new_dict = context.Base_addLocalVariableDict()
-# Add new ActiveResult object and add it to the activeprocess concerned with ...
-# Data Notebook in concern
-result = ActiveResult(summary=new_dict)
-active_process.activateResult(result)
+# Creating new context via external method to save results in ZODB
+notebook_context = context.Base_createNotebookContext()
 
 # Create new notebook
 notebook = context.newContent(
     title=title,
     reference=reference,
-    process=active_process_id,
+    notebook_context=notebook_context,
     portal_type='Data Notebook'
   )