https://codemirror.net/demo/matchtags.html

makes editions of web pages and xml easier

This first reverts the following 2 commits:
  566c0c5f
  3a08c758

to at least fix the issue that the context method takes 2 arguments that aren't
available from FormBoxEditor.edit() without slowing things even more.

About deferred style, widget editor objects can be ignored for the moment. If
we ever need to restore fully functional objects, the regression in commit
  a5a2f1cd
could be solved with the following patch:

--- a/product/ERP5Form/FormBox.py
+++ b/product/ERP5Form/FormBox.py
@@ -131,22 +131,32 @@ def render(self, field, key, value, REQUEST, render_prefix=None):
 class FormBoxEditor:
   """An editor returned from FormBox validation able to `edit` document."""
 
+  path = None
+
   def __init__(self, result, context):
     """Initialize with all necessary information for editing.
 
     Keep a reference to the correct context and don't expect the caller to provide it
     during the edit phase because they don't have access to the widget anymore.
     """
     self.attr_dict, self.editor_list = result
+    self.context = context
     self.edit = lambda _: self._edit(context)
 
   def __getstate__(self):
-    pass
+    return (self.attr_dict, self.editor_list,
+            self.path or self.context.getPhysicalPath())
+
+  def __setstate__(self, state):
+    self.attr_dict, self.editor_list, self.path = state
 
   def __call__(self, REQUEST):
     # Called by Base_edit in case of FormValidationError
     pass
 
+  def edit(self, context):
+    return self._edit(context.unrestrictedTraverse(self.path))
+
   def _edit(self, context):
     """Edit inside correct context."""
     context.edit(**self.attr_dict)

This commit also removes the unused view() method on editors.

/reviewed-on nexedi/erp5!622

/reviewed-on nexedi/erp5!626

/reviewed-on nexedi/erp5!625

erp5_l10n_fa is needed to test the RTL direction.

same as b6eec549

Since 76cc938e we have diffs each time we edit ZSQL Methods. This is a "big commit" to change all remaining ZSQL methods. I checked diffs one by one, but I would appreciate others check this as well and confirm that it's OK to do so.

From the root of repository, I did :

```
grep -rl Products.ZSQLMethods.SQL  . | grep '\.xml$' | xargs ~/bin/zsql_backslash_n.py
```

With a script containing
```python

from lxml import etree
import sys

for filename in sys.argv[1:]:
  with open(filename) as f:
    tree = etree.parse(f)
    root = tree.getroot()
    for el in tree.xpath('//item/key/string[text() = "arguments_src"]/../../value/string'):
      if el.text:
        el.text = el.text.replace(r'\r\n', r'\n')

    # force <string> element to have a text, so that they export as <string></string> and not <string/>
    for el in tree.xpath('//string[not(text())]'):
      el.text = ''

  with open(filename, 'w') as f:
    f.write(
        '<?xml version="1.0"?>\n'
        + etree.tostring(root, pretty_print=True, encoding="utf-8"))

  print filename
```
( script snippet: nexedi/erp5$277 )

/cc @jm @georgios.dagkakis @Nicolas

/reviewed-on nexedi/erp5!496

Conflicts:
	product/ERP5/bootstrap/erp5_mysql_innodb_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/z_getitem_by_path.xml
	product/ERP5/bootstrap/erp5_mysql_innodb_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/z_getitem_by_uid.xml
	product/ERP5/bootstrap/erp5_mysql_innodb_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/z_produce_reserved_uid_list.xml

When document has pending activities, we refuse changing ID ( because there might be pending `updateRelatedContent` activities if I remember correctly ), but it's done in a way that breaks the "atomic" aspect of the transaction a bit, because we

As a result, this happens sometimes that not all properties user changed are modified. In the example below, the change to *Include Documents in Site Map* is not saved (and also change to *ID*):

![erp5-sorryPendingActivitiesSavePartially](/uploads/ff4bfd6ad0e8a42ba3684cccdc450e21/erp5-sorryPendingActivitiesSavePartially.gif)

( screencast of editing a document to change ids and several other
properties - after clicking save, we can see that changing id is refused
because there is pending activities. Other properties that where changes
at the same times are not all modified, which breaks the
transactionality we can usally expect when editing documents in ERP5 )

The changed here is to use a field validator that refused editing when there are pending activities, so that user gets a:

![erp5-pending-activiities](/uploads/bfe825560bdee34f0443e8e36884f21c/erp5-pending-activiities.png)

( screenshot of the change: now edition is rejected )

and the result is either all changes are applied or no change is applied at all.

This is done by:
 * introducing a new `my_view_mode_id` field in `erp5_core`'s `Base_viewFieldLibrary`
 * using this field as proxy field of all editable `my_id` fields. Maybe I forgot some business templates, I changed only the most common ones. I intentionally did not change all fields of `erp5_ui_test` because I think they are used to compare speed of proxy fields vs traditional fields.

0352f50fd543fda2712bb8ca93d8a8814f975a26 introduces a Zelenium test exercising this new behavior.

/reviewed-on nexedi/erp5!504

Conflicts:
	product/ERP5/bootstrap/erp5_core/SkinTemplateItem/portal_skins/erp5_core/Base_viewFieldLibrary.xml

This is a fix for this issue:

Create a payment transaction group, a source reference will be generated (something like `PTG0123`. The field is read only on the form, so it's clearly not something we expect user to edit, but a kind of unique generated reference.

Clone that payment transaction group, on the cloned payment transaction group, source reference is empty.

The fix here is to generate a new source reference.

A side effect of these changes is that destination reference is no longer reset after clone, because before this `Delivery_afterClone` was used and this script reset both source and destination reference.

/reviewed-on nexedi/erp5!535

Numbering tests can be used to control test execution order, but developers should write independent tests that should not rely on test execution order.

`getTitle` is useless. Test runner will print the class name.

/reviewed-on nexedi/erp5!534

The fact that ERP5 does not remember the selected language has been reported as a bug to us.

Instead of using a session cookie, this change make the cookie persists. For one year. This is a bit arbitrary, I copied that behavior from the "remember my username" [cookie](https://lab.nexedi.com/nexedi/erp5/blob/557c20bdf0e/product/ERP5Type/patches/CookieCrumbler.py#L138) that we use to have on the login page.

/reviewed-on nexedi/erp5!506

In internal transactions, we can have a payment line where source is using a bank account and destination is using another bank account.

Bank Reconciliation did not support these lines properly, because it simply assumed that *"if there's a bank reconciliation on this line, it means the line is reconciled"*, but this was incorrect, because source and destination needs to be able to reconcile this line independently.

The fixes is to change the rule from *"if there's a bank reconciliation on this line, it means the line is reconciled"* to *"if there's a bank reconciliation with the same bank account as this line's source payment, it means the line is reconciled for source"* and *"if there's a bank reconciliation with the same bank account as this line's destination payment, it means the line is reconciled for destination"*.

The related  key used to select the already reconciled lines or lines to reconcile becomes a bit more complex, it joins with category table to ensure the bank reconciliation's bank account  (which is `source_payment` on the bank reconciliation) is the same as the `section_uid` of the line in stock table that we are considering. This also has a side effect that this related key only work with queries on stock table, but it was never intended to be used elsewhere.

Another problem is that queries made by bank reconciliation module become a bit more slower. For sites where this is a problem, the recommended approach is to replace the related keys by columns on stock table. In cfec8d2b (which is part of this MR) we add indexation methods. In 5afb002090cd965bd17ffcf5c4c54ba73dfcbfc4 (which is not part of this MR and not planned for inclusion in "default" ERP5) we show an example of how to enable these methods to actually add the columns in stock table.

/reviewed-on nexedi/erp5!495

We realized that the references of notification messages used in credential request management had a typo (`crendential` vs `credential`) and that the message used for a new credential request was not using the same `credential_request` prefix as others.

This MR changes the messages as follow:

| Wrong Reference | Correct Reference |
| --- | --- |
| erp5-subscription.notification | credential_request-subscription |
| crendential_request-confirmation-without-password | credential_request-confirmation-without-password |
| crendential_request-confirmation-with-password | credential_request-confirmation-with-password |
| crendential_request-confirmation-without-password | credential_request-confirmation-without-password |
| crendential_recovery-reset-link | credential_recover-reset-link |
| crendential_recovery-username | credential_recovery-username |

This is an incompatible change that can affect projects that have defined some custom notification messages without explicitly setting the references on the system preferences (ie. getting the default value from the property definition).
I found one project using custom notification messages, but preference was defined.
Other projects I checked did not override these notification messages.

In our projects, we mostly use messages for credential requests, but the reference for the notification message is usually defined as a property of the web section.

So I'm confident this should not affect projects and we can proceed with this clean up without causing too much troubles. To check if you need to adjust notification messages, check if you have customized notification message with reference *Wrong Reference*.

/reviewed-on nexedi/erp5!492

The background:

Following up nexedi/erp5!393 , we updated security configuration used in Nexedi ERP5 , we wanted to add a security rule so that users uploading a `Personal/Private` document are allowed to "share" that document (technically, it's sharing with themselves only because it's a private document, but it's useful to distinguish from drafts).

Then we discovered that in the current workflow configuration, only `Assignor` role is allowed to *share*, but `Assignor` is also allowed to *publish*, but we did not want users to publish their personal documents..

---
Original commit message:

Previously, only Assignor was able to publish, share and release, this
make it impossible to have security configuration where some user can
only share and not publish documents.

To address this issue in the more backward compatible way possible, we
enable these transitions for Associate and keep them enabled for
Assignor role.

/reviewed-on nexedi/erp5!497

My use case is that we have an ERP5 configuration where a PDF document is "implictly" created when user validate an invoice. Later this PDF becomes "secret" and we want to remove permissions on the PDF to all except a small  group of users.

Please also read commit message for more uses cases.

My idea is to change globally document publication workflow to remove permissions for Owner, because usually in workflow we don't have security for Owner, except in draft states.
For cases where the user who created the document must have certain permissions for the whole lifetime of the document, we can create a security rule where this user would be Associate.
Also, for the case of documents, maybe we would want to use *Contributors* fields instead of Owner, as it gives more flexibility.

In what I am suggesting, the permissions by state would change from:

![Screenshot_2017-09-15_at_16.12.53](/uploads/5b3664a2663deb893ea4f8fc9858a52f/Screenshot_2017-09-15_at_16.12.53.png)

to:

![Screenshot_2017-09-15_at_17.34.34](/uploads/95803dc89e1a2501c29872a6a5131c33/Screenshot_2017-09-15_at_17.34.34.png)

The full updated `document_publication_workflow` specification would be:

[P-ERP5.Workflow.Security.After.Removing.Owner.pdf](/uploads/02eac46ec436385d0d0577695803b3b5/P-ERP5.Workflow.Security.After.Removing.Owner.pdf)

But this is an incompatible change, because some users will loose access to some documents they use to have access.

/reviewed-on nexedi/erp5!393

/reviewed-on nexedi/erp5!621

Translate worklist and history page

/reviewed-on nexedi/erp5!620

Use reindexObject which can group activities.

Reduce duplicated code.

Use the hardcoded https://validator.erp5.net service
This service is installed with: https://lab.nexedi.com/nexedi/slapos/tree/master/software/htmlvalidatorserver

Add "Post Modules", a serie of modules whose purpose is to track exchanges in/out of ERP5.

Currently, users write Mail Messages (usually as a HTML content), and then send the Mail Message. The RFC5322 version of the email (what is given to MailHost and sent over the network) is generated on the fly and never stored anywhere. About Letter, no output is generated. It means that exchanges are not tracked, and if an Event is sent 2 times, we don't know their exact content on both times.

Posts have the purpose to fix this issue, by creating objects storing the content of the sent Events **exactly as they are transmitted out of ERP5**. In exemple, sent Mail Messages will be converted to a  multi-part message, stored as a Internet Message Post, which content will be transmitted as such to MailHost. For Letters, their content can be processed (by erp5_corporate_identity for exemple), then converted to PDF, and stored as Letter Post. We can then imagine actions/alarms to gather all exportable Letters Posts and mail them to a printer.

Posts follow their own workflow, shared in between all post types, which allow to track which posts are in the sending queue, which have been sent and which have been received. It must be easy to know when an event has been sent several time, and retrieve the content of each of these sendings. Design of the post modules/workflow should also allow an easy customization (for exemple, we may want to send all emails by bulk, instead of  1 by 1, in order to set up mass-emailing policies).

/reviewed-on nexedi/erp5!532

This mixin contains all the methods returning values based on the mail message
itself contained in the data property of the EmailDocument.

erp5_crm: display standard preview on Mail Message only if internet_message_post_module isn't installed

The attachment field now uses the property aggregate_document_title of Event.

It will be used to differenciate attachments from Mime Post.