See merge request nexedi/erp5!2003

In 1b555dbf (py2/py3: Make Products code compatible with both python2
and python3 (nexedi/erp5!1751)., 2024-07-16) we changed the code in a
way that on python2 it would first node text to a python2 str and only
later convert it to unicode, but setting text to a str during the first
step can fail with:

    ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters

This change to set directly unicode on py2.

`sender@customer.com <sender@customer.com>` used in the test is not a
valid email address. We have updated to python3.9.20 which comes with a
fix for CVE-2023-27043 and no longer allow this kind of broken addresses.

Replace the address with a similar valid address,
`"sender@customer.com" <sender@customer.com>`, that was probably the
original intention of this test.

On python3, the type of selected columns depend on the data type from
mariadb side, VARCHAR will be str, BINARY/BLOB will be bytes, etc

These SQL method select path that is first evaluated from a variable
that is NULL and in that case, mariadb seems to select LONGBLOB as data
type:

    MariaDB [test]> set @defined_as_null=null; drop table if exists tmp; create table tmp as (select @defined_as_null); show create table tmp;
    +-------+------------------------------------------------------------------------------------------------------------------------------------+
    | Table | Create Table                                                                                                                       |
    +-------+------------------------------------------------------------------------------------------------------------------------------------+
    | tmp   | CREATE TABLE `tmp` (
      `@defined_as_null` longblob DEFAULT NULL
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci |
    +-------+------------------------------------------------------------------------------------------------------------------------------------+

By casting to CHAR in SQL, on the python side we always have the str
that we expect here, because this is used as path attribute of a SQL
brain.

This is similar to 18deb716 (ProcessingNodeTestCase: also setRequest
in processing_node, 2023-05-02), as said in that commit message,
timerserver loop calls setRequest, but before entering the loop, we
wait for the portal to be created with:

 447   │           try:
 448   │             self.portal = self.app[self.app.test_portal_name]
 449   │           except (AttributeError, KeyError):
 450   │             continue

While accessing like this, this will load classes and initialize dynamic
modules, on python2 this was OK, but on python3 this was raising an
error because `getRequest` returned None:

  File "./parts/erp5/product/ERP5Type/dynamic/component_package.py", line 449, in load_module
    return self.__load_module(fullname)
  File "./parts/erp5/product/ERP5Type/dynamic/component_package.py", line 416, in __load_module
    erp5.component.ref_manager.add_module(module)
  File "./parts/erp5/product/ERP5Type/dynamic/dynamic_module.py", line 86, in add_module
    self.add_request(get_request())
  File "./parts/erp5/product/ERP5Type/dynamic/dynamic_module.py", line 64, in add_request
    self.setdefault(last_sync, (WeakSet(), set()))[0].add(request_obj)
  File "./lib/python3.9/_weakrefset.py", line 89, in add
    self.data.add(ref(item, self._remove))
TypeError: cannot create weak reference to 'NoneType' object

On python2, this was actually raising as well, but this error is hidden
by a `hasattr`, because on python2 `hasattr` ignores all exceptions and
on python3 it only ignores only `AttributeError`.

  File "./parts/erp5/product/ERP5Type/Core/PropertySheet.py", line 61, in createAccessorHolder
    self.applyOnAccessorHolder(accessor_holder, expression_context, portal)
  File "./parts/erp5/product/ERP5Type/Core/PropertySheet.py", line 175, in applyOnAccessorHolder
    for property_definition in self.contentValues():
  File "./parts/erp5/product/ERP5Type/Core/Folder.py", line 1570, in contentValues
    portal_type_id_list = self._getTypesTool().listContentTypes()
  File "./parts/erp5/product/ERP5Type/Tool/TypesTool.py", line 173, in listContentTypes
    provider_value = _getOb(provider, None)
  File "./eggs/Zope-4.8.7-py2.7.egg/OFS/ObjectManager.py", line 323, in _getOb
    if id[:1] != '_' and hasattr(aq_base(self), id):
  File "./parts/erp5/product/ERP5Type/dynamic/lazy_class.py", line 120, in __getattribute__
    self.__class__.loadClass()

And simplify just a bit the code

FutureWarning: This search incorrectly ignores the root element, and will be
fixed in a future version.  If you rely on the current behaviour, change it to
'.//role'.

py3: TestTradeModelLineMixin inherited from UserDict() to store values on the class direcly (!1751).

This does not work with py3:
    File "parts/erp5/Products/ERP5Type/tests/runUnitTest.py", line 941, in main
      result = runUnitTestList(test_list=args,
    File "parts/erp5/Products/ERP5Type/tests/runUnitTest.py", line 703, in runUnitTestList
      result = TestRunner(verbosity=verbosity).run(suite)
    [...]
    File "parts/python3/lib/python3.9/unittest/runner.py", line 184, in run
      test(result)
    [...]
    File "parts/python3/lib/python3.9/unittest/suite.py", line 84, in __call__
      return self.run(*args, **kwds)
    File "parts/python3/lib/python3.9/unittest/suite.py", line 111, in run
      if _isnotsuite(test):
    File "parts/python3/lib/python3.9/unittest/suite.py", line 369, in _isnotsuite
      iter(test)
    File "parts/python3/lib/python3.9/collections/__init__.py", line 1067, in __iter__
      return iter(self.data)
  AttributeError: 'TestComplexTradeModelLineUseCaseSale' object has no attribute 'data'

Use a dedicated dict() to store these values as there was no strong reasons to
inherit from UserDict() here and this makes test implementation easier too...

group_id is used as key of OOBtree and as documented, it's not possible to mix
keys that can not be compared, so we can not have a mix of string and bytes, for
consistency with other BTrees, such as the ones used for OFS.

group_id is also used in a SQL column which is BINARY, this is problematic on
py3 because the selected values will be returned as bytes, but we expect str
here. Because we don't want to run a data migration, we adjust the select
methods to convert to str while selecting.

Since years there was a warning that id_group must be a string, now we make it a
bit stricter, we also enforce that the id_group is valid UTF-8.

A few more tests and assertions were also added.

Since Zope commit 020685087 (`Allow ZPublisher to handle a query string together
with a request body (#1124)`, 2023-05-15) Zope tries to process all XML HTTP
requests as XML-RPC and we need to tell that these SOAP requests are not XML-RPC.

py2/py3: Base64 encode inventory cache, as Shared.DC.ZRDB.DA.SQL tries to decode bytes to str (!1751).

Co-authored-by: Arnaud Fontaine <arnaud.fontaine@nexedi.com>

Followup of 94739085.

key()-based sorting is now used instead, available since Python 2.4.

Jupyter integration is more or less abandoned. Just do the minimum to make tests pass.

Also, remove `if True/1/False/0` statement raising pylint warnings.
Co-authored-by: Arnaud Fontaine <arnaud.fontaine@nexedi.com>

* Zope API changes:
  + publish():
    - stdin is now BytesIO rather than StringIO.
    - Returned value of a script is passed to str() in python2, not in python3 anymore.
  + HTTPResponse `body` property is now bytes().
  + OFS.Image.File file parameter is bytes().
  + zope.interface implements() is now @implementer decorator.
  + Python standard logging module recommended instead of zLOG.
* Python3 API changes:
  + builtin reduce() was removed.
  + urlnorm is now available.
  + Use BytesIO rather than StringIO to follow py3 API.
  + hmac.new() requires digestmod argument from Python 3.8.
  + Use six.moves library to handle moved objects from py2 to py3.
  + `modernize -f xrange_six` then slightly adjusted manually to just use range
     where it does not make a significant difference (for example in test).
  + base64.b64encode() now expects bytes().
  + UserDict() interface changed:
    - New parameter in update() and pop().
    - `failobj` setdefault parameter renamed to `default`.
  + ensure_list() on dict.{values,items}() and list(dict) for dict.keys() when
    we really need a list and not an iterable (Python3).
  + Make dict iteration works on both version of Python.
    - Use six.iter{items,values,keys}().
    - has_key() has been removed.
    - Make sure that dict.{items,values,keys}() returns a real list when
    modified (ensure_list()).
  + Comparisons between int and NoneType raises TypeError.
    + BTrees key must be str() not int() (_getOb()).
  + No more unbound methods in python3 so use six.get_unbound_function().
  + Exceptions:
    - No longer behave as sequences. Using `args` attribute instead.
    - When an exception has been assigned using `as target`, it is cleared at
      the end of the except clause.
  + file: py2 was returning `str` upon reading, now it returns text strings.
    Also, opening mode is text strings by default.
  + Data strings are bytes().
    - Replace str() by bytes().
  + iterators no longer have next() method, instead there is next() builtin.
  + New ConnectionError exception so rename existing one to not clash.
  + Integer division is now with //.
  + __nonzero__ is now __bool__.
  + apply() does not exist anymore.
  + Deprecated threading.Thread isAlive() has been removed.
  + im_func replaced by __func__.
  + Use six.with_metaclass() to define metaclass in a cross-compatible way with py2 and py3.
  + Only test method can be marked as expectedFailure(), not assert statement anymore.
  + os.path.walk() removed.
  + HTMLParser never fails: no strict mode nor HTMLParseError anymore (Python #15114).
  + Unpickler.find_global() is now Unpickler.find_class().
Co-Authored-by: Kazuhiko SHIOZAKI <kazuhiko@nexedi.com>
Co-Authored-by: Arnaud Fontaine <arnaud.fontaine@nexedi.com>
Co-Authored-by: Carlos Ramos Carreño <carlos.ramos@nexedi.com>
Co-Authored-by: Emmeline Vouriot <emmeline.vouriot@nexedi.com>

Some conversion (TextDocument) depend on the content type, so it's necessary to
first update the content type before trying to convert to base format.

Transform: Show better traceback on debugging failure of transform module import (nexedi/erp5!1751).

zope5: Since 5.8.1 missing Content-Type HTTP header is interpreted as application/x-www-form-urlencoded (nexedi/erp5!1751).

Zope.git:
  commit 5b324f6c461f5ea1cc069739b6c32a1a5ff59df9
  Date:   Thu Jan 19 07:15:18 2023 +0100
    replace `cgi.FieldStorage` by `multipart` (#1094)
    * interpret a missing `CONTENT_TYPE` as `application/x-www-form-urlencoded`

With cgi.FieldStorage, it was interpretated as text/plain so Content-Type has to
be provided now (even for WebDAV despite RFC 4918 stating that it *SHOULD* be
provided).

Some Unit Tests did not provide such header at all but this was wrong (such as
erp5_stripe:testStripe where the real request has application/json as
Content-Type and not text/plain).

See merge request nexedi/erp5!1988

to test the translation of the editor's "Maximize" button

Notably showned as a companion of the editor gadget