because Domain.getRelativeUrl (and Category.getRelativeUrl) cannot be
restrictedTraverse'd

some work in progress changes to improve developer experience

monaco_editor: increase debounce timeout for pylint checks XXX

on very large python files (>1000 lines) sometimes they queue up and we
have to wait for all requests that were queued by zope.

XXX maybe this does not happen when accessing through haproxy/apache, I
am observing this when hitting zope directly

jedi: generate stubs WIP

ERP5: "quick and dirty" type annotations XXX

IIRC the only thing needed is that ERP5TypeTestCase.getPortal is an
ERP5Site

monaco_editor: also enable jedi for codelens ( WIP: ZMI only )

core: pass "language support url" to text editors XXX

For now this is just the portal_url, but I'm thinking it could be a
proper tool.

monaco_editor: enable pylint in gadget version

because pylint is a bit slow on large components, debounce every 2
seconds. TODO: this is too slow.

monaco_editor: enable formatting provider for python

This makes "Format Document" / "Format Selection" work.

monaco_editor: enable completion provider for python

this makes completions works when using Ctrl+space

monaco_editor: pass portal_type to checkPythonSourceCode

python_support: new business template to act as a language server for python

checkPythonSourceCode: add a cache

When using checkPythonSourceCode integrated in the source code editor,
for a scenario where developer edit a component and save we can benefit
from caching the check message for the source code content, because the
same check that the one happening in the editor will happen when the
component is saved.

This cache varies on:
 - "component_packages" cache cookie which is reset every time some
component code is edited.
 - zope startup time to take into account editions of file system code.
This assumes that after reseting file system code zope will be
restarted.
 - portal_type, because the checks performed by this function also
depend on portal type.

jedi wip

administration: keep using pylint only for now

monaco_editor: jedi WIP

core: use mypy to check python code ( WIP experiment )

Revert "core: use mypy to check python code ( WIP experiment )"

This reverts commit cfa27232.

ERP5TypeTestCase: jedi workarounds

monaco_editor: WIP reference provider for python

monaco_editor: jedi wip (no longer use /tmp/)

yapf: adjust config following up Gabriel feedback

see discussion on !1422 (comment 134105)

See https://zope.readthedocs.io/en/latest/migrations/zope4/zodb.html

The approach here is to have a ZODB patch to apply zodbupdate on object
load, decoding with UTF-8 encoding because it will be correct most of
the time with ERP5

And workaround safeimage impacted by this round2 patch.

This reverts commit 5e21f77f.

This was done too quickly based on a wrong assumption that simulation
movements to build would always be in planned state and that we could
have an efficient way of selecting them by catalog with index on
portal_type and simulation state, but it does not work this way.

Maybe the change is useful for something else, but since we don't have
any use case for now, let's just revert.

See merge request nexedi/erp5!2001

See merge request !2003

In 1b555dbf (py2/py3: Make Products code compatible with both python2
and python3 (nexedi/erp5!1751)., 2024-07-16) we changed the code in a
way that on python2 it would first node text to a python2 str and only
later convert it to unicode, but setting text to a str during the first
step can fail with:

    ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters

This change to set directly unicode on py2.

and move the implementation of SimulationMovement.getSimulationState
here. This makes it possible for rules to generate simulation movements
that are not necessary planned.

Applied rules must have a rule as specialise

`sender@customer.com <sender@customer.com>` used in the test is not a
valid email address. We have updated to python3.9.20 which comes with a
fix for CVE-2023-27043 and no longer allow this kind of broken addresses.

Replace the address with a similar valid address,
`"sender@customer.com" <sender@customer.com>`, that was probably the
original intention of this test.

On python3, the type of selected columns depend on the data type from
mariadb side, VARCHAR will be str, BINARY/BLOB will be bytes, etc

These SQL method select path that is first evaluated from a variable
that is NULL and in that case, mariadb seems to select LONGBLOB as data
type:

    MariaDB [test]> set @defined_as_null=null; drop table if exists tmp; create table tmp as (select @defined_as_null); show create table tmp;
    +-------+------------------------------------------------------------------------------------------------------------------------------------+
    | Table | Create Table                                                                                                                       |
    +-------+------------------------------------------------------------------------------------------------------------------------------------+
    | tmp   | CREATE TABLE `tmp` (
      `@defined_as_null` longblob DEFAULT NULL
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci |
    +-------+------------------------------------------------------------------------------------------------------------------------------------+

By casting to CHAR in SQL, on the python side we always have the str
that we expect here, because this is used as path attribute of a SQL
brain.

This is similar to 18deb716 (ProcessingNodeTestCase: also setRequest
in processing_node, 2023-05-02), as said in that commit message,
timerserver loop calls setRequest, but before entering the loop, we
wait for the portal to be created with:

 447   │           try:
 448   │             self.portal = self.app[self.app.test_portal_name]
 449   │           except (AttributeError, KeyError):
 450   │             continue

While accessing like this, this will load classes and initialize dynamic
modules, on python2 this was OK, but on python3 this was raising an
error because `getRequest` returned None:

  File "./parts/erp5/product/ERP5Type/dynamic/component_package.py", line 449, in load_module
    return self.__load_module(fullname)
  File "./parts/erp5/product/ERP5Type/dynamic/component_package.py", line 416, in __load_module
    erp5.component.ref_manager.add_module(module)
  File "./parts/erp5/product/ERP5Type/dynamic/dynamic_module.py", line 86, in add_module
    self.add_request(get_request())
  File "./parts/erp5/product/ERP5Type/dynamic/dynamic_module.py", line 64, in add_request
    self.setdefault(last_sync, (WeakSet(), set()))[0].add(request_obj)
  File "./lib/python3.9/_weakrefset.py", line 89, in add
    self.data.add(ref(item, self._remove))
TypeError: cannot create weak reference to 'NoneType' object

On python2, this was actually raising as well, but this error is hidden
by a `hasattr`, because on python2 `hasattr` ignores all exceptions and
on python3 it only ignores only `AttributeError`.

  File "./parts/erp5/product/ERP5Type/Core/PropertySheet.py", line 61, in createAccessorHolder
    self.applyOnAccessorHolder(accessor_holder, expression_context, portal)
  File "./parts/erp5/product/ERP5Type/Core/PropertySheet.py", line 175, in applyOnAccessorHolder
    for property_definition in self.contentValues():
  File "./parts/erp5/product/ERP5Type/Core/Folder.py", line 1570, in contentValues
    portal_type_id_list = self._getTypesTool().listContentTypes()
  File "./parts/erp5/product/ERP5Type/Tool/TypesTool.py", line 173, in listContentTypes
    provider_value = _getOb(provider, None)
  File "./eggs/Zope-4.8.7-py2.7.egg/OFS/ObjectManager.py", line 323, in _getOb
    if id[:1] != '_' and hasattr(aq_base(self), id):
  File "./parts/erp5/product/ERP5Type/dynamic/lazy_class.py", line 120, in __getattribute__
    self.__class__.loadClass()

And simplify just a bit the code

FutureWarning: This search incorrectly ignores the root element, and will be
fixed in a future version.  If you rely on the current behaviour, change it to
'.//role'.

py3: TestTradeModelLineMixin inherited from UserDict() to store values on the class direcly (nexedi/erp5!1751).

This does not work with py3:
    File "parts/erp5/Products/ERP5Type/tests/runUnitTest.py", line 941, in main
      result = runUnitTestList(test_list=args,
    File "parts/erp5/Products/ERP5Type/tests/runUnitTest.py", line 703, in runUnitTestList
      result = TestRunner(verbosity=verbosity).run(suite)
    [...]
    File "parts/python3/lib/python3.9/unittest/runner.py", line 184, in run
      test(result)
    [...]
    File "parts/python3/lib/python3.9/unittest/suite.py", line 84, in __call__
      return self.run(*args, **kwds)
    File "parts/python3/lib/python3.9/unittest/suite.py", line 111, in run
      if _isnotsuite(test):
    File "parts/python3/lib/python3.9/unittest/suite.py", line 369, in _isnotsuite
      iter(test)
    File "parts/python3/lib/python3.9/collections/__init__.py", line 1067, in __iter__
      return iter(self.data)
  AttributeError: 'TestComplexTradeModelLineUseCaseSale' object has no attribute 'data'

Use a dedicated dict() to store these values as there was no strong reasons to
inherit from UserDict() here and this makes test implementation easier too...

group_id is used as key of OOBtree and as documented, it's not possible to mix
keys that can not be compared, so we can not have a mix of string and bytes, for
consistency with other BTrees, such as the ones used for OFS.

group_id is also used in a SQL column which is BINARY, this is problematic on
py3 because the selected values will be returned as bytes, but we expect str
here. Because we don't want to run a data migration, we adjust the select
methods to convert to str while selecting.

Since years there was a warning that id_group must be a string, now we make it a
bit stricter, we also enforce that the id_group is valid UTF-8.

A few more tests and assertions were also added.

Since Zope commit 020685087 (`Allow ZPublisher to handle a query string together
with a request body (#1124)`, 2023-05-15) Zope tries to process all XML HTTP
requests as XML-RPC and we need to tell that these SOAP requests are not XML-RPC.

py2/py3: Base64 encode inventory cache, as Shared.DC.ZRDB.DA.SQL tries to decode bytes to str (nexedi/erp5!1751).

Co-authored-by: Arnaud Fontaine <arnaud.fontaine@nexedi.com>

Followup of 94739085.

key()-based sorting is now used instead, available since Python 2.4.

Jupyter integration is more or less abandoned. Just do the minimum to make tests pass.

Also, remove `if True/1/False/0` statement raising pylint warnings.
Co-authored-by: Arnaud Fontaine <arnaud.fontaine@nexedi.com>

* Zope API changes:
  + publish():
    - stdin is now BytesIO rather than StringIO.
    - Returned value of a script is passed to str() in python2, not in python3 anymore.
  + HTTPResponse `body` property is now bytes().
  + OFS.Image.File file parameter is bytes().
  + zope.interface implements() is now @implementer decorator.
  + Python standard logging module recommended instead of zLOG.
* Python3 API changes:
  + builtin reduce() was removed.
  + urlnorm is now available.
  + Use BytesIO rather than StringIO to follow py3 API.
  + hmac.new() requires digestmod argument from Python 3.8.
  + Use six.moves library to handle moved objects from py2 to py3.
  + `modernize -f xrange_six` then slightly adjusted manually to just use range
     where it does not make a significant difference (for example in test).
  + base64.b64encode() now expects bytes().
  + UserDict() interface changed:
    - New parameter in update() and pop().
    - `failobj` setdefault parameter renamed to `default`.
  + ensure_list() on dict.{values,items}() and list(dict) for dict.keys() when
    we really need a list and not an iterable (Python3).
  + Make dict iteration works on both version of Python.
    - Use six.iter{items,values,keys}().
    - has_key() has been removed.
    - Make sure that dict.{items,values,keys}() returns a real list when
    modified (ensure_list()).
  + Comparisons between int and NoneType raises TypeError.
    + BTrees key must be str() not int() (_getOb()).
  + No more unbound methods in python3 so use six.get_unbound_function().
  + Exceptions:
    - No longer behave as sequences. Using `args` attribute instead.
    - When an exception has been assigned using `as target`, it is cleared at
      the end of the except clause.
  + file: py2 was returning `str` upon reading, now it returns text strings.
    Also, opening mode is text strings by default.
  + Data strings are bytes().
    - Replace str() by bytes().
  + iterators no longer have next() method, instead there is next() builtin.
  + New ConnectionError exception so rename existing one to not clash.
  + Integer division is now with //.
  + __nonzero__ is now __bool__.
  + apply() does not exist anymore.
  + Deprecated threading.Thread isAlive() has been removed.
  + im_func replaced by __func__.
  + Use six.with_metaclass() to define metaclass in a cross-compatible way with py2 and py3.
  + Only test method can be marked as expectedFailure(), not assert statement anymore.
  + os.path.walk() removed.
  + HTMLParser never fails: no strict mode nor HTMLParseError anymore (Python #15114).
  + Unpickler.find_global() is now Unpickler.find_class().
Co-Authored-by: Kazuhiko SHIOZAKI <kazuhiko@nexedi.com>
Co-Authored-by: Arnaud Fontaine <arnaud.fontaine@nexedi.com>
Co-Authored-by: Carlos Ramos Carreño <carlos.ramos@nexedi.com>
Co-Authored-by: Emmeline Vouriot <emmeline.vouriot@nexedi.com>