in ec835d86 (payroll_l10n_fr: add missing sorts in DSN report, 2024-02-28)
we added sorts so that the report is deterministic when hash randomization
is enabled, but this was not enough, when iterating through employee_ctp
to find the social_contribution_organisation, not all records have a
corporate_registration_code, so we check that it's present before
accessing it.

Also, one `sorted` that was added was incorrect, because the elements
are dict, so they can not be compared on python3.

Instead of overriding the method and adjusting edit order in some class,
define "_default_edit_order" as a class value and use it in Base._edit
as default value when caller do not explicitly pass edit_order.

This was made to keep the default edit order consistent with the order
of edits on python2 for properties where the edit order matters.

This affects mostly scripts, when for example in a script we do:

  delivery.edit(start_date=d, stop_date=d)

on python2 without PYTHONHASHSEED, stop_date is also set, so we keep
this behavior (that is assumed by some tests).

We also change the order of edit for other properties not constraint
by edit_order to edit them in alphabetic order, to have a constant
deterministic behavior.
Co-authored-by: Kazuhiko SHIOZAKI <kazuhiko@nexedi.com>

Because this was using a set, the order in which property sheets were
used when generating classes was not deterministic.
This can make a different if two property sheets define the same
property. This was the case for short_title property, which was defined
with different default values in DublinCore and SimpleItem property
sheets. With this change the behavior is always same when hash
randomization is enabled.

Property sheets can be defined in classes or on the portal type. The
test only covers the case that a property defined on the portal type
must override the same property defined on a class, but the expected
behavior is also that property sheets defined on a class override
property sheets from parent class.

With a perfect configuration, this should probably does not matter,
because the order of interactions should not be important (if there
are dependencies they should be handled more explicitly), but this
makes the behavior deterministic.
Co-authored-by: Kazuhiko SHIOZAKI <kazuhiko@nexedi.com>

movement.hasCategory('quantity_unit') was always false, it was a
misunderstanding of hasCategory API. This was working anyway because
of the order of edit (which depends on iteration order of **kw arguments
passed to edit), this was only working when quantity unit was set before
resource.

( this was revealed by test_01_quantity_unit_copied from testInvoice
running on python2 with PYTHONHASHSEED=164 )

We have DublinCore.short_title and SimpleItem.short_title, make the two
use the same default value for consistency.

This is what happens with PYTHONHASHSEED=0, but when set to something
random, the order was not guaranteed.

Revealed by erp5_accounting_ui_test:testERP5CurrencyExchangeLine
PYTHONHASHSEED: 438 on py2
Co-authored-by: Kazuhiko SHIOZAKI <kazuhiko@nexedi.com>

When searching for something like for example "A AND B", make sure that
the AND complex query contains query for ["A", "B"], not ["B", "A"].
This is probably always equivalent in generated SQL, but this makes
testing easier.

Revealed by testSQLCatalog.TestSQLCatalog with PYTHONHASHSEED 279 on py2

When cutting with bytes, we might cut in the middle of a multi-bytes
character and later decoding it would cause an error. We could switch
to sending bytes encoded to base64, because the limit would be more
natural to be number of bytes, but just decoding and cutting the string
is easier and in practice not so different.

This bug was revealed by testERP5SyncML.TestERP5SyncML.test_28_PartialData
with PYTHONHASHSEED 872 and test_28_PartialData with PYTHONHASHSEED 238
( on python2 )

The order is used during syncml and if "title" comes before "first_name"
and "last_name", synchronizing persons causes conflicts (because title
of persons is computed from first_name and last_name).
Luckily, sorting properties gives an order that is OK for syncml tests,
so we don't need to invent a new mechanism to control the order of
properties in asXML.

if their content type ended in xml ( such as image/svg+xml ),
TextDocument._convertToBaseFormat was resetting the content type to
application/xml.

When doing something like:

  wp = web_page_module.newContent(
    portal_type='Web Page',
    content_type='image/svg+xml',
    text_content=svg
  )

wp.getContentType() was sometimes 'image/svg+xml' sometimes
'application/xml', depending on the order of edit.

for erp5_web_renderjs_ui_test:testFunctionalRJSPage failures with
PYTHONHASHSEED 347

This produces stable output that we can assert more easily in the test

This solves a failure with testBudget, it should only impact test, but
it's probably better to have a deterministic ordering here.

When a business template has multiple dependencies, these dependencies
were installed in an order defined by iteration in dictionaries, which
is not deterministic when using a random PYTHONHASHSEED on python2.

As an example, when we have business template "b" depending on "b1" and
"b2", the only guarantee was that "b" would be installed after "b1" and
"b2", but which one of "b1" and "b2" is installed first was undefined,
now it is stable, "b1" should be installed before "b2", because we added
a sort. In practice it should not matter, because if in that example
"b2" really needs to be installed after "b1", then "b2" should declare a
dependency on "b1".

test_resolveBusinessTemplateListDependency was failing on python3
because the order on python3 was different from the one using
PYTHONHASHSEED=0 on python2.

In that case, the order of actions rendering is undefined, we want to
have unique priorities to have deterministic order of actions.

This probably happened only in the test, but it's safer not to retry
for ever.

With PYTHONHASHSEED=0, editing a document like this:

    foo.edit(
      title="live_test_%s" % new_id,
      reference="live_test_%s" % new_id
    )

only sets the reference, because it sets the reference first and then
when setting title it already has the "live_test_%s" % new_id value (
because title is acquired from reference).

With PYTHONHASHSEED set, the order is not guaranteed and sometimes
title is edited before reference. This is revealed by
testHalJsonStyle.TestERP5Document_getHateoas_mode_search with
PYTHONHASHSEED 465.

Co-authored-by: Kazuhiko SHIOZAKI <kazuhiko@nexedi.com>

otherwise edit is called with both (new) resource_uid and (old) resource_value.

revealed by erp5_web_renderjs_ui_test:testFunctionalRJSDeveloperMode
with PYTHONHASHSEED: 977 on py2

erp5_workflow_test:testWorkflowAndDCWorkflow PYTHONHASHSEED=872 fails
with:

    ValueError: UID mismatch for <Workflow Script at /erp5_portal_7faf002ae74ecbab0dce45c9fca0f5e1/portal_workflow/testing_erp5_interaction_workflow/script_afterValidate>

Using portal_transforms is not deterministic, because we don't have
direct chain from ods to text, so the conversion is two steps, either
ods=>html=>text or ods=>pdf=>text.

This caused problem with hash randomization, revealed by
testDeferredStyle.TestODSDeferredStyle.test_report_method_access_request
with PYTHONHASHSEED=538 on python2. When using PYTHONHASHSEED=0 , this
was using ods=>html=>text and the assertion from the test was OK, but
when using ods=>pdf=>text the test fails because the words are cut at
end of line.

use a valid image in testERP5ImageProcessor, this test evolved and
became a test for ERP5 behavior with broken images.

This indirectly solves a failure with testERP5ImageProcessor
and PYTHONHASHSEED: 805