For historical reasons, PDF to text involved conversion first of the PDF to
png, then this png to tiff and the tiff was sent to tesseract. This works, but
it consumes a lot of resources with large PDFs, especially because the
intermediate png/tiff are created with a resolution of 300 DPI, which easily
needs serveral Go of RAM and temporary disk space.
This was obsorved with the PDF created by erp5_document_scanner, which are
usually high quality (1 or 2Mo per page) and even a one page PDF sometimes
took more than one minute to OCR.

Since 9.53 ghostscript integrates tesseract engine directly, we don't need to
prepare a tiff beforehand, we can directly send the PDF data to ghostscript.

These change use ghostscript if available and otherwise fallback to the same
pipeline as before. This will allow the transition until all ERP5 instances
are running a recent enough SlapOS with ghostscript 9.54. Fortunately, before
SlapOS include ghostscript 9.54, ERP5 software release did not have ghostscript
in $PATH, so we don't have to check ghostscript version, we assume that if gs
is in $PATH, it means we have a recent enough SlapOS.

This new approach was less tolerant regarding broken/password-protected PDFs
so we perform a new check that the PDF is valid and not encrypted before
trying to use OCR.

The missing workflow resulted in a bug when moving quantities multiple times, first to one delivery, then to another delivery. Then the same solver process with same decision was used as for the first time even if the first delivery was already delivered or movement was deleted. The reason was that Quantity Split Move Solver did not change to succeeded state after solving, so Solver Process did not change to solved state.

When a first migration was reverted for any reason (e.g. ConflictError),
the assertion failed because __setstate__ was not called yet to fix the
class back to the old one.

OOIBtree can accept None key only on newer version of software

Since 264ded5c in ERP5JS we render the form directly after a successful edit, but this was done sometimes too early, namely, the next form was rendered before interaction workflows and this leads to problems like the ones discussed in nexedi/erp5!982 (comment 92893)

/reviewed-on nexedi/erp5!1040

This test click on the delete button which deletes by an ajax request
and immediatly after open the "wait for activities" page. Sometimes the
second request starts before the first is committed, so there are no
activities to wait for.

When using ZServer, we had only one worker thread, so this was not
visible, but with wsgi we have more than one so it happened sometimes.

Use an old jQuery trick to wait that the first request is no longer in
flight.

/reviewed-on nexedi/erp5!1038

Also covers traversal (and a lot more).

https://tools.ietf.org/html/draft-ietf-httpbis-rfc6265bis-02

SameSite=None breaks the compatibility with some browser versions.
https://www.chromium.org/updates/same-site/incompatible-clients

We choose Lax and not Strict so that we can open links to ERP5 from
external applications and so that OAuth Logins work. Implementing the
"two cookies, one for read one for write" approach suggested in
https://tools.ietf.org/html/draft-ietf-httpbis-rfc6265bis-02#section-8.8.2
would be too big change at this point.

Allow instances to surcharge the SameSite value for some specific domains if needed,
by surcharging the ERP5Site_getAuthCookieSameSite script.

We want to render the next page after interaction workflows are
executed.

This uses a foo_interaction_workflow that sets a short title when title
is set to a special value. We use it to check that when user set the
title to that special value and save, next time the form is displayed,
the values displayed are the ones after interaction workflow modified.

This is a simple and a bit unrealistic scenario, but we have for example
interaction workflows that update security definitions when some
properties are modified, we want the next form to display the values
after modifications.

This is also the case of editions of portal components which sets the
document to "modified" state and "re-validate" it at the end of
transaction, unless pylint detects error.

action added in a previous commit was removed, probably by mistake
because only the metadata was removed.

workflows were not sorted and some metadata files have an extra \n,
probably because of manual edit

This uses a different approach that what I was using in the past. I'm no longer trying to rename skins that does not match our naming conventions, but just adding them the the "ignore list".

This would still allow to detect when new wrongly named scripts are introduced without risking regressions when we rename a script that was used from some places we did not change (like project customisations).

/reviewed-on nexedi/erp5!1041

This was a false positive

We should be able to configure a group calendar saying that the pattern
is "from 9:00 to 12:00, repeat every monday morning" with a group
calendar assignment saying "use this pattern from 01/01/2016 until
31/12/2016" and then create another group calendar assignment for 2017
without having to change the periodicity stop date on all presence
periods of the group calendar.

I think it should repeat from group calendar assignment's start date
until min(group calendar assignment's stop date, presence period's
periodicity stop date).


/reviewed-on nexedi/erp5!125

/reviewed-on nexedi/erp5!940

 - update PresencePeriod.getNextPeriodicalDate with fixes from 6155f7ff
 - do not use addToDate, but simply DateTime arithmetics that unlike addToDate, works correctly

Shared parts received from test node will be passed as
SLAPOS_TEST_SHARED_PART_LIST environment variable to egg tests.

This will be useful for SLAPOS-SR tests.

Some test suites who install software during the test, such as SLAPOS-SR
tests, could benefit from reusing already installed shared parts.

The convention is that --shared_part_list is a os.pathsep (:) separated
list of paths of read-only shared parts in which the test is not allowed
to write.

Shared parts speed up compilation time and is becoming the standard in
SlapOS software installations, so it makes sense to use it in our test
nodes, as it also gives one more opportunity to test this feature.

erp5testnode configuration file supports a new shared_part_list option,
that can be set to a \n separated list of paths to use for shared parts,
following the same rules as slapos.core and slapos.recipe.cmmi (ie. the
first ones are read-only and the last one is read-write).

This shared_part_list option will be set in slapos.cfg used to compile
both the "software for testnode" (ie. selenium-runner) and later the
softwares under tests.
The software under tests will also use a local directory for each test
suite to install shared suite.

The directory structure is now:

  srv/
    shared/
      (shared parts to install selenium runner)
    slapos/
      soft/
        (selenium-runner software)
    testnode/
      foo/ # test suite with reference foo
        inst/
          (partitions of tested software)
        shared/
          (shared parts to install tested software)
        soft/
          (tested software)

and in the configuration srv/shared will be set as initial
shared_part_list.

When installing selenium-runner, srv/shared/ is used to write shared
parts. These shared parts are never removed.

When installing software under test, srv/shared/ and
srv/testnode/foo/shared/ are used. If parts are found in srv/shared they
are used, if they are not found, they are installed in
srv/testnode/foo/shared/.

In practice, this should mean that the shared parts installed by
selenium-runner will be reused for all tested softwares and this should
speed up initial installation of these softwares.

Currently, nothing is implemented regarding removal of unused shared
parts, but in our case:
 - srv/testnode/foo/shared/ will be removed when "foo" is removed.
 - srv/shared/ should be used only when installing selenium-runner.

If this starts to use too much disk space, one quick and dirty
workaround could be to destroy the test node instance and re-create it.

Use monaco editor with https://github.com/joe-re/sql-language-server
completion provider bundled in
https://lab.nexedi.com/jerome/monaco-editor-sql-completion-provider
This completion provider is not perfect, but codemirror's one was not so
good either. At least we can use monaco editor.

Also add a quick "copy to cliboard" feature, which copies the table as
html or in markdown format for text.

And also fix Server-Timing header that was an obsolete syntax no longer
supported by chrome.

/reviewed-on nexedi/erp5!1036

The fixes from nexedi/erp5!630
were not enough, exceptions set on days were calendar did not repeat
were also confusing other exceptions after this.

Simplify implementation a lot, instead of keeping track of the next
exception date, start by building a set of all exceptions dates and
use membership of this set as a criterion to skip exceptions.

/reviewed-on nexedi/erp5!1030

Allow python's cryptographically secure pseudorandom number generator for usage in restricted python and use it where it makes sense.

This also change the API of `Person_generatePassword` which no longer allow to control the number of letters and numbers.

/reviewed-on nexedi/erp5!847