An error occurred fetching the project authors.
  1. 18 Sep, 2024 1 commit
  2. 02 Jul, 2024 2 commits
    • Vincent Pelletier's avatar
      Products.CMFActivity: Distribute all activities pending distribution · 4cbc4eea
      Vincent Pelletier authored
      Activity distribution (actually: validation) is fundamentally a bottleneck
      of the current CMFActivity design: there can be only one distribution/
      validation node, and it must check every single activity which express a
      dependency over any activity, until these dependencies are satisfied.
      As a result, distribution/validation is in the critical path between an
      activity being spawned and it being executed, and this work cannot be
      parallelised. So care should be taken to waste as little time as possible,
      in order to reduce the activity execution latency.
      
      Before this change, CMFActivity would distribute at most 1000 activities
      (MAX_VALIDATED_LIMIT) per queue per timerserver wake-up.
      In a typical ERP5 setup, timerserver ticks once per second, which means
      CMFActivity was unable to validate more than 1000 activities per second
      per queue. Maybe there are more activities which are possible to validate
      but still the code was forcing the node to sleep until the next wake-up,
      which is a tremendous waste of time.
      
      This change fixes that issue by having ActivityTool.distribute keep looping
      until there has been an iteration over all activity queues which
      simultaneously did not find any activity they could validate. This lets
      Zope yield CPU control when it would be better spent processing activities
      (those which are preventing the validation of any activities remaining to
      validate) without imposing a maximum effective validation-per-second
      hard limit.
      4cbc4eea
    • Vincent Pelletier's avatar
      Products.CMFActivity: Inverse dequeueMessage return value · 02791ba1
      Vincent Pelletier authored
      dequeueMessage used to return False when something was done, and True
      otherwise. The only caller which uses that value then reverses it, causing
      a double-negative which makes the code confusing to read.
      Inverse the meaning of the return value, getting rid of this double-
      negation.
      02791ba1
  3. 03 Jun, 2024 1 commit
  4. 07 Mar, 2024 2 commits
  5. 21 Feb, 2024 2 commits
  6. 06 Feb, 2024 1 commit
  7. 03 Mar, 2023 1 commit
  8. 01 Mar, 2023 1 commit
  9. 27 Feb, 2023 1 commit
    • Jérome Perrin's avatar
      CMFActivity,syncml: stop using a dedicated log file · fcd26be0
      Jérome Perrin authored
      $INSTANCE_HOME/log only exists in runUnitTest instances, this code had
      no effect in slapos instances.
      
      It is a future problem because it depends on the Signals module (part
      of ZServer package) and we are removing this dependency in the python 3
      port.
      
      For the SynchronizationTool, it was even more problematic because it
      can cause the component not to loed, with an error like:
      
          ImportError: erp5.component.tool.SynchronizationTool: cannot load Component SynchronizationTool (signal only works in main thread)
      fcd26be0
  10. 30 Sep, 2022 1 commit
  11. 13 Jun, 2022 2 commits
  12. 01 Jun, 2022 1 commit
  13. 04 May, 2022 2 commits
    • Arnaud Fontaine's avatar
      py3: _mysql.string_literal() returns bytes(). · 94739085
      Arnaud Fontaine authored
      And _mysql/mysqldb API (_mysql.connection.query()) converts the query string to
      bytes() (additionally, cursor.execute(QUERY, ARGS) calls query() after
      converting everything to bytes() too).
      94739085
    • Arnaud Fontaine's avatar
      py2/py3: Make Products code compatible with both python2 and python3. · a17bb910
      Arnaud Fontaine authored
      Done through various 2to3 fixers (zope.fixers, modernize, future) and manual
      changes. This is a single commit so that we have a clearer picture of how code
      converted with my2to3 should look like.
      
      Except straightforward @implementer decorator 2to3 fixer, only product/ folder
      was considered as the goal was to be able to create an ERP5Site.
      
      * Use @implementer decorator introduced in zope.interface 3.6.0 (2010):
      
        The implements syntax used under Python 2.X does not work under 3.X, since it
        depends on how metaclasses are implemented and this has changed. Instead it
        now supports a decorator syntax (also under Python 2.X).
      
        Applied thanks to 2to3 `zope.fixers` package.
      
      * Use `six.moves` rather than `future` install_aliases() feature because the
        latter use unicode_literals and "wraps" module aliases so that unicode() are
        returned for text rather than str() (Python2 standard library). This notably
        breaks BusinessTemplate code which uses urllib quote() for filesystem paths...
      
      * No more unbound methods in python3 so use six.get_unbound_function().
      
      * dict.(iteritems,iterkeys,itervalues)() => six.\1(dict) thanks to `dict_six`
        2to3 fixer from `modernize`:
        $ python-modernize -w -f dict_six product/
      
      * Manually make sure that dict.{items,values,keys}() returns a real list when it
        is latter modified rather than a dict_{items,values,keys} (ensure_list()). By
        default, 2to3 blindly does list(dict.{items,values,keys}()) which is not
        acceptable from performances point of view. With my2to3, this will be possible
        to handle such case automatically.
      
      * Replace cStringIO.StringIO() by six.moves.cStringIO() (a module alias for
        cStringIO.StringIO() on py2 and io.StringIO() on py3).
      
      * Use six.text_type which maps to unicode() on py2 and str() on py3. This also
        makes a clearer difference between text and binary strings.
      
      * Replace map()/filter() with lambda function by list comprehension (this has
        the benefit to avoid casting to list for py3 as it returns iterators).
      a17bb910
  14. 27 Apr, 2022 1 commit
    • Vincent Pelletier's avatar
      Products.CMFActivity.ActivityTool: Store user object in activity. · f363ac65
      Vincent Pelletier authored
      When spawning an activity, store the current security context's user in
      the Message object itself, so the activity security context can be
      re-created with the same security during activity execution.
      This allows a user to be modified (different groups, global roles, maybe
      removed altogether) after they spawned activities and before these activities
      could run.
      It also means that any temporary custom group or global role granted to
      that user (by a privilege elevation mechanism out of the scope of this
      change) will still be effective during the activity execution.
      This follows the principle that
        foo.activate(...).bar(...)
      should be equivalent to its "immediate execution" version
        foo.bar(...)
      by ensuring that the security context of the activity is the same as the
      one which was applied to the code which spawned that activity,
      independently of any intermediate configuration change - hence improving
      (deferred and fragmentary) transaction isolation.
      
      This also removes the need to look the user up, then looking up their
      assignments (and other documents involved in group computation), etc,
      saving the cost of these calls.
      
      Also, remove redundant user_name argument of Message.changeUser method.
      f363ac65
  15. 21 Feb, 2022 1 commit
  16. 17 Feb, 2022 2 commits
  17. 07 Jan, 2022 1 commit
    • Vincent Pelletier's avatar
      Revert "Products.CMFActivity.ActivityTool: Improve behaviour on single-node instances." · 340acac6
      Vincent Pelletier authored
      Also revert related fixup commits:
        "CMFActivity: fixup do not loop on tic if the node is the distribution node"
        "Products.ERP5Type.tests: Follow-up on ActivityTool.tic signature change."
      
      While the original commit did improve the specific workload it was
      designed to improve, it turned out to degrade too much intensive activity
      workloads, like initial ERP5 site creation and tests (which, for the
      purposes of this change, are the same as a single-zope instance).
      Given how easy it is to get a multi-Zope instance, which would solve the
      original issue and also provide the better performance necessary anyway
      for an instance managing a non-trivial amount of documents, I choose to
      revert this change.
      
      I am not reverting several loosely-related changes I applied, which
      rather fix real bugs uncovered by the different activity execution
      scheme this change provided, especially by letting tests'
      "stop_condition" callback being executed a lot more often between
      activities, uncovering missing dependencies and unrealistic test
      expectations, whose fixes should be beneficial independently from the
      reverted code.
      
      This reverts commit 4dfafbc9.
      This reverts commit 4eb26017.
      This reverts commit 041642d0.
      340acac6
  18. 04 Jan, 2022 1 commit
  19. 23 Dec, 2021 1 commit
    • Vincent Pelletier's avatar
      Products.CMFActivity.ActivityTool: Improve behaviour on single-node instances. · 041642d0
      Vincent Pelletier authored
      - Ignore node preference when spawning activities.
        Otherwise, activities which are not spawned with a preferred node will
        get an effective priority penalty compared to same-priority activities
        spawned *with* a node preference, despite both being to execute by the
        same processing node.
      - Break activity processing loop when the current processing node is also
        the activity validation node.
        This avoids pathological cases of activity accumulation, for example when
        reindexing an entire site: _recurseCallMethod is spawned in
        processing_node=0, but immediateReindexObject is spawned in
        processing_node=-1 because of serialization_tag dependency, so with such
        loop _recurseCallMethod will be executed over and over, piling indexation
        activities up until _recurseCallMethod does not self-respawn.
        In turn, such activity accumulation lead to an increased overhead, and
        decreased activity processing efficiency.
        This may also allow multi-node instances to more reliably use the
        validation node as a processing node.
      
      The cost for multi-node instances of these changes should be absolutely
      minimal (no extra IO necessary, minimal extra code).
      A possible drawback on single-node instances is that tic period may become
      more important because process_timer will return more often.
      041642d0
  20. 29 Apr, 2021 1 commit
  21. 29 Mar, 2021 1 commit
    • Jérome Perrin's avatar
      CMFActivity: Always set zope.globalrequest · 9651e553
      Jérome Perrin authored
      ERP5 uses a mix of context.REQUEST and Products.ERP5Type.Global.get_request(),
      which now uses zope.globalrequest.getRequest().
      CMFActivity reconstruct the original request before executing activity, so that
      the activity is executed with a request equivalent to the request at the time
      where the method was activated. For this, context.REQUEST was properly restored,
      but get_request()/globalrequest was only restored when the REQUEST had some
      Accept-Language header, so that it replays the language negociation for
      Localizer.
      
      With browser requests, I guess every browser pass an Accept-Language header (
      according to https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language
      all major browser support it), but within unit tests such header is not present.
      So this lead to activities with unit test requests being slightly different,
      because as they don't have Accept-Language, context.REQUEST was set, but the
      global request was not set, so they were running with two different requests
      in context.REQUEST and global request, leading to some problems for example
      with formulator fields.
      9651e553
  22. 25 Jan, 2021 1 commit
  23. 18 Jan, 2021 1 commit
    • Vincent Pelletier's avatar
      CMFActivity: Optimise validation queries. · a016ed04
      Vincent Pelletier authored
      See SQLBase._getExecutableMessageSet for operation principle.
      Removes the notion of order_validation_text: activity validation is no
      longer evaluated per-activity , but per-dependency for multiple activities
      at a time. In this context, order_validation_text does not make sense as
      it flattens all dependency types for a given activity.
      Rework activity-dependency-to-SQL methods: use a dict rather
      dynamically-generated method names.
      Based on initial work by Julien Muchembled.
      a016ed04
  24. 11 Dec, 2020 1 commit
  25. 13 Jul, 2020 1 commit
  26. 05 Jun, 2020 1 commit
  27. 25 May, 2020 1 commit
  28. 26 Dec, 2019 1 commit
    • Vincent Pelletier's avatar
      CMFActivity.ActivityTool: Use uid for identity check instead of oid. · 06aee0c2
      Vincent Pelletier authored
      uids are a way one can signal that different objects (from a ZODB point of
      view, and hence an oid point of view) are actually to be considered as the
      same objet (from a higher abstration point of view). For example, another
      variant of the same object, as imported over an older variant.
      In turn, this allows extending the protection to activities spawned from
      brains, and not just from live objects.
      06aee0c2
  29. 14 Nov, 2019 1 commit
  30. 13 Nov, 2019 1 commit
  31. 13 Sep, 2019 2 commits
  32. 25 Jul, 2019 1 commit
    • Vincent Pelletier's avatar
      CMFActivity.ActivityTool: Set exc_type default value. · 3123e1a9
      Vincent Pelletier authored
      This property is only conditionally set, which causes distracting error
      when something goes wrong. If we really want to check that the expected
      code path was followed there has to be a better check and a more debugging-
      friendly error than an AttributeError.
      3123e1a9
  33. 28 Jun, 2019 1 commit