1. 07 Nov, 2019 10 commits
    • Kirill Smelkov's avatar
      *: Suppress DeprecationWarning on `import imp` · ec2e882b
      Kirill Smelkov authored
      As of 20191107 imp is present in Python 3.8 source tree and is _used_
      there - see e.g.
      
          https://github.com/python/cpython/blob/v3.8.0-66-g7c20888e713/Lib/pkgutil.py#L188-L192
      
      And they say imp will be there until at least Python 4 - see e.g.
      
          https://github.com/python/cpython/commit/e4f41deccf94
      
      This fixes test_defer_excchain_dump failure under python3-dbg, where
      
      E           Failed: not equal:
      E           Differences (unified diff with -expected +actual):
      E               @@ -1,4 +1,6 @@
      E               +PYGOLANG/golang/_gopath.py:37: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
      E               +  import imp
      
      was added into traceback printed on stderr.
      
      Fixes: bb9a94c3 (golang: Teach defer to chain exceptions (PEP 3134) even on Python2)
      ec2e882b
    • Kirill Smelkov's avatar
      time: Move/Port timers to C++/Pyx nogil · b073f6df
      Kirill Smelkov authored
      Provide time.Ticker, time.Timer and friends that can be used directly
      from C++ and Pyx/nogil codes. Python-level classes become small wrapper
      around pyx/nogil ones.
      
      This is the first patch that moves to Pyx/nogil classes that are
      dynamically allocated on heap. refptr<T> is used to automatically manage
      lifetime of such objects. At Pyx level exposed API is very similar to
      Python-one, while internally it uses refptr<T> and friends.
      b073f6df
    • Kirill Smelkov's avatar
      libgolang: Switch _chan refcounting to refobj · e614d641
      Kirill Smelkov authored
      Refobj was just added in previous patch as the class that provides base
      functionality for reference-counted objects.
      e614d641
    • Kirill Smelkov's avatar
      libgolang: Objects refcounting (initial draft) · e82b4fab
      Kirill Smelkov authored
      Since we are going to move more pygolang functionality into C++ (timers,
      context, ...), and since C++ does not have garbage collector, we will
      need to find a way to automatically manage memory in leak/error free way.
      
      Let's do this via refptr<T> smart pointer (inspired by WebKit's RefPtr<T>),
      which, similarly to chan<T> automatically manages pointed object's
      lifetime through reference counting.
      
      refptr<T> will be used in follow-up patches.
      
      Top-level documentation is TODO.
      e82b4fab
    • Kirill Smelkov's avatar
      golang: Provide way to wrap pyx/nogil-level chan[T] into pychan · 07f9430d
      Kirill Smelkov authored
      This will be needed in Python-wrappers for C-level code which exposes
      channels. For example in time package whose implementation will soon be
      moved to nogil world fully.
      07f9430d
    • Kirill Smelkov's avatar
      readme: Provide more detailed example for pychan -> chan[T] conversion · 69b80926
      Kirill Smelkov authored
      3121b290 (golang: Teach pychan to work with channels of C types, not
      only PyObjects) added ability to convert pychan to chan[T], but
      described that ability only briefly in README. Provide more details in
      that description before we add description and implementation for
      reflexive pychan <- chan[T] wrapping.
      69b80926
    • Kirill Smelkov's avatar
      libgolang: Split time and sync into their own packages · 1a9dae3b
      Kirill Smelkov authored
      We already have some sync functionality implemented in C++ (e.g.
      sync.Once, sync.WaitGroup) and we are going to add more and also move
      timers implementation to C++. It is getting crowded for that
      functionality to still live in libgolang.{h,cpp}
      
      -> Split sync & time functionality into their own C++ packages (still
      built-into libgolang.so)
      1a9dae3b
    • Kirill Smelkov's avatar
      time: Provide package documentation · 419c8950
      Kirill Smelkov authored
      Briefly describe provided functionality instead of only referring to
      https://golang.org/pkg/time.
      419c8950
    • Kirill Smelkov's avatar
      sync: Provide package documentation · 40f3b90c
      Kirill Smelkov authored
      Briefly describe provided functionality instead of only referring to
      https://golang.org/pkg/sync.
      40f3b90c
    • Kirill Smelkov's avatar
      pyx.build: Extension.depends += sync.pxd, time.pxd · b06ec3d4
      Kirill Smelkov authored
      Make built extensions depend on golang/{sync,time}.pxd and rebuild if
      those files change. A bit overkill to depend globally, but more correct,
      since an extension that is using pygolang can be using those packages.
      
      Before the patch - without those depends - since distutils does not
      implement proper dependency tracking, extensions - even those that
      actually depend on sync/time pxd, were not rebuilt.
      
      sync.pxd was missed to be added in 34b7a1f4 (golang: Expose Sema and
      Mutex as public Python and Cython/nogil API). time.pxd was missed to be
      added in ce8152a2 (pyx api: Provide sleep).
      b06ec3d4
  2. 06 Nov, 2019 3 commits
  3. 04 Nov, 2019 6 commits
  4. 31 Oct, 2019 1 commit
    • Kirill Smelkov's avatar
      libgolang: Fix build with old gcc · 77719d8a
      Kirill Smelkov authored
      Gcc 5.4 from Ubuntu 16.04 LTS complains e.g.
      
          ./golang/libgolang.h: In function ‘golang::_selcase golang::_selrecv_(golang::_chan*, void*, bool*)’:
          ./golang/libgolang.h:227:5: sorry, unimplemented: non-trivial designated initializers not supported
               };
               ^
      
      The problem started to appear after 47111d3e (libgolang: Teach select to accept
      inplace tx data) when we moved _selcase.ptxrx inside union.
      
      Let's add workaround for older compilers, even though e.g. gcc 8.3 from Debian
      10 accepts existing code just fine.
      77719d8a
  5. 30 Oct, 2019 1 commit
    • Kirill Smelkov's avatar
      sync: Move/Port WaitGroup to C++/Pyx nogil · b316e504
      Kirill Smelkov authored
      Provide sync.WaitGroup that can be used directly from C++ and Pyx/nogil codes.
      Python-level sync.WaitGroup becomes small wrapper around pyx/nogil one.
      
      Python-level tests should be enough to cover C++/Pyx functionality at
      zero-level approximation.
      b316e504
  6. 24 Oct, 2019 1 commit
  7. 22 Oct, 2019 1 commit
    • Kirill Smelkov's avatar
      libgolang: Expose defer as public C++ API · 1d153a45
      Kirill Smelkov authored
      Libgolang, since 3b241983 (Port/move channels to C/C++/Pyx), already had
      defer macro implemented and used, but only internally. The reason it was
      not yet exposed as public API is that there is a difference with Go's
      defer in that deferred function is called at end of current scope
      instead of end of current function, and I was a bit reluctant to expose
      defer with different-than-Go semantic.
      
      However even with this difference defer is useful, and the difference
      can be documented. Unfortunately it is not easy to correctly fix the
      difference, so the most practical way for now is to expose defer as it is.
      
      I've also contemplated how to avoid using macro, but without a macro,
      users will have to explicitly declare placeholder variable for every
      defer call which goes against usability.
      
      Since defer is exposed as macro, I've also contemplated to expose it as
      something like `libgolang_defer` with the idea to avoid name conflicts,
      and so that users - that are using defer - will be doing `#define defer
      libgolang_defer`. However I ended up not doing that and exposing `defer`
      macro with its own name. My rationale is:
      
      - grepping /usr/include/ for \<defer\> on my system did not showed any
        real usage.
      - Qt also #defines `slots` and `signals` and that does not cause
        problems in practice.
      
      -> expose `defer` macro into public C++ API as is, so that it can be
      used not only inside libgolang.cpp . For example I myself need defer
      functionality in C++ part of wendelin.core.
      1d153a45
  8. 18 Oct, 2019 3 commits
    • Kirill Smelkov's avatar
      golang: Fix recover to clear current exception · 9e6ff8bd
      Kirill Smelkov authored
      Else the exception, even if it was recovered from, will be included as
      cause for next raised exception. See added test for details.
      9e6ff8bd
    • Kirill Smelkov's avatar
      golang: Teach defer to chain exceptions (PEP 3134) even on Python2 · bb9a94c3
      Kirill Smelkov authored
      Python3 chains exceptions, so that e.g. if exc1 is raised and, while it
      was not handled, another exc2 is raised, exc2 will be linked to exc1 via
      exc2.__context__ attribute and exc1 will be included into exc2 traceback
      printout. However many projects still use Python2 and there is no
      similar chaining functionality there. This way exc1 is completely lost.
      
      Since defer code is in our hands, we can teach it to implement exception
      chaining even on Python2 by carefully analyzing what happens in
      _GoFrame.__exit__().
      
      Implementing chaining itself is relatively easy, but is only part of the
      story. Even if an exception is chained with its cause, but exception
      dump does not show the cause, the chaining will be practically useless.
      With this in mind this patches settles not only on implementing chaining
      itself, but on also giving a promise that chained cause exceptions will
      be included into traceback dumps as well.
      
      To realize this promise we adjust all exception dumping funcitons in
      traceback module and carefully install adjusted
      traceback.print_exception() into sys.excepthook. This amends python
      interactive sessions and programs run by python interpreter to include
      causes in exception dumps. "Careful" here means that we don't change
      sys.excepthook if on golang module load we see that sys.excepthook was already
      changed by some other module - e.g. due to IPython session running
      because IPython installs its own sys.excepthook. In such cases we don't
      install our sys.excepthook, but we also provide integration patches that
      add exception chaining support for traceback dump functionality in
      popular third-party software. The patches (currently for IPython and
      Pytest) are activated automatically, but only when/if corresponding
      software is imported and actually used. This should give practically
      good implementation of the promise - a user can now rely on seeing
      exception cause in traceback dump whatever way python programs are run.
      
      The implementation takes https://pypi.org/project/pep3134/ experience
      into account [1]. peak.utils.imports [2,3] is used to be notified when/if
      third-party module is imported.
      
      [1] https://github.com/9seconds/pep3134/
      [2] https://pypi.org/project/Importing/
      [3] http://peak.telecommunity.com/DevCenter/Importing
      
      This patch originally started as hacky workaround in wendelin.core
      because in wcfs tests I was frequently hitting situations, where
      exception raised by an assert was hidden by another exception raised in
      further generic teardown check. For example wcfs tests check that wcfs
      is unmounted after every test run [4] and if that fails it was hiding
      problems raised by an assert. As the result I was constantly guessing
      and adding code like [5] to find what was actually breaking. At some
      point I added hacky workaround for defer to print cause exception not to
      loose it [6]. [7] has more context and background discussion on this topic.
      
      [4] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L70
      [5] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L853-857
      [6] kirr/wendelin.core@c00d94c7
      [7] nexedi/zodbtools!13 (comment 81553)
      
      After this patch, on Python2
      
          defer(cleanup1)
          defer(cleanup2)
          defer(cleanup3)
          ...
      
      is no longer just a syntatic sugar for
      
          try:
              try:
                  try:
                      ...
                  finally:
                      cleanup3()
              finally:
                  cleanup2()
          finally:
              cleanup1()
      bb9a94c3
    • Kirill Smelkov's avatar
      golang_test: Split pyrun into -> _pyrun & pyrun · 6729fe92
      Kirill Smelkov authored
      - _pyrun runs the command and returns full information: exitcode, stdout, stderr.
      - pyrun  runs the command and raises exception if ran command fails.
      
      We will need _pyrun in the next patch to test that particular command
      fails and access its stderr.
      6729fe92
  9. 15 Oct, 2019 2 commits
  10. 14 Oct, 2019 9 commits
    • Kirill Smelkov's avatar
      time: Switch internals to pyx/nogil · 8c2ac5e9
      Kirill Smelkov authored
      - use .c.chan_double() which gives chan[double] pyx/nogil way to access
        Ticker.c and Timer.c. Use the channels via pyx/nogil API from inside.
      - use pyx/nogil sleep and now;
      
      This gets time.pyx codebase closer to be used from pyx/nogil mode.
      
      NOTE: unless something like pyx/nogil memory management emerges[1] we
      are relying on Python to manage memory of Ticker and Timer classes.
      If we just spawn e.g. Ticker.__tick via pyx/nogil go, the thread that is
      spawned won't be holding a reference to Ticker object, and once the
      ticker goes out of scope in original thread (while its channel .c might
      be still in scope), __tick will segfault accessing freed Ticker object.
      
      To workaround it we use the following pattern:
      
          nogilready = chan(dtype='C.structZ')
          pygo(mymeth)
          nogilready.recv()
      
          def mymeth(MyObject self, pychan nogilready)
              with nogil:
                  nogilready.chan_structZ().close()
                  self._mymeth()
          cdef void _mymeth(MyObject self) nogil:
              ...
      
      where python reference to MyObject will be held in spawned thread during
      its lifetime, while the service provided by mymeth will be done under
      nogil.
      
      [1] https://www.nexedi.com/blog/NXD-Document.Blog.Cypclass
      8c2ac5e9
    • Kirill Smelkov's avatar
      time: Switch Timer/Ticker channels from chan() -> chan(dtype='C.double') · 7c929b25
      Kirill Smelkov authored
      This will allow to use the channels in nogil mode including in followup
      patches touching time.pyx codebase.
      7c929b25
    • Kirill Smelkov's avatar
      context: Switch done channel from chan() to -> chan(dtype='C.structZ') · 149ae661
      Kirill Smelkov authored
      C.structZ is empty structure and chan[C.structZ] can be used in
      pyx/nogil world. This way context tree initially created from Python can
      be extended in pyx/nogil and e.g. root context can be canceled from
      Python, which will correctly transfer the cancelation signal to pyx/nogil
      world too.
      149ae661
    • Kirill Smelkov's avatar
      golang: Teach pychan to work with channels of C types, not only PyObjects · 3121b290
      Kirill Smelkov authored
      Introduce notion of data type (dtype similarly to NumPy) into pychan and
      teach it to accept for send objects only matching that dtype. Likewise
      teach pychan to decode raw bytes received from underlying channel into
      Python object correspodningg to pychan dtype. For C dtypes, e.g.
      'C.int', 'C.double' etc, contrary to chan of python objects, the
      transfer can be done without depending on Python GIL. This way channels
      of such C-level dtypes can be used to implement interaction in between
      Python and nogil worlds.
      3121b290
    • Kirill Smelkov's avatar
      *: Channels must be compared by ==, not by "is" even for nilchan · 2c8063f4
      Kirill Smelkov authored
      In a followup commit we are going to add channel types (e.g. chan of
      double, chan of int, etc) and accordingly there will be several nil
      channel objects, e.g. nil(dtype=int), nil(dtype=double) etc, which will
      be separate python objects. Even without data types, another planned
      change is to add directional channels, e.g. a channel instance that can
      only send, but not receive and vice versa(*).
      
      This way for the same underlying channel object, there can be several
      pychan objects that point to it - even for nil channel - e.g. nilchan
      and `niltx = nilchan.txonly()` that creates another pychan object
      pointing to the same underlying nil.
      
      Since we want all channels (of the same type) that point to the same
      underlying channel to compare as same, we cannot use "is" for comparison
      and have to use ==. In other words channels, both at C and Python level,
      should be perceived as pointers, of which there can be multiple ones
      pointing to the same location, and thus == has to be used to compare
      them.
      
      (*) see https://golang.org/ref/spec#Channel_types for details.
      2c8063f4
    • Kirill Smelkov's avatar
      *: Annotate method receiver with corresponding type for all cdef classes · f6fab7b5
      Kirill Smelkov authored
      For example
      
          -    def send(pych, obj):
          +    def send(pychan pych, obj):
      
      Even though Cython allows not to annotate self with type, this makes the
      code more clear.
      f6fab7b5
    • Kirill Smelkov's avatar
      golang: pyselect: Switch into using inplace tx data · 30561db4
      Kirill Smelkov authored
      This prepares pyselect codebase for future logic where all channel
      element types will be sent via inplace _selcase data. For PyObject we
      could previously go with "wiring ptx through pycase[1]", but for
      arbitrary type, that has to be first converted from Python object to
      C-level object, we would have to store the result somewhere, and that
      would mean extra allocation and pyselect code complexity increase, even
      for cases that don't use anything but chan[object].
      
      So do the preparation and switch pyselect into using inplace tx.
      30561db4
    • Kirill Smelkov's avatar
      libgolang: Teach select to accept inplace tx data · 47111d3e
      Kirill Smelkov authored
      Currently select, via _selcase, requires users to provide pointers to tx
      and rx buffers. However if element type itself can fit into a pointer
      word, we can put the element directly into _selcase and mark the case
      with a flag, that it contains inplace data instead of referring to
      external storage. This will be helpful in upcoming patch where we'll
      teach pychan to work with several element types, not only pyobject
      elements.
      
      This patch does careful introduction of _selcase.flags - in such a way
      that the size of _selcase stays the same as it was before by using
      bitfields. The .ptxrx pointer is unioned with newly introduced inplace
      uint64 .itxrx data, which is used by select instead of .ptxrx if the
      flag is set. The usage of uint64 should not increase _selcase size on
      64-bit platforms.
      
      Then _selcase.ptx() and .prx() accessors are adapted accordingly and
      the rest of the changes are corresponding test and
      _chanselect2<onstack=false> adaptation.
      
      This functionality is kind of low-level and is not exposed via any
      _selsend() or chan.sends() API changes. Whenever inplace tx should be
      used, the case should be prepared either completely manually, or with
      e.g. first calling _selsend() and then manually changing .flags and
      .itxrx.  Added test serves as the example on how to do it.
      
      Inplace rx is currently forbidden - because supporting that would require
      to drop const from casev select argument. However in the future, for
      symmetry, we might want to support that as well.
      
      P.S.
      
      Since write to selcase.itxrx requires casting pointers e.g. like this:
      
      	*(int *)&sel[0].itxrx = 12345;
      
      it breaks C99 strict aliasing and by default compiler can generate bad
      code on such pattern. To the problem we adapt the build system
      to default compiler to no-strict-aliasing (as many other projects do,
      e.g. Linux kernel) with the idea that in many cases where strict
      aliasing was intended to help it actually does not, because e.g. pointer
      types are the same, while explicitly marking pointers with `restrict`
      keyword does help indeed.
      
      Nothing new for Python2 here, as it is using -fno-strict-aliasing by
      itself. However Python3 is compiling without -fno-strict-aliasing:
      https://python.org/dev/peps/pep-3123 .
      47111d3e
    • Kirill Smelkov's avatar
      libgolang: _selcase: Introduce accessors to get pointers to for-send and for-recv data · 2590e9a7
      Kirill Smelkov authored
      In the next patch we are going to teach _selcase to support both
      external and inplace data. Before that let's do a couple of preparatory
      things.
      
      This patch: introduces .ptx() and .prx() accessors to get to
      corresponding data buffer associated with _selcase. Convert _selcase
      users to use .ptx() and .prx() instead of doing direct .ptxrx access.
      This way when we'll add inplace support to _selcase, we'll need to adapt
      only accessors, not clients.
      
      The only place that is left using .ptxrx directly is one tricky place in
      _chanselect2<onstack=false> where gevent-related code needs to carefully
      deal with proxying tx/rx buffers due to STACK_DEAD_WHILE_PARKED.
      
      NOTE even though new accessors may panic, libgolang.cpp always calls them
      after checking that the conditions for ptx/prx calls are valid.
      2590e9a7
  11. 13 Oct, 2019 3 commits
    • Kirill Smelkov's avatar
      libgolang: _selcase: Rename .data -> .ptxrx · d6c8862d
      Kirill Smelkov authored
      In the next patches we are going to teach _selcase to support both
      external and inplace data. Before that let's do a couple of preparatory
      things.
      
      This patch: rename .data -> .ptxrx . The new name is more clear:
      
      - "p" prefix aligns with other libgolang style, e.g. ptx or prx.
      - "txrx" suffix says that this is used for both send and recv.
      
      Just plain renaming, nothing else in this patch.
      d6c8862d
    • Kirill Smelkov's avatar
      golang: pyselect: Fix tx object reference leak on error exit · e9180de1
      Kirill Smelkov authored
      Before passing objects to _chanselect for send, pyselect increfs them, as just
      send does, to indicate that one object reference is passed to channel buffer.
      On exit, since only one case is actually executed by select, pyselect needs to
      decref incref'ed object from not executed cases.
      
      Pyselect already implements the latter cleanup, but currently the cleanup is
      executed only if control flow reaches _chanselect at all. Which is a bug, since
      pyselect can panic or raise an exception just in the middle of preparation phase.
      
      -> Fix it by associating the finally-decref cleanup with whole
      prepare+_chanselect code.
      
      Without the fix, the second part of added test (abnormal exit) fails e.g. like:
      
              @mark.skipif(not hasattr(sys, 'getrefcount'),   # skipped e.g. on PyPy
                           reason="needs sys.getrefcount")
              def test_select_refleak():
                  ch1 = chan()
                  ch2 = chan()
                  obj1 = object()
                  obj2 = object()
                  tx1 = (ch1.send, obj1)
                  tx2 = (ch2.send, obj2)
      
                  # normal exit
                  gc.collect()
                  nref1 = sys.getrefcount(obj1)
                  nref2 = sys.getrefcount(obj2)
                  _, _rx = select(
                      tx1,        # 0
                      tx2,        # 1
                      default,    # 2
                  )
                  assert (_, _rx) == (2, None)
                  gc.collect()
                  assert sys.getrefcount(obj1) == nref1
                  gc.collect()
                  assert sys.getrefcount(obj1) == nref2
      
                  # abnormal exit
                  with raises(AttributeError) as exc:
                      select(
                          tx1,        # 0
                          tx2,        # 1
                          'zzz',      # 2 causes pyselect to panic
                      )
                  assert exc.value.args == ("'str' object has no attribute '__self__'",)
                  gc.collect()
          >       assert sys.getrefcount(obj1) == nref1
          E       assert 4 == 3
          E         -4
          E         +3
      
          golang/golang_test.py:690: AssertionError
      
      The bug was introduced in 3b241983 (Port/move channels to C/C++/Pyx).
      e9180de1
    • Kirill Smelkov's avatar
      golang: Rework pychan to use C-level channel API · f2847307
      Kirill Smelkov authored
      We will soon rework pychan to be python wrapper not only for
      chan<object>, but also for other channels of various C types - e.g.
      chan<structZ>, chan<int>, etc.
      
      To prepare for this let's first rework pychan from using chan[PyObject*]
      into raw _chan* functions. This will allow us to use the same functions
      over raw channels while dynamically dispatching on channel element type.
      f2847307