https://bitbucket.org/pypy/pypy/commits/251b47698e8e
https://bitbucket.org/pypy/pypy/issues/3096#comment-54707695

Which in turn fixes recover to clear current exception (see 9e6ff8bd
"golang: Fix recover to clear current exception") for context.

e.g. Context -> PyContext, with_cancel -> pywith_cancel, etc. This will
make the followup transition of context to pyx/nogil more clear.

As of 20191107 imp is present in Python 3.8 source tree and is _used_
there - see e.g.

    https://github.com/python/cpython/blob/v3.8.0-66-g7c20888e713/Lib/pkgutil.py#L188-L192

And they say imp will be there until at least Python 4 - see e.g.

    https://github.com/python/cpython/commit/e4f41deccf94

This fixes test_defer_excchain_dump failure under python3-dbg, where

E           Failed: not equal:
E           Differences (unified diff with -expected +actual):
E               @@ -1,4 +1,6 @@
E               +PYGOLANG/golang/_gopath.py:37: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
E               +  import imp

was added into traceback printed on stderr.

Fixes: bb9a94c3 (golang: Teach defer to chain exceptions (PEP 3134) even on Python2)

Provide time.Ticker, time.Timer and friends that can be used directly
from C++ and Pyx/nogil codes. Python-level classes become small wrapper
around pyx/nogil ones.

This is the first patch that moves to Pyx/nogil classes that are
dynamically allocated on heap. refptr<T> is used to automatically manage
lifetime of such objects. At Pyx level exposed API is very similar to
Python-one, while internally it uses refptr<T> and friends.

Refobj was just added in previous patch as the class that provides base
functionality for reference-counted objects.

Since we are going to move more pygolang functionality into C++ (timers,
context, ...), and since C++ does not have garbage collector, we will
need to find a way to automatically manage memory in leak/error free way.

Let's do this via refptr<T> smart pointer (inspired by WebKit's RefPtr<T>),
which, similarly to chan<T> automatically manages pointed object's
lifetime through reference counting.

refptr<T> will be used in follow-up patches.

Top-level documentation is TODO.

This will be needed in Python-wrappers for C-level code which exposes
channels. For example in time package whose implementation will soon be
moved to nogil world fully.

3121b290 (golang: Teach pychan to work with channels of C types, not
only PyObjects) added ability to convert pychan to chan[T], but
described that ability only briefly in README. Provide more details in
that description before we add description and implementation for
reflexive pychan <- chan[T] wrapping.

We already have some sync functionality implemented in C++ (e.g.
sync.Once, sync.WaitGroup) and we are going to add more and also move
timers implementation to C++. It is getting crowded for that
functionality to still live in libgolang.{h,cpp}

-> Split sync & time functionality into their own C++ packages (still
built-into libgolang.so)

Briefly describe provided functionality instead of only referring to
https://golang.org/pkg/time.

Briefly describe provided functionality instead of only referring to
https://golang.org/pkg/sync.

Make built extensions depend on golang/{sync,time}.pxd and rebuild if
those files change. A bit overkill to depend globally, but more correct,
since an extension that is using pygolang can be using those packages.

Before the patch - without those depends - since distutils does not
implement proper dependency tracking, extensions - even those that
actually depend on sync/time pxd, were not rebuilt.

sync.pxd was missed to be added in 34b7a1f4 (golang: Expose Sema and
Mutex as public Python and Cython/nogil API). time.pxd was missed to be
added in ce8152a2 (pyx api: Provide sleep).

Just cosmetics - easier to oversee the code.

Ticker: self -> pytx
Timer:  self -> pyt

e.g. tick -> pytick, Timer -> PyTimer. etc. This will make the followup
transition of time to pyx/nogil more clear.

e.g. go -> pygo, select -> pyselect, etc. This will make the followup
transition to non-py mode more clear.

Similarly to 0e838833 (time: Turn Ticker and Timer into cdef classes)
rework all classes in context package to be cdef classes:

- cdef their attributes (else accessing any of them raises
  AttributeError). But don't change any of them to be used in cimport
  mode yet - this will be done later as separate steps.

In preparation to start migrating, at least partially, context
functionality to nogil mode, first, similarly to package time (see
32f34607 "time: Move code to pyx"), move the code from context.py to
_context.pyx.

This is straight code movement + associated  setup.py & co adjusts.

We don't move just to context.pyx (note no _ prefix), since we will need
to continue distinguishing pyx/nogil from py objects/functions.

Gcc 5.4 from Ubuntu 16.04 LTS complains e.g.

    ./golang/libgolang.h: In function ‘golang::_selcase golang::_selrecv_(golang::_chan*, void*, bool*)’:
    ./golang/libgolang.h:227:5: sorry, unimplemented: non-trivial designated initializers not supported
         };
         ^

The problem started to appear after 47111d3e (libgolang: Teach select to accept
inplace tx data) when we moved _selcase.ptxrx inside union.

Let's add workaround for older compilers, even though e.g. gcc 8.3 from Debian
10 accepts existing code just fine.

Provide sync.WaitGroup that can be used directly from C++ and Pyx/nogil codes.
Python-level sync.WaitGroup becomes small wrapper around pyx/nogil one.

Python-level tests should be enough to cover C++/Pyx functionality at
zero-level approximation.

Provide sync.Once that can be used directly from C++ and Pyx/nogil worlds.
Python-level sync.Once becomes small wrapper around pyx/nogil sync.Once .

Libgolang, since 3b241983 (Port/move channels to C/C++/Pyx), already had
defer macro implemented and used, but only internally. The reason it was
not yet exposed as public API is that there is a difference with Go's
defer in that deferred function is called at end of current scope
instead of end of current function, and I was a bit reluctant to expose
defer with different-than-Go semantic.

However even with this difference defer is useful, and the difference
can be documented. Unfortunately it is not easy to correctly fix the
difference, so the most practical way for now is to expose defer as it is.

I've also contemplated how to avoid using macro, but without a macro,
users will have to explicitly declare placeholder variable for every
defer call which goes against usability.

Since defer is exposed as macro, I've also contemplated to expose it as
something like `libgolang_defer` with the idea to avoid name conflicts,
and so that users - that are using defer - will be doing `#define defer
libgolang_defer`. However I ended up not doing that and exposing `defer`
macro with its own name. My rationale is:

- grepping /usr/include/ for \<defer\> on my system did not showed any
  real usage.
- Qt also #defines `slots` and `signals` and that does not cause
  problems in practice.

-> expose `defer` macro into public C++ API as is, so that it can be
used not only inside libgolang.cpp . For example I myself need defer
functionality in C++ part of wendelin.core.

Else the exception, even if it was recovered from, will be included as
cause for next raised exception. See added test for details.

Python3 chains exceptions, so that e.g. if exc1 is raised and, while it
was not handled, another exc2 is raised, exc2 will be linked to exc1 via
exc2.__context__ attribute and exc1 will be included into exc2 traceback
printout. However many projects still use Python2 and there is no
similar chaining functionality there. This way exc1 is completely lost.

Since defer code is in our hands, we can teach it to implement exception
chaining even on Python2 by carefully analyzing what happens in
_GoFrame.__exit__().

Implementing chaining itself is relatively easy, but is only part of the
story. Even if an exception is chained with its cause, but exception
dump does not show the cause, the chaining will be practically useless.
With this in mind this patches settles not only on implementing chaining
itself, but on also giving a promise that chained cause exceptions will
be included into traceback dumps as well.

To realize this promise we adjust all exception dumping funcitons in
traceback module and carefully install adjusted
traceback.print_exception() into sys.excepthook. This amends python
interactive sessions and programs run by python interpreter to include
causes in exception dumps. "Careful" here means that we don't change
sys.excepthook if on golang module load we see that sys.excepthook was already
changed by some other module - e.g. due to IPython session running
because IPython installs its own sys.excepthook. In such cases we don't
install our sys.excepthook, but we also provide integration patches that
add exception chaining support for traceback dump functionality in
popular third-party software. The patches (currently for IPython and
Pytest) are activated automatically, but only when/if corresponding
software is imported and actually used. This should give practically
good implementation of the promise - a user can now rely on seeing
exception cause in traceback dump whatever way python programs are run.

The implementation takes https://pypi.org/project/pep3134/ experience
into account [1]. peak.utils.imports [2,3] is used to be notified when/if
third-party module is imported.

[1] https://github.com/9seconds/pep3134/
[2] https://pypi.org/project/Importing/
[3] http://peak.telecommunity.com/DevCenter/Importing

This patch originally started as hacky workaround in wendelin.core
because in wcfs tests I was frequently hitting situations, where
exception raised by an assert was hidden by another exception raised in
further generic teardown check. For example wcfs tests check that wcfs
is unmounted after every test run [4] and if that fails it was hiding
problems raised by an assert. As the result I was constantly guessing
and adding code like [5] to find what was actually breaking. At some
point I added hacky workaround for defer to print cause exception not to
loose it [6]. [7] has more context and background discussion on this topic.

[4] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L70
[5] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L853-857
[6] kirr/wendelin.core@c00d94c7
[7] nexedi/zodbtools!13 (comment 81553)

After this patch, on Python2

    defer(cleanup1)
    defer(cleanup2)
    defer(cleanup3)
    ...

is no longer just a syntatic sugar for

    try:
        try:
            try:
                ...
            finally:
                cleanup3()
        finally:
            cleanup2()
    finally:
        cleanup1()

- _pyrun runs the command and returns full information: exitcode, stdout, stderr.
- pyrun  runs the command and raises exception if ran command fails.

We will need _pyrun in the next patch to test that particular command
fails and access its stderr.

_makesema declaration was giving a warning when compiling with C:

	In file included from golang/runtime/libgolang_test_c.c:26:
	./golang/libgolang.h:237:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
	 LIBGOLANG_API _sema *_makesema();
	 ^~~~~~~~~~~~~

Fix the prototype to be valid for both C and C++.
Fixes: 34b7a1f4.

E.g. wendelin.core amends build_ext class and so it needs a way to
import it.

- use .c.chan_double() which gives chan[double] pyx/nogil way to access
  Ticker.c and Timer.c. Use the channels via pyx/nogil API from inside.
- use pyx/nogil sleep and now;

This gets time.pyx codebase closer to be used from pyx/nogil mode.

NOTE: unless something like pyx/nogil memory management emerges[1] we
are relying on Python to manage memory of Ticker and Timer classes.
If we just spawn e.g. Ticker.__tick via pyx/nogil go, the thread that is
spawned won't be holding a reference to Ticker object, and once the
ticker goes out of scope in original thread (while its channel .c might
be still in scope), __tick will segfault accessing freed Ticker object.

To workaround it we use the following pattern:

    nogilready = chan(dtype='C.structZ')
    pygo(mymeth)
    nogilready.recv()

    def mymeth(MyObject self, pychan nogilready)
        with nogil:
            nogilready.chan_structZ().close()
            self._mymeth()
    cdef void _mymeth(MyObject self) nogil:
        ...

where python reference to MyObject will be held in spawned thread during
its lifetime, while the service provided by mymeth will be done under
nogil.

[1] https://www.nexedi.com/blog/NXD-Document.Blog.Cypclass

This will allow to use the channels in nogil mode including in followup
patches touching time.pyx codebase.

C.structZ is empty structure and chan[C.structZ] can be used in
pyx/nogil world. This way context tree initially created from Python can
be extended in pyx/nogil and e.g. root context can be canceled from
Python, which will correctly transfer the cancelation signal to pyx/nogil
world too.

Introduce notion of data type (dtype similarly to NumPy) into pychan and
teach it to accept for send objects only matching that dtype. Likewise
teach pychan to decode raw bytes received from underlying channel into
Python object correspodningg to pychan dtype. For C dtypes, e.g.
'C.int', 'C.double' etc, contrary to chan of python objects, the
transfer can be done without depending on Python GIL. This way channels
of such C-level dtypes can be used to implement interaction in between
Python and nogil worlds.

In a followup commit we are going to add channel types (e.g. chan of
double, chan of int, etc) and accordingly there will be several nil
channel objects, e.g. nil(dtype=int), nil(dtype=double) etc, which will
be separate python objects. Even without data types, another planned
change is to add directional channels, e.g. a channel instance that can
only send, but not receive and vice versa(*).

This way for the same underlying channel object, there can be several
pychan objects that point to it - even for nil channel - e.g. nilchan
and `niltx = nilchan.txonly()` that creates another pychan object
pointing to the same underlying nil.

Since we want all channels (of the same type) that point to the same
underlying channel to compare as same, we cannot use "is" for comparison
and have to use ==. In other words channels, both at C and Python level,
should be perceived as pointers, of which there can be multiple ones
pointing to the same location, and thus == has to be used to compare
them.

(*) see https://golang.org/ref/spec#Channel_types for details.

For example

    -    def send(pych, obj):
    +    def send(pychan pych, obj):

Even though Cython allows not to annotate self with type, this makes the
code more clear.

This prepares pyselect codebase for future logic where all channel
element types will be sent via inplace _selcase data. For PyObject we
could previously go with "wiring ptx through pycase[1]", but for
arbitrary type, that has to be first converted from Python object to
C-level object, we would have to store the result somewhere, and that
would mean extra allocation and pyselect code complexity increase, even
for cases that don't use anything but chan[object].

So do the preparation and switch pyselect into using inplace tx.

Currently select, via _selcase, requires users to provide pointers to tx
and rx buffers. However if element type itself can fit into a pointer
word, we can put the element directly into _selcase and mark the case
with a flag, that it contains inplace data instead of referring to
external storage. This will be helpful in upcoming patch where we'll
teach pychan to work with several element types, not only pyobject
elements.

This patch does careful introduction of _selcase.flags - in such a way
that the size of _selcase stays the same as it was before by using
bitfields. The .ptxrx pointer is unioned with newly introduced inplace
uint64 .itxrx data, which is used by select instead of .ptxrx if the
flag is set. The usage of uint64 should not increase _selcase size on
64-bit platforms.

Then _selcase.ptx() and .prx() accessors are adapted accordingly and
the rest of the changes are corresponding test and
_chanselect2<onstack=false> adaptation.

This functionality is kind of low-level and is not exposed via any
_selsend() or chan.sends() API changes. Whenever inplace tx should be
used, the case should be prepared either completely manually, or with
e.g. first calling _selsend() and then manually changing .flags and
.itxrx.  Added test serves as the example on how to do it.

Inplace rx is currently forbidden - because supporting that would require
to drop const from casev select argument. However in the future, for
symmetry, we might want to support that as well.

P.S.

Since write to selcase.itxrx requires casting pointers e.g. like this:

	*(int *)&sel[0].itxrx = 12345;

it breaks C99 strict aliasing and by default compiler can generate bad
code on such pattern. To the problem we adapt the build system
to default compiler to no-strict-aliasing (as many other projects do,
e.g. Linux kernel) with the idea that in many cases where strict
aliasing was intended to help it actually does not, because e.g. pointer
types are the same, while explicitly marking pointers with `restrict`
keyword does help indeed.

Nothing new for Python2 here, as it is using -fno-strict-aliasing by
itself. However Python3 is compiling without -fno-strict-aliasing:
https://python.org/dev/peps/pep-3123 .

In the next patch we are going to teach _selcase to support both
external and inplace data. Before that let's do a couple of preparatory
things.

This patch: introduces .ptx() and .prx() accessors to get to
corresponding data buffer associated with _selcase. Convert _selcase
users to use .ptx() and .prx() instead of doing direct .ptxrx access.
This way when we'll add inplace support to _selcase, we'll need to adapt
only accessors, not clients.

The only place that is left using .ptxrx directly is one tricky place in
_chanselect2<onstack=false> where gevent-related code needs to carefully
deal with proxying tx/rx buffers due to STACK_DEAD_WHILE_PARKED.

NOTE even though new accessors may panic, libgolang.cpp always calls them
after checking that the conditions for ptx/prx calls are valid.

In the next patches we are going to teach _selcase to support both
external and inplace data. Before that let's do a couple of preparatory
things.

This patch: rename .data -> .ptxrx . The new name is more clear:

- "p" prefix aligns with other libgolang style, e.g. ptx or prx.
- "txrx" suffix says that this is used for both send and recv.

Just plain renaming, nothing else in this patch.