1. 24 Mar, 2019 1 commit
    • Kirill Smelkov's avatar
      gpython: Workaround PyPy2 vs Gevent patch_thread crash · 704d99f0
      Kirill Smelkov authored
      	pypy-gevent runtests: commands[0] | gpython -m pytest gpython/ golang/
      	Traceback (most recent call last):
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy-gevent/bin/gpython", line 10, in <module>
      	    sys.exit(main())
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy-gevent/site-packages/gpython/__init__.py", line 153, in main
      	    _ = monkey.patch_all()      # XXX sys=True ?
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy-gevent/site-packages/gevent/monkey.py", line 976, in patch_all
      	    patch_thread(Event=Event, _warnings=_warnings)
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy-gevent/site-packages/gevent/monkey.py", line 178, in ignores
      	    return func(*args, **kwargs)
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy-gevent/site-packages/gevent/monkey.py", line 588, in patch_thread
      	    _patch_existing_locks(threading_mod)
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy-gevent/site-packages/gevent/monkey.py", line 492, in _patch_existing_locks
      	    if o._RLock__owner is not None:
      	AttributeError: 'thread.RLock' object has no attribute '_RLock__owner'
      
      It will be fixed on next Gevent release, however until then the crash is there
      and we have to workaround: we can skip patching existing locks as it will be
      the same behaviour of next Gevent release, since its just not possible to patch
      locks created via instantiated C-level classes:
      
      https://github.com/gevent/gevent/commit/d0e04658
      
      We are doing monkey-patching very early, so it should be safe.
      704d99f0
  2. 20 Mar, 2019 1 commit
    • Kirill Smelkov's avatar
      golang: Deflake channel tests · 731f39e3
      Kirill Smelkov authored
      Channel tests are passing on an idle machine. However if there is
      another load - e.g. in the presence of simultaneous Firefox start,
      channel tests break, for example:
      
      	    def test_select():
      	        ...
      
      	        # non-blocking try recv: ok
      	        ch = chan()
      	        done = chan()
      	        def _():
      	            for i in range(N):
      	                ch.send(i)
      	            done.close()
      	        go(_)
      
      	        for i in range(N):
      	            tdelay()
      	            if i % 2:
      	                _, _rx = select(
      	                        ch.recv,
      	                        default,
      	                )
      	                assert (_, _rx) == (0, i)
      	            else:
      	                _, _rx = select(
      	                        ch.recv_,
      	                        default,
      	                )
      	>               assert (_, _rx) == (0, (i, True))
      	E               assert (1, None) == (0, (320, True))
      	E                 At index 0 diff: 1 != 0
      	E                 Use -v to get the full diff
      
      	golang_test.py:209: AssertionError
      
      The failure here is that it was default case selected, not ch.recv_. The
      default was selected because the sending goroutine was not fast to
      enqueue next send before we tried to receive. We were trying to make
      sure that the sender will be enqueued first via adding 1ms time delay
      before trying to receive, but in the presence of concurrent load spikes
      that turns out to be not enough.
      
      We could try to fix the test by increasing the time to sleep in tdelay,
      make the tests more slow and still not 100% reliable. However we can
      change the tests to instead actually wait for the condition that is
      semantically required: a sender enqueued on the channel.
      
      Do that everywhere where tdelay was used.
      
      Now tests are faster (it was ~3s total, now it is ~ 0.5s total) and pass
      irregardless of whether the machine is idle or otherwise loaded.
      731f39e3
  3. 19 Mar, 2019 3 commits
    • Kirill Smelkov's avatar
      tox -= Python3.6 · 469f21a9
      Kirill Smelkov authored
      Debian dropped Python3.6 from python3-distutils. Without it I'm no
      longer being able to run tests on that python version:
      
      	(neo) (z-dev) (g.env) kirr@deco:~/src/tools/go/pygolang$ tox -e py36-thread
      	execution failed:  -- Traceback (most recent call last):
      	  File "<string>", line 1, in <module>
      	ModuleNotFoundError: No module named 'distutils.sysconfig'
      
      	GLOB sdist-make: /home/kirr/src/tools/go/pygolang/setup.py
      	py36-thread create: /home/kirr/src/tools/go/pygolang/.tox/py36-thread
      	ERROR: invocation failed (exit code 1), logfile: /home/kirr/src/tools/go/pygolang/.tox/py36-thread/log/py36-thread-0.log
      	ERROR: actionid: py36-thread
      	msg: getenv
      	cmdargs: '/usr/bin/python3 -m virtualenv --python /usr/bin/python3.6 py36-thread'
      
      	Traceback (most recent call last):
      	  File "/usr/lib/python3/dist-packages/virtualenv.py", line 25, in <module>
      	    import distutils.sysconfig
      	ModuleNotFoundError: No module named 'distutils.sysconfig'
      	Running virtualenv with interpreter /usr/bin/python3.6
      
      	ERROR: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python3 -m virtualenv --python /usr/bin/python3.6 py36-thread (see /home/kirr/src/tools/go/pygolang/.tox/py36-thread/log/py36-thread-0.log)', 1)
      
      https://bugs.debian.org/918881#19
      469f21a9
    • Kirill Smelkov's avatar
      gpython: Don't check for time not being pre-imported on PyPy · e847c550
      Kirill Smelkov authored
      PyPy7 always pre-imports the time module. Without the change tests were
      failing:
      
      	Traceback (most recent call last):
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy3-gevent/bin/gpython", line 10, in <module>
      	    sys.exit(main())
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy3-gevent/site-packages/gpython/__init__.py", line 145, in main
      	    '\n\n\t%s\n\nsys.modules:\n\n\t%s' % (bad, sysmodv))
      	RuntimeError: gpython: internal error: the following modules are pre-imported, but must be not:
      
      	        ['time']
      
      	sys.modules:
      
      	        ['__future__', '__main__', '__pypy__', '__pypy__._pypydatetime', '__pypy__.builders', '__pypy__.intop', '__pypy__.os', '__pypy__.thread', '__pypy__.time', '_ast', '_bootlocale', '_codecs', '_collections', '_collections_abc', '_continuation', '_csv', '_frozen_importlib', '_frozen_importlib_external', '_imp', '_io', '_locale', '_multibytecodec', '_operator', '_rawffi', '_rawffi.alt', '_signal', '_sre', '_structseq', '_thread', '_warnings', '_weakref', '_weakrefset', 'abc', 'array', 'builtins', 'codecs', 'copyreg', 'encodings', 'encodings.aliases', 'encodings.ascii', 'encodings.latin_1', 'encodings.utf_8', 'errno', 'gc', 'genericpath', 'gpython', 'marshal', 'os', 'os.path', 'posix', 'posixpath', 'pwd', 're', 'site', 'sre_compile', 'sre_constants', 'sre_parse', 'stat', 'sys', 'time', 'unicodedata']
      e847c550
    • Kirill Smelkov's avatar
      tox += PyPy3 · da68a8ae
      Kirill Smelkov authored
      Debian started to ship PyPy3:
      
      	https://packages.debian.org/search?keywords=pypy3
      da68a8ae
  4. 15 Mar, 2019 1 commit
    • Kirill Smelkov's avatar
      golang: Deprecate @method(cls) in favour of @func(cls) · 942ee900
      Kirill Smelkov authored
      Since we already have func (see 5146eb0b "Add support for defer &
      recover") we can use @func for both plain functions and for methods.
      
      For example instead of
      
      	@func
      	def my_function(...):
      	    ...
      
      	@method(MyClass)			<-- NOTE
      	def my_method(self, ...):
      	    ...
      
      have it as
      
      	@func
      	def my_function(...):
      	    ...
      
      	@func(MyClass)				<-- NOTE
      	def my_method(self, ...):
      	    ...
      
      which looks more similar to Go and exposes less golang keywords to a user.
      942ee900
  5. 13 Mar, 2019 1 commit
    • Kirill Smelkov's avatar
      gpython: Fix pymain to properly setup sys.path[0] · 6b4990f6
      Kirill Smelkov authored
      Pymain was not adding e.g. directory of executed file to sys.path, and
      as the result if there were e.g. 2 files
      
      	dir/hello.py	# imports world
      	dir/world.py
      
      running `gpython dir/hello.py` would fail to import world.
      
      The case for interactive console was also failing to setup sys.argv as
      empty, so it was containing ['/path/to/gpython']. It was also, contrary
      to standard python, unconditionally showing '>>>' prompt even when stdin
      was not a tty.
      
      Fix all that and add a test to cover pymain functionality.
      6b4990f6
  6. 16 Jan, 2019 6 commits
  7. 30 Dec, 2018 1 commit
  8. 13 Dec, 2018 4 commits
    • Kirill Smelkov's avatar
      pygolang v0.0.0.dev6 · 1ebbbfee
      Kirill Smelkov authored
      1ebbbfee
    • Kirill Smelkov's avatar
      strconv += unquote(), unquote_next() · ed6b7895
      Kirill Smelkov authored
      This are functions to decode quotation that was produced by
      strconv.quote().
      ed6b7895
    • Kirill Smelkov's avatar
      strconv: New package + expose .quote() there · f09701b0
      Kirill Smelkov authored
      Move quote implementation from gcompat to strconv. Make quote work on both
      unicode|bytes string input and produce the same output type. Preserve qq
      (still remaining in gcompat) to always produce str, since that is used
      in prints.
      f09701b0
    • Kirill Smelkov's avatar
      readme: Overhaul · 94bae8e8
      Kirill Smelkov authored
      Stop hiding qq and benchmarking bits in changelog only - add their
      description into additional sections in readme.
      
      Since py.bench description is now in readme, there is no need to keep
      py.bench example in the changelog.
      94bae8e8
  9. 31 Oct, 2018 1 commit
  10. 30 Oct, 2018 6 commits
    • Kirill Smelkov's avatar
      pygolang v0.0.0.dev5 · 8424abe0
      Kirill Smelkov authored
      8424abe0
    • Kirill Smelkov's avatar
      Add support for defer & recover · 5146eb0b
      Kirill Smelkov authored
      `defer` allows to schedule a cleanup to be executed when current function
      returns. It is similar to `try`/`finally` but does not force the cleanup part
      to be far away in the end. For example::
      
         wc = wcfs.join(zurl)    │     wc = wcfs.join(zurl)
         defer(wc.close)         │     try:
                                 │        ...
         ...                     │        ...
         ...                     │        ...
         ...                     │     finally:
                                 │        wc.close()
      
      For completeness there is `recover` and `panic` that allow to program with
      Go-style error handling, for example::
      
         def _():
            r = recover()
            if r is not None:
               print("recovered. error was: %s" % (r,))
         defer(_)
      
         ...
      
         panic("aaa")
      
      But `recover` and `panic` are probably of less utility since they can be
      practically natively modelled with `try`/`except`.
      
      If `defer` is used, the function that uses it must be wrapped with `@func` or
      `@method` decorators.
      
      The implementation is partly inspired by work of Denis Kolodin:
      
      - https://habr.com/post/191786
      - https://stackoverflow.com/a/43028386/9456786
      5146eb0b
    • Kirill Smelkov's avatar
      select: Don't let both a queued and a tried cases win at the same time · f0b592b4
      Kirill Smelkov authored
      While the second phase of select is running we queue send/recv cases to
      corresponding channels. At some point - when some of the cases are
      already queued - a peer goroutine might try to send/recv on that
      channel. And it will succeed because a waiter was queued to the channel.
      
      At the same time select is continuing its enqueue loop and before enqueuing
      to a channel it tries to send/recv there. If that channel became just ready
      (i.e. just after select poll phase) the try to send/recv will succeed. This
      means that actually 2 select cases could be executed at the same time.
      
      Fix it by carefully checking whether some case already won before trying
      to send/recv on a channel.
      
      This fixes the test failures that were demonstrated by previous 2 patches.
      f0b592b4
    • Kirill Smelkov's avatar
      select: Another select-select test which reuses channels during iterations · 2fc6797c
      Kirill Smelkov authored
      Previous select-select test uses 2 channels and recreats them every
      iteration. However the bug we just saw in b51b8d5d (select: Run tests
      more thoroughly) is of parasitic nature - where misbehaviour depends on
      what state has been left there from previous select and whether the race
      to get to that state fast enough succeeded.
      
      So add a more secialized test which tries to trigger the parasitic
      effects that depend on previous state left by select on the channels.
      
      The test, similarly to b51b8d5d, currently fails.
      2fc6797c
    • Kirill Smelkov's avatar
      select: Run tests more thoroughly · b51b8d5d
      Kirill Smelkov authored
      With more iterations it triggers a bug:
      
      	kirr@deco:~/src/tools/go/pygolang$ py.test -vsx
      	============================================== test session starts ===============================================
      	platform linux2 -- Python 2.7.15+, pytest-3.6.4, py-1.6.0, pluggy-0.6.0 -- /usr/bin/python2
      	cachedir: .pytest_cache
      	rootdir: /home/kirr/src/tools/go/pygolang, inifile:
      	collected 7 items
      
      	golang/_gopath_test.py::test_import_module PASSED
      	golang/_gopath_test.py::test_import_package PASSED
      	golang/gcompat_test.py::test_qq PASSED
      	golang/golang_test.py::test_go PASSED
      	golang/golang_test.py::test_chan PASSED
      	golang/golang_test.py::test_select Exception in thread Thread-117:
      	Traceback (most recent call last):
      	  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
      	    self.run()
      	  File "/usr/lib/python2.7/threading.py", line 754, in run
      	    self.__target(*self.__args, **self.__kwargs)
      	  File "/home/kirr/src/tools/go/pygolang/golang/golang_test.py", line 320, in _
      	    assert (_, _rx) == (1, 'b')
      	AssertionError: assert (1, 'xxx1') == (1, 'b')
      	  At index 1 diff: 'xxx1' != 'b'
      	  Full diff:
      	  - (1, 'xxx1')
      	  + (1, 'b')
      b51b8d5d
    • Kirill Smelkov's avatar
      go: Fix test_go when running in-tree without pygolang being installed · ec929991
      Kirill Smelkov authored
      This test was failing when pygolang was not dev-mode installed (`pip install -e .`):
      
      	kirr@deco:~/src/tools/go/pygolang$ py.test
      	=========================================== test session starts ============================================
      	platform linux2 -- Python 2.7.15+, pytest-3.6.4, py-1.6.0, pluggy-0.6.0
      	rootdir: /home/kirr/src/tools/go/pygolang, inifile:
      	collected 8 items
      
      	golang/_gopath_test.py ..                                                                            [ 25%]
      	golang/gcompat_test.py .                                                                             [ 37%]
      	golang/golang_test.py F....                                                                          [100%]
      
      	================================================= FAILURES =================================================
      	_________________________________________________ test_go __________________________________________________
      
      	    def test_go():
      	        # leaked goroutine behaviour check: done in separate process because we need
      	        # to test process termination exit there.
      	        subprocess.check_call([sys.executable,
      	>           dirname(__file__) + "/golang_test_goleaked.py"])
      
      	golang/golang_test.py:38:
      	_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      
      	popenargs = (['/usr/bin/python2', '/home/kirr/src/tools/go/pygolang/golang/golang_test_goleaked.py'],)
      	kwargs = {}, retcode = 1
      	cmd = ['/usr/bin/python2', '/home/kirr/src/tools/go/pygolang/golang/golang_test_goleaked.py']
      
      	    def check_call(*popenargs, **kwargs):
      	        """Run command with arguments.  Wait for command to complete.  If
      	        the exit code was zero then return, otherwise raise
      	        CalledProcessError.  The CalledProcessError object will have the
      	        return code in the returncode attribute.
      
      	        The arguments are the same as for the Popen constructor.  Example:
      
      	        check_call(["ls", "-l"])
      	        """
      	        retcode = call(*popenargs, **kwargs)
      	        if retcode:
      	            cmd = kwargs.get("args")
      	            if cmd is None:
      	                cmd = popenargs[0]
      	>           raise CalledProcessError(retcode, cmd)
      	E           CalledProcessError: Command '['/usr/bin/python2', '/home/kirr/src/tools/go/pygolang/golang/golang_test_goleaked.py']' returned non-zero exit status 1
      
      	/usr/lib/python2.7/subprocess.py:190: CalledProcessError
      	------------------------------------------- Captured stderr call -------------------------------------------
      	Traceback (most recent call last):
      	  File "/home/kirr/src/tools/go/pygolang/golang/golang_test_goleaked.py", line 23, in <module>
      	    from golang import go, chan
      	ImportError: No module named golang
      	==================================== 1 failed, 7 passed in 0.10 seconds ====================================
      
      Fix it by injecting top-level pygolang/ into $PYTHONPATH when testing
      via external processes.
      
      Fixes 69cef96e (go: Don't allow leaked goroutines to prevent program to exit)
      ec929991
  11. 25 Oct, 2018 1 commit
    • Kirill Smelkov's avatar
      @method: Fix it for Python3 · ab69e0fa
      Kirill Smelkov authored
      There was no test for @method so far and that's why it went unnoticed.
      But on Python3 it breaks on f.func_name:
      
      	In [3]: def f(): pass
      
      	In [4]: f.func_name
      	---------------------------------------------------------------------------
      	AttributeError                            Traceback (most recent call last)
      	<ipython-input-4-662dcbac1531> in <module>()
      	----> 1 f.func_name
      
      	AttributeError: 'function' object has no attribute 'func_name'
      
      Fix it by using f.__name__ which works on both py2 and py3.
      
      Add test for @method to make sure it doesn't break unnoticed.
      ab69e0fa
  12. 24 Oct, 2018 1 commit
    • Kirill Smelkov's avatar
      go: Don't allow leaked goroutines to prevent program to exit · 69cef96e
      Kirill Smelkov authored
      This is the Go behaviour, as demonstratd by the following program:
      
      ---- 8< ----
      package main
      
      import (
      	"fmt"
      	"time"
      )
      
      func work(w int) {
      	for i := 0; ; i++ {
      		fmt.Printf("w%d: %d\n", w, i)
      		time.Sleep(1*time.Second)
      	}
      }
      
      func main() {
      	for i := 0; i < 100; i++ {
      		go work(i)
      	}
      
      	time.Sleep(3*time.Second)
      	println("main: exit")
      }
      ---- 8< ----
      69cef96e
  13. 04 Jul, 2018 2 commits
    • Kirill Smelkov's avatar
      pygolang v0.0.0.dev4 · f2905909
      Kirill Smelkov authored
      f2905909
    • Kirill Smelkov's avatar
      py.bench: New command to benchmark python code similarly to `go test -bench` · 9bf03d9c
      Kirill Smelkov authored
      + corresponding bits in golang.testing package.
      
      py.bench description follows:
      
      """
      py.bench, similarly to py.test, discovers bench_* functions and Bench* classes
      and then runs each discovered benchmark via golang.testing.benchmark. Similarly
      to `go test -bench`, benchmarking results are printed in Go benchmark format.
      
      For example, running py.bench on the following code::
      
          def bench_add(b):
              x, y = 1, 2
              for i in xrange(b.N):
                  x + y
      
      gives something like::
      
          $ py.bench --count=3 x.py
          ...
          pymod: bench_add.py
          Benchmarkadd    50000000        0.020 µs/op
          Benchmarkadd    50000000        0.020 µs/op
          Benchmarkadd    50000000        0.020 µs/op
      """
      
      The implementation is based on t/py.bench from Wendelin.core - see
      following commits for its history:
      
      lab.nexedi.com/nexedi/wendelin.core/commit/51f252d4
      lab.nexedi.com/nexedi/wendelin.core/commit/074ce24d
      lab.nexedi.com/nexedi/wendelin.core/commit/ed13c3f9
      lab.nexedi.com/nexedi/wendelin.core/commit/fc08766d
      lab.nexedi.com/nexedi/wendelin.core/commit/5a1ed45a
      lab.nexedi.com/nexedi/wendelin.core/commit/bcab1246
      9bf03d9c
  14. 02 Jul, 2018 4 commits
  15. 20 Jun, 2018 2 commits
    • Kirill Smelkov's avatar
      golang/x/perf/benchlib: New module to load & work with data in Go benchmark format · 812e7ed7
      Kirill Smelkov authored
      Go benchmark format is described here:
      
      	https://github.com/golang/proposal/blob/master/design/14313-benchmark-format.md
      
      Additionally we support extension to that format:
      
      	- a line starting with `*** neotest:` denotes start of neotest extension block.
      	  The block consists of labels describing e.g. hardware and software on a node.
      	  The block ends with a blank line.
      	  Labels in the block are not added to benchmarking lines from main stream.
      	  The block itself should not contain benchmark lines.
      
      and upon processing benchmark units are normalized to common base,
      similarly to
      
      	golang.org/cl/82955
      	golang.org/cl/82956
      	golang.org/cl/82957
      
      Orignally implemented here:
      
      	kirr/neo@502d9477
      	kirr/neo@a9b10a45
      	kirr/neo@f5fec740
      	kirr/neo@916782b6
      812e7ed7
    • Kirill Smelkov's avatar
      Turn pygopath into full pygolang · afa46cf5
      Kirill Smelkov authored
      Not only we can import modules by full path, but now we can also spawn
      threads/coroutines and exchange data in between them with the same
      primitives and semantic as in Go.
      
      The bulk of new functionality is copied from here:
      
      	kirr/go123@9e1aa6ab
      
      Original commit description follows:
      
      """
      golang: New _Python_ package to provide Go-like features to Python language
      - `go` spawns lightweight thread.
      - `chan` and `select` provide channels with Go semantic.
      - `method` allows to define methods separate from class.
      - `gimport` allows to import python modules by full path in a Go workspace.
      
      The focus of first draft was on usage interface and on correctness, not speed.
      In particular select should be fully working.
      
      If there is a chance I will maybe try to followup with gevent-based
      implementation in the future.
      Hide whitespace changes
      """
      afa46cf5
  16. 21 May, 2018 2 commits
    • Kirill Smelkov's avatar
      pygopath: Initial draft · 9c61f254
      Kirill Smelkov authored
      Module gopath provides way to import python modules by full path in a Go workspace.
      
      For example
      
          lonet = gopath.gimport('lab.nexedi.com/kirr/go123/xnet/lonet')
      
      will import either
      
          lab.nexedi.com/kirr/go123/xnet/lonet.py, or
          lab.nexedi.com/kirr/go123/xnet/lonet/__init__.py
      
      located somewhere under $GOPATH.
      9c61f254
    • Kirill Smelkov's avatar
      Start of pygopath.git · 6d47012e
      Kirill Smelkov authored
      A simple project to allow python imports to be done by full path in a Go
      workspace.
      6d47012e