Skip to content

pygolang
pygolang
Commit 4dc1a7f0 authored by Kirill Smelkov's avatar Kirill Smelkov
3 Browse files
Options

tox += ThreadSanitizer, AddressSanitizer, Python debug builds 

- ThreadSanitizer helps to detect races and some memory errors,
- AddressSanitizer helps to detect memory errors,
- Python debug builds help to detect e.g reference counting errors.

Adding all those tools to testing coverage discovers e.g. the following
bugs (not a full list):

---- 8< ----

py27-thread-tsan:
WARNING: ThreadSanitizer: data race (pid=7143)
  Write of size 8 at 0x7b1400000650 by main thread:
    #0 free ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:649 (libtsan.so.0+0x2b46a)
    #1 free ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:643 (libtsan.so.0+0x2b46a)
    #2 golang::_chan::decref() golang/runtime/libgolang.cpp:470 (liblibgolang.so.0.1+0x47f2)
    #3 _chanxdecref golang/runtime/libgolang.cpp:452 (liblibgolang.so.0.1+0x484a)
    #4 _test_go_c golang/runtime/libgolang_test_c.c:86 (_golang_test.so+0x13a2e)
    #5 __pyx_pf_6golang_12_golang_test_12test_go_c golang/_golang_test.cpp:3340 (_golang_test.so+0xcbaa)
    #6 __pyx_pw_6golang_12_golang_test_13test_go_c golang/_golang_test.cpp:3305 (_golang_test.so+0xcbaa)
    #7 PyEval_EvalFrameEx <null> (python2.7+0xf68b4)

  Previous read of size 8 at 0x7b1400000650 by thread T8:
    #0 golang::Sema::acquire() golang/runtime/libgolang.cpp:164 (liblibgolang.so.0.1+0x410a)
    #1 golang::Mutex::lock() golang/runtime/libgolang.cpp:175 (liblibgolang.so.0.1+0x4c82)
    #2 golang::_chan::close() golang/runtime/libgolang.cpp:754 (liblibgolang.so.0.1+0x4c82)
    #3 _chanclose golang/runtime/libgolang.cpp:732 (liblibgolang.so.0.1+0x4d1a)
    #4 _work golang/runtime/libgolang_test_c.c:92 (_golang_test.so+0x136cc)
    #5 <null> <null> (python2.7+0x1929e3)

  Thread T8 (tid=7311, finished) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:915 (libtsan.so.0+0x2be1b)
    #1 PyThread_start_new_thread <null> (python2.7+0x19299f)
    #2 _taskgo golang/runtime/libgolang.cpp:119 (liblibgolang.so.0.1+0x3f68)
    #3 _test_go_c golang/runtime/libgolang_test_c.c:84 (_golang_test.so+0x13a1c)
    #4 __pyx_pf_6golang_12_golang_test_12test_go_c golang/_golang_test.cpp:3340 (_golang_test.so+0xcbaa)
    #5 __pyx_pw_6golang_12_golang_test_13test_go_c golang/_golang_test.cpp:3305 (_golang_test.so+0xcbaa)
    #6 PyEval_EvalFrameEx <null> (python2.7+0xf68b4)

py37-thread-asan:
==22205==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000002cd0 at pc 0x7fd3732a7679 bp 0x7fd3723c8c50 sp 0x7fd3723c8c48
READ of size 8 at 0x607000002cd0 thread T7
    #0 0x7fd3732a7678 in golang::Sema::acquire() golang/runtime/libgolang.cpp:164
    #1 0x7fd3732a8644 in golang::Mutex::lock() golang/runtime/libgolang.cpp:175
    #2 0x7fd3732a8644 in golang::_chan::close() golang/runtime/libgolang.cpp:754
    #3 0x7fd3724004b2 in golang::chan<golang::structZ>::close() const golang/libgolang.h:323
    #4 0x7fd3724004b2 in operator() golang/runtime/libgolang_test.cpp:262
    #5 0x7fd3724004b2 in __invoke_impl<void, _test_chan_vs_stackdeadwhileparked()::<lambda()>&> /usr/include/c++/8/bits/invoke.h:60
    #6 0x7fd3724004b2 in __invoke<_test_chan_vs_stackdeadwhileparked()::<lambda()>&> /usr/include/c++/8/bits/invoke.h:95
    #7 0x7fd3724004b2 in __call<void> /usr/include/c++/8/functional:400
    #8 0x7fd3724004b2 in operator()<> /usr/include/c++/8/functional:484
    #9 0x7fd3724004b2 in _M_invoke /usr/include/c++/8/bits/std_function.h:297
    #10 0x7fd3723fdc6e in std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687
    #11 0x7fd3723fdc6e in operator() golang/libgolang.h:273
    #12 0x7fd3723fdc6e in _FUN golang/libgolang.h:271
    #13 0x62ddf3  (/home/kirr/src/tools/go/pygolang-master/.tox/py37-thread-asan/bin/python3+0x62ddf3)
    #14 0x7fd377393fa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
    #15 0x7fd376eda4ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce)

0x607000002cd0 is located 16 bytes inside of 72-byte region [0x607000002cc0,0x607000002d08)
freed by thread T0 here:
    #0 0x7fd377519fb0 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8fb0)
    #1 0x7fd372401335 in golang::chan<golang::structZ>::~chan() golang/libgolang.h:292
    #2 0x7fd372401335 in _test_chan_vs_stackdeadwhileparked() golang/runtime/libgolang_test.cpp:222

previously allocated by thread T0 here:
    #0 0x7fd37751a518 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9518)
    #1 0x7fd3732a7d0b in zalloc golang/runtime/libgolang.cpp:1185
    #2 0x7fd3732a7d0b in _makechan golang/runtime/libgolang.cpp:413

Thread T7 created by T0 here:
    #0 0x7fd377481db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
    #1 0x62df39 in PyThread_start_new_thread (/home/kirr/src/tools/go/pygolang-master/.tox/py37-thread-asan/bin/python3+0x62df39)

SUMMARY: AddressSanitizer: heap-use-after-free golang/runtime/libgolang.cpp:164 in golang::Sema::acquire()
Shadow bytes around the buggy address:
  0x0c0e7fff8540: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
  0x0c0e7fff8550: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c0e7fff8560: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fd fd
  0x0c0e7fff8570: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
  0x0c0e7fff8580: fd fd fd fd fd fa fa fa fa fa fd fd fd fd fd fd
=>0x0c0e7fff8590: fd fd fd fa fa fa fa fa fd fd[fd]fd fd fd fd fd
  0x0c0e7fff85a0: fd fa fa fa fa fa 00 00 00 00 00 00 00 00 00 fa
  0x0c0e7fff85b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff85c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff85d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff85e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb

---- 8< ----

The bugs will be addressed in the followup patches.
parent 78e38690
Hide whitespace changes
Inline Side-by-side
Showing with 219 additions and 16 deletions
MANIFEST.in
View file @ 4dc1a7f0
include COPYING README.rst CHANGELOG.rst tox.ini pyproject.toml
include COPYING README.rst CHANGELOG.rst tox.ini pyproject.toml trun
include golang/libgolang.h
include golang/runtime/libgolang.cpp
recursive-include golang *.py *.pxd *.pyx *.toml
......
setup.py
View file @ 4dc1a7f0
......@@ -28,23 +28,19 @@ from setuptools.command.develop import develop as _develop
from os.path import dirname, join
import sys, re
# read file content
def readfile(path):
with open(path, 'r') as f:
return f.read()
# reuse golang.pyx.build to build pygolang extensions.
# we have to be careful and inject synthetic golang package in order to be
# able to import golang.pyx.build without built/working golang.
import imp, pkgutil
golang = imp.new_module('golang')
golang.__package__ = 'golang'
golang.__path__ = ['golang']
golang.__file__ = 'golang/__init__.py'
golang.__loader__ = pkgutil.ImpLoader('golang', None, 'golang/__init__.py',
[None, None, imp.PY_SOURCE])
sys.modules['golang'] = golang
trun = {}
exec(readfile('trun'), trun)
trun['ximport_empty_golangmod']()
from golang.pyx.build import setup, Extension as Ext
# read file content
def readfile(path):
with open(path, 'r') as f:
return f.read()
# grep searches text for pattern.
# return re.Match object or raises if pattern was not found.
......
tox.ini
View file @ 4dc1a7f0
[tox]
envlist = {py27,py36,py37,pypy,pypy3}-{thread,gevent}
envlist =
{ py27,py36,py37d,py37,pypy,pypy3}-{thread,gevent}
# XXX gevent fails to install with python2-dbg (https://github.com/gevent/gevent/issues/1461)
{py27d }-{thread }
# ThreadSanitizer
# XXX under pypy tsan does not observe the GIL lock/release(*), and this
# way reports even plain T1.Py_INCREF / T2.Py_DECREF as a race.
# (*) PyPy locks its GIL (see RPyGilAcquire) by manually doing atomic cmpxchg
# and other games, which TSAN cannot see if PyPy itself was not compiled with
# -fsanitize=thread.
{ py27,py36,py37d,py37 }-{thread }-tsan
# XXX see ^^^ about gevent vs python2-dbg
{py27d }-{thread }-tsan
# XXX py*-gevent-tsan would be nice to have, but at present TSAN is not
# effective with gevent, because it does not understand greenlet "thread"
# switching and so perceives the program as having only one thread where races
# are impossible. Disabled to save time.
# { py27,py36,py37d,py37 }-{ gevent}-tsan
# AddressSanitizer
# XXX asan does not work with gevent: https://github.com/python-greenlet/greenlet/issues/113
{py27d,py27,py36,py37d,py37,pypy,pypy3}-{thread }-asan
[testenv]
basepython =
py27d: python2.7-dbg
py27: python2.7
py36: python3.6
py37d: python3.7-dbg
py37: python3.7
pypy: pypy
pypy3: pypy3
setenv =
# distutils take CFLAGS for both C and C++.
# distutils use CFLAGS also at link stage -> we don't need to set LDFLAGS separately.
tsan: CFLAGS=-g -fsanitize=thread
asan: CFLAGS=-g -fsanitize=address
# XXX however distutils' try_link, which is used by numpy.distutils use only CC
# as linker without CFLAGS and _without_ LDFLAGS, which fails if *.o were
# compiled with -fsanitize=X and linked without that option. Work it around
# with also adjusting CC.
# XXX better arrange to pass CFLAGS to pygolang only, e.g. by adding --race or
# --sanitize=thread to `setup.py build_ext`.
tsan: CC=cc -fsanitize=thread
asan: CC=cc -fsanitize=address
# always compile pygolang from source and don't reuse binary pygolang wheels as
# we compile each case with different CFLAGS.
install_command =
python -m pip install --no-binary pygolang {opts} {packages}
deps =
.[all]
......@@ -10,5 +64,12 @@ deps =
changedir = {envsitepackagesdir}
commands=
thread: {envpython} -m pytest gpython/ golang/
gevent: gpython -m pytest gpython/ golang/
{toxinidir}/trun \
thread: {envpython} \
gevent: gpython \
-m pytest \
# asan/tsan: tell pytest not to capture output - else it is not possible to see
# reports from sanitizers because they crash tested process on error.
# likewise for python debug builds.
asan,tsan,py{27,37}d: -s \
gpython/ golang/
trun 0 → 100755
View file @ 4dc1a7f0
#!/usr/bin/env python
# Copyright (C) 2019 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your
# option) any later version, as published by the Free Software Foundation.
#
# You can also Link and Combine this program with other software covered by
# the terms of any of the Free Software licenses or any of the Open Source
# Initiative approved licenses and Convey the resulting work. Corresponding
# source of such a combination shall include the source code for all other
# software used.
#
# This program is distributed WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See COPYING file for full licensing terms.
# See https://www.nexedi.com/licensing for rationale and options.
""" `trun ...` - run `...` while testing pygolang
For example it is not possible to import sanitized libgolang.so if non-sanitized
python was used to start tests - it will fail with
ImportError: /usr/lib/x86_64-linux-gnu/libtsan.so.0: cannot allocate memory in static TLS block
trun cares to run python with LD_PRELOAD set appropriately to /path/to/libtsan.so
"""
# TODO integrate startup code into e.g. `gpython -race` which should arrange
# for _golang.xxx-race.so to be loaded instead of _golang.xxx.so, and also
# arrange for initial $LD_PRELOAD. Always build both _golang.xxx.so and
# _golang.xxx-race.so as part of standard pygolang build.
from __future__ import print_function, absolute_import
import os, sys, re, subprocess, imp, pkgutil
PY3 = (bytes is not str)
# env_prepend prepends value to ${name} environment variable.
#
# the value is prepended with " " separator.
# the value is prepended - not appended - so that defaults set by trun can be overridden by user.
def env_prepend(name, value):
_ = os.environ.get(name, "")
if _ != "":
value += " " + _
os.environ[name] = value
# grep1 searches for the first line, that matches pattern, from text.
def grep1(pattern, text): # -> re.Match|None
p = re.compile(pattern)
for l in text.splitlines():
m = p.search(l)
if m is not None:
return m
return None
# ximport_empty_golangmod injects synthetic golang package in order to be able
# to import e.g. golang.pyx.build, or locate golang._golang, without built/working golang.
def ximport_empty_golangmod():
assert 'golang' not in sys.modules
golang = imp.new_module('golang')
golang.__package__ = 'golang'
golang.__path__ = ['golang']
golang.__file__ = 'golang/__init__.py'
golang.__loader__ = pkgutil.ImpLoader('golang', None, 'golang/__init__.py',
[None, None, imp.PY_SOURCE])
sys.modules['golang'] = golang
def main():
# install synthetic golang package so that we can import golang.X even if
# golang._golang (which is imported by golang/__init__.py) is not built/functional.
# Use golang.pyx.build._findpkg to locate _golang.so that corresponds to our python.
ximport_empty_golangmod()
from golang.pyx import build
_golang_so = build._findpkg('golang._golang')
# determine if _golang.so is linked to a sanitizer, and if yes, to which
# particular sanitizer DSO. Set LD_PRELOAD appropriately.
ld_preload = None
if 'linux' in sys.platform:
p = subprocess.Popen(["ldd", _golang_so.path], stdout=subprocess.PIPE)
out, _ = p.communicate()
if PY3:
out = out.decode('utf-8')
_ = grep1(r"lib.san\.so\.. => ([^\s]+)", out)
if _ is not None:
ld_preload = ("LD_PRELOAD", _.group(1))
elif 'darwin' in sys.platform:
# on darwin there is no ready out-of-the box analog of ldd, but
# sanitizer runtimes print instruction what to preload, e.g.
# ==973==ERROR: Interceptors are not working. This may be because ThreadSanitizer is loaded too late (e.g. via dlopen). Please launch the executable with:
# DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/10.0.1/lib/darwin/libclang_rt.tsan_osx_dynamic.dylib
# "interceptors not installed" && 0./test.sh: line 6: 973 Abort trap: 6 ./trun python -m pytest "$@"
# try to `import golang` to retrieve that.
p = subprocess.Popen(["python", "-c", "import golang"], stderr=subprocess.PIPE)
_, err = p.communicate()
if p.returncode != 0:
if PY3:
err = err.decode('utf-8')
_ = grep1("DYLD_INSERT_LIBRARIES=(.*)$", err)
if _ is not None:
ld_preload = ("DYLD_INSERT_LIBRARIES", _.group(1))
else:
print("trun %r: `import golang` failed with unexpected error:" % sys.argv[1:], file=sys.stderr)
print(err, file=sys.stderr)
sys.exit(2)
# ld_preload has e.g. ("LD_PRELOAD", "/usr/lib/x86_64-linux-gnu/libtsan.so.0")
if ld_preload is not None:
#print('env <-', ld_preload)
env_prepend(*ld_preload)
# $LD_PRELOAD setup; ready to exec `...`
# if TSAN/ASAN detects a bug - make it fail loudly on the first bug
env_prepend("TSAN_OPTIONS", "halt_on_error=1")
env_prepend("ASAN_OPTIONS", "halt_on_error=1")
# tweak TSAN/ASAN defaults:
# enable TSAN deadlock detector
# (unfortunately it caughts only few _potential_ deadlocks and actually
# gets stuck on any real deadlock)
env_prepend("TSAN_OPTIONS", "detect_deadlocks=1")
env_prepend("TSAN_OPTIONS", "second_deadlock_stack=1")
# many python allocations, whose lifetime coincides with python interpreter
# lifetime and which are not explicitly freed on python shutdown, are
# reported as leaks. Disable leak reporting to avoid huge non-pygolang
# related printouts.
env_prepend("ASAN_OPTIONS", "detect_leaks=0")
# tune ASAN to check more aggressively by default
env_prepend("ASAN_OPTIONS", "detect_stack_use_after_return=1")
# exec `...`
os.execvp(sys.argv[1], sys.argv[1:])
if __name__ == '__main__':
main()
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7