1. 11 May, 2018 2 commits
    • Kirill Smelkov's avatar
      tracing/runtime: Teach g_typedef to extract type definition for non-struct types · 2c2c5cb6
      Kirill Smelkov authored
      With previous sed expression it was failing e.g. on runtime.guintptr,
      giving much more after type definition:
      
      	$ go doc -c -u runtime.muintptr | sed -n -e '/^type /,/^}/p'
      	type muintptr uintptr
      	    muintptr is a *m that is not tracked by the garbage collector.
      
      	    Because we do free Ms, there are some additional constrains on muintptrs:
      
      	    1. Never hold an muintptr locally across a safe point.
      
      	    2. Any muintptr in the heap must be owned by the M itself so it can
      
      	    ensure it is not in use when the last true *m is released.
      
      	func (mp muintptr) ptr() *m
      	func (mp *muintptr) set(m *m)
      
      Fix it since we'll need to extract more single-line types for Go1.11 support.
      
      To verify it works de-stub {g,p,m}uintptr.
      2c2c5cb6
    • Kirill Smelkov's avatar
      tracing/gotrace: Fix tests for Go1.10 where CGo is now supported · b7e99d87
      Kirill Smelkov authored
      Go1.10 now modifies CGo sources as text, not AST, and this way comments
      are not removed and gotrace can see its '//trace:event ' comments in a
      CGo *.go files: https://github.com/golang/go/commit/85c3ebf4.
      
      This way with Go1.10 tests were failing because `//trace:event
      traceHello()` in a/pkg1/pkg1c.go was now noticed by gotrace and more
      then expected trace events produces.
      
      I have not noticed this in 65c399f0 (tracing/runtime: Add support for
      Go1.10 (preliminary)) probably because at that time above Go commit was
      not in my local Go1.10 tree.
      
      For the reference: tracing/runtime support regenerated for Go1.10.2
      stays exactly the same as in 65c399f0 and so does not need updating.
      b7e99d87
  2. 10 May, 2018 2 commits
  3. 25 Apr, 2018 3 commits
    • Kirill Smelkov's avatar
      xnet: Add event corresponding to network dial start · 66edb630
      Kirill Smelkov authored
      We already have TraceConnect which corresponds to event of established
      network connection.
      
      However in many test scenarios it is required to know and pause
      execution right before dial is going to happen. For this introduce
      TraceDial which represents event corresponding to network dialing start.
      66edb630
    • Kirill Smelkov's avatar
      xnet: Networker += Name() · c95f2373
      Kirill Smelkov authored
      Networker.Name should return name of access-point Networker represents
      on the network.
      
      We will need this information in the next patch for tracing dial events,
      when there is no connection established yet, and thus dialer location
      has to be taken from somewhere. It is also generally a good idea for
      Networker to have a name.
      
      For NetPlain networker the name is local hostname.
      c95f2373
    • Kirill Smelkov's avatar
      xnet/pipenet: Add tests for Host.Network · 8c6b8ad0
      Kirill Smelkov authored
      Should be in d3a7a196 (xnet/pipenet: Package for TCP-like synchronous
      in-memory network of net.Pipes)
      8c6b8ad0
  4. 24 Apr, 2018 1 commit
  5. 09 Apr, 2018 1 commit
  6. 15 Mar, 2018 1 commit
  7. 15 Jan, 2018 1 commit
  8. 12 Jan, 2018 2 commits
    • Kirill Smelkov's avatar
      xbufio += SeqReaderAt - buffering wrapper for a io.ReaderAt optimized for sequential access · 3956445e
      Kirill Smelkov authored
      For example in ZODB FileStorage format reader routines are written
      working with io.ReaderAt for the following reasons:
      
      - for loads random-access is required,
      - there can be several concurrent loads in flight simultaneously.
      
      At the same time various database iterations (APIs additional to load)
      use sequential access pattern and can be served by the same record
      reading routines. However with them we cannot use e.g. bufio.Reader
      because it works with plain io.Reader, not io.ReaderAt.
      
      Here comes SeqReaderAt: it adds a buffer, by default 2·4K, on top of
      original io.Reader, automatically detects direction of sequential
      access which can be forward, backward, or interleaved forward-backward
      patterns, and buffers data accordingly to avoid many syscalls e.g. in
      os.File case.
      3956445e
    • Kirill Smelkov's avatar
      xio, xbufio: New packages amending std packages io & bufio · eadf5c4a
      Kirill Smelkov authored
      Currently by direct & buffered readers that also can report current
      offset in input stream.
      
      This functionality is handy to have when for example one needs to report
      an error after finding decoding problem and wants to include particular
      stream position in the report.
      eadf5c4a
  9. 09 Jan, 2018 2 commits
  10. 21 Dec, 2017 4 commits
  11. 27 Oct, 2017 5 commits
    • Kirill Smelkov's avatar
      cmd/gmigrate: Tool to show number of times G migrates to another M (OS thread) · 9195e30a
      Kirill Smelkov authored
      This tool analyzes `go tool trace -d` output and take notices on
      ProcStart and GoStart events to track G->P->M relation to check how
      often a G changes M.
      
      Unfortunately it seems to be a frequent event and changing M probably means
      changing CPU and thus loosing CPU caches.
      9195e30a
    • Kirill Smelkov's avatar
      xnet/pipenet: Package for TCP-like synchronous in-memory network of net.Pipes · d3a7a196
      Kirill Smelkov authored
      This patch adds pipenet - virtual network of net.Pipes.
      
      Addresses on pipenet are host:port pairs. A host is xnet.Networker and so
      can be worked with similarly to regular TCP network with Dial/Listen/Accept/...
      
      Example:
      
          net := pipenet.New("")
          h1 := net.Host("abc")
          h2 := net.Host("def")
      
          l, err := h1.Listen(":10")       // starts listening on address "abc:10"
          go func() {
                  csrv, err := l.Accept()  // csrv will have LocalAddr "abc:10"
          }()
          ccli, err := h2.Dial("abc:10")   // ccli will have RemoteAddr "def:10"
      
      Pipenet might be handy for testing interaction of networked applications in 1
      process without going to OS networking stack.
      d3a7a196
    • Kirill Smelkov's avatar
      xnet: Draft tracing layer that can be applied on top of networker · 621150c1
      Kirill Smelkov authored
      This patch adds xnet.NetTrace which wraps a networker calling
      notification functions on Connect/Listen/Tx events.
      
      The code is draft and I'm not sure adding this functionality is good
      idea, but I still add it for completeness and because there is one user
      for it.
      
      Please do not expect the interface of xnet tracing to be stable.
      621150c1
    • Kirill Smelkov's avatar
      xnet: New package that provides uniform access for working with both TCP and TLS/TCP · 7f62584e
      Kirill Smelkov authored
      Std net.Conn already can represent both plain TCP connections and
      connections wrapped with TLS. However the connections itself need to be
      created differently. This might become inconvenient when establishing
      connections should be inside server logic where it is desirable to have
      one codepath which works uniformly via interfaces.
      
      This patch introduces Networker - new interface which represents
      access-point to a streaming network. One can Dial or Listen on it and
      get underlying network name.
      
      Two functions are also provided to create networkers for plain TCP
      and to wrap a networker with TLS layer.
      
      This way one can initially decide and setup a networker, pass it to
      server logic, and server inside uses just Networker interface
      transparently either listening/connecting via regular sockets, or via
      sockets wrapped with TLS layer.
      7f62584e
    • Kirill Smelkov's avatar
      exc += XFunc(), Funcx() - functional counterparts to XRun and Runx · 546119d3
      Kirill Smelkov authored
      XRun (added in db941f12) and Runx (added in 486ede30) run a function and
      convert error <-> exception back and forth. However sometimes it is
      handy to only convert a function but not run it - e.g. for passing into
      
      	x/sync/errgroup.Group.Go
      
      To do so this patch adds XFunc and Funcx - functional counterparts to
      XRun and Runx.
      
      No new tests are needed because now XRun and Runx are just tiny wrappers
      around new functions and we already have tests for XRun and Runx.
      546119d3
  12. 25 Oct, 2017 5 commits
    • Kirill Smelkov's avatar
      tracing: Part 3 - Silence race-detector about probe.Detach · c0f14991
      Kirill Smelkov authored
      Race-detector does not know Probe.Detach works under world stopped and
      that this way it cannot break consistency of probes list attached to a
      trace event - on event signalling either a probe will be run or not run
      at all.
      
      And we do not mind that e.g. while Detach was in progress a probe was
      read from traceevent list and decided to be run and the probe
      function was actually called just after Detach finished.
      
      For this reason tell race-detector to not take into account all memory
      read/write that are performed while the world is stopped.
      
      If we do not it complains e.g. this way:
      
          ==================
          WARNING: DATA RACE
          Read at 0x00c42000d760 by goroutine 7:
            lab.nexedi.com/kirr/neo/go/zodb/storage._traceCacheGCFinish_run()
                /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/xcommon/tracing/tracing.go:265 +0x81
            lab.nexedi.com/kirr/neo/go/zodb/storage.traceCacheGCFinish()
                /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/ztrace.go:22 +0x63
            lab.nexedi.com/kirr/neo/go/zodb/storage.(*Cache).gc()
                /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/cache.go:497 +0x62c
            lab.nexedi.com/kirr/neo/go/zodb/storage.(*Cache).gcmain()
                /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/cache.go:478 +0x4c
      
          Previous write at 0x00c42000d760 by goroutine 6:
            lab.nexedi.com/kirr/neo/go/xcommon/tracing.(*Probe).Detach()
                /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/xcommon/tracing/tracing.go:319 +0x103
            lab.nexedi.com/kirr/neo/go/xcommon/tracing.(*ProbeGroup).Done()
                /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/xcommon/tracing/tracing.go:344 +0xa5
            lab.nexedi.com/kirr/neo/go/zodb/storage.TestCache()
                /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/cache_test.go:576 +0x7f94
            testing.tRunner()
                /home/kirr/src/tools/go/go/src/testing/testing.go:746 +0x16c
      
          Goroutine 7 (running) created at:
            lab.nexedi.com/kirr/neo/go/zodb/storage.NewCache()
                /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/cache.go:129 +0x227
            lab.nexedi.com/kirr/neo/go/zodb/storage.TestCache()
                /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/cache_test.go:165 +0x7b1
            testing.tRunner()
                /home/kirr/src/tools/go/go/src/testing/testing.go:746 +0x16c
      
          Goroutine 6 (finished) created at:
            testing.(*T).Run()
                /home/kirr/src/tools/go/go/src/testing/testing.go:789 +0x568
            testing.runTests.func1()
                /home/kirr/src/tools/go/go/src/testing/testing.go:1004 +0xa7
            testing.tRunner()
                /home/kirr/src/tools/go/go/src/testing/testing.go:746 +0x16c
            testing.runTests()
                /home/kirr/src/tools/go/go/src/testing/testing.go:1002 +0x521
            testing.(*M).Run()
                /home/kirr/src/tools/go/go/src/testing/testing.go:921 +0x206
            main.main()
                lab.nexedi.com/kirr/neo/go/zodb/storage/_test/_testmain.go:44 +0x1d3
          ==================
      c0f14991
    • Kirill Smelkov's avatar
      tracing: Part 2 - gotrace utility · d89b1be1
      Kirill Smelkov authored
      As it was said in the previous patch here goes gotrace utility to
      process `//trace:event ...` and other tracing related directives.
      
      Related excerpt from the documentation:
      
      ---- 8< ----
      Gotrace
      
      The way //trace:event and //trace:import works is via additional code being
      generated for them. Whenever a package uses any //trace: directive,
      it has to organize to run `gotrace gen` on its sources for them to work,
      usually with the help of //go:generate. For example:
      
      	package hello
      
      	//go:generate gotrace gen .
      
      	//trace:event ...
      
      Besides `gotrace gen` gotrace has other subcommands also related to tracing,
      for example `gotrace list` lists trace events a package provides.
      ---- 8< ----
      
      Gotrace works by parsing and typechecking go sources via go/parse &
      go/types and then for special comments generating corresponding
      additional code that is supported by tracing runtime.
      d89b1be1
    • Kirill Smelkov's avatar
      tracing: Part 1 - runtime · 3cf17be3
      Kirill Smelkov authored
      Package tracing will provide usage and runtime support for Linux-style
      traceevents/tracepoints for Go.
      
      This patch comes with the runtime support to attach/detach probes to/from
      trace-events and stop/restart the world so that attaching/detaching can
      be done safely while nothing else is running. Having attach/detach under
      STW allows regular probe list iteration to be done without locks.
      
      The next patch will add gotrace utility which automatically turns
      //trace:event in-source comments into proper trace-event definitions.
      
      Below is excerpt from tracing usage. Please refer to tracing.go for full
      text.
      
      ---- 8< ----
      Package tracing provides usage and runtime support for Go tracing facilities.
      
      Trace events
      
      A Go package can define several events of interest to trace via special
      comments. With such definition a tracing event becomes associated with trace
      function that is used to signal when the event happens. For example:
      
      	package hello
      
      	//trace:event traceHelloPre(who string)
      	//trace:event traceHello(who string)
      
      	func SayHello(who string) {
      		traceHelloPre(who)
      		fmt.Println("Hello, %s", who)
      		traceHello(who)
      	}
      
      By default trace function does nothing and has very small overhead.
      
      Probes
      
      However it is possible to attach probing functions to events. A probe, once
      attached, is called whenever event is signalled in the context which triggered
      the event and pauses original code execution until the probe is finished. It is
      possible to attach several probing functions to the same event and dynamically
      detach/(re-)attach them at runtime. Attaching/detaching probes must be done
      under tracing.Lock. For example:
      
      	type saidHelloT struct {
      		who  string
      		when time.Time
      	}
      	saidHello := make(chan saidHelloT)
      
      	tracing.Lock()
      	p := traceHello_Attach(nil, func(who string) {
      		saidHello <- saidHelloT{who, time.Now()}
      	})
      	tracing.Unlock()
      
      	go func() {
      		for hello := range saidHello {
      			fmt.Printf("Said hello to %v @ %v\n", hello.who, hello.when)
      		}
      	}()
      
      	SayHello("JP")
      	SayHello("Kirr")
      	SayHello("Varya")
      
      	tracing.Lock()
      	p.Detach()
      	tracing.Unlock()
      
      	close(saidHello)
      
      For convenience it is possible to keep group of attached probes and detach them
      all at once using ProbeGroup:
      
      	pg := &tracing.ProbeGroup{}
      
      	tracing.Lock()
      	traceHelloPre_Attach(pg, func(who string) { ... })
      	traceHello_Attach(pg, func(who string) { ... })
      	tracing.Unlock()
      
      	// some activity
      
      	// when probes needs to be detached (no explicit tracing.Lock needed):
      	pg.Done()
      
      Probes is general mechanism which allows various kinds of trace events usage.
      Three ways particularly are well-understood and handy:
      
      	- recording events stream
      	- profiling
      	- synchronous tracing
      
      ...
      3cf17be3
    • Kirill Smelkov's avatar
      prog: Simple package to implement programs with subcommands. · f1c839d8
      Kirill Smelkov authored
      Initial draft. The implementation is modelled after `git` and `go`.
      f1c839d8
    • Kirill Smelkov's avatar
      my: Fixup tests after relicensing · 0db94061
      Kirill Smelkov authored
      4d9a613c (Relicense to GPLv3+ with wide exception for all Free Software
      / Open Source projects + Business options.) added more lines to
      my/my_test.go than removed (@@ -5,16 +5,18 @@) so the line number of
      
              myline := Line()
      
      changed by 2 (= 18 - 16).
      
      Fix the test accordingly.
      0db94061
  13. 24 Oct, 2017 1 commit
    • Kirill Smelkov's avatar
      Relicense to GPLv3+ with wide exception for all Free Software / Open Source... · 4d9a613c
      Kirill Smelkov authored
      Relicense to GPLv3+ with wide exception for all Free Software / Open Source projects + Business options.
      
      Nexedi stack is licensed under Free Software licenses with various exceptions
      that cover three business cases:
      
      - Free Software
      - Proprietary Software
      - Rebranding
      
      As long as one intends to develop Free Software based on Nexedi stack, no
      license cost is involved. Developing proprietary software based on Nexedi stack
      may require a proprietary exception license. Rebranding Nexedi stack is
      prohibited unless rebranding license is acquired.
      
      Through this licensing approach, Nexedi expects to encourage Free Software
      development without restrictions and at the same time create a framework for
      proprietary software to contribute to the long term sustainability of the
      Nexedi stack.
      
      Please see https://www.nexedi.com/licensing for details, rationale and options.
      4d9a613c
  14. 11 Sep, 2017 1 commit
  15. 08 Aug, 2017 1 commit
  16. 25 Jul, 2017 1 commit
  17. 19 Jul, 2017 1 commit
    • Kirill Smelkov's avatar
      xerr += First, Merge · 79e328c5
      Kirill Smelkov authored
      This are handy utilities to reduce several errors into only 1 either
      picking the first error or merging, if there are several, into Errorv.
      
      It is unfortunate but an issue with Errorv was realized that even though
      it satisfies the error interface, it cannot be generally worked with as
      error because it (being []error) is uncomparable. Thus e.g. the following
      code, if err dynamic type is Errorv, will panic:
      
      	if err == io.EOF
      
      It is pity Go slices are uncomparable.
      79e328c5
  18. 07 Jun, 2017 1 commit
    • Kirill Smelkov's avatar
      xerr += Context, Contextf · 8f170959
      Kirill Smelkov authored
      This are handy utilities to automatically prepend context on error
      return. For example in
      
      	func myfunc(...) (..., err error) {
      		defer xerr.Context(&err, "error context")
      		...
      		if ... {
      			return ..., errors.New("an error")
      		}
      		return ..., nil
      	}
      
      while preserving nil error return on successful execution, myfunc will
      return error with string "error context: an error" on failure case.
      8f170959
  19. 20 Apr, 2017 4 commits
    • Kirill Smelkov's avatar
      xfmt: Qpy & friends to quote string the way Python would do · c0bbd06e
      Kirill Smelkov authored
      This is somtimes needed for checking programs output bit-to-bit where on
      python side repr(x), `x` or %r is used for output.
      c0bbd06e
    • Kirill Smelkov's avatar
      xfmt: Addons to fmt and strconv with focus on formatting text without allocations · 1aa677c8
      Kirill Smelkov authored
      Std fmt works ok unless you need to do text formatting in hot codepaths.
      There fmt becomes inappropriate as it is slow and does allocations on
      every formatting.
      
      strconv also does not have append routines for every needed case, e.g.
      there is no strconv.AppendRune, no strconv.AppendHex etc.
      
      So xfmt
      
      1. provides append routines for builtin types lacking in strconv
      
      2. introduces xfmt.Stringer interface which types can implement to hook
         into general formatting via xfmt.Append()
      
      3. provides xfmt.Buffer which is []byte with syntatic sugar for
         formatting in a way similar to printf: For example if in fmt speak
         you have
      
      	s := fmt.Sprintf("hello %q %d %x", "world", 1, []byte("data"))
      
         xfmt analog would be
      
      	buf := xfmt.Buffer{}
      	buf .S("hello ") .Q("world") .C(' ') .D(1) .C(' ') .Xb([]byte("data"))
      	s := buf.Bytes()
      
         and xfmt.Buffer can be reused several times via Buffer.Reset() .
      
      The above xfmt.Buffer usage is more uglier than fmt.Printf but much less uglier
      than direct strconv.Append* and friends calls, and works faster and without
      allocations compared to fmt.Printf:
      
      	BenchmarkXFmt/%c(0x41)-4                20000000                65.4 ns/op             1 B/op          1 allocs/op
      	BenchmarkXFmt/.Cb(0x41)-4               200000000                5.96 ns/op            0 B/op          0 allocs/op
      	BenchmarkXFmt/%c(-1)-4                  20000000                70.1 ns/op             3 B/op          1 allocs/op
      	BenchmarkXFmt/.C(-1)-4                  100000000               12.9 ns/op             0 B/op          0 allocs/op
      	BenchmarkXFmt/%c(66)-4                  20000000                65.8 ns/op             1 B/op          1 allocs/op
      	BenchmarkXFmt/.C(66)-4                  100000000               12.7 ns/op             0 B/op          0 allocs/op
      	BenchmarkXFmt/%c(1080)-4                20000000                67.2 ns/op             2 B/op          1 allocs/op
      	BenchmarkXFmt/.C(1080)-4                100000000               12.8 ns/op             0 B/op          0 allocs/op
      	BenchmarkXFmt/%c(8364)-4                20000000                69.4 ns/op             3 B/op          1 allocs/op
      	BenchmarkXFmt/.C(8364)-4                100000000               13.8 ns/op             0 B/op          0 allocs/op
      	BenchmarkXFmt/%c(65537)-4               20000000                70.5 ns/op             4 B/op          1 allocs/op
      	BenchmarkXFmt/.C(65537)-4               100000000               14.3 ns/op             0 B/op          0 allocs/op
      	BenchmarkXFmt/%s("hello")-4             20000000                72.3 ns/op             5 B/op          1 allocs/op
      	BenchmarkXFmt/.S("hello")-4             200000000                9.40 ns/op            0 B/op          0 allocs/op
      	...
      1aa677c8
    • Kirill Smelkov's avatar
      xbytes: Additional utilities for working with byte slices · 64936209
      Kirill Smelkov authored
      - small addons over std bytes package: xbytes.ContainsByte
      - (re)allocation routines for byte slices
      64936209
    • Kirill Smelkov's avatar
      xmath: Addon math routines · b1893d27
      Kirill Smelkov authored
      So far only one of them: CeilPow2 to return min(y) >= x: y = 2^i - i.e.
      next power of two >= x. This is handy to have in reallocation routines
      to allocate buffers from 2^i classes.
      
      Two implementations:
      
      - fast for go19
      - slower fallback for go18
      b1893d27
  20. 19 Apr, 2017 1 commit