1. 16 Apr, 2019 1 commit
    • Kirill Smelkov's avatar
      tracing: call/return overhead related to using trace functions is gone · 79711643
      Kirill Smelkov authored
      As of Go1.12 the check for whether particular trace event is enabled or
      not is inlined into caller. For example:
      
      	---- 8< ---- (connection.go)
      	//trace:event traceMsgSendPre(l *NodeLink, connId uint32, msg proto.Msg)
      
      	func (link *NodeLink) sendMsg(connId uint32, msg proto.Msg) error {
      		traceMsgSendPre(link, connId, msg)
      
      		buf := msgPack(connId, msg)
      		return link.sendPkt(buf)
      	}
      
      	---- 8< ---- (ztrace.go)
      	// traceevent: traceMsgSendPre(l *NodeLink, connId uint32, msg proto.Msg)
      
      	type _t_traceMsgSendPre struct {
      	        tracing.Probe
      	        probefunc     func(l *NodeLink, connId uint32, msg proto.Msg)
      	}
      
      	var _traceMsgSendPre *_t_traceMsgSendPre
      
      	func traceMsgSendPre(l *NodeLink, connId uint32, msg proto.Msg) {
      	        if _traceMsgSendPre != nil {
      	                _traceMsgSendPre_run(l, connId, msg)
      	        }
      	}
      
      	func _traceMsgSendPre_run(l *NodeLink, connId uint32, msg proto.Msg) {
      	        for p := _traceMsgSendPre; p != nil; p = (*_t_traceMsgSendPre)(unsafe.Pointer(p.Next())) {
      	                p.probefunc(l, connId, msg)
      	        }
      	}
      
      	---- 8< ---- (connection.o)
      	TEXT lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·(*NodeLink).sendMsg(SB), ABIInternal, $40-48 // connection.go:1368
      	        // MOVQ    (TLS), CX (stack growth prologue)
      	        // CMPQ    SP, 16(CX)
      	        // JLS     177
      	        // SUBQ    $40, SP
      	        // MOVQ    BP, 32(SP) (BP save)
      	        // LEAQ    32(SP), BP (BP init)
      	        // FUNCDATA $0, gclocals·dbae954985b577993f0eafab1347dd21(SB) (args)
      	        FUNCDATA   $1, gclocals·f6bd6b3389b872033d462029172c8612(SB) (locals)
      	        FUNCDATA   $3, gclocals·47f0f3578d0f16ef296c57af2564832c(SB)
      	        PCDATA     $2, $0             // connection.go:1369
      	        PCDATA     $0, $0
      	        XCHGL      AX, AX
      	        CMPQ       lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·_traceMsgSendPre(SB), $0	<-- inlined traceMsgSendPre
      	        JNE        pc130              // ztrace.go:50					<--
      	pc44:
      	        MOVL       connId+56(SP), AX  // connection.go:1371				<-- normal path
      	        MOVL       AX, (SP)
      	        MOVQ       msg+64(SP), AX
      	        MOVQ       AX, 8(SP)
      	        PCDATA     $2, $1
      	        PCDATA     $0, $1
      	        MOVQ       msg+72(SP), AX
      	        PCDATA     $2, $0
      	        MOVQ       AX, 16(SP)
      	        CALL       lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·msgPack(SB)
      	        PCDATA     $2, $1
      	        MOVQ       24(SP), AX
      	        PCDATA     $2, $2             // connection.go:1372
      	        PCDATA     $0, $2
      	        MOVQ       link+48(SP), CX
      	        PCDATA     $2, $1
      	        MOVQ       CX, (SP)
      	        PCDATA     $2, $0
      	        MOVQ       AX, 8(SP)
      	        CALL       lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·(*NodeLink).sendPkt(SB)
      	        PCDATA     $2, $1
      	        MOVQ       24(SP), AX
      	        MOVQ       16(SP), CX
      	        PCDATA     $0, $3
      	        MOVQ       CX, _r2+80(SP)
      	        PCDATA     $2, $0
      	        MOVQ       AX, _r2+88(SP)
      	        // MOVQ    32(SP), BP (BP restore)
      	        // ADDQ    $40, SP (SP restore)
      	        RET
      	pc130:											<-- slow path (tracepoint attached)
      	        PCDATA     $2, $1             // ztrace.go:51
      	        PCDATA     $0, $0
      	        MOVQ       link+48(SP), AX
      	        PCDATA     $2, $0
      	        MOVQ       AX, (SP)
      	        MOVL       connId+56(SP), CX
      	        MOVL       CX, 8(SP)
      	        MOVQ       msg+64(SP), DX
      	        MOVQ       DX, 16(SP)
      	        PCDATA     $2, $3
      	        MOVQ       msg+72(SP), BX
      	        PCDATA     $2, $0
      	        MOVQ       BX, 24(SP)
      	        CALL       lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·_traceMsgSendPre_run(SB)
      	        JMP        pc44
      	        // NOP (stack growth)
      	        // PCDATA  $0, $-1            // connection.go:1368
      	        // PCDATA  $2, $-1
      	        // CALL    runtime.morestack_noctxt(SB)
      	        // JMP     0
      79711643
  2. 12 Apr, 2019 1 commit
  3. 11 Apr, 2019 2 commits
  4. 24 Jan, 2019 1 commit
  5. 17 Jan, 2019 1 commit
  6. 16 Jan, 2019 1 commit
  7. 19 Nov, 2018 1 commit
  8. 06 Jul, 2018 1 commit
  9. 03 Jul, 2018 1 commit
    • Kirill Smelkov's avatar
      tracing/pyruntraced: New tool to run Python code with tracepoints activated (draft) · 7a476082
      Kirill Smelkov authored
      We will hopefully need it in the future to test external python code
      similarly to when testing in-process go code with the help of
      tracepoints. See module description for details (tracetest is not yet
      done).
      
      There is no yet Go client API for interacting with such-traced python
      program. For the record: manually inspecting traced python process could
      be done via e.g.
      
      	socat EXEC:"./pyruntraced 3 neo.trace.py -- ../../../t/backup-play/N1-writer",fdin=3,fdout=3 TCP:localhost:8888
      
      and telnetting to localhost:8888 from another xterm.
      7a476082
  10. 20 Jun, 2018 1 commit
  11. 19 Jun, 2018 11 commits
    • Kirill Smelkov's avatar
      xnet/lonet: Draft _Python_ counterpart · 5f16012d
      Kirill Smelkov authored
      This patch brings lonet implementation in Python with the idea that Go
      and Python programs could interoperate via lonet network and thus mixed
      Go/Python application cluster could be tested.
      
      Implementation quality is lower compared to Go version, but still it
      should be more or less ok.
      5f16012d
    • Kirill Smelkov's avatar
      xerr: Draft _Python_ counterpart · fa0f9048
      Kirill Smelkov authored
      Provide infrastructure, similar to what we already have in xerr, to Python
      programs: to wrap errors with error context and to extract the cause
      from a wrapped error.
      
      We will need it in the next patch.
      fa0f9048
    • Kirill Smelkov's avatar
      golang/gcompat: New _Python_ package that provides Go-compatibility layer for Python · 7c4f365c
      Kirill Smelkov authored
      Let's start with substitute for %q that is lacking in python.
      The implementation is copied from zodbtools from here:
      
      https://lab.nexedi.com/nexedi/zodbtools/blob/2801fae9/zodbtools/util.py#L39
      
      It is not good for golang to depend on zodbtools and the function is
      minor. In the future maybe zodbtools in turn will change to import qq
      from golang.
      7c4f365c
    • Kirill Smelkov's avatar
      golang: New _Python_ package to provide Go-like features to Python language · 9e1aa6ab
      Kirill Smelkov authored
      - `go` spawns lightweight thread.
      - `chan` and `select` provide channels with Go semantic.
      - `method` allows to define methods separate from class.
      - `gimport` allows to import python modules by full path in a Go workspace.
      
      The focus of first draft was on usage interface and on correctness, not speed.
      In particular select should be fully working.
      
      If there is a chance I will maybe try to followup with gevent-based
      implementation in the future.
      9e1aa6ab
    • Kirill Smelkov's avatar
      xnet/lonet: New package to provide TCP network simulated on top of localhost TCP loopback. · fd4f9a4e
      Kirill Smelkov authored
      Lonet is the virtnet network that, contrary to pipenet, could be used
      when there are several OS-level processes involved.
      
      It uses SQLite for its registry and native OS-level TCP over loopback
      for data exchange. There is small text-based connection handshake
      protocol prelude that have to be carried out when a connection is tried
      to be established to host via its subnetwork, but after it data exchange
      goes directly through OS TCP stack.
      
      Lonet documentation follows:
      
      """
      Package lonet provides TCP network simulated on top of localhost TCP loopback.
      
      For testing distributed systems it is sometimes handy to imitate network of
      several TCP hosts. It is also handy that ports allocated on Dial/Listen/Accept on
      that hosts be predictable - that would help tests to verify network events
      against expected sequence. When whole system could be imitated in 1 OS-level
      process, package lab.nexedi.com/kirr/go123/xnet/pipenet serves the task via
      providing TCP-like synchronous in-memory network of net.Pipes. However
      pipenet cannot be used for cases where tested system consists of 2 or more
      OS-level processes. This is where lonet comes into play:
      
      Similarly to pipenet addresses on lonet are host:port pairs and several
      hosts could be created with different names. A host is xnet.Networker and
      so can be worked with similarly to regular TCP network access-point with
      Dial/Listen/Accept. Host's ports allocation is predictable: ports of a host
      are contiguous integer sequence starting from 1 that are all initially free,
      and whenever autobind is requested the first free port of the host will be
      used.
      
      Internally lonet network maintains registry of hosts so that lonet
      addresses could be resolved to OS-level addresses, for example α:1 and β:1
      to 127.0.0.1:4567 and 127.0.0.1:8765, and once lonet connection is
      established it becomes served by OS-level TCP connection over loopback.
      
      Example:
      
      net, err := lonet.Join(ctx, "mynet")
      hα, err := net.NewHost(ctx, "α")
      hβ, err := net.NewHost(ctx, "β")
      
      // starts listening on address "α:10"
      l, err := hα.Listen(":10")
      go func() {
       csrv, err := l.Accept()   // csrv will have LocalAddr "α:1"
      }()
      ccli, err := hβ.Dial(ctx, "α:10") // ccli will be connection between "β:1" - "α:1"
      
      Once again lonet is similar to pipenet, but since it works via OS TCP stack
      it could be handy for testing networked application when there are several
      OS-level processes involved.
      """
      
      """
      Lonet organization
      
      For every lonet network there is a registry with information about hosts
      available on the network, and for each host its OS-level listening address.
      The registry is kept as SQLite database under
      
      /<tmp>/lonet/<network>/registry.db
      
      Whenever host α needs to establish connection to address on host β, it
      queries the registry for β and further talks to β on that address.
      Correspondingly when a host joins the network, it announces itself to the
      registry so that other hosts could see it.
      
      Handshake protocol
      
      After α establishes OS-level connection to β via main β address, it sends
      request to further establish lonet connection on top of that:
      
      > lonet "<network>" dial "<α:portα>" "<β:portβ>"\n
      
      β checks whether portβ is listening, and if yes, accepts the connection on
      corresponding on-β listener with giving feedback to α that connection was
      accepted:
      
      < lonet "<network>" connected "<β:portβ'>"\n
      
      After that connection is considered to be lonet-established and all further
      exchange on it is directly controlled by corresponding lonet-level
      Read/Write on α and β.
      
      If, on the other hand, lonet-level connection cannot be established, β replies:
      
      < lonet "<networkβ>" E "<error>"\n
      
      where <error> could be:
      
      - connection refused if <β:portβ> is not listening
      - network mismatch if β thinks it works on different lonet network than α
      - protocol error if β thinks that α send incorrect dial request
      - ...
      """
      fd4f9a4e
    • Kirill Smelkov's avatar
      xnet/pipenet: Generalize it into xnet/virtnet · 40120cb0
      Kirill Smelkov authored
      As we are going to implement another virtual network it would be good to
      share common code between implementations. For this generalize pipenet
      implementation to also cover the case when one does not own full network
      and owns only some hosts of it.
      
      An example of such situation is when one process handles one group of
      virtual hosts and another process handles another group of virtual
      hosts. Below a group of virtual hosts handled as part of network is
      called subnetwork.
      
      If hosts are not created in one place, we need a way to communicate
      information about new hosts in between subnetworks. This leads to using
      some kind of "registry" (see Registry interface).
      
      Then for the common code to be reused by a virtual network
      implementation it has to provide its working in the form of Engine
      interface to that common code. In turn the common code exposes another
      - Notifier - interface for particular network implementation to notify
      common code of events that come from outside to the subnetwork.
      
      Pipenet is reworked to be just a client of the common virtnet
      infrastructure.
      
      Virtnet documentation follows:
      
      """
      Package virtnet provides infrastructure for TCP-like virtual networks.
      
      For testing distributed systems it is sometimes handy to imitate network of
      several TCP hosts. It is also handy that ports allocated on Dial/Listen/Accept
      on that hosts be predictable - that would help tests to verify network
      events against expected sequence.
      
      Package virtnet provides infrastructure for using and implementing such
      TCP-like virtual networks.
      
      Using virtnet networks
      
      Addresses on a virtnet network are host:port pairs represented by Addr.
      A network conceptually consists of several SubNetworks each being home for
      multiple Hosts. Host is xnet.Networker and so can be worked with similarly
      to regular TCP network access-point with Dial/Listen/Accept. Host's ports
      allocation is predictable: ports of a host are contiguous integer sequence
      starting from 1 that are all initially free, and whenever autobind is
      requested the first free port of the host will be used.
      Virtnet ensures that host names are unique throughout whole network.
      
      To work with a virtnet network, one uses corresponding package for
      particular virtnet network implementation. Such packages provide a way to
      join particular network and after joining give back SubNetwork to user.
      Starting from SubNetwork one can create Hosts and from those exchange data
      throughout whole network.
      
      Please see package lab.nexedi.com/kirr/go123/xnet/pipenet for particular
      well-known virtnet-based network.
      
      Implementing virtnet networks
      
      To implement a virtnet-based network one need to implement Engine and Registry.
      
      A virtnet network implementation should provide Engine and Registry
      instances to SubNetwork when creating it. The subnetwork will use provided
      engine and registry for its operations. A virtnet network implementation
      receives instance of Notifier together with SubNetwork when creating it. The
      network implementation should use provided Notifier to notify the subnetwork
      to handle incoming events.
      
      Please see Engine, Registry and Notifier documentation for details.
      """
      
      Another virtnet-based network that is not limited to be used only in 1
      OS process will follow next.
      40120cb0
    • Kirill Smelkov's avatar
      xnet/pipenet: Polish a bit · f04d243b
      Kirill Smelkov authored
      - clarify docstrings;
      - unexport NetPrefix.
      f04d243b
    • Kirill Smelkov's avatar
      xnet/pipenet: TODO to fixit for TCP semantic · ca56bb08
      Kirill Smelkov authored
      On TCP port(accepted) = port(listen), i.e.  When we connect
      www.nexedi.com:80, remote addr of socket will have port 80.  Likewise on
      server side accepted socket will have local port 80.  The connection
      should be thus fully identified by src-dst address pair.
      
      However currently pipenet, when accepting connection, allocates another
      free port at acceptor side.
      ca56bb08
    • Kirill Smelkov's avatar
      xcontext: New package that complements std package context. · 8e09e376
      Kirill Smelkov authored
      Quoting documentation:
      
      """
      Package xcontext provides addons to std package context.
      
      Merging contexts
      
      Merge could be handy in situations where spawned job needs to be canceled
      whenever any of 2 contexts becomes done. This frequently arises with service
      methods that accept context as argument, and the service itself, on another
      control line, could be instructed to become non-operational. For example:
      
      	func (srv *Service) DoSomething(ctx context.Context) (err error) {
      		defer xerr.Contextf(&err, "%s: do something", srv)
      
             		// srv.serveCtx is context that becomes canceled when srv is
             		// instructed to stop providing service.
             		origCtx := ctx
             		ctx, cancel := xcontext.Merge(ctx, srv.serveCtx)
             		defer cancel()
      
             		err = srv.doJob(ctx)
             		if err != nil {
             			if ctx.Err() != nil && origCtx.Err() == nil {
             				// error due to service shutdown
             				err = ErrServiceDown
             			}
             			return err
             		}
      
      		...
      	}
      """
      8e09e376
    • Kirill Smelkov's avatar
      xnet: Polish documentation a bit · 65ab3d34
      Kirill Smelkov authored
      - clarify Networker.Name on example of NetPlain;
      - we don't need Networker.Addr since we have Networker.Name;
      - format Dial/Listen docstrings so that they render properly under godoc;
      - format list in NetTLS doc so that it renders properly under godoc.
      65ab3d34
    • Kirill Smelkov's avatar
      .gitignore: Initial draft · c311277e
      Kirill Smelkov authored
      profile/trace outputs + files left after `go test -c`.
      c311277e
  12. 11 May, 2018 3 commits
    • Kirill Smelkov's avatar
      tracing/runtime: Add support for Go1.11 (preliminary) · d9250d63
      Kirill Smelkov authored
      As of go 47be3d49 there are 2 new types in g compared to go1.10:
      waitReason and ancestorInfo.
      
      Preliminary because Go1.11 has not been released yet, so
      zruntime_g_go1.11.go will need to be potentially regenerated after
      the release.
      d9250d63
    • Kirill Smelkov's avatar
      tracing/runtime: Teach g_typedef to extract type definition for non-struct types · 2c2c5cb6
      Kirill Smelkov authored
      With previous sed expression it was failing e.g. on runtime.guintptr,
      giving much more after type definition:
      
      	$ go doc -c -u runtime.muintptr | sed -n -e '/^type /,/^}/p'
      	type muintptr uintptr
      	    muintptr is a *m that is not tracked by the garbage collector.
      
      	    Because we do free Ms, there are some additional constrains on muintptrs:
      
      	    1. Never hold an muintptr locally across a safe point.
      
      	    2. Any muintptr in the heap must be owned by the M itself so it can
      
      	    ensure it is not in use when the last true *m is released.
      
      	func (mp muintptr) ptr() *m
      	func (mp *muintptr) set(m *m)
      
      Fix it since we'll need to extract more single-line types for Go1.11 support.
      
      To verify it works de-stub {g,p,m}uintptr.
      2c2c5cb6
    • Kirill Smelkov's avatar
      tracing/gotrace: Fix tests for Go1.10 where CGo is now supported · b7e99d87
      Kirill Smelkov authored
      Go1.10 now modifies CGo sources as text, not AST, and this way comments
      are not removed and gotrace can see its '//trace:event ' comments in a
      CGo *.go files: https://github.com/golang/go/commit/85c3ebf4.
      
      This way with Go1.10 tests were failing because `//trace:event
      traceHello()` in a/pkg1/pkg1c.go was now noticed by gotrace and more
      then expected trace events produces.
      
      I have not noticed this in 65c399f0 (tracing/runtime: Add support for
      Go1.10 (preliminary)) probably because at that time above Go commit was
      not in my local Go1.10 tree.
      
      For the reference: tracing/runtime support regenerated for Go1.10.2
      stays exactly the same as in 65c399f0 and so does not need updating.
      b7e99d87
  13. 10 May, 2018 2 commits
  14. 25 Apr, 2018 3 commits
    • Kirill Smelkov's avatar
      xnet: Add event corresponding to network dial start · 66edb630
      Kirill Smelkov authored
      We already have TraceConnect which corresponds to event of established
      network connection.
      
      However in many test scenarios it is required to know and pause
      execution right before dial is going to happen. For this introduce
      TraceDial which represents event corresponding to network dialing start.
      66edb630
    • Kirill Smelkov's avatar
      xnet: Networker += Name() · c95f2373
      Kirill Smelkov authored
      Networker.Name should return name of access-point Networker represents
      on the network.
      
      We will need this information in the next patch for tracing dial events,
      when there is no connection established yet, and thus dialer location
      has to be taken from somewhere. It is also generally a good idea for
      Networker to have a name.
      
      For NetPlain networker the name is local hostname.
      c95f2373
    • Kirill Smelkov's avatar
      xnet/pipenet: Add tests for Host.Network · 8c6b8ad0
      Kirill Smelkov authored
      Should be in d3a7a196 (xnet/pipenet: Package for TCP-like synchronous
      in-memory network of net.Pipes)
      8c6b8ad0
  15. 24 Apr, 2018 1 commit
  16. 09 Apr, 2018 1 commit
  17. 15 Mar, 2018 1 commit
  18. 15 Jan, 2018 1 commit
  19. 12 Jan, 2018 2 commits
    • Kirill Smelkov's avatar
      xbufio += SeqReaderAt - buffering wrapper for a io.ReaderAt optimized for sequential access · 3956445e
      Kirill Smelkov authored
      For example in ZODB FileStorage format reader routines are written
      working with io.ReaderAt for the following reasons:
      
      - for loads random-access is required,
      - there can be several concurrent loads in flight simultaneously.
      
      At the same time various database iterations (APIs additional to load)
      use sequential access pattern and can be served by the same record
      reading routines. However with them we cannot use e.g. bufio.Reader
      because it works with plain io.Reader, not io.ReaderAt.
      
      Here comes SeqReaderAt: it adds a buffer, by default 2·4K, on top of
      original io.Reader, automatically detects direction of sequential
      access which can be forward, backward, or interleaved forward-backward
      patterns, and buffers data accordingly to avoid many syscalls e.g. in
      os.File case.
      3956445e
    • Kirill Smelkov's avatar
      xio, xbufio: New packages amending std packages io & bufio · eadf5c4a
      Kirill Smelkov authored
      Currently by direct & buffered readers that also can report current
      offset in input stream.
      
      This functionality is handy to have when for example one needs to report
      an error after finding decoding problem and wants to include particular
      stream position in the report.
      eadf5c4a
  20. 09 Jan, 2018 2 commits
  21. 21 Dec, 2017 2 commits