• Cong Wang's avatar
    tracing: Introduce trace event injection · 6c3edaf9
    Cong Wang authored
    We have been trying to use rasdaemon to monitor hardware errors like
    correctable memory errors. rasdaemon uses trace events to monitor
    various hardware errors. In order to test it, we have to inject some
    hardware errors, unfortunately not all of them provide error
    injections. MCE does provide a way to inject MCE errors, but errors
    like PCI error and devlink error don't, it is not easy to add error
    injection to each of them. Instead, it is relatively easier to just
    allow users to inject trace events in a generic way so that all trace
    events can be injected.
    
    This patch introduces trace event injection, where a new 'inject' is
    added to each tracepoint directory. Users could write into this file
    with key=value pairs to specify the value of each fields of the trace
    event, all unspecified fields are set to zero values by default.
    
    For example, for the net/net_dev_queue tracepoint, we can inject:
    
      INJECT=/sys/kernel/debug/tracing/events/net/net_dev_queue/inject
      echo "" > $INJECT
      echo "name='test'" > $INJECT
      echo "name='test' len=1024" > $INJECT
      cat /sys/kernel/debug/tracing/trace
      ...
       <...>-614   [000] ....    36.571483: net_dev_queue: dev= skbaddr=00000000fbf338c2 len=0
       <...>-614   [001] ....   136.588252: net_dev_queue: dev=test skbaddr=00000000fbf338c2 len=0
       <...>-614   [001] .N..   208.431878: net_dev_queue: dev=test skbaddr=00000000fbf338c2 len=1024
    
    Triggers could be triggered as usual too:
    
      echo "stacktrace if len == 1025" > /sys/kernel/debug/tracing/events/net/net_dev_queue/trigger
      echo "len=1025" > $INJECT
      cat /sys/kernel/debug/tracing/trace
      ...
          bash-614   [000] ....    36.571483: net_dev_queue: dev= skbaddr=00000000fbf338c2 len=0
          bash-614   [001] ....   136.588252: net_dev_queue: dev=test skbaddr=00000000fbf338c2 len=0
          bash-614   [001] .N..   208.431878: net_dev_queue: dev=test skbaddr=00000000fbf338c2 len=1024
          bash-614   [001] .N.1   284.236349: <stack trace>
     => event_inject_write
     => vfs_write
     => ksys_write
     => do_syscall_64
     => entry_SYSCALL_64_after_hwframe
    
    The only thing that can't be injected is string pointers as they
    require constant string pointers, this can't be done at run time.
    
    Link: http://lkml.kernel.org/r/20191130045218.18979-1-xiyou.wangcong@gmail.com
    
    Cc: Ingo Molnar <mingo@redhat.com>
    Signed-off-by: default avatarCong Wang <xiyou.wangcong@gmail.com>
    Signed-off-by: default avatarSteven Rostedt (VMware) <rostedt@goodmis.org>
    6c3edaf9
trace.h 63 KB