Skip to content

L
linux
Switch branch/tag
  1. 08 Feb, 2015 2 commits
    • Steven Rostedt (Red Hat)'s avatar
      x86/tlb/trace: Do not trace on CPU that is offline · 6c8465a8
      Steven Rostedt (Red Hat) authored 
      When taking a CPU down for suspend and resume, a tracepoint may be called
when the CPU has been designated offline. As tracepoints require RCU for
protection, they must not be called if the current CPU is offline.

Unfortunately, trace_tlb_flush() is called in this scenario as was noted
by LOCKDEP:

...

 Disabling non-boot CPUs ...
 intel_pstate CPU 1 exiting

 ===============================
 smpboot: CPU 1 didn't die...
 [ INFO: suspicious RCU usage. ]
 3.19.0-rc7-next-20150204.1-iniza-small #1 Not tainted
 -------------------------------
 include/trace/events/tlb.h:35 suspicious rcu_dereference_check() usage!

 other info that might help us debug this:

 RCU used illegally from offline CPU!
 rcu_scheduler_active = 1, debug_locks = 0
 no locks held by swapper/1/0.

 stack backtrace:
 CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.19.0-rc7-next-20150204.1-iniza-small #1
 Hardware name: SAMSUNG ELECTRONICS CO., LTD. 530U3BI/530U4BI/530U4BH/530U3BI/530U4BI/530U4BH, BIOS 13XK 03/28/2013
  0000000000000001 ffff88011a44fe18 ffffffff817e370d 0000000000000011
  ffff88011a448290 ffff88011a44fe48 ffffffff810d6847 ffff8800c66b9600
  0000000000000001 ffff88011a44c000 ffffffff81cb3900 ffff88011a44fe78
 Call Trace:
  [<ffffffff817e370d>] dump_stack+0x4c/0x65
  [<ffffffff810d6847>] lockdep_rcu_suspicious+0xe7/0x120
  [<ffffffff810b71a5>] idle_task_exit+0x205/0x2c0
  [<ffffffff81054c4e>] play_dead_common+0xe/0x50
  [<ffffffff81054ca5>] native_play_dead+0x15/0x140
  [<ffffffff8102963f>] arch_cpu_idle_dead+0xf/0x20
  [<ffffffff810cd89e>] cpu_startup_entry+0x37e/0x580
  [<ffffffff81053e20>] start_secondary+0x140/0x150
 intel_pstate CPU 2 exiting

...

By converting the tlb_flush tracepoint to a TRACE_EVENT_CONDITION where the
condition is cpu_online(smp_processor_id()), we can avoid calling RCU protected
code when the CPU is offline.

Link: http://lkml.kernel.org/r/CA+icZUUGiGDoL5NU8RuxKzFjoLjEKRtUWx=JB8B9a0EQv-eGzQ@mail.gmail.com

Cc: stable@vger.kernel.org # 3.17+
Fixes: d17d8f9d "x86/mm: Add tracepoints for TLB flushes"
Reported-by: default avatarSedat Dilek <sedat.dilek@gmail.com>
Tested-by: default avatarSedat Dilek <sedat.dilek@gmail.com>
Suggested-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: default avatarDave Hansen <dave@sr71.net>
Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      6c8465a8
    • Steven Rostedt (Red Hat)'s avatar
      tracing: Add condition check to RCU lockdep checks · a05d59a5
      Steven Rostedt (Red Hat) authored 
      The trace_tlb_flush() tracepoint can be called when a CPU is going offline.
When a CPU is offline, RCU is no longer watching that CPU and since the
tracepoint is protected by RCU, it must not be called. To prevent the
tlb_flush tracepoint from being called when the CPU is offline, it was
converted to a TRACE_EVENT_CONDITION where the condition checks if the
CPU is online before calling the tracepoint.

Unfortunately, this was not enough to stop lockdep from complaining about
it. Even though the RCU protected code of the tracepoint will never be
called, the condition is hidden within the tracepoint, and even though the
condition prevents RCU code from being called, the lockdep checks are
outside the tracepoint (this is to test tracepoints even when they are not
enabled).

Even though tracepoints should be checked to be RCU safe when they are not
enabled, the condition should still be considered when checking RCU.

Link: http://lkml.kernel.org/r/CA+icZUUGiGDoL5NU8RuxKzFjoLjEKRtUWx=JB8B9a0EQv-eGzQ@mail.gmail.com

Fixes: 3a630178 "tracing: generate RCU warnings even when tracepoints are disabled"
Cc: stable@vger.kernel.org # 3.18+
Acked-by: default avatarDave Hansen <dave@sr71.net>
Reported-by: default avatarSedat Dilek <sedat.dilek@gmail.com>
Tested-by: default avatarSedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      a05d59a5
  3. 15 Jan, 2015 5 commits
    • Steven Rostedt (Red Hat)'s avatar
      tracing: Fix enabling of syscall events on the command line · ce1039bd
      Steven Rostedt (Red Hat) authored 
      Commit 5f893b26 "tracing: Move enabling tracepoints to just after
rcu_init()" broke the enabling of system call events from the command
line. The reason was that the enabling of command line trace events
was moved before PID 1 started, and the syscall tracepoints require
that all tasks have the TIF_SYSCALL_TRACEPOINT flag set. But the
swapper task (pid 0) is not part of that. Since the swapper task is the
only task that is running at this early in boot, no task gets the
flag set, and the tracepoint never gets reached.

Instead of setting the swapper task flag (there should be no reason to
do that), re-enabled trace events again after the init thread (PID 1)
has been started. It requires disabling all command line events and
re-enabling them, as just enabling them again will not reset the logic
to set the TIF_SYSCALL_TRACEPOINT flag, as the syscall tracepoint will
be fooled into thinking that it was already set, and wont try setting
it again. For this reason, we must first disable it and re-enable it.

Link: http://lkml.kernel.org/r/1421188517-18312-1-git-send-email-mpe@ellerman.id.au
Link: http://lkml.kernel.org/r/20150115040506.216066449@goodmis.orgReported-by: default avatarMichael Ellerman <mpe@ellerman.id.au>
Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      ce1039bd
    • Steven Rostedt (Red Hat)'s avatar
      tracing: Remove extra call to init_ftrace_syscalls() · 83829b74
      Steven Rostedt (Red Hat) authored 
      trace_init() calls init_ftrace_syscalls() and then calls trace_event_init()
which also calls init_ftrace_syscalls(). It makes more sense to only
call it from trace_event_init().

Calling it twice wastes memory, as it allocates the syscall events twice,
and loses the first copy of it.

Link: http://lkml.kernel.org/r/54AF53BD.5070303@huawei.com
Link: http://lkml.kernel.org/r/20150115040505.930398632@goodmis.orgReported-by: default avatarWang Nan <wangnan0@huawei.com>
Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      83829b74
    • Steven Rostedt (Red Hat)'s avatar
      ftrace/jprobes/x86: Fix conflict between jprobes and function graph tracing · 237d28db
      Steven Rostedt (Red Hat) authored 
      If the function graph tracer traces a jprobe callback, the system will
crash. This can easily be demonstrated by compiling the jprobe
sample module that is in the kernel tree, loading it and running the
function graph tracer.

 # modprobe jprobe_example.ko
 # echo function_graph > /sys/kernel/debug/tracing/current_tracer
 # ls

The first two commands end up in a nice crash after the first fork.
(do_fork has a jprobe attached to it, so "ls" just triggers that fork)

The problem is caused by the jprobe_return() that all jprobe callbacks
must end with. The way jprobes works is that the function a jprobe
is attached to has a breakpoint placed at the start of it (or it uses
ftrace if fentry is supported). The breakpoint handler (or ftrace callback)
will copy the stack frame and change the ip address to return to the
jprobe handler instead of the function. The jprobe handler must end
with jprobe_return() which swaps the stack and does an int3 (breakpoint).
This breakpoint handler will then put back the saved stack frame,
simulate the instruction at the beginning of the function it added
a breakpoint to, and then continue on.

For function tracing to work, it hijakes the return address from the
stack frame, and replaces it with a hook function that will trace
the end of the call. This hook function will restore the return
address of the function call.

If the function tracer traces the jprobe handler, the hook function
for that handler will not be called, and its saved return address
will be used for the next function. This will result in a kernel crash.

To solve this, pause function tracing before the jprobe handler is called
and unpause it before it returns back to the function it probed.

Some other updates:

Used a variable "saved_sp" to hold kcb->jprobe_saved_sp. This makes the
code look a bit cleaner and easier to understand (various tries to fix
this bug required this change).

Note, if fentry is being used, jprobes will change the ip address before
the function graph tracer runs and it will not be able to trace the
function that the jprobe is probing.

Link: http://lkml.kernel.org/r/20150114154329.552437962@goodmis.org

Cc: stable@vger.kernel.org # 2.6.30+
Acked-by: default avatarMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      237d28db
    • Steven Rostedt (Red Hat)'s avatar
      ftrace: Check both notrace and filter for old hash · 7485058e
      Steven Rostedt (Red Hat) authored 
      Using just the filter for checking for trampolines or regs is not enough
when updating the code against the records that represent all functions.
Both the filter hash and the notrace hash need to be checked.

To trigger this bug (using trace-cmd and perf):

 # perf probe -a do_fork
 # trace-cmd start -B foo -e probe
 # trace-cmd record -p function_graph -n do_fork sleep 1

The trace-cmd record at the end clears the filter before it disables
function_graph tracing and then that causes the accounting of the
ftrace function records to become incorrect and causes ftrace to bug.

Link: http://lkml.kernel.org/r/20150114154329.358378039@goodmis.org

Cc: stable@vger.kernel.org
[ still need to switch old_hash_ops to old_ops_hash ]
Reviewed-by: default avatarMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      7485058e
    • Steven Rostedt (Red Hat)'s avatar
      ftrace: Fix updating of filters for shared global_ops filters · 8f86f837
      Steven Rostedt (Red Hat) authored 
      As the set_ftrace_filter affects both the function tracer as well as the
function graph tracer, the ops that represent each have a shared
ftrace_ops_hash structure. This allows both to be updated when the filter
files are updated.

But if function graph is enabled and the global_ops (function tracing) ops
is not, then it is possible that the filter could be changed without the
update happening for the function graph ops. This will cause the changes
to not take place and may even cause a ftrace_bug to occur as it could mess
with the trampoline accounting.

The solution is to check if the ops uses the shared global_ops filter and
if the ops itself is not enabled, to check if there's another ops that is
enabled and also shares the global_ops filter. In that case, the
modification still needs to be executed.

Link: http://lkml.kernel.org/r/20150114154329.055980438@goodmis.org

Cc: stable@vger.kernel.org # 3.17+
Reviewed-by: default avatarMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      8f86f837
  5. 06 Jan, 2015 1 commit
  7. 05 Jan, 2015 3 commits
  9. 04 Jan, 2015 4 commits
  11. 02 Jan, 2015 3 commits
  13. 31 Dec, 2014 10 commits
  15. 30 Dec, 2014 12 commits
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7