• Peter Zijlstra's avatar
    sched/idle: Avoid spurious wakeup IPIs · fd99f91a
    Peter Zijlstra authored
    Because mwait_idle_with_hints() gets called from !idle context it must
    call current_clr_polling(). This however means that resched_task() is
    very likely to send an IPI even when we were polling:
    
      CPU0					CPU1
    
      if (current_set_polling_and_test())
        goto out;
    
      __monitor(&ti->flags);
      if (!need_resched())
        __mwait(eax, ecx);
    					set_tsk_need_resched(p);
    					smp_mb();
    out:
      current_clr_polling();
    					if (!tsk_is_polling(p))
    					  smp_send_reschedule(cpu);
    
    So while it is correct (extra IPIs aren't a problem, whereas a missed
    IPI would be) it is a performance problem (for some).
    
    Avoid this issue by using fetch_or() to atomically set NEED_RESCHED
    and test if POLLING_NRFLAG is set.
    
    Since a CPU stuck in mwait is unlikely to modify the flags word,
    contention on the cmpxchg is unlikely and thus we should mostly
    succeed in a single go.
    Signed-off-by: default avatarPeter Zijlstra <peterz@infradead.org>
    Acked-by: default avatarNicolas Pitre <nico@linaro.org>
    Cc: Andy Lutomirski <luto@amacapital.net>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Link: http://lkml.kernel.org/n/tip-kf5suce6njh5xf5d3od13rr0@git.kernel.orgSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
    fd99f91a
core.c 189 KB