• Nicholas Piggin's avatar
    mm: add PageWaiters indicating tasks are waiting for a page bit · 62906027
    Nicholas Piggin authored
    Add a new page flag, PageWaiters, to indicate the page waitqueue has
    tasks waiting. This can be tested rather than testing waitqueue_active
    which requires another cacheline load.
    
    This bit is always set when the page has tasks on page_waitqueue(page),
    and is set and cleared under the waitqueue lock. It may be set when
    there are no tasks on the waitqueue, which will cause a harmless extra
    wakeup check that will clears the bit.
    
    The generic bit-waitqueue infrastructure is no longer used for pages.
    Instead, waitqueues are used directly with a custom key type. The
    generic code was not flexible enough to have PageWaiters manipulation
    under the waitqueue lock (which simplifies concurrency).
    
    This improves the performance of page lock intensive microbenchmarks by
    2-3%.
    
    Putting two bits in the same word opens the opportunity to remove the
    memory barrier between clearing the lock bit and testing the waiters
    bit, after some work on the arch primitives (e.g., ensuring memory
    operand widths match and cover both bits).
    Signed-off-by: default avatarNicholas Piggin <npiggin@gmail.com>
    Cc: Dave Hansen <dave.hansen@linux.intel.com>
    Cc: Bob Peterson <rpeterso@redhat.com>
    Cc: Steven Whitehouse <swhiteho@redhat.com>
    Cc: Andrew Lutomirski <luto@kernel.org>
    Cc: Andreas Gruenbacher <agruenba@redhat.com>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Mel Gorman <mgorman@techsingularity.net>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    62906027
internal.h 15.3 KB