• Paolo Abeni's avatar
    x86/uaccess: Optimize copy_user_enhanced_fast_string() for short strings · b260feec
    Paolo Abeni authored
    [ Upstream commit 236222d3 ]
    
    According to the Intel datasheet, the REP MOVSB instruction
    exposes a pretty heavy setup cost (50 ticks), which hurts
    short string copy operations.
    
    This change tries to avoid this cost by calling the explicit
    loop available in the unrolled code for strings shorter
    than 64 bytes.
    
    The 64 bytes cutoff value is arbitrary from the code logic
    point of view - it has been selected based on measurements,
    as the largest value that still ensures a measurable gain.
    
    Micro benchmarks of the __copy_from_user() function with
    lengths in the [0-63] range show this performance gain
    (shorter the string, larger the gain):
    
     - in the [55%-4%] range on Intel Xeon(R) CPU E5-2690 v4
     - in the [72%-9%] range on Intel Core i7-4810MQ
    
    Other tested CPUs - namely Intel Atom S1260 and AMD Opteron
    8216 - show no difference, because they do not expose the
    ERMS feature bit.
    Signed-off-by: default avatarPaolo Abeni <pabeni@redhat.com>
    Acked-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    Cc: Alan Cox <gnomes@lxorguk.ukuu.org.uk>
    Cc: Andy Lutomirski <luto@kernel.org>
    Cc: Borislav Petkov <bp@alien8.de>
    Cc: Brian Gerst <brgerst@gmail.com>
    Cc: Denys Vlasenko <dvlasenk@redhat.com>
    Cc: H. Peter Anvin <hpa@zytor.com>
    Cc: Hannes Frederic Sowa <hannes@stressinduktion.org>
    Cc: Josh Poimboeuf <jpoimboe@redhat.com>
    Cc: Kees Cook <keescook@chromium.org>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Link: http://lkml.kernel.org/r/4533a1d101fd460f80e21329a34928fad521c1d4.1498744345.git.pabeni@redhat.com
    [ Clarified the changelog. ]
    Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
    Signed-off-by: default avatarSasha Levin <alexander.levin@verizon.com>
    b260feec
copy_user_64.S 5.52 KB