• Thomas Gleixner's avatar
    jump_label: Fix concurrency issues in static_key_slow_dec() · 83ab38ef
    Thomas Gleixner authored
    The commit which tried to fix the concurrency issues of concurrent
    static_key_slow_inc() failed to fix the equivalent issues
    vs. static_key_slow_dec():
    
    CPU0                     CPU1
    
    static_key_slow_dec()
      static_key_slow_try_dec()
    
    	key->enabled == 1
    	val = atomic_fetch_add_unless(&key->enabled, -1, 1);
    	if (val == 1)
    	     return false;
    
      jump_label_lock();
      if (atomic_dec_and_test(&key->enabled)) {
         --> key->enabled == 0
       __jump_label_update()
    
    			 static_key_slow_dec()
    			   static_key_slow_try_dec()
    
    			     key->enabled == 0
    			     val = atomic_fetch_add_unless(&key->enabled, -1, 1);
    
    			      --> key->enabled == -1 <- FAIL
    
    There is another bug in that code, when there is a concurrent
    static_key_slow_inc() which enables the key as that sets key->enabled to -1
    so on the other CPU
    
    	val = atomic_fetch_add_unless(&key->enabled, -1, 1);
    
    will succeed and decrement to -2, which is invalid.
    
    Cure all of this by replacing the atomic_fetch_add_unless() with a
    atomic_try_cmpxchg() loop similar to static_key_fast_inc_not_disabled().
    
    [peterz: add WARN_ON_ONCE for the -1 race]
    Fixes: 4c5ea0a9 ("locking/static_key: Fix concurrent static_key_slow_inc()")
    Reported-by: default avatarYue Sun <samsun1006219@gmail.com>
    Reported-by: default avatarXingwei Lee <xrivendell7@gmail.com>
    Signed-off-by: default avatarThomas Gleixner <tglx@linutronix.de>
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Link: https://lkml.kernel.org/r/20240610124406.422897838@linutronix.de
    83ab38ef
jump_label.c 22.8 KB