• David Hildenbrand's avatar
    proc/vmcore: fix possible deadlock on concurrent mmap and read · 5039b170
    David Hildenbrand authored
    Lockdep noticed that there is chance for a deadlock if we have concurrent
    mmap, concurrent read, and the addition/removal of a callback.
    
    As nicely explained by Boqun:
     "Lockdep warned about the above sequences because rw_semaphore is a
      fair read-write lock, and the following can cause a deadlock:
    
    	TASK 1			TASK 2		TASK 3
    	======			======		======
    	down_write(mmap_lock);
    				down_read(vmcore_cb_rwsem)
    						down_write(vmcore_cb_rwsem); // blocked
    	down_read(vmcore_cb_rwsem); // cannot get the lock because of the fairness
    				down_read(mmap_lock); // blocked
    
      IOW, a reader can block another read if there is a writer queued by
      the second reader and the lock is fair"
    
    To fix this, convert to srcu to make this deadlock impossible.  We need
    srcu as our callbacks can sleep.  With this change, I cannot trigger any
    lockdep warnings.
    
        ======================================================
        WARNING: possible circular locking dependency detected
        5.17.0-0.rc0.20220117git0c947b89.68.test.fc36.x86_64 #1 Not tainted
        ------------------------------------------------------
        makedumpfile/542 is trying to acquire lock:
        ffffffff832d2eb8 (vmcore_cb_rwsem){.+.+}-{3:3}, at: mmap_vmcore+0x340/0x580
    
        but task is already holding lock:
        ffff8880af226438 (&mm->mmap_lock#2){++++}-{3:3}, at: vm_mmap_pgoff+0x84/0x150
    
        which lock already depends on the new lock.
    
        the existing dependency chain (in reverse order) is:
    
        -> #1 (&mm->mmap_lock#2){++++}-{3:3}:
               lock_acquire+0xc3/0x1a0
               __might_fault+0x4e/0x70
               _copy_to_user+0x1f/0x90
               __copy_oldmem_page+0x72/0xc0
               read_from_oldmem+0x77/0x1e0
               read_vmcore+0x2c2/0x310
               proc_reg_read+0x47/0xa0
               vfs_read+0x101/0x340
               __x64_sys_pread64+0x5d/0xa0
               do_syscall_64+0x43/0x90
               entry_SYSCALL_64_after_hwframe+0x44/0xae
    
        -> #0 (vmcore_cb_rwsem){.+.+}-{3:3}:
               validate_chain+0x9f4/0x2670
               __lock_acquire+0x8f7/0xbc0
               lock_acquire+0xc3/0x1a0
               down_read+0x4a/0x140
               mmap_vmcore+0x340/0x580
               proc_reg_mmap+0x3e/0x90
               mmap_region+0x504/0x880
               do_mmap+0x38a/0x520
               vm_mmap_pgoff+0xc1/0x150
               ksys_mmap_pgoff+0x178/0x200
               do_syscall_64+0x43/0x90
               entry_SYSCALL_64_after_hwframe+0x44/0xae
    
        other info that might help us debug this:
    
         Possible unsafe locking scenario:
    
               CPU0                    CPU1
               ----                    ----
          lock(&mm->mmap_lock#2);
                                       lock(vmcore_cb_rwsem);
                                       lock(&mm->mmap_lock#2);
          lock(vmcore_cb_rwsem);
    
         *** DEADLOCK ***
    
        1 lock held by makedumpfile/542:
         #0: ffff8880af226438 (&mm->mmap_lock#2){++++}-{3:3}, at: vm_mmap_pgoff+0x84/0x150
    
        stack backtrace:
        CPU: 0 PID: 542 Comm: makedumpfile Not tainted 5.17.0-0.rc0.20220117git0c947b89.68.test.fc36.x86_64 #1
        Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
        Call Trace:
         __lock_acquire+0x8f7/0xbc0
         lock_acquire+0xc3/0x1a0
         down_read+0x4a/0x140
         mmap_vmcore+0x340/0x580
         proc_reg_mmap+0x3e/0x90
         mmap_region+0x504/0x880
         do_mmap+0x38a/0x520
         vm_mmap_pgoff+0xc1/0x150
         ksys_mmap_pgoff+0x178/0x200
         do_syscall_64+0x43/0x90
    
    Link: https://lkml.kernel.org/r/20220119193417.100385-1-david@redhat.com
    Fixes: cc5f2704 ("proc/vmcore: convert oldmem_pfn_is_ram callback to more generic vmcore callbacks")
    Signed-off-by: default avatarDavid Hildenbrand <david@redhat.com>
    Reported-by: default avatarBaoquan He <bhe@redhat.com>
    Acked-by: default avatarBaoquan He <bhe@redhat.com>
    Cc: Vivek Goyal <vgoyal@redhat.com>
    Cc: Dave Young <dyoung@redhat.com>
    Cc: "Paul E. McKenney" <paulmck@kernel.org>
    Cc: Josh Triplett <josh@joshtriplett.org>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Boqun Feng <boqun.feng@gmail.com>
    Cc: <stable@vger.kernel.org>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    5039b170
vmcore.c 41.2 KB