• Rik van Riel's avatar
    dma-debug: avoid deadlock between dma debug vs printk and netconsole · bd44ca3d
    Rik van Riel authored
    Currently the dma debugging code can end up indirectly calling printk
    under the radix_lock. This happens when a radix tree node allocation
    fails.
    
    This is a problem because the printk code, when used together with
    netconsole, can end up inside the dma debugging code while trying to
    transmit a message over netcons.
    
    This creates the possibility of either a circular deadlock on the same
    CPU, with that CPU trying to grab the radix_lock twice, or an ABBA
    deadlock between different CPUs, where one CPU grabs the console lock
    first and then waits for the radix_lock, while the other CPU is holding
    the radix_lock and is waiting for the console lock.
    
    The trace captured by lockdep is of the ABBA variant.
    
    -> #2 (&dma_entry_hash[i].lock){-.-.}-{2:2}:
                      _raw_spin_lock_irqsave+0x5a/0x90
                      debug_dma_map_page+0x79/0x180
                      dma_map_page_attrs+0x1d2/0x2f0
                      bnxt_start_xmit+0x8c6/0x1540
                      netpoll_start_xmit+0x13f/0x180
                      netpoll_send_skb+0x20d/0x320
                      netpoll_send_udp+0x453/0x4a0
                      write_ext_msg+0x1b9/0x460
                      console_flush_all+0x2ff/0x5a0
                      console_unlock+0x55/0x180
                      vprintk_emit+0x2e3/0x3c0
                      devkmsg_emit+0x5a/0x80
                      devkmsg_write+0xfd/0x180
                      do_iter_readv_writev+0x164/0x1b0
                      vfs_writev+0xf9/0x2b0
                      do_writev+0x6d/0x110
                      do_syscall_64+0x80/0x150
                      entry_SYSCALL_64_after_hwframe+0x4b/0x53
    
    -> #0 (console_owner){-.-.}-{0:0}:
                      __lock_acquire+0x15d1/0x31a0
                      lock_acquire+0xe8/0x290
                      console_flush_all+0x2ea/0x5a0
                      console_unlock+0x55/0x180
                      vprintk_emit+0x2e3/0x3c0
                      _printk+0x59/0x80
                      warn_alloc+0x122/0x1b0
                      __alloc_pages_slowpath+0x1101/0x1120
                      __alloc_pages+0x1eb/0x2c0
                      alloc_slab_page+0x5f/0x150
                      new_slab+0x2dc/0x4e0
                      ___slab_alloc+0xdcb/0x1390
                      kmem_cache_alloc+0x23d/0x360
                      radix_tree_node_alloc+0x3c/0xf0
                      radix_tree_insert+0xf5/0x230
                      add_dma_entry+0xe9/0x360
                      dma_map_page_attrs+0x1d2/0x2f0
                      __bnxt_alloc_rx_frag+0x147/0x180
                      bnxt_alloc_rx_data+0x79/0x160
                      bnxt_rx_skb+0x29/0xc0
                      bnxt_rx_pkt+0xe22/0x1570
                      __bnxt_poll_work+0x101/0x390
                      bnxt_poll+0x7e/0x320
                      __napi_poll+0x29/0x160
                      net_rx_action+0x1e0/0x3e0
                      handle_softirqs+0x190/0x510
                      run_ksoftirqd+0x4e/0x90
                      smpboot_thread_fn+0x1a8/0x270
                      kthread+0x102/0x120
                      ret_from_fork+0x2f/0x40
                      ret_from_fork_asm+0x11/0x20
    
    This bug is more likely than it seems, because when one CPU has run out
    of memory, chances are the other has too.
    
    The good news is, this bug is hidden behind the CONFIG_DMA_API_DEBUG, so
    not many users are likely to trigger it.
    Signed-off-by: default avatarRik van Riel <riel@surriel.com>
    Reported-by: default avatarKonstantin Ovsepian <ovs@meta.com>
    Signed-off-by: default avatarChristoph Hellwig <hch@lst.de>
    bd44ca3d
debug.c 42 KB