-
Roland Dreier authored
Using the vmxnet3 driver produces a lockdep warning because vmxnet3_set_mc(), which is called with mc->mca_lock held, takes adapter->cmd_lock. However, there are a couple of places where adapter->cmd_lock is taken with softirqs enabled, lockdep warns that a softirq that tries to take mc->mca_lock could happen while adapter->cmd_lock is held, leading to an AB-BA deadlock. I'm not sure if this is a real potential deadlock or not, but the simplest and best fix seems to be simply to make sure we take cmd_lock with spin_lock_irqsave() everywhere -- the places with plain spin_lock just look like oversights. The full enormous lockdep warning is: ========================================================= [ INFO: possible irq lock inversion dependency detected ] 2.6.39-rc6+ #1 --------------------------------------------------------- ifconfig/567 just changed the state of lock: (&(&mc->mca_lock)->rlock){+.-...}, at: [<ffffffff81531e9f>] mld_ifc_timer_expire+0xff/0x280 but this lock took another, SOFTIRQ-unsafe lock in the past: (&(&adapter->cmd_lock)->rlock){+.+...} and interrupts could create inverse lock ordering between them. other info that might help us debug this: 4 locks held by ifconfig/567: #0: (rtnl_mutex){+.+.+.}, at: [<ffffffff8147d547>] rtnl_lock+0x17/0x20 #1: ((inetaddr_chain).rwsem){.+.+.+}, at: [<ffffffff810896cf>] __blocking_notifier_call_chain+0x5f/0xb0 #2: (&idev->mc_ifc_timer){+.-...}, at: [<ffffffff8106f21b>] run_timer_softirq+0xeb/0x3f0 #3: (&ndev->lock){++.-..}, at: [<ffffffff81531dd2>] mld_ifc_timer_expire+0x32/0x280 the shortest dependencies between 2nd lock and 1st lock: -> (&(&adapter->cmd_lock)->rlock){+.+...} ops: 11 { HARDIRQ-ON-W at: [<ffffffff8109ad86>] __lock_acquire+0x7f6/0x1e10 [<ffffffff8109ca4d>] lock_acquire+0x9d/0x130 [<ffffffff81571156>] _raw_spin_lock+0x36/0x70 [<ffffffffa000d212>] vmxnet3_alloc_intr_resources+0x22/0x230 [vmxnet3] [<ffffffffa0014031>] vmxnet3_probe_device+0x5f6/0x15c5 [vmxnet3] [<ffffffff812df67f>] local_pci_probe+0x5f/0xd0 [<ffffffff812dfde9>] pci_device_probe+0x119/0x120 [<ffffffff81373df6>] driver_probe_device+0x96/0x1c0 [<ffffffff81373fcb>] __driver_attach+0xab/0xb0 [<ffffffff81372a1e>] bus_for_each_dev+0x5e/0x90 [<ffffffff81373a2e>] driver_attach+0x1e/0x20 [<ffffffff813735b8>] bus_add_driver+0xc8/0x290 [<ffffffff813745b6>] driver_register+0x76/0x140 [<ffffffff812e0046>] __pci_register_driver+0x66/0xe0 [<ffffffffa001b03a>] serio_raw_poll+0x3a/0x60 [serio_raw] [<ffffffff81002165>] do_one_initcall+0x45/0x190 [<ffffffff810aa76b>] sys_init_module+0xfb/0x250 [<ffffffff8157a142>] system_call_fastpath+0x16/0x1b SOFTIRQ-ON-W at: [<ffffffff8109adb7>] __lock_acquire+0x827/0x1e10 [<ffffffff8109ca4d>] lock_acquire+0x9d/0x130 [<ffffffff81571156>] _raw_spin_lock+0x36/0x70 [<ffffffffa000d212>] vmxnet3_alloc_intr_resources+0x22/0x230 [vmxnet3] [<ffffffffa0014031>] vmxnet3_probe_device+0x5f6/0x15c5 [vmxnet3] [<ffffffff812df67f>] local_pci_probe+0x5f/0xd0 [<ffffffff812dfde9>] pci_device_probe+0x119/0x120 [<ffffffff81373df6>] driver_probe_device+0x96/0x1c0 [<ffffffff81373fcb>] __driver_attach+0xab/0xb0 [<ffffffff81372a1e>] bus_for_each_dev+0x5e/0x90 [<ffffffff81373a2e>] driver_attach+0x1e/0x20 [<ffffffff813735b8>] bus_add_driver+0xc8/0x290 [<ffffffff813745b6>] driver_register+0x76/0x140 [<ffffffff812e0046>] __pci_register_driver+0x66/0xe0 [<ffffffffa001b03a>] serio_raw_poll+0x3a/0x60 [serio_raw] [<ffffffff81002165>] do_one_initcall+0x45/0x190 [<ffffffff810aa76b>] sys_init_module+0xfb/0x250 [<ffffffff8157a142>] system_call_fastpath+0x16/0x1b INITIAL USE at: [<ffffffff8109a9e9>] __lock_acquire+0x459/0x1e10 [<ffffffff8109ca4d>] lock_acquire+0x9d/0x130 [<ffffffff81571156>] _raw_spin_lock+0x36/0x70 [<ffffffffa000d212>] vmxnet3_alloc_intr_resources+0x22/0x230 [vmxnet3] [<ffffffffa0014031>] vmxnet3_probe_device+0x5f6/0x15c5 [vmxnet3] [<ffffffff812df67f>] local_pci_probe+0x5f/0xd0 [<ffffffff812dfde9>] pci_device_probe+0x119/0x120 [<ffffffff81373df6>] driver_probe_device+0x96/0x1c0 [<ffffffff81373fcb>] __driver_attach+0xab/0xb0 [<ffffffff81372a1e>] bus_for_each_dev+0x5e/0x90 [<ffffffff81373a2e>] driver_attach+0x1e/0x20 [<ffffffff813735b8>] bus_add_driver+0xc8/0x290 [<ffffffff813745b6>] driver_register+0x76/0x140 [<ffffffff812e0046>] __pci_register_driver+0x66/0xe0 [<ffffffffa001b03a>] serio_raw_poll+0x3a/0x60 [serio_raw] [<ffffffff81002165>] do_one_initcall+0x45/0x190 [<ffffffff810aa76b>] sys_init_module+0xfb/0x250 [<ffffffff8157a142>] system_call_fastpath+0x16/0x1b } ... key at: [<ffffffffa0017590>] __key.42516+0x0/0xffffffffffffda70 [vmxnet3] ... acquired at: [<ffffffff8109ca4d>] lock_acquire+0x9d/0x130 [<ffffffff81571bb5>] _raw_spin_lock_irqsave+0x55/0xa0 [<ffffffffa000de27>] vmxnet3_set_mc+0x97/0x1a0 [vmxnet3] [<ffffffff8146ffa0>] __dev_set_rx_mode+0x40/0xb0 [<ffffffff81470040>] dev_set_rx_mode+0x30/0x50 [<ffffffff81470127>] __dev_open+0xc7/0x100 [<ffffffff814703c1>] __dev_change_flags+0xa1/0x180 [<ffffffff81470568>] dev_change_flags+0x28/0x70 [<ffffffff814da960>] devinet_ioctl+0x730/0x800 [<ffffffff814db508>] inet_ioctl+0x88/0xa0 [<ffffffff814541f0>] sock_do_ioctl+0x30/0x70 [<ffffffff814542a9>] sock_ioctl+0x79/0x2f0 [<ffffffff81188798>] do_vfs_ioctl+0x98/0x570 [<ffffffff81188d01>] sys_ioctl+0x91/0xa0 [<ffffffff8157a142>] system_call_fastpath+0x16/0x1b -> (_xmit_ETHER){+.....} ops: 6 { HARDIRQ-ON-W at: [<ffffffff8109ad86>] __lock_acquire+0x7f6/0x1e10 [<ffffffff8109ca4d>] lock_acquire+0x9d/0x130 [<ffffffff8157124b>] _raw_spin_lock_bh+0x3b/0x70 [<ffffffff81475618>] __dev_mc_add+0x38/0x90 [<ffffffff814756a0>] dev_mc_add+0x10/0x20 [<ffffffff81532c9e>] igmp6_group_added+0x10e/0x1b0 [<ffffffff81533f2d>] ipv6_dev_mc_inc+0x2cd/0x430 [<ffffffff81515e17>] ipv6_add_dev+0x357/0x450 [<ffffffff81519f27>] addrconf_notify+0x2f7/0xb10 [<ffffffff81575c1c>] notifier_call_chain+0x8c/0xc0 [<ffffffff81089586>] raw_notifier_call_chain+0x16/0x20 [<ffffffff814689b7>] call_netdevice_notifiers+0x37/0x70 [<ffffffff8146a944>] register_netdevice+0x244/0x2d0 [<ffffffff8146aa0f>] register_netdev+0x3f/0x60 [<ffffffffa001419b>] vmxnet3_probe_device+0x760/0x15c5 [vmxnet3] [<ffffffff812df67f>] local_pci_probe+0x5f/0xd0 [<ffffffff812dfde9>] pci_device_probe+0x119/0x120 [<ffffffff81373df6>] driver_probe_device+0x96/0x1c0 [<ffffffff81373fcb>] __driver_attach+0xab/0xb0 [<ffffffff81372a1e>] bus_for_each_dev+0x5e/0x90 [<ffffffff81373a2e>] driver_attach+0x1e/0x20 [<ffffffff813735b8>] bus_add_driver+0xc8/0x290 [<ffffffff813745b6>] driver_register+0x76/0x140 [<ffffffff812e0046>] __pci_register_driver+0x66/0xe0 [<ffffffffa001b03a>] serio_raw_poll+0x3a/0x60 [serio_raw] [<ffffffff81002165>] do_one_initcall+0x45/0x190 [<ffffffff810aa76b>] sys_init_module+0xfb/0x250 [<ffffffff8157a142>] system_call_fastpath+0x16/0x1b INITIAL USE at: [<ffffffff8109a9e9>] __lock_acquire+0x459/0x1e10 [<ffffffff8109ca4d>] lock_acquire+0x9d/0x130 [<ffffffff8157124b>] _raw_spin_lock_bh+0x3b/0x70 [<ffffffff81475618>] __dev_mc_add+0x38/0x90 [<ffffffff814756a0>] dev_mc_add+0x10/0x20 [<ffffffff81532c9e>] igmp6_group_added+0x10e/0x1b0 [<ffffffff81533f2d>] ipv6_dev_mc_inc+0x2cd/0x430 [<ffffffff81515e17>] ipv6_add_dev+0x357/0x450 [<ffffffff81519f27>] addrconf_notify+0x2f7/0xb10 [<ffffffff81575c1c>] notifier_call_chain+0x8c/0xc0 [<ffffffff81089586>] raw_notifier_call_chain+0x16/0x20 [<ffffffff814689b7>] call_netdevice_notifiers+0x37/0x70 [<ffffffff8146a944>] register_netdevice+0x244/0x2d0 [<ffffffff8146aa0f>] register_netdev+0x3f/0x60 [<ffffffffa001419b>] vmxnet3_probe_device+0x760/0x15c5 [vmxnet3] [<ffffffff812df67f>] local_pci_probe+0x5f/0xd0 [<ffffffff812dfde9>] pci_device_probe+0x119/0x120 [<ffffffff81373df6>] driver_probe_device+0x96/0x1c0 [<ffffffff81373fcb>] __driver_attach+0xab/0xb0 [<ffffffff81372a1e>] bus_for_each_dev+0x5e/0x90 [<ffffffff81373a2e>] driver_attach+0x1e/0x20 [<ffffffff813735b8>] bus_add_driver+0xc8/0x290 [<ffffffff813745b6>] driver_register+0x76/0x140 [<ffffffff812e0046>] __pci_register_driver+0x66/0xe0 [<ffffffffa001b03a>] serio_raw_poll+0x3a/0x60 [serio_raw] [<ffffffff81002165>] do_one_initcall+0x45/0x190 [<ffffffff810aa76b>] sys_init_module+0xfb/0x250 [<ffffffff8157a142>] system_call_fastpath+0x16/0x1b } ... key at: [<ffffffff827fd868>] netdev_addr_lock_key+0x8/0x1e0 ... acquired at: [<ffffffff8109ca4d>] lock_acquire+0x9d/0x130 [<ffffffff8157124b>] _raw_spin_lock_bh+0x3b/0x70 [<ffffffff81475618>] __dev_mc_add+0x38/0x90 [<ffffffff814756a0>] dev_mc_add+0x10/0x20 [<ffffffff81532c9e>] igmp6_group_added+0x10e/0x1b0 [<ffffffff81533f2d>] ipv6_dev_mc_inc+0x2cd/0x430 [<ffffffff81515e17>] ipv6_add_dev+0x357/0x450 [<ffffffff81519f27>] addrconf_notify+0x2f7/0xb10 [<ffffffff81575c1c>] notifier_call_chain+0x8c/0xc0 [<ffffffff81089586>] raw_notifier_call_chain+0x16/0x20 [<ffffffff814689b7>] call_netdevice_notifiers+0x37/0x70 [<ffffffff8146a944>] register_netdevice+0x244/0x2d0 [<ffffffff8146aa0f>] register_netdev+0x3f/0x60 [<ffffffffa001419b>] vmxnet3_probe_device+0x760/0x15c5 [vmxnet3] [<ffffffff812df67f>] local_pci_probe+0x5f/0xd0 [<ffffffff812dfde9>] pci_device_probe+0x119/0x120 [<ffffffff81373df6>] driver_probe_device+0x96/0x1c0 [<ffffffff81373fcb>] __driver_attach+0xab/0xb0 [<ffffffff81372a1e>] bus_for_each_dev+0x5e/0x90 [<ffffffff81373a2e>] driver_attach+0x1e/0x20 [<ffffffff813735b8>] bus_add_driver+0xc8/0x290 [<ffffffff813745b6>] driver_register+0x76/0x140 [<ffffffff812e0046>] __pci_register_driver+0x66/0xe0 [<ffffffffa001b03a>] serio_raw_poll+0x3a/0x60 [serio_raw] [<ffffffff81002165>] do_one_initcall+0x45/0x190 [<ffffffff810aa76b>] sys_init_module+0xfb/0x250 [<ffffffff8157a142>] system_call_fastpath+0x16/0x1b -> (&(&mc->mca_lock)->rlock){+.-...} ops: 6 { HARDIRQ-ON-W at: [<ffffffff8109ad86>] __lock_acquire+0x7f6/0x1e10 [<ffffffff8109ca4d>] lock_acquire+0x9d/0x130 [<ffffffff8157124b>] _raw_spin_lock_bh+0x3b/0x70 [<ffffffff81532bd5>] igmp6_group_added+0x45/0x1b0 [<ffffffff81533f2d>] ipv6_dev_mc_inc+0x2cd/0x430 [<ffffffff81515e17>] ipv6_add_dev+0x357/0x450 [<ffffffff81ce0d16>] addrconf_init+0x4e/0x183 [<ffffffff81ce0ba1>] inet6_init+0x191/0x2a6 [<ffffffff81002165>] do_one_initcall+0x45/0x190 [<ffffffff81ca4d3f>] kernel_init+0xe3/0x168 [<ffffffff8157b2e4>] kernel_thread_helper+0x4/0x10 IN-SOFTIRQ-W at: [<ffffffff8109ad5e>] __lock_acquire+0x7ce/0x1e10 [<ffffffff8109ca4d>] lock_acquire+0x9d/0x130 [<ffffffff8157124b>] _raw_spin_lock_bh+0x3b/0x70 [<ffffffff81531e9f>] mld_ifc_timer_expire+0xff/0x280 [<ffffffff8106f2a9>] run_timer_softirq+0x179/0x3f0 [<ffffffff810666d0>] __do_softirq+0xc0/0x210 [<ffffffff8157b3dc>] call_softirq+0x1c/0x30 [<ffffffff8100d42d>] do_softirq+0xad/0xe0 [<ffffffff81066afe>] irq_exit+0x9e/0xb0 [<ffffffff8157bd40>] smp_apic_timer_interrupt+0x70/0x9b [<ffffffff8157ab93>] apic_timer_interrupt+0x13/0x20 [<ffffffff8149d857>] rt_do_flush+0x87/0x2a0 [<ffffffff814a16b6>] rt_cache_flush+0x46/0x60 [<ffffffff814e36e0>] fib_disable_ip+0x40/0x60 [<ffffffff814e5447>] fib_inetaddr_event+0xd7/0xe0 [<ffffffff81575c1c>] notifier_call_chain+0x8c/0xc0 [<ffffffff810896e8>] __blocking_notifier_call_chain+0x78/0xb0 [<ffffffff81089736>] blocking_notifier_call_chain+0x16/0x20 [<ffffffff814d8021>] __inet_del_ifa+0xf1/0x2e0 [<ffffffff814d8223>] inet_del_ifa+0x13/0x20 [<ffffffff814da731>] devinet_ioctl+0x501/0x800 [<ffffffff814db508>] inet_ioctl+0x88/0xa0 [<ffffffff814541f0>] sock_do_ioctl+0x30/0x70 [<ffffffff814542a9>] sock_ioctl+0x79/0x2f0 [<ffffffff81188798>] do_vfs_ioctl+0x98/0x570 [<ffffffff81188d01>] sys_ioctl+0x91/0xa0 [<ffffffff8157a142>] system_call_fastpath+0x16/0x1b INITIAL USE at: [<ffffffff8109a9e9>] __lock_acquire+0x459/0x1e10 [<ffffffff8109ca4d>] lock_acquire+0x9d/0x130 [<ffffffff8157124b>] _raw_spin_lock_bh+0x3b/0x70 [<ffffffff81532bd5>] igmp6_group_added+0x45/0x1b0 [<ffffffff81533f2d>] ipv6_dev_mc_inc+0x2cd/0x430 [<ffffffff81515e17>] ipv6_add_dev+0x357/0x450 [<ffffffff81ce0d16>] addrconf_init+0x4e/0x183 [<ffffffff81ce0ba1>] inet6_init+0x191/0x2a6 [<ffffffff81002165>] do_one_initcall+0x45/0x190 [<ffffffff81ca4d3f>] kernel_init+0xe3/0x168 [<ffffffff8157b2e4>] kernel_thread_helper+0x4/0x10 } ... key at: [<ffffffff82801be2>] __key.40877+0x0/0x8 ... acquired at: [<ffffffff810997bc>] check_usage_forwards+0x9c/0x110 [<ffffffff8109a32c>] mark_lock+0x19c/0x400 [<ffffffff8109ad5e>] __lock_acquire+0x7ce/0x1e10 [<ffffffff8109ca4d>] lock_acquire+0x9d/0x130 [<ffffffff8157124b>] _raw_spin_lock_bh+0x3b/0x70 [<ffffffff81531e9f>] mld_ifc_timer_expire+0xff/0x280 [<ffffffff8106f2a9>] run_timer_softirq+0x179/0x3f0 [<ffffffff810666d0>] __do_softirq+0xc0/0x210 [<ffffffff8157b3dc>] call_softirq+0x1c/0x30 [<ffffffff8100d42d>] do_softirq+0xad/0xe0 [<ffffffff81066afe>] irq_exit+0x9e/0xb0 [<ffffffff8157bd40>] smp_apic_timer_interrupt+0x70/0x9b [<ffffffff8157ab93>] apic_timer_interrupt+0x13/0x20 [<ffffffff8149d857>] rt_do_flush+0x87/0x2a0 [<ffffffff814a16b6>] rt_cache_flush+0x46/0x60 [<ffffffff814e36e0>] fib_disable_ip+0x40/0x60 [<ffffffff814e5447>] fib_inetaddr_event+0xd7/0xe0 [<ffffffff81575c1c>] notifier_call_chain+0x8c/0xc0 [<ffffffff810896e8>] __blocking_notifier_call_chain+0x78/0xb0 [<ffffffff81089736>] blocking_notifier_call_chain+0x16/0x20 [<ffffffff814d8021>] __inet_del_ifa+0xf1/0x2e0 [<ffffffff814d8223>] inet_del_ifa+0x13/0x20 [<ffffffff814da731>] devinet_ioctl+0x501/0x800 [<ffffffff814db508>] inet_ioctl+0x88/0xa0 [<ffffffff814541f0>] sock_do_ioctl+0x30/0x70 [<ffffffff814542a9>] sock_ioctl+0x79/0x2f0 [<ffffffff81188798>] do_vfs_ioctl+0x98/0x570 [<ffffffff81188d01>] sys_ioctl+0x91/0xa0 [<ffffffff8157a142>] system_call_fastpath+0x16/0x1b stack backtrace: Pid: 567, comm: ifconfig Not tainted 2.6.39-rc6+ #1 Call Trace: <IRQ> [<ffffffff810996f6>] print_irq_inversion_bug+0x146/0x170 [<ffffffff81099720>] ? print_irq_inversion_bug+0x170/0x170 [<ffffffff810997bc>] check_usage_forwards+0x9c/0x110 [<ffffffff8109a32c>] mark_lock+0x19c/0x400 [<ffffffff8109ad5e>] __lock_acquire+0x7ce/0x1e10 [<ffffffff8109a383>] ? mark_lock+0x1f3/0x400 [<ffffffff8109b497>] ? __lock_acquire+0xf07/0x1e10 [<ffffffff81012255>] ? native_sched_clock+0x15/0x70 [<ffffffff8109ca4d>] lock_acquire+0x9d/0x130 [<ffffffff81531e9f>] ? mld_ifc_timer_expire+0xff/0x280 [<ffffffff8109759d>] ? lock_release_holdtime+0x3d/0x1a0 [<ffffffff8157124b>] _raw_spin_lock_bh+0x3b/0x70 [<ffffffff81531e9f>] ? mld_ifc_timer_expire+0xff/0x280 [<ffffffff8157170b>] ? _raw_spin_unlock+0x2b/0x40 [<ffffffff81531e9f>] mld_ifc_timer_expire+0xff/0x280 [<ffffffff8106f2a9>] run_timer_softirq+0x179/0x3f0 [<ffffffff8106f21b>] ? run_timer_softirq+0xeb/0x3f0 [<ffffffff810122b9>] ? sched_clock+0x9/0x10 [<ffffffff81531da0>] ? mld_gq_timer_expire+0x30/0x30 [<ffffffff810666d0>] __do_softirq+0xc0/0x210 [<ffffffff8109455f>] ? tick_program_event+0x1f/0x30 [<ffffffff8157b3dc>] call_softirq+0x1c/0x30 [<ffffffff8100d42d>] do_softirq+0xad/0xe0 [<ffffffff81066afe>] irq_exit+0x9e/0xb0 [<ffffffff8157bd40>] smp_apic_timer_interrupt+0x70/0x9b [<ffffffff8157ab93>] apic_timer_interrupt+0x13/0x20 <EOI> [<ffffffff81571f14>] ? retint_restore_args+0x13/0x13 [<ffffffff810974a7>] ? lock_is_held+0x17/0xd0 [<ffffffff8149d857>] rt_do_flush+0x87/0x2a0 [<ffffffff814a16b6>] rt_cache_flush+0x46/0x60 [<ffffffff814e36e0>] fib_disable_ip+0x40/0x60 [<ffffffff814e5447>] fib_inetaddr_event+0xd7/0xe0 [<ffffffff81575c1c>] notifier_call_chain+0x8c/0xc0 [<ffffffff810896e8>] __blocking_notifier_call_chain+0x78/0xb0 [<ffffffff81089736>] blocking_notifier_call_chain+0x16/0x20 [<ffffffff814d8021>] __inet_del_ifa+0xf1/0x2e0 [<ffffffff814d8223>] inet_del_ifa+0x13/0x20 [<ffffffff814da731>] devinet_ioctl+0x501/0x800 [<ffffffff8108a3af>] ? local_clock+0x6f/0x80 [<ffffffff81575898>] ? do_page_fault+0x268/0x560 [<ffffffff814db508>] inet_ioctl+0x88/0xa0 [<ffffffff814541f0>] sock_do_ioctl+0x30/0x70 [<ffffffff814542a9>] sock_ioctl+0x79/0x2f0 [<ffffffff810dfe87>] ? __call_rcu+0xa7/0x190 [<ffffffff81188798>] do_vfs_ioctl+0x98/0x570 [<ffffffff8117737e>] ? fget_light+0x33e/0x430 [<ffffffff81571ef9>] ? retint_swapgs+0x13/0x1b [<ffffffff81188d01>] sys_ioctl+0x91/0xa0 [<ffffffff8157a142>] system_call_fastpath+0x16/0x1b Signed-off-by:Roland Dreier <roland@purestorage.com> Signed-off-by:
Shreyas N Bhatewara <sbhatewara@vmware.com> Signed-off-by:
Scott J. Goldman <scottjg@vmware.com> Signed-off-by:
David S. Miller <davem@davemloft.net>
e328d410
