Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1702811 > unrolled thread
| Started by | Jiri Kosina <jikos@kernel.org> |
|---|---|
| First post | 2017-08-03 11:20 +0200 |
| Last post | 2017-08-03 13:40 +0200 |
| Articles | 6 — 3 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: x86/thermal: AB-BA dependency between mvm->mutex and tz->lock Jiri Kosina <jikos@kernel.org> - 2017-08-03 11:20 +0200
Re: [linuxwifi] x86/thermal: AB-BA dependency between mvm->mutex and tz->lock "Coelho, Luciano" <luciano.coelho@intel.com> - 2017-08-03 11:50 +0200
Re: [linuxwifi] x86/thermal: AB-BA dependency between mvm->mutex and tz->lock Kalle Valo <kvalo@codeaurora.org> - 2017-08-03 12:10 +0200
Re: [linuxwifi] x86/thermal: AB-BA dependency between mvm->mutex and tz->lock Jiri Kosina <jikos@kernel.org> - 2017-08-03 13:40 +0200
Re: [linuxwifi] x86/thermal: AB-BA dependency between mvm->mutex and tz->lock Jiri Kosina <jikos@kernel.org> - 2017-08-03 15:10 +0200
Re: [linuxwifi] x86/thermal: AB-BA dependency between mvm->mutex and tz->lock "Coelho, Luciano" <luciano.coelho@intel.com> - 2017-08-03 13:40 +0200
| From | Jiri Kosina <jikos@kernel.org> |
|---|---|
| Date | 2017-08-03 11:20 +0200 |
| Subject | Re: x86/thermal: AB-BA dependency between mvm->mutex and tz->lock |
| Message-ID | <uaksq-6mu-11@gated-at.bofh.it> |
On Mon, 31 Jul 2017, Jiri Kosina wrote:
> Hi,
>
> booting current Linus' tree, I'm seeing lockdep splat (see the end of this
> mail).
>
> Apparently, there is AB-BA between tz->lock and mvm->mutex through the CPU
> hotplug lock.
>
> The obivous depency is: thermal_zone_get_temp() acquires tz->lock, and
> then calls iwl_mvm_tzone_get_temp() (through tz->ops->get_temp()
> callback), which acquires mvm->mutex
>
> The less obvious dependency is primarily caused by iwl_op_mode_mvm_start()
> allocating workqueue (#2 stacktrace) while holding mvm->mutex (which is
> broken, because that mutex is being taken also from CPU hotplug callback
> path, hence the AB-BA).
As the "central" part of the dependency is being added by iwlwifi driver
(_iwl_pcie_rx_init() allocating workqueue while holding
trans_pcie->mutex), I'm adding iwlwifi folks as well to CC.
>
> ======================================================
> WARNING: possible circular locking dependency detected
> 4.13.0-rc2-00110-g0b5477d #347 Not tainted
> ------------------------------------------------------
> modprobe/881 is trying to acquire lock:
> (&mvm->mutex){+.+.+.}, at: [<ffffffffc0c504d2>] iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
>
> but task is already holding lock:
> (&tz->lock){+.+.+.}, at: [<ffffffff926a9351>] thermal_zone_get_temp+0x41/0x70
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #5 (&tz->lock){+.+.+.}:
> lock_acquire+0xbd/0x220
> __mutex_lock+0x6e/0x900
> mutex_lock_nested+0x1b/0x20
> thermal_zone_get_temp+0x41/0x70
> thermal_zone_device_update+0x3c/0x280
> thermal_zone_device_register+0x3b8/0x610
> pkg_thermal_cpu_online+0x20b/0x284 [x86_pkg_temp_thermal]
> cpuhp_invoke_callback+0xac/0x900
> cpuhp_thread_fun+0x79/0x160
> smpboot_thread_fn+0x156/0x220
> kthread+0x114/0x150
> ret_from_fork+0x2a/0x40
>
> -> #4 (cpuhp_state){+.+.+.}:
> lock_acquire+0xbd/0x220
> cpuhp_issue_call+0xea/0x170
> __cpuhp_setup_state_cpuslocked+0x12a/0x190
> __cpuhp_setup_state+0x46/0xc0
> page_writeback_init+0x43/0x67
> pagecache_init+0x39/0x3c
> start_kernel+0x45a/0x4ae
> x86_64_start_reservations+0x24/0x26
> x86_64_start_kernel+0x13d/0x14c
> verify_cpu+0x0/0xf1
>
> -> #3 (cpuhp_state_mutex){+.+.+.}:
> lock_acquire+0xbd/0x220
> __mutex_lock+0x6e/0x900
> mutex_lock_nested+0x1b/0x20
> __cpuhp_setup_state_cpuslocked+0x4f/0x190
> __cpuhp_setup_state+0x46/0xc0
> page_alloc_init+0x28/0x30
> start_kernel+0x186/0x4ae
> x86_64_start_reservations+0x24/0x26
> x86_64_start_kernel+0x13d/0x14c
> verify_cpu+0x0/0xf1
>
> -> #2 (cpu_hotplug_lock.rw_sem){++++++}:
> lock_acquire+0xbd/0x220
> cpus_read_lock+0x46/0x90
> apply_workqueue_attrs+0x17/0x50
> __alloc_workqueue_key+0x195/0x4d0
> _iwl_pcie_rx_init+0x384/0x390 [iwlwifi]
> iwl_pcie_rx_init+0x1e/0x380 [iwlwifi]
> iwl_trans_pcie_start_fw+0x295/0x6f0 [iwlwifi]
> iwl_mvm_load_ucode_wait_alive+0xe7/0x390 [iwlmvm]
> iwl_run_init_mvm_ucode+0x84/0x320 [iwlmvm]
> iwl_op_mode_mvm_start+0x964/0xd30 [iwlmvm]
> _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
> iwl_opmode_register+0xaa/0xd0 [iwlwifi]
> iwl_mvm_init+0x37/0x1000 [iwlmvm]
> do_one_initcall+0x51/0x1a9
> do_init_module+0x60/0x20e
> load_module+0x203f/0x2b50
> SYSC_finit_module+0x96/0xd0
> SyS_finit_module+0xe/0x10
> entry_SYSCALL_64_fastpath+0x23/0xc2
>
> -> #1 (&trans_pcie->mutex){+.+.+.}:
> lock_acquire+0xbd/0x220
> __mutex_lock+0x6e/0x900
> mutex_lock_nested+0x1b/0x20
> iwl_trans_pcie_start_fw+0x130/0x6f0 [iwlwifi]
> iwl_mvm_load_ucode_wait_alive+0xe7/0x390 [iwlmvm]
> iwl_run_init_mvm_ucode+0x84/0x320 [iwlmvm]
> iwl_op_mode_mvm_start+0x964/0xd30 [iwlmvm]
> _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
> iwl_opmode_register+0xaa/0xd0 [iwlwifi]
> iwl_mvm_init+0x37/0x1000 [iwlmvm]
> do_one_initcall+0x51/0x1a9
> do_init_module+0x60/0x20e
> load_module+0x203f/0x2b50
> SYSC_finit_module+0x96/0xd0
> SyS_finit_module+0xe/0x10
> entry_SYSCALL_64_fastpath+0x23/0xc2
>
> -> #0 (&mvm->mutex){+.+.+.}:
> __lock_acquire+0x13e1/0x1400
> lock_acquire+0xbd/0x220
> __mutex_lock+0x6e/0x900
> mutex_lock_nested+0x1b/0x20
> iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> thermal_zone_get_temp+0x51/0x70
> thermal_zone_device_update+0x3c/0x280
> thermal_zone_device_register+0x3b8/0x610
> iwl_mvm_thermal_initialize+0x1d1/0x3a0 [iwlmvm]
> iwl_op_mode_mvm_start+0xa1d/0xd30 [iwlmvm]
> _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
> iwl_opmode_register+0xaa/0xd0 [iwlwifi]
> iwl_mvm_init+0x37/0x1000 [iwlmvm]
> do_one_initcall+0x51/0x1a9
> do_init_module+0x60/0x20e
> load_module+0x203f/0x2b50
> SYSC_finit_module+0x96/0xd0
> SyS_finit_module+0xe/0x10
> entry_SYSCALL_64_fastpath+0x23/0xc2
>
> other info that might help us debug this:
>
> Chain exists of:
> &mvm->mutex --> cpuhp_state --> &tz->lock
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(&tz->lock);
> lock(cpuhp_state);
> lock(&tz->lock);
> lock(&mvm->mutex);
>
> *** DEADLOCK ***
>
> 2 locks held by modprobe/881:
> #0: (&iwlwifi_opmode_table_mtx){+.+.+.}, at: [<ffffffffc0aa2df4>] iwl_opmode_register+0x24/0xd0 [iwlwifi]
> #1: (&tz->lock){+.+.+.}, at: [<ffffffff926a9351>] thermal_zone_get_temp+0x41/0x70
>
> stack backtrace:
> CPU: 3 PID: 881 Comm: modprobe Not tainted 4.13.0-rc2-00110-g0b5477d #347
> Hardware name: LENOVO 20K5S22R00/20K5S22R00, BIOS R0IET38W (1.16 ) 05/31/2017
> Call Trace:
> dump_stack+0x85/0xc9
> print_circular_bug+0x1f9/0x207
> __lock_acquire+0x13e1/0x1400
> lock_acquire+0xbd/0x220
> ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> __mutex_lock+0x6e/0x900
> ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> ? thermal_zone_get_temp+0x41/0x70
> ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> ? thermal_zone_get_temp+0x41/0x70
> ? find_held_lock+0x39/0xb0
> mutex_lock_nested+0x1b/0x20
> iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> thermal_zone_get_temp+0x51/0x70
> thermal_zone_device_update+0x3c/0x280
> thermal_zone_device_register+0x3b8/0x610
> iwl_mvm_thermal_initialize+0x1d1/0x3a0 [iwlmvm]
> iwl_op_mode_mvm_start+0xa1d/0xd30 [iwlmvm]
> _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
> iwl_opmode_register+0xaa/0xd0 [iwlwifi]
> iwl_mvm_init+0x37/0x1000 [iwlmvm]
> ? 0xffffffffc0c87000
> do_one_initcall+0x51/0x1a9
> ? rcu_read_lock_sched_held+0x98/0xa0
> ? kmem_cache_alloc_trace+0x2a5/0x340
> do_init_module+0x60/0x20e
> load_module+0x203f/0x2b50
> ? __symbol_put+0x50/0x50
> SYSC_finit_module+0x96/0xd0
> SyS_finit_module+0xe/0x10
> entry_SYSCALL_64_fastpath+0x23/0xc2
> RIP: 0033:0x7f2ef067cc89
> RSP: 002b:00007ffea2ea3d78 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
> RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2ef067cc89
> RDX: 0000000000000000 RSI: 000000000041af06 RDI: 0000000000000001
> RBP: 0000000000000005 R08: 0000000000000000 R09: 000000000096b230
> R10: 0000000000000001 R11: 0000000000000246 R12: 00007ffea2ea2d80
> R13: 00007ffea2ea2d60 R14: 0000000000000005 R15: 000000000096f3c0
> thermal thermal_zone3: failed to read out thermal zone (-5)
>
>
> --
> Jiri Kosina
> SUSE Labs
>
--
Jiri Kosina
SUSE Labs
[toc] | [next] | [standalone]
| From | "Coelho, Luciano" <luciano.coelho@intel.com> |
|---|---|
| Date | 2017-08-03 11:50 +0200 |
| Subject | Re: [linuxwifi] x86/thermal: AB-BA dependency between mvm->mutex and tz->lock |
| Message-ID | <uakVs-6xq-21@gated-at.bofh.it> |
| In reply to | #1702811 |
On Thu, 2017-08-03 at 11:10 +0200, Jiri Kosina wrote:
> On Mon, 31 Jul 2017, Jiri Kosina wrote:
>
> > Hi,
> >
> > booting current Linus' tree, I'm seeing lockdep splat (see the end of this
> > mail).
> >
> > Apparently, there is AB-BA between tz->lock and mvm->mutex through the CPU
> > hotplug lock.
> >
> > The obivous depency is: thermal_zone_get_temp() acquires tz->lock, and
> > then calls iwl_mvm_tzone_get_temp() (through tz->ops->get_temp()
> > callback), which acquires mvm->mutex
> >
> > The less obvious dependency is primarily caused by iwl_op_mode_mvm_start()
> > allocating workqueue (#2 stacktrace) while holding mvm->mutex (which is
> > broken, because that mutex is being taken also from CPU hotplug callback
> > path, hence the AB-BA).
>
> As the "central" part of the dependency is being added by iwlwifi driver
> (_iwl_pcie_rx_init() allocating workqueue while holding
> trans_pcie->mutex), I'm adding iwlwifi folks as well to CC.
>
> >
> > ======================================================
> > WARNING: possible circular locking dependency detected
> > 4.13.0-rc2-00110-g0b5477d #347 Not tainted
> > ------------------------------------------------------
> > modprobe/881 is trying to acquire lock:
> > (&mvm->mutex){+.+.+.}, at: [<ffffffffc0c504d2>] iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> >
> > but task is already holding lock:
> > (&tz->lock){+.+.+.}, at: [<ffffffff926a9351>] thermal_zone_get_temp+0x41/0x70
> >
> > which lock already depends on the new lock.
> >
> >
> > the existing dependency chain (in reverse order) is:
> >
> > -> #5 (&tz->lock){+.+.+.}:
> > lock_acquire+0xbd/0x220
> > __mutex_lock+0x6e/0x900
> > mutex_lock_nested+0x1b/0x20
> > thermal_zone_get_temp+0x41/0x70
> > thermal_zone_device_update+0x3c/0x280
> > thermal_zone_device_register+0x3b8/0x610
> > pkg_thermal_cpu_online+0x20b/0x284 [x86_pkg_temp_thermal]
> > cpuhp_invoke_callback+0xac/0x900
> > cpuhp_thread_fun+0x79/0x160
> > smpboot_thread_fn+0x156/0x220
> > kthread+0x114/0x150
> > ret_from_fork+0x2a/0x40
> >
> > -> #4 (cpuhp_state){+.+.+.}:
> > lock_acquire+0xbd/0x220
> > cpuhp_issue_call+0xea/0x170
> > __cpuhp_setup_state_cpuslocked+0x12a/0x190
> > __cpuhp_setup_state+0x46/0xc0
> > page_writeback_init+0x43/0x67
> > pagecache_init+0x39/0x3c
> > start_kernel+0x45a/0x4ae
> > x86_64_start_reservations+0x24/0x26
> > x86_64_start_kernel+0x13d/0x14c
> > verify_cpu+0x0/0xf1
> >
> > -> #3 (cpuhp_state_mutex){+.+.+.}:
> > lock_acquire+0xbd/0x220
> > __mutex_lock+0x6e/0x900
> > mutex_lock_nested+0x1b/0x20
> > __cpuhp_setup_state_cpuslocked+0x4f/0x190
> > __cpuhp_setup_state+0x46/0xc0
> > page_alloc_init+0x28/0x30
> > start_kernel+0x186/0x4ae
> > x86_64_start_reservations+0x24/0x26
> > x86_64_start_kernel+0x13d/0x14c
> > verify_cpu+0x0/0xf1
> >
> > -> #2 (cpu_hotplug_lock.rw_sem){++++++}:
> > lock_acquire+0xbd/0x220
> > cpus_read_lock+0x46/0x90
> > apply_workqueue_attrs+0x17/0x50
> > __alloc_workqueue_key+0x195/0x4d0
> > _iwl_pcie_rx_init+0x384/0x390 [iwlwifi]
> > iwl_pcie_rx_init+0x1e/0x380 [iwlwifi]
> > iwl_trans_pcie_start_fw+0x295/0x6f0 [iwlwifi]
> > iwl_mvm_load_ucode_wait_alive+0xe7/0x390 [iwlmvm]
> > iwl_run_init_mvm_ucode+0x84/0x320 [iwlmvm]
> > iwl_op_mode_mvm_start+0x964/0xd30 [iwlmvm]
> > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
> > iwl_opmode_register+0xaa/0xd0 [iwlwifi]
> > iwl_mvm_init+0x37/0x1000 [iwlmvm]
> > do_one_initcall+0x51/0x1a9
> > do_init_module+0x60/0x20e
> > load_module+0x203f/0x2b50
> > SYSC_finit_module+0x96/0xd0
> > SyS_finit_module+0xe/0x10
> > entry_SYSCALL_64_fastpath+0x23/0xc2
> >
> > -> #1 (&trans_pcie->mutex){+.+.+.}:
> > lock_acquire+0xbd/0x220
> > __mutex_lock+0x6e/0x900
> > mutex_lock_nested+0x1b/0x20
> > iwl_trans_pcie_start_fw+0x130/0x6f0 [iwlwifi]
> > iwl_mvm_load_ucode_wait_alive+0xe7/0x390 [iwlmvm]
> > iwl_run_init_mvm_ucode+0x84/0x320 [iwlmvm]
> > iwl_op_mode_mvm_start+0x964/0xd30 [iwlmvm]
> > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
> > iwl_opmode_register+0xaa/0xd0 [iwlwifi]
> > iwl_mvm_init+0x37/0x1000 [iwlmvm]
> > do_one_initcall+0x51/0x1a9
> > do_init_module+0x60/0x20e
> > load_module+0x203f/0x2b50
> > SYSC_finit_module+0x96/0xd0
> > SyS_finit_module+0xe/0x10
> > entry_SYSCALL_64_fastpath+0x23/0xc2
> >
> > -> #0 (&mvm->mutex){+.+.+.}:
> > __lock_acquire+0x13e1/0x1400
> > lock_acquire+0xbd/0x220
> > __mutex_lock+0x6e/0x900
> > mutex_lock_nested+0x1b/0x20
> > iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> > thermal_zone_get_temp+0x51/0x70
> > thermal_zone_device_update+0x3c/0x280
> > thermal_zone_device_register+0x3b8/0x610
> > iwl_mvm_thermal_initialize+0x1d1/0x3a0 [iwlmvm]
> > iwl_op_mode_mvm_start+0xa1d/0xd30 [iwlmvm]
> > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
> > iwl_opmode_register+0xaa/0xd0 [iwlwifi]
> > iwl_mvm_init+0x37/0x1000 [iwlmvm]
> > do_one_initcall+0x51/0x1a9
> > do_init_module+0x60/0x20e
> > load_module+0x203f/0x2b50
> > SYSC_finit_module+0x96/0xd0
> > SyS_finit_module+0xe/0x10
> > entry_SYSCALL_64_fastpath+0x23/0xc2
> >
> > other info that might help us debug this:
> >
> > Chain exists of:
> > &mvm->mutex --> cpuhp_state --> &tz->lock
> >
> > Possible unsafe locking scenario:
> >
> > CPU0 CPU1
> > ---- ----
> > lock(&tz->lock);
> > lock(cpuhp_state);
> > lock(&tz->lock);
> > lock(&mvm->mutex);
> >
> > *** DEADLOCK ***
> >
> > 2 locks held by modprobe/881:
> > #0: (&iwlwifi_opmode_table_mtx){+.+.+.}, at: [<ffffffffc0aa2df4>] iwl_opmode_register+0x24/0xd0 [iwlwifi]
> > #1: (&tz->lock){+.+.+.}, at: [<ffffffff926a9351>] thermal_zone_get_temp+0x41/0x70
> >
> > stack backtrace:
> > CPU: 3 PID: 881 Comm: modprobe Not tainted 4.13.0-rc2-00110-g0b5477d #347
> > Hardware name: LENOVO 20K5S22R00/20K5S22R00, BIOS R0IET38W (1.16 ) 05/31/2017
> > Call Trace:
> > dump_stack+0x85/0xc9
> > print_circular_bug+0x1f9/0x207
> > __lock_acquire+0x13e1/0x1400
> > lock_acquire+0xbd/0x220
> > ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> > ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> > __mutex_lock+0x6e/0x900
> > ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> > ? thermal_zone_get_temp+0x41/0x70
> > ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> > ? thermal_zone_get_temp+0x41/0x70
> > ? find_held_lock+0x39/0xb0
> > mutex_lock_nested+0x1b/0x20
> > iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
> > thermal_zone_get_temp+0x51/0x70
> > thermal_zone_device_update+0x3c/0x280
> > thermal_zone_device_register+0x3b8/0x610
> > iwl_mvm_thermal_initialize+0x1d1/0x3a0 [iwlmvm]
> > iwl_op_mode_mvm_start+0xa1d/0xd30 [iwlmvm]
> > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
> > iwl_opmode_register+0xaa/0xd0 [iwlwifi]
> > iwl_mvm_init+0x37/0x1000 [iwlmvm]
> > ? 0xffffffffc0c87000
> > do_one_initcall+0x51/0x1a9
> > ? rcu_read_lock_sched_held+0x98/0xa0
> > ? kmem_cache_alloc_trace+0x2a5/0x340
> > do_init_module+0x60/0x20e
> > load_module+0x203f/0x2b50
> > ? __symbol_put+0x50/0x50
> > SYSC_finit_module+0x96/0xd0
> > SyS_finit_module+0xe/0x10
> > entry_SYSCALL_64_fastpath+0x23/0xc2
> > RIP: 0033:0x7f2ef067cc89
> > RSP: 002b:00007ffea2ea3d78 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
> > RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2ef067cc89
> > RDX: 0000000000000000 RSI: 000000000041af06 RDI: 0000000000000001
> > RBP: 0000000000000005 R08: 0000000000000000 R09: 000000000096b230
> > R10: 0000000000000001 R11: 0000000000000246 R12: 00007ffea2ea2d80
> > R13: 00007ffea2ea2d60 R14: 0000000000000005 R15: 000000000096f3c0
> > thermal thermal_zone3: failed to read out thermal zone (-5)
CCing David Weinehall who also just reported this to me.
We'll check this ASAP. Thanks for reporting!
--
Luca.
[toc] | [prev] | [next] | [standalone]
| From | Kalle Valo <kvalo@codeaurora.org> |
|---|---|
| Date | 2017-08-03 12:10 +0200 |
| Subject | Re: [linuxwifi] x86/thermal: AB-BA dependency between mvm->mutex and tz->lock |
| Message-ID | <ualeQ-6Uy-55@gated-at.bofh.it> |
| In reply to | #1702849 |
"Coelho, Luciano" <luciano.coelho@intel.com> writes:
> On Thu, 2017-08-03 at 11:10 +0200, Jiri Kosina wrote:
>> On Mon, 31 Jul 2017, Jiri Kosina wrote:
>>
>> > Hi,
>> >
>> > booting current Linus' tree, I'm seeing lockdep splat (see the end of this
>> > mail).
>> >
>> > Apparently, there is AB-BA between tz->lock and mvm->mutex through the CPU
>> > hotplug lock.
>> >
>> > The obivous depency is: thermal_zone_get_temp() acquires tz->lock, and
>> > then calls iwl_mvm_tzone_get_temp() (through tz->ops->get_temp()
>> > callback), which acquires mvm->mutex
>> >
>> > The less obvious dependency is primarily caused by iwl_op_mode_mvm_start()
>> > allocating workqueue (#2 stacktrace) while holding mvm->mutex (which is
>> > broken, because that mutex is being taken also from CPU hotplug callback
>> > path, hence the AB-BA).
>>
>> As the "central" part of the dependency is being added by iwlwifi driver
>> (_iwl_pcie_rx_init() allocating workqueue while holding
>> trans_pcie->mutex), I'm adding iwlwifi folks as well to CC.
>>
>> >
>> > ======================================================
>> > WARNING: possible circular locking dependency detected
>> > 4.13.0-rc2-00110-g0b5477d #347 Not tainted
>> > ------------------------------------------------------
>> > modprobe/881 is trying to acquire lock:
>> > (&mvm->mutex){+.+.+.}, at: [<ffffffffc0c504d2>] iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
>> >
>> > but task is already holding lock:
>> > (&tz->lock){+.+.+.}, at: [<ffffffff926a9351>] thermal_zone_get_temp+0x41/0x70
>> >
>> > which lock already depends on the new lock.
>> >
>> >
>> > the existing dependency chain (in reverse order) is:
>> >
>> > -> #5 (&tz->lock){+.+.+.}:
>> > lock_acquire+0xbd/0x220
>> > __mutex_lock+0x6e/0x900
>> > mutex_lock_nested+0x1b/0x20
>> > thermal_zone_get_temp+0x41/0x70
>> > thermal_zone_device_update+0x3c/0x280
>> > thermal_zone_device_register+0x3b8/0x610
>> > pkg_thermal_cpu_online+0x20b/0x284 [x86_pkg_temp_thermal]
>> > cpuhp_invoke_callback+0xac/0x900
>> > cpuhp_thread_fun+0x79/0x160
>> > smpboot_thread_fn+0x156/0x220
>> > kthread+0x114/0x150
>> > ret_from_fork+0x2a/0x40
>> >
>> > -> #4 (cpuhp_state){+.+.+.}:
>> > lock_acquire+0xbd/0x220
>> > cpuhp_issue_call+0xea/0x170
>> > __cpuhp_setup_state_cpuslocked+0x12a/0x190
>> > __cpuhp_setup_state+0x46/0xc0
>> > page_writeback_init+0x43/0x67
>> > pagecache_init+0x39/0x3c
>> > start_kernel+0x45a/0x4ae
>> > x86_64_start_reservations+0x24/0x26
>> > x86_64_start_kernel+0x13d/0x14c
>> > verify_cpu+0x0/0xf1
>> >
>> > -> #3 (cpuhp_state_mutex){+.+.+.}:
>> > lock_acquire+0xbd/0x220
>> > __mutex_lock+0x6e/0x900
>> > mutex_lock_nested+0x1b/0x20
>> > __cpuhp_setup_state_cpuslocked+0x4f/0x190
>> > __cpuhp_setup_state+0x46/0xc0
>> > page_alloc_init+0x28/0x30
>> > start_kernel+0x186/0x4ae
>> > x86_64_start_reservations+0x24/0x26
>> > x86_64_start_kernel+0x13d/0x14c
>> > verify_cpu+0x0/0xf1
>> >
>> > -> #2 (cpu_hotplug_lock.rw_sem){++++++}:
>> > lock_acquire+0xbd/0x220
>> > cpus_read_lock+0x46/0x90
>> > apply_workqueue_attrs+0x17/0x50
>> > __alloc_workqueue_key+0x195/0x4d0
>> > _iwl_pcie_rx_init+0x384/0x390 [iwlwifi]
>> > iwl_pcie_rx_init+0x1e/0x380 [iwlwifi]
>> > iwl_trans_pcie_start_fw+0x295/0x6f0 [iwlwifi]
>> > iwl_mvm_load_ucode_wait_alive+0xe7/0x390 [iwlmvm]
>> > iwl_run_init_mvm_ucode+0x84/0x320 [iwlmvm]
>> > iwl_op_mode_mvm_start+0x964/0xd30 [iwlmvm]
>> > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
>> > iwl_opmode_register+0xaa/0xd0 [iwlwifi]
>> > iwl_mvm_init+0x37/0x1000 [iwlmvm]
>> > do_one_initcall+0x51/0x1a9
>> > do_init_module+0x60/0x20e
>> > load_module+0x203f/0x2b50
>> > SYSC_finit_module+0x96/0xd0
>> > SyS_finit_module+0xe/0x10
>> > entry_SYSCALL_64_fastpath+0x23/0xc2
>> >
>> > -> #1 (&trans_pcie->mutex){+.+.+.}:
>> > lock_acquire+0xbd/0x220
>> > __mutex_lock+0x6e/0x900
>> > mutex_lock_nested+0x1b/0x20
>> > iwl_trans_pcie_start_fw+0x130/0x6f0 [iwlwifi]
>> > iwl_mvm_load_ucode_wait_alive+0xe7/0x390 [iwlmvm]
>> > iwl_run_init_mvm_ucode+0x84/0x320 [iwlmvm]
>> > iwl_op_mode_mvm_start+0x964/0xd30 [iwlmvm]
>> > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
>> > iwl_opmode_register+0xaa/0xd0 [iwlwifi]
>> > iwl_mvm_init+0x37/0x1000 [iwlmvm]
>> > do_one_initcall+0x51/0x1a9
>> > do_init_module+0x60/0x20e
>> > load_module+0x203f/0x2b50
>> > SYSC_finit_module+0x96/0xd0
>> > SyS_finit_module+0xe/0x10
>> > entry_SYSCALL_64_fastpath+0x23/0xc2
>> >
>> > -> #0 (&mvm->mutex){+.+.+.}:
>> > __lock_acquire+0x13e1/0x1400
>> > lock_acquire+0xbd/0x220
>> > __mutex_lock+0x6e/0x900
>> > mutex_lock_nested+0x1b/0x20
>> > iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
>> > thermal_zone_get_temp+0x51/0x70
>> > thermal_zone_device_update+0x3c/0x280
>> > thermal_zone_device_register+0x3b8/0x610
>> > iwl_mvm_thermal_initialize+0x1d1/0x3a0 [iwlmvm]
>> > iwl_op_mode_mvm_start+0xa1d/0xd30 [iwlmvm]
>> > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
>> > iwl_opmode_register+0xaa/0xd0 [iwlwifi]
>> > iwl_mvm_init+0x37/0x1000 [iwlmvm]
>> > do_one_initcall+0x51/0x1a9
>> > do_init_module+0x60/0x20e
>> > load_module+0x203f/0x2b50
>> > SYSC_finit_module+0x96/0xd0
>> > SyS_finit_module+0xe/0x10
>> > entry_SYSCALL_64_fastpath+0x23/0xc2
>> >
>> > other info that might help us debug this:
>> >
>> > Chain exists of:
>> > &mvm->mutex --> cpuhp_state --> &tz->lock
>> >
>> > Possible unsafe locking scenario:
>> >
>> > CPU0 CPU1
>> > ---- ----
>> > lock(&tz->lock);
>> > lock(cpuhp_state);
>> > lock(&tz->lock);
>> > lock(&mvm->mutex);
>> >
>> > *** DEADLOCK ***
>> >
>> > 2 locks held by modprobe/881:
>> > #0: (&iwlwifi_opmode_table_mtx){+.+.+.}, at: [<ffffffffc0aa2df4>] iwl_opmode_register+0x24/0xd0 [iwlwifi]
>> > #1: (&tz->lock){+.+.+.}, at: [<ffffffff926a9351>] thermal_zone_get_temp+0x41/0x70
>> >
>> > stack backtrace:
>> > CPU: 3 PID: 881 Comm: modprobe Not tainted 4.13.0-rc2-00110-g0b5477d #347
>> > Hardware name: LENOVO 20K5S22R00/20K5S22R00, BIOS R0IET38W (1.16 ) 05/31/2017
>> > Call Trace:
>> > dump_stack+0x85/0xc9
>> > print_circular_bug+0x1f9/0x207
>> > __lock_acquire+0x13e1/0x1400
>> > lock_acquire+0xbd/0x220
>> > ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
>> > ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
>> > __mutex_lock+0x6e/0x900
>> > ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
>> > ? thermal_zone_get_temp+0x41/0x70
>> > ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
>> > ? thermal_zone_get_temp+0x41/0x70
>> > ? find_held_lock+0x39/0xb0
>> > mutex_lock_nested+0x1b/0x20
>> > iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm]
>> > thermal_zone_get_temp+0x51/0x70
>> > thermal_zone_device_update+0x3c/0x280
>> > thermal_zone_device_register+0x3b8/0x610
>> > iwl_mvm_thermal_initialize+0x1d1/0x3a0 [iwlmvm]
>> > iwl_op_mode_mvm_start+0xa1d/0xd30 [iwlmvm]
>> > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
>> > iwl_opmode_register+0xaa/0xd0 [iwlwifi]
>> > iwl_mvm_init+0x37/0x1000 [iwlmvm]
>> > ? 0xffffffffc0c87000
>> > do_one_initcall+0x51/0x1a9
>> > ? rcu_read_lock_sched_held+0x98/0xa0
>> > ? kmem_cache_alloc_trace+0x2a5/0x340
>> > do_init_module+0x60/0x20e
>> > load_module+0x203f/0x2b50
>> > ? __symbol_put+0x50/0x50
>> > SYSC_finit_module+0x96/0xd0
>> > SyS_finit_module+0xe/0x10
>> > entry_SYSCALL_64_fastpath+0x23/0xc2
>> > RIP: 0033:0x7f2ef067cc89
>> > RSP: 002b:00007ffea2ea3d78 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
>> > RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2ef067cc89
>> > RDX: 0000000000000000 RSI: 000000000041af06 RDI: 0000000000000001
>> > RBP: 0000000000000005 R08: 0000000000000000 R09: 000000000096b230
>> > R10: 0000000000000001 R11: 0000000000000246 R12: 00007ffea2ea2d80
>> > R13: 00007ffea2ea2d60 R14: 0000000000000005 R15: 000000000096f3c0
>> > thermal thermal_zone3: failed to read out thermal zone (-5)
>
> CCing David Weinehall who also just reported this to me.
>
> We'll check this ASAP. Thanks for reporting!
Adding linux-wireless also to the loop.
--
Kalle Valo
[toc] | [prev] | [next] | [standalone]
| From | Jiri Kosina <jikos@kernel.org> |
|---|---|
| Date | 2017-08-03 13:40 +0200 |
| Subject | Re: [linuxwifi] x86/thermal: AB-BA dependency between mvm->mutex and tz->lock |
| Message-ID | <uamDU-7Ls-23@gated-at.bofh.it> |
| In reply to | #1702891 |
On Thu, 3 Aug 2017, Coelho, Luciano wrote:
> Okay, so as I understand it the problem has been there for a long time,
> but the splat is only coming up now because of Thomas' patch that adds
> the lockdep map[1], right?
Yeah, sorry, forgot to mention that pre-49dfe2a67797 kernels wouldn't
produce this, as there would not be aware of the fact that
cpus_read_lock() is actually semantically a lock.
> I see the workqueue allocation you mentioned. I'll try to move this
> allocation out of the mutex and see how it goes.
I have been briefly looking into this as well -- it'll basically have to
be moved out of the trans_pcie->mutex context, but
(a) I'm not sure whether that's actually safe
(b) iwl_pcie_rx_reuse_rbd() (which is where corresponding work is being
queued) is not a proper context either (it's atomic context)
Thanks,
--
Jiri Kosina
SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Jiri Kosina <jikos@kernel.org> |
|---|---|
| Date | 2017-08-03 15:10 +0200 |
| Subject | Re: [linuxwifi] x86/thermal: AB-BA dependency between mvm->mutex and tz->lock |
| Message-ID | <uao32-n3-29@gated-at.bofh.it> |
| In reply to | #1702970 |
On Thu, 3 Aug 2017, Jiri Kosina wrote: > > I see the workqueue allocation you mentioned. I'll try to move this > > allocation out of the mutex and see how it goes. > > I have been briefly looking into this as well -- it'll basically have to > be moved out of the trans_pcie->mutex context, but > > (a) I'm not sure whether that's actually safe > (b) iwl_pcie_rx_reuse_rbd() (which is where corresponding work is being > queued) is not a proper context either (it's atomic context) Actually moving it out of trans_pcie->mutex is likely not to be enough, the dependency would still be there, just the graph will have one vertex less, with the dependency going directly from mvm mutex to cpu_hotplug_lock. -- Jiri Kosina SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | "Coelho, Luciano" <luciano.coelho@intel.com> |
|---|---|
| Date | 2017-08-03 13:40 +0200 |
| Subject | Re: [linuxwifi] x86/thermal: AB-BA dependency between mvm->mutex and tz->lock |
| Message-ID | <uamDU-7Ls-25@gated-at.bofh.it> |
| In reply to | #1702891 |
On Thu, 2017-08-03 at 13:02 +0300, Kalle Valo wrote:
> "Coelho, Luciano" <luciano.coelho@intel.com> writes:
>
> > On Thu, 2017-08-03 at 11:10 +0200, Jiri Kosina wrote:
> > > On Mon, 31 Jul 2017, Jiri Kosina wrote:
> > >
> > > > Hi,
> > > >
> > > > booting current Linus' tree, I'm seeing lockdep splat (see the end of this
> > > > mail).
> > > >
> > > > Apparently, there is AB-BA between tz->lock and mvm->mutex through the CPU
> > > > hotplug lock.
> > > >
> > > > The obivous depency is: thermal_zone_get_temp() acquires tz->lock, and
> > > > then calls iwl_mvm_tzone_get_temp() (through tz->ops->get_temp()
> > > > callback), which acquires mvm->mutex
> > > >
> > > > The less obvious dependency is primarily caused by iwl_op_mode_mvm_start()
> > > > allocating workqueue (#2 stacktrace) while holding mvm->mutex (which is
> > > > broken, because that mutex is being taken also from CPU hotplug callback
> > > > path, hence the AB-BA).
> > >
> > > As the "central" part of the dependency is being added by iwlwifi driver
> > > (_iwl_pcie_rx_init() allocating workqueue while holding
> > > trans_pcie->mutex), I'm adding iwlwifi folks as well to CC.
[...]
> > > > -> #2 (cpu_hotplug_lock.rw_sem){++++++}:
> > > > lock_acquire+0xbd/0x220
> > > > cpus_read_lock+0x46/0x90
> > > > apply_workqueue_attrs+0x17/0x50
> > > > __alloc_workqueue_key+0x195/0x4d0
> > > > _iwl_pcie_rx_init+0x384/0x390 [iwlwifi]
> > > > iwl_pcie_rx_init+0x1e/0x380 [iwlwifi]
> > > > iwl_trans_pcie_start_fw+0x295/0x6f0 [iwlwifi]
> > > > iwl_mvm_load_ucode_wait_alive+0xe7/0x390 [iwlmvm]
> > > > iwl_run_init_mvm_ucode+0x84/0x320 [iwlmvm]
> > > > iwl_op_mode_mvm_start+0x964/0xd30 [iwlmvm]
> > > > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi]
> > > > iwl_opmode_register+0xaa/0xd0 [iwlwifi]
> > > > iwl_mvm_init+0x37/0x1000 [iwlmvm]
> > > > do_one_initcall+0x51/0x1a9
> > > > do_init_module+0x60/0x20e
> > > > load_module+0x203f/0x2b50
> > > > SYSC_finit_module+0x96/0xd0
> > > > SyS_finit_module+0xe/0x10
> > > > entry_SYSCALL_64_fastpath+0x23/0xc2
Okay, so as I understand it the problem has been there for a long time,
but the splat is only coming up now because of Thomas' patch that adds
the lockdep map[1], right?
I see the workqueue allocation you mentioned. I'll try to move this
allocation out of the mutex and see how it goes.
[1] http://lkml.kernel.org/r/20170524081549.709375845@linutronix.de
--
Cheers,
Luca.
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web