Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1696350
| From | Greg Kroah-Hartman <gregkh@linuxfoundation.org> |
|---|---|
| Newsgroups | linux.kernel |
| Subject | [PATCH 4.9 068/125] ipmi: use rcu lock around call to intf->handlers->sender() |
| Date | 2017-07-25 22:40 +0200 |
| Message-ID | <u7eMy-7zW-19@gated-at.bofh.it> (permalink) |
| References | <u7dGN-6Sz-9@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
4.9-stable review patch. If anyone has any objections, please let me know.
------------------
From: Tony Camuso <tcamuso@redhat.com>
commit cdea46566bb21ce309725a024208322a409055cc upstream.
A vendor with a system having more than 128 CPUs occasionally encounters
the following crash during shutdown. This is not an easily reproduceable
event, but the vendor was able to provide the following analysis of the
crash, which exhibits the same footprint each time.
crash> bt
PID: 0 TASK: ffff88017c70ce70 CPU: 5 COMMAND: "swapper/5"
#0 [ffff88085c143ac8] machine_kexec at ffffffff81059c8b
#1 [ffff88085c143b28] __crash_kexec at ffffffff811052e2
#2 [ffff88085c143bf8] crash_kexec at ffffffff811053d0
#3 [ffff88085c143c10] oops_end at ffffffff8168ef88
#4 [ffff88085c143c38] no_context at ffffffff8167ebb3
#5 [ffff88085c143c88] __bad_area_nosemaphore at ffffffff8167ec49
#6 [ffff88085c143cd0] bad_area_nosemaphore at ffffffff8167edb3
#7 [ffff88085c143ce0] __do_page_fault at ffffffff81691d1e
#8 [ffff88085c143d40] do_page_fault at ffffffff81691ec5
#9 [ffff88085c143d70] page_fault at ffffffff8168e188
[exception RIP: unknown or invalid address]
RIP: ffffffffa053c800 RSP: ffff88085c143e28 RFLAGS: 00010206
RAX: ffff88017c72bfd8 RBX: ffff88017a8dc000 RCX: ffff8810588b5ac8
RDX: ffff8810588b5a00 RSI: ffffffffa053c800 RDI: ffff8810588b5a00
RBP: ffff88085c143e58 R8: ffff88017c70d408 R9: ffff88017a8dc000
R10: 0000000000000002 R11: ffff88085c143da0 R12: ffff8810588b5ac8
R13: 0000000000000100 R14: ffffffffa053c800 R15: ffff8810588b5a00
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
<IRQ stack>
[exception RIP: cpuidle_enter_state+82]
RIP: ffffffff81514192 RSP: ffff88017c72be50 RFLAGS: 00000202
RAX: 0000001e4c3c6f16 RBX: 000000000000f8a0 RCX: 0000000000000018
RDX: 0000000225c17d03 RSI: ffff88017c72bfd8 RDI: 0000001e4c3c6f16
RBP: ffff88017c72be78 R8: 000000000000237e R9: 0000000000000018
R10: 0000000000002494 R11: 0000000000000001 R12: ffff88017c72be20
R13: ffff88085c14f8e0 R14: 0000000000000082 R15: 0000001e4c3bb400
ORIG_RAX: ffffffffffffff10 CS: 0010 SS: 0018
This is the corresponding stack trace
It has crashed because the area pointed with RIP extracted from timer
element is already removed during a shutdown process.
The function is smi_timeout().
And we think ffff8810588b5a00 in RDX is a parameter struct smi_info
crash> rd ffff8810588b5a00 20
ffff8810588b5a00: ffff8810588b6000 0000000000000000 .`.X............
ffff8810588b5a10: ffff880853264400 ffffffffa05417e0 .D&S......T.....
ffff8810588b5a20: 24a024a000000000 0000000000000000 .....$.$........
ffff8810588b5a30: 0000000000000000 0000000000000000 ................
ffff8810588b5a30: 0000000000000000 0000000000000000 ................
ffff8810588b5a40: ffffffffa053a040 ffffffffa053a060 @.S.....`.S.....
ffff8810588b5a50: 0000000000000000 0000000100000001 ................
ffff8810588b5a60: 0000000000000000 0000000000000e00 ................
ffff8810588b5a70: ffffffffa053a580 ffffffffa053a6e0 ..S.......S.....
ffff8810588b5a80: ffffffffa053a4a0 ffffffffa053a250 ..S.....P.S.....
ffff8810588b5a90: 0000000500000002 0000000000000000 ................
Unfortunately the top of this area is already detroyed by someone.
But because of two reasonns we think this is struct smi_info
1) The address included in between ffff8810588b5a70 and ffff8810588b5a80:
are inside of ipmi_si_intf.c see crash> module ffff88085779d2c0
2) We've found the area which point this.
It is offset 0x68 of ffff880859df4000
crash> rd ffff880859df4000 100
ffff880859df4000: 0000000000000000 0000000000000001 ................
ffff880859df4010: ffffffffa0535290 dead000000000200 .RS.............
ffff880859df4020: ffff880859df4020 ffff880859df4020 @.Y.... @.Y....
ffff880859df4030: 0000000000000002 0000000000100010 ................
ffff880859df4040: ffff880859df4040 ffff880859df4040 @@.Y....@@.Y....
ffff880859df4050: 0000000000000000 0000000000000000 ................
ffff880859df4060: 0000000000000000 ffff8810588b5a00 .........Z.X....
ffff880859df4070: 0000000000000001 ffff880859df4078 ........x@.Y....
If we regards it as struct ipmi_smi in shutdown process
it looks consistent.
The remedy for this apparent race is affixed below.
Signed-off-by: Tony Camuso <tcamuso@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This was first introduced in 7ea0ed2b5be817 ipmi: Make the
message handler easier to use for SMI interfaces
where some code was moved outside of the rcu_read_lock()
and the lock was not added.
Signed-off-by: Corey Minyard <cminyard@mvista.com>
---
drivers/char/ipmi/ipmi_msghandler.c | 5 +++++
1 file changed, 5 insertions(+)
--- a/drivers/char/ipmi/ipmi_msghandler.c
+++ b/drivers/char/ipmi/ipmi_msghandler.c
@@ -3877,6 +3877,9 @@ static void smi_recv_tasklet(unsigned lo
* because the lower layer is allowed to hold locks while calling
* message delivery.
*/
+
+ rcu_read_lock();
+
if (!run_to_completion)
spin_lock_irqsave(&intf->xmit_msgs_lock, flags);
if (intf->curr_msg == NULL && !intf->in_shutdown) {
@@ -3899,6 +3902,8 @@ static void smi_recv_tasklet(unsigned lo
if (newmsg)
intf->handlers->sender(intf->send_info, newmsg);
+ rcu_read_unlock();
+
handle_new_recv_msgs(intf);
}
Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
[PATCH 4.9 000/125] 4.9.40-stable review Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 068/125] ipmi: use rcu lock around call to intf->handlers->sender() Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 046/125] PCI: rockchip: Use normal register bank for config accessors Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 042/125] powerpc/asm: Mark cr0 as clobbered in mftb() Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 035/125] scsi: Add STARGET_CREATED_REMOVE state to scsi_target_state Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 044/125] af_key: Fix sadb_x_ipsecrequest parsing Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 037/125] xen/scsiback: Fix a TMR related use-after-free Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 053/125] usb: storage: return on error to avoid a null pointer dereference Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 030/125] ASoC: compress: Derive substream from stream based on direction Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 008/125] ath9k: fix tx99 bus error Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 027/125] wlcore: fix 64K page support Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 056/125] usb: renesas_usbhs: gadget: disable all eps when the driver stops Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 047/125] PCI/PM: Restore the status of PCI devices across hibernation Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 049/125] xhci: fix 20000ms port resume timeout Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 028/125] btrfs: Dont clear SGID when inheriting ACLs Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 033/125] PM / Domains: Fix unsafe iteration over modified list of domains Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 052/125] [media] mxl111sf: Fix driver to use heap allocate buffers for USB messages Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 009/125] ath9k: fix an invalid pointer dereference in ath9k_rng_stop() Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 041/125] powerpc: Fix emulation of mfocrf in emulate_step() Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 043/125] powerpc/mm/radix: Properly clear process table entry Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 039/125] powerpc/64: Fix atomic64_inc_not_zero() to return an int Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 034/125] scsi: ses: do not add a device to an enclosure if enclosure_add_links() fails. Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 029/125] igb: Explicitly select page 0 at initialization Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 032/125] PM / Domains: Fix unsafe iteration over modified list of domain providers Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 036/125] iscsi-target: Add login_keys_workaround attribute for non RFC initiators Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 054/125] USB: cdc-acm: add device-id for quirky printer Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 071/125] f2fs: sanity check size of nat and sit cache Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 050/125] xhci: Fix NULL pointer dereference when cleaning up streams for removed host Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 069/125] ipmi:ssif: Add missing unlock in error branch Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 026/125] Bluetooth: use constant time memory comparison for secret values Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 031/125] PM / Domains: Fix unsafe iteration over modified list of device links Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 040/125] powerpc: Fix emulation of mcrf in emulate_step() Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:40 +0200
[PATCH 4.9 023/125] perf intel-pt: Ensure never to set last_ip when packet count is zero Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 003/125] dm mpath: cleanup -Wbool-operation warning in choose_pgpath() Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 017/125] NFC: Add sockaddr length checks before accessing sa_family in bind handlers Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 018/125] perf intel-pt: Move decoder error setting into one condition Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 005/125] thermal: max77620: fix device-node reference imbalance Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 006/125] thermal: cpu_cooling: Avoid accessing potentially freed structures Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 004/125] [media] s5p-jpeg: dont return a random width/height Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 001/125] disable new gcc-7.1.1 warnings for now Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 015/125] nfc: Ensure presence of required attributes in the activate_target handler Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 020/125] perf intel-pt: Fix missing stack clear Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 002/125] [media] ir-core: fix gcc-7 warning on bool arithmetic Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 012/125] NFC: nfcmrvl: do not use device-managed resources Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 007/125] ath9k: fix tx99 use after free Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 021/125] perf intel-pt: Ensure IP is zero when state is INTEL_PT_STATE_NO_IP Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 011/125] NFC: nfcmrvl_uart: add missing tty-device sanity check Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 014/125] NFC: nfcmrvl: fix firmware-management initialisation Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 016/125] nfc: Fix the sockaddr length sanitization in llcp_sock_connect Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 019/125] perf intel-pt: Improve sample timestamp Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 010/125] NFC: fix broken device allocation Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
[PATCH 4.9 013/125] NFC: nfcmrvl: use nfc-device for firmware download Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-25 22:50 +0200
Re: [PATCH 4.9 000/125] 4.9.40-stable review Guenter Roeck <linux@roeck-us.net> - 2017-07-26 05:00 +0200
Re: [PATCH 4.9 000/125] 4.9.40-stable review Sumit Semwal <sumit.semwal@linaro.org> - 2017-07-26 16:20 +0200
Re: [PATCH 4.9 000/125] 4.9.40-stable review Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-26 22:00 +0200
Re: [PATCH 4.9 000/125] 4.9.40-stable review Shuah Khan <shuahkh@osg.samsung.com> - 2017-07-26 16:30 +0200
csiph-web