Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1293536
| From | Kamal Mostafa <kamal@canonical.com> |
|---|---|
| Newsgroups | linux.kernel |
| Subject | [PATCH 3.13.y-ckt 01/78] tty: fix stall caused by missing memory barrier in drivers/tty/n_tty.c |
| Date | 2015-12-17 02:20 +0100 |
| Message-ID | <qGvoB-108-15@gated-at.bofh.it> (permalink) |
| References | <qGuVA-Ae-3@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
3.13.11-ckt32 -stable review patch. If anyone has any objections, please let me know.
------------------
From: Kosuke Tatsukawa <tatsu@ab.jp.nec.com>
commit e81107d4c6bd098878af9796b24edc8d4a9524fd upstream.
My colleague ran into a program stall on a x86_64 server, where
n_tty_read() was waiting for data even if there was data in the buffer
in the pty. kernel stack for the stuck process looks like below.
#0 [ffff88303d107b58] __schedule at ffffffff815c4b20
#1 [ffff88303d107bd0] schedule at ffffffff815c513e
#2 [ffff88303d107bf0] schedule_timeout at ffffffff815c7818
#3 [ffff88303d107ca0] wait_woken at ffffffff81096bd2
#4 [ffff88303d107ce0] n_tty_read at ffffffff8136fa23
#5 [ffff88303d107dd0] tty_read at ffffffff81368013
#6 [ffff88303d107e20] __vfs_read at ffffffff811a3704
#7 [ffff88303d107ec0] vfs_read at ffffffff811a3a57
#8 [ffff88303d107f00] sys_read at ffffffff811a4306
#9 [ffff88303d107f50] entry_SYSCALL_64_fastpath at ffffffff815c86d7
There seems to be two problems causing this issue.
First, in drivers/tty/n_tty.c, __receive_buf() stores the data and
updates ldata->commit_head using smp_store_release() and then checks
the wait queue using waitqueue_active(). However, since there is no
memory barrier, __receive_buf() could return without calling
wake_up_interactive_poll(), and at the same time, n_tty_read() could
start to wait in wait_woken() as in the following chart.
__receive_buf() n_tty_read()
------------------------------------------------------------------------
if (waitqueue_active(&tty->read_wait))
/* Memory operations issued after the
RELEASE may be completed before the
RELEASE operation has completed */
add_wait_queue(&tty->read_wait, &wait);
...
if (!input_available_p(tty, 0)) {
smp_store_release(&ldata->commit_head,
ldata->read_head);
...
timeout = wait_woken(&wait,
TASK_INTERRUPTIBLE, timeout);
------------------------------------------------------------------------
The second problem is that n_tty_read() also lacks a memory barrier
call and could also cause __receive_buf() to return without calling
wake_up_interactive_poll(), and n_tty_read() to wait in wait_woken()
as in the chart below.
__receive_buf() n_tty_read()
------------------------------------------------------------------------
spin_lock_irqsave(&q->lock, flags);
/* from add_wait_queue() */
...
if (!input_available_p(tty, 0)) {
/* Memory operations issued after the
RELEASE may be completed before the
RELEASE operation has completed */
smp_store_release(&ldata->commit_head,
ldata->read_head);
if (waitqueue_active(&tty->read_wait))
__add_wait_queue(q, wait);
spin_unlock_irqrestore(&q->lock,flags);
/* from add_wait_queue() */
...
timeout = wait_woken(&wait,
TASK_INTERRUPTIBLE, timeout);
------------------------------------------------------------------------
There are also other places in drivers/tty/n_tty.c which have similar
calls to waitqueue_active(), so instead of adding many memory barrier
calls, this patch simply removes the call to waitqueue_active(),
leaving just wake_up*() behind.
This fixes both problems because, even though the memory access before
or after the spinlocks in both wake_up*() and add_wait_queue() can
sneak into the critical section, it cannot go past it and the critical
section assures that they will be serialized (please see "INTER-CPU
ACQUIRING BARRIER EFFECTS" in Documentation/memory-barriers.txt for a
better explanation). Moreover, the resulting code is much simpler.
Latency measurement using a ping-pong test over a pty doesn't show any
visible performance drop.
Signed-off-by: Kosuke Tatsukawa <tatsu@ab.jp.nec.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
[jsalisbury: Backported to 3.13.y:
- Use wake_up_interruptible(), not wake_up_interruptible_poll()
- There are only two spurious uses of waitqueue_active() to remove]
BugLink: http://bugs.launchpad.net/bugs/1512815
Signed-off-by: Joseph Salisbury <joseph.salisbury@canonical.com>
Signed-off-by: Kamal Mostafa <kamal@canonical.com>
---
drivers/tty/n_tty.c | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
index 0b2e1ff..d40c9e3 100644
--- a/drivers/tty/n_tty.c
+++ b/drivers/tty/n_tty.c
@@ -1383,8 +1383,7 @@ handle_newline:
put_tty_queue(c, ldata);
ldata->canon_head = ldata->read_head;
kill_fasync(&tty->fasync, SIGIO, POLL_IN);
- if (waitqueue_active(&tty->read_wait))
- wake_up_interruptible(&tty->read_wait);
+ wake_up_interruptible(&tty->read_wait);
return 0;
}
}
@@ -1669,8 +1668,7 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
if ((!ldata->icanon && (read_cnt(ldata) >= ldata->minimum_to_wake)) ||
L_EXTPROC(tty)) {
kill_fasync(&tty->fasync, SIGIO, POLL_IN);
- if (waitqueue_active(&tty->read_wait))
- wake_up_interruptible(&tty->read_wait);
+ wake_up_interruptible(&tty->read_wait);
}
}
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
[3.13.y-ckt stable] Linux 3.13.11-ckt32 stable review Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 38/78] vfs: Make sendfile(2) killable even better Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 56/78] sched/core: Clear the root_domain cpumasks in init_rootdomain() Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 72/78] packet: only allow extra vlan len on ethernet devices Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 62/78] tcp: md5: fix lockdep annotation Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 63/78] ARM: dts: Kirkwood: Fix QNAP TS219 power-off Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 47/78] drm/radeon: make some dpm errors debug only Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 04/78] iio: lpc32xx_adc: fix warnings caused by enabling unprepared clock Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 27/78] USB: option: add XS Stick W100-2 from 4G Systems Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 68/78] net: mvneta: fix bit assignment for RX packet irq enable Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 75/78] net: ip6mr: fix static mfc/dev leaks on table destruction Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 08/78] x86/fpu: Fix 32-bit signal frame handling Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 64/78] isdn: Partially revert debug format string usage clean up Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 65/78] remoteproc: avoid stack overflow in debugfs file Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 76/78] unix: avoid use-after-free in ep_remove_wait_queue Kamal Mostafa <kamal@canonical.com> - 2015-12-17 01:50 +0100 [PATCH 3.13.y-ckt 35/78] powerpc/tm: Block signal return setting invalid MSR state Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 58/78] mmc: remove bondage between REQ_META and reliable write Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 55/78] wan/x25: Fix use-after-free in x25_asy_open_tty() Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 45/78] RDS: fix race condition when sending a message on unbound socket Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 46/78] btrfs: fix signed overflows in btrfs_sync_file Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 37/78] fix sysvfs symlinks Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 51/78] target: Fix race for SCF_COMPARE_AND_WRITE_POST checking Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 48/78] nfs: if we have no valid attrs, then don't declare the attribute cache valid Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 49/78] xen/gntdev: Grant maps should not be subject to NUMA balancing Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 54/78] Fix a memory leak in scsi_host_dev_release() Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 57/78] x86/signal: Fix restart_syscall number for x32 tasks Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 43/78] arm64: KVM: Fix AArch32 to AArch64 register mapping Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 50/78] iscsi-target: Fix rx_login_comp hang after login failure Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 60/78] usb: musb: core: fix order of arguments to ulpi write callback Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 53/78] block: Always check queue limits for cloned requests Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 03/78] stackprotector: Introduce CONFIG_CC_STACKPROTECTOR_STRONG Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 59/78] sctp: translate host order to network order when setting a hmacid Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 52/78] target: fix COMPARE_AND_WRITE non zero SGL offset data corruption Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:00 +0100 [PATCH 3.13.y-ckt 36/78] ARC: dw2 unwind: Remove falllback linear search thru FDE entries Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 31/78] fat: fix fake_offset handling on error path Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 42/78] ring-buffer: Update read stamp with first real commit on page Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 15/78] mac80211: mesh: fix call_rcu() usage Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 44/78] drm/radeon: make rv770_set_sw_state failures non-fatal Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 30/78] ALSA: hda - Apply HP headphone fixups more generically Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 33/78] parisc: Drop unused MADV_xxxK_PAGES flags from asm/mman.h Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 19/78] MIPS: KVM: Fix ASID restoration logic Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 23/78] xhci: Fix a race in usb2 LPM resume, blocking U3 for usb2 devices Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 32/78] kernel/signal.c: unexport sigsuspend() Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 21/78] MIPS: KVM: Uninit VCPU in vcpu_create error path Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 12/78] USB: serial: option: add support for Novatel MiFi USB620L Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 22/78] xhci: Workaround to get Intel xHCI reset working more reliably Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 16/78] usb: dwc3: gadget: let us set lower max_speed Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 40/78] nfs4: start callback_ident at idr 1 Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 14/78] ASoC: wm8962: correct addresses for HPF_C_0/1 Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 41/78] ALSA: hda - Fix headphone noise after Dell XPS 13 resume back from S3 Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 13/78] USB: ti_usb_3410_5052: Add Honeywell HGI80 ID Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 28/78] usblp: do not set TASK_INTERRUPTIBLE before lock Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 20/78] MIPS: KVM: Fix CACHE immediate offset sign extension Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 39/78] vfs: Avoid softlockups with sendfile(2) Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 25/78] ALSA: hda - Add fixup for Acer Aspire One Cloudbook 14 Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 29/78] mac: validate mac_partition is within sector Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 24/78] x86/cpu: Fix SMAP check in PVOPS environments Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 26/78] arm64: restore bogomips information in /proc/cpuinfo Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:10 +0100 [PATCH 3.13.y-ckt 05/78] iio:ad5064: Make sure ad5064_i2c_write() returns 0 on success Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:20 +0100 [PATCH 3.13.y-ckt 10/78] ALSA: usb-audio: prevent CH345 multiport output SysEx corruption Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:20 +0100 [PATCH 3.13.y-ckt 11/78] ALSA: usb-audio: work around CH345 input SysEx corruption Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:20 +0100 [PATCH 3.13.y-ckt 07/78] iio:ad7793: Fix ad7785 product ID Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:20 +0100 [PATCH 3.13.y-ckt 01/78] tty: fix stall caused by missing memory barrier in drivers/tty/n_tty.c Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:20 +0100 [PATCH 3.13.y-ckt 09/78] ALSA: usb-audio: add packet size quirk for the Medeli DD305 Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:20 +0100 [PATCH 3.13.y-ckt 06/78] iio: ad5064: Fix ad5629/ad5669 shift Kamal Mostafa <kamal@canonical.com> - 2015-12-17 02:20 +0100
csiph-web