Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1466192

[PATCH 3.12 004/100] printk: do cond_resched() between lines while outputting to consoles

Show all headers | View raw

From: Tejun Heo <tj@kernel.org>

3.12-stable review patch.  If anyone has any objections, please let me know.

===============

commit 8d91f8b15361dfb438ab6eb3b319e2ded43458ff upstream.

@console_may_schedule tracks whether console_sem was acquired through
lock or trylock.  If the former, we're inside a sleepable context and
console_conditional_schedule() performs cond_resched().  This allows
console drivers which use console_lock for synchronization to yield
while performing time-consuming operations such as scrolling.

However, the actual console outputting is performed while holding
irq-safe logbuf_lock, so console_unlock() clears @console_may_schedule
before starting outputting lines.  Also, only a few drivers call
console_conditional_schedule() to begin with.  This means that when a
lot of lines need to be output by console_unlock(), for example on a
console registration, the task doing console_unlock() may not yield for
a long time on a non-preemptible kernel.

If this happens with a slow console devices, for example a serial
console, the outputting task may occupy the cpu for a very long time.
Long enough to trigger softlockup and/or RCU stall warnings, which in
turn pile more messages, sometimes enough to trigger the next cycle of
warnings incapacitating the system.

Fix it by making console_unlock() insert cond_resched() between lines if
@console_may_schedule.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Calvin Owens <calvinowens@fb.com>
Acked-by: Jan Kara <jack@suse.com>
Cc: Dave Jones <davej@codemonkey.org.uk>
Cc: Kyle McMartin <kyle@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Charles (Chas) Williams <ciwillia@brocade.com>
Signed-off-by: Jiri Slaby <jslaby@suse.cz>
---
 include/linux/console.h |  1 +
 kernel/panic.c          |  3 +--
 kernel/printk/printk.c  | 35 ++++++++++++++++++++++++++++++++++-
 3 files changed, 36 insertions(+), 3 deletions(-)

diff --git a/include/linux/console.h b/include/linux/console.h
index 7571a16bd653..ac1599bda9fc 100644
--- a/include/linux/console.h
+++ b/include/linux/console.h
@@ -150,6 +150,7 @@ extern int console_trylock(void);
 extern void console_unlock(void);
 extern void console_conditional_schedule(void);
 extern void console_unblank(void);
+extern void console_flush_on_panic(void);
 extern struct tty_driver *console_device(int *);
 extern void console_stop(struct console *);
 extern void console_start(struct console *);
diff --git a/kernel/panic.c b/kernel/panic.c
index 19cdd89ab78a..de5924c75b1b 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -140,8 +140,7 @@ void panic(const char *fmt, ...)
 	 * buffer.  Try to acquire the lock then release it regardless of the
 	 * result.  The release will also print the buffers out.
 	 */
-	console_trylock();
-	console_unlock();
+	console_flush_on_panic();
 
 	if (!panic_blink)
 		panic_blink = no_blink;
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index e736e50d2d08..44a8df70c0ec 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -2012,13 +2012,24 @@ void console_unlock(void)
 	static u64 seen_seq;
 	unsigned long flags;
 	bool wake_klogd = false;
-	bool retry;
+	bool do_cond_resched, retry;
 
 	if (console_suspended) {
 		up(&console_sem);
 		return;
 	}
 
+	/*
+	 * Console drivers are called under logbuf_lock, so
+	 * @console_may_schedule should be cleared before; however, we may
+	 * end up dumping a lot of lines, for example, if called from
+	 * console registration path, and should invoke cond_resched()
+	 * between lines if allowable.  Not doing so can cause a very long
+	 * scheduling stall on a slow console leading to RCU stall and
+	 * softlockup warnings which exacerbate the issue with more
+	 * messages practically incapacitating the system.
+	 */
+	do_cond_resched = console_may_schedule;
 	console_may_schedule = 0;
 
 	/* flush buffered message fragment immediately to console */
@@ -2075,6 +2086,9 @@ skip:
 		call_console_drivers(level, text, len);
 		start_critical_timings();
 		local_irq_restore(flags);
+
+		if (do_cond_resched)
+			cond_resched();
 	}
 	console_locked = 0;
 	mutex_release(&console_lock_dep_map, 1, _RET_IP_);
@@ -2143,6 +2157,25 @@ void console_unblank(void)
 	console_unlock();
 }
 
+/**
+ * console_flush_on_panic - flush console content on panic
+ *
+ * Immediately output all pending messages no matter what.
+ */
+void console_flush_on_panic(void)
+{
+	/*
+	 * If someone else is holding the console lock, trylock will fail
+	 * and may_schedule may be set.  Ignore and proceed to unlock so
+	 * that messages are flushed out.  As this can be called from any
+	 * context and we don't want to get preempted while flushing,
+	 * ensure may_schedule is cleared.
+	 */
+	console_trylock();
+	console_may_schedule = 0;
+	console_unlock();
+}
+
 /*
  * Return the console tty driver structure and its associated index
  */
-- 
2.9.3

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH 3.12 000/100] 3.12.63-stable review Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:20 +0200
  [PATCH 3.12 001/100] x86/mm: Add barriers and document switch_mm()-vs-flush synchronization Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
    [PATCH 3.12 002/100] mm: migrate dirty page without clear_page_dirty_for_io etc Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
    [PATCH 3.12 004/100] printk: do cond_resched() between lines while outputting to consoles Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
  [PATCH 3.12 009/100] libceph: set 'exists' flag for newly up osd Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
    [PATCH 3.12 024/100] ALSA: ctl: Stop notification after disconnection Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
    [PATCH 3.12 021/100] tty/vt/keyboard: fix OOB access in do_compute_shiftstate() Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
    [PATCH 3.12 015/100] iio: Fix error handling in iio_trigger_attach_poll_func Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
    [PATCH 3.12 020/100] iio:ad7266: Fix probe deferral for vref Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
    [PATCH 3.12 011/100] kvm: Fix irq route entries exceeding KVM_MAX_IRQ_ROUTES Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
    [PATCH 3.12 025/100] scsi: fix race between simultaneous decrements of ->host_failed Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
    [PATCH 3.12 022/100] ALSA: dummy: Fix a use-after-free at closing Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
    [PATCH 3.12 023/100] ALSA: au88x0: Fix calculation in vortex_wtdma_bufshift() Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
    [PATCH 3.12 016/100] staging: iio: accel: fix error check Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
    [PATCH 3.12 026/100] Fix reconnect to not defer smb3 session reconnect long after socket reconnect Jiri Slaby <jslaby@suse.cz> - 2016-08-19 09:40 +0200
  [PATCH 3.12 009/101] netfilter: x_tables: speed up jump target validation Jiri Slaby <jslaby@suse.cz> - 2016-08-19 10:00 +0200
  Re: [PATCH 3.12 000/100] 3.12.63-stable review Guenter Roeck <linux@roeck-us.net> - 2016-08-19 21:10 +0200

csiph-web