Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1362029 > unrolled thread

[RFC][PATCH v6 0/2] printk: Make printk() completely async

Back to article view | Back to linux.kernel

Contents

  [RFC][PATCH v6 0/2] printk: Make printk() completely async Sergey Senozhatsky <sergey.senozhatsky@gmail.com> - 2016-03-21 18:30 +0100
    Re: [RFC][PATCH v6 0/2] printk: Make printk() completely async Jan Kara <jack@suse.cz> - 2016-03-22 07:50 +0100
      Re: [RFC][PATCH v6 0/2] printk: Make printk() completely async Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-03-22 09:00 +0100
        Re: [RFC][PATCH v6 0/2] printk: Make printk() completely async Jan Kara <jack@suse.cz> - 2016-03-22 09:20 +0100

#1362029 — [RFC][PATCH v6 0/2] printk: Make printk() completely async

Hello,

 RFC

 The patch set is based on slightly updated Jan Kara's patches.

This patch set makes printk() completely asynchronous: new messages
are getting upended to the kernel printk buffer, but instead of 'direct'
printing the actual print job is performed by a dedicated kthread.
This has the advantage that printing always happens from a schedulable
context and thus we don't lockup any particular CPU or even interrupts.

The patch set is against next-20160321

the series in total has 3 patches:
- printk: Make printk() completely async
- printk: Make wake_up_klogd_work_func() async
- printk: make console_unlock() async

per discussion, "printk: make console_unlock() async" will be posted
later on.

v7:
-- move wake_up_process out of logbuf lock (Jan, Byungchul)
-- do not disable async printk in recursion handling code.
-- rebase against next-20160321 (w/NMI patches)

v5:
-- make printk.synchronous RO (Petr)
-- make printing_func() correct and do not use wait_queue (Petr)
-- do not panic() when can't allocate printing thread (Petr)
-- do not wake_up_process() only in IRQ, prefer vprintk_emit() (Jan)
-- move wake_up_klogd_work_func() to a separate patch (Petr)
-- move wake_up_process() under logbuf lock so printk recursion logic can
   help us out
-- switch to sync_print mode if printk recursion occured
-- drop "printk: Skip messages on oops" patch

v4:
-- do not directly wake_up() the printing kthread from vprintk_emit(), need
   to go via IRQ->wake_up() to avoid sched deadlocks (Jan)

v3:
-- use a dedicated kthread for printing instead of using wq (Jan, Tetsuo, Tejun)

v2:
- use dedicated printk workqueue with WQ_MEM_RECLAIM bit
- fallback to system-wide workqueue only if allocation of printk_wq has
  failed
- do not use system_wq as a fallback wq. both console_lock() and onsole_unlock()
  can spend a significant amount of time; so we need to use system_long_wq.
- rework sync/!sync detection logic
  a) we can have deferred (in_sched) messages before we allocate printk_wq,
     so the only way to handle those messages is via IRQ context
  b) even in printk.synchronous mode, deferred messages must not be printed
     directly, and should go via IRQ context
  c) even if we allocated printk_wq and have !sync_printk mode, we must route
     deferred messages via IRQ context
- so this adds additional bool flags to vprint_emit() and introduces a new
  pending bit to `printk_pending'
- fix build on !PRINTK configs



Jan Kara (2):
  printk: Make printk() completely async
  printk: Make wake_up_klogd_work_func() async

 Documentation/kernel-parameters.txt |  10 ++
 kernel/printk/printk.c              | 216 ++++++++++++++++++++++++++----------
 2 files changed, 168 insertions(+), 58 deletions(-)

-- 
2.8.0.rc3.12.g047057b

[toc] | [next] | [standalone]

#1362351

Hi,

On Tue 22-03-16 02:25:28, Sergey Senozhatsky wrote:
>  The patch set is based on slightly updated Jan Kara's patches.
> 
> This patch set makes printk() completely asynchronous: new messages
> are getting upended to the kernel printk buffer, but instead of 'direct'
> printing the actual print job is performed by a dedicated kthread.
> This has the advantage that printing always happens from a schedulable
> context and thus we don't lockup any particular CPU or even interrupts.
> 
> The patch set is against next-20160321
> 
> the series in total has 3 patches:
> - printk: Make printk() completely async
> - printk: Make wake_up_klogd_work_func() async
> - printk: make console_unlock() async
> 
> per discussion, "printk: make console_unlock() async" will be posted
> later on.

Patches look good to me. I don't think you need to mention the
console_unlock() async patch when it is not part of the series.  BTW, you
seemed to have dropped my patch to skip if there are too many buffered
messages when oops is in progress. Any reason for that?

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [next] | [standalone]

#1362404

Hello Jan,

On (03/22/16 07:49), Jan Kara wrote:
> Hi,
> 
> On Tue 22-03-16 02:25:28, Sergey Senozhatsky wrote:
> >  The patch set is based on slightly updated Jan Kara's patches.
> > 
> > This patch set makes printk() completely asynchronous: new messages
> > are getting upended to the kernel printk buffer, but instead of 'direct'
> > printing the actual print job is performed by a dedicated kthread.
> > This has the advantage that printing always happens from a schedulable
> > context and thus we don't lockup any particular CPU or even interrupts.
> > 
> > The patch set is against next-20160321
> > 
> > the series in total has 3 patches:
> > - printk: Make printk() completely async
> > - printk: Make wake_up_klogd_work_func() async
> > - printk: make console_unlock() async
> > 
> > per discussion, "printk: make console_unlock() async" will be posted
> > later on.
> 
> Patches look good to me. I don't think you need to mention the
> console_unlock() async patch when it is not part of the series.

thanks.

> BTW, you seemed to have dropped my patch to skip if there are too
> many buffered messages when oops is in progress. Any reason for that?

I did. seems that my box has some problems with emails delivery recently:
http://marc.info/?l=linux-kernel&m=145821214108806

in short,
I'm afraid we can lose valuable data in some cases.
for example

: hardlockup detector with sysctl_hardlockup_all_cpu_backtrace.
:
: static void watchdog_overflow_callback(...)
: {
:         ...
:         if (is_hardlockup()) {
:         ...
:                 if (sysctl_hardlockup_all_cpu_backtrace &&
:                                !test_and_set_bit(0, &hardlockup_allcpu_dumped))
:                         trigger_allbutself_cpu_backtrace();
:
:                 nmi_panic(regs, msg);
:         ...
:         }
:         ...
: }
:
: trigger_allbutself_cpu_backtrace() can be much more than 100 lines.
: trigger_allbutself_cpu_backtrace() may or may not be implemented via
: NMI. for example arch/sparc/kernel/process_64.c

on a large enough system trigger_allbutself_cpu_backtrace() can easily be more
than 100 lines, and trigger_allbutself_cpu_backtrace() may not be able to print
the backtraces to console (for example this hardlockuped CPU owns the console_sem)
before CPU declares oops_in_progress in nmi_panic()->panic().

	-ss

[toc] | [prev] | [next] | [standalone]

#1362413

On Tue 22-03-16 16:57:38, Sergey Senozhatsky wrote:
> : hardlockup detector with sysctl_hardlockup_all_cpu_backtrace.
> :
> : static void watchdog_overflow_callback(...)
> : {
> :         ...
> :         if (is_hardlockup()) {
> :         ...
> :                 if (sysctl_hardlockup_all_cpu_backtrace &&
> :                                !test_and_set_bit(0, &hardlockup_allcpu_dumped))
> :                         trigger_allbutself_cpu_backtrace();
> :
> :                 nmi_panic(regs, msg);
> :         ...
> :         }
> :         ...
> : }
> :
> : trigger_allbutself_cpu_backtrace() can be much more than 100 lines.
> : trigger_allbutself_cpu_backtrace() may or may not be implemented via
> : NMI. for example arch/sparc/kernel/process_64.c
> 
> on a large enough system trigger_allbutself_cpu_backtrace() can easily be more
> than 100 lines, and trigger_allbutself_cpu_backtrace() may not be able to print
> the backtraces to console (for example this hardlockuped CPU owns the console_sem)
> before CPU declares oops_in_progress in nmi_panic()->panic().

OK, makes sense. We'd probably need more clever logic in detecting when to
skip. Let's drop it for now and revisit it later if it is an issue. Thanks
for explanation.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web