Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1322632

Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the debug code

Show all headers | View raw

On (01/29/16 15:37), Sergey Senozhatsky wrote:
> 
> panic()->console_panic_mode()->{for_each_console()->reset(), zap_locks()}->console_trelock()->console_unlock().

Hello,

This is not a final submission, just a RFC, so we can settle a better
plan. the patches are not signed off, have known problems (and likely
some unknown). I put a summary in here and send them out as a reply to
this email, so it'll be easier to review/comment/discuss.

patch 0001
***************
CPU stop IPI issued from panic() on CPUA, can leave console_sem locked
on CPUB if that cpu was holding the console_sem lock at the time when
IPI arrived. console_flush_on_panic() is trying to workaround it by
ignoring the return status of console_trylock() and unconditionally
executing console_unlock().

console_unlock() has a dependency on at least one more
lock - `logbuf_lock', which can be corrupted, for example, thus
console_unlock() may not be able to print anything afterall.

Introduce console_reset_on_panic() function to zap (re-init) printk
locks and call this function from panic().

WARNING
=======
This must be improved. console_reset_on_panic() is called before
smp_send_stop(), so:
a) we can have several CPU looping in console_unlock(), which is not
so critical.
b) we can re-init logbuf_lock while other CPU is holding it. Which
is more serious and needs to fixed.

The reason why console_reset_on_panic() is called this early is that
panicing CPU does pr_emerg("Kernel panic...") and dump_stack()
before it sends out smp_send_stop(). So if console_sem or logbug_lock,
or some console device driver lock is/are corrupted then panic() may
never smp_send_stop().




patch 0002
***************
Console driver(-s) can be in any state when CPU stop IPI
arrives from panic() issued on another CPU, so
console_flush_on_panic()->console_unlock() can call
con->write() callback on a locked console driver.

Introduce reset_console_drivers() that attempts to reset()
every console in via a console driver specific ->reset()
call.

Invoke reset_console_drivers() from console_reset_on_panic().

WARNING
=======
console_reset_on_panic() needs to be fixed.




patch 0003 -- detect recursive spin_dump() and panic() the system
***************
spin_dump() calls printk() which can attempt to reacquire the
'buggy' lock (one of printk's lock, or console device driver lock,
etc.) and thus spin_dump() will recursive into itself. Steal most
significant bit of spin_lock->owner_cpu to keep there a mark
that spin_dump() is in progress for that particular spin_lock.
spin_dump() will now set SPIN_DUMP_IN_PROGRESS bit at the
beginning of spin_dump() and clear it at the end, so it's
possible to detect recursive spin_dump() calls by checking if
lock's owner_cpu already has SPIN_DUMP_IN_PROGRESS bit already
set. panic() the system when spin_dump() recursion occurs.



	-ss

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Byungchul Park <byungchul.park@lge.com> - 2016-01-28 03:00 +0100
  Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-28 03:40 +0100
    RE: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the debug code "byungchul.park" <byungchul.park@lge.com> - 2016-01-28 06:00 +0100
      Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-28 07:10 +0100
        Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Byungchul Park <byungchul.park@lge.com> - 2016-01-28 09:30 +0100
          Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-28 11:50 +0100
            Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-28 12:00 +0100
              Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky@gmail.com> - 2016-01-28 16:50 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the  debug code Peter Hurley <peter@hurleysoftware.com> - 2016-01-29 00:10 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Byungchul Park <byungchul.park@lge.com> - 2016-01-29 01:00 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-29 02:00 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Byungchul Park <byungchul.park@lge.com> - 2016-01-29 04:10 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-29 05:10 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Byungchul Park <byungchul.park@lge.com> - 2016-01-29 13:20 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-29 01:30 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the  debug code Peter Hurley <peter@hurleysoftware.com> - 2016-01-29 05:40 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-29 06:30 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the  debug code Peter Hurley <peter@hurleysoftware.com> - 2016-01-29 06:50 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-29 07:20 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-29 07:40 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-31 13:30 +0100
                [PATCH 2/3] printk: introduce reset_console_drivers() Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-31 13:40 +0100
                Re: [PATCH 2/3] printk: introduce reset_console_drivers() kbuild test robot <lkp@intel.com> - 2016-01-31 13:50 +0100
                [PATCH 1/3] printk: introduce console_reset_on_panic() function Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-31 13:40 +0100
                [PATCH 3/3] spinlock_debug: panic on recursive lock spin_dump() Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-31 13:40 +0100
                Re: [PATCH 3/3] spinlock_debug: panic on recursive lock spin_dump() Sergey Senozhatsky <sergey.senozhatsky@gmail.com> - 2016-02-01 17:20 +0100
                Re: [PATCH 3/3] spinlock_debug: panic on recursive lock spin_dump() Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-02-02 09:00 +0100
                Re: [PATCH 1/3] printk: introduce console_reset_on_panic() function kbuild test robot <lkp@intel.com> - 2016-01-31 13:50 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Byungchul Park <byungchul.park@lge.com> - 2016-01-29 08:00 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-01-29 08:20 +0100
                Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in  the debug code Byungchul Park <byungchul.park@lge.com> - 2016-01-29 09:20 +0100

csiph-web