Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1241694 > unrolled thread

[PATCH] panic: release stale console lock to always get the logbuf printed out

Back to article view | Back to linux.kernel

Contents

  [PATCH] panic: release stale console lock to always get the logbuf printed out Vitaly Kuznetsov <vkuznets@redhat.com> - 2015-10-07 19:10 +0200
    Re: [PATCH] panic: release stale console lock to always get the  logbuf printed out Andrew Morton <akpm@linux-foundation.org> - 2015-10-08 00:40 +0200
      Re: [PATCH] panic: release stale console lock to always get the  logbuf printed out Jan Kara <jack@suse.cz> - 2015-10-08 11:10 +0200
        Re: [PATCH] panic: release stale console lock to always get the logbuf printed out Vitaly Kuznetsov <vkuznets@redhat.com> - 2015-10-08 12:10 +0200
          Re: [PATCH] panic: release stale console lock to always get the  logbuf printed out Andrew Morton <akpm@linux-foundation.org> - 2015-10-08 23:00 +0200
            Re: [PATCH] panic: release stale console lock to always get the logbuf printed out Vitaly Kuznetsov <vkuznets@redhat.com> - 2015-10-09 12:20 +0200
              Re: [PATCH] panic: release stale console lock to always get the logbuf printed out Vitaly Kuznetsov <vkuznets@redhat.com> - 2015-10-09 14:50 +0200
        Re: Re: [PATCH] panic: release stale console lock to always get the  logbuf printed out kbuild test robot <lkp@intel.com> - 2015-10-12 05:10 +0200
      Re: [PATCH] panic: release stale console lock to always get the logbuf printed out Vitaly Kuznetsov <vkuznets@redhat.com> - 2015-10-08 12:00 +0200
        Re: [PATCH] panic: release stale console lock to always get the  logbuf printed out Andrew Morton <akpm@linux-foundation.org> - 2015-10-08 23:00 +0200
          Re: [PATCH] panic: release stale console lock to always get the logbuf printed out Vitaly Kuznetsov <vkuznets@redhat.com> - 2015-10-09 12:20 +0200

#1241694 — [PATCH] panic: release stale console lock to always get the logbuf printed out

In some cases we may end up killing the CPU holding the console lock
while still having valuable data in logbuf. E.g. I'm observing the
following:
- A crash is happening on one CPU and console_unlock() is being called on
  some other.
- console_unlock() tries to print out the buffer before releasing the lock
  and on slow console it takes time.
- in the meanwhile crashing CPU does lots of printk()-s with valuable data
  (which go to the logbuf) and sends IPIs to all other CPUs.
- console_unlock() finishes printing previous chunk and enables interrupts
  before trying to print out the rest, the CPU catches the IPI and never
  releases console lock.
This is not the only possible case: in VT/fb subsystems we have many other
console_lock()/console_unlock() users. Non-masked interrupts (or receiving
NMI in case of extreme slowness) will have the same result. Getting the
whole console buffer printed out on crash should be top priority.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 kernel/panic.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/panic.c b/kernel/panic.c
index 04e91ff..f94525f 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -23,6 +23,7 @@
 #include <linux/sysrq.h>
 #include <linux/init.h>
 #include <linux/nmi.h>
+#include <linux/console.h>
 
 #define PANIC_TIMER_STEP 100
 #define PANIC_BLINK_SPD 18
@@ -147,6 +148,15 @@ void panic(const char *fmt, ...)
 
 	bust_spinlocks(0);
 
+	/*
+	 * We may have ended up killing the CPU holding the lock and still have
+	 * some valuable data in console buffer. Try to acquire the lock and
+	 * release it regardless of the result. The release will also print the
+	 * buffers out.
+	 */
+	console_trylock();
+	console_unlock();
+
 	if (!panic_blink)
 		panic_blink = no_blink;
 
-- 
2.4.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1241831 — Re: [PATCH] panic: release stale console lock to always get the logbuf printed out

(cc Jan)

On Wed,  7 Oct 2015 19:02:22 +0200 Vitaly Kuznetsov <vkuznets@redhat.com> wrote:

> In some cases we may end up killing the CPU holding the console lock
> while still having valuable data in logbuf. E.g. I'm observing the
> following:
> - A crash is happening on one CPU and console_unlock() is being called on
>   some other.
> - console_unlock() tries to print out the buffer before releasing the lock
>   and on slow console it takes time.
> - in the meanwhile crashing CPU does lots of printk()-s with valuable data
>   (which go to the logbuf) and sends IPIs to all other CPUs.
> - console_unlock() finishes printing previous chunk and enables interrupts
>   before trying to print out the rest, the CPU catches the IPI and never
>   releases console lock.

Why doesn't the lock-owning CPU release the console lock?  Because it
was stopped by smp_send_stop() in panic()?

I don't recall why we stop CPUs in panic(), and of course we didn't
document the reason.  I guess it makes sense from the "what else can we
do" point of view, but I wonder if we can just do it later on - that
would fix this problem?

(dumb aside: why doesn't smp_send_stop() stop the calling CPU?)

> This is not the only possible case: in VT/fb subsystems we have many other
> console_lock()/console_unlock() users. Non-masked interrupts (or receiving
> NMI in case of extreme slowness) will have the same result. Getting the
> whole console buffer printed out on crash should be top priority.

Yes, this is a pretty big hole in the logic.

> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -23,6 +23,7 @@
>  #include <linux/sysrq.h>
>  #include <linux/init.h>
>  #include <linux/nmi.h>
> +#include <linux/console.h>
>  
>  #define PANIC_TIMER_STEP 100
>  #define PANIC_BLINK_SPD 18
> @@ -147,6 +148,15 @@ void panic(const char *fmt, ...)
>  
>  	bust_spinlocks(0);
>  
> +	/*
> +	 * We may have ended up killing the CPU holding the lock and still have
> +	 * some valuable data in console buffer. Try to acquire the lock and
> +	 * release it regardless of the result. The release will also print the
> +	 * buffers out.
> +	 */
> +	console_trylock();
> +	console_unlock();
> +

"killing the CPU" is a bit vague.  How's this look?

--- a/kernel/panic.c~panic-release-stale-console-lock-to-always-get-the-logbuf-printed-out-fix
+++ a/kernel/panic.c
@@ -149,10 +149,10 @@ void panic(const char *fmt, ...)
 	bust_spinlocks(0);
 
 	/*
-	 * We may have ended up killing the CPU holding the lock and still have
-	 * some valuable data in console buffer. Try to acquire the lock and
-	 * release it regardless of the result. The release will also print the
-	 * buffers out.
+	 * We may have ended up stopping the CPU holding the lock (in
+	 * smp_send_stop()) while still having some valuable data in the console
+	 * buffer.  Try to acquire the lock then release it regardless of the
+	 * result.  The release will also print the buffers out.
 	 */
 	console_trylock();
 	console_unlock();
_


Does the console_trylock() guarantee that the console lock is now held?
If the console_lock-holding CPU is still running then there's a window
where the above code could enter console_unlock() when nobody's holding
console_lock.  If smp_send_stop() always works (synchronously) then
that won't happen.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1242096 — Re: [PATCH] panic: release stale console lock to always get the logbuf printed out

[Multipart message — attachments visible in raw view] — view raw

On Wed 07-10-15 15:34:08, Andrew Morton wrote:
> (cc Jan)
> 
> On Wed,  7 Oct 2015 19:02:22 +0200 Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
> 
> > In some cases we may end up killing the CPU holding the console lock
> > while still having valuable data in logbuf. E.g. I'm observing the
> > following:
> > - A crash is happening on one CPU and console_unlock() is being called on
> >   some other.
> > - console_unlock() tries to print out the buffer before releasing the lock
> >   and on slow console it takes time.
> > - in the meanwhile crashing CPU does lots of printk()-s with valuable data
> >   (which go to the logbuf) and sends IPIs to all other CPUs.
> > - console_unlock() finishes printing previous chunk and enables interrupts
> >   before trying to print out the rest, the CPU catches the IPI and never
> >   releases console lock.
> 
> Why doesn't the lock-owning CPU release the console lock?  Because it
> was stopped by smp_send_stop() in panic()?
> 
> I don't recall why we stop CPUs in panic(), and of course we didn't
> document the reason.  I guess it makes sense from the "what else can we
> do" point of view, but I wonder if we can just do it later on - that
> would fix this problem?
> 
> (dumb aside: why doesn't smp_send_stop() stop the calling CPU?)
> 
> > This is not the only possible case: in VT/fb subsystems we have many other
> > console_lock()/console_unlock() users. Non-masked interrupts (or receiving
> > NMI in case of extreme slowness) will have the same result. Getting the
> > whole console buffer printed out on crash should be top priority.
> 
> Yes, this is a pretty big hole in the logic.
> 
> > --- a/kernel/panic.c
> > +++ b/kernel/panic.c
> > @@ -23,6 +23,7 @@
> >  #include <linux/sysrq.h>
> >  #include <linux/init.h>
> >  #include <linux/nmi.h>
> > +#include <linux/console.h>
> >  
> >  #define PANIC_TIMER_STEP 100
> >  #define PANIC_BLINK_SPD 18
> > @@ -147,6 +148,15 @@ void panic(const char *fmt, ...)
> >  
> >  	bust_spinlocks(0);
> >  
> > +	/*
> > +	 * We may have ended up killing the CPU holding the lock and still have
> > +	 * some valuable data in console buffer. Try to acquire the lock and
> > +	 * release it regardless of the result. The release will also print the
> > +	 * buffers out.
> > +	 */
> > +	console_trylock();
> > +	console_unlock();
> > +
> 
> "killing the CPU" is a bit vague.  How's this look?
> 
> --- a/kernel/panic.c~panic-release-stale-console-lock-to-always-get-the-logbuf-printed-out-fix
> +++ a/kernel/panic.c
> @@ -149,10 +149,10 @@ void panic(const char *fmt, ...)
>  	bust_spinlocks(0);
>  
>  	/*
> -	 * We may have ended up killing the CPU holding the lock and still have
> -	 * some valuable data in console buffer. Try to acquire the lock and
> -	 * release it regardless of the result. The release will also print the
> -	 * buffers out.
> +	 * We may have ended up stopping the CPU holding the lock (in
> +	 * smp_send_stop()) while still having some valuable data in the console
> +	 * buffer.  Try to acquire the lock then release it regardless of the
> +	 * result.  The release will also print the buffers out.
>  	 */
>  	console_trylock();
>  	console_unlock();
> _
> 
> 
> Does the console_trylock() guarantee that the console lock is now held?
> If the console_lock-holding CPU is still running then there's a window
> where the above code could enter console_unlock() when nobody's holding
> console_lock.  If smp_send_stop() always works (synchronously) then
> that won't happen.

We have this mechanism using zap_locks() in kernel/printk/printk.c when
crash happens on the CPU holding console_sem. Can't we use the same
mechanism for this case? Something like adding:

	zap_locks();
	console_lock();
	console_unlock();

to panic? If we picked up patch "kernel: Avoid softlockups in
stop_machine() during heavy printing" from my series (it's completely
independent, I've attached the latest version), the result would look less
hacky to me (attached). Thoughts?

Warning, the result is untested... I can do some testing and official
posting of the two patches if people agree we want to got down this path.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [next] | [standalone]

#1242142

Jan Kara <jack@suse.cz> writes:

>> --- a/kernel/panic.c~panic-release-stale-console-lock-to-always-get-the-logbuf-printed-out-fix
>> +++ a/kernel/panic.c
>> @@ -149,10 +149,10 @@ void panic(const char *fmt, ...)
>>  	bust_spinlocks(0);
>>  
>>  	/*
>> -	 * We may have ended up killing the CPU holding the lock and still have
>> -	 * some valuable data in console buffer. Try to acquire the lock and
>> -	 * release it regardless of the result. The release will also print the
>> -	 * buffers out.
>> +	 * We may have ended up stopping the CPU holding the lock (in
>> +	 * smp_send_stop()) while still having some valuable data in the console
>> +	 * buffer.  Try to acquire the lock then release it regardless of the
>> +	 * result.  The release will also print the buffers out.
>>  	 */
>>  	console_trylock();
>>  	console_unlock();
>> _
>> 
>> 
>> Does the console_trylock() guarantee that the console lock is now held?
>> If the console_lock-holding CPU is still running then there's a window
>> where the above code could enter console_unlock() when nobody's holding
>> console_lock.  If smp_send_stop() always works (synchronously) then
>> that won't happen.
>
> We have this mechanism using zap_locks() in kernel/printk/printk.c when
> crash happens on the CPU holding console_sem. Can't we use the same
> mechanism for this case? Something like adding:
>
> 	zap_locks();
> 	console_lock();
> 	console_unlock();
>
> to panic?

I thought it doesn't really matter who holds the lock at this point --
all other cpus were already stopped with IPIs/NMIs.

> If we picked up patch "kernel: Avoid softlockups in
> stop_machine() during heavy printing" from my series (it's completely
> independent, I've attached the latest version), the result would look less
> hacky to me (attached). Thoughts?

Haven't tested it but should also work...

Thanks,

-- 
  Vitaly
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1242730 — Re: [PATCH] panic: release stale console lock to always get the logbuf printed out

On Thu, 08 Oct 2015 12:03:25 +0200 Vitaly Kuznetsov <vkuznets@redhat.com> wrote:

> > If we picked up patch "kernel: Avoid softlockups in
> > stop_machine() during heavy printing" from my series (it's completely
> > independent, I've attached the latest version), the result would look less
> > hacky to me (attached). Thoughts?
> 
> Haven't tested it but should also work...

Well, I expect we'll be merging Jan's lockup-avoidance code fairly
soon.  There's no point in solving the same problem twice.

Sp please, do take a closer look at the proposed patches and if
possible, runtime test them?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1243214

Andrew Morton <akpm@linux-foundation.org> writes:

> On Thu, 08 Oct 2015 12:03:25 +0200 Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>
>> > If we picked up patch "kernel: Avoid softlockups in
>> > stop_machine() during heavy printing" from my series (it's completely
>> > independent, I've attached the latest version), the result would look less
>> > hacky to me (attached). Thoughts?
>> 
>> Haven't tested it but should also work...
>
> Well, I expect we'll be merging Jan's lockup-avoidance code fairly
> soon.  There's no point in solving the same problem twice.
>
> Sp please, do take a closer look at the proposed patches and if
> possible, runtime test them?

Sure, I'll take a look.

-- 
  Vitaly
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1243315

Vitaly Kuznetsov <vkuznets@redhat.com> writes:

> Andrew Morton <akpm@linux-foundation.org> writes:
>
>> On Thu, 08 Oct 2015 12:03:25 +0200 Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>>
>>> > If we picked up patch "kernel: Avoid softlockups in
>>> > stop_machine() during heavy printing" from my series (it's completely
>>> > independent, I've attached the latest version), the result would look less
>>> > hacky to me (attached). Thoughts?
>>> 
>>> Haven't tested it but should also work...
>>
>> Well, I expect we'll be merging Jan's lockup-avoidance code fairly
>> soon.  There's no point in solving the same problem twice.
>>
>> Sp please, do take a closer look at the proposed patches and if
>> possible, runtime test them?
>
> Sure, I'll take a look.

Jan,

I've just tested patches you suggested and it seems they also solve the
issue I observe. Feel free to add:

Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>

Thanks!

-- 
  Vitaly
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1244361 — Re: Re: [PATCH] panic: release stale console lock to always get the logbuf printed out

[Multipart message — attachments visible in raw view] — view raw

Hi Jan,

[auto build test ERROR on v4.3-rc4 -- if it's inappropriate base, please ignore]

config: arm64-allnoconfig (attached as .config)
reproduce:
        wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=arm64 

All errors (new ones prefixed by >>):

   kernel/printk/printk.c: In function 'printk_log_buf_drain':
>> kernel/printk/printk.c:2410:3: error: implicit declaration of function 'zap_locks' [-Werror=implicit-function-declaration]
      zap_locks();
      ^
   cc1: some warnings being treated as errors

vim +/zap_locks +2410 kernel/printk/printk.c

  2404	void printk_log_buf_drain(bool panic)
  2405	{
  2406		bool retry;
  2407		unsigned long flags;
  2408	
  2409		if (panic)
> 2410			zap_locks();
  2411	
  2412		while (1) {
  2413			raw_spin_lock_irqsave(&logbuf_lock, flags);

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[toc] | [prev] | [next] | [standalone]

#1242140

Andrew Morton <akpm@linux-foundation.org> writes:

> (cc Jan)
>
> On Wed,  7 Oct 2015 19:02:22 +0200 Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>
>> In some cases we may end up killing the CPU holding the console lock
>> while still having valuable data in logbuf. E.g. I'm observing the
>> following:
>> - A crash is happening on one CPU and console_unlock() is being called on
>>   some other.
>> - console_unlock() tries to print out the buffer before releasing the lock
>>   and on slow console it takes time.
>> - in the meanwhile crashing CPU does lots of printk()-s with valuable data
>>   (which go to the logbuf) and sends IPIs to all other CPUs.
>> - console_unlock() finishes printing previous chunk and enables interrupts
>>   before trying to print out the rest, the CPU catches the IPI and never
>>   releases console lock.
>
> Why doesn't the lock-owning CPU release the console lock?  Because it
> was stopped by smp_send_stop() in panic()?
>
> I don't recall why we stop CPUs in panic(), and of course we didn't
> document the reason.  I guess it makes sense from the "what else can we
> do" point of view, but I wonder if we can just do it later on - that
> would fix this problem?

We don't know for how long should we wait for the other CPU to finish
the output and it can take some time. In case we're rebooting after a
short timeout we can still end up with something in the logbuf.

>
> (dumb aside: why doesn't smp_send_stop() stop the calling CPU?)
>

Because we do send_IPI_allbutself() ? :-)

>> This is not the only possible case: in VT/fb subsystems we have many other
>> console_lock()/console_unlock() users. Non-masked interrupts (or receiving
>> NMI in case of extreme slowness) will have the same result. Getting the
>> whole console buffer printed out on crash should be top priority.
>
> Yes, this is a pretty big hole in the logic.
>
>> --- a/kernel/panic.c
>> +++ b/kernel/panic.c
>> @@ -23,6 +23,7 @@
>>  #include <linux/sysrq.h>
>>  #include <linux/init.h>
>>  #include <linux/nmi.h>
>> +#include <linux/console.h>
>>  
>>  #define PANIC_TIMER_STEP 100
>>  #define PANIC_BLINK_SPD 18
>> @@ -147,6 +148,15 @@ void panic(const char *fmt, ...)
>>  
>>  	bust_spinlocks(0);
>>  
>> +	/*
>> +	 * We may have ended up killing the CPU holding the lock and still have
>> +	 * some valuable data in console buffer. Try to acquire the lock and
>> +	 * release it regardless of the result. The release will also print the
>> +	 * buffers out.
>> +	 */
>> +	console_trylock();
>> +	console_unlock();
>> +
>
> "killing the CPU" is a bit vague.  How's this look?
>
> --- a/kernel/panic.c~panic-release-stale-console-lock-to-always-get-the-logbuf-printed-out-fix
> +++ a/kernel/panic.c
> @@ -149,10 +149,10 @@ void panic(const char *fmt, ...)
>  	bust_spinlocks(0);
>
>  	/*
> -	 * We may have ended up killing the CPU holding the lock and still have
> -	 * some valuable data in console buffer. Try to acquire the lock and
> -	 * release it regardless of the result. The release will also print the
> -	 * buffers out.
> +	 * We may have ended up stopping the CPU holding the lock (in
> +	 * smp_send_stop()) while still having some valuable data in the console
> +	 * buffer.  Try to acquire the lock then release it regardless of the
> +	 * result.  The release will also print the buffers out.

Thanks, looks better)

>  	 */
>  	console_trylock();
>  	console_unlock();
> _
>
> Does the console_trylock() guarantee that the console lock is now held?
> If the console_lock-holding CPU is still running
> then there's a window
> where the above code could enter console_unlock() when nobody's holding
> console_lock.  If smp_send_stop() always works (synchronously) then
> that won't happen.

Well, in smp_send_stop() we send IPIs and if some CPU is still running
we send NMIs... In case someone ignored that I'm not sure what can be
done...

-- 
  Vitaly
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1242732 — Re: [PATCH] panic: release stale console lock to always get the logbuf printed out

On Thu, 08 Oct 2015 11:51:13 +0200 Vitaly Kuznetsov <vkuznets@redhat.com> wrote:

> > On Wed,  7 Oct 2015 19:02:22 +0200 Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
> >
> >> In some cases we may end up killing the CPU holding the console lock
> >> while still having valuable data in logbuf. E.g. I'm observing the
> >> following:
> >> - A crash is happening on one CPU and console_unlock() is being called on
> >>   some other.
> >> - console_unlock() tries to print out the buffer before releasing the lock
> >>   and on slow console it takes time.
> >> - in the meanwhile crashing CPU does lots of printk()-s with valuable data
> >>   (which go to the logbuf) and sends IPIs to all other CPUs.
> >> - console_unlock() finishes printing previous chunk and enables interrupts
> >>   before trying to print out the rest, the CPU catches the IPI and never
> >>   releases console lock.
> >
> > Why doesn't the lock-owning CPU release the console lock?  Because it
> > was stopped by smp_send_stop() in panic()?
> >
> > I don't recall why we stop CPUs in panic(), and of course we didn't
> > document the reason.  I guess it makes sense from the "what else can we
> > do" point of view, but I wonder if we can just do it later on - that
> > would fix this problem?
> 
> We don't know for how long should we wait for the other CPU to finish
> the output and it can take some time. In case we're rebooting after a
> short timeout we can still end up with something in the logbuf.

I don't understand what you're saying here.

If we move panic()'s call to smp_send_stop() so it occurs later in
panic(), won't this result in this CPU's messages being properly
displayed?  The currently-printing CPU will still be running and all
the printks will proceed in the normal fashion?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1243216

Andrew Morton <akpm@linux-foundation.org> writes:

> On Thu, 08 Oct 2015 11:51:13 +0200 Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>
>> > On Wed,  7 Oct 2015 19:02:22 +0200 Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>> >
>> >> In some cases we may end up killing the CPU holding the console lock
>> >> while still having valuable data in logbuf. E.g. I'm observing the
>> >> following:
>> >> - A crash is happening on one CPU and console_unlock() is being called on
>> >>   some other.
>> >> - console_unlock() tries to print out the buffer before releasing the lock
>> >>   and on slow console it takes time.
>> >> - in the meanwhile crashing CPU does lots of printk()-s with valuable data
>> >>   (which go to the logbuf) and sends IPIs to all other CPUs.
>> >> - console_unlock() finishes printing previous chunk and enables interrupts
>> >>   before trying to print out the rest, the CPU catches the IPI and never
>> >>   releases console lock.
>> >
>> > Why doesn't the lock-owning CPU release the console lock?  Because it
>> > was stopped by smp_send_stop() in panic()?
>> >
>> > I don't recall why we stop CPUs in panic(), and of course we didn't
>> > document the reason.  I guess it makes sense from the "what else can we
>> > do" point of view, but I wonder if we can just do it later on - that
>> > would fix this problem?
>> 
>> We don't know for how long should we wait for the other CPU to finish
>> the output and it can take some time. In case we're rebooting after a
>> short timeout we can still end up with something in the logbuf.
>
> I don't understand what you're saying here.
>
> If we move panic()'s call to smp_send_stop() so it occurs later in
> panic(), won't this result in this CPU's messages being properly
> displayed?

If some other CPU is printing, for how long do we need to wait before we
try to stop it? It can take *any* amount of time to print out the buffer
-- we can even reboot the host earlier.

> The currently-printing CPU will still be running and all
> the printks will proceed in the normal fashion?

It will be running till we reboot the host, and we need to make sure
there is nothing in the buffer when we do that. I see only two viable
options: make sure the crashing cpu prints out the buffer before we
reboot (natural serialization) or some sort of lock-waiting to make sure
the printing CPU is done with its job.

-- 
  Vitaly
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web