Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1531728 > unrolled thread

Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-28 23:00 +0100
    Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 01:50 +0100
      Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-29 07:00 +0100
        Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-11-29 10:20 +0100
          Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 15:10 +0100
            Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-29 16:10 +0100
              Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Petr Mladek <pmladek@suse.com> - 2016-11-29 17:20 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 19:10 +0100
              Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 18:00 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-29 18:20 +0100
                  Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 18:40 +0100
                    Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Petr Mladek <pmladek@suse.com> - 2016-11-30 11:00 +0100
        Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 11:30 +0100
        Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-11-29 13:50 +0100
          Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 16:20 +0100
            Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Petr Mladek <pmladek@suse.com> - 2016-11-29 17:30 +0100
              Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-11-29 18:20 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 20:50 +0100
                  Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-11-29 21:00 +0100
                    Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 21:10 +0100
                      Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-29 21:40 +0100
                        Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Josh Poimboeuf <jpoimboe@redhat.com> - 2016-11-30 20:20 +0100
                          Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-11-30 21:00 +0100
                          Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-12-01 07:00 +0100
                            Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-12-01 13:40 +0100
                              Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-12-01 17:50 +0100
                                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-12-01 18:10 +0100
                Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Petr Mladek <pmladek@suse.com> - 2016-11-30 11:10 +0100
                  Re: perf: fuzzer BUG: KASAN: stack-out-of-bounds in __unwind_start Peter Zijlstra <peterz@infradead.org> - 2016-11-30 12:10 +0100

Page 2 of 2 — ← Prev page 1 [2]

#1532747

On Tue, Nov 29, 2016 at 12:07:11PM -0800, Paul E. McKenney wrote:
> On Tue, Nov 29, 2016 at 08:52:04PM +0100, Peter Zijlstra wrote:
> > On Tue, Nov 29, 2016 at 11:39:35AM -0800, Paul E. McKenney wrote:
> > > On Tue, Nov 29, 2016 at 06:10:38PM +0100, Peter Zijlstra wrote:
> > 
> > > > It mostly works, most of the time, and that seems to be what Linus
> > > > wants, since its really the best we can have given the constraints. But
> > > > for debugging, when you have a UART, it totally blows.
> > > 
> > > UART???  They still make those things???  ;-)
> > 
> > Yes, most computer like devices actually have them, trouble is, most
> > consumer devices don't have the pins exposed. Luckily most server class
> > hardware still does.
> > 
> > And they're absolutely _awesome_ for debugging; getting data out is a
> > matter of trivial MMIO poll loops. Rock solid stuff.
> 
> They very clearly need to bring the baud rate into the current millenium,
> many tens of Mbaud at the -very- least.

On a more practical note...

Currently, cond_resched_rcu_qs() is not permitted to be invoked until
after the scheduler has started.  However, it appears that there is some
kernel code that can loop for quite some time at runtime, but which also
executes during early boot.  So it would be good to make it so that
cond_resched_rcu_qs() could be called at boot.

One approach would be to check rcu_scheduler_active, but this isn't
defined in normal Tiny RCU builds.  I can expand Tiny RCU, or I can
kludge the non-CONFIG_DEBUG_LOCK_ALLOC value of rcu_scheduler_active
to false (with this latter being the current state).  But it occurred
to me that I could also condition on !is_idle_task(), given that idle
tasks shouldn't ever be invoking the scheduler anyway.

So is the following a sensible approach, or should I look elsewhere?

	#define cond_resched_rcu_qs() \
	do { \
		if (!is_idle_task(current) && !cond_resched()) \
			rcu_note_voluntary_context_switch(current); \
	} while (0)

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1533518

On Tue, Nov 29, 2016 at 12:32:59PM -0800, Paul E. McKenney wrote:
> On Tue, Nov 29, 2016 at 12:07:11PM -0800, Paul E. McKenney wrote:
> > On Tue, Nov 29, 2016 at 08:52:04PM +0100, Peter Zijlstra wrote:
> > > On Tue, Nov 29, 2016 at 11:39:35AM -0800, Paul E. McKenney wrote:
> > > > On Tue, Nov 29, 2016 at 06:10:38PM +0100, Peter Zijlstra wrote:
> > > 
> > > > > It mostly works, most of the time, and that seems to be what Linus
> > > > > wants, since its really the best we can have given the constraints. But
> > > > > for debugging, when you have a UART, it totally blows.
> > > > 
> > > > UART???  They still make those things???  ;-)
> > > 
> > > Yes, most computer like devices actually have them, trouble is, most
> > > consumer devices don't have the pins exposed. Luckily most server class
> > > hardware still does.
> > > 
> > > And they're absolutely _awesome_ for debugging; getting data out is a
> > > matter of trivial MMIO poll loops. Rock solid stuff.
> > 
> > They very clearly need to bring the baud rate into the current millenium,
> > many tens of Mbaud at the -very- least.
> 
> On a more practical note...
> 
> Currently, cond_resched_rcu_qs() is not permitted to be invoked until
> after the scheduler has started.  However, it appears that there is some
> kernel code that can loop for quite some time at runtime, but which also
> executes during early boot.  So it would be good to make it so that
> cond_resched_rcu_qs() could be called at boot.
> 
> One approach would be to check rcu_scheduler_active, but this isn't
> defined in normal Tiny RCU builds.  I can expand Tiny RCU, or I can
> kludge the non-CONFIG_DEBUG_LOCK_ALLOC value of rcu_scheduler_active
> to false (with this latter being the current state).  But it occurred
> to me that I could also condition on !is_idle_task(), given that idle
> tasks shouldn't ever be invoking the scheduler anyway.

This question was probably intended for other folks, but I should point
out that idle tasks *do* invoke the scheduler.  cpu_idle_loop() calls
schedule_preempt_disabled().

> 
> So is the following a sensible approach, or should I look elsewhere?
> 
> 	#define cond_resched_rcu_qs() \
> 	do { \
> 		if (!is_idle_task(current) && !cond_resched()) \
> 			rcu_note_voluntary_context_switch(current); \
> 	} while (0)
> 
> 							Thanx, Paul
> 

-- 
Josh

[toc] | [prev] | [next] | [standalone]

#1533539

On Wed, Nov 30, 2016 at 01:13:03PM -0600, Josh Poimboeuf wrote:
> On Tue, Nov 29, 2016 at 12:32:59PM -0800, Paul E. McKenney wrote:
> > On Tue, Nov 29, 2016 at 12:07:11PM -0800, Paul E. McKenney wrote:
> > > On Tue, Nov 29, 2016 at 08:52:04PM +0100, Peter Zijlstra wrote:
> > > > On Tue, Nov 29, 2016 at 11:39:35AM -0800, Paul E. McKenney wrote:
> > > > > On Tue, Nov 29, 2016 at 06:10:38PM +0100, Peter Zijlstra wrote:
> > > > 
> > > > > > It mostly works, most of the time, and that seems to be what Linus
> > > > > > wants, since its really the best we can have given the constraints. But
> > > > > > for debugging, when you have a UART, it totally blows.
> > > > > 
> > > > > UART???  They still make those things???  ;-)
> > > > 
> > > > Yes, most computer like devices actually have them, trouble is, most
> > > > consumer devices don't have the pins exposed. Luckily most server class
> > > > hardware still does.
> > > > 
> > > > And they're absolutely _awesome_ for debugging; getting data out is a
> > > > matter of trivial MMIO poll loops. Rock solid stuff.
> > > 
> > > They very clearly need to bring the baud rate into the current millenium,
> > > many tens of Mbaud at the -very- least.
> > 
> > On a more practical note...
> > 
> > Currently, cond_resched_rcu_qs() is not permitted to be invoked until
> > after the scheduler has started.  However, it appears that there is some
> > kernel code that can loop for quite some time at runtime, but which also
> > executes during early boot.  So it would be good to make it so that
> > cond_resched_rcu_qs() could be called at boot.
> > 
> > One approach would be to check rcu_scheduler_active, but this isn't
> > defined in normal Tiny RCU builds.  I can expand Tiny RCU, or I can
> > kludge the non-CONFIG_DEBUG_LOCK_ALLOC value of rcu_scheduler_active
> > to false (with this latter being the current state).  But it occurred
> > to me that I could also condition on !is_idle_task(), given that idle
> > tasks shouldn't ever be invoking the scheduler anyway.
> 
> This question was probably intended for other folks, but I should point
> out that idle tasks *do* invoke the scheduler.  cpu_idle_loop() calls
> schedule_preempt_disabled().

Good point.  My next fallback is that idle loops should not be running
for long periods of time within RCU_NONIDLE().  Does that work?

							Thanx, Paul

> > So is the following a sensible approach, or should I look elsewhere?
> > 
> > 	#define cond_resched_rcu_qs() \
> > 	do { \
> > 		if (!is_idle_task(current) && !cond_resched()) \
> > 			rcu_note_voluntary_context_switch(current); \
> > 	} while (0)
> > 
> > 							Thanx, Paul
> > 
> 
> -- 
> Josh
> 

[toc] | [prev] | [next] | [standalone]

#1533826

On Wed, Nov 30, 2016 at 01:13:03PM -0600, Josh Poimboeuf wrote:
> This question was probably intended for other folks, but I should point
> out that idle tasks *do* invoke the scheduler.  cpu_idle_loop() calls
> schedule_preempt_disabled().

Right, but that doesn't matter I think. The below will simply not call
rcu_note_voluntary_context_switch() from the idle task, which would be
fine I think.

> > So is the following a sensible approach, or should I look elsewhere?
> > 
> > 	#define cond_resched_rcu_qs() \
> > 	do { \
> > 		if (!is_idle_task(current) && !cond_resched()) \
> > 			rcu_note_voluntary_context_switch(current); \

You should reverse your conditions though:

		if (!cond_resched() && !is_idle_task(current))
			rcu_note_voluntary_context_switch(current);

That way we'll still do cond_resched() and you only gate the RCU call.

[toc] | [prev] | [next] | [standalone]

#1534036

On Thu, Dec 01, 2016 at 06:52:35AM +0100, Peter Zijlstra wrote:
> On Wed, Nov 30, 2016 at 01:13:03PM -0600, Josh Poimboeuf wrote:
> > This question was probably intended for other folks, but I should point
> > out that idle tasks *do* invoke the scheduler.  cpu_idle_loop() calls
> > schedule_preempt_disabled().
> 
> Right, but that doesn't matter I think. The below will simply not call
> rcu_note_voluntary_context_switch() from the idle task, which would be
> fine I think.
> 
> > > So is the following a sensible approach, or should I look elsewhere?
> > > 
> > > 	#define cond_resched_rcu_qs() \
> > > 	do { \
> > > 		if (!is_idle_task(current) && !cond_resched()) \
> > > 			rcu_note_voluntary_context_switch(current); \
> 
> You should reverse your conditions though:
> 
> 		if (!cond_resched() && !is_idle_task(current))
> 			rcu_note_voluntary_context_switch(current);
> 
> That way we'll still do cond_resched() and you only gate the RCU call.

This makes it illegal at early boot.  This is not a problem with the
surviving cond_resched_rcu_qs(), but one of the candidates really was
called at boot time.  If I reverse the order as you say, I can just as
well leave of the "!is_idle_task(current)".

So I will just drop this patch until such time as someone actually needs
to add a cond_resched_rcu_qs() that sometimes gets invoked at boot time.

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1534280

On Thu, Dec 01, 2016 at 04:33:16AM -0800, Paul E. McKenney wrote:
> On Thu, Dec 01, 2016 at 06:52:35AM +0100, Peter Zijlstra wrote:
> > On Wed, Nov 30, 2016 at 01:13:03PM -0600, Josh Poimboeuf wrote:
> > > This question was probably intended for other folks, but I should point
> > > out that idle tasks *do* invoke the scheduler.  cpu_idle_loop() calls
> > > schedule_preempt_disabled().
> > 
> > Right, but that doesn't matter I think. The below will simply not call
> > rcu_note_voluntary_context_switch() from the idle task, which would be
> > fine I think.
> > 
> > > > So is the following a sensible approach, or should I look elsewhere?
> > > > 
> > > > 	#define cond_resched_rcu_qs() \
> > > > 	do { \
> > > > 		if (!is_idle_task(current) && !cond_resched()) \
> > > > 			rcu_note_voluntary_context_switch(current); \
> > 
> > You should reverse your conditions though:
> > 
> > 		if (!cond_resched() && !is_idle_task(current))
> > 			rcu_note_voluntary_context_switch(current);
> > 
> > That way we'll still do cond_resched() and you only gate the RCU call.
> 
> This makes it illegal at early boot.

Humm, how early are we talking?

[toc] | [prev] | [next] | [standalone]

#1534311

On Thu, Dec 01, 2016 at 05:41:07PM +0100, Peter Zijlstra wrote:
> On Thu, Dec 01, 2016 at 04:33:16AM -0800, Paul E. McKenney wrote:
> > On Thu, Dec 01, 2016 at 06:52:35AM +0100, Peter Zijlstra wrote:
> > > On Wed, Nov 30, 2016 at 01:13:03PM -0600, Josh Poimboeuf wrote:
> > > > This question was probably intended for other folks, but I should point
> > > > out that idle tasks *do* invoke the scheduler.  cpu_idle_loop() calls
> > > > schedule_preempt_disabled().
> > > 
> > > Right, but that doesn't matter I think. The below will simply not call
> > > rcu_note_voluntary_context_switch() from the idle task, which would be
> > > fine I think.
> > > 
> > > > > So is the following a sensible approach, or should I look elsewhere?
> > > > > 
> > > > > 	#define cond_resched_rcu_qs() \
> > > > > 	do { \
> > > > > 		if (!is_idle_task(current) && !cond_resched()) \
> > > > > 			rcu_note_voluntary_context_switch(current); \
> > > 
> > > You should reverse your conditions though:
> > > 
> > > 		if (!cond_resched() && !is_idle_task(current))
> > > 			rcu_note_voluntary_context_switch(current);
> > > 
> > > That way we'll still do cond_resched() and you only gate the RCU call.
> > 
> > This makes it illegal at early boot.
> 
> Humm, how early are we talking?

The case I saw was during start_kernel(), IIRC.  But again, it turned
out that the patch putting cond_resched_rcu_qs() in that early was
(1) broken and (2) unnecessary.

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1533171

On Tue 2016-11-29 18:10:38, Peter Zijlstra wrote:
> On Tue, Nov 29, 2016 at 05:29:20PM +0100, Petr Mladek wrote:
> 
> > > > People are very busy polishing the turd we call printk, but from where
> > > > I'm sitting its terminally and unfixably broken.
> > 
> > I still hope that we could do better :-)
> 
> How? The console drivers are a complete trainwreck, you simply cannot
> build anything sensible ontop of a trainwreck.

I am afraid that I will not persuade you but...


> And from what I understood from talking to someone (I again forgot who)
> at LPC, the whole reason people were poking at this is that the block
> layer (or something thereabouts) prints a gazillion lines of crap when
> you attach a stupid amount of devices (through FC or other SAN like
> things).

This is crazy indeed if it happens on a production system.


> The way we've 'fixed' that in the scheduler (a fairly long time ago)
> when SGI complained about our printks taking too long (because they had
> 4096 CPUs), is to simply remove the printks (they're now hidden behind
> the sched_debug boot param).

This is a solution. But what if you want to enable debugging and the
system does not boot because the printing takes too long.


> In any case, as long as printk has a globally serialized 'log', it, per
> design, will be worse than the console drivers its build upon. And them
> being shit precludes the entire stack from being useful.

I probably still do not understand all the problems with console
drivers. My understanding is that the problem is that they have
its own locking and are slow. It means that they are prone to
a deadlock and they might block for a long time.

In compare, the serialized log buffer has one lock and writing
is fast. It means that it suffers "only" from the deadlocks.
And we try to address the deadlocks by using the temporary
per-CPU buffers in critical situations (NMI, locked sections).

Of course, it is useless if you have the messages in a buffer
and can't reach them. But we do the best effort to push them
to consoles and crash dump. Also it might be very useful to
have the log buffer on persistent memory.


> It mostly works, most of the time, and that seems to be what Linus
> wants, since its really the best we can have given the constraints. But
> for debugging, when you have a UART, it totally blows.

I believe that the early console is the last resort for debugging
some type of bugs. But many other bugs can be debugged with the
classic printk(). And there are (production) systems where you
cannot (easily) or do not want to use early printk all the time.

Another question is the complexity of the printk() code. Especially,
the big effort to get "perfect" (non-mixed) output is questionable.

Best Regards,
Petr

[toc] | [prev] | [next] | [standalone]

#1533223

On Wed, Nov 30, 2016 at 11:01:29AM +0100, Petr Mladek wrote:
> On Tue 2016-11-29 18:10:38, Peter Zijlstra wrote:

> > In any case, as long as printk has a globally serialized 'log', it, per
> > design, will be worse than the console drivers its build upon. And them
> > being shit precludes the entire stack from being useful.
> 
> I probably still do not understand all the problems with console
> drivers. My understanding is that the problem is that they have
> its own locking and are slow. It means that they are prone to
> a deadlock and they might block for a long time.

Slow isn't a problem; just limit the crap you want to push down them.

Them taking locks, them using the scheduler and them depending on entire
subsystem state to be 'sane' are the problems.

Take for instance the usb-serial console driver (everybody agrees its
crap, and gregkh did it as a lark, but still it exists), that takes
locks, relies on the scheduler, depends on the USB subsystem, which in
turn depends on the PCI subsystem.

Now imagine trying to use that for something halfway sensible.

Even the DRM based consoles suffer much the same problems.

Heck, even the 'normal' UART drivers do this :-(

> In compare, the serialized log buffer has one lock and writing
> is fast. It means that it suffers "only" from the deadlocks.
> And we try to address the deadlocks by using the temporary
> per-CPU buffers in critical situations (NMI, locked sections).

The temporary buffers are crap when you never get around to flushing
them. You need a fully lockless and wait free buffer or you're screwed.

> Of course, it is useless if you have the messages in a buffer
> and can't reach them. But we do the best effort to push them
> to consoles and crash dump. Also it might be very useful to
> have the log buffer on persistent memory.

Nothing will crash if you do while (1); in NMI context.

Been there, done that (of course it wasn't _that_ blatant, but the
effect was much the same).

I also have WARN()s in scheduler code, now most of those will not
indicate fatal conditions, but given the above state of console drivers
they have a very real chance of deadlocking the system.

And no, we're not going to do WARN_DEFERRED and similar crap. That just
proliferates the utter fail of printk() down the stack and creates more
mess.

> > It mostly works, most of the time, and that seems to be what Linus
> > wants, since its really the best we can have given the constraints. But
> > for debugging, when you have a UART, it totally blows.
> 
> I believe that the early console is the last resort for debugging
> some type of bugs. But many other bugs can be debugged with the
> classic printk(). And there are (production) systems where you
> cannot (easily) or do not want to use early printk all the time.
> 
> Another question is the complexity of the printk() code. Especially,
> the big effort to get "perfect" (non-mixed) output is questionable.

I'm saying its an entirely wasted effort, because no matter how much
complexity you pile on, you can never get it into a useful state because
its all build on shit.

So yes, I much prefer to not add more and more complexity into printk.
We should strip it down, not pile more junk on.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web