Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1707326

Re: RCU stall when using function_graph

Show all headers | View raw

On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote:
> 
> 
> On Sunday 06 August 2017 10:32 PM, Paul E. McKenney wrote:
> >On Sat, Aug 05, 2017 at 02:24:21PM +0900, 김동현 wrote:
> >>Dear All
> >>
> >>As for me, after configuring function_graph as below, crash disappears.
> >>"echo 0 > d/tracing/tracing_on"
> >>"sleep 1"
> >>
> >>"echo function_graph > d/tracing/current_tracer"
> >>"sleep 1"
> >>
> >>"echo smp_call_function_single > d/tracing/set_ftrace_filter"
> 
> It will limit trace output to only for the filtered function
> (smp_call_function_single).
> 
> >>adb shell "sleep 1"
> >>
> >>"echo 1 > d/tracing/tracing_on"
> >>adb shell "sleep 1"
> >>
> >>Right after function_graph is enabled, too many logs are traced upon IRQ
> >>transaction which many times eventually causes stall.
> >
> >That would do it!
> >
> >Hmmm...
> >
> >Steven, would it be helpful if RCU were to inform tracing (say) halfway
> >through the RCU CPU stall interval, allowing the tracer to do something
> >like cond_resched_rcu_qs()?  I can imagine all sorts of reasons why this
> >wouldn't work, for example, if all the tracing was with irqs disabled
> >or some such, but figured I should ask.
> >
> >Does Guillermo's approach work for others?
> 
> Limited output with a couple of filtered function will definitely
> not cause RCU schedule stall. But the question is whether we should
> expect a full function graph trace working on every platform or not
> (specially the one which generates high interrupts)?

It might well be that the user must disable RCU CPU stall warnings via
the rcu_cpu_stall_suppress sysfs entry (or increase their timeout via the
rcu_cpu_stall_timeout sysfs entry) before doing something that greatly
increases overhead.  Like enabling large quantities of tracing.  ;-)

It -might- be possible to do this automatically, but reliable
automation would require that tracing understand how often each
function was called, which sounds to me to be a bit of a stretch.

Thoughts?

							Thanx, Paul

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: RCU stall when using function_graph "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-02 00:10 +0200
  Re: RCU stall when using function_graph Daniel Lezcano <daniel.lezcano@linaro.org> - 2017-08-02 00:20 +0200
    Re: RCU stall when using function_graph Steven Rostedt <rostedt@goodmis.org> - 2017-08-02 02:20 +0200
      Re: RCU stall when using function_graph Daniel Lezcano <daniel.lezcano@linaro.org> - 2017-08-02 14:50 +0200
        Re: RCU stall when using function_graph Steven Rostedt <rostedt@goodmis.org> - 2017-08-02 15:10 +0200
          Re: RCU stall when using function_graph "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-03 04:50 +0200
          Re: RCU stall when using function_graph Daniel Lezcano <daniel.lezcano@linaro.org> - 2017-08-03 13:50 +0200
            Re: RCU stall when using function_graph "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-03 14:50 +0200
              Re: RCU stall when using function_graph Daniel Lezcano <daniel.lezcano@linaro.org> - 2017-08-03 16:40 +0200
                Re: RCU stall when using function_graph "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-06 19:10 +0200
                Re: RCU stall when using function_graph Pratyush Anand <panand@redhat.com> - 2017-08-09 11:20 +0200
                Re: RCU stall when using function_graph "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-09 15:00 +0200
                Re: RCU stall when using function_graph Daniel Lezcano <daniel.lezcano@linaro.org> - 2017-08-09 15:30 +0200
                Re: RCU stall when using function_graph "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-09 16:50 +0200
                Re: RCU stall when using function_graph Daniel Lezcano <daniel.lezcano@linaro.org> - 2017-08-09 18:00 +0200
                Re: RCU stall when using function_graph "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-09 19:30 +0200
                Re: RCU stall when using function_graph Daniel Lezcano <daniel.lezcano@linaro.org> - 2017-08-10 11:50 +0200
                Re: RCU stall when using function_graph "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-10 23:40 +0200
                Re: RCU stall when using function_graph Daniel Lezcano <daniel.lezcano@linaro.org> - 2017-08-11 11:40 +0200
        Re: RCU stall when using function_graph "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-02 19:00 +0200
  Re: RCU stall when using function_graph "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-02 14:50 +0200

csiph-web