Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > linux.kernel > #1466835 > unrolled thread

runaway latency detection

Started by"T. Smith" <tlsmith3777@gmail.com>
First post2016-08-20 19:10 +0200
Last post2016-08-21 05:50 +0200
Articles 3 — 3 participants

Back to article view | Back to linux.kernel


Contents

  runaway latency detection "T. Smith" <tlsmith3777@gmail.com> - 2016-08-20 19:10 +0200
    Re: runaway latency detection Nikolay Borisov <n.borisov.lkml@gmail.com> - 2016-08-20 20:00 +0200
      Re: runaway latency detection Steven Rostedt <rostedt@goodmis.org> - 2016-08-21 05:50 +0200

#1466835 — runaway latency detection

From"T. Smith" <tlsmith3777@gmail.com>
Date2016-08-20 19:10 +0200
Subjectrunaway latency detection
Message-ID<s8hWp-5Mt-1@gated-at.bofh.it>
Trying to isolate PREEMPT_RT (full preemption) runaway latency conditions seen when using the cyclictest on the 4.1.8 SMP PREEMPT_RT kernel.  Seen average latency of 3uS, but runaway or outlier max latency seems indeterminate.  Noted 72uS with one run of the cyclictest for example. Is this a known issue, the cause understood, and is there a patch for this?  

The goal is to isolate causes of indeterminism when using the PREEMPT_RT kernel configuration with full preemption and to characterize latency and jitter using ftrace, any recommendations?

[toc] | [next] | [standalone]


#1466839

FromNikolay Borisov <n.borisov.lkml@gmail.com>
Date2016-08-20 20:00 +0200
Message-ID<s8iIN-64d-13@gated-at.bofh.it>
In reply to#1466835

On 20.08.2016 20:03, T. Smith wrote:
> The goal is to isolate causes of indeterminism when using the PREEMPT_RT kernel configuration with full preemption and to characterize latency and jitter using ftrace, any recommendations?

What hardware is it ? If it's x86 it's entirely possible you are seeing
SMI. Perhaps you should look into disabling those if you are on x86?

[toc] | [prev] | [next] | [standalone]


#1466913

FromSteven Rostedt <rostedt@goodmis.org>
Date2016-08-21 05:50 +0200
Message-ID<s8rVL-3wx-1@gated-at.bofh.it>
In reply to#1466839

On August 20, 2016 3:04:43 PM EDT, Tracy Smith 


>Using ftace it is difficult to catch the latency since it can occur at
>any
>point during the cyclictest. It is an uncomfortable level of
>indeterminism
>without knowing the cause.
>
>Is there a way to trace random latency jumps on a single core using
>trace,
>or any other suggestion to isolate it? I know the functions and tasks
>running on the core using ftrace, but can't catch the random latency
>jumps.

Have you tried using the -b option of cyclictest?  It stops the tracing when it detects a latency. It shows what happened up to that point. I can discuss more about what to do later but I'll be traveling to LinuxCon tomorrow.

-- Steve


>
>Thx, Tracy
>On Aug 20, 2016 12:54 PM, "Nikolay Borisov" <n.borisov.lkml@gmail.com>
>wrote:
>
>>
>>
>> On 20.08.2016 20:03, T. Smith wrote:
>> > The goal is to isolate causes of indeterminism when using the
>PREEMPT_RT
>> kernel configuration with full preemption and to characterize latency
>and
>> jitter using ftrace, any recommendations?
>>
>> What hardware is it ? If it's x86 it's entirely possible you are
>seeing
>> SMI. Perhaps you should look into disabling those if you are on x86?
>>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[toc] | [prev] | [standalone]


Back to top | Article view | linux.kernel


csiph-web