Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1490527 > unrolled thread

Re: Regression in 4.8 - CPU speed set very low

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-24 04:50 +0200
    Re: Regression in 4.8 - CPU speed set very low "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-09-26 13:40 +0200
      Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-26 18:20 +0200
        Re: Regression in 4.8 - CPU speed set very low "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-09-26 23:10 +0200
          Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-26 23:30 +0200
            Re: Regression in 4.8 - CPU speed set very low "Rafael J. Wysocki" <rafael@kernel.org> - 2016-09-26 23:40 +0200
              Re: Regression in 4.8 - CPU speed set very low Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com> - 2016-09-26 23:50 +0200
                Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-27 00:20 +0200
              Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-27 00:20 +0200
                Re: Regression in 4.8 - CPU speed set very low "Rafael J. Wysocki" <rafael@kernel.org> - 2016-09-27 00:20 +0200
                  Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-27 02:00 +0200
                    Re: Regression in 4.8 - CPU speed set very low "Rafael J. Wysocki" <rafael@kernel.org> - 2016-09-27 02:30 +0200
                      Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-27 02:50 +0200
                        Re: Regression in 4.8 - CPU speed set very low Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com> - 2016-09-27 03:40 +0200
                          Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-27 05:00 +0200
                        RE: Regression in 4.8 - CPU speed set very low "Doug Smythies" <dsmythies@telus.net> - 2016-09-27 05:20 +0200
                          Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-27 10:50 +0200
                            Re: Regression in 4.8 - CPU speed set very low "Rafael J. Wysocki" <rafael@kernel.org> - 2016-09-27 13:50 +0200
                              Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-29 04:30 +0200
                                Re: Regression in 4.8 - CPU speed set very low "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-09-29 14:20 +0200
                                  Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-29 17:10 +0200
                                  Re: Regression in 4.8 - CPU speed set very low Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com> - 2016-09-29 18:00 +0200
                                    Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-29 18:30 +0200
                                    Re: Regression in 4.8 - CPU speed set very low "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-09-29 22:50 +0200
                                    Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-10-01 00:00 +0200
            Re: Regression in 4.8 - CPU speed set very low lsorense@csclub.uwaterloo.ca (Lennart Sorensen) - 2016-09-27 17:00 +0200
              Re: Regression in 4.8 - CPU speed set very low Larry Finger <Larry.Finger@lwfinger.net> - 2016-09-29 04:30 +0200
                Re: Regression in 4.8 - CPU speed set very low lsorense@csclub.uwaterloo.ca (Lennart Sorensen) - 2016-09-29 16:20 +0200
          Re: Regression in 4.8 - CPU speed set very low Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com> - 2016-09-26 23:30 +0200
            Re: Regression in 4.8 - CPU speed set very low "Rafael J. Wysocki" <rafael@kernel.org> - 2016-09-26 23:40 +0200
              Re: Regression in 4.8 - CPU speed set very low Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com> - 2016-09-26 23:50 +0200
                Re: Regression in 4.8 - CPU speed set very low "Rafael J. Wysocki" <rafael@kernel.org> - 2016-09-26 23:50 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1493530

On 09/29/2016 07:19 AM, Rafael J. Wysocki wrote:
> On Wednesday, September 28, 2016 09:22:59 PM Larry Finger wrote:
>> On 09/27/2016 06:46 AM, Rafael J. Wysocki wrote:
>>> On Tue, Sep 27, 2016 at 10:48 AM, Larry Finger
>>> <Larry.Finger@lwfinger.net> wrote:
>>>> On 09/26/2016 10:12 PM, Doug Smythies wrote:
>>>>>
>>>>> On 2016.09.26 18:31 Srinivas Pandruvada wrote:
>>>>>>
>>>>>> On Mon, 2016-09-26 at 19:48 -0500, Larry Finger wrote:
>>>>>>>
>>>>>>> On 09/26/2016 07:21 PM, Rafael J. Wysocki wrote:
>>>>>>>>
>>>>>>>> On Tue, Sep 27, 2016 at 1:53 AM, Larry Finger wrote:
>>>>>>>> But for both we need a reproducer anyway.
>>>>>>>
>>>>>>> I do not have a reliable reproducer. The condition has always
>>>>>>> happened when
>>>>>>> running a high-compute job such as a 'make -j8' on the kernel, or
>>>>>>> building the
>>>>>>> RPM for openSUSE's implementation of VirtualBox. The latter is what
>>>>>>> I'm using
>>>>>>> for most of my testing.
>>>>>
>>>>>
>>>>> Run some CPU stressor and get all your CPU's going at 100% load.
>>>>> And watch your core temperatures while you do so.
>>>>
>>>>
>>>> for i in 1 2 3 4; do while : ; do : ; done & done
>>>>
>>>> triggered the fault in a few minutes.
>>>>>
>>>>>
>>>>>>
>>>>>>>> It also would be good to rule out the thermal throttling (as per
>>>>>>>> the Srinivas' comments).
>>>>>
>>>>>
>>>>> It is almost certainly thermal throttling, or similar causing
>>>>> Clock modulation, of it seems 50%.
>>>>
>>>>
>>>> While the infinite loops were running, the temps were:
>>>>
>>>> finger@linux-1t8h:~/rtlwifi_new> sensors
>>>> coretemp-isa-0000
>>>> Adapter: ISA adapter
>>>> Physical id 0:  +83.0°C  (high = +84.0°C, crit = +100.0°C)
>>>> Core 0:         +83.0°C  (high = +84.0°C, crit = +100.0°C)
>>>> Core 1:         +74.0°C  (high = +84.0°C, crit = +100.0°C)
>>>
>>> It looks like the trip point (high) temperature was exceeded causing
>>> thermal throttling to kick in.
>>>
>>>> After the fault occurs, I get
>>>>
>>>> finger@linux-1t8h:~/rtlwifi_new> sensors
>>>> coretemp-isa-0000
>>>> Adapter: ISA adapter
>>>> Physical id 0:  +44.0°C  (high = +84.0°C, crit = +100.0°C)
>>>> Core 0:         +43.0°C  (high = +84.0°C, crit = +100.0°C)
>>>> Core 1:         +41.0°C  (high = +84.0°C, crit = +100.0°C)
>>>
>>> So after that it stays at 400 MHz forever, right?
>>>
>>>>>>>>
>>>>>>>> For now, please tell me what's in
>>>>>>>> /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq
>>>>>>>
>>>>>>> 800000
>>>>>>
>>>>>> Your effective freq is lower than 800MHz. One of the possible reason is
>>>>>> thermal throttling.
>>>>>>
>>>>>> What distro you are using?
>>>>>
>>>>>
>>>>> And what make and model of LapTop?
>>>>
>>>>
>>>> Toshiba Tecra A50-A with CPU Model: 6.60.3 "Intel(R) Core(TM) i7-4600M CPU @
>>>> 2.90GHz. That is a dual-core unit with hyperthreading.
>>>>
>>>> @Rafael: As I write this, the system has been running the infinite loop test
>>>> for almost 5 hours with kernel 4.7. I will leave that running while I'm
>>>> gone, but I am certain that it is OK.
>>>
>>> OK, and what temperatures do you see while doing this?
>>
>> finger@linux-1t8h:~/linux-2.6> sensors
>> coretemp-isa-0000
>> Adapter: ISA adapter
>> Physical id 0:  +90.0°C  (high = +84.0°C, crit = +100.0°C)
>> Core 0:         +90.0°C  (high = +84.0°C, crit = +100.0°C)
>> Core 1:         +78.0°C  (high = +84.0°C, crit = +100.0°C)
>>
>> Once again, the CPU temp is greater than the "high" value; however, the clock
>> rate continues to hold near 3600 MHz.
>>
>> My laptop was inadvertently put to sleep while I was gone. I forgot to leave a
>> note for my wife and she quieted the noisy cpu fan. :)
>
> It looks like in 4.8-rc we made a change that caused the "high" trip point to
> be acted on.
>
> Srinivas, Rui, do you recall what that can be?
>
> One more question (I think I asked it previously): In the failing case
> (4.8-rc1 and later), when the frequency drops down to the 400 MHz, does it
> ever go back higher or is it stuck at that level forever?
>
> In any case, it may help to file a bug at bugzilla.kernel.org against
> CPU/thermal or similar and let me know the bug number.  We'll need to
> collect some tracepoint data to debug this and some place to put them
> into for easy reference.

Sorry if I missed that earlier question. The CPU is stuck at that lower 
frequency until I reboot.

Bug report at https://bugzilla.kernel.org/show_bug.cgi?id=173361. I tried to 
cover the main points of the discussion. Please add the ones that I missed.

Larry

[toc] | [prev] | [next] | [standalone]

#1493543

On Thu, 2016-09-29 at 14:19 +0200, Rafael J. Wysocki wrote:

[...]

> > My laptop was inadvertently put to sleep while I was gone. I forgot
> > to leave a 
> > note for my wife and she quieted the noisy cpu fan. :)
> It looks like in 4.8-rc we made a change that caused the "high" trip
> point to
> be acted on.
This high trip point we don't expose in thermal subsystem (the thermal
 zone dump didn't show this anywhere as a trip). This is exposed by
core-dts driver only. This is the point BIOS is supposed to act, I
guess that's why you are seeing 50% clock modulation. 


Are you running thermald 

What is?
# ps -e | grep thermald


> 
> Srinivas, Rui, do you recall what that can be?
> 
> One more question (I think I asked it previously): In the failing
> case
> (4.8-rc1 and later), when the frequency drops down to the 400 MHz,
> does it
> ever go back higher or is it stuck at that level forever?
> 
> In any case, it may help to file a bug at bugzilla.kernel.org against
> CPU/thermal or similar and let me know the bug number.  We'll need to
> collect some tracepoint data to debug this and some place to put them
> into for easy reference.
Yes, this is good idea.

Thanks,
Srinivas

[toc] | [prev] | [next] | [standalone]

#1493561

On 09/29/2016 10:56 AM, Srinivas Pandruvada wrote:
> On Thu, 2016-09-29 at 14:19 +0200, Rafael J. Wysocki wrote:
>
> [...]
>
>>> My laptop was inadvertently put to sleep while I was gone. I forgot
>>> to leave a
>>> note for my wife and she quieted the noisy cpu fan. :)
>> It looks like in 4.8-rc we made a change that caused the "high" trip
>> point to
>> be acted on.
> This high trip point we don't expose in thermal subsystem (the thermal
>  zone dump didn't show this anywhere as a trip). This is exposed by
> core-dts driver only. This is the point BIOS is supposed to act, I
> guess that's why you are seeing 50% clock modulation.
>
>
> Are you running thermald
>
> What is?
> # ps -e | grep thermald

The output is blank. I am not running thermald.

Larry

[toc] | [prev] | [next] | [standalone]

#1493708

On Thursday, September 29, 2016 08:56:16 AM Srinivas Pandruvada wrote:
> On Thu, 2016-09-29 at 14:19 +0200, Rafael J. Wysocki wrote:
> 
> [...]
> 
> > > My laptop was inadvertently put to sleep while I was gone. I forgot
> > > to leave a 
> > > note for my wife and she quieted the noisy cpu fan. :)
> > It looks like in 4.8-rc we made a change that caused the "high" trip
> > point to
> > be acted on.
> This high trip point we don't expose in thermal subsystem (the thermal
>  zone dump didn't show this anywhere as a trip). This is exposed by
> core-dts driver only. This is the point BIOS is supposed to act, I
> guess that's why you are seeing 50% clock modulation. 

Right.  That's SMM kicking in.

The real problem is that we get stuck at 400 MHz.

Thanks,
Rafael

[toc] | [prev] | [next] | [standalone]

#1494309

On 09/29/2016 10:56 AM, Srinivas Pandruvada wrote:
> On Thu, 2016-09-29 at 14:19 +0200, Rafael J. Wysocki wrote:
>
> [...]
>
>>> My laptop was inadvertently put to sleep while I was gone. I forgot
>>> to leave a
>>> note for my wife and she quieted the noisy cpu fan. :)
>> It looks like in 4.8-rc we made a change that caused the "high" trip
>> point to
>> be acted on.
> This high trip point we don't expose in thermal subsystem (the thermal
>  zone dump didn't show this anywhere as a trip). This is exposed by
> core-dts driver only. This is the point BIOS is supposed to act, I
> guess that's why you are seeing 50% clock modulation.
>
>
> Are you running thermald
>
> What is?
> # ps -e | grep thermald
>
>
>>
>> Srinivas, Rui, do you recall what that can be?
>>
>> One more question (I think I asked it previously): In the failing
>> case
>> (4.8-rc1 and later), when the frequency drops down to the 400 MHz,
>> does it
>> ever go back higher or is it stuck at that level forever?
>>
>> In any case, it may help to file a bug at bugzilla.kernel.org against
>> CPU/thermal or similar and let me know the bug number.  We'll need to
>> collect some tracepoint data to debug this and some place to put them
>> into for easy reference.
> Yes, this is good idea.

To complete the record in this thread, the problem also happened with kernel 
4.7, thus it is not a regression in 4.8-rcX. The full discussion is at 
https://bugzilla.kernel.org/show_bug.cgi?id=173361.

Larry

[toc] | [prev] | [next] | [standalone]

#1491966

On Mon, Sep 26, 2016 at 04:28:29PM -0500, Larry Finger wrote:
> Mostly I use a KDE applet named "System load" and look at the "average
> clock", but the same info is also available in /proc/cpuinfo as "cpu MHz".
> When the bug triggers, the system gets very slow, and the cpu fan stops even
> though the cpu is still busy.
> 
> Commit f7816ad, which had run for 7 days without showing the bug, failed
> after about 2 hours today. All my testing since Sept. 9 has been wasted. Oh
> well, that's the way it goes!

Is it possible there is no bug and instead you have a hardware problem?

What I am thinking:

CPU fan stops, then CPU gets busy, CPU overheats, thermal throtling
kicks in to protect CPU and it gets VERY slow.

So maybe you have a bad CPU fan that is getting stuck.  Perhaps even if
you have a motherboard that varies the CPU fan depending on need and the
fan doesn't like the lowest speed and sometimes gets stuck when asked
to go slow.

Of course if the CPU fan is the problem that could explain why it takes
varying amounts of time to see the problem.

I suggest checking what the cpu temperature sensors are showing next
time it gets slow.

-- 
Len Sorensen

[toc] | [prev] | [next] | [standalone]

#1493105

On 09/27/2016 09:51 AM, Lennart Sorensen wrote:
> On Mon, Sep 26, 2016 at 04:28:29PM -0500, Larry Finger wrote:
>> Mostly I use a KDE applet named "System load" and look at the "average
>> clock", but the same info is also available in /proc/cpuinfo as "cpu MHz".
>> When the bug triggers, the system gets very slow, and the cpu fan stops even
>> though the cpu is still busy.
>>
>> Commit f7816ad, which had run for 7 days without showing the bug, failed
>> after about 2 hours today. All my testing since Sept. 9 has been wasted. Oh
>> well, that's the way it goes!
>
> Is it possible there is no bug and instead you have a hardware problem?
>
> What I am thinking:
>
> CPU fan stops, then CPU gets busy, CPU overheats, thermal throtling
> kicks in to protect CPU and it gets VERY slow.
>
> So maybe you have a bad CPU fan that is getting stuck.  Perhaps even if
> you have a motherboard that varies the CPU fan depending on need and the
> fan doesn't like the lowest speed and sometimes gets stuck when asked
> to go slow.
>
> Of course if the CPU fan is the problem that could explain why it takes
> varying amounts of time to see the problem.
>
> I suggest checking what the cpu temperature sensors are showing next
> time it gets slow.

By the time it gets slow, the CPU's cool, and one cannot see the temp just 
before that event happened.

The reason I suspect a bug is that it fails with 4.8-rcX, but not with 4.7. Of 
course, it could be something subtle that slightly changes the heat load, which 
causes the CPU temp to be a little higher so that the effect is triggered.

I am reasonably confident that it is not a hardware problem, but we may have to 
wait until 4.8 is released and gets wider usage. If no one else reports a 
problem, then I am certainly wrong.

Larry

[toc] | [prev] | [next] | [standalone]

#1493480

On Wed, Sep 28, 2016 at 09:26:42PM -0500, Larry Finger wrote:
> By the time it gets slow, the CPU's cool, and one cannot see the temp just
> before that event happened.

Hmm, I would not expect the CPU to drop from 80 to 40 degrees in a few
seconds if the fan is not spinning.  I wouldn't even expect it if the
fan was spinning.  I would think at least 30 to 60 seconds if not more.

The only way I would think the temperature could change quickly would
be if the heatsink isn't even touching the CPU anymore so there is very
little material to hold the heat in the CPU.

> The reason I suspect a bug is that it fails with 4.8-rcX, but not with 4.7.
> Of course, it could be something subtle that slightly changes the heat load,
> which causes the CPU temp to be a little higher so that the effect is
> triggered.
> 
> I am reasonably confident that it is not a hardware problem, but we may have
> to wait until 4.8 is released and gets wider usage. If no one else reports a
> problem, then I am certainly wrong.

Well hard to reproduce bugs are always really annoying.

This old bug sounds a lot like what you are seeing:
https://bugzilla.redhat.com/show_bug.cgi?id=924570
and it links to this:
https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.6-Thermal-Updates

Apparently turning off turbo boost seems to stop the problem for a lot
of people in that case.  Doesn't explain why it started happening
recently.  And of course that may have been a different problem in
the past.

-- 
Len Sorensen

[toc] | [prev] | [next] | [standalone]

#1491557

On Mon, 2016-09-26 at 23:06 +0200, Rafael J. Wysocki wrote:
> On Monday, September 26, 2016 11:15:45 AM Larry Finger wrote:

[...]

> > I have been reluctant to post my entire .config on the list. It is
> > available at 
> > http://pastebin.com/aMZaAKwL.
> 
> If the governor is ondemand, the driver is acpi-cpufreq, most likely.
> 
> How do you measure the frequency?
> 
Also
When you get into this situation, please dump:
# cat /sys/devices/system/cpu/cpufreq/policy?/scaling_max_freq
# cat /sys/devices/system/cpu/intel_pstate/*


Thanks,
Srinivas

[toc] | [prev] | [next] | [standalone]

#1491562

On Mon, Sep 26, 2016 at 11:26 PM, Srinivas Pandruvada
<srinivas.pandruvada@linux.intel.com> wrote:
> On Mon, 2016-09-26 at 23:06 +0200, Rafael J. Wysocki wrote:
>> On Monday, September 26, 2016 11:15:45 AM Larry Finger wrote:
>
> [...]
>
>> > I have been reluctant to post my entire .config on the list. It is
>> > available at
>> > http://pastebin.com/aMZaAKwL.
>>
>> If the governor is ondemand, the driver is acpi-cpufreq, most likely.
>>
>> How do you measure the frequency?
>>
> Also
> When you get into this situation, please dump:
> # cat /sys/devices/system/cpu/cpufreq/policy?/scaling_max_freq
> # cat /sys/devices/system/cpu/intel_pstate/*

The driver is not intel_pstate.

Thanks,
Rafael

[toc] | [prev] | [next] | [standalone]

#1491565

On Mon, 2016-09-26 at 23:30 +0200, Rafael J. Wysocki wrote:
> On Mon, Sep 26, 2016 at 11:26 PM, Srinivas Pandruvada
> <srinivas.pandruvada@linux.intel.com> wrote:
> > 
> > On Mon, 2016-09-26 at 23:06 +0200, Rafael J. Wysocki wrote:
> > > 
> > > On Monday, September 26, 2016 11:15:45 AM Larry Finger wrote:
> > 
> > [...]
> > 
> > > 
> > > > 
> > > > I have been reluctant to post my entire .config on the list. It
> > > > is
> > > > available at
> > > > http://pastebin.com/aMZaAKwL.
> > > 
> > > If the governor is ondemand, the driver is acpi-cpufreq, most
> > > likely.
> > > 
> > > How do you measure the frequency?
> > > 
> > Also
> > When you get into this situation, please dump:
> > # cat /sys/devices/system/cpu/cpufreq/policy?/scaling_max_freq
> > # cat /sys/devices/system/cpu/intel_pstate/*
> 
> The driver is not intel_pstate.
I guessed from
CONFIG_X86_INTEL_PSTATE=y
and
Frequency is not 400 but something like 396.130


Thanks,
Srinivas

[toc] | [prev] | [next] | [standalone]

#1491566

On Mon, Sep 26, 2016 at 11:41 PM, Srinivas Pandruvada
<srinivas.pandruvada@linux.intel.com> wrote:
> On Mon, 2016-09-26 at 23:30 +0200, Rafael J. Wysocki wrote:
>> On Mon, Sep 26, 2016 at 11:26 PM, Srinivas Pandruvada
>> <srinivas.pandruvada@linux.intel.com> wrote:
>> >
>> > On Mon, 2016-09-26 at 23:06 +0200, Rafael J. Wysocki wrote:
>> > >
>> > > On Monday, September 26, 2016 11:15:45 AM Larry Finger wrote:
>> >
>> > [...]
>> >
>> > >
>> > > >
>> > > > I have been reluctant to post my entire .config on the list. It
>> > > > is
>> > > > available at
>> > > > http://pastebin.com/aMZaAKwL.
>> > >
>> > > If the governor is ondemand, the driver is acpi-cpufreq, most
>> > > likely.
>> > >
>> > > How do you measure the frequency?
>> > >
>> > Also
>> > When you get into this situation, please dump:
>> > # cat /sys/devices/system/cpu/cpufreq/policy?/scaling_max_freq
>> > # cat /sys/devices/system/cpu/intel_pstate/*
>>
>> The driver is not intel_pstate.
> I guessed from
> CONFIG_X86_INTEL_PSTATE=y
> and
> Frequency is not 400 but something like 396.130

Ah.  Good catch!

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web