Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1553681 > unrolled thread

Re: [linux-sunxi] [PATCH] clk: sunxi-ng: fix PLL_CPUX adjusting on H3

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [linux-sunxi] [PATCH] clk: sunxi-ng: fix PLL_CPUX adjusting on H3 Ondřej Jirman <megous@megous.com> - 2017-01-07 17:00 +0100
    Re: [linux-sunxi] [PATCH] clk: sunxi-ng: fix PLL_CPUX adjusting on H3 Maxime Ripard <maxime.ripard@free-electrons.com> - 2017-01-09 11:10 +0100
      Re: [linux-sunxi] [PATCH] clk: sunxi-ng: fix PLL_CPUX adjusting on H3 Ondřej Jirman <megous@megous.com> - 2017-01-09 16:00 +0100
        Re: [linux-sunxi] [PATCH] clk: sunxi-ng: fix PLL_CPUX adjusting on H3 Maxime Ripard <maxime.ripard@free-electrons.com> - 2017-01-16 17:50 +0100
          Re: [linux-sunxi] [PATCH] clk: sunxi-ng: fix PLL_CPUX adjusting on H3 Ondřej Jirman <megous@megous.com> - 2017-01-16 18:00 +0100

#1553681 — Re: [linux-sunxi] [PATCH] clk: sunxi-ng: fix PLL_CPUX adjusting on H3

[Multipart message — attachments visible in raw view] — view raw

Maxime,

Dne 25.11.2016 v 01:28 megous@megous.com napsal(a):
> From: Ondrej Jirman <megous@megous.com>
> 
> When adjusting PLL_CPUX on H3, the PLL is temporarily driven
> too high, and the system becomes unstable (oopses or hangs).
> 
> Add a notifier to avoid this situation by temporarily switching
> to a known stable 24 MHz oscillator.

I have done more thorough testing on H3 and this approach with switching
to 24MHz oscillator does not work. Motivation being that my Orange Pi
One still gets lockups even with this patch under certain circumstances.

So I have created a small test program for CPUS (additional OpenRISC CPU
on the SoC) which randomly changes PLL_CPUX settings while main CPU is
running a loop that sends messages to CPUS via msgbox.

Assumption being that while CPUS is successfully receiving messages via
msgbox, the main CPU didn't lock up, yet.

With this I am able to quickly and thoroughly test various PLL_CPUX
change and factor selection algorithms.

Results are that bypassing CPUX clock by switching to 24 MHz oscillator
does not work at all. Main CPU locks up in about 1 second into the test.
Don't ask me why.

What works is selecting NKMP factors so that M is always 1 and P is
anything other than /1 only for frequencies under 288MHz. As mandated by
the H3 datasheet. Mainline ccu_nkmp_find_best doesn't respect these
conditions. With that I can change CPUX frequencies randomly 20x a
second so far indefinitely without the main CPU ever locking up.

Please drop or revert this patch. It is not a correct approach to the
problem. I'd suggest dropping the entire clock notifier mechanism, too,
unless it can be proven to work reliably. The bypass makes some
intuitive sense, but for some reason it doesn't work in practice (on H3
at least).

Aside from this, uboot also needs to be changed to set that it uses M
and P factors correctly.

Whatever else I try, I always hit lockups sooner or later with the test
program. I tried 24MHz bypass and staged application of multipliers and
dividers as discussed before.

I'll send a proper patch for nkmp clock driver and u-boot later.

regards,
  o.

> Signed-off-by: Ondrej Jirman <megous@megous.com>
> Tested-by: Lutz Sammer <johns98@gmx.net>
> ---
>  drivers/clk/sunxi-ng/ccu-sun8i-h3.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/drivers/clk/sunxi-ng/ccu-sun8i-h3.c b/drivers/clk/sunxi-ng/ccu-sun8i-h3.c
> index 614d47c..cf266c9 100644
> --- a/drivers/clk/sunxi-ng/ccu-sun8i-h3.c
> +++ b/drivers/clk/sunxi-ng/ccu-sun8i-h3.c
> @@ -809,6 +809,13 @@ static const struct sunxi_ccu_desc sun8i_h3_ccu_desc = {
>  	.num_resets	= ARRAY_SIZE(sun8i_h3_ccu_resets),
>  };
>  
> +static struct ccu_mux_nb sun8i_h3_cpu_nb = {
> +	.common		= &cpux_clk.common,
> +	.cm		= &cpux_clk.mux,
> +	.delay_us	= 1, /* > 8 clock cycles at 24 MHz */
> +	.bypass_index	= 1, /* index of 24 MHz oscillator */
> +};
> +
>  static void __init sun8i_h3_ccu_setup(struct device_node *node)
>  {
>  	void __iomem *reg;
> @@ -827,6 +834,9 @@ static void __init sun8i_h3_ccu_setup(struct device_node *node)
>  	writel(val | (3 << 16), reg + SUN8I_H3_PLL_AUDIO_REG);
>  
>  	sunxi_ccu_probe(node, reg, &sun8i_h3_ccu_desc);
> +
> +	ccu_mux_notifier_register(pll_cpux_clk.common.hw.clk,
> +				  &sun8i_h3_cpu_nb);
>  }
>  CLK_OF_DECLARE(sun8i_h3_ccu, "allwinner,sun8i-h3-ccu",
>  	       sun8i_h3_ccu_setup);
> 

[toc] | [next] | [standalone]

#1554192

[Multipart message — attachments visible in raw view] — view raw

On Sat, Jan 07, 2017 at 04:49:18PM +0100, Ondřej Jirman wrote:
> Maxime,
> 
> Dne 25.11.2016 v 01:28 megous@megous.com napsal(a):
> > From: Ondrej Jirman <megous@megous.com>
> > 
> > When adjusting PLL_CPUX on H3, the PLL is temporarily driven
> > too high, and the system becomes unstable (oopses or hangs).
> > 
> > Add a notifier to avoid this situation by temporarily switching
> > to a known stable 24 MHz oscillator.
> 
> I have done more thorough testing on H3 and this approach with switching
> to 24MHz oscillator does not work. Motivation being that my Orange Pi
> One still gets lockups even with this patch under certain circumstances.
> 
> So I have created a small test program for CPUS (additional OpenRISC CPU
> on the SoC) which randomly changes PLL_CPUX settings while main CPU is
> running a loop that sends messages to CPUS via msgbox.
> 
> Assumption being that while CPUS is successfully receiving messages via
> msgbox, the main CPU didn't lock up, yet.
> 
> With this I am able to quickly and thoroughly test various PLL_CPUX
> change and factor selection algorithms.
> 
> Results are that bypassing CPUX clock by switching to 24 MHz oscillator
> does not work at all. Main CPU locks up in about 1 second into the test.
> Don't ask me why.

You mean that you are changing the frequency behind Linux' back? That
won't work. There's more to cpufreq than just changing the frequency,
but also adusting the number of loops per jiffy for the new frequency
for example. I don't really expect that setup to work even on a
perfectly stable system. CPUFreq *has* to be involved, otherwise, that
alone might introduce bugs, and you cannot draw any conclusions
anymore.

> What works is selecting NKMP factors so that M is always 1 and P is
> anything other than /1 only for frequencies under 288MHz. As mandated by
> the H3 datasheet. Mainline ccu_nkmp_find_best doesn't respect these
> conditions. With that I can change CPUX frequencies randomly 20x a
> second so far indefinitely without the main CPU ever locking up.
> 
> Please drop or revert this patch. It is not a correct approach to the
> problem. I'd suggest dropping the entire clock notifier mechanism, too,
> unless it can be proven to work reliably.

It has been proven to work reliably on a number of other SoCs.

Thanks!
Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[toc] | [prev] | [next] | [standalone]

#1554383

Dne 9.1.2017 v 10:59 Maxime Ripard napsal(a):
> On Sat, Jan 07, 2017 at 04:49:18PM +0100, Ondřej Jirman wrote:
>> Maxime,
>>
>> Dne 25.11.2016 v 01:28 megous@megous.com napsal(a):
>>> From: Ondrej Jirman <megous@megous.com>
>>>
>>> When adjusting PLL_CPUX on H3, the PLL is temporarily driven
>>> too high, and the system becomes unstable (oopses or hangs).
>>>
>>> Add a notifier to avoid this situation by temporarily switching
>>> to a known stable 24 MHz oscillator.
>>
>> I have done more thorough testing on H3 and this approach with switching
>> to 24MHz oscillator does not work. Motivation being that my Orange Pi
>> One still gets lockups even with this patch under certain circumstances.
>>
>> So I have created a small test program for CPUS (additional OpenRISC CPU
>> on the SoC) which randomly changes PLL_CPUX settings while main CPU is
>> running a loop that sends messages to CPUS via msgbox.
>>
>> Assumption being that while CPUS is successfully receiving messages via
>> msgbox, the main CPU didn't lock up, yet.
>>
>> With this I am able to quickly and thoroughly test various PLL_CPUX
>> change and factor selection algorithms.
>>
>> Results are that bypassing CPUX clock by switching to 24 MHz oscillator
>> does not work at all. Main CPU locks up in about 1 second into the test.
>> Don't ask me why.
> 
> You mean that you are changing the frequency behind Linux' back? That
> won't work. There's more to cpufreq than just changing the frequency,
> but also adusting the number of loops per jiffy for the new frequency
> for example. I don't really expect that setup to work even on a
> perfectly stable system. CPUFreq *has* to be involved, otherwise, that
> alone might introduce bugs, and you cannot draw any conclusions
> anymore.

No, this has nothing to do with linux. I'm not running linux for this
test. I'm running a small program on CPUS (Open RISC CPU) on the SoC
loaded using FEL from USB.

The main cpu is just pushing messages into msgbox in a loop, so that
CPUS can determine that the main CPU is still running ok and give
feedback to me over UART. Not even DRAM is involved. The programs are
running from SRAM.

This is the most direct test of PLL change stability that can be done on
this SoC regardless of the OS. Not even CPU voltage switching is
involved. I just set the maximum voltage and fiddle with CPU_PLL
frequencies randomly, while waiting for the main CPU to lock up.

It does lock up quickly with mainline ccu_nkmp_find_best algorithm for
finding factors.

Even with linux kernel, it breaks. It's just more difficult to hit the
right conditions. I got oops only right after boot when running cpuburn
to trigger thermal_zone issued OPP change, if I first run some cpupower
commands. That's why I wrote this program to stress test various CPU_PLL
change/factor selection algorithms independently of everything else, to
get more predictable and quicker testing results.

>> What works is selecting NKMP factors so that M is always 1 and P is
>> anything other than /1 only for frequencies under 288MHz. As mandated by
>> the H3 datasheet. Mainline ccu_nkmp_find_best doesn't respect these
>> conditions. With that I can change CPUX frequencies randomly 20x a
>> second so far indefinitely without the main CPU ever locking up.
>>
>> Please drop or revert this patch. It is not a correct approach to the
>> problem. I'd suggest dropping the entire clock notifier mechanism, too,
>> unless it can be proven to work reliably.
> 
> It has been proven to work reliably on a number of other SoCs.

Unless it was stress tested like this with randomy changed settings, I
doubt you can call it reliable. It may just be very hard to hit the
issue on linux with particular OPP/thermal zone configuration. That's
because the issue is dependent on before and after NKMP values. People
may have just been lucky so far.

regards,
  o.

> Thanks!
> Maxime
> 

[toc] | [prev] | [next] | [standalone]

#1559888

[Multipart message — attachments visible in raw view] — view raw

Hi Ondrej,

Sorry for the late reply,

On Mon, Jan 09, 2017 at 03:50:42PM +0100, Ondřej Jirman wrote:
> Dne 9.1.2017 v 10:59 Maxime Ripard napsal(a):
> > On Sat, Jan 07, 2017 at 04:49:18PM +0100, Ondřej Jirman wrote:
> >> Maxime,
> >>
> >> Dne 25.11.2016 v 01:28 megous@megous.com napsal(a):
> >>> From: Ondrej Jirman <megous@megous.com>
> >>>
> >>> When adjusting PLL_CPUX on H3, the PLL is temporarily driven
> >>> too high, and the system becomes unstable (oopses or hangs).
> >>>
> >>> Add a notifier to avoid this situation by temporarily switching
> >>> to a known stable 24 MHz oscillator.
> >>
> >> I have done more thorough testing on H3 and this approach with switching
> >> to 24MHz oscillator does not work. Motivation being that my Orange Pi
> >> One still gets lockups even with this patch under certain circumstances.
> >>
> >> So I have created a small test program for CPUS (additional OpenRISC CPU
> >> on the SoC) which randomly changes PLL_CPUX settings while main CPU is
> >> running a loop that sends messages to CPUS via msgbox.
> >>
> >> Assumption being that while CPUS is successfully receiving messages via
> >> msgbox, the main CPU didn't lock up, yet.
> >>
> >> With this I am able to quickly and thoroughly test various PLL_CPUX
> >> change and factor selection algorithms.
> >>
> >> Results are that bypassing CPUX clock by switching to 24 MHz oscillator
> >> does not work at all. Main CPU locks up in about 1 second into the test.
> >> Don't ask me why.
> > 
> > You mean that you are changing the frequency behind Linux' back? That
> > won't work. There's more to cpufreq than just changing the frequency,
> > but also adusting the number of loops per jiffy for the new frequency
> > for example. I don't really expect that setup to work even on a
> > perfectly stable system. CPUFreq *has* to be involved, otherwise, that
> > alone might introduce bugs, and you cannot draw any conclusions
> > anymore.
> 
> No, this has nothing to do with linux. I'm not running linux for this
> test. I'm running a small program on CPUS (Open RISC CPU) on the SoC
> loaded using FEL from USB.
> 
> The main cpu is just pushing messages into msgbox in a loop, so that
> CPUS can determine that the main CPU is still running ok and give
> feedback to me over UART. Not even DRAM is involved. The programs are
> running from SRAM.
> 
> This is the most direct test of PLL change stability that can be done on
> this SoC regardless of the OS. Not even CPU voltage switching is
> involved. I just set the maximum voltage and fiddle with CPU_PLL
> frequencies randomly, while waiting for the main CPU to lock up.

Ok.

> It does lock up quickly with mainline ccu_nkmp_find_best algorithm
> for finding factors.
> 
> Even with linux kernel, it breaks. It's just more difficult to hit the
> right conditions. I got oops only right after boot when running cpuburn
> to trigger thermal_zone issued OPP change, if I first run some cpupower
> commands. That's why I wrote this program to stress test various CPU_PLL
> change/factor selection algorithms independently of everything else, to
> get more predictable and quicker testing results.

Understood. Do you have the code available somewhere?

> >> What works is selecting NKMP factors so that M is always 1 and P is
> >> anything other than /1 only for frequencies under 288MHz. As mandated by
> >> the H3 datasheet. Mainline ccu_nkmp_find_best doesn't respect these
> >> conditions. With that I can change CPUX frequencies randomly 20x a
> >> second so far indefinitely without the main CPU ever locking up.
> >>
> >> Please drop or revert this patch. It is not a correct approach to the
> >> problem. I'd suggest dropping the entire clock notifier mechanism, too,
> >> unless it can be proven to work reliably.
> > 
> > It has been proven to work reliably on a number of other SoCs.
> 
> Unless it was stress tested like this with randomy changed settings, I
> doubt you can call it reliable. It may just be very hard to hit the
> issue on linux with particular OPP/thermal zone configuration. That's
> because the issue is dependent on before and after NKMP values. People
> may have just been lucky so far.

Yes, or maybe we just have OPPs that just don't trigger a low enough P
factor.

There's no rush anyway, the H3 cpufreq support is not enabled at the
moment, so that code basically does nothing for the moment.

What's your current plan to fix that? I guess the easiest (and most
likely to be reusable) would be to allow for clock tables, instead of
using the generic approach. We might have some other clocks (like
audio or video) that would need such a precise tuning in the future
too.

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[toc] | [prev] | [next] | [standalone]

#1559892

Hi Maxime,

Dne 16.1.2017 v 17:43 Maxime Ripard napsal(a):
>> It does lock up quickly with mainline ccu_nkmp_find_best algorithm
>> for finding factors.
>>
>> Even with linux kernel, it breaks. It's just more difficult to hit the
>> right conditions. I got oops only right after boot when running cpuburn
>> to trigger thermal_zone issued OPP change, if I first run some cpupower
>> commands. That's why I wrote this program to stress test various CPU_PLL
>> change/factor selection algorithms independently of everything else, to
>> get more predictable and quicker testing results.
> 
> Understood. Do you have the code available somewhere?

It's a bit messy, but it's here:

https://github.com/megous/h3-firmware/blob/master/main.c

>>>> What works is selecting NKMP factors so that M is always 1 and P is
>>>> anything other than /1 only for frequencies under 288MHz. As mandated by
>>>> the H3 datasheet. Mainline ccu_nkmp_find_best doesn't respect these
>>>> conditions. With that I can change CPUX frequencies randomly 20x a
>>>> second so far indefinitely without the main CPU ever locking up.
>>>>
>>>> Please drop or revert this patch. It is not a correct approach to the
>>>> problem. I'd suggest dropping the entire clock notifier mechanism, too,
>>>> unless it can be proven to work reliably.
>>>
>>> It has been proven to work reliably on a number of other SoCs.
>>
>> Unless it was stress tested like this with randomy changed settings, I
>> doubt you can call it reliable. It may just be very hard to hit the
>> issue on linux with particular OPP/thermal zone configuration. That's
>> because the issue is dependent on before and after NKMP values. People
>> may have just been lucky so far.
> 
> Yes, or maybe we just have OPPs that just don't trigger a low enough P
> factor.
> 
> There's no rush anyway, the H3 cpufreq support is not enabled at the
> moment, so that code basically does nothing for the moment.
> 
> What's your current plan to fix that? I guess the easiest (and most
> likely to be reusable) would be to allow for clock tables, instead of
> using the generic approach. We might have some other clocks (like
> audio or video) that would need such a precise tuning in the future
> too.

My proposed solution is this for M factor (H3 specific solution):

https://github.com/megous/linux/commit/88be3d421e958579026135d8acec4b3983958738

and this for P factor:

https://github.com/megous/linux/commit/d7f274ed0c13fa9b4099445cb6bf9b2f8f2cfa8a

Perhaps it should be configurable if the P limitation is not universal
for all NKMP clocks. But I haven't read all the datasheets.

It was verified with the above arisc firmware to work reliably, so it
should be enough. It is simpler and the factor table should not be
necessary in this case.

regards,
  o.

> Maxime
> 

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web