Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1510585 > unrolled thread

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PREEMPT-RT] Oops in rapl_cpu_prepare() "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-10-27 21:10 +0200
    Re: [PREEMPT-RT] Oops in rapl_cpu_prepare() Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-10-28 10:10 +0200
      Re: [PREEMPT-RT] Oops in rapl_cpu_prepare() "M. Vefa Bicakci" <m.v.b@runbox.com> - 2016-11-01 11:40 +0100
        Re: [PREEMPT-RT] Oops in rapl_cpu_prepare() Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-11-02 18:30 +0100
          Re: [PREEMPT-RT] Oops in rapl_cpu_prepare() "M. Vefa Bicakci" <m.v.b@runbox.com> - 2016-11-03 19:30 +0100
      Re: [PREEMPT-RT] Oops in rapl_cpu_prepare() "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-11-02 10:40 +0100
        Re: [PREEMPT-RT] Oops in rapl_cpu_prepare() Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-11-02 11:00 +0100

#1510585 — Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

On 10/25/2016 08:22 AM, Sebastian Andrzej Siewior wrote:
> On 2016-10-21 17:03:56 [-0400], Charles (Chas) Williams wrote:
>> 	[    3.107126] init_rapl_pmus: maxpkg 4
> there! vmware bug. It probably worked by chance.

Yes, the behavior is a bit random.

> I assume "init_rapl_pmus: maxpkg 4" is from init_rapl_pmus() returning
> topology_max_packages(). So it says 4 but then returns 65535 for CPU 2
> and 3. That -1 comes probably from topology_update_package_map(). Could
> you please send a complete boot log and try the following patch? This
> one should fix your boot problem and disable RAPL if the info is
> invalid.

But sometimes the topology info is correct and if I get lucky, the
package id could be valid for all the CPU's.  Given the behavior,
I have seen so far it makes me thing the RAPL isn't being emulated.
So even if I did boot onto a "valid" set of cores, would I always be
certain that I will be on those cores?

> diff --git a/arch/x86/events/intel/rapl.c b/arch/x86/events/intel/rapl.c
> index 0a535cea8ff3..f5d85f2853d7 100644
> --- a/arch/x86/events/intel/rapl.c
> +++ b/arch/x86/events/intel/rapl.c
> @@ -682,6 +682,15 @@ static int __init init_rapl_pmus(void)
>  {
>  	int maxpkg = topology_max_packages();
>  	size_t size;
> +	unsigned int cpu;
> +
> +	for_each_possible_cpu(cpu) {
> +		if (topology_logical_package_id(cpu) >= maxpkg) {
> +			pr_err("rapl pmu error: max package: %u but CPU%d belongs to %u\n",
> +			       maxpkg, cpu, topology_logical_package_id(cpu));
> +			return -EINVAL;
> +		}
> +	}
>
>  	size = sizeof(*rapl_pmus) + maxpkg * sizeof(struct rapl_pmu *);
>  	rapl_pmus = kzalloc(size, GFP_KERNEL);

Per your request in your next email:

>One thing I forgot to ask: Could you please check if you get the same
>pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
>rework).

Our previous kernel was 4.4, and didn't use the logical package id:

         /* check if phys_is is already covered */
         for_each_cpu(i, &rapl_cpu_mask) {
                 if (phys_id == topology_physical_package_id(i))
                         return;

[toc] | [next] | [standalone]

#1510928

On 2016-10-27 15:00:32 [-0400], Charles (Chas) Williams wrote:
> > I assume "init_rapl_pmus: maxpkg 4" is from init_rapl_pmus() returning
> > topology_max_packages(). So it says 4 but then returns 65535 for CPU 2
> > and 3. That -1 comes probably from topology_update_package_map(). Could
> > you please send a complete boot log and try the following patch? This
> > one should fix your boot problem and disable RAPL if the info is
> > invalid.
> 
> But sometimes the topology info is correct and if I get lucky, the
> package id could be valid for all the CPU's.  Given the behavior,
> I have seen so far it makes me thing the RAPL isn't being emulated.
> So even if I did boot onto a "valid" set of cores, would I always be
> certain that I will be on those cores?

I don't what vmware does here. Nor do they ship source to check. So if
you have a big HW box with say two packages, it might make sense to give
this information to the guest _if_ the CPUs are pinned and the guest
never migrates.

> Per your request in your next email:
> 
> > One thing I forgot to ask: Could you please check if you get the same
> > pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
> > rework).
> 
> Our previous kernel was 4.4, and didn't use the logical package id:
I see.

Did the patch I sent fixed it for you and were you not able to test?

Sebastian

[toc] | [prev] | [next] | [standalone]

#1513208

> On 2016-10-27 15:00:32 [-0400], Charles (Chas) Williams wrote:
>>
>> [snip]
>>
>> But sometimes the topology info is correct and if I get lucky, the
>> package id could be valid for all the CPU's.  Given the behavior,
>> I have seen so far it makes me thing the RAPL isn't being emulated.
>> So even if I did boot onto a "valid" set of cores, would I always be
>> certain that I will be on those cores?
> 
> I don't what vmware does here. Nor do they ship source to check. So if
> you have a big HW box with say two packages, it might make sense to give
> this information to the guest _if_ the CPUs are pinned and the guest
> never migrates.
> 
>> Per your request in your next email:
>> 
>> > One thing I forgot to ask: Could you please check if you get the same
>> > pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
>> > rework).
>> 
>> Our previous kernel was 4.4, and didn't use the logical package id:
>
> I see.
> 
> Did the patch I sent fixed it for you and were you not able to test?

Hello Sebastian,

The patch fixes the kernel oops for me.

I am using a custom 4.8.5-based kernel on Qubes OS R3.2, which is based
on Xen 4.6.3. Apparently, Xen also has a similar bug/flaw/quirk regarding
the allocation of package identifiers for the virtual CPUs.

Prior to your patch, my Xen-based virtual machines would intermittently
crash most of the time at boot-up with the backtrace reported by Charles.
Due to this, I was under the impression that this is a subtle race
condition.

With your patch, the virtual machines boot-up successfully, all the time.
Here are the relevant excerpts from dmesg:

=== 8< ===
[    0.263936] RAPL PMU: rapl pmu error: max package: 1 but CPU0 belongs to 65535
...
[    2.213669] intel_rapl: Found RAPL domain package
[    2.213689] intel_rapl: Found RAPL domain core
[    2.216337] intel_rapl: Found RAPL domain uncore
[    2.216370] intel_rapl: RAPL package 0 domain package locked by BIOS
=== >8 ===

Thank you,

Vefa

Please note: I am not subscribed to the Linux kernel mailing list, so
I had to manually construct the headers of this reply with the proper
In-Reply-To and References values (which were extracted from marc.info).
As a result, this e-mail may not show up as a reply to your earlier
conversation with Charles.

[toc] | [prev] | [next] | [standalone]

#1514040

On 2016-11-01 13:15:53 [+0300], M. Vefa Bicakci wrote:
> Hello Sebastian,
Hi,

> The patch fixes the kernel oops for me.
> 
> I am using a custom 4.8.5-based kernel on Qubes OS R3.2, which is based
> on Xen 4.6.3. Apparently, Xen also has a similar bug/flaw/quirk regarding
> the allocation of package identifiers for the virtual CPUs.
> 
> Prior to your patch, my Xen-based virtual machines would intermittently
> crash most of the time at boot-up with the backtrace reported by Charles.
> Due to this, I was under the impression that this is a subtle race
> condition.

how hard is it to get such a xen setup up and running?

Sebastian

[toc] | [prev] | [next] | [standalone]

#1514771

On 11/02/2016 08:23 PM, Sebastian Andrzej Siewior wrote:
> On 2016-11-01 13:15:53 [+0300], M. Vefa Bicakci wrote:
>> Hello Sebastian,
>
> Hi,
> 
>> The patch fixes the kernel oops for me.
>>
>> I am using a custom 4.8.5-based kernel on Qubes OS R3.2, which is based
>> on Xen 4.6.3. Apparently, Xen also has a similar bug/flaw/quirk regarding
>> the allocation of package identifiers for the virtual CPUs.
>>
>> Prior to your patch, my Xen-based virtual machines would intermittently
>> crash most of the time at boot-up with the backtrace reported by Charles.
>> Due to this, I was under the impression that this is a subtle race
>> condition.
> 
> how hard is it to get such a xen setup up and running?

Hello Sebastian,

Sorry about my late reply!

The set-up I use is a bit involved/complicated. To replicate it, you
would need to install Qubes OS R3.2 (assuming that you have compatible
hardware with a lot of RAM) and then build a custom 4.8.y-based kernel
with a set of cherry-picked commits. After installing this kernel in
dom0 with dnf or rpm, you would need to run:

  # Generate a domU initrd and copy the kernel image and the generated
  # initrd to Qubes OS's domU kernel directory (/var/lib/qubes/...)
  $ sudo /usr/sbin/qubes-prepare-vm-kernel <kernel_version>

  # From now on, use kernel_version when starting domU instances.
  $ qubes-prefs -s default-kernel <kernel_version>

Afterwards, starting a domU (i.e., AppVM in Qubes OS terminology) should
exhibit the issue in question related to RAPL:

  $ sudo truncate -s0 /var/log/xen/console/guest-<app_vm_name>.log
  $ qvm-start --debug <app_vm_name>
  $ cat /var/log/xen/console/guest-<app_vm_name>.log

As you may appreciate, the set-up is a bit involved. Nevertheless, in
case you would like to replicate my set-up, I can try to publish my
linux-4.8.y-based git branch so that you can build a similar kernel as
the one I use.

Vefa

[toc] | [prev] | [next] | [standalone]

#1513779

On 10/28/2016 04:03 AM, Sebastian Andrzej Siewior wrote:
> On 2016-10-27 15:00:32 [-0400], Charles (Chas) Williams wrote:
>>> I assume "init_rapl_pmus: maxpkg 4" is from init_rapl_pmus() returning
>>> topology_max_packages(). So it says 4 but then returns 65535 for CPU 2
>>> and 3. That -1 comes probably from topology_update_package_map(). Could
>>> you please send a complete boot log and try the following patch? This
>>> one should fix your boot problem and disable RAPL if the info is
>>> invalid.
>>
>> But sometimes the topology info is correct and if I get lucky, the
>> package id could be valid for all the CPU's.  Given the behavior,
>> I have seen so far it makes me thing the RAPL isn't being emulated.
>> So even if I did boot onto a "valid" set of cores, would I always be
>> certain that I will be on those cores?
>
> I don't what vmware does here. Nor do they ship source to check. So if
> you have a big HW box with say two packages, it might make sense to give
> this information to the guest _if_ the CPUs are pinned and the guest
> never migrates.

Yes, I agree _if_.  That's why it simply isn't clear to me that we should
attempt do any RAPL at all for VMWare.  The current behavior doesn't seem
to make sense and I don't expect it to suddenly start acting reasonable.
Since I don't understand why some package id's are valid and others
are not, I would prefer not to trust any of the information as far as
enabling/disabling the RAPL monitoring.

>
>> Per your request in your next email:
>>
>>> One thing I forgot to ask: Could you please check if you get the same
>>> pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
>>> rework).
>>
>> Our previous kernel was 4.4, and didn't use the logical package id:
> I see.
>
> Did the patch I sent fixed it for you and were you not able to test?

Yes, it does prevent RAPL from starting and loading.  From the boot log:

[    2.711481] RAPL PMU: rapl pmu error: max package: 4 but CPU2 belongs to 65535
[    2.711639] rapl pmu error: max package: 4 but CPU2 belongs to 65535

This was consistent across several reboots.  I poked around in the
VM settings.  Apparently this guest is configured for four virtual
sockets with one core per socket.  Testing with two virtual sockets,
one core per socket:

[    2.163177] RAPL PMU: rapl pmu error: max package: 2 but CPU1 belongs to 65535
[    2.163304] rapl pmu error: max package: 2 but CPU1 belongs to 65535

Booting with 1 virtual socket, 1 core per socket:

[    1.750311] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer
[    1.750312] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules
[    1.750313] RAPL PMU: hw unit of domain package 2^-0 Joules
[    1.750314] RAPL PMU: hw unit of domain dram 2^-0 Joules

Booting with 1 virtual socket, 4 cores per socket:

[    3.527298] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer
[    3.527302] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules
[    3.527304] RAPL PMU: hw unit of domain package 2^-0 Joules
[    3.527307] RAPL PMU: hw unit of domain dram 2^-0 Joules

So, it looks like VMWare tends to always get something wrong if you have
more than one virtual socket.  The above behavior was consistent across
several reboots.

[toc] | [prev] | [next] | [standalone]

#1513782

On 2016-11-02 05:16:03 [-0400], Charles (Chas) Williams wrote:
> Yes, it does prevent RAPL from starting and loading.  From the boot log:
please send the whole bootlog. offlist if you want.

Sebastian

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web